In comments to both the Privacy and Civil Liberties Oversight Board and the Review Group on Intelligence and Communications Technologies, USACM recommended the use of an independent systems engineering analysis to review the design and operation of complex processes and systems. Our members spent some time further considering what should go into such an analysis, outlined the elements for one and described how such an analysis could be performed in the context of national surveillance programs. This description was submitted in a letter to the Director of National Intelligence earlier this month, and links such an analysis to several of the recommendations by the Review Group.
A systems engineering analysis will be focused on data flowing into, through and out of a system. Analyses should be concerned with the following system attributes:
- Auditability (including provenance)—the ability to associate system actions and data with their sources
- Confidentiality—the extent to which the system ensures that data is only accessible for authorized purposes and to those who are authorized
- Integrity—the resistance of data to unauthorized modification
- Data quality (of both collected and derived data)—the fitness of data for its intended purpose, including its accuracy and currency
- Functional completeness—the extent to which the set of functions addresses all objectives
- Functional correctness—the extent to which the system produces correct results with the necessary degree of precision
- Functional appropriateness—the extent to which the functions achieve objectives
The letter goes into further detail about how each of those attributes can be analyzed. By conducting such an analysis, USACM believes that relevant policy tradeoffs can be highlighted, hopefully before systems are deployed and problems arise.