Starting in the 1980s, the US government established a program to evaluate the security of computer operating systems. Since then, other governments have established programs. In the late 1990s, major governments agreed to recognize a Common Criteria for security product evaluations. Since then, the number of evaluations have skyrocketed.

The following figure summarizes the number of government endorsed security evaluations completed every year since the first was completed in 1984. The different colors represent different evaluation criteria, with “CC” representing today’s Common Criteria.

Starting in 1999, I have occasionally run projects where I have tracked down every report I could find of a security product evaluation. At the Last National Information Systems Security Conference (23rd NISSC) in October, 2000, I presented a paper (PDF) that surveyed the trends shown in the previous 16 years of formal computer security evaluations.

In 2006, I ran another survey that yielded the nearby chart and a paper (PDF) reviewing current trends. This was published in Information Systems Security (v. 16, n. 4) in 2007. The work was done with the help of several undergrads at the Univerisity of St. Thomas.

Other Observations

For additional insight, I’d suggest looking at Section 23.3.2 of Ross Anderson’s book Security Engineering, which describes the process from the UK point of view. Ross isn’t impressed with the way the process works in practice; while the process may be somewhat more stringent in the US, the US process simply produces different failure modes.

In the US, cryptographic products are certified under the FIPS 140 process, administered by theNational Institute for Standards and Technology (NIST). Evaluation experts are quick to point out that the process and intentions are different between FIPS 140 and the Common Criteria. For the end user, they may appear to yield a similar result: a third-party assessment of a security device or product. In practice, different communities have different requirements. In the US and Canada, there is no substitute for an up-to-date FIPS 140 certification when we look at cryptographic products. Other countries or communities may acknowledge FIPS 140 certifications or they may require Common Criteria certifications.

In any case, security evaluations and certifications simply illustrate a form of due diligence. They do not guarantee the safety of a device or system. In 2009, for example, researchers found that many self-encrypting USB drives contained an identical, fatal security flaw. All of the drives had completed a FIPS 140 evaluation that did not highlight the failure.

This article by Rick Smith is licensed under a

Creative Commons Attribution 3.0 United States License.