Security Evaluations

Starting in the 1980s, the US government established a program to evaluate the security of computer operating systems. Since then, other governments have established programs. In the late 1990s, major governments agreed to recognize a Common Criteria for security product evaluations. Since then, the number of evaluations have skyrocketed.

The following figure summarizes the number of government endorsed security evaluations completed every year since the first was completed in 1984. The different colors represent different evaluation criteria, with "CC" representing today's Common Criteria.

 

Number of evaluations completed per year since their inception

Starting in 1999, I have occasionally run projects where I have tracked down every report I could find of a security product evaluation. My first project led to some preliminary results.

At the Last National Information Systems Security Conference (23rd NISSC) in October, 2000, I presented a paper (PDF) that surveyed the trends shown in the previous 16 years of formal computer security evaluations. I also produced a summary page of those results.

In 2006, I ran another survey that yielded the chart above and a paper (PDF) reviewing current trends. This was published in Information Systems Security (v. 16, n. 4) in 2007. The work was done with the help of several undergrads at the Univerisity of St. Thomas.

Other Observations

For additional insight, I'd suggest looking at Section 23.3.2 of Ross Anderson's book Security Engineering, which describes the process from the UK point of view. Ross isn't impressed with the way the process works in practice; while the process may be somewhat more stringent in the US, the US process simply produces different failure modes.

In the US, cryptographic products are certified under the FIPS 140 process, administered by the National Institute for Standards and Technology (NIST). Evaluation experts are quick to point out that the process and intentions are different between FIPS 140 and the Common Criteria. For the end user, they may appear to yield a similar result: a third-party assessment of a security device or product. In practice, different communities have different requirements. In the US and Canada, there is no substitute for an up-to-date FIPS 140 certification when we look at cryptographic products. Other countries or communities may acknowledge FIPS 140 certifications or they may require Common Criteria certifications.

In any case, security evaluations and certifications simply illustrate a form of due diligence. They do not guarantee the safety of a device or system. In 2009, for example, researchers found that many self-encrypting USB drives contained an identical, fatal security flaw. All of the drives had completed a FIPS 140 evaluation that did not highlight the failure.

Creative Commons License

This article by Rick Smith is licensed under a

Creative Commons Attribution 3.0 United States License.

Earlier evaluation notes

For a more recent view of this topic, visit my Security Evaluations page.

Date: 4/20/2005.

In 1999, I tracked down every report I could find of a product that had completed a published, formal security evaluation in accordance with trusted systems evaluation criteria. This led to some preliminary results. At the Last National Information Systems Security Conference (23rd NISSC) in October, 2000, I presented a paper (PDF) that surveyed the trends shown in the previous 16 years of formal computer security evaluations.

I collected all of my data in an Excel 97/98 spreadsheet that contained an entry for every evaluation I could find through the end of 1999. At the moment the spreadsheet includes the reported evaluations by the United States (TCSEC/NCSC and Common Criteria), United Kingdom (ITSEC and Common Criteria), Australia, and whatever evaluations were reported from Canada, France, and Germany by the US, UK, and Australian sites. I am not convinced that this is every published evaluation that took place, but it's every report I could find.

Key Observations

  • Product evaluations still take place: There was a big surge of evaluations in 1994 followed by a drop, but the number of evaluations never fell back to 1993 levels, and a few dozen still take place every year.
  • Few products actually bother with evaluation: Even the most limited estimates would suggest that hundreds of security products enter the market every year, yet it appears that no more than a tenth of them ever enter the evaluation process.
  • Flight from the US: Despite the fact that the US pioneered security evaluation and US companies arguably dominate the international market in information technology, most products, including US products, are evaluated overseas. The trend is increasing in this direction. This is probably because evaluations in the US tend to be far more expensive and time consuming.
  • Popularity of middle ratings: The trend in the vast majority of evaluations is to seek EAL 3 or EAL 4 ratings, or their rough equivalent in ITSEC ratings. Very few go for lower or higher ratings. The higher ratings appear to all go to government or military specific products.
  • Network security products (including access control) have come to dominate evaluations: originally, operating systems were the main type of product evaluated. Then the trusted DBMS criteria were published, and several DBMSes were evaluated. However, other products like access control products, data comm security devices, and networking devices like firewalls have come to dominate evaluations in recent years.

For additional insight, I'd suggest looking at Section 23.3.2 of Ross Anderson's book Security Engineering, which describes the process from the UK point of view. Ross isn't impressed with the way the process works in practice; while the process may be somewhat more stringent in the US, the US process simply produces different failure modes.

I would be thrilled if anyone interested in a weird research project would use my spreadsheet as a starting point to further analyze the phenomenon of security evaluations. There are probably other facts to be gleaned from the existing data, or other information to be collected. As noted, I stopped collecting data at the end of the last century.