You are here

Earlier evaluation notes

For a more recent view of this topic, visit my Security Evaluations page.

Date: 4/20/2005.

In 1999, I tracked down every report I could find of a product that had completed a published, formal security evaluation in accordance with trusted systems evaluation criteria. This led to some preliminary results. At the Last National Information Systems Security Conference (23rd NISSC) in October, 2000, I presented a paper (PDF) that surveyed the trends shown in the previous 16 years of formal computer security evaluations.

I collected all of my data in an Excel 97/98 spreadsheet that contained an entry for every evaluation I could find through the end of 1999. At the moment the spreadsheet includes the reported evaluations by the United States (TCSEC/NCSC and Common Criteria), United Kingdom (ITSEC and Common Criteria), Australia, and whatever evaluations were reported from Canada, France, and Germany by the US, UK, and Australian sites. I am not convinced that this is every published evaluation that took place, but it's every report I could find.

Key Observations

  • Product evaluations still take place: There was a big surge of evaluations in 1994 followed by a drop, but the number of evaluations never fell back to 1993 levels, and a few dozen still take place every year.
  • Few products actually bother with evaluation: Even the most limited estimates would suggest that hundreds of security products enter the market every year, yet it appears that no more than a tenth of them ever enter the evaluation process.
  • Flight from the US: Despite the fact that the US pioneered security evaluation and US companies arguably dominate the international market in information technology, most products, including US products, are evaluated overseas. The trend is increasing in this direction. This is probably because evaluations in the US tend to be far more expensive and time consuming.
  • Popularity of middle ratings: The trend in the vast majority of evaluations is to seek EAL 3 or EAL 4 ratings, or their rough equivalent in ITSEC ratings. Very few go for lower or higher ratings. The higher ratings appear to all go to government or military specific products.
  • Network security products (including access control) have come to dominate evaluations: originally, operating systems were the main type of product evaluated. Then the trusted DBMS criteria were published, and several DBMSes were evaluated. However, other products like access control products, data comm security devices, and networking devices like firewalls have come to dominate evaluations in recent years.

For additional insight, I'd suggest looking at Section 23.3.2 of Ross Anderson's book Security Engineering, which describes the process from the UK point of view. Ross isn't impressed with the way the process works in practice; while the process may be somewhat more stringent in the US, the US process simply produces different failure modes.

I would be thrilled if anyone interested in a weird research project would use my spreadsheet as a starting point to further analyze the phenomenon of security evaluations. There are probably other facts to be gleaned from the existing data, or other information to be collected. As noted, I stopped collecting data at the end of the last century.

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer