Tag Archives: CERT

US-CERT: A disgrace to vulnerability statistics

Several people have asked OSVDB about their thoughts on the recent US-CERT Cyber Security Bulletin 2005 Summary. Producing vulnerability statistics is trivial to do. All it takes is your favorite data set, a few queries, and off you go. Producing meaningful and useful vulnerability statistics is a real chore. I’ve long been interested in vulnerability statistics, especially related to how they are used and the damage they cause. Creating and maintaining a useful statistics project has been on the OSVDB to-do list for some time, and I personally have not followed up with some folks that had the same interest (Ejovi et al). Until I see such statistics “done right”, I will of course continue to voice my opinion at other efforts.

Some of the following comments are in the form of questions, that have what I feel to be fairly obvious answers. At some point I will no doubt flesh out some of these ideas a bit better. Until then..

Information in the US-CERT Cyber Security Bulletin is a compilation and includes information published by outside sources, so the information should not be considered the result of US-CERT analysis.

Read: Our disclaimer so you can’t blame us for shoddy stats!

Software vulnerabilities are categorized in the appropriate section reflecting the operating system on which the vulnerability was reported; however, this does not mean that the vulnerability only affects the operating system reported since this information is obtained from open-source information.

What’s the point then? If you can’t categorize this software by specific operating system or at least distinguish which are multiple vendor (with better accuracy), is it really that helpful or useful?

Vulnerabilities
* Windows Operating System
* Unix/ Linux Operating System
* Multiple Operating System

A decade later and we’re still doing the “windows vs unix” thing? I guess after the last year of hype, Mac OS X still isn’t mature enough to get it’s own category, or is it now considered “Unix”? To answer that question, yes this list lumps it in with “Unix”, and yes it is curious they don’t break it out after SANS gave it it’s own entry on their list. Not even an “other” category to cover VMS, AS/400, routers or other devices?

Now, let’s look at the very first entry on the list out of curiosity: A Windows Operating System vulnerability titled “1Two Livre d’Or Input Validation Errors Permit Cross-Site Scripting”. This issue can be referenced by CVE 2005-1644, SecurityTracker 1013971, Bugtraq ID or OSVDB 16717. The CERT bulletin labels this issue “High Risk” which is baffling to say the least. A cross-site scripting issue in the error output of a very low distribution guestbook is high risk? What do they label a remote code execution vulnerability in windows? Looking at the first MSIE vuln on the list, it too is rated as “high risk”. Great, way to completely invalidate your entire risk rating.

OK, on to the fun part.. the statistics! Unfortunately, the bulletin is very lacking on wording, explanation, details or additional disclaimers. We get two very brief paragraphs, and the list of vulnerabilities that link to their summary entries. Very unfortunate. No, let me do one better. US-CERT, you are a disgrace to vulnerability databases. I can’t fathom why you even bothered to create this list, and why anyone in their right mind would actually use, reference or quote this trash. The only ‘statistics’ provided by this bulletin:

This bulletin provides a year-end summary of software vulnerabilities that were identified between January 2005 and December 2005. The information is presented only as a index with links to the US-CERT Cyber Security Bulletin the information was published in. There were 5198 reported vulnerabilities: 812 Windows operating system vulnerabilities; 2328 Unix/Linux operating vulnerabilities; and 2058 Multiple operating system vulnerabilities.

The simple truth is that 99.99% of the people reading this document will see the first two paragraphs and move on. They will not read every single entry on that page. That means they walk away with the idea that Unix/Linux is roughly 3x more vulnerable than Windows, when it simply is not the case. While scrolling down, I ran across a section of the Unix/Linux vulnerability list that jumped out at me:

# ImageMagick Photoshop Document Buffer Overflow (Updated)
# ImageMagick Photoshop Document Buffer Overflow (Updated)
# ImageMagick Photoshop Document Buffer Overflow (Updated)
# ImageMagick Photoshop Document Buffer Overflow (Updated)
# ImageMagick Photoshop Document Buffer Overflow (Updated)
# ImageMagick Remote EXIF Parsing Buffer Overflow (Updated)
# ImageMagick Remote EXIF Parsing Buffer Overflow (Updated)
# ImageMagick Remote EXIF Parsing Buffer Overflow (Updated)
# ImageMagick Remote EXIF Parsing Buffer Overflow (Updated)
# Info-ZIP UnZip File Permission Modification
# Info-ZIP UnZip File Permission Modification (Updated)
# Info-ZIP UnZip File Permission Modification (Updated)
# Info-ZIP UnZip File Permission Modification (Updated)
# Info-ZIP UnZip File Permission Modification (Updated)
# Info-ZIP UnZip File Permission Modification (Updated)
# Info-ZIP UnZip File Permission Modification (Updated)
# Info-ZIP Zip Remote Recursive Directory Compression Buffer Overflow (Updated)
# Info-ZIP Zip Remote Recursive Directory Compression Buffer Overflow (Updated)
# Info-ZIP Zip Remote Recursive Directory Compression Buffer Overflow (Updated)
# Info-ZIP Zip Remote Recursive Directory Compression Buffer Overflow (Updated)

So the list includes the same entries over and over because they are [updated]. The quoted part above represents four vulnerabilities but appears to have been responsible for twenty entries instead. The PCRE overflow issue gets twelve entries on the CERT list. Why is the Unix/Linux section full of this type of screw up, yet magically the Windows section contains very few? Could it be that the Unix/Linux vendors actually respond to every issue in a more timely fashion? Or is US-CERT intentionally trying to harm the reputation of Unix/Linux? What, don’t like those two conclusions? TOUGH, that is what happens when you release such shoddy ‘research’ or ‘statistics’ (used very lightly).

Fortunately for me, someone at Slashdot noticed the same thing and did some calculations based on removing the [updated] entries; Windows drops from 813 to 671, Unix/Linux drops from 2328 to 891, and Multiple drops from 2057 to 1512. This gives us a total of 3074 vulnerabilities reported (by US-CERT standards), not 5198. With a margin for error so large, how can anyone take them seriously? More to the point, how can the mainstream media journalists over at the Washington Post blog about this, but not challenge the statistics?

A decade later, and the security community still lacks any meaningful statistics for vulnerabilities. Why can’t these outfits with commercial or federal funding actually do a good job and produce solid data that helps instead of confuses and misleads?!

Open Source Vulnerability Database Opens for Public Access

The Open Source Vulnerability Database (OSVDB), a project to catalog and describe the Internet’s security vulnerabilities, opened for public use on 31 March 2004.

The OSVDB project was launched in 2002 following a realization in the security community that no independent, community-operated vulnerability database existed. There were, and still are, numerous vulnerability databases. Some of these databases are managed by private interests to meet their own requirements, while others contain a limited subset of vulnerabilities or have significant restrictions on their content. None are simultaneously comprehensive, open for free use, and answerable to the community. The OSVDB’s organizers set out to implement a vulnerability database that meets all those requirements.

The OSVDB project has been successful in fulfilling its original objectives. The project concentrated at first on establishing a core group of project organizers, on creating the technical infrastructure to collect and validate vulnerability data, and on building a team of contributors to create the open-source vulnerability records. These goals have been met, and the OSVDB team is now planning its next stage of growth. After a significant period of development – in effect, an “alpha” release – it has been opened to the public as of 31 March 2004 at http://www.osvdb.org/.

A GROWING PROBLEM

According to CERT’s statistics, the number of computer security vulnerabilities found each year has risen over two thousand percent since 1995. Tracking these vulnerabilities and their cures is critical for those who protect networked systems against accidental misuse and deliberate attack, from home users and small businesses to globe-spanning enterprises.

Annual vulnerability announcements number in the thousands, well beyond the capacity for human memory to manage. Well-organized databases, with verified contents and flexible search abilities, are required if these vulnerabilities are to be controlled by the security community. The OSVDB provides the necessary structure, technology, and content to support that community requirement for vulnerability management.

AN OPEN SOLUTION

The OSVDB’s main goal is to be complete and to be without bias. It should serve as one-stop shopping for all vulnerability needs. Developers creating vulnerability-assessment tools, system administrators protecting servers and networks, business staff assessing risks and remedies, academic researchers documenting analyzing the past and future of network security: all expend effort to identify vulnerabilities, all work to document them consistently, all can benefit from a single, comprehensive source of vulnerability data. The OSVDB is this source, reducing duplication of effort while it promotes data consistency.

The OSVDB is unbiased and neutral in its practices for accepting, reviewing, and publishing vulnerabilities. Its open acceptance of community input and internal review processes ensure that the vulnerability database is not colored by vendor-related biases. OSVDB organizers believe that more than one vulnerability database is needed to meet the full variety of community requirements. While it references the other vulnerability databases, it develops its own database entries to ensure that there are no restrictions on distribution and re-use of the OSVDB vulnerability data: its contents are free of cost and free of restrictions on use.

FUTURE DIRECTIONS

Licensing

Research and analysis of licensing alternatives for the OSVDB products and services are underway. The OSVDB project team expects to produce the final project license in the second quarter of 2004. In the meantime, a working-draft license is in force (see the OSVDB website at http://www.osvdb.org/license.php).

Formal non-profit standing

The OSVDB team is currently working to provide the required legal status by incorporating an organization under United States law. The organization, tentatively named the Open Security Foundation, will be a private not-for-profit foundation. Its mission is to make information-technology (IT) security information and services freely available to all who need it. The foundation’s initial project will be the Open Source Vulnerability Database, but it will be capable of hosting additional security projects and will actively seek out suitable ones.

OSVDB ethical vulnerability disclosure

The OSVDB’s policy on the release of vulnerability information will incorporate clear guidelines on the timing of notification to the product developer, and of notification to the open security community. The OSVDB’s approach will support an ethical and predictable process for this release. The policy is expected to be published in the second quarter of 2004.

Recruitment

An open-source project succeeds or fails based on the support of its volunteer participants. The long-term viability of the OSVDB project depends on continuous success in recruiting new participants, and in recognizing the contributions of those who work within the project. Programs and initiatives to publicize the OSVDB’s work and to recruit new participants will be pursued in the second quarter of 2004 and continuously after that.

Expansion of the vulnerability database

In its initial development phase, the OSVDB project created an online content-management system to add vulnerability records to the database. The system supports the initial research and creation of records, the review process, and incorporation of the finalized records into the public database. Throughout initial use and testing, the system has been improved continuously to streamline the needed tasks and to make it easier to perform the research and cross-referencing needed to complete a vulnerability record. This focus on ease of use will help contributors work efficiently and will speed the creation of vulnerability records, leading to the desired expansion of the vulnerability database.

Advanced vulnerability retrieval

The vulnerability database is currently available in its entirety from the OSVDB website. The OSVDB is developing tools to make it easy to search the vulnerability database on-line so that straightforward queries are easy to make. For those requiring a higher degree of automation in querying and retrieving vulnerabilities, an XML-formatted version of the database will be developed so that automated processes can query it remotely. The OSVDB system will also prototype automated posting of vulnerabilities through an RSS-like “push” mechanism. Subscribers will receiver each new vulnerability at the moment it is cleared into the database, and can choose to set customized filters to receive a subset of those records as needed. These new features are intended to be put in place over the second and third quarters of 2004.

Active integration with vulnerability tools

Tracking existing and new vulnerabilities is one of the toughest challenges for developers of security tools. OSVDB is working to streamline the process of identifying and setting priorities for the vulnerabilities it provides to tool developers like the Nessus, Snort, and Nikto projects. In brief, the OSVDB will assist vulnerability-tool developers to identify vulnerabilities that are not already represented in their products, and will provide a way to identify the high-priority vulnerabilities for immediate attention.

CONCLUSION

The OSVDB is relatively new in the arena of open-source projects. It was first conceived in the summer of 2002, and has already put in place much of the organization, technology, and process needed to meet its initial goals. Continuing to build on that foundation, however, will allow the OSVDB to become more useful and more central to the information-technology security community. The upcoming year promises not just incremental improvements to the OSVDB, but also innovations to the existing legal and organizational structure of the project, a focus on recruitment of project participants, and technical advances to make the project even more valuable to the security community. The OSVDB online system can be found at http://www.OSVDB.org.

Complete information on the OSVDB’s aims and objectives can be found at: http://osvdb.org/documentation.php

MORE INFORMATION

Jake Kouns Open Source Vulnerability Database Project jkouns@osvdb.org

JOIN THE PROJECT

The network needs YOU! Check out the project FAQs at http://www.osvdb.org/faq.php, then join using the form at http://www.osvdb.org/newuser.php.

Follow

Get every new post delivered to your Inbox.

Join 5,028 other followers