The Open Security Foundation (OSF) and Risk Based Security wrote an open letter to FIRST regarding the upcoming Common Vulnerability Scoring System (CVSS) version 3 proposal. While we were not formally asked to provide input, given the expertise of managing vulnerability databases, along with the daily use of CVSS, we felt the feedback would provide valuable insight to improve CVSS in the future.
Some of the areas discussed include:
- Introducing 4 levels for granularity
- Better definitions for terminology for more accurate scoring
- Re-examining the pitfalls of “Access Complexity”
- Limitations of the current Access Vector breakdown
- The challenge of scoring authentication
- And a variety of other considerations to improve vulnerability scoring
Our conclusion points to the need for CVSS to be overhauled as CVSSv2 has too many current shortcomings to provide an adequate and useful risk scoring model. You can download the full letter in PDF format.
Readers may recall that I blogged about a similar topic just over a month ago, in an article titled Advisories != Vulnerabilities, and How It Affects Statistics. In this installment, instead of “advisories”, we have “CVEs” and the inherent problems when using CVE identifiers in the place of “vulnerabilities”. Doing so is technically inaccurate, and it negatively influences statistics, ultimately leading to bad conclusions.
NSS Labs just released an extensive report titled “Vulnerability Threat Trends; A Decade in Review, Transition on the Way“, by Stefan Frei. While the report is interesting, and the fundamental methodology is sound, Frei uses a dataset that is not designed for true vulnerability statistics. Additionally, I believe that some factors that Frei attributes to trends are incorrect. I offer this blog as open feedback to bring additional perspective to the realm of vulnerability stats, which is a long ways from approaching maturity.
Vulnerabilities versus CVE
In the NSS Labs paper, they define a vulnerability as “a weakness in software that enables an attacker to compromise the integrity, availability, or confidentiality of the software or the data that it processes.” This is as good a definition as any. The key point here is a weakness, singular. What Frei fails to point out, is that the CVE dictionary is not a vulnerability database in the same sense as many others. It is a specialty database designed primarily to assign a unique identifier to a vulnerability, or a group of vulnerabilities, to coordinate tracking and discussion. While CVE says “CVE Identifiers are unique, common identifiers for publicly known information security vulnerabilities” , it is more important to note the way CVE abstracts, which is covered in great detail. From the CVE page on abstraction:
CVE Abstraction Content Decisions (CDs) provide guidelines about when to combine multiple reports, bugs, and/or attack vectors into a single CVE name (“MERGE”), and when to create separate CVE names (“SPLIT”).
This clearly denotes that a single CVE may represent multiple vulnerabilities. With that in mind, every statistic generated by NSS Labs for this report is not accurate, and their numbers are not reproduceable using any other vulnerability dataset (unless it too is only based on CVE data and does not abstract differently, e.g. NVD). This distinction puts the report’s statements and conclusions in a different light:
As of January 2013 the NVD listed 53,489 vulnerabilities ..
In the last ten years on average 4,660 vulnerabilities were disclosed per year ..
.. with an all-‐time high of 6,462 vulnerabilities counted in 2006 ..
The abstraction distinction means that these numbers aren’t just technically inaccurate (i.e. terminology), they are factually inaccurate (i.e. actual stats when abstracting on a per-vulnerability basis). In each case where Frei uses the term “vulnerability”, he really means “CVE”. When you consider that a single CVE may cover as many as 66 or more distinct vulnerabilities, it really invalidates any statistic generated using this dataset as he did. For example:
However, in 2012 alone the number of vulnerabilities increased again to a considerable 5,225 (80% of the all-‐time high), which is 12% above the ten-‐year average. This is the largest increase observed in the past six years and ends the trend of moderate declines since 2006.
Based on my explanation, what does 5,225 really mean? If we agree for the sake of argument, that CVE averages two distinct vulnerabilities per CVE assignment, that is now over 10,000 vulnerabilities. How does that in turn change any observations on trending?
The report’s key findings offer 7 high-level conclusions based on the CVE data. To put all of the above in more perspective, I will examine a few of them and use an alternate dataset, OSVDB, that abstracts entries on a per-vulnerability basis. With those numbers, we can see how the findings stand. NSS Labs report text is quoted below.
The five year long trend in decreasing vulnerability disclosures ended abruptly in 2012 with a +12% increase
Based on OSVDB data, this is incorrect. Both 2009 (7,879) -> 2010 (8,835) as well as 2011 (7,565) -> 2012 (8,919) showed an upward trend.
More than 90 percent of the vulnerabilities disclosed are moderately or highly critical – and therefore relevant
If we assume “moderately” is “Medium” criticality, as later defined in the report, is 4.0 -‐ 6.9 then OSVDB shows 57,373 entries that are CVSSv2 4.0 – 10.0, out of 82,123 total. That means 90% is considerably higher than we show. Note: we do not have complete CVSSv2 data for 100% of our entries, but we do have them for all entries affiliated with the ones Frei examined and more. If “moderately critical” and “highly critical” refer to different ranges, then they should be more clearly defined.
It is also important to note that this finding is a red herring, due to the way CVSS scoring works. A remote path disclosure in a web application scores a 5.0 base score (CVSS2#AV:N/AC:L/Au:N/C:P/I:N/A:N). This skews the scoring data considerably higher than many in the industry would agree with, as 5.0 is the same score you get for many XSS vulnerabilities that can have more serious impact.
9 percent of vulnerabilities disclosed in 2012 are extremely critical (with CVSS score>9.9) paired with low attack/exploitation complexity
This is another red herring, because any CVSS 10.0 score means that “low complexity” was factored in. The wording in the report implies that a > 9.9 score could be paired with higher complexity, which isn’t possible. Further, CVSS is scored for the worst case scenario when details are not available (e.g. CVE-2012-5895). Given the number of “unspecified” issues, this may seriously skew the number of CVSSv2 10.0 scores.
Finally, there was one other element to this report that was used in the overview, and later in the document, that is used to attribute a shift in disclosure trends. From the overview:
The parallel and massive drop of vulnerability disclosures by the two long established purchase programs iDefense VCP and TippingPoint ZDI indicate a transition in the way vulnerability and exploit information is handled in the industry.
I believe this is a case of “correlation does not mean causation“. While these are the two most recognized third-party bug bounty programs around, there are many variables at play here. In the bigger picture, shifts in these programs do not necessarily mean anything. Some of the factors that may have influenced disclosure numbers for those two programs include:
- There are more bug bounty programs available. Some may offer better price or incentive for disclosing through them, stealing business from iDefense/ZDI.
- Both companies have enjoyed their share of internal politics that affected at least one program. In 2012, several people involved in the ZDI program left the company to form their startup. It has been theorized that since their departure, ZDI has not built the team back up and that disclosures were affected as a result.
- ZDI had a small bout of external politics, in which one of their most prevalent bounty collectors (Luigi Auriemma) had a serious disagreement about ZDI’s handling of a vulnerability, as relates to Portnoy and Exodus. Auriemma’s shift to disclose via his own company would dramatically affect ZDI disclosure totals alone.
- Both of these companies have a moving list of software that they offer a bounty on. As it changes, it may result in spikes of disclosures via their programs.
Regardless, iDefense and ZDI represent a small percentage of overall disclosures, it is curious that Frei opted to focus on this so prominently as a reason for vulnerability trends changing without considering some influencing factors. Even during a good year, 2011 for example, iDefense (42) and ZDI (297) together accounted for 339 out of 7,565 vulnerabilities, only ~ 4.5% of the overall disclosures. There are many other trends that could just as easily explain relatively small shifts in disclosure totals. When making statements about trends in vulnerability disclosure and how it affects statistics, it isn’t something that should be done by casual observers. They simply miss a lot of the low-level details you glean on the day-to-day vulnerability handling and cataloging.
To be clear, I am not against using CVE/NVD data to generate statistics. However, when doing so, it is important that the dataset be explained and qualified before going into analysis. The perception and definition of what “a vulnerability” is changes based on the person or VDB. In vulnerability statistics, not all vulnerabilities are created equal.
I’ve written about the various problems with generating vulnerability statistics in the past. There are countless factors that contribute to, or skew vulnerability stats. This is an ongoing problem for many reasons. First, important numbers are thrown around in the media and taken as gospel, creating varying degrees of bias in administrators and owners. Second, these stats are rarely explained to show how they were derived. In short, no one shows their work, shows potential bias, caveats, or other issues that should be included as a responsible security professional. A recent article has highlighted this problem again. To better show why vulnerability stats are messy, but important, I will show you how it is trivial to skew numbers simply by using different criteria, along with several pitfalls that must be factored into any set of stats you generate. The fun part is that the word used to describe the differences can be equally nebulous and they are all valid, if properly disclaimed!
I noticed a Tweet from @SCMagazine about an article titled “The ghosts of Microsoft: Patch, present and future”. The article is by Alex Horan, security strategist, CORE Security and discusses Microsoft’s vulnerabilities this year. Reading down, the first line of the second paragraph immediately struck me as being incorrect.
Based on my count, there were 83 vulnerabilities announced by Microsoft over the past year. This averages out to a little more than six per month, a reasonable number of patches (and reboots) to apply to your systems over the course of a year.
It is difficult to tell if Horan means “vulnerabilities” or “patches”, as he appears to use the same word to mean both, when they are quite different. The use of ’83’ makes it very clear, Horan is referencing Microsoft advisories, not vulnerabilities. This is an important distinction as a single advisory can contain multiple vulnerabilities.
A cursory look at the data in OSVDB showed there were closer to 170 vulnerabilities verified by Microsoft in 2012. Doing a search to include references for “MS12” (used in their advisory designation), 160 results. This is how it was easy to determine the number Horan used was inaccurate, or his wording was. If you generate statistics based on advisories versus independent vulnerabilities, results will vary greatly. To add a third perspective, we must also consider the total number of disclosed vulnerabilities in Microsoft products. This means ones that did not correspond to a Microsoft advisory (e.g. perhaps a KB only), did not receive a CVE designation, or were missed completely by the company. On Twitter, Space Rogue (@spacerog) asked about severity breakdowns over the last few years. Since that would take considerable time to generate, I am going to stay focused on 2012 as it demonstrates the issues. Hopefully this will give him a few numbers though!
If we look at the 2012 Microsoft advisories versus 2012 Microsoft CVE versus 2012 Microsoft total vulnerabilities, and do a percentage breakdown by severity, you can see heavy bias. We will use the following breakdown of CVSS scores to determine severity: 9 – 10 = critical, 7 – 8.9 = important, 4 – 6.9 = moderate, 0 – 3.9 = low.
|2012 Advisories (83)||35 (42.2%)||46 (55.4%)||2 (2.4%)||—|
|2012 CVE (160)||100 (62.5%)||18 (11.3%)||39 (24.4%)||3 (1.8%)|
|2012 Total (176)||101 (57.4%)||19 (10.8%)||41 (23.3%)||15 (8.5%)|
It isn’t easy to see the big shifts in totals in a chart, but it is important to establish the numbers involved when displaying any type of chart or visual representation. If we look at those three breakdowns using simple pie charts, the shifts become much more apparent:
The visual jump in critical vulnerabilities from the first to the second two charts is distinct. In addition, notice the jump from the first two charts to the third in regards to the low severity vulnerabilities and that they didn’t even make an appearance on the first chart. This is a simple example of how the “same” vulnerabilities can be represented, based on terminology and the source of data. If you want to get pedantic, there are additional considerations that must be factored into these vulnerabilities.
In no particular order, these are other points that should not only be considered, but disclaimed in any presentation of the data above. While it may seem minor, at least one of these points could further skew vulnerability counts and severity distribution.
- MS12-080 Only contains 1 CVE if you look at immediate identifiers, but also contains 2 more CVE in the fine print related to Oracle Outside In, which is used by the products listed in the advisory.
- MS12-058 actually has no immediate CVEs! If you read the fine print, it actually covers 13 vulnerabilities. Again, these are vulnerabilities in Oracle Outside In, which is used in some Microsoft products.
- Of the 176 Microsoft vulnerabilities in 2012, as tracked by OSVDB, 10 do not have CVE identifiers assigned.
- OSVDB 83750 may or may not be a vulnerability, as it is based on a Microsoft KB with uncertain wording. Vague vulnerability disclosures can skew statistics.
- Most of these CVSS scores are taken from the National Vulnerability Database (NVD). NVD outsources CVSS score generation to junior analysts from a large consulting firm. Just as we occasionally have mistakes in our CVSS scores, so does NVD. Overall, the number of scores that have serious errors are low, but they can still introduce a level of error into statistics.
- One of the vulnerabilities (OSVDB 88774 / CVE-2012-4792) has no formal Microsoft advisory, because it is a 0-day that was just discovered two days ago. There will almost certainly be a formal Microsoft advisory in January 2013 that covers it. This highlights a big problem with using vendor advisories for any statistic generation. Vendors generally release advisories when their investigation of the issue has completed, and a formal solution is made available. Generating statistics or graphics off the same vulnerabilities, but using disclosure versus solution date will give two different results.
These are just a few ways that statistics can be manipulated, often by accident, and why presenting as much data and explanation is beneficial to everyone. I certainly hope that SCMagazine and/or CORE will issue a small correction or explanation as to the what the “83” number really represents.
I always mean to post changes more frequently, but apathy and other tasks seem to win the day. Here is a brief list of OSVDB change highlights over the past few months.
- The database currently covers 59,833 vulnerabilities, spanning 26,179 products from 4,735 researchers, over 44 years.
- Sequoia, ES&S and InkaVote e-voting machine audit documents integrated (all text search “electronic voting machine”)
- Happy 10th birthday CVE! We are now “fully” mapped for all years
- Integrated the historic Phage mail list content (http://securitydigest.org/phage/)
- Our Snort X-ref import script was borked (since Sep). Fixed, added almost 500 recent Snort IDs to references
- Apache bug system scoured, over 150 Apache vulns added from the last eight years (http://osvdb.org/ref/blog/apache-scouring.txt)
- Metasploit static references supported (http://blog.osvdb.org/2009/10/23/metasploit-reference-support-added-more)
- Exploit-DB references supported (http://www.exploit-db.com/)
- VuPEN references supported (http://www.vupen.com/)
- Many vulnerability and solution templates overhauled
- Search engine rebuilds are considerably faster, will auto-tweet when rebuilding (as it may affect search results)
- Reference search for full URL works
- Title search for multiple words fixed (was temporarily matching on some but not all words)
- New search filters and custom exports (http://blog.osvdb.org/2009/11/09/search-filters-custom-exports)
- Inverse search filtering enabled (http://blog.osvdb.org/2009/10/30/not-it)
- Search by CVSS scores (http://blog.osvdb.org/2009/10/28/search-enhance-by-cvss-score-or-attribute)
- Any search can be turned into a ‘Watch List’. Left nav menu has this option, new results are mailed to you as entered in the system
- New menu system (top and left nav)
- Twitter feed more actively used for project updates
- Twitter feed displays on front page
- ‘About’ page is updated, expect more static pages to be updated to better reflect project status soon
- CVSSv2 scoring support added, including:
- CVSS scoring history (historically track NVD, OSVDB and other sources)
- Anyone can submit scores for entries without CVE/NVD (over 13,000)
- Updating CVSS scores for entries without are worth .25 points for now, to encourage mangling
- Moderation system in place for submitted CVSS scores
- Creditee system overhaul (http://blog.osvdb.org/2009/11/21/creditee-system-overhauled)
- “Vulnerabilities in OSVDB disclosed by type by quarter” graphs added to front page
- More fixes to continue support for IE6. Don’t expect this to last!
Using the ‘Advanced Search‘, you can now search the database by entering a CVSSv2 score range (e.g., 8 to 10) or by a specific CVSSv2 attribute (e.g., Confidentiality : Partial). To search for entries with only a 10 score, use the search range 10 to 10.
Using this search mechanism, we can see there are 3,217 entries in the database with a score of 10 and 9,266 entries that involve a complete loss of availability.
We hope this flexibility allows for even more refined searches to better help your project or organization. Stay tuned, this is one of many new search features planned.
Along with the score, we display the date that NVD generated it and give users a method for recommending updates if they feel the score is inaccurate. While this is long overdue, this is one of many CVSS related features we will be adding in the near future. For those wondering about the delay in adding CVSS support, the cliff notes answer is “we had reservations about the scoring system”. Back in 2005, Jake and I had a long chat with a couple of the creators of CVSS and brought up our concerns. Our goal was to create our own scoring system, but internal debate (and procrastination) lead to neither being implemented. Rather than creating our own system, we finally opted to use what has become an industry standard. Some of our planned CVSS score enhancements on the to-do list, no particular order:
- Method for adding our own CVSS score. There are thousands of entries in OSVDB that do not have a CVE assignment, and as a result, no NVD based CVSS score.
- A more robust moderation queue to handle proposed changes. This may optionally have a one-click method for us to notify NVD of our change so they may consider revising their score.
- Ensure the CVSSv2 score is part of the database dumps, available for download.
- Method for tracking CVSS score historically. As NVD revises their score, or we do, a user should be able to see the history of changes.
- Compare our/NVD scores with other public tools, display discrepancy if different. For example, if a Nessus plugin scores an issue differently than NVD, show both scores so users may consider which is more accurate.
- Track researcher generated CVSS score. While infrequent, some advisories provide scoring. If different than NVD, display the discrepancy.
As always, if you have ideas on how we could better handle CVSS scoring, or have additional ideas for features, please contact us!
Oracle’s last quarterly critical patch update included some changes and started using CVSS to rate the severity of their vulnerabilities. Anyone that has ever tried to truly understand Oracle vulnerabilities most likely thought this would be a much needed improvement. The whole easy, difficult, wide, low, high ratings Oracle used previously made it almost impossible to figure out just how critical are the issues and then to prioritize the patch implementation.
Shortly after the October CPU was released, researchers started to question the CVSS ratings leading many to believe that Oracle is downplaying the true risk of the vulnerabilities.
Oracle also patched 13 remotely exploitable holes in its Application Server software, the highest of which the vendor rated as 4.7 out of 10. However, a closer examination of the flaws suggest that many of the ratings should be in the 8.0 range, said Caleb Sima, CTO of SPI Dynamics, an Atlanta-based security vendor that also reported bugs to Oracle. “The problem is, Oracle didn’t give enough details [for third parties] to be able to say exactly what the score should be,” Sima said. – Source
Oracle claims that they are listening to their customers and trying to help organizations really understand the true risk. However, it appears that for many of the vulnerabilities there contained even less detail with the new format than previously. Was the only real improvement to the advisories that questionable CVSS ratings were included?