I’ve written about the various problems with generating vulnerability statistics in the past. There are countless factors that contribute to, or skew vulnerability stats. This is an ongoing problem for many reasons. First, important numbers are thrown around in the media and taken as gospel, creating varying degrees of bias in administrators and owners. Second, these stats are rarely explained to show how they were derived. In short, no one shows their work, shows potential bias, caveats, or other issues that should be included as a responsible security professional. A recent article has highlighted this problem again. To better show why vulnerability stats are messy, but important, I will show you how it is trivial to skew numbers simply by using different criteria, along with several pitfalls that must be factored into any set of stats you generate. The fun part is that the word used to describe the differences can be equally nebulous and they are all valid, if properly disclaimed!
I noticed a Tweet from @SCMagazine about an article titled “The ghosts of Microsoft: Patch, present and future”. The article is by Alex Horan, security strategist, CORE Security and discusses Microsoft’s vulnerabilities this year. Reading down, the first line of the second paragraph immediately struck me as being incorrect.
Based on my count, there were 83 vulnerabilities announced by Microsoft over the past year. This averages out to a little more than six per month, a reasonable number of patches (and reboots) to apply to your systems over the course of a year.
It is difficult to tell if Horan means “vulnerabilities” or “patches”, as he appears to use the same word to mean both, when they are quite different. The use of ’83′ makes it very clear, Horan is referencing Microsoft advisories, not vulnerabilities. This is an important distinction as a single advisory can contain multiple vulnerabilities.
A cursory look at the data in OSVDB showed there were closer to 170 vulnerabilities verified by Microsoft in 2012. Doing a search to include references for “MS12″ (used in their advisory designation), 160 results. This is how it was easy to determine the number Horan used was inaccurate, or his wording was. If you generate statistics based on advisories versus independent vulnerabilities, results will vary greatly. To add a third perspective, we must also consider the total number of disclosed vulnerabilities in Microsoft products. This means ones that did not correspond to a Microsoft advisory (e.g. perhaps a KB only), did not receive a CVE designation, or were missed completely by the company. On Twitter, Space Rogue (@spacerog) asked about severity breakdowns over the last few years. Since that would take considerable time to generate, I am going to stay focused on 2012 as it demonstrates the issues. Hopefully this will give him a few numbers though!
If we look at the 2012 Microsoft advisories versus 2012 Microsoft CVE versus 2012 Microsoft total vulnerabilities, and do a percentage breakdown by severity, you can see heavy bias. We will use the following breakdown of CVSS scores to determine severity: 9 – 10 = critical, 7 – 8.9 = important, 4 – 6.9 = moderate, 0 – 3.9 = low.
|2012 Advisories (83)||35 (42.2%)||46 (55.4%)||2 (2.4%)||–|
|2012 CVE (160)||100 (62.5%)||18 (11.3%)||39 (24.4%)||3 (1.8%)|
|2012 Total (176)||101 (57.4%)||19 (10.8%)||41 (23.3%)||15 (8.5%)|
It isn’t easy to see the big shifts in totals in a chart, but it is important to establish the numbers involved when displaying any type of chart or visual representation. If we look at those three breakdowns using simple pie charts, the shifts become much more apparent:
The visual jump in critical vulnerabilities from the first to the second two charts is distinct. In addition, notice the jump from the first two charts to the third in regards to the low severity vulnerabilities and that they didn’t even make an appearance on the first chart. This is a simple example of how the “same” vulnerabilities can be represented, based on terminology and the source of data. If you want to get pedantic, there are additional considerations that must be factored into these vulnerabilities.
In no particular order, these are other points that should not only be considered, but disclaimed in any presentation of the data above. While it may seem minor, at least one of these points could further skew vulnerability counts and severity distribution.
- MS12-080 Only contains 1 CVE if you look at immediate identifiers, but also contains 2 more CVE in the fine print related to Oracle Outside In, which is used by the products listed in the advisory.
- MS12-058 actually has no immediate CVEs! If you read the fine print, it actually covers 13 vulnerabilities. Again, these are vulnerabilities in Oracle Outside In, which is used in some Microsoft products.
- Of the 176 Microsoft vulnerabilities in 2012, as tracked by OSVDB, 10 do not have CVE identifiers assigned.
- OSVDB 83750 may or may not be a vulnerability, as it is based on a Microsoft KB with uncertain wording. Vague vulnerability disclosures can skew statistics.
- Most of these CVSS scores are taken from the National Vulnerability Database (NVD). NVD outsources CVSS score generation to junior analysts from a large consulting firm. Just as we occasionally have mistakes in our CVSS scores, so does NVD. Overall, the number of scores that have serious errors are low, but they can still introduce a level of error into statistics.
- One of the vulnerabilities (OSVDB 88774 / CVE-2012-4792) has no formal Microsoft advisory, because it is a 0-day that was just discovered two days ago. There will almost certainly be a formal Microsoft advisory in January 2013 that covers it. This highlights a big problem with using vendor advisories for any statistic generation. Vendors generally release advisories when their investigation of the issue has completed, and a formal solution is made available. Generating statistics or graphics off the same vulnerabilities, but using disclosure versus solution date will give two different results.
These are just a few ways that statistics can be manipulated, often by accident, and why presenting as much data and explanation is beneficial to everyone. I certainly hope that SCMagazine and/or CORE will issue a small correction or explanation as to the what the “83″ number really represents.
Perhaps it is the fine tequila this evening, but I really don’t get how our industry can latch on to the recent ‘Aurora’ incident and try to take Microsoft to task about it. The amount of news on this has been overwhelming, and I will try to very roughly summarize:
- News surfaces Google, Adobe and 30+ companies hit by “0-day” attack
- Google uses this for political overtones
- Originally thought to be Adobe 0-day, revealed it was MSIE 0-day
- Jan 14, confirmed it is MSIE vuln, shortly after dubbed “aurora”
- Jan 21, uproar over MS knowing about the vuln since Sept
Now, here is where we get to the whole forest, trees and some analogy about eyesight. Oh, I’ll warn (and surprise) you in advance, I am giving Microsoft the benefit of the doubt here (well, for half the blog post) and throwing this back at journalists and the security community instead. Let’s look at this from a different angle.
The big issue that is newsworthy is that Microsoft knew of this vulnerability in September, and didn’t issue a patch until late January. What is not clear, is if Microsoft knew it was being exploited. The wording of the Wired article doesn’t make it clear: “aware months ago of a critical security vulnerability well before hackers exploited it to breach Google, Adobe and other large U.S. companies” and “Microsoft confirmed it learned of the so-called ‘zero-day’ flaw months ago”. Errr, nice wording. Microsoft was aware of the vulnerability (technically), before hackers exploited it, but doesn’t specifically say if they KNEW hackers were exploiting it. Microsoft learned of the “0-day” months ago? No, bad bad bad. This is taking an over-abused term and making it even worse. If a vulnerability is found and reported to the vendor before it is exploited, is it still 0-day (tree, forest, no one there to hear it falling)?
Short of Microsoft admitting they knew it was being exploited, we can only speculate. So, for fun, let’s give them a pass on that one and assume it was like any other privately disclosed bug. They were working it like any other issue, fixing, patching, regression testing, etc. Good Microsoft!
Bad Microsoft! But, before you jump on the bandwagon, bad journalists! Bad security community!
Why do you care they sat on this one vulnerability for six months? Why is that such a big deal? Am I the only one who missed the articles pointing out that they actually sat on five code execution bugs for longer? Where was the outpour of blogs or news articles mentioning that “aurora” was one of six vulnerabilities reported to them during or before September, all in MSIE, all that allowed remote code execution (tree, forest, not seeing one for the other)?
|CVE||Reported to MS||Disclosed||Time to Patch|
|CVE-2010-0244||2009-07-14||2010-01-21||6 Months, 7 Days (191 days)|
|CVE-2010-0245||2009-07-14||2010-01-21||6 Months, 7 Days (191 days)|
|CVE-2010-0246||2009-07-16||2010-01-21||6 Months, 5 Days (189 days)|
|CVE-2010-0248||2009-08-14||2010-01-21||5 Months, 7 days (160 days)|
|CVE-2010-0247||2009-09-03||2010-01-21||4 Months, 18 days (140 days)|
|CVE-2010-0249||2009-09-??||2010-01-14||4 Months, 11 days (133 days) – approx|
|CVE-2010-0027||2009-11-15||2010-01-21||2 Months, 6 days (67 days)|
|CVE-2009-4074||2009-11-20||2009-11-21||2 Months, 1 day (62 days)|
Remind me again, why the “Aurora” conspiracy is noteworthy? If Microsoft knew of six remote code execution bugs, all from the September time-frame, why is one any more severe than the other? Is it because one was used to compromise hosts, detected and published in an extremely abnormal fashion? Are we actually trying to hold Microsoft accountable on that single vulnerability when the five others just happened not to be used to compromise Google, Adobe and others?
Going back to the Wired article, they say on the second to last paragraph: “On Thursday, meanwhile, Microsoft released a cumulative security update for Internet Explorer that fixes the flaw, as well as seven other security vulnerabilities that would allow an attacker to remotely execute code on a victim’s computer.” Really, Wired? That late in the article, you gloss over “seven other vulnerabilities” that would allow remote code execution? And worse, you don’t point out that Microsoft was informed of five of them BEFORE AURORA?
Seriously, I am the first one to hold Microsoft over the flames for bad practices, but that goes beyond my boundaries. If you are going to take them to task over all this, at least do it right. SIX CODE EXECUTION VULNERABILITIES that they KNEW ABOUT FOR SIX MONTHS. Beating them up over just one is amateur hour in this curmudgeonly world.
A while back, Microsoft announced they were moving to release patches on the second Tuesday of each month, lovingly called Patch Tuesday. Soon after, Oracle announced that they too would be moving to scheduled releases of patches on the Tuesday closest to the 15th day of January, April, July and October. Now, Cisco has announced they are moving to scheduled patches on the fourth Wednesday of the month in March and September of each calendar year.
In the attempt to make life easier on administrators and help avoid installing patches every few days, these scheduled releases are now causing organizations to enjoy life between monster patches.
Mar 11 – Microsoft
Mar 26 – Cisco
Apr 8 – Microsoft
Apr 15 – Oracle
May 13 – Microsoft
June 10 – Microsoft
July 8 – Microsoft
July 15 – Oracle
August 12 – Microsoft
September 9 – Microsoft
September 24 – Cisco
October 14 – Microsoft, Oracle
November 11 – Microsoft
December 9 – Microsoft
As you can see, October 14 promises to be a lot of fun for companies running Oracle products on Microsoft systems. While the scheduled dates look safe, I can’t wait until we see the ”perfect storm” of vendor patches.
Steven Christey of CVE recently commented on the fact that Microsoft, Adobe, Cisco, Sun and HP all released multi-issue advisories on the same day (Feb 13). My first reaction was to come up with an amusing graphic depicting this perfect storm. Due to not having any graphic editing skills and too much cynicism, I now wonder if these are the same vendors that continually bitch about irresponsible disclosure and it “hurting their customers”?
These same customers are now being subjected to patches for at least five major vendors on the same day. In some IT shops, this is devastating and difficult to manage and recover from. If a single patch has problems it forces the entire upgrade schedule to come to a halt until the problem can be resolved. If these vendors cared for their customers like they pretend to when someone releases a critical issue w/o vendor coordination, then they would consider staggering the patches to help alleviate the burden it causes on their beloved customers.
Microsoft: Responsible Vulnerability Disclosure Protects Users
By Mark Miller, Director, Microsoft Security Response Center
Responsible disclosure, reporting a vulnerability directly to the vendor and allowing sufficient time to produce an update, benefits the users and everyone else in the security ecosystem by providing the most comprehensive and highest-quality security update possible.
Provided “sufficient time” doesn’t drag out too long, else the computer criminal (who are in the ‘security ecosystem’) benefit greatly from responsible disclosure too.
From my experience helping customers digest and respond to full disclosure reports, I can tell you that responsible disclosure, while not perfect, doesn’t increase risk as full disclosure can.
Except “your experience” wouldn’t take full disclosure cases into account appropriately. Look at some of the vulnerabilities reported in Windows, Real, Novell and other big vendors. Notice that in more and more cases, we’re seeing the vendor acknowledge multiple researchers who found the issues independantly. That is proof that multiple people know about vulnerabilities pre-disclosure, be it full or responsible. If a computer criminal has such vulnerability information that remains unpatched for a year due to the vendor producing “the most comprehensive and highest-quality security update possible”, then the risk is far worse than the responsible disclosure your experience encompasses.
Vendors only take these shortcuts because we have to, knowing that once vulnerability details are published the time to exploit can be exceedingly short-many times in the range of days or hours.
See above, the bolded “proof” I mention. If vendors are going to move along with their head in the sand, pretending that there is a single person with the vulnerability or exploit details, and pretending that they alone control the disclosure, the vendors are naive beyond imagination.
The security researcher community is an integral part of this change, with Microsoft products experiencing approximately 75 percent responsible disclosure.
I’d love to see the chart showing issues in Microsoft products (as listed in OSVDB), relevant dates (disclosed to vendor, patch date, public disclosure) and the resulting statistics. My gut says it would be less than 75%.
McAfee is reporting that Microsoft patched 133 Critical / Important vulnerabilities in 2006. They also compare this number against previous years to presumably demonstrate that security isn’t getting better at Microsoft.
This paper will examine the differences between the security posture of Microsoft’s SQL Server and Oracle’s RDBMS based upon flaws reported by external security researchers and since fixed by the vendor in question. Only flaws affecting the database server software itself have been considered in compiling this data so issues that affect, for example, Oracle Application Server have not been included. The sources of information used whilst compiling the data that forms the basis of this document include:
The Microsoft Security Bulletins web page
The Oracle Security Alerts web page
The CVE website at Mitre.
The SecurityFocus.com website
A general comparison is made covering Oracle 8, 9 and 10 against SQL Server 7, 2000 and 2005. The vendors� flagship database servers are then compared.
Microsoft is finding themselves under increasing pressure to release fixes for critical vulnerabilities. This week, Microsoft broke from tradition again and opted to release and early fix for a critical Internet Explorer vulnerability. Since we’ve seen other critical vulnerabilities come up before this one, some of which were being exploited in the wild, why the change of policy? One factor that might be influencing this decision is the sudden availability of third-party patches. Back in March, eEye released an unofficial patch for the MSIE createTextRange() flaw which drew criticism and contempt from Microsoft. Windows/IE users were under no pressure to use the patch, but it gave some an alternative to disabling Active Scripting entirely.
This time around, we’re seeing multiple third parties come up with alternative patches that may help some companies while they wait for Microsoft to officially fix a vulnerability. This week the Internet Explorer setSlice vulnerability is being exploited in the wild with more than two weeks before Microsoft possibly releases a patch for it. With this reoccuring trend of critical vulnerabilities going unpatched for “too long”, a group of security professionals has created a new response team called ZERT to help consumers. Determina has also released a patch for the setSlice vulnerability, giving consumers even more choices in helping to mitigate the threat while waiting for Microsoft to patch.
With more and more third party patches available, will it pressure Microsoft to step up and break the monthly patch cycle more often? Will they realize that making patches available for critical vulnerabilities being exploited in the wild, even if not fully tested, is a better option than consumers finding themselves under the control of computer criminals and botnets? After all, we know that Microsoft is perfectly capable of producing fast patches when they think it is a serious issue.
This blog will serve as a dumping ground for browser-based security research and vulnerability disclosure. To kick off this blog, we are announcing the Month of Browser Bugs (MoBB), where we will publish a new browser hack, every day, for the entire month of July. The hacks we publish are carefully chosen to demonstrate a concept without disclosing a direct path to remote code execution. Enjoy!
There are already two Microsoft IE vulnerabilities posted. This should prove to be a fun blog!
p.s. Anyone else have flashbacks to SoD and the HP Bug of the Week? =)