The notion of expertise in any field is fascinating. It crosses so many aspects of humans and our perception. For example, two people in the same discipline, each with the highest honors academic can grant, can still have very different expertise within that field. Society and science have advanced so we don’t have just have “science” experts and medical doctors can specialize to extreme degrees. Within Information Security, we see the same where there are experts in penetration testing, malware analysis, reverse engineering, security administration, and more.
In the context of a software company, especially one that does not specifically specialize in security (and is trivial to argue was late to the security game), you cannot shoehorn them into any specific discipline or expertise. We can all absolutely agree there is an absolute incredible level of expertise across a variety of disciplines within Microsoft. So when Microsoft releases yet another report that speaks to vulnerability disclosures, the only thing I can think of is duality. Especially in the context of a report that puts forth some expertise that they are uniquely qualified to speak on, while mixed with a topic that pre-dates Microsoft and they certainly aren’t qualified to speak on to some degree.
A Tweet from Carsten Eiram pointed me to the latest report, and brought up the obvious fact that it seemed to be way off when it comes to vulnerability disclosures.
It’s always amusing to me that you get legal disclaimers in such analysis papers before you even get a summary of the paper:
Basically, the take away is that they don’t stand behind their data. Honestly, the fact I am blogging about this suggests that is a good move and that they should not. The next thing that is fascinating is that it was written by 33 authors and 14 contributors. Since you don’t know which of them worked on the vulnerability section, it means we get to hold them all accountable. Either they can break it down by author and section, or they all signed off on the entire paper. Sorry, the joys of academic papers!
After the legal disclaimers, then you start to get the analysis disclaimers, which are more telling to me. Companies love to blindly throw legal disclaimers on anything and everything (e.g. I bet you still get legal disclaimers in the footer of emails you receive, that have no merit). When they start to explain their results via disclaimers while not actually including their methodology, anyone reading the report should be concerned. From their “About this report” section:
This volume of the Microsoft Security Intelligence Report focuses on the first and second quarters of 2016, with trend data for the last several quarters presented on a quarterly basis. Because vulnerability disclosures can be highly inconsistent from quarter to quarter and often occur disproportionately at certain times of the year, statistics about vulnerability disclosures are presented on a half-yearly basis.
This is a fairly specific statement that speaks as if it is fact that vulnerability trends vary by quarter (they do!), but potentially ignores the fact that they can also vary by half-year or year. We have seen that impact not only a year, but the comparison to every year prior (e.g. Will Dormann in 2014 and his Tapioca project). Arbitrarily saying that it is a ‘quarter’ or ‘half-year’ does not demonstrate experience in aggregating vulnerabilities, instead it is a rather arbitrary and short time-frame. Focusing on a quarter can easily ignore some of the biases that impact vulnerability aggregation as outlined by Steve Christey and my talk titled “Buying Into the Bias: Why Vulnerability Statistics Suck” (PPT).
Jumping down to the “Ten years of exploits: A long-term study of exploitation of vulnerabilities in Microsoft software” section, Microsoft states:
However, despite the increasing number of disclosures, the number of remote code execution (RCE) and elevation of privilege (EOP) vulnerabilities in Microsoft software has declined
Doing a title search of Risk Based Security’s VulnDB for “microsoft windows local privilege escalation” tells a potentially different story. While 2015 is technically lower than 2011 and 2013, it is significantly higher than 2012 and 2014. I can’t say for certain why these dips occur, but they are very interesting.
Thousands of vulnerabilities are publicly disclosed across the industry every year. The 4,512 vulnerabilities disclosed during the second half of 2014 (2H14) is the largest
number of vulnerabilities disclosed in any half-year period since the Common Vulnerabilities and Exposures system was launched in 1999.
This quote from the report explicitly shows serious bias in their source data, and further shows that they do not consider their wording. This would be a bit more accurate saying “The 4,512 vulnerabilities aggregated by MITRE during the second half of 2014…” The simple fact is, a lot more than 4,512 vulnerabilities were disclosed during that time. VulnDB shows that they aggregated 8,836 vulnerabilities in that same period, but less than the 9,016 vulnerabilities aggregated in the second half of 2015. Microsoft also doesn’t disclaim that the second half of 2014 is when the aforementioned Will Dormann released the results of his ‘Tapioca’ project totaling over 20,500 vulnerabilities, only 1,384 of which received CVE IDs. Why? Because CVE basically said “it isn’t worth it”, and they weren’t the only vulnerability database to do so. With all of this in mind, Microsoft’s comment about the second half of 2014 becomes a lot more complicated.
The information in this section is compiled from vulnerability disclosure data that is published in the National Vulnerability Database (NVD), the US government’s repository of standards-based vulnerability management data at nvd.nist.gov. The NVD represents all disclosures that have a published CVE (Common Vulnerabilities and Exposures) identifier.
This is a curious statement, since CVE is run by MITRE under a contract from the Department of Homeland Security (DHS), making it a “US government repository” too. More importantly, NVD is essentially a clone of CVE that just wraps additional meta-data around each entry (e.g. CPE, CWE, and CVSS scoring). This also reminds us that they opted to use a limited data set, one that is well known in the Information Security field as being woefully incomplete. So even a company as large as Microsoft, with expected expertise in vulnerabilities, opts to use a sub-par data set which drastically influences statistics.
Figure 23. Remote code executable (RCE) and elevation of privilege (EOP) vulnerability disclosures in Microsoft software known to be exploited before the corresponding security update release or within 30 days afterward, 2006–2015
The explanation for Figure 23 is problematic in several ways. Does it cherry pick RCE and EOP while ignoring context-dependent (aka user-assisted) issues? Or does this represent all Microsoft vulnerabilities? This is important to ask as most web browser exploits are considered to be context-dependent and coveted by the bad guys. This could be Microsoft conveniently leaving out a subset of vulnerabilities that would make the stats look worse. Next, looking at 2015 as an example from their chart, they say 18 vulnerabilities were exploited and 397 were not. Of the 560 Microsoft vulnerabilities aggregated by VulnDB in 2015, 48 have a known public exploit. Rather than check each one to determine the time from disclosure to exploit publication, I’ll ask a more important question. What is the provenance of Microsoft’s exploitation data? That isn’t something CVE or NVD track.
Figure 25 illustrates the number of vulnerability disclosures across the software industry for each half-year period since 2H13
Once again, Microsoft fails to use the correct wording. This is not the number of vulnerability disclosures, this is the number of disclosures aggregated by MITRE/CVE. Here is their chart from the report:
Under the chart they claim:
Vulnerability disclosures across the industry decreased 9.8 percent between 2H15 and 1H16, to just above 3,000.
As mentioned earlier, since Microsoft is using a sub-par data set, I feel it is important to see what this chart would look like using more complete data. More importantly, watch how it invalidates their claim about an industry decrease of 9.8 percent between 2H15 and 1H16, since RBS shows the drop is closer to 18%.
I have blogged about vulnerability statistics, focusing on these types of reports, for quite a while now. And yet, every year we see the exact same mistakes made by just about every company publishing statistics on vulnerabilities. Remember, unless they are aggregating vulnerabilities every day, they are losing a serious understanding of the data they work with.
March 19, 2017 Update – Carsten Eiram (@carsteneiram) pointed out that the pattern of local privilege escalation numbers actually follow an expected pattern with knowledge of researcher activity and trends:
In 2011, Tarjei Mandt while he was at Norman found a metric ton of LPEs in win32k.sys as part of a project.
In 2013, it was Mateusz Jurczyk’s turn to also hit win32k.sys by focusing on a bug-class he dubbed “double-fetch” (he’s currently starting that project up again to see if he with tweaks can find more vulns).
And in 2015, Nils Sommer (reported via Google P0) was hitting win32k.sys again along with a few other drivers and churned out a respectable amount of LPE vulnerabilities.
So 2012 and 2014 represent “standard” years while 2011, 2013, and 2015 had specific high-profile researchers focus on Windows LPE flaws via various fuzzing projects.
So the explanation is the same explanation we almost always see: vulnerability disclosures and statistics are so incredibly researcher driven based on which product / component / vulnerability type a researcher or group of researchers decides to focus on.
Today, we pushed OSVDB 82447 which covers a backdoor in the Multics Operating System. For those not familiar with this old OS, there is an entire domain covering the fascinating history behind the development of Multics. OSVDB 82447 is titled “Multics Unspecified Third-party Backdoor” and gives an interesting insight into backdoors distributed by vendors. In this case, a third-party planted it, told the vendor, and Honeywell still distributed the operating system anyway. I encourage you to read the full paper by Lieutenant Colonel Roger R. Schell, a member of the tiger team that carried out the attack.
During a US Air Force sanctioned penetration test of mainframe computers, sometime before 1979, the tiger team ended up penetrating a Multics installation at Honeywell. In an account of what happened later, a paper said that the tiger team “modified the manufacturer’s master copy of the Multics operating system itself” and injected a backdoor. The backdoor code was described as being small, “fewer than 10 instructions out of 100,000” and required a password for use. The report continues, saying that even though Honeywell was told it was there and how it worked, their technicians could not find it. Subsequently, the backdoor was distributed in future installations of Multics.
It would be interesting to know why Honeywell didn’t ask for, or didn’t receive, the specific modified code from the Air Force tiger team, and why they opted to distribute it to customers. Perhaps they thought if their own technicians couldn’t find the backdoor, no one else could. Even more interesting is why a tiger team was sanctioned to carry out a penetration test that not only gave them access to the “master copy” of Multics, but why they were allowed to actually place the backdoor there. When they heard Honeywell couldn’t find it, why didn’t they insist on ensuring it was removed before installation at customer locations? This brings a new twist to the ethics of penetration testing, at least in a historical context.
NVD announced this week that they are now going to expand and provide vulnerability information in Spanish. I found this a bit amusing since OSVDB once thought that translating the database was a critical feature that needed to be delivered back in 2002. In fact, all of the language support was in the original OSVDB database schema and the backend code was created to handle it as we truly thought this would be implemented.
However, we quickly realized there were several issues with this concept including finding people to perform the translations! Additional concerns were raised as we spoke to more people in the security industry which included many conversations with non-US based security professionals (including a long ranting conversation with FX at Defcon). The critical concern was that much of the true meaning of the vulnerabilty is lost when the information is translated. The bottom line is that it was strongly believed that the vulnerability information in OSVDB should remain only in English.
OSVDB decided that we would not proceed any further with official plans to to translate the database, however, we have been contacted from other people that have wanted to translate OSVDB and we have provided permission to do so…
Here is a copy of the NVD announcement:
The National Vulnerability Database (NVD) is expanding to provide vulnerability translations. The first translation data feed is in Spanish and is being provided in cooperation with Inteco (http://www.inteco.es/), an entity of the Spanish government’s Ministry of Industry, Tourism, and Commerce (http://www.mityc.es/). Inteco is providing the translations and is solely responsible for the translation content. NVD is providing the translation infrastructure. The result of this cooperative effort is that NVD now contains an XML feed with 7,858 Spanish translations for the Common Vulnerabilities and Exposures (CVE) dictionary of security related software flaws. This feed will be maintained with translations for all new CVE vulnerabilities and, as with the other NVD data feeds, the data can be incorporated into commercial products and services with no licensing fees or restrictions. The translations are available through translation XML feeds at http://nvd.nist.gov/download.cfm#transxml.
We would love to hear any further thoughts (good and bad) on the value of translating vulnerability information into other languages.
I’ve mentioned the sociology aspect of the hacker, vuln researcher and security companies before, specifically how they interact, how one will influence another and more. The list of fun ideas I have on these topics is great, and maybe some day i’ll find the time to write more on them. In the mean time, this obvious one popped up and focuses on vulnerability researchers, how they find bugs, and how some feed off the work of others. We see this often where ResearcherA will find a vulnerability in one script, disclose the information, and ResearcherB will followup shortly after with the same type of vulnerability in a different script of the same product.
Recently we’ve seen a rash of remote file inclusion bugs in various add-ons to Mambo and Joomla. These add-ons are typically not written by the same developers nor are they distributed with the base installation of each product. However, they all seem to have one thing in common: “mosConfig_absolute_path” (or sometimes “absolute_path”). The same variable is being exploited in dozens of different add-ons and being found by different people. If we examine the chain of disclosure, can we see patterns in who consistently does followup research (low hanging fruit) instead of finding original vulnerabilities? Are there more observations in the way they are disclosed such as reporting to exploit sites vs Bugtraq or Full Disclosure? Are there misplaced signs of ego that accompany what amounts to trivial vulnerability finds while others are more modest and take it for what it is? Is it surprising that as people jump on the bandwagon, more and more reports end up being inaccurate and not a real vulnerability?
While skimming the list,
strike-out text indicates the vulnerability has been disputed or proven false. The names of the researchers who didn’t fully check their find are in bold (and I’m curious if the other disclosures hold up under scrutiny). There is one occurrence of italics that potentially shows this type of “research” being used in the wild.
2006-08-21 bigAPE-Backup for Mambo – mdx
2006-08-20 Display MOSBot Manager for Mambo – O.U.T.L.A.W (Aria-security)
2006-08-20 EstateAgent for Mambo – O.U.T.L.A.W (Aria-security)
2006-08-19 CatalogShop for Mambo – O.U.T.L.A.W (Aria-security)
2006-08-18 Joomla x-shop – Crackers_Child
2006-08-18 Joomla Rssxt – Crackers_Child
2006-08-18 Kochsuite for Joomla – camino (Insecurity Research Team)
2006-08-18 mtg_myhomepage For Mambo – O.U.T.L.A.W (Aria-security)
2006-08-18 mambo-phphop Product Scroller – O.U.T.L.A.W (Aria-security)
2006-08-17 contentpublisher for Mambo – Crackers_Child
2006-08-17 MambelFish for Mambo – mdx
2006-08-17 JIM for Joomla – XORON
2006-08-17 mosListMessenger for Mambo – Crackers_Child
2006-08-17 anjel for Mambo – Crackers_Child
2006-08-16 Coppermine for Mambo – k1tk4t
2006-08-16 Reporter for Mambo – Crackers_Child
2006-08-16 com_lm for Mambo – Crackers_Child
2006-08-14 MMP for Mambo – mdx
2006-08-14 PeopleBook for Mambo – Matdhule
2006-08-10 Remository for Mambo – camino (Insecurity Research Team)
2006-08-07 JD-Wiki for Joomla – jank0 (hackbsd crew)
2006-07-31 Mambatstaff for Mambo – Dr.Jr7
2006-07-30 UHP for Mambo – Kurdish Security
2006-07-29 artlinks for Mambo – Dr.Jr7
2006-07-29 Colophon for Joomla – Drago84 (Exclusive Security Italian Security)
2006-07-28 Security Images for Joomla – Drago84
2006-07-28 MGM for Mambo – A-S-T TEAM
2006-07-28 Guestbook for Mambo – Matdhule
2006-07-24 PrinceClan Chess for Mambo – Tr_ZiNDaN
2006-07-20 MultiBanners for Mambo – Blue|Spy
2006-07-17 Mambo-SMF Forum – ASIANEAGLE
2006-07-17 VideoDB for Mambo – h4ntu (#batamhacker crew)
2006-07-17 LoudMouth for Mambo – h4ntu (#batamhacker crew)
2006-07-17 PollXT for Joomla – vitux
2006-07-17 Calendar for Mambo – Matdhule
2006-07-17 New Article for Mambo – Ahmad Maulana a.k.a Matdhule
2006-07-13 perForms for Joomla – “Vuln founded in a log file: lazy 0day!!! :D”
2006-07-12 Hashcash for Joomla – Ahmad Maulana a.k.a Matdhule
2006-07-12 SiteMap for Mambo – Ahmad Maulana a.k.a Matdhule
2006-07-12 HTMLArea3 for Mambo – Ahmad Maulana a.k.a Matdhule
2006-07-10 PccookBook for Mambo – Ahmad Maulana a.k.a Matdhule
2006-07-07 ExtCalendar for Mambo – Ahmad Maulana a.k.a Matdhule
2006-07-03 Galleria for Mambo – sikunYuk
2006-06-26 CBSMS Mambo Module – Kw3[R]Ln (Romanian Security Team)
2006-06-13 Jobline for Mambo – SpC-x
While all of this not necessarily useful to many, this line of research and observation is fascinating.