[Note: This blog had been sitting as a 99% completed draft since early September. I lost track of time and forgot to finish it off then. Since this is still a relevant topic, I am publishing now despite it not being quite as timely in the context of the articles cited in it.]
An article by Kim Zetter in Wired, titled “When Security Experts Gather to Talk Consensus, Chaos Ensues“, talks about a recent meeting that tried to address the decades-old problem of vulnerability disclosure.
This article covers a recent meetings organized by the National Telecommunications and Information Administration (NTIA), a division of the US Commerce Department (DoC). This is not the main reason I am blogging, when most would assume I would speak up on this topic. I’ll give you an odd insight into this from my perspective, then get to the real point. The person who organized this, was virtually introduced to me on July 31, 2015. He replied the same day, and had a great introduction that showed intent to make this a worthwhile effort:
The US Dept of Commerce is trying to help in the old dance between security researchers and system/SW vendors. We think that folks on most sides want a better relationship, with more trust, more efficiency, and better results. Our goal is to bring the voices together in a “multistakeholder process” to find common ground in an open participant-led forum.
I’ve been trying to reach out to many people in this area, on all sides. I know that you’re both veterans of many discussions like this, and I’d like to chat with you about lessons learned, how things may (or may not) have changed, and generally get your insight into how to make this not suck.
That level of understanding from a U.S. government employee is rewarding and encouraging. But, in the interest of fairness and pointing out the obvious, the first thing I asked:
In the past couple of weeks, I have heard mention of the Dept. of Commerce becoming involved in vulnerability disclosure. One thing that is not clear to myself, and many others, is what your intended role or “jurisdiction” (for lack of better words) is?
That is an important question that the DoC must consider, and understand how to respond to.
That question should still be on everyone’s minds, as it has not been answered in a factual manner. Good intentions only get you so far. Having another government player in this mess, who has no power, no jurisdiction, and only “the best of intentions” can only muddy the waters at best. The notes of the ~ 6 hour meeting are now online. I haven’t read them, don’t plan to. This is pure academic masturbation that has escaped from academia, and missed the age old point that so many refuse to admit. “When the $government outlaws chinchillas, only the outlaws will have chinchillas.” Seriously, stupidly simple concept that basically as much the basis for our society as anything else you hold dear.
I’m jumping past that, because a vulnerability tourist said something else that needs addressing. If you haven’t seen me use that term, ‘vulnerability tourist’ refers to someone who dabbles in the bigger world of vulnerability aggregation, disclosure, and the higher-level ideas surrounding it. This doesn’t speak to finding vulns (although it helps), disclosing them (although it helps), or aggregating them for long-term analysis (although it helps). It speaks to someone who thinks they are doing good speaking as some form of expert, but are actually continuing to harm it due to a lack of real knowledge on the topic. When someone who has next to no experience in those realms speaks on behalf of the industry, it becomes difficult to take them seriously… but unfortunately, journalists do. Especially when it comes to hot topics or the dreaded “0day” vulnerabilities, that happen every day… while the media cherry picks the occasional one. As usual in our industry, give such a person a bit of rope and you are likely to find them hanging a few days or weeks later.
Focusing on a single aspect of the Wired article:
Members of the audience snickered, for example, when a representative from the auto industry pleaded that researchers should consider “safety” when testing for vulnerabilities. At a bar after the event, some attendees said automakers are the ones who don’t seem concerned about consumer safety when they sell cars that haven’t been pen-tested for vulnerabilities or when it takes them five years to fix a known vulnerability.
And when Corman sought community support for new companies entering the bug bounty arena, some attendees responded with derision. He noted that after United Airlines launched its bug bounty program this year—the first for the airline industry—it suffered backlash from the security community instead of support.
Just Google “volkswagen emissions software” and you will see why the public cannot trust auto manufacturers. Nothing to do with vulnerability research, and at least one is manipulating their software to defraud the public which may hurt the world environment in ways we can’t comprehend. If that isn’t enough for you, consider the same company spent two years doing everything in their power to hide a vulnerability in their software, that may have allowed criminals to more easily steal consumer’s cars.
So first, ask yourself why anyone from the security arena is so gung-ho in supporting the auto industry. Sure, it would benefit our industry, and more importantly benefit the average consumer. Effecting that change would be incredible! But given the almost three decades of disclosure debate centered on questionable dealings between researchers and vendors, it is not a fight you can win fast. And more importantly, it is not one you capitulate to for your own gain.
Moving past that, the quote from Corman speaks to me personally. When United announced their bug bounty program, it received a lot of media attention. And this is critical to note. A company not known for a bug bounty program, in an industry not known for it, offering perks that were new to the industry (i.e. United, airlines, and frequent flier miles). Sure, that is interesting! Unfortunately, none of the journalists that covered that new bounty program read their terms, or if they did, couldn’t comprehend why it was a horrible program that put researchers at great risk. To anyone mildly familiar with bounty programs, it screamed “run, not walk, away…”. I was one of the more vocal in criticizing the program on social media. When a new bounty opens up, there are hundreds of (largely) neophyte researchers that flock to it, trying to find the lowest hanging fruit, to get the quick and easy bounties. If you have run a vulnerability reporting program, even one without bounties, you likely have witnessed this (I have). I was also very quick to start reaching out to my security contacts, trying to find back-channel contacts to United to give them feedback on their offering.
United’s original bounty program was not just a little misleading, but entirely irresponsible and misleading. It basically said, in laymen’s terms, “if you find a vuln in our site, report it and you may get up to 1 million airline miles!” It also said, you cannot test ANY united.com site, you cannot test our airplanes (on the back of the Chris Roberts / United / FBI drama), our mobile application, or anything else remotely worth testing. The program had a long list of what you could NOT test, that excluded every single target a malicious hacker would target. Worse? It did not offer a test / Dev network, a test airplane, or any other ‘safe’ target to test. In fact, it even excluded the “beta” United site! Uh… what were bounty-seekers supposed to do here? If United thinks that bug bounty seekers read past the “mail bugs to” and “this is the potential bounty”, they need to reconsider their bounty program. The original bounty excluded:
“Code injection on live systems”
Bugs on customer-facing websites such as:
Yet, despite that exclusionary list, they did NOT include what was ALLOWED to be tested. That, in modern security terms, is called a honeypot. Don’t even begin to talk about “intent”, because in a court of law with some 17 year old facing CFAA charges, intent doesn’t enter the picture until way too late. The original United program was set up so that they could trivially file a case with the FBI and go after anyone attacking any of their Internet-addressable systems, their mobile apps, or their airplanes. Shortly after the messy public drama of a white hat hacker telling the media he tested the airplane systems, and could have done “bad things” in so many words.
My efforts to reach out to United via back-channels worked. One of their security engineers that is part of the bounty program was happy to open a dialogue with me. See, this is where we get to the bits that are the most important. We traded a few mails, where I outlined my issues with the bounty program and gave them extensive feedback on how to better word it, so that researchers could not only trust the program, but help them with their efforts. The engineer replied quickly saying they would review my feedback and I never heard back. That is the norm for me, when I reach out and try to help a company. So now, when I go to write this blog, of course I look to see if United revised their bounty program! Let’s look at the current United bug bounty offering:
Bugs that are eligible for submission:
Bugs on United-operated, customer-facing websites such as:
Bugs on the United app
Bugs in third-party programs loaded by united.com or its other online properties
Wow… simply WOW. That is a full 180 from the original program! Not only do they allow testing of the sites they excluded before, but they opened up more sites to the program. They better defined what was allowed (considerably more lenient for testing) as far as technical attacks, and targets. I still disagree a bit on a few things that are “not allowed”, but completely understand their reasons for doing so. They have to balance the safety of their systems and customers with the possibility that a vulnerability exists. And they are wise to consider that a vulnerability may exist.
So after all that, let’s jump back to the quote which drew my ire.
And when [Josh] Corman sought community support for new companies entering the bug bounty arena, some attendees responded with derision. He noted that after United Airlines launched its bug bounty program this year—the first for the airline industry—it suffered backlash from the security community instead of support.
This is why a soundbyte in a media article doesn’t work. It doesn’t help vendors, and it doesn’t help researchers. It only helps someone who has largely operated outside the vulnerability world for a long time. Remember, “security researcher” is an overly vague term that has many meanings. Nothing about that press release suggests Corman has experience in any of the disciplines that matter in this debate. As a result, his lack of experience shows clearly here. Perhaps it was his transition from being a “DevOps” expert for a couple years, to being a “Rugged Software” expert, to becoming a “vulnerability disclosure” expert shortly after?
First, United’s original bug bounty offering was horrible in every way. There was basically zero merit to it, and only served to put researchers at risk. Corman’s notion that scaring companies away from this example is ‘bad’ is irresponsible and contradictory to his stated purpose with the I Am The Cavalry initiative. While Corman’s intentions are good, the delivery simply wasn’t helpful to our industry, the automobile industry, or the airline industry. Remember, “the road to hell is paved with good intentions“.
Keeping up with the latest vulnerabilities — especially in the context of the latest threats — can be a real challenge.
One would hope this article would help with that challenge, and it most certainly is one. First, a disclaimer; I was involved with OSVDB for roughly 10 years and the primary curator over that time. Further, I am now involved with Risk Based Security’s VulnDB commercial vulnerability database offering. Both of these are mentioned in the article, so my comments below will most certainly have some level of bias.
To help readers, Sean Martin writes “In no particular order, here are nine key vulnerability data sources for your consideration.” With that, flip to the next page of the article.
It’s important to understand the source — and backing for your source — to avoid getting left without a solid vulnerability database. A good example is the case where many had to say goodbye to their vulnerability feed when minority-player Open Source Vulnerability Database (OSVDB) was shut down.
“Not having OSVDB any longer, while sad for those that relied on it, may actually reduce the complexity in making sure there is integration across all products, MSSPs, services, and SIEMs,” says Fred Wilmot, chief technology officer at PacketSled
I am not sure how OSVDB constituted a “minority-player” in any sense of the term given the broad coverage for a decade. While historical entries were often incomplete, the database was commercially maintained from just before January, 2012, and the information was still given away for free, despite competing with the company providing the support and updates. Since the quote specifically mentions that OSVDB shut down, and it did on April 5, 2016, it’s nice to hear people give belated appreciation to the project. OSVDB shutting down, I would argue, does not reduce the complexity of anything for those knowledgeable about vulnerability disclosure. On the surface, sure! One less set of IDs to integrate across products sounds like a good thing. However, you have to also remember that OSVDB was cataloging thousands of vulnerabilities a year that were not found in the other sources listed in this article. That means there is a level of complexity here that is horrible for companies trying to keep up with vulnerabilities.
Page 3 tells readers about NIST’s National Vulnerability Database (NVD):
NVD is the US government repository of standards-based vulnerability management data. This data enables automation of vulnerability management, security measurement, and compliance. NVD is based on and synchronized with the CVE List (see next slide).
First, since NVD is synchronized with CVE, it is curious that they are listed as separate sources. For those not aware, NVD is a sort of ‘value add’ to CVE in that they generate CVSS and CPE data for the vulnerabilities cataloged by MITRE for the CVE project. Monitoring NVD means you are already monitoring all of CVE and getting the additional meta-data. It is also important to note that the meta-data is outsourced to a contractor who employs ‘junior analysts’ to do the work. This becomes apparent if you consume there data and actually look at their CVSS scores over the last ~ 8 years. Personally, I stopped emailing them corrections many years back due to the volume involved. To this day, you can still often see them scoring Adobe Flash vulnerabilities as CVSSv2 10.0, meaning they miss the ‘context-dependent’ (a.k.a. ‘user-assisted’) aspect which means the access complexity moves from ‘L’ow to ‘M’edium per the CVSSv2 scoring guide on FIRST, resulting in a 9.3 score. Seems minor, but that reclassifies a vulnerability from ‘Critical’ to ‘High’ for many organizations, and should make you question their scoring on more complex issues.
Page 4 tells us more about CVE and offers some “insight” into it that is horribly wrong:
CVE is a dictionary of publicly known information security vulnerabilities and exposures. CVE’s common identifiers enable data exchange between security products and provide a baseline index point for evaluating coverage of tools and services.
Morey Haber, VP of technology at BeyondTrust, offers these examples:
Scanning tools most commonly use CVEs for classification
SIEM technologies understand their applicability in reporting
Risk frameworks use them as a calculation vehicle for applied risk to the business
First, I cannot over-share Steve Ragan’s recent article titled “Over 6,000 vulnerabilities went unassigned by MITRE’s CVE project in 2015“. Consider just the headline, and then think about the fact that CVE does not catalog at least 47,267 vulnerabilities historically. Now, re-read Haber’s examples of how CVE is used and what kind of Achilles’ heel that is for any organization using security software based on CVE.
Fred Wilmot’s quote about CVE is what prompted me to write this entire blog. This is so incredibly wrong and misleading:
“Now that you have a common calculator for interoperability among vendors, the fact that CVE is maintained completely transparently to the community is a HUGE pro,” says Fred Wilmot, chief technology officer at PacketSled. “There is no holdout of exploits for vulnerabilities based on financial gain or intent. It’s altruism at its best. The weakness in the CVE comes in the weaponization of that information and the lack of disclosure for profit and activism, as two examples.”
Where to start…
- There is no common calculator for “interoperability among vendors” in the context of CVE. That isn’t what CVE is or does.
- CVE is most certainly not maintained transparently to the community. It is not maintained transparently to the volunteer Editorial Board (now known simply as the ‘CVE Board’) either. The backroom workings and decisions MITRE makes on behalf of CVE without Board or public input have been documented before. The last decision that lacked any transparency was their recent catastrophic decision to change the CVE format to a new ‘federated’ scheme. If you have any doubt about this being a backroom decision, look at the first reply from CVE Board member Kurt Seifried.
- Wilmot’s characterization that CVE is “altruism at its best” also speaks to a lack of knowledge of CVE. While MITRE, the organization that maintains CVE, is technically a not-for-profit organization, they only take non-compete contracts at incredible expense to the U.S. taxpayer. CVE, and a handful of other ‘C’ projects related to information security, bring in considerable money to the company. In 2015, they enjoyed over $1.4 billion in revenue and maintained $788 million in assets. The fact that the contract to maintain CVE is non-compete, and cannot be bid on by companies more qualified to run the project, speaks to where the real interest lies and it isn’t altruistic.
- The weakness in CVE is certainly not the “weaponization” of that information. A significant majority of weaponized exploits that lead to the thousands of data breaches and organizations being compromised are typically done with functional exploits that enjoy little technical information being made public. For example, phishing attacks that rely on Adobe Reader or Adobe Flash are usually patched by Adobe eventually, and the subsequent disclosure has no technical details. Even if researchers post more details down the road, the entries in CVE are rarely updated to include the additional details.
- The last bit of Wilmot’s quote, I will need someone to explain to me. “The weakness in the CVE [..] comes in the lack of disclosure for profit and activism.” I don’t know what that means.
Page 5 tells readers about the CERT Vulnerability Notes Database:
The Vulnerability Notes Database provides information about software vulnerabilities. Vulnerability notes include summaries, technical details, remediation information, and lists of affected vendors. Most Vulnerability notes are the result of private coordination and disclosure efforts. For more comprehensive coverage of public vulnerability reports, consider the National Vulnerability Database (NVD).
“This is nice to have, but it still uses CVEs as reference,” says Fred Wilmot, chief technology officer at PacketSled. “NVD is not nearly as practical to consume directly as CVE — the disclosure form is fine, but why would I go there and not directly to MITRE for CVE establishment first? However, it’s probably a good place to spend time during an investigation.”
The CERT VNDB is not a comprehensive vulnerability database, and does not aim to be one. As mentioned, their information is primarily via their assisting researchers in coordinating a disclosure with a vendor. Since CERT is a CNA, meaning they can assign CVE IDs to vulnerabilities they coordinate, it means that over 99% of their entries are covered by CVE and thus NVD. Monitoring NVD will get you all of CVE and almost all of CERT VNDB. The very few CERT VU that do not get CVE IDs assigned before disclosure are rare, and I believe they get assignments shortly after from MITRE.
Once again, Wilmot speaks about these sources and doesn’t appear to have real working knowledge which personifies my term ‘vulnerability tourist’. CERT VNDB disclosures appear on their site before they appear in CVE or NVD. It may be 24 – 72 hours before they appear in fact, meaning that while it still uses CVEs as a reference, for timely monitoring of vulnerabilities it may be important to keep an eye out on CERT directly. Next, Wilmot goes on to say “NVD is not nearly as practical to consume directly as CVE”, apparently not realizing that NVD makes its data available in XML. While MITRE makes the CVE data in several formats, it doesn’t mean NVD is not easy to consume. The most important distinction here is that NVD comes with CPE data where CVE does not. For any medium to large organization, this is basically mandatory meta-data for actually putting the information to use.
Page 6 tells readers about Risk Based Security’s VulnDB offering. The curious bit to me is the quote from Morey Haber:
“VulnDB does not contain audit information, but it is a good source for solutions that need to reference vulnerability information in their products such as firewalls or IDS/IPS and do not want to rely on open source or to build/maintain a library,” said Morey Haber, VP of technology at BeyondTrust.
First, BeyondTrust is not a user of VulnDB, which is a commercial offering unlike CVE/NVD/CERT. They did a short-term trial in 2012, during the beginning of the offering and opted not pursue it as a source of vulnerability intelligence. Second, what does “audit information” even mean in the context of a VDB? Audit information about your own environment maybe? Something that a vulnerability intelligence provider can’t possibly deliver. An audit trail is maintained for each vulnerability entry and is available to customers, but I doubt that is what he means since calling VulnDB out on this doesn’t make sense and the other sources of vulnerabilities listed in this article don’t maintain such a trail.
While VulnDB can certainly be used to reference vulnerabilities in security products as he says, that is the tip of the iceberg. With over 47,000 vulnerabilities not found in CVE or NVD, the breadth of information is incredible. Further, VulnDB has made a concerted effort for years to track vulnerabilities in third-party libraries, and builds on top of the robust meta-data that has been generated for over a decade. Haber’s comments do not reflect actual knowledge of the VulnDB offering.
Page 7 tells readers about the DISA IAVA Database And STIGS. Haber gives commentary on this as well:
“IAVA, the DISA-based vulnerability mapping database, is based on existing SCAP sources, and once in a while it contains details for government systems that are not a part of the commercial world,” says Morey Haber, VP of technology at BeyondTrust. “For any vendor doing .gov or .mil work, this reference is a must.”
While some of the IAVA advisories may contain additional detail, it is important to note that these will not provide any vulnerabilities above and beyond CVE/NVD, and their advisories lag well behind the issues being published in CVE/NVD. Haber is right, that this is a vital resource for .gov and .mil contractors, for several reasons.
Page 8 tells readers about SecurityTracker.com, which is a long-running site that aggregates vulnerabilities to a degree. Haber once again provides commentary:
“The website tends to focus on non-OS vulnerabilities, but they are certainly included in the feed,” says Morey Haber, VP of technology at BeyondTrust. “Infrastructure and IoT tend to make the front page the most, and this site is a good third-party reference for new flaws.”
Actually, they do focus on OS vulnerabilities and that can routinely be seen on their site. As I write this, 2 of the 5 vulnerabilities listed on the front page are in operating systems. The biggest thing to note about this offering is that publicly, they just don’t aggregate a significant volume of vulnerabilities for public consumption. Their most recent ID is 1037091, meaning they have ~ 37,000 entries in their database. Note that they operate like CVE though, where multiple vulnerabilities can be associated with a single ID. Regardless, reading their Weekly Vulnerability Summary emails for the past five weeks shows their volume: Sep 26 2016 (32 alerts), Oct 3 2016 (17 alerts), Oct 10 2016 (26), Oct 17 2016 (31), and Oct 24 2016 (32 alerts). To put that into perspective, VulnDB averages 46 new entries a day in 2016, with the most being 224 in a single day in 2016. For the most part, SecurityTracker may beat CVE on adding to the database, but they are almost entirely covering entries that have CVE IDs.
Page 9 tells readers about the Open Vulnerability And Assessment Language (OVAL) Interpreter And Repository. This is a curious addition to the article because it is not a source of vulnerability intelligence. Instead, it is a standard for reporting about systems. From the OVAL page:
OVAL® International in scope and free for public use, OVAL is an information security community effort to standardize how to assess and report upon the machine state of computer systems. OVAL includes a language to encode system details, and an assortment of content repositories held throughout the community.
While OVAL is certainly useful to some organizations, it does not belong in a list of vulnerability sources.
Page 10 tells readers about Information Sharing And Analysis Centers (ISACs). This is another curious addition as ISACs typically trade information on active attacks, threat actors, and which vulnerabilities may be targeted more heavily. They are generally not a source of vulnerabilities in the same context as most of the resources above.
In summary, Sean Martin’s article says it will share “9 Sources For Tracking New Vulnerabilities”. In reality, based on that quote and context, the article only tells readers about CVE/NVD, CERT VNDB, RBS VulnDB, and SecurityTracker. Several of the sources listed are not really for tracking new vulnerabilities, rather augment the vulnerability threat intelligence in various ways.
Earlier this week, Michael Roytman of Kenna Security wrote a blog with more details about the vulnerability section of the Verizon DBIR report, partially in response to my last blog here questioning how some of the data was generated and the conclusions put forth. The one real criticism I will note, is that Roytman’s blog does not acknowledge or warn that the list of CVE IDs included in the DBIR had typos, causing the wrong IDs to be included. In the world of vulnerability databases, that unique ID is designed specifically to avoid such confusion. Carrying the wrong IDs undermines the integrity of the data being presented.
In addition to my comments, Roytman had a long call with Adrian Sanabria which led to generating a new set of data with a different scope. From the Kenna blog:
We had an excellent offline discussion in which he dove deeply into the assumptions of my work, asked thoughtful, deep questions in private, and together, we came up with a better metric for generating a top 10 vulnerabilities list. To address these issues, I scaled the total successful exploitation count for every vulnerability in 2015 by the number of observed occurrences of that vulnerability in Kenna’s aggregate dataset. Sifting through 265 million vulnerabilities gives us a top 10 list perhaps more in line with what was expected – but equally unexpected! The takeaway here is that datasets like the one explored in the DBIR might be noisy, might have false positives and the like, but carefully applied to your enterprise the additional context successful exploitation data lends to vulnerability management is priceless.
I won’t go into much detail in this blog, but will say that I disagree with the statement that severely flawed data can produce takeaways that are “priceless”. Organizations acting on these top 10 lists may be spending time and resources chasing vulnerabilities that do not impact them, or pose very little risk compared to other threats they face. That action most certainly has a price; one that can be enumerated to some degree due to the cost of the employee time required. With that in mind, let’s look at the methodology which is spelled out in more detail than the DBIR, before we consider the new top 10 list Roytman generated. First, his notes on the data examined:
The first is a convenience sample that includes 2,442,792 assets (defined as: workstations, servers, databases, ips, mobile devices, etc) and 264,912,235 vulnerabilities associated to those assets. The vulnerabilities are generated by 8 different scanners, they are: Beyond Security, Tripwire, McAfee VM, Qualys, Tenable, BeyondTrust, Rapid7, and OpenVAS . This dataset is used in determining remediation rates and the normalized open rate of vulnerabilities.
I am curious if it is normal practice to consider an IP address an asset, when the system that addresses the IP is largely considered the asset. Moving past that, one point that sticks in my mind is the tools that generate the data. From the list above, consider that Beyond Security claims to have “what is arguably the world’s most complete database of security vulnerabilities.” Click around their site and you see that “the AVDS database includes over 10,000 known vulnerabilities and the updates include discoveries by our own team and those discovered by corporate and private security teams around the world.” That is less than 25% of what CVE has, and less than 10% of what VulnDB has. They even show you that they only cover 200 CVE IDs for 2016, as compared to the 1,474 open 2016 CVEs.
They don’t specify which Tripwire product, include McAfee Vulnerability Manager which was declared End of Life in October last year, and don’t specify which Qualys product. So it is a start as far as explaining what tools generate the data, but still leaves a lot of guess-work.
Roytman describes the second data set used as:
The second is a convenience sample that includes 3,615,706,022 successful exploitation events which all take place in 2015 which come from Dell Secureworks’ Counter-Threat Unit and Alienvault’s Open Threat Exchange.
The third qualification, describing the methodology is perhaps the most important, and was lacking in the DBIR:
Please note the methodology of data collection: Successful Exploitation is defined as one successful technical exploitation of a vulnerability on one machine at a particular timestamp. The event is defined as: 1. An asset has a known CVE open. 2. An attack come in that matches the signature for that CVE on that asset and 3. One or more IOCs are detected/correlated post attack. It is not necessarily a loss of a data, or even root on a machine. It is just the successful use of whatever condition is outlined in a CVE.
I have reached out to Michael and requested a sampling of data for two of the CVE IDs on his new list, and he is going to do so. In the meantime, I had a discussion with several people more familiar with IDS than myself and asked about how they would detect attacks for CVE-2013-0229 and CVE-2001-0540 as an example. Sure, detecting a specific type of packet meant to trigger this is one thing, but what is the threshold for the IDS to say it is an attack, when exploitation requires “a large number of malformed Remote Desktop Protocol (RDP) requests“. Is there a specific number of packets for it to flag an attack in progress? If too low, it may be prone to a high number of false positives. If too high, it may not detect a successful exploitation of the issue. Which leads into the second part of the methodology, comparing it with “one or more IOCs [that] are detected/correlated post attack”. In this case, it would presumably be the targeted service not responding, which could be detected a number of ways (e.g. probing the port, seeing specific errors in the logs, noticing a given process not running). Once the service is down, is the IDS that isn’t aware of the host state still cataloging a single attack? Or is it generating alerts every X minutes it detects an attack ongoing? Hopefully the data Michael sends will help me better understand how that correlation is being made, as it represents a source of incredible bias for the resulting data analysis.
Moving on to Roytman’s new list using the above data and methodology, here is the top 10 list he sees in the data:
- 2015-03-05 – 2015-1637 – Microsoft Windows Secure Channel (Schannel) RSA Temporary Key Handling EXPORT_RSA Ciphers Downgrade MitM (FREAK)
- 2015-01-06 – 2015-0204 – OpenSSL RSA Temporary Key Handling EXPORT_RSA Ciphers Downgrade MitM (FREAK)
- 2014-04-07 – 2014-0160 – OpenSSL TLS Heartbeat Extension Packets Handling Out-of-bounds Read Remote Memory Disclosure (Heartbleed)
- 2012-03-13 – 2012-0152 – Microsoft Windows Remote Desktop Protocol Terminal Server RDP Packet Parsing Remote DoS
- 2009-05-16 – 2013-0229 – MiniUPnPd SSDP Handler minissdp.c ProcessSSDPRequest Function Malformed Input Handling Remote DoS
- 2002-02-12 – 2002-0012 – Multiple Vendor Malformed SNMP Trap Handling DoS
- 2002-02-12 – 2002-0013 – Multiple Vendor Malformed SNMP Message-Handling Remote DoS
- 2001-12-20 – 2001-0876 – Microsoft Windows Universal Plug and Play NOTIFY Directive URL Handling Remote Overflow
- 2001-12-20 – 2001-0877 – Microsoft Windows Universal Plug and Play NOTIFY Request Remote DoS
- 2001-07-25 – 2001-0540 – Microsoft Windows Terminal Server RDP Request Handling Memory Exhaustion Remote DoS
Roytman prefaces that list with a comment that the “top 10 list perhaps more in line with what was expected – but equally unexpected!” Indeed, that is certainly true. Expected? A high-profile vulnerability disclosed in 2014 that was seen being widely exploited then, and subsequent years, something sorely missing from the DBIR list even after the corrected CVE IDs were factored in. This is the type of vulnerability almost everyone expected to top the list due to how easy it was to exploit, and knowing that it had been used heavily. This speaks to the original point and reason for the first blog; all the data ‘science’ in the world that produces highly questionable results should not be taken as gospel. Even if the methodology was sound, it doesn’t mean the data being used was.
Unfortunately, Roytman’s revised list deviates quickly into the “equally unexpected“. The first DBIR report list had a single denial of service (DoS) attack on the list, which stood out as odd. The revised list bumped that number to two which was a bit odd. The most recent list expands that to six DoS attacks, which is highly questionable for one reason or another. First, you can question the data set and methodology leading to this conclusion, but let’s say that passes muster solely for the sake of argument. Second, you can question why so many denial of service attacks are on a top 10 most exploited vulnerabilities list, framed in a context that uses the word ‘compromise’, riding on the back of a report centered around data breaches. These attacks are not causing attackers to gain privileges or steal data. They are a nuisance most of the time, or potentially used in serious DoS attacks other times. It makes one question why DoS attacks weren’t dropped from the results completely, and disclaimed as such! Or, generate two lists; one with results based on raw data, DoS and everything, and a second list that focuses on vulnerabilities that may allow for privileges to be gained and a real compromise to happen.
I cannot stress this enough. Using a term like ‘Indicator of Compromise’ (IOC) in the context of DoS attacks is disingenuous and misleading. Going back to Roytman’s introduction into this section where he makes a comment about seeing the trees (referring to the classic metaphor), I find it ironic as that sums up the purpose of my blog. The DBIR was written as if they could only see the trees (data points), and not the forest (bigger picture), which is what many people took issue with.
One point that I overlooked on the original list, and still appears on the new list, is the presence of FREAK (twice even). Fortunately for me, Thomas Ptacek does a great job explaining why FREAK is likely on the list, but absolutely should not be. Using Roytman’s blog and data, he calculates that attackers would have spent $332,183,325 using Amazon EC2 to exploit FREAK. He continues by citing one of the researchers who discovered FREAK explaining one way that a high number of false positives are generated on that particular vulnerability. He goes on to drive the point home, quoting the researcher and commenting that it likely has not been exploited in the wild by an average attacker.
Roytman essentially dismisses all of this in his blog post while saying that I am correct, that “IDS alerts generate a ton of false positives, vulnerability scanners often don’t revisit signatures, CVE is not a complete list of vulnerability definitions. But those are just the trees, and we’ll get to them later.” Unfortunately, he doesn’t get to it later in a way that provides meaningful insight into the questions about the data and conclusions. Dan Guido wrote an excellent summary of why the DBIR vulnerability section has issues, and factors in Roytman’s latest blog, breaking it all down in a manner that highlights the flaws. Even with the revised list, it is still missing the US-CERT top 30 previously cited, the Microsoft data, and the recently disclosed ‘top PoC exploits distributed on social media‘. At some point, one would logically conclude these lists should have more overlap. One thing I would love to see from Verizon and Kenna is a detailed explanation of their methodology as it relates to detecting client-side exploits, that appear to be the defacto standard for infecting tens, maybe hundreds of thousands of hosts, every year to create botnets.
I want to look at this from one more perspective, because I think it beautifully highlights how vulnerability analysis is a moving target, but in this case for all the wrong reasons. While most vulnerability aggregators and analysts are constantly adapting to new variations of vulnerabilities, new sources of vulnerability information, and new players in the game with wildly different styles of disclosure, others that come along after the data is generated and do analysis frequently seem to lose perspective in my experience. I believe this is such a case and is best illustrated in what the DBIR top 10 looks like over three revisions in less than two weeks. Yes, I know Roytman’s list isn’t officially the DBIR list, but he generated the initial data and then opted to do a different form of analysis putting it forth as something that is more representative due to applying his analysis to a more limited dataset that he presumably trusts more (i.e. the Kenna aggregate dataset). One has to wonder if this was brought up to Verizon as a better way to approach the list, and if so, why was it rejected.
The following lists show the evolution of the CVEs that appear on the top 10, with strikeout denoting the typos between the original DBIR and Gabe’s clarification which is reflected in current downloads, underlining to show any denial of service attacks, and bold to show the new CVE IDs that appear with Roytman’s reworking.
DBIR - DBIR Revised - Roytman Blog 2015-1637 – 2015-1637 - 2015-1637 2015-0204 – 2015-0204 - 2015-0204 20
12-1054 – 2002-1054 - 2014-0160 20 11-0877 – 2001-0877 - 2012-0152 2003-0818 – 2003-0818 - 2013-0229 2002-0126 – 2002-0126 - 2002-0012 2002-0953 – 2002-0953 - 2002-0013 2001-0876 – 2001-0876 - 2001-0876 2001-0680 – 2001-0680 - 2001-0877 1999-1058 – 1999-1058 - 2001-0540
In my mail to Roytman asking for a sample data set, I suggested that it would be interesting to see him generate a list using his methodology, but remove any DoS attack (six of his ten), so the top list only included exploits that could achieve remote privileges of some sort. He replied to me with:
… and again, awesome idea. One of those all-too-simple in retrospect, damnit why didn’t I think of it earlier things.
I found this interesting thinking back to his use of the forest and the trees metaphor.
Verizon released their yearly Data Breach Investigations Report (DBIR) and it wasn’t too long before I started getting asked about their “Vulnerabilities” section (page 13). After bringing up some highly questionable points about last year’s report regarding vulnerabilities, several people felt that the report did not stand up to scrutiny. With a few questions leveled at me, I was curious if Verizon and partners learned from last year.
This year’s vulnerability data was provided by Kenna Security (formerly Risk I/O), and Verizon “also utilized vulnerability scan data provided by Beyond Trust, Qualys and Tripwire in support of this section.” So the data isn’t from a single vendor, but at least four vendors, giving the impression that the data should be well-rounded, and have less questions than last year.
From the report:
Secondly, attackers automate certain weaponized vulnerabilities and spray and pray them across the internet, sometimes yielding incredible success. The distribution is very similar to last year, with the top 10 vulnerabilities accounting for 85% of successful exploit traffic. While being aware of and fixing these mega-vulns is a solid first step, don’t forget that the other 15% consists of over 900 CVEs, which are also being actively exploited in the wild.
This is not encouraging, as they have 10 vulnerabilities that account for an incredible amount of traffic, and the footnoted list of CVE IDs suggests the same problems as last year. And just like last year, the report does not explain the methodology for detecting the vulnerabilities, does not include details about the generation of the statistics, and provides a loose definition of what “successfully exploited” means. Without more detail it is impossible for others to reproduce their results, and extremely difficult to explain or disclaim them as a third party reading the report. Going to the Kenna Security page about this report doesn’t really yield much clarity, but does highlight another potential flaw in the methodology:
Kenna’s Chief Data Scientist Michael Roytman was the primary author of this year’s “Vulnerabilities” chapter, analyzing a correlated threat data set that spans 200M+ successful exploitations across 500+ common vulnerabilities and exposures from over 20,000 enterprises in more than 150 countries.
It’s subtle, but notice they went through a data set that spans exploitations across “500+ common vulnerabilities and exposures”, also known as CVE. If the data is only looking for CVEs, then there is an incredibly large bias at play from the start, since they are missing at least half of the disclosed vulnerabilities. More importantly, this becomes a game of fractions that the industry is keen to overlook at every opportunity:
- CVE represents approximately half of the disclosed vulnerabilities.
- Vulnerability scanners and IPS/IDS don’t have signatures for all CVE IDs, so they look for some fraction of CVE.
- Detection signatures are often flawed, leading to false positives and false negatives, meaning they are actually detecting a fraction of the CVE IDs they intend to.
Another crucial factor in how this data is generated is in the detection of the exploits. Of the four companies contributing data, one was founded in 2009 (Risk I/O / Kenna) and another in 2006 (Beyond Trust, in the context of this discussion). That leaves Qualys (founded 1999) and Tripwire (founded 1997), who are likely the sources of the signatures that detected the vulnerabilities. For those around in the late 90s, the vulnerability landscape was very different than today, and security products based on signatures back then are in some ways considered rudimentary compared to today. Over time, most security products do not revisit older signatures to improve them unless they have to, often due to customer demand. Newly formed companies basically never go back and write signatures for vulnerabilities from 1999. So it stands to reason that the detection of these issues are based on Qualys and/or Tripwire’s detection capabilities, and the signatures detecting these vulnerabilities are likely outdated and not as well-constructed as compared to their more recent signatures.
That leads us to ask, how many vulnerabilities are these companies really looking for? Where did the detection signatures originate and how accurate are they? While the DBIR does disclaim that the data used is a sample, they also admit “bias undoubtedly exists”. However, they don’t warn the reader of these extremely limiting caveats that put the entire data set into a perspective clearly showing strong bias. This, combined with the lack of detailed methodology for how these vulnerabilities are detected and correlated to measure ‘success’, ultimately means this data has little value other than for inclusion in pedestrian reports on vulnerabilities.
With that in mind, I can only go by what information is available. We’ll start with the concise list of the top 10 CVE IDs these four vulnerability intelligence providers say are being exploited the most, and Verizon labels as “successfully exploited”:
- 2015-03-05 – CVE-2015-1637 – Microsoft Windows Secure Channel (Schannel) RSA Temporary Key Handling EXPORT_RSA Ciphers Downgrade MitM (FREAK)
- 2015-01-06 – CVE-2015-0204 – OpenSSL RSA Temporary Key Handling EXPORT_RSA Ciphers Downgrade MitM (FREAK)
- 2012-02-25 – CVE-2012-1054 – Puppet k5login File Symlink File Overwrite Local Privilege Escalation
- 2011-07-19 – CVE-2011-0877 – Oracle Enterprise Manager Grid Control Instance Management Unspecified Remote Issue (2011-0877)
- 2004-02-10 – CVE-2003-0818 – Microsoft Windows ASN.1 Library (MSASN1.DLL) BER Encoding Handling Remote Integer Overflows
- 2002-01-15 – CVE-2002-0126 – BlackMoon FTP Server Multiple Command Remote Overflow
- 2001-12-26 – CVE-2002-0953 – PHPAddress globals.php LangCookie Parameter Remote File Inclusion
- 2001-12-20 – CVE-2001-0876 – Microsoft Windows Universal Plug and Play NOTIFY Directive URL Handling Remote Overflow
- 2001-04-13 – CVE-2001-0680 – QVT/Net / Term FTP Server LIST Command Traversal Remote File Access
- 1999-11-22 – CVE-1999-1058 – Vermillion FTPD Long CWD Command Handling Remote Overflow DoS
This list should raise serious red flags for anyone passingly familiar with vulnerabilities. Not only do we have very odd ‘top 10’ lists from last year and this year, but there is little overlap between them. How does 2015 show a top 10 list exploiting eight vulnerabilities with CVE identifiers between 1999 and 2002, meaning they had been exploited so much as many as thirteen years later, only to see them all drop off the list this year, to be replaced by new 15+ year old vulnerabilities? In addition to this oddity, there are more considerations leading to my top 10 list of questions about their list:
- How does a local vulnerability based on a symlink overwrite flaw (CVE-2012-1054) make it into a top 10 list of “85% of successful exploit traffic“?
- How does a local vulnerability in Puppet rank #3 on this list, given the install base of Puppet as compared to Adobe or Java?
- If they are detecting exploits on the wire, shouldn’t we see Java, Adobe Reader, and Adobe Flash somewhere on the list? The “Slow and steady—but how slow?” section even talks about time-to-exploit for Adobe.
- Why doesn’t this list remotely match US-CERT’s “Top 30 Targeted High Risk Vulnerabilities” that includes vulnerabilities back to 2006, but not a single one listed above?
- How does a vulnerability that by all accounts is so vague, that it has to be distinguished by the vendor issued CVE ID (CVE-2011-0877), have a signature and get exploited so much?
- How does a vulnerability in Oracle Enterprise Manager Grid Control show up as #4, when no Oracle Database vulnerabilities appear?
- How do you distinguish an FTP LIST command exploit from one vendor to another? (e.g. CVE-2005-2726, CVE-2002-0558, CVE-2001-0933, CVE-2001-0680) According to the one-liner methodology, this is done via pairing SIEM data, suggesting that BlackMoon and Vermillion are that popular today.
- Yet, how does a remote DoS in an Windows-based FTP program that doesn’t appear to have been distributed for a decade make it on this list? Are people really conducting targeted DoS attacks against this software?
- Is BlackMoon FTP Server really that prevalent to be exploited so often?
- Or is there a problem in generating this data, which would be more easily attributed to loose signatures detecting FTP attacks regardless of vendor?
Figure 12 in the report, which is described as “Count of CVEs exploited in 2015 by CVE publication date” is a curious thing to include, as the CVE publication date is very distinct from the vulnerability disclosure date. While a large percentage of CVE publication dates are within seven days of disclosure, many are not (e.g. CVE-2015-8852 disclosed 2015-03-23 and CVE publication on 2016-04-26). Enough to make this chart questionable as far as the insight it provides. Taking the data as presented, are they really saying that only ~ 73 vulnerabilities with a 2015 ID were successfully exploited in 2015 across “millions of successful real-world exploitations“? Given that 40 vulnerabilities were discovered in the wild, 33 of which have 2015 CVE IDs, that means that only ~40 other 2015 vulnerabilities were successfully exploited? If that is the takeaway, how is the security industry unable to stop the increasing wave of data breaches, the same kinds that led to this report? Something doesn’t add up here.
While people are cheering about the DBIR disclaiming there is sample bias (and not really enumerating it), they ignore the measurement bias, don’t speak to publication bias, don’t explain the attrition bias between 2015 and 2016, or potential chaining bias. As usual, the media is happy to glom onto such reports without asking any of these questions or providing critical analysis. As an industry, we need to keep challenging metrics and statistics to ensure they are not only accurate, but provide meaning that benefits us.
4/28/2016 – According to Gabe (@gdbassett), the list of CVEs in the DBIR is incorrect. He posted a new list of CVEs (mostly the same) via Twitter in a reply to Andreas Lindh who was surprised at the top ten list of vulnerabilities as well. Gabe also confirmed that afterwards they “compared the figure CVEs (listed above) against the raw data. After removing non-confirmed breaches, they match.” He went on to link to another source showing “data” about one of the CVEs, that really doesn’t mean anything without more context. Meanwhile, Michael Roytman, who did the vulnerability section of the report confirmed that he/Kenna would be responding to this blog with one of their own.
I hate to harp on simple transposition style mistakes in a report, but given the severity behind using numeric identifiers for vulnerabilities, it seems like that should have been double and triple-checked. Even then, I don’t understand how someone familiar with vulnerabilities could see either list and not ask many of the same questions I did, or provide more information in the report to back the claims. That said, let’s look at Gabe’s new list of CVEs. Bold and links are used to highlight the new ones:
- 2015-03-05 – 2015-1637 – Microsoft Windows Secure Channel (Schannel) RSA Temporary Key Handling EXPORT_RSA Ciphers Downgrade MitM (FREAK)
- 2015-01-06 – 2015-0204 – OpenSSL RSA Temporary Key Handling EXPORT_RSA Ciphers Downgrade MitM (FREAK)
- 2004-02-10 – 2003-0818 – Microsoft Windows ASN.1 Library (MSASN1.DLL) BER Encoding Handling Remote Integer Overflows
- 2002-07-22 – 2002-1054 – Pablo FTP Server LIST Command Arbitrary Directory Listing Remote Information Disclosure
- 2002-01-15 – 2002-0126 – BlackMoon FTP Server Multiple Command Remote Overflow
- 2001-12-26 – 2002-0953 – PHPAddress globals.php LangCookie Parameter Remote File Inclusion
- 2001-12-20 – 2001-0877 – Microsoft Windows Universal Plug and Play NOTIFY Request Remote DoS
- 2001-12-20 – 2001-0876 – Microsoft Windows Universal Plug and Play NOTIFY Directive URL Handling Remote Overflow
- 2001-04-13 – 2001-0680 – QVT/Net / Term FTP Server LIST Command Traversal Remote File Access
- 1999-11-22 – 1999-1058 – Vermillion FTPD Long CWD Command Handling Remote Overflow DoS
The list gained another FTP server issue that doesn’t necessarily lead to privileges, and another remote denial of service attack, while losing the Puppet symlink (CVE-2012-1054) and the vague Oracle Enterprise Manager issue (CVE-2011-0877). All said and done, the list is just as confusing as before, perhaps more so. That gives us four FTP vulnerabilities, only one of which leads to remote code execution, and two denial of service attacks, that gain no real privileges for an attacker. As Andreas Lindh points out, that I failed to highlight, having a man-in-the-middle vulnerability occupy two spots on this list is also baffling in the context of the volume of attacks stated. Also note that with the addition of CVE-2002-1054 (Pablo FTP), there are now two vulnerabilities that appear on the DBIR 2015 and DBIR 2016 top ten CVE list.
Hopefully the forthcoming blog from Michael Roytman will shed some light on these issues.
As of today, a decision has been made to shut down the Open Sourced Vulnerability Database (OSVDB), and will not return. We are not looking for anyone to offer assistance at this point, and it will not be resurrected in its previous form.This was not an easy decision, and several of us struggled for well over ten years trying to make it work at great personal expense. The industry simply did not want to contribute and support such an effort. The OSVDB blog will continue to be a place for providing commentary on all things related to the vulnerability world.
Today, The Register wrote an article on MITRE’s announcement of a new CVE ID scheme, and got many things wrong about the situation. As I began to write out the errata in an email, someone asked that I make it public so they could learn from the response as well.
From The Register:
The pilot platform will implement a new structure for issuance of Common Vulnerabilities and Exposures numbers. MITRE will directly issue the identifiers and bypass its editorial board.
The old system bypassed the Editorial Board. The Board isn’t there to give input on assignments, not for over 10 years. The criticism right now is that the new system was rolled out without consulting the Board, which the board is supposedly there for:
The MITRE Corporation created the CVE Editorial Board, moderates Board discussions, and provides guidance throughout the process to ensure that CVE serves the public interest.
The last ID scheme change was after more than a year of discussion, debate, and ultimately a vote by the board (11/2/2012 Start –> 6/7/2013 End). This ID scheme change was a knee-jerk response to recent criticism (only after it became more public than the Editorial Board list), without any board member input. From The Register article:
The CVE numbers are the numerical tags assigned to legitimate verified bugs that act as a single source of truth for security companies and engineers as they seek to describe and patch problems.
No… CVE IDs are assigned to security issues, period. Alleged, undetermined, possible, or validated. Many IDs are assigned to bogus / illegitimate issues, which are generally discovered after the assignment has been made. There is no rule or expectation that IDs are only assigned to legitimate verified bugs. To wit, search the CVE database for the word “REJECTED”.
More importantly, the notion it is a “single source of truth” is perhaps the worst characterization of CVE one could imagine. Many companies use alternate vulnerability databases that offer more features, better consumption methods, and/or much better coverage. With MITRE falling behind other VDBs by as many as 6,000 vulnerabilities in 2015 alone, it isn’t a single source for anything more than training wheels for your vulnerability program.
The platform will exist alongside the current slower but established CVE system.
At this point, with what has been published by MITRE, this is pure propoganda in line with the platitudes they have been giving the Editorial Board for the last year. The new format does not make things easier for them, in any way. The ‘old ID scheme’ was never the slowdown or choke point in their process. In fact, read their press release and they say they will be the only ones issuing federated IDs, meaning the same problem will happen in the interim. If it doesn’t, and they start issuing faster, it wasn’t the new scheme that fixed it. More important to this part of the story is that it wasn’t just about assignment speed. The last six months have seen MITRE flat out refuse assignments to more and more researchers, citing the vulnerable software isn’t on their list of monitored products. Worse, they actually blame the Editorial Board to a degree, which is a blatant and pathetic attempt at scape-goating. The list they refer to was voted on by the board, yes. But the list given to the board was tragically small and it was about arranging the last chairs on the Titanic so to speak.
He says MITRE is aiming for automated vulnerability identification, description, and processing, and welcomed input from board members and the security community.
This is amusing, disgusting, and absurd. MITRE, including the new person in the mix (Sain) have NOT truly welcomed input from the board. The board has been giving input, non-stop, and MITRE has been ignoring it every single time. Only after repeated mails asking for an update, do they give us the next brief platitude. Sain’s title, “MITRE CVE communications and adoption lead”, is also ironic, given that just last week Sain told the board he would contact them via telephone to discuss the issues facing CVE, and never did. Instead, there was no communication, and MITRE decided to roll out a scheme that was horribly designed without any input from the Board or the industry.
For those who want a better glimpse into just how bad MITRE is handling the CVE project, I encourage you to read the Editorial Board traffic (and hope it is updated, as MITRE still manually runs a script to update that archive).
Risk Based Security released the VulnDB QuickView report that shows 2015 broke the previous all-time record for the highest number of reported vulnerabilities. The 14,185 vulnerabilities cataloged during 2015 by Risk Based Security eclipsed the total covered by the National Vulnerability Database (NVD) and CVE by over 6,000.
Risk Based Security’s newly released 2015 Year End VulnDB QuickView report shows that 20.5% of reported vulnerabilities received CVSS scores between 9.0 and 10.0 and the number of vulnerabilities and the CVSS scores are both trending higher over the last four years.
It comes as no real surprise that Web-related vulnerabilities account for nearly 60% of the total reported in 2015, with cross-site scripting (XSS) making up 39% of those.
The VulnDB QuickView report also revealed that vulnerabilities disclosed in a coordinated fashion with the vendor rose to 42% in 2015 compared to 28% in 2014, the previous record. Another interesting fact in the report is that third-party Bug Bounty programs outpaced Vendor managed bounty programs 4:1 in 2015, when details were made available to the public.
David Weinstein, a researcher at NowSecure, has posted a blog titled “Ruminations on App CVEs“. Thanks to Will Dormann’s Tweet it came to our attention, and he is correct! We have opinions on this. Quoted material below is from Weinstein’s blog unless otherwise attributed.
CVE is well-positioned to play a critical role in tracking the risk level of mobile computing.
This is trivially to debate and counter in 2015. Until last year, CVE maintained a public list of sources that was virtually unchanged since its inception in 1999. After pressure from the public and the CVE Editorial Board, compromised of non-MITRE industry “advisors”, CVE revised their coverage policy and shifted to a new system of ‘full’ or ‘partial’ coverage based on the vendor, product, and/or vulnerability source. On the surface, this list looks promising, but upon any significant scrutiny, it is utterly lacking in adequate coverage. In order to head off complaints, their webpage even qualifies ‘full coverage’ sources by using the phrasing “nearly all issues disclosed” to “allow the flexibility to potentially postpone coverage of minor issues”.
In fact, Steve Christey may disagree with your assertion. In 2013, he posted to the VIM list saying:
Unfortunately, CVE can no longer guarantee full coverage of all public vulnerabilities… we are not well-prepared to handle the full volume of CVEs for all publicly-disclosed vulnerabilities… – Steve Christey, Editor of CVE
Given the 39,280 vulnerabilities we track that do not have a CVE assignment, assignment volume is specifically a problem for them. Further, the current state of CVE assignments is a disaster, and I have been in mails with CVE staff actively encouraging them to figure out the problem, as my now 27 day wait for three different CVE assignments is getting ridiculous.
However, CVEs have largely focused on tracking server-side and related flaws and yet the security community has evolved to track client-side vulnerabilities as a critical aspect of dealing with risk.
While I can’t offer you a set of handy statistics to back this claim, know that CVE, or their CNAs, assign identifiers to a considerable number of client applications. In fact, with the release of Apple iOS 9 yesterday, the public learned of 91 _new_ vulnerabilities affecting the platform, many of them client-side. I do not believe this statement to be a fair criticism of CVE’s assignment history and policy.
While CVE has worked well for tracking system flaws in mobile operating systems, there remain quirks with regard to mobile app flaws.
There are around 90 vulnerabilities in Google Android alone, that do not have CVE assignments, and that represents a fairly significant percentage overall. However, in this case we can’t blame CVE at all for this shortcoming, as the fault lies entirely with Google instead.
It seems like using CVE is the “right thing” here but MITRE has been very cryptic (in my opinion) about what qualifies as something that gets a CVE and what doesn’t. These discussions should not happen in some closed door meeting in my opinion, or at least should not end there…
First, please factor in that CVE and every other vulnerability database has gone through a world of headache, dealing with researchers who do not fully understand the concept of ‘crossing privilege boundaries’. The number of “not-a-vuln” vulnerability reports we see has skyrocketed this year. Further, even from individuals that are very technical, don’t always convey their information in terms that are easily understood. They clearly see a vulnerability, the report they send doesn’t show it to our eyes. It requires back-and-forth to figure it out and better understand the exact vulnerability. Bottom line, some are clear-cut cases and typically get a quick assignment. Other cases are murky at best.
Second, I fully agree that such a discussion should not be behind closed doors. At the very least, it should include the CVE editorial board, and ideally should include anyone in the industry who has an interest in it. If you want to bring up the discussion on the VIM mail list, which has over one hundred members, including at least one person from each of the major public vulnerability databases, please do.
Is MITRE consistently responding to other researchers’ requests for CVEs?
No. The turnaround time for requesting a CVE has gone up considerably. As mentioned above, I have been waiting for three assignments for almost one month now, while CVE assigned two others filed on the same day within hours. They are not currently consistent with the same researcher, even one familiar with CVE and their abstraction and assignment policy.
Attaching a CVE to a vulnerable app, even if it’s an old version, is actually a big part of tracking the reputation of the developer as well!
Just quoting to signal-boost this, as it is a very important comment, and absolutely true.
Who acts as a certified numbering authority (CNA) on behalf of all the app developers on the market?
Based on the current list of CNAs, there is no CNA that covers third-party apps really, regardless of platform. Also note that many CNAs are not currently fulfilling their duties properly, and I have been trying to address this with MITRE for months. There is no current policy on filing a complaint against a CNA, there is no method for warning or revoking CNA status, and MITRE has dropped the ball replying to me on several different threads on this topic.
Should Google step in here as a formal body to assist with coordinating CVEs etc for these apps?
Absolutely not. While they are positioned to be the ideal CNA, their history in CVE assignments has been dismal. Further, they have so many diverse products, each with their own mechanism for bug reporting (if one exists), own disclosure policy (if one exists), and disclosure method (which doesn’t exist for many). We can’t rely on Google to disclose the vulnerabilities, let alone act as a CNA for standardized and formalized ID assignment. Could Google fix this? Absolutely. They are obviously overflowing with brilliant individuals. However, this would require a central security body in the company that crosses all departments, and fulfills such a duty. Due to the volume of software and vulnerabilities they deal with, it would require a small team. The same kind that IBM, Cisco, and other large companies already stand up and have used for years.
Is the process MITRE established designed and prepared to handle the mountain of bugs that will be thrown at it when the community really focuses on this problem as we have?
No, as covered above. CVE is already backpedaling on coverage, and has been for a couple years. That does not give any level of comfort that they would be willing to, or even could handle such a workload.
Can the community better crowd source this effort with confirmation of vulnerability reporting in a more scalable and distributed manner that doesn’t place a single entity in a potentially critical path?
Possibly. However, in the context of mobile, Google is a better choice than crowd-sourcing via the community. The community largely doesn’t understand the standards and abstraction policy that helps define CVE and the value. Many CNAs do not either unfortunately, despite them being in the best position to handle such assignments. Moving to a model that relies on more trusted CNAs has merit, but it would also require better documentation and training from MITRE to ensure the CNA follows the standards.
Every vendor distributing an app on the Play Store should be required to provide a security related contact e.g., email@example.com
Except, statistically, I bet that not even half of them have a domain! The app stores are full of a wide variety of applications that are little more than a hobby of one person. They don’t get a web page or a formal anything. Given the raw number of apps, that is also a tall order. Maybe apps that enjoy a certain amount of success (via download count?) would then be subjected to such rules?
Google could be a little clearer about when firstname.lastname@example.org should be used for organized disclosure of bugs and consider taking a stronger position in the process.
“Little clearer” is being more than generous. Remember, until a month or two ago, they had no formal place or process to make announcements of security updates. Disclosures related to Android have been all over the board and it is clear that Google has no process around it.
MITRE and/or whoever runs CVE these days should clarify what is appropriate for CVEs so that we know where we should be investing our efforts
Absolutely, and CVE’s documentation the last decade or more has not been properly updated. Having such documentation would also potentially cut down on their headache, as requests are made for issues clearly not a vulnerability. But newcomers to security research don’t have a well-written guide that explain it.
Should we pursue a decentralized CVE request process based on crowdsourcing and reputation?
I cannot begin to imagine what this would entail or what you have in mind. On the surface, won’t happen, won’t work.
Hopefully others in the industry chime in on this discussion, and again, I encourage you to take it to the CVE Editorial Board and the VIM list, to solicit feedback from more. I appreciate you bringing this topic up and getting it attention.
This post is in response to the drama the last few days, where Mary Ann Davidson posted an inflammatory blog about security researchers that send Oracle vulnerabilities while violating their End-user License Agreement (EULA… that thing you click without reading for every piece of software you install). The post was deleted promptly by Oracle, then Oracle said it was not the corporate line, and due to the crazy journalists who of course felt obligated to cover. You can read up on the background elsewhere, because it has absolutely no bearing on reality, which this very brief blog covers.
This is such an absurdly simple concept to grasp, yet the CISO of Oracle (among others) are oblivious to it. Part of me wants to write a scathing 8 page “someone is wrong on the Internet” blog. The other part of me says sleep is more valuable than dealing with these mouth-breathing idiots, which Davidson is one of. Sleep will win, so here is the cold hard facts and reality of the situation. Anything else should be debated at some obscure academic conference, but we know Oracle pays big money to debate it to politicians. Think about that.
Reality #1: Now, let’s start with an age-old saying… “when chinchillas are outlawed, only outlaws will have chinchillas.” Fundamentally, the simple fact that cannot be argued by any rational, logical human, is that laws apply to law-abiding citizens. Those who break the law (i.e. criminal, malefactor, evildoer, transgressor, culprit, felon, crook, hoodlum, gangster, whatever…) do not follow laws. Those who ignore criminal law, likely do not give two fucks about civil law, which a EULA violation would fall under.
Reality #2: Researchers get access to crappy Oracle software in the process of performing their job duties. A penetration test or audit may give them temporary access, and they may find a vulnerability. If the client doesn’t mandate they keep it in-house, the researcher may opt to share it with the vendor, doing the right thing. Where exactly does the EULA fit in here? It was agreed to by the customer, not the third-party researcher. Even if there is a provision in the EULA for such a case, if the company doesn’t warn the researcher of said provision, how can they be held liable?
Reality #3: Tying back into #1 here, what are the real consequences? This is civil law, not criminal. When it comes to criminal law, which is a lot more clear, the U.S. doesn’t have solid extradition case-law backing them. We tend to think “run to Argentina!” when it comes to evading U.S. law. In reality, you can possibly just run to the U.K. instead. Ignore the consequences, that is not relevant when it comes to the law in this context. If you focus on “oh but the death penalty was involved”, you are not understanding Law 101.
In the case of Soering v. United Kingdom, the European Court of Human Rights ruled that the United Kingdom was not permitted under its treaty obligations to extradite an individual to the United States, because the United States’ federal government was constitutionally unable to offer binding assurances that the death penalty would not be sought in Virginia courts.
Now, consider all of the countries that have no extradition treaty with the U.S. There are a lot. How many? Think less on the volume, think more on how well-known this is… a quick Google shows that U.S. news tells us where to run! CNBC says “10 hideout cities for fugitives” and DailyFinance says “Know Where to Run to: The 5 Best Countries With No Extradition“. Not enough? Let’s look at the absolute brilliance that local news can deliver, since my search was intended to find a short list of countries with no extradition, and Wikipedia failed me. Leave it to WSFA 12 in Alabama, to give us a very concise list of countries with no extradition treaty with the US! Criminals, send a spoofed email of thanks to this station for cliff-noting this shit.
These countries currently have no extradition treaty with the United States:
Afghanistan, Algeria, Andorra, Angola, Armenia, Bahrain, Bangladesh, Belarus, Bosnia and Herzegovina, Brunei, Burkina Faso, Burma, Burundi, Cambodia, Cameroon, Cape Verde, the Central
African Republic, Chad, Mainland China, Comoros, Congo (Kinshasa), Congo (Brazzaville), Djibouti, Equatorial Guinea, Eritrea, Ethiopia, Gabon, Guinea, Guinea-Bissau, Indonesia, Ivory Coast, Kazakhstan, Kosovo, Kuwait, Laos, Lebanon, Libya, Macedonia, Madagascar, Maldives, Mali, Marshall Islands, Mauritania, Micronesia, Moldova, Mongolia, Montenegro, Morocco, Mozambique, Namibia, Nepal, Niger, Oman, Qatar, Russia, Rwanda, Samoa, São Tomé & Príncipe, Saudi Arabia, Senegal, Serbia, Somalia, Sudan, Syria, Togo, Tunisia, Uganda, Ukraine, United Arab Emirates, Uzbekistan, Vanuatu, Vatican, Vietnam and Yemen.
Now, can anyone arguing in favor of Davidson’s “EULA speech”, that Oracle officially disagreed with, explain how a EULA protects a company in any way, in a real-world scenario?
Quite simply, there are two major issues at play. First, the absurd idea that a EULA will protect you from anything, other than chasing Intellectual Property (IP) lawsuits against other companies. That happens a lot, to be sure. But it has no bearing, in any way, on security research.
Second, I think back to something an old drunk friend told me a few times. “Never lick a gift-whore in the mouse.” I said he was a drunk friend. Security researchers who ply their trade, find vulnerabilities in your product, report them to you, and wait for you to release a patch? Embrace them. Hug them. Pay them if you can. They are your allies… and every vulnerability they help you squash, is one less vulnerability a bad guy can use to pop your customers. No one in their right mind would ever alienate such a process.
Yesterday, we experienced a catastrophic failure on the primary system hosting OSVDB that initially appears to be related to a drive failure. We are working with our hosting provider but have yet to determine root cause and we currently do not have an ETA as to when the site will be restored.
6/12/2015 Update: After fighting with the disks, the site has been restored and functioning. Apologies for the downtime!
6/14/2015 Update: Days after getting it fixed, the box tanked again. We’re working on it, while cursing technology.
8/3/2015 Update: The system has been back up since ~ 6/16. We just spaced it and forgot to update the blog. Apologies!