Recently, we got feedback via Twitter that we come across as “immature”. On the surface, perhaps. Not all of our Tweets are critical of CVE though. I replied pretty quickly that said criticism is also us “pushing for them to improve since so much of the industry relies on them.” When I Tweeted that, a post to the CVE Editorial Board wasn’t public on the web site, so I couldn’t quote it. But it was a great and timely example of one way our team is pushing CVE to improve.
That said, let me better explain our criticism. It isn’t the first time, it won’t be the last time, but I am not sure if some of these thoughts have been published via blog before. Put simply, we would not be so critical of CVE if it wasn’t a crumbling cornerstone of the Information Security industry. Countless organizations and products use CVE as a bible of public vulnerabilities. A majority of security technology, including firewalls, vulnerability scanners, IDS, IPS, and everything else is built on their database. Our industry assumes they are doing their job, doing their best, and cataloging public vulnerabilities. That simply is not the case, and it hasn’t been for more than a year. Everyone uses it as a benchmark, trusts it, and relies on it. Yet no one questions it except us. Last I checked, our profession was built on “trust but verify“. We verify, thus, we’re critical.
That said, we have a long history of sending corrections, feedback, and improvement ideas to all of the major VDBs including CVE, BID, Secunia, ISS, SecTracker, EDB, PacketStorm, and more. We currently have a great relationship with ISS and EDB. We have had a great-to-courteous relationship with CVE. SecTracker has been receptive of our feedback in the past, but given their minimal output we don’t try to provide cross-references to them anymore. PacketStorm is bad about approving our comments on their published disclosures. BID is and has been a lost cause for almost a decade. Further, we have decent relationships with US-CERT, IBM PSIRT, CERT (CM), and other companies teams. We give continued feedback to some of these organizations that are designed to help their process, help their disclosures, and thus help the industry. I mention this because for the most part, a majority of them are competitors to us. We have a commercial model, it is the only way to fund the database. Despite that, and a decade of disillusionment of the ‘open source’ model, we still give away a significant portion of our data. Until recently, our biggest competitor in licensing our data was ourselves, as potential customers would freely admit they could take our data from osvdb.org. Next to ourselves? CVE and NVD. It is astounding and scary how many companies will pass up on superior vulnerability intelligence because what they are using is free. In many cases, they tell their customers they provide the best intelligence, provide the most security, and a wide variety of other platitudes. In reality, they don’t want to spend a sliver of a fraction of their profits, to truly help their customers. It is a level of greed and unethical behavior that is absolutely disgusting. If I could expose them, I would, but they fall under dreaded NDAs.
Point is, we strive to push all of these organizations to be better. Jake and I gave a presentation back in 2005 at CanSecWest where we said that VDBs need to evolve. They still do. Ten years later, we’re still pushing them to do so while they resist with every ounce of their being. Fortunately, we have been pushing ourselves to be better during that time, and it shows. We’re no more critical of the other VDBs than we are of ourselves. It doesn’t help us one bit. In fact, it only serves to hurt us. Yet, it is the right thing to do for the industry, so we do it.
Nothing like waking up to a new article purporting to show vulnerability statistics and having someone ask us for comment. But hey, we love giving additional perspective on such statistics since they are often without proper context and disclaimers. This morning, the new article comes from Help Net Security and is titled “SQL injection vulnerabilities surge to highest levels in three years“. It cites “DB Networks’ research” who did the usual, parsed NVD data. As we all know, that data comes from CVE who is a frequent topic of rant on Twitter and occasional blogs. Cliff notes: CVE does not promise comprehensive coverage of public disclosures. They openly admit this and Steve Christey has repeatedly said that CVE may not be the best for statistics, even as far back as 2006. Early in 2013, Christey again publicly stated that CVE “can no longer guarantee full coverage of all public vulnerabilities.” I bring this up to remind everyone, again, that it is important to add such disclaimers about your data set, and ultimately your published research.
Getting back to the statistics and DB Research, the second thing I wondered after their data source, was who they were and why they were doing this analysis (SQLi specifically). I probably shouldn’t be surprised to learn this comes over a year after they announced they have an appliance that blocks SQL injection attacks. This too should be disclaimed in any article covering their analysis, as it adds perspective why they are choosing a single vulnerability type, and may also explain why they chose NVD over another data sources (since it fits more neatly into their narrative). Following that, they paid Ponemon to help them conduct a study on the threat of SQL injection. Or wait, was it two of them, the second with a bent towards retail breaches? Here are the results of the first one it seems, which again shows that they have a pretty specific bias toward SQL injection.
Now that we know they have a vested interest in the results of their analysis coming out a specific way, let’s look at the results. First, the Help Net Security article does not describe their methodology, at all. Reading their self-written, paid-for news announcement (PR Newswire) about the analysis, it becomes very clear this is a gimmick for advertising, not actual vulnerability statistics research. It’s 2015, years after Steve Christey and I have both ranted about such statistics, and they don’t explain their methodology. This makes it apparent that “CVE abstraction bias” is possibly the biggest factor here. I have blogged about using CVE/NVD as a dataset before, because it contains one of the biggest pitfalls in such statistic generation.
Rather than debunk these stats directly, since it has been done many times in the past, I can say that they are basically meaningless at this point. Even without their methodology, I am sure someone can trivially reproduce their results and figure out if they abstracted per CVE, or per actual SQLi mentioned. As a recent example, CVE-2014-7137 is a single entry that actually covers 54 distinct SQL injection vulnerabilities. If you count just the CVE candidate versus the vulnerabilities that may be listed within them, your numbers will vary greatly. That said, I will assume that their results can be reproduced since we know their data source and their bias in desired results.
With that in mind, let’s first look at what the numbers look like when using a database that clearly abstracts those issues, and covers a couple thousand sources more than CVE officially does:
“SQL Injection Vulnerabilities Surge to Highest Levels in Three Years” is the title of DB Networks’ press release, and is summarized by Help Net Security as “last year produced the most SQL vulnerabilities identified since 2011 and 104% more than were identified in 2013.” In reality (or at least, using a more comprehensive data set), we see that isn’t the case since the available statistics a) don’t show 2014 being more than 2011 and b) 2011 not holding a candle to 2010. No big surprise really, given that we actually do vulnerability aggregation while NVD and DB Networks does not. But really, I digress. These ‘statistics’ are nothing more than a thinly veiled excuse to further advertise themselves. Notice in the press release they go from the statistics that help justify their products right into the third paragraph quoting the CEO being “truly honored to be selected as a finalist for SC Magazine’s Best Database Security Solution.” He follows that up with another line calling out their specific product that helps “database security in some of the world’s largest mission critical datacenters.”
Yep, these statistics are very transparent. They are based on a convenient data source, maintained by an agency that doesn’t actually aggregate the information, that doesn’t have the experience their data-benefactors have (CVE). They are advertised with a single goal in mind; selling their product. The fact they use the word “research” in the context of generating these statistics is a joke.
As always, I encourage companies and individuals to keep publishing vulnerability statistics. But I stress that it should be done responsibly. Disclaim your data source, explain your methodology, be clear if you are curious about the results coming out one way (bias is fine, just disclaim it), and realize that different data sources will produce different results. Dare to use multiple sources and compare the results, even if it doesn’t fully back your desired opinion. Why? Because if you disclaim your data sources and results, the logical and simple conclusion is that you may still be right. We just don’t have the perfect vulnerability disclosure data source yet. Fortunately, some of us are working harder than others to find that unicorn.
Earlier today, Chris Betz, senior director of the Microsoft Security Response Center (MSRC), posted a blog calling for “better coordinated vulnerability disclosure“.
Before I begin a rebuttal of sorts, let me be absolutely clear. The entire OSVDB team is very impressed with Microsoft’s transition over the last decade as far as security response goes. The MSRC has evolved and matured greatly, which is a benefit to both Microsoft and their customers world-wide. This post is not meant to undermine their efforts at large, rather to point out that since day one, propaganda is still a valuable tool for the company. I will preface this with a reminder that this is not a new issue. I have personally blogged about this as far back as 2001, after Scott Culp (Microsoft at the time) wrote a polarizing piece about “information anarchy” that centered around disclosure issues. At some point Microsoft realized this was a bad position to take and that it didn’t endear them to the researchers providing free vulnerability information to them. Despite that, it took almost ten years for Microsoft to drop the term “responsible” disclosure (also biased against researchers) in favor of “coordinated” disclosure. Again, Microsoft has done a phenomenal job advancing their security program, especially the last three to five years. But… it is on the back of a confrontational policy toward researchers.
Reading yesterday’s blog, there are bits and pieces that stand out to me for various reasons. It is easy to gloss over many of these if you aren’t a masochist and spend most of your waking time buried in vulnerability aggregation and related topics.
In terms of the software industry at large and each player’s responsibility, we believe in Coordinated Vulnerability Disclosure (CVD).
Not sure I have seen “CVD” as a formal initialism until now, which is interesting. After trying to brand “information anarchy” and pushing the “responsible disclosure” term, good to see you embrace a better term.
Ultimately, vulnerability collaboration between researchers and vendors is about limiting the field of opportunity so customers and their data are better protected against cyberattacks.
And this line, early on in the blog, demonstrates you do not live in the real world of vulnerability disclosure. Microsoft has enjoyed their ‘ivory’ tower so to speak. Many researchers find and disclose vulnerabilities for entirely selfish reasons (e.g. bug bounties), which you basically do not offer. Yes, you have a bounty program, but it is very different from most and does not reward a vast majority of vulnerabilities reported to you. Microsoft has done well in creating a culture of “report vulnerabilities to us for free for the honor of being mentioned in one of our advisories”. And I get that! Being listed as a creditee in a Microsoft advisory is advertising itself as far as researcher talent. However… you are talking about a minority of researchers in the greater picture, that chase that honor.
Those in favor of full, public disclosure believe that this method pushes software vendors to fix vulnerabilities more quickly and makes customers develop and take actions to protect themselves. We disagree.
Oh sorry, let me qualify, your black and white tower. This absolutely does work for some vendors, especially those who have a poor history in dealing with vulnerability reports. You may not be one of them for the last 10 years, but you once were. Back in the late ’90s, Microsoft had a reputation for being horrible when dealing with researchers. No vulnerability disclosure policy, no bug bounty (even five years after Netscape had implemented one), and no standard process for receiving and addressing reports. Yes, you have a formal and mature process now, but many of us in the industry remember your beginnings.
It is necessary to fully assess the potential vulnerability, design and evaluate against the broader threat landscape, and issue a “fix” before it is disclosed to the public, including those who would use the vulnerability to orchestrate an attack.
This is a great point. But, let’s read on and offer some context using your own words…
Of the vulnerabilities privately disclosed through coordinated disclosure practices and fixed each year by all software vendors, we have found that almost none are exploited before a “fix” has been provided to customers, and even after a “fix” is made publicly available only a very small amount are ever exploited.
Wait, if only a very small amount of vulnerabilities are exploited after a fix, and ‘almost none’ are exploited before a fix… why do you care if it is coordinated? You essentially invalidate any argument for a researcher coordinating disclosure with you. Why do they care if you clearly state that coordination doesn’t matter, and that the vulnerability will “almost [never]” be exploited? You can’t have this both ways.
CVD philosophy and action is playing out today as one company – Google – has released information about a vulnerability in a Microsoft product, two days before our planned fix on our well known and coordinated Patch Tuesday cadence, despite our request that they avoid doing so.
And this is where you move from propaganda to an outright lie. The issue in question was disclosed on December 29, 2014. That is 15 days, not two days, before your January patch Tuesday. I’d love to hold my breath waiting for MSRC or Betz to explain this minor ’rounding error’ on dates, but I have a feeling I would come out on the losing side. Or is Microsoft simply not aware of public vulnerability disclosures and should perhaps invest in a solution for such vulnerability intelligence? Yes, blatant sales opportunity, but they are desperately begging for it given this statement. =)
[Update. Apparently Microsoft is unhappy over Issue 123 which was auto-published on January 11, as opposed to Issue 118 linked above auto-published on December 29. So they are correct on two days, but curious they aren’t complaining over 118 at the same time when both are local privilege escalation vulnerabilities.]
One could also argue that this is a local privilege escalation vulnerability, which requires a level of access to exploit that simply does not apply to a majority of Windows users. Betz goes on to say that software is complicated (it is), and that not every vulnerability is equal (also true), but also glosses over the fact that Google is in the same boat they are. A little over four years ago, the Google security team posted a blog talking about “rebooting” responsible disclosure and say this:
As software engineers, we understand the pain of trying to fix, test and release a product rapidly; this especially applies to widely-deployed and complicated client software. Recognizing this, we put a lot of effort into keeping our release processes agile so that security fixes can be pushed out to users as quickly as possible.
To be fair, Google also did not publish a timeline of any sorts with this disclosure. We don’t know anything that happened after the September 30, 2014 report to Microsoft. Did you ask for more time Google? Did Microsoft say it was being patched in January? If so, you look like total assholes, disclosure policy be damned. If they didn’t mentioned January specifically and only asked for more time, maybe it was fair you kept to your schedule. One of the two parties should publish all of the correspondence now. What’s the harm, the issue is public! Come on.. someone show their cards, prove the other wrong. Back to Microsoft’s blog…
What’s right for Google is not always right for customers.
This is absolutely true. But you forgot the important qualifier; what is is right for Microsoft, is not always right for customers.
For example, look at CVE-2010-3889 (heavily referenced) aka “Microsoft Windows on 32-bit win32k.sys Keyboard Layout Loading Local Privilege Escalation”. This is one of four vulnerabilities used by Stuxnet. Unfortunately, Microsoft has no clear answer if this is even patched, four years later. That CVE identifier doesn’t seem to exist in any Microsoft security advisory. Why not? Did you really let a vulnerability that may have aided an attack on an Iranian nuclear power plant go unpatched? Think of the ethics questions there! Or is this a case of the Microsoft security response process not being as mature as I give them credit, and this is a dupe of CVE-2010-2743? Why does it take a third-party four years to figure this out while writing a blog on a whim?
It is a zero sum game where all parties end up injured.
What does this even mean, other than propaganda? It is rarely, if ever, a case where “all parties” are injured. If a researcher discloses something to you and publishes prematurely, or publishes on their own without contacting you, usually that party is not ‘injured’ in doing so. That is simple fact.
Betz’ blog goes on to quote the Microsoft CVD policy which states:
Microsoft’s Approach to Coordinated Vulnerability Disclosure
Under the principle of Coordinated Vulnerability Disclosure, finders disclose newly discovered vulnerabilities in hardware, software, and services directly to the vendors of the affected product; to a national CERT or other coordinator who will report to the vendor privately; or to a private service that will likewise report to the vendor privately.
Perhaps you should qualify that statement, as US-CERT has a 45 day disclosure policy in most cases. That is half the time Google gave you. Quoting from the US-CERT policy:
Q: Will all vulnerabilities be disclosed within 45 days?
A: No. There may often be circumstances that will cause us to adjust our publication schedule. Threats that are especially serious or for which we have evidence of exploitation will likely cause us to shorten our release schedule. Threats that require “hard” changes (changes to standards, changes to core operating system components) will cause us to extend our publication schedule. We may not publish every vulnerability that is reported to us.
Note that it does not qualify “the vendor asks for more time”. That is the United States government saying a vendor gets 45 days to patch with rare exception. Oh wait Mr. Betz, before you go quoting “changes to core operating system components”, I will stop you there. Vulnerabilities in win32k.sys are not new. That 3.1 meg binary (on Windows 7) is the cause for a lot of grief for Windows users in that file alone. Given that history, you cannot say that changes to that file meet the US-CERT criteria.
Finally, this isn’t the first pissing match between Google and Microsoft on vulnerability disclosure. While Microsoft has routinely played the victim card and Google certainly seems more aggressive on their disclosure policy, there is a more than one bit of irony if one looks deeper. In random order…
Microsoft disclosed a vulnerability in Google Chrome, but didn’t do proper research. This vulnerability may be in WebKit as one person notes, meaning it could affect other browsers like Apple Safari. If it does, then Apple would get blindsided in this disclosure, and it would not be ‘coordinated’ or ‘responsible’, and would qualify as ‘information anarchy’ as Microsoft once called it. While we don’t know if it was ultimately in WebKit, we do know this vulnerability exists because Google Chrome was trying to work around issues with Microsoft software.
Look at MSVR11-011 and MSVR11-012 from 2011, where Microsoft “coordinated” two vulnerabilities with the FFmpeg team. To be sure, the FFmpeg team is outstanding at responding to and fixing vulnerabilities. However, in the real world, there are thousands of vendors that use FFmpeg as a library in their own products. While it may have been fixed in the base code, it can easily take somewhere between months and a decade for vendors to learn about and upgrade the library in their software. Only in a completely naive world could Microsoft call this “coordinated”.
Even better, let’s go back to the inaugural Microsoft Vulnerability Research (MSVR) advisory, MSVR11-001. This was a “Use-After-Free Object Lifetime Vulnerability in Chrome” that in reality was a vulnerability in WebKit, the underlying rendering library used by Chrome. The problem is that WebKit is used by a lot more than Chrome. So the first advisory from MSVR conveniently targets a Google product, but completely botches the “coordinated” disclosure, going to a single vendor using WebKit code, because the Microsoft researchers apparently didn’t diagnose the problem fully. No big deal right?
Wrong. I am sure Adobe, Samsung, Amazon, Tizen, Symbian, BlackBerry, Midori, and Android web browser users would disagree strongly. Do you really want to compare the number of users you blindsided with this “coordinated” disclosure to the ones you protected? Microsoft was a bigger jackass on this disclosure than Google ever was, plain and simple.
Finally, do I even need to go into the absolute mess than you call the “Advanced Notification Service” (ANS)? In case readers aren’t aware, this is not a single program. This is several different programs with various names like MAPP and others. Just three days ago, you Mr. Betz announced that ANS was changing. This is after another program got changed drastically, multiple companies were kicked out of the MAPP program, and who knows what else happened. All of which was founded on Microsoft giving advanced and sometimes detailed vulnerability information to questionable companies, that may not be friendly parties.
The entire notion of “coordinated” disclosure went out the window as far as Microsoft goes, when they first implemented these programs. You specifically gave a very limited number of organizations details about vulnerabilities, before other customers had access. That, by definition, is not coordination. That is favoritism in the name of the bottom line, and speaks strongly against any intent you outline in yesterday’s blog post.
While Microsoft has taken great effort to improve their security process, it is disingenuous to call this anything but propaganda.
CVE, managed by MITRE, a ‘sole-source’ government contractor, who gets as much as one million dollars a year from the government (or more) to run the project, is a confusing entity. Researchers who have reached out to CVE for assignment or clarification on current assignments, have gone 10 days without answer (as of 2014-11-15 late night). Yet look at their actual assignments the past week:
X N 924 Nov 10 email@example.com (22K) [CVENEW] New CVE CANs: 2014/11/10 06:00 ; count=17
X N 931 Nov 11 firstname.lastname@example.org (25K) [CVENEW] New CVE CANs: 2014/11/11 17:00 ; count=32
X N 932 Nov 11 email@example.com (18K) [CVENEW] New CVE CANs: 2014/11/11 18:00 ; count=18
X N 938 Nov 12 firstname.lastname@example.org (7514) [CVENEW] New CVE CANs: 2014/11/12 11:00 ; count=5
X N 962 Nov 13 email@example.com (12K) [CVENEW] New CVE CANs: 2014/11/13 10:00 ; count=9
X N 986 Nov 13 firstname.lastname@example.org (7139) [CVENEW] New CVE CANs: 2014/11/13 19:00 ; count=4
X N 995 Nov 14 email@example.com (7191) [CVENEW] New CVE CANs: 2014/11/14 10:00 ; count=3
X N 1015 Nov 14 firstname.lastname@example.org (6076) [CVENEW] New CVE CANs: 2014/11/14 21:00 ; count=3
X N 1035 Nov 15 email@example.com (5859) [CVENEW] New CVE CANs: 2014/11/15 15:00 ; count=2
X N 1037 Nov 15 firstname.lastname@example.org (9615) [CVENEW] New CVE CANs: 2014/11/15 16:00 ; count=7
X N 1043 Nov 15 email@example.com (9034) [CVENEW] New CVE CANs: 2014/11/15 19:00 ; count=4
X N 1045 Nov 15 firstname.lastname@example.org (7539) [CVENEW] New CVE CANs: 2014/11/15 20:00 ; count=3
X N 1046 Nov 15 email@example.com (5885) [CVENEW] New CVE CANs: 2014/11/15 21:00 ; count=2
Impressive! Until you read between the lines. Mon, 17 entries. Tues, which is MS (32) / Adobe (18) release day, and those numbers are obvious… as it only covers those two vendors. Then 5 on Wed, 13 on Thurs, 6 on Friday… and then 21 on the weekend? Why is a government contractor, who has a long history of not working or answering mails on the weekend, doing what appears to be overtime on a weekend?
Meanwhile, the 10th we have 32 entries, 11th we have 100 entries, 12th we have 92 entries, 13th we have 56 entries, 14th we have 42 entries, and the 15th we have 11 entries. That is 109 entries this week from CVE, where 50 of them (almost half) were Microsoft and Adobe. Meanwhile, we have 337 entries over those same days. That doesn’t count our backfill for historical entries, from those ‘old days’ back in earlier 2014 or 2013, that we are constantly doing.
Tonight, when matching up the Nov 15 CVE entries, we had 100% of the CVE assignments already. Remind me, where is the value of CVE exactly? They are assigning these identifiers in advance, that is obvious. But for most disclosures they are simple and straight-forward. They aren’t being used to coordinate among multiple vendors. The researchers or vendors are including the CVE identifier *before* CVE actually publishes them.
What led me to this post is that CVE is actually working on a weekend, which is very odd. Unless you mail Steve directly you generally don’t hear back from CVE until later in the week. OSVDB / RBS has outstanding mail to both Steve and CVE regarding previous assignments and other things, un-answered for 10 or more days currently. The entire purpose of CVE is to provide this ID for coordination and clarity. When they ignore such a mail, especially from a ‘trusted’ source, it speaks poorly on them. Given the level of government funding they receive, how are they not keeping up with disclosures throughout the week and instead, turning to a Saturday?
And please remember, the Saturday CVE assignments mentioned above won’t appear on CVE’s site for another day, and won’t be in NVD for at least 24 hours. Once NVD gets them, they won’t have a CVSS score or CPE data for a bit after. By a ‘bit’ I mean between a few hours and a few weeks.
This is fail on top of fail. And your security solutions are built on top of this. Yeah, of course this is a losing battle.
Based on a Twitter thread started by Aaron Portnoy that was replied to by @4Dgifts asking why people would debunk vulnerability reports, I offer this quick high-level summary of what we see, and how we handle it.
Note that OSVDB uses an extensive classification system (that is very close to being overhauled greatly for more clarity and granularity), in addition to CVSS scoring. Part of our classification system allows us to flag an entry as ‘not-a-vuln’ or ‘myth/fake’. I’d like to briefly explain the different, but also in the bigger picture. When we process vulnerability reports, we only have time to go through the information disclosed usually. In some cases we will spend extra time validating or debunking the issue, as well as digging up information the researcher left out such as vendor URL, affected version, script name, parameter name, etc. That leads to the high-level types of disclosures:
- Invalid / Not Enough – We are seeing cases where a disclosure doesn’t have enough actionable information. There is no vendor URL, the stated product name doesn’t come up on various Google searches, the proof-of-concept (PoC) provided is only for one live site, etc. If we can’t replicate it or dig up the vendor in five minutes, we have to move on.
- Site-specific – Some of the disclosures from above end up being specific to one web site. In a few rare cases, they impact several web sites due to the companies all using the same web hosting / design shop that re-uses templates. Site-specific does not qualify for inclusion in any of the big vulnerability databases (e.g. CVE, BID, Secunia, X-Force, OSVDB). We aggregate vulnerabilities in software and hardware that is available to multiple consumers, on their premises. That means that big offerings like Dropbox or Amazon or Facebook don’t get included either. OSF maintains a separate project that documents site-specific issues.
- Vulnerability – There is enough actionable information to consider it valid, and nothing that sets off warnings that it may be an issue. This is the run-of-the-mill event we deal with in large volumes.
- Not a Vulnerability – While a valid report, the described issue is just considered a bug of some kind. The most common example is a context-dependent ‘DoS’ that simply crashes the software, such as media player or browser. The issue was reported to crash the software, so that is valid. But in ‘exploiting’ the issue, the attacker has gained nothing. They have not crossed privilege boundaries, as the issue can quickly be recovered from. Note that if the issue is a persistent DoS condition, that becomes a valid issue.
- Myth/Fake – This was originally created to handle rumors of older vulnerabilities that simply were not true. “Do you remember that remote Solaris 2.5 bug in squirreld??” Since then, we have started using this classification more to denote when a described issue is simply invalid. For example, the researcher claims code execution and provides a PoC that only shows a DoS. Subsequent analysis shows that it is not exploitable.
Before you start sending emails, as @4DGifts reminds us, you can rarely say with 100% assurance that something isn’t exploitable. We understand and agree with that completely. But it is also not our job to prove a negative. If a researcher is claiming code execution, then they must provide the evidence to back their claim. Either an additional PoC that is more than a stability crash, or fully explain the conditions required to exploit it. Often times when a researcher does this, we see that while it is an issue of some sort, it may not cross privilege boundaries. “So you need admin privs to exploit this…” and “If you get a user to type in that shell code into a prompt on local software, it executes code…” Sure, but that doesn’t cross privilege boundaries.
That is why we encourage people like Aaron to help debunk invalid vulnerability reports. We’re all about accuracy, and we simply don’t have time to test and figure out every vulnerability disclosed. If it is a valid issue but requires dancing with a chicken at midnight, we want that caveat in our entry. If it is a code execution issue, but only with the same privileges as the attacker exploiting it, we want to properly label that too. We do not use CVSS to score bogus reports as valid. Instead, we reflect that they do not impact confidentiality, integrity, or availability which gives it a 0.0 score.
[2014-05-09 Update: We’d like to thank both McAfee and S21sec for promptly reaching out to work with us and to inform us that they are both investigating the incident, and taking steps to ensure that future access and data use complies with our license.]
Every day we get requests for an account on OSVDB, and every day we have to turn more and more people away. In many cases the intended use is clearly commercial, so we tell them they can license our data via our commercial partner Risk Based Security. While we were a fully open project for many years, the volunteer model we wanted didn’t work out. People wanted our data, but largely did not want to give their time or resources. A few years back we restricted exports and limited the API due to ongoing abuse from a variety of organizations. Our current model is designed to be free for individual, non-commercial use. Anything else requires a license and paying for the access and data usage. This is the only way we can keep the project going and continue to provide superior vulnerability intelligence.
As more and more organizations rely on automated scraping of our data in violation of our license, it has forced us to restrict some of the information we provide. As the systematic abuse rises, one of our only options is to further restrict the information while trying to find a balance of helping the end user, but crippling commercial (ab)use. We spend about half an hour a week looking at our logs to identify abusive behavior and block them from accessing the database to help curb those using our data without a proper license. In most cases we simply identify and block them, and move on. In other cases, it is a stark reminder of just how far security companies will go to to take our information. Today brought us two different cases which illustrate what we’re facing, and why their unethical actions ultimately hurt the community as we further restrict access to our information.
This is not new in the VDB world. Secunia has recently restricted almost all unauthenticated free access to their database while SecurityFocus’ BID database continues to have a fraction of the information they make available to paying customers. Quite simply, the price of aggregating and normalizing this data is high.
In the first case, we received a routine request for an account from a commercial security company, S21sec, that wanted to use our data to augment their services:
From: Marcos xxxxxx (firstname.lastname@example.org)
To: moderators osvdb.org
Date: Thu, 16 May 2013 11:26:28 +0200
Subject: [OSVDB Mods] Request for account on OSVDB.org
I’m working on e-Crime and Malware Research for S21Sec (www.s21sec.com), a lead IT Security company from Spain. I would like to obtain an API key to use in research of phishing cases we need to investigate phishing and compromised sites. We want to use tools like “cms-explorer” and create our own internal tools.
Tlf: +34 902 222 521
http://www.s21sec.com , blog.s21sec.com
As with most requests like this, they received a form letter reply indicating that our commercial partner would be in touch to figure out licensing:
From: Brian Martin (brian opensecurityfoundation.org)
To: Marcos xxxxxx (email@example.com)
Cc: RBS Sales (sales riskbasedsecurity.com)
Date: Thu, 16 May 2013 15:26:04 -0500 (CDT)
Subject: Re: [OSVDB Mods] Request for account on OSVDB.org
The use you describe is considered commercial by the Open Security
We have partnered with Risk Based Security (in the CC) to handle
commercial licensing. In addition to this, RBS provides a separate portal
with more robust features, including an expansive watch list capability,
as well as a considerably more powerful API and database export options.
The OSVDB API is very limited in the number of calls due to a wide variety
of abuse over the years, and also why the free exports are no longer
available. RBS also offers additional analysis of vulnerabilities
including more detailed technical notes on conditions for exploitation and
OSF / OSVDB
He came back pretty quickly saying that he had no budget for this, and didn’t even wait to get a price quote or discuss options:
From: Marcos xxxxxx (firstname.lastname@example.org)
Date: Mon, May 20, 2013 at 10:55 AM
Subject: Re: [OSVDB Mods] Request for account on OSVDB.org
To: Brian Martin (brian opensecurityfoundation.org)
Cc: RBS Sales (sales riskbasedsecurity.com)
Thanks for the answer, but I have no budget to get the license.
We figured that was the end of it really. Instead, jump to today when we noticed someone scraping our data and trying to hide their tracks to a limited degree. Standard enumeration of our entries, but they were forging the user-agent:
184.108.40.206 – – [07/May/2014:09:37:06 -0500] “GET /show/osvdb/106231 HTTP/1.1” 200 20415 “-” “Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0”
220.127.116.11 – – [07/May/2014:09:37:06 -0500] “GET /show/osvdb/106232 HTTP/1.1” 200 20489 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko”
18.104.22.168 – – [07/May/2014:09:37:07 -0500] “GET /show/osvdb/106233 HTTP/1.1” 200 20409 “-” “Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)”
22.214.171.124 – – [07/May/2014:09:37:08 -0500] “GET /show/osvdb/106235 HTTP/1.1” 200 20463 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36”
Visiting that IP told us who it was:
So after requesting data, and hearing that it would require a commercial license, they figure they will just scrape the data and use it without paying. 3,600 accesses between 09:18:30 and 09:43:19.
In the second case, and substantially more offensive, is the case of security giant McAfee. They approached us last year about obtaining a commercial feed to our data that culminated in a one hour phone call with someone who ran an internal VDB there. On the call, we discussed our methodology and our data set. While we had superior numbers to any other solution, they were hung up on the fact that we weren’t fully automated. The fact that we did a lot of our process manually struck them as odd. In addition to that, we employed less people than they did to aggregate and maintain the data. McAfee couldn’t wrap their heads around this, saying there was “no way” we could maintain the data we do. We offered them a free 30 day trial to utilize our entire data set and to come back to us if they still thought it was lacking.
They didn’t even give it a try. Instead they walked away thinking our solution must be inferior. Jump to today…
126.96.36.199 – – [04/May/2014:07:22:14 -0500] “GET /90703 HTTP/1.1” 200 6042 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36”
188.8.131.52 – – [04/May/2014:07:22:16 -0500] “GET /90704 HTTP/1.1” 200 6040 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36”
184.108.40.206 – – [04/May/2014:07:22:18 -0500] “GET /90705 HTTP/1.1” 200 6039 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36”
220.127.116.11 – – [04/May/2014:07:22:20 -0500] “GET /90706 HTTP/1.1” 200 6052 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36”
They made 2,219 requests between 06:25:24 on May 4 and 21:18:26 on May 6. Excuse us, you clearly didn’t want to try our service back then. If you would like to give a shot then we kindly ask you to contact RBS so that you can do it using our API, customer portal, and/or exports as intended.
Overall, it is entirely frustrating and disappointing to see security companies who sell their services based on reputation and integrity, who claim to have ethics completely disregard them in favor of saving a buck.
We know SCADA is virtual swiss cheese, ready to be owned if someone can reach a device. We have preached airgaps for decades, even before we knew how bad the software was. Back then it was just, “this is so critical, it has to be separate!”
The last five years have proven how bad it is, with the rise of SCADA vulnerabilities. Sure, we can overlook the bad coding, proprietary protocols, no evidence of a SDLC, and the incredible amount of time it can take to patch. For some silly reason we put up with “forever-day bugs” because something is so critical it can’t be rebooted (forgetting how absurd that design choice is). But, what if we go a step beyond that?
An ICS-CERT 14-084-01 advisory released yesterday on vulnerabilities in Festo products is a good reminder of just how bad the problem is, and how much deeper it goes. First, the product has a backdoor in the FTP service allowing unauthenticated access (CVSSv2 9.3). This can allow a remote attacker to crash the device or execute arbitrary code. Second, the device is vulnerable due to bundling the 3S CoDeSys Runtime Toolkit which does not require authentication for admin functions (CVSSv2 10.0), and a traversal flaw that allows file manipulation leading to code execution (CVSSv2 10.0). Those two issues were reported in January of 2013, making this report as relates to Festo products over a year late.
So we have a vendor backdoor, unauthenticated administrator access, and a way to bypass authentication if it was there to gain privileges. So realistically, what type of organizations does this potentially impact? From the ICS-CERT advisory:
This product is used industrywide as a programmable logic controller with inclusion of a multiaxis controller for automated assembly and automated manufacturing. Identified customers are in solar cell manufacturing, automobile assembly, general assembly and parts control, and airframe manufacturing where tolerances are particularly critical to end product operations.
Now to dig the hole deeper. Under the “Mitigation” section, we see how serious Festo considers these vulnerabilities. Paraphrased from two lines in the advisory:
Festo has decided not to resolve these vulnerabilities, placing critical infrastructure asset owners using this product at risk … because of compatibility reasons with existing engineering tools.
The two 3S CoDeSys vulnerabilities have a fix available and just need to be integrated into the Festo products. What does “compatibility with existing engineering tools” really mean in the context of software? The ICS-CERT advisory also says:
According to the Festo product web page, other products are using newer versions of CoDeSys software and may not be vulnerable to the CoDeSys vulnerability, but this has not been evaluated by the researcher.
The researcher already spent time finding the issues, reporting them to a coordinating body, and following coordinated disclosure practices. Expecting them to also evaluate which products are not vulnerable is ridiculous. This is a case of the vendor just being lazy and irresponsible.
A company that makes vulnerable critical components that affect our infrastructure and directly impact our safety, but refuses to fix them. Why is this allowed to exist in our society?
After John Cartwright abruptly announced the closure of the Full Disclosure mail list, there was a lot of speculation as to why. I mailed John Cartwright the day after and asked some general questions. In so many words he indicated it was essentially the emotional wear and tear of running the list. While he did not name anyone specifically, the two biggest names being speculated were ‘NetDev’ due to years of being a headache, and the more recent thread started by Nicholas Lemonias. Through other channels, not via Cartwright, I obtained a copy of a legal threat made against at least one hosting provider for having copies of the mails he sent. This mail was no doubt sent to Cartwright among others. As such, I believe this is the “straw that broke the camels back” so to speak. A copy of that mail can be found at the bottom of this post and it should be a stark lesson that disclosure mail list admins are not only facing threats from vendors trying to stifle research, but now security researchers. This includes researchers who openly post to a list, have a full discussion about the issue, desperately attempt to defend their research, and then change their mind and want to erase it all from public record.
As I previously noted, relying on Twitter and Pastebin dumps are not a reliable alternative to a mail list. Others agree with me including Gordon Lyon, the maintainer of seclists.org and author of Nmap. He has launched a replacement Full Disclosure list to pick up the torch. Note that if you were previously subscribed, the list users were not transferred. You will need to subscribe to the new list if you want to continue participating. The new list will be lightly moderated by a small team of volunteers. The community owes great thanks to both John and now Gordon for their service in helping to ensure that researchers have an outlet to disclose. Remember, it is a mail list on the surface; behind the scenes, they deal with an incredible number of trolls, headache, and legal threats. Until you run a list or service like this, you won’t know how emotionally draining it is.
Note: The following mail was voluntarily shared with me and I was granted permission to publish it by a receiving party. It is entirely within my legal right to post this mail.
From: Nicholas Lemonias. (email@example.com)
Date: Tue, Mar 18, 2014 at 9:11 PM
Subject: Abuse from $ISP hosts
I am writing you to launch an official complaint relating to Data
Protection Directives / and Data Protection Act (UK).
Therefore my request relates to the retention of personal and confidential
information by websites hosted by Secunia.
These same information are also shared by UK local and governmental
authorities and financial institutions, and thus there are growing
concerns of misuse of such information.
Consequently we would like to request that you please delete ALL records
containing our personal information (names, emails, etc..) in whole, from
your hosted websites (seclists.org) and that distribution of our
information is ceased . We have mistakenly posted to the site, and however
reserve the creation rights to that thread, and also reserve the right to
have all personal information deleted, and ceased from any electronic
dissemination, use either partially or in full.
I hope that the issue is resolved urgently without the involvement of local
I look forward to hearing from you soon.
Thanks in advance,
Update 7:30P EST: Andrew Wallace (aka NetDev) has released a brief statement regarding Full Disclosure. Further, Nicholas Lemonias has threatened me in various ways in a set of emails, all public now.
This morning I woke to the news that the Full-Disclosure mail list was closing its doors. Assuming this is not a hoax (dangerously close to April 1st) and not spoofed mail that somehow got through, there seems to be perspective missing on the importance of this event. Via Facebook posts and Twitter I see casual disappointment, insults that the list was low signal to noise, and that many had stopped reading it a while back. I don’t begrudge the last comment one bit. The list has certainly had its share of noise, but that is the price we pay as a community and industry for having a better source for vulnerability disclosure. Speaking to the point of mail lists specifically, there were three lists that facilitated this: Bugtraq, Full-Disclosure, and Open Source Security (OSS). Bugtraq has been around the longest and is the only alternative to Full-Disclosure really (remember that VulnWatch didn’t last, and was ultimately low traffic). OSS is a list that caters to open source software and does not traffic in commercial software. A majority of the posts come from open source vendors (e.g. Linux distributions), the software’s maintainer, etc. It is used as much for disclosure as coordination between vendors and getting a CVE assigned.
One of the first things that should be said is a sincere “thank you” to John Cartwright for running the list so long. For those of you who have not moderated a list, especially a high-traffic list, it is no picnic. The amount of spam alone makes list moderation a pain in the ass. Add to that the fake exploits, discussions that devolve into insults, and topics that are on the fringe of the list’s purpose. Trying to sort out which should be allowed becomes more difficult than you would think. More importantly, he has done it in a timely manner for so long. Read the bold part again, because that is absolutely critical here. When vulnerability information goes out, it is important that it goes out to everyone equally. Many mails sent to Bugtraq and Full-Disclosure are also sent to other parties at the same time. For example, every day we get up to a dozen mails to the OSVDB Moderators with new vulnerability information, and those lists and other sources (e.g. Exploit-DB, OffSec, 1337day) are in the CC. If you use one or a few of those places as your primary source for vulnerability intelligence, you want that information as fast as anyone else. A mail sent on Friday afternoon may hit just one of them, before appearing two days later on the rest. This is due to the sites being run with varying frequency, work schedules, and dedication. Cartwright’s quick moderation made sure those mails went out quickly, often at all hours of the day and over weekends.
While many vulnerability disclosers will send to multiple sources, you cannot assume that every disclosure will hit every source. Some of these sites specialize in a type of vulnerability (e.g. web-based), while some accept most but ignore a subset (e.g. some of the more academic disclosures). Further, not every discloser sends to all these sources. Many will send to a single mail list (e.g. Bugtraq or FD), or to both of them. This is where the problem arises. For many of the people still posting to the two big disclosure lists, they are losing out on the list that was basically guaranteed to post their work. Make no mistake, that isn’t the case for both lists.
This goes back to why Full-Disclosure was created in the first place (July 11, 2002). This was days before Symantec announced they were acquiring SecurityFocus (July 17, 2002). That was not a coincidence. While I can’t put a finger on when BugTraq changed for the worse exactly, I can assure you it has. Back in 2003, security researchers were noticing curious delays in their information being posted. One company challenged SecurityFocus/Bugtraq publicly, forcing them to defend themselves.
“The problem with SecurityFocus is not that they moderate the lists, but the fact that they deliberately delay and partially censor the information,” said Thomas Kristensen, CTO of Secunia, based in Copenhagen, Denmark. “Since they were acquired by Symantec they changed their policy regarding BugTraq. Before they used to post everything to everybody at the same time. Now they protect the interests of Symantec, delay information and inform their customers in advance.” Wong says there is no truth to these accusations. “The early warnings that our DeepSight customers get come from places like BugTraq and events and incidents that we monitor,” Wong said. “We dont give those alerts [from BugTraq] to our customers any sooner than anyone else gets them.”
Unfortunately for our community, Mr. Wong is absolutely incorrect. I have witnessed this behavior first hand several times over the years, as have others. From a series of mails in 2006:
* mudge (mudge @ uidzero org) [060120 20:04]:
Actually, this advisory is missing some important information. bugtraq engaged in this prior to the “buy out”. Security Focus engaged in this practice as well where there were some advisories that would go out only to the Security Focus paid private list and not be forwarded to the public list to which they were posted.
On Fri, 20 Jan 2006, H D Moore wrote:
FWIW, I have noticed that a few of my own BT posts will not reach my mailbox until they have already been added to the securityfocus.com BID database. It could be my subscriber position in the delivery queue, but it does seem suspicious sometimes. Could just be paranoia, but the list behavior/delivery delays definitely contribute to it.
In each case, moderators of Bugtraq vehemently denied the allegations. In one case, Al Huger (with Symantec at the time) reminded everyone that the combined lists of SecurityFocus were delivering over 7 million mails a day. That alone can cause issues in delivery of course. On the other hand, Symantec surely has the resources to ensure they run a set of mail servers that can churn out mail in such volume to ensure prompt delivery. Jump to more recently and you can still see incredible delay that has nothing to do with delivery issues. For example, RBS posted an advisory simultaneously to both Bugtraq and Full-Disclosure. Notice that the mail was posted on Sep 10 for Full-Disclosure and Sep 19 for Bugtraq. A nine day delay in moderating vulnerability information is not acceptable in today’s landscape of threats and bad actors. Regardless of intent, such delays simply don’t cut it.
In addition to the Bugtraq moderators having such delays, they will sometimes reject a post for trivial reasons such as “using a real IP address” in an example (one time using the vendor’s IP, another time using a public IP I control). They rejected those posts, while frequently allowing “target.com” in disclosures which is a real company.
With the death of Full-Disclosure, Bugtraq is now our primary source of vulnerability disclosure in the scope of mail lists, and only source for vulnerabilities in commercial software (out of scope for OSS). To those who argue that people “use mail a lot less now”, I suggest you look at the volume of Bugtraq, Full-Disclosure, and OSS. That is a considerable amount of disclosures made through that mechanism. Another mindset is that disclosing vulnerabilities can be done with a Tweet using a hash tag and a link to pastebin or other hosting site. To this I can quickly say that you have never run a VDB (and try finding a full set of your original l0pht or @stake advisories, many have largely vanished). Pastebin dumps are routinely removed. Researcher blogs, even hosted on free services such as WordPress and Blogger, disappear routinely. Worse, vendors that host advisories in their own products will sometimes remove their own historical advisories. The “Tweet + link” method simply does not cut it unless you want vulnerability provenance to vanish in large amounts. It is bad enough that VDBs have to rely on the Internet Archive so often (speaking of, donate to them!), but forcing us to set up a system to mirror all original disclosures is a burden. Last, for those who argue that nothing good is posted to Full-Disclosure, Lucian Constantin points out a couple good examples to counter the argument in his article on the list closing.
Instead, mail lists provide an open distributed method for releasing information. As you can see, these lists are typically mirrored on multiple sites as well as personal collections of incoming email. It is considerably easier and safer to use such a method for vulnerability disclosures going forward. In my eyes, and the eyes of others that truly appreciate what Full-Disclosure has done, the loss of that list is devastating in the short term. Not only will it introduce a small amount of bias in vulnerability aggregation, it will take time to recover. Even if someone else picks up the torch under the same name, or starts a new list to replace it, it will take time for people to transition to the new list.
To conclude, I would also ask that John Cartwright practice full disclosure himself. Shuttering the list is one thing, but blaming the action on an unnamed person with no real details isn’t what the spirit of the list is about. Give us details in a concise and factual manner, so that the industry can better understand what you are facing and what they may be getting into should they opt to run such a list.
On February 26, Secunia released their annual vulnerability report (link to report PDF) summarizing the computer security vulnerabilities they had cataloged over the 2013 calendar year. For those not familiar with their vulnerability database (VDB), we consider them a ‘specialty’ VDB rather than a ‘comprehensive’ VDB (e.g. OSVDB, X-Force). They may disagree on that point, but it is a simple matter of numbers that leads us to designate them as such. That also tends to explain why some of our conclusions and numbers are considerably different and complete than theirs.
In past years this type of blog post would not need a disclaimer, but it does now. OSVDB, while the website is mostly open to the public, is also the foundation of the VulnDB offering from our commercial partner and sponsor Risk Based Security (RBS). As such, we are now a direct competitor to Secunia, so any criticism leveled at them or their report may be biased. On the other hand, many people know that I am consistently critical of just about any vulnerability statistics published. Poor vulnerability statistics have plagued our industry for a long time. So much so that Steven Christey from CVE and I gave a presentation last year at the BlackHat briefings in Las Vegas on the topic.
One of the most important messages and take-aways from that talk is that all vulnerability statistics should be disclaimed and explained in advance. That means that a vulnerability report should start out by explaining where the data came from, applicable definitions, and the methodology of generating the statistics. This puts the subsequent statistics in context to better explain and disclaim them, as a level of bias enters any set of vulnerability statistics. Rather than follow the Secunia report in the order they publish them, I feel it is important to skip to the very end first. For that is where they finally explain their methodology to some degree, which is absolutely critical in understanding how their statistics were derived.
On page 16 (out of 20) of the report, in the Appendix “Secunia Vulnerability Tracking Process”, Secunia qualifies their methodology for counting vulnerabilities.
A vulnerability count is added to each Secunia Advisory to indicate the number of vulnerabilities covered by the Secunia Advisory. Using this count for statistical purposes is more accurate than counting CVE identifiers. Using vulnerability counts is, however, also not ideal as this is assigned per advisory. This means that one advisory may cover multiple products, but multiple advisories may also cover the same vulnerabilities in the same code-base shared across different programs and even different vendors.
First, the ‘vulnerability count’ referenced is not part of a public Secunia advisory, so their results cannot be realistically duplicated. The next few lines are important, as they invalidate the Secunia data set for making any type of real conclusion on the state of vulnerabilities. Not only can one advisory cover multiple products, multiple advisories can cover the same single vulnerability, just across different major versions. This high rate of duplicates and lack of unique identifiers make the data set too convoluted for meaningful statistics.
CVE has become a de facto industry standard used to uniquely identify vulnerabilities which have achieved wide acceptance in the security industry.
This is interesting to us because Secunia is not fully mapped to CVE historically. Meaning, there are thousands of vulnerabilities that CVE has cataloged, that Secunia has not included. CVE is a de facto industry standard, but also a drastically incomplete one. At the bare minimum, Secunia should have a 100% mapping to them and they do not. This further calls into question any statistics generated off this data set, when they knowingly ignore such a large number of vulnerabilities.
From remote describes other vulnerabilities where the attacker is not required to have access to the system or a local network in order to exploit the vulnerability. This category covers services that are acceptable to be exposed and reachable to the Internet (e.g. HTTP, HTTPS, SMTP). It also covers client applications used on the Internet and certain vulnerabilities where it is reasonable to assume that a security conscious user can be tricked into performing certain actions.
Classification for the location of vulnerability exploitation is important as this heavily factors into criticality; either via common usage, or through scoring systems such as CVSS. In their methodology, we see that Secunia does not make a distinction between ‘remote’ and ‘context-dependent’ (or ‘user-assisted’ by some). This means that the need for user interaction is not factored into an issue and ultimately, scoring and statistics become based on network, local (adjacent) network, or local vectors.
Secunia further breaks down their classification in the appendix under “Secunia Vulnerability Criticality Classification“. However, it is important to note that their breakdown does not really jibe with any other scoring system. Looking past the flaw of using the word ‘critical’ in all five classifications, the distinction between ‘Extremely Critical’ and ‘Highly Critical’ is minor; it appears to be solely based on if Secunia is aware of exploit code existing for that issue based on their descriptions. This mindset is straight out of the mid 90s in regards to threat modeling. In today’s landscape, if details are available about a vulnerability then it is a given that a skilled attacker can either write or purchase a vulnerability for the issue within a few days, for a majority of disclosed issues. In many cases, even when details aren’t public but a patch is, that is enough to reliably reverse it and leverage it for working exploit code in a short amount of time. Finally, both of these designations still do not abstract on if user interaction is required. In each case, it may or may not be. In reality, I imagine that the difference between ‘Extremely’ and ‘Highly’ is supposed to be based on if exploits are happening in the wild at time of disclosure (i.e. zero day).
Now that we have determined their statistics cannot be reproduced, use a flawed methodology, and are based on drastically incomplete data, let’s examine their conclusions anyway!
The blog announcing the report is titled “1,208 vulnerabilities in the 50 most popular programs – 76% from third-party programs” and immediately calls into question their perspective. Reading down a bit, we find out what they mean by “third-party programs”:
“And the findings in the Secunia Vulnerability Review 2014 support that, once again, the biggest vulnerability threat to corporate and private security comes from third-party – i.e. non-Microsoft – programs.”
Unfortunately, this is not the definition of a third-party program by most in our industry. On a higher more general level, a “third-party software component” is a “is a reusable software component developed to be either freely distributed or sold by an entity other than the original vendor of the development platform” (Wikipedia). In the world of VDBs, we frequently refer to a third-party component a ‘library‘ that is integrated into a bigger package. For example, Adobe Reader 10 which is found on many desktop computers is actually built on Adobe’s own code, but also as many as 212 other pieces of software. The notion that “non-Microsoft” software is “third-party” is very weird for lack of better words, and shows the mindset and perspective of Secunia. This completely discounts users of Apple, Linux, VMs (e.g. Oracle, VMware, Citrix), and mobile devices among others. Such a Microsoft-centric report should clearly be labeled as such, not as a general vulnerability report.
In the Top 50 programs, a total of 1,208 vulnerabilities were discovered in 2013. Third-party programs were responsible for 76% of those vulnerabilities, although these programs only account for 34% of the 50 most popular programs on private PCs. The share of Microsoft programs (including the Windows 7 operating system) in the Top 50 is a prominent 33 products – 66%. Even so, Microsoft programs are only responsible for 24% of the vulnerabilities in the Top 50 programs in 2013.
This is aiming for the most convoluted summary award apparently. I really can’t begin to describe how poorly this comes across. If you want to know the ‘Top 50 programs’, you have to read down to page 18 of the PDF and then resolve a lot of questions, some of which will be touched on below. When you read the list, and see that several ‘Microsoft’ programs actually had 0 vulnerabilities, it will call into question the “prominent 33 products” and show how the 66% is incorrectly weighted.
“However, another very important security factor is how easy it is to update Microsoft programs compared to third-party programs.” — Secunia CTO, Morten R. Stengaard.
When debunking vulnerability statistics, I tend to focus on the actual numbers. This is a case where I simply have to branch out and question how a ‘CTO’ could make this absurd statement. In one sentence, he implies that updating Microsoft is easy, while third-party programs (i.e. non-Microsoft programs per their definition) are not. Apparently Mr. Stengaard does not use Oracle Java, Adobe Flash player, Adobe Air, Adobe Reader, Mozilla Firefox, Mozilla Thunderbird, Google Chrome, Opera, or a wide range of other non-Microsoft desktop software, all of which have the same one-click patching/upgrade ability. Either Mr. Stengaard is not qualified to speak on this topic, or he is being extremely disingenuous in his characterization of non-Microsoft products to suit the needs of supporting this report and patch management business model. If he means that patching Windows is easier on an enterprise scale (e.g. via SCCM or WSUS), then that is frequently true, but such qualifications should be clear.
This is a case where using a valid and accepted definition of ‘third-party programs’ (e.g. a computing library) would make this quote more reasonable. Trying to upgrade ffmpeg, libav, or WebKit in the context of the programs that rely on them as libraries is not something that can be done by the average user. The problem is further compounded when portions of desktop software are used as a library in another program, such as AutoCad which appears in the Adobe Reader third-party license document linked above. However, these are the kinds of distinctions that any VDB should be fully aware of, and be able to disclaim and explain more readily.
Moving on to the actual ‘Secunia Vulnerability Review 2014‘ report, the very first line opens up a huge can of worms as the number is incorrect and entirely misleading. The flawed methodology used to generate the statistic cascades down into a wide variety of other incorrect conclusions.
The absolute number of vulnerabilities detected was 13,073, discovered in 2,289 products from 539 vendors.
It is clear that there are a significant amount of vulnerabilities that are being counted multiple times. While this number is generated from Secunia’s internal ‘vulnerability count’ number associated with each advisory, they miss the most obvious flaw; that many of their advisories cover the exact same vulnerability. Rather than abstract so that one advisory is updated to reflect additional products impacted, Secunia will release additional advisories. This is immediately visible in cases where a protocol is found to have a vulnerability, such as the “TLS / DTLS Protocol CBC-mode Ciphersuite Timing Analysis Plaintext Recovery Cryptanalysis Attack” (OSVDB 89848). This one vulnerability impacts any product that implements that protocol, so it is expected to be widespread. As such, that one vulnerability tracks to 175 different Secunia advisories. This is not a case where 175 different vendors coded the same vulnerability or the issue is distinct in their products. This is a case of a handful of base products (e.g. OpenSSL, GnuTLS, PolarSSL) implementing the flawed protocol, and hundreds of vendors using that software bundled as part of their own.
While that is an extreme example, the problem is certainly front-and-center due to their frequent multi-advisory coverage of the same issue. Consider that one OpenSSL vulnerability may be covered in 11 Secunia advisories. Then look at other products that are frequently used as libraries or found on multiple Linux distributions, each of which get their own advisory. Below is a quick chart showing examples of a single vulnerability in one of several products, along with the number of Secunia advisories that references that one vulnerability:
|Example w/ 1 Vuln||# of Secunia Adv|
|CVE-2013-6367 Linux Kernel||15|
|CVE-2013-6644 Google Chrome||5|
|CVE-2013-6415 Ruby on Rails||10|
|CVE-2014-0368 Oracle Java||27|
This problem is further compounded when you consider the number of vulnerabilities in those products in 2013, where each one received multiple Secunia advisories. This table shows the products from above, and the number of unique vulnerabilities as tracked by OSVDB for that product in 2013 that had at least one associated Secunia advisory:
|Software||# of Vulns in product in 2013 w/ Secunia Ref|
|Ruby on Rails||14|
It is easy to see how Secunia quickly jumped to 13,073 vulnerabilities while only issuing 3,327 advisories in 2013. If there is any doubt about vulnerability count inflation, consider these four Secunia advisories that cover the same set of vulnerabilities, each titled “WebSphere Application Server Multiple Java Vulnerabilities“. Secunia created four advisories for the same vulnerabilities simply to abstract based on the major versions affected, as seen in this table:
|Secunia Advisory||# of Vulns in product in 2013|
|56778||reported in versions 18.104.22.168 through 22.214.171.124.|
|56852||reported in versions 126.96.36.199 through 188.8.131.52.|
|56891||reported in version 184.108.40.206 through 220.127.116.11.|
|56897||reported in versions 18.104.22.168 through 22.214.171.124.|
The internal ‘vulnerability count’ for these advisories are very likely 25, 25, 25, and 27, adding up to 102. Applied against IBM, you have 27 vulnerabilities inflated greatly and counting for 102 instead. Then consider that IBM has several hundred products that use Java, OpenSSL, and other common software. It is easy to see how Secunia could jump to erroneous conclusions:
The 32% year-on-year increase in the total number of vulnerabilities from 2012 to 2013 is mainly due to a vulnerability increase in IBM products of 442% (from 772 vulnerabilities in 2012 to 4,181 in 2013).
The next set of statistics is convoluted on the surface, but even more confusing when you read the details and explanations for how they were derived:
Numbers – Top 50 portfolio
The number of vulnerabilities in the Top 50 portfolio was 1,208, discovered in 27 products from 7 vendors plus the most used operating system, Microsoft Windows 7.
To assess how exposed endpoints are, we analyze the types of products typically found on an endpoint. Throughout 2013, anonymous data has been gathered from scans of the millions of private computers which have the Secunia Personal Software Inspector (PSI) installed. Secunia data shows that the computer of a typical PSI user has an average of 75 programs installed on it. Naturally, there are country- and region-based variations regarding which programs are installed. Therefore, for the sake of clarity, we chose to focus on a representative portfolio of the 50 most common products found on a typical computer and the most used operating system, and analyze the state of this portfolio and operating system throughout the course of 2013. These 50 programs are comprised of 33 Microsoft programs and 17 non-Microsoft (third-party) programs.
Reading down to page 18 of the full report, you see the table listing the “Top 50” software installed as determined by their PSI software. On the list is a wide variety of software that are either components of Windows (meaning they come installed by default, but show up in the “Programs” list e.g. Microsoft Visual C++ Redistributable) or in a few cases third-party software (e.g. Google Toolbar), many of which have 0 associated vulnerabilities. In other cases they include product driver support tools (e.g. Realtek AC 97 Update and Remove Driver Tool) or ActiveX components that are generally not installed via traditional means (e.g. comdlg32 ActiveX Control). With approximately half of the Top 50 software having vulnerabilities, and mixing different types of software components, it causes summary put forth by Secunia to be misleading. Since they include Google Chrome on the list, by their current logic, they should also include WebKit which is a third-party library wrapped into Chrome, just as they include ‘Microsoft Powerpoint Viewer’ (33) which is a component of ‘Microsoft Powerpoint’ (14) and does not install separately.
Perhaps the most disturbing thing about this Top 50 summary is that Secunia only counts 7 vendors in their list. Reading through the list carefully, you see that there are actually 10 vendors represented: Microsoft, Adobe, Oracle, Mozilla, Google, Realtek, Apple, Piriform (CCleaner), VideoLAN, and Flexera (InstallShield). This seriously calls into question any conclusions put forth by Secunia regarding their Top 50 list and challenges their convoluted and irreproducible methodology.
Rather than offer a rebuttal line by line for the rest of the report and blog, we’ll just look at some of the included statistics that are questionable, wrong, or just further highlight that Secunia has missed some vulnerabilities.
In 2013, 727 vulnerabilities were discovered in the 5 most popular browsers: Google Chrome, Mozilla Firefox, Internet Explorer, Opera and Safari.
By our count, there were at least 756 vulnerabilities in these browsers: Google Chrome (295), Mozilla Firefox (155), Internet Explorer (138), Opera (9), Apple Safari (8 on desktop, 4 on mobile), and WebKit (component of Chrome and Safari, 147). The count in Opera is likely very low though. In July 2013, Opera issued the first browser based on Blink, so it’s very likely that it has been affected by the vast majority of the Blink vulnerability fixes by Google. However, Opera is not very good at clearly reporting vulnerabilities, so this very likely accounts for the very low count that both we and Secunia have; something they should clearly have disclaimed.
In 2013, 70 vulnerabilities were discovered in the 5 most popular PDF readers: Adobe Reader, Foxit Reader, PDF-XChange Viewer, Sumatra PDF and Nitro PDF Reader.
By our count, there were at least 76 vulnerabilities in these PDF readers: Adobe Reader (69), Foxit (2), PDF-XChange (1), Sumatra (0), and Nitro (4).
The actual vulnerability count in Microsoft programs was 192 in 2013; 128.6% higher than in 2012.
Based on our data, there were 363 vulnerabilities in Microsoft software in 2013, not 192. This is up from 207 in 2012, giving us a 175.3% increase.
As in 2012, not many zero-day vulnerabilities were identified in 2013: 10 in total in the Top 50 software portfolio, and 14 in All products.
A zero-day vulnerability is a vulnerability that is actively exploited by hackers before it is publicly known, and before the vendor has published a patch for it.
By that definition, which we share, we tracked 72 vulnerabilities that were “discovered in the wild” in 2013. To be fair, our number is considerably higher because we actually track mobile vulnerabilities, something Secunia typically ignores. More curious is that based on a cursory search, we find 17 of their advisories that qualify as 0-day by their definition, suggesting they do not have a method for accurately counting them: SA51820 (1), SA52064 (1), SA52116 (2), SA52196 (2), SA52374 (2), SA52451 (1), SA53314 (1), SA54060 (1), SA54274 (1), SA54884 (2), SA55584 (1), SA55611 (1), and SA55809 (1).
Find out how quickly software vendors issue fixes – so-called patches – when vulnerabilities are discovered in All products.
This comes from their “Time to Patch for all products” summary page. This statement seems pretty clear; How fast do vendors issue fixes when vulnerabilities are discovered? However, Secunia does not track that specifically! The more appropriate question that can be answered by their data is “When are patches available at or after the time of public disclosure?” These are two very different metrics. The information on this page is generated using PSI/CSI statistics. So if a vulnerability is disclosed and a fix is already available at that time, it counts as within 24 hours. It doesn’t factor in that the vendor may have spent months fixing the issue before disclosure and patch.
In conclusion, while we appreciate companies sharing vulnerability intelligence, the Secunia 2013 vulnerability report is ultimately fluff that provides no benefit to organizations. The flawed methodology and inability for them to parse their own data means that the conclusions cannot be relied upon for making business decisions. When generating vulnerability statistics, a wide variety of bias will always be present. It is absolutely critical that your vulnerability aggregation methodology be clearly explained so that results are qualified and have more meaning.