In our pursuit of a more complete historical record of vulnerabilities, we’re offering a bounty! We don’t want your 0-day really. OK sure we do, but we know you are stingy with that, so we’ll settle on your ~ 12,775 day exploits!
First, the bounty. This is coming out my pocket since it is legacy and doesn’t immediately benefit people using us as a vulnerability feed. As such, this isn’t going to be a profit center for you. In addition to the personal satisfaction of helping preserve history, shout outs on this blog and multiple Twitter feeds, I will send you something. Want a gift card for Amazon? Something else I have that you want? I’ll make my best effort to make it reasonably worth your while. I know it isn’t a cool $1,337 Google style unfortunately, but I will try!
Now, what am I after. Not “a” vulnerability, but any of several lists of vulnerabilities from decades ago. These were maintained in the 1980’s most likely, one of which was internal at the time. I am hoping that given the time that has passed, and that the vulnerabilities have long since been patched and most products EOL’d, they can be disclosed. If you don’t have a copy but know someone might, send me a virtual introduction please! Any lead that results in me getting my hands on a list will be rewarded in some fashion as well. If you have a copy but it is buried in a box in the garage, let me know. I will see about traveling to help you dig through junk to find it. Seriously, that is how bad I want these historic lists!
- The Unix Known Problem List (this was not one of the vendor-specific lists, but those may be groovy)
- UC Santa Cruz hack method list
- Mt. Xinu bug list (later than 4.2 or with more details than this copy)
- Matt Bishop’s UNIX Hole List
- Sun Microsystems Bug-List (internal at the time no doubt)
- ISIS mail list archive (one run by Andrew Burt in 80’s)
- Bjorn Satedevas’ systems administration mailing list archive
- The “inner” Zardoz mail list archive (split from the main one, less members)
Any public-referenced vulnerability before 1980 that we do not have in the database. I know there has to be more out there, help us find them!
Bonus bonus bounty (for SCADA types):
Any SCADA or ICS vulnerability before 1985-06-01!
That’s it! Pretty simple, but may require some digging mentally or physically.
When referencing vulnerabilities in your products, you have a habit of only using an internal tracking number that is kept confidential between the reporter (e.g. ICS-CERT, ZDI) and you. For example, from your HotFix page (that requires registration):
WI2815: Directory Traversal Buffer overflow. Provided and/or discovered by: ICS-CERT ticket number ICS-VU-579709 created by Anthony …
The ICS-CERT ticket number is assigned as an internal tracking ID while the relevant parties figure out how to resolve the issue. Ultimately, that ticket number is not published by ICS-CERT. I have already sent a mail to them suggesting they include it in advisories moving forward, to help third parties match up vulnerabilities to fixes to initial reports. Instead of using that, you should use the public ICS-CERT advisory ID. The details you provide there are not specific enough to know which issue this corresponds to.
In another example:
WI2146: Improved the Remote Agent utility (CEServer.exe) to implement authentication between the development application and the target system, to ensure secure downloading, running, and stopping of projects. Also addressed problems with buffer overrun when downloading large files. Credits: ZDI reports ZDI-CAN-1181 and ZDI-CAN-1183 created by Luigi Auriemma
In this case, these likely correspond to OSVDB 77178 and 77179, but it would be nice to know that for sure. Further, we’d like to associate those internal tracking numbers to the entries but vendors do not reliably put them in order, so we don’t know if ZDI-CAN-1181 corresponds to the first or second.
WI1944: ISSymbol Virtual Machine buffer overflow Provided and/or discovered by: ZDI report ZDI-CAN-1341 and ZDI-CAN-1342
In this case, you give two ZDI tracking identifiers, but only mention a single vulnerability. ZDI has a history of abstracting issues very well. The presence of two identifiers, to us, means there are two distinct vulnerabilities.
This is one of the primary reasons CVE exists, and why ZDI, ICS-CERT, and most vendors now use it. In most cases, these larger reporting bodies will have a CVE number to share with you during the process, or if not, will have one at the time of disclosure.
Like your customers do, we appreciate clear information regarding vulnerabilities. Many large organizations will use a central clearing house like ours for vulnerability alerting, rather than trying to monitor hundreds of vendor pages. Helping us to understand your security patches in turn helps your customers.
Finally, adding a date the patch was made available will help to clarify these issues and give another piece of information that is helpful to organizations.
Thank you for your consideration in improving your process!
Anyone who knows me in the context of vulnerability databases will find this post a tad shocking, even if they have endured my rants about it before.
For the first time ever, I am making it policy that we will no longer put any priority on Vulnerability Labs advisories. For those unfamiliar with the site, it is run by Benjamin Kunz Mejri who now has a new company Evolution Security.
If you read that web site, and even a history of his/VL disclosures, it looks impressive on the surface. Yes, they have found some legitimate vulnerabilities, even in high-profile vendors. Most, if not all, are pedestrian web application vulnerabilities such as cross-site scripting, traversals, or file upload issues. More complex vulnerabilities like overflows typically end up being what we call “self hacks”, and do not result in the crossing of privilege boundaries. Many of their published vulnerabilities require excessive conditions and offer no real exploit scenario.
During the past 10 months, I know of three other vulnerability databases that officially gave up on adding their advisories. Nothing public, but the internal memo was “don’t bother”. OSVDB was the holdout. We did our best to keep up with their stream of horrible advisories. I personally offered to help them re-write and refine their advisory process several times. I started out nicely, giving a sincere offer of my time and experience, and it went unanswered. I slowly escalated, primarily on Twitter, giving them grief over their disclosures. Eventually, their advisories became nothing but an annoyance and incredible time sink. Then I got ugly, and I have been to this day. No, not my proudest moment, but I stand by it 100%.
As of tonight, we are giving in as well. Vulnerability Lab advisories represent too much of a time sink, trying to decipher their meaning, that they simply aren’t worth adding. For cases where the software is more notable, we will continue to slam our head against the wall and figure them out. For the rest, they are getting deprioritized in the form of a “to do when we run out of other import sources”. Since we monitor over 1,100 sources including blogs, web sites, changelogs, and bug trackers, this is not happening for a long time.
I truly regret having to do this. One of my biggest joys of running a vulnerability database is in cataloging all the vulnerabilities. ALL OF THEM.
So this also serves as my final offer Benjamin. Search the VDBs out there and notice how few of your advisories end up in them. Think about why that is. If you are as smart as you think you are, you will choke down your pride and accept my offer of help. I am willing to sink a lot of time into helping you improve your advisories. This will in turn help the rest of the community, and what I believe are your fictitious customers. As I have told you several times before, there is no downside to this for you, just me. I care about helping improve security. Do you?
Tonight, shortly before retiring from a long day of vulnerability import, I caught a tweet mentioning a web site about reporting vulnerabilities. Created on 15-aug-2013 per whois, the footer shows it was written by Fraser Scott, aka @zeroXten on Twitter.
This time, the web site is directly related to what we do. I want to be very clear here; I like the goal of this site. I like the simplistic approach to helping the reader decide which path is best for them. I want to see this site become the top result when searching for “how do I disclose a vulnerability?” This commentary is only meant to help the author improve the site. Please, take this advice to heart, and don’t hesitate if you would like additional feedback. [Update: After starting this blog last night, before publishing this morning, he already reached out. Awesome.]
Under the ‘What’ category, there are three general disclosure options:
NON DISCLOSURE, RESPONSIBLE DISCLOSURE, and FULL DISCLOSURE
First, you are missing a fourth option of ‘limited disclosure’. Researchers can announce they have found a vulnerability in given software, state the implications, and be done with it. Public reports of code execution in some software will encourage the vendor to prioritize the fix, as customers may begin putting pressure on them. Adding a video showing the code execution reinforces the severity. It often doesn’t help a VDB like ours, because such a disclosure typically doesn’t have enough actionable information. However, it is one way a researcher can disclose, and still protect themselves.
Second, “responsible”? No. The term was possibly coined by Steve Christey, further used by Russ Cooper, that was polarized by Cooper as well as Scott Culp at Microsoft (“Information Anarchy”, really?), in a (successful) effort to brand researchers as “irresponsible” if they don’t conform to vendor disclosure demands. The appropriate term more widely recognized, and fair to both sides, is that of “coordinated” disclosure. Culp’s term forgets that vendors can be irresponsible if they don’t prioritize critical vulnerabilities while customers are known to be vulnerable with public exploit code floating about. Since then, Microsoft and many other companies have adopted “coordinated” to refer to the disclosure process.
Under the ‘Who’ category, there are more things to consider:
SEND AN EMAIL
These days, it is rare to see domains following RFC-compliant addresses. That is a practice mostly lost to the old days. Telling readers to try to “Contact us” tab/link that invariably shows on web pages is better. Oh wait, you do that. However, that comes well after the big header reading TECHNICAL SUPPORT which may throw people off.
As a quick side note: “how to notifying them of security issues”. This is one of many spelling or grammar errors. Please run the text through a basic grammar checker.
Under the ‘How’ category:
This is excellent advice, except that using Tor bit since there are serious questions about the security/anonymity of it. If researchers are worried, they should look at a variety of options including using a coffee shop’s wireless, hotel wireless, etc.
This is also a great point, but more to the point, make sure your mail is polite and NOT THREATENING. Don’t threaten to disclose on your own timeline. See how the vendor handles the vulnerability report without any indication of disclosing it. Give them benefit of the doubt. If you get hints they are stalling at some point, then gently suggest it may be in the best interest of their customers to disclose. Remind them that vulnerabilities are rarely discovered by a single person and that they can’t assume you are the only one who has found it. You are just the only one who apparently decided to help the vendor.
Post to Full-Disclosure sure, or other options that may be more beneficial to you. Bugtraq has a history of stronger moderation, they tend to weed out crap. Send it directly to vulnerability databases and let them publish it anonymously. VDBs like Secunia generally validate all vulnerabilities before posting to their database. That may help you down the road if your intentions are called into question. Post to the OSS-security mail list if the vulnerability is in open-source software, so you get the community involved. For that list, getting a CVE identifier and having others on the list verifying or sanity checking your findings, it gives more positive attention to the technical issues instead of the politics of disclosure.
Using a bug bounty system is a great idea as it keeps the new researcher from dealing with disclosure politics generally. Let people experienced with the process, who have an established relationship and history with the vendor handle it. However, don’t steer newcomers to ZDI immediately. In fact, don’t name them specifically unless you have a vested interest in helping them, and if so, state it. Instead, break it down into vendor bug bounty programs and third-party programs. Provide a link to Bugcrowd’s excellent resource on a list of current bounty programs.
The fine print of course. Under CITATIONS, I love that you reference the Errata legal threats page, but this should go much higher on the page. Make sure new disclosers know the potential mess they are getting into. We know people don’t read the fine print. This could also be a good lead-in to using a third-party bounty or vulnerability handling service.
It’s great that you make this easy to share with everyone and their dog, but please consider getting a bit more feedback before publishing a site like this. It appears you did this in less than a day, when an extra 24 hours shows you could have made a stronger offering. You are clearly eager to make it better. You have already reached out to me, and likely Steve Christey if not others. As I said, with some edits and fix-ups, this will be a great resource.
Last week, Steve Christey and I gave a presentation at Black Hat Briefings 2013 in Las Vegas about vulnerability statistics. We submitted a brief whitepaper on the topic, reproduced below, to accompany the slides that are now available.
Buying Into the Bias: Why Vulnerability Statistics Suck
By Steve Christey (MITRE) and Brian Martin (Open Security Foundation)
July 11, 2013
Academic researchers, journalists, security vendors, software vendors, and professional analysts often analyze vulnerability statistics using large repositories of vulnerability data, such as “Common Vulnerabilities and Exposures” (CVE), the Open Sourced Vulnerability Database (OSVDB), and other sources of aggregated vulnerability information. These statistics are claimed to demonstrate trends in vulnerability disclosure, such as the number or type of vulnerabilities, or their relative severity. Worse, they are typically misused to compare competing products to assess which one offers the best security.
Most of these statistical analyses demonstrate a serious fault in methodology, or are pure speculation in the long run. They use the easily-available, but drastically misunderstood data to craft irrelevant questions based on wild assumptions, while never figuring out (or even asking the sources about) the limitations of the data. This leads to a wide variety of bias that typically goes unchallenged, that ultimately forms statistics that make headlines and, far worse, are used to justify security budget and spending.
As maintainers of two well-known vulnerability information repositories, we’re sick of hearing about research that is quickly determined to be sloppy after it’s been released and gained public attention. In almost every case, the research casts aside any logical approach to generating the statistics. They frequently do not release their methodology, and they rarely disclaim the serious pitfalls in their conclusions. This stems from their serious lack of understanding about the data source they use, and how it operates. In short, vulnerability databases (VDBs) are very different and very fickle creatures. They are constantly evolving and see the world of vulnerabilities through very different glasses.
This paper and its associated presentation introduce a framework in which vulnerability statistics can be judged and improved. The better we get about talking about the issues, the better the chances of truly improving how vulnerability statistics are generated and interpreted.
Bias, We All Have It
Bias is inherent in everything humans do. Even the most rigorous and well-documented process can be affected by levels of bias that we simply do not understand are working against us. This is part of human nature. As with all things, bias is present in the creation of the VDBs, how the databases are populated with vulnerability data, and the subsequent analysis of that data. Not all bias is bad; for example, VDBs have a bias to avoid providing inaccurate information whenever possible, and each VDB effectively has a customer base whose needs directly drive what content is published.
Bias comes in many forms that we see as strongly influencing vulnerability statistics, via a number of actors involved in the process. It is important to remember that VDBs catalog the public disclosure of security vulnerabilities by a wide variety of people with vastly different skills and motivations. The disclosure process varies from person to person and introduces bias for sure, but even before the disclosure occurs, bias has already entered the picture.
Consider the general sequence of events that lead to a vulnerability being cataloged in a VDB.
- A researcher chooses a piece of software to examine.
- Each researcher operates with a different skill set and focus, using tools or techniques with varying strengths and weaknesses; these differences can impact which vulnerabilities are capable of being discovered.
- During the process, the researcher will find at least one vulnerability, often more.
- The researcher may or may not opt for vendor involvement in verifying or fixing the issue.
- At some point, the researcher may choose to disclose the vulnerability. That disclosure will not be in a common format, may suffer from language barriers, may not be technically accurate, may leave out critical details that impact the severity of the vulnerability (e.g. administrator authentication required), may be a duplicate of prior research, or introduce a number of other problems.
- Many VDBs attempt to catalog all public disclosures of information. This is a “best effort” activity, as there are simply too many sources for any one VDB to monitor, and accuracy problems can increase the expense of analyzing a single disclosure.
- If the VDB maintainers see the disclosure mentioned above, they will add it to the database if it meets their criteria, which is not always public. If the VDB does not see it, they will not add it. If the VDB disagrees with the disclosure (i.e. believes it to be inaccurate), they may not add it.
By this point, there are a number of criteria that may prevent the disclosure from ever making it into a VDB. Without using the word, the above steps have introduced several types of bias that impact the process. These biases carry forward into any subsequent examination of the database in any manner.
Types of Bias
Specific to the vulnerability disclosure aggregation process that VDBs go through every day, there are four primary types of bias that enter the picture. Note that while each of these can be seen in researchers, vendors, and VDBs, some are more common to one than the others. There are other types of bias that could also apply, but they are beyond the scope of this paper.
Selection bias covers what gets selected for study. In the case of disclosure, this refers to the researcher’s bias in selecting software and the methodology used to test the software for vulnerabilities; for example, a researcher might only investigate software written in a specific language and only look for a handful of the most common vulnerability types. In the case of VDBs, this involves how the VDB discovers and handles vulnerability disclosures from researchers and vendors. Perhaps the largest influence on selection bias is that many VDBs monitor a limited source of disclosures. It is not necessary to argue what “limited” means. Suffice it to say, no VDB is remotely complete on monitoring every source of vulnerability data that is public on the net. Lack of resources – primarily the time of those working on the database – causes a VDB to prioritize sources of information. With an increasing number of regional or country-based CERT groups disclosing vulnerabilities in their native tongue, VDBs have a harder time processing the information. Each vulnerability that is disclosed but does not end up in the VDB, ultimately factors into statistics such as “there were X vulnerabilities disclosed last year”.
Publication bias governs what portion of the research gets published. This ranges from “none”, to sparse information, to incredible technical detail about every finding. Somewhere between selection and publication bias, the researcher will determine how much time they are spending on this particular product, what vulnerabilities they are interested in, and more. All of this folds into what gets published. VDBs may discover a researcher’s disclosure, but then decide not to publish the vulnerability due to other criteria.
Abstraction bias is a term that we crafted to explain the process that VDBs use to assign identifiers to vulnerabilities. Depending on the purpose and stated goal of the VDB, the same 10 vulnerabilities may be given a single identifier by one database, and 10 identifiers by a different one. This level of abstraction is an absolutely critical factor when analyzing the data to generate vulnerability statistics. This is also the most prevalent source of problems for analysis, as researchers rarely understand the concept of abstraction, why it varies, and how to overcome it as an obstacle in generating meaningful statistics. Researchers will use whichever abstraction is most appropriate or convenient for them; after all, there are many different consumers for a researcher advisory, not just VDBs. Abstraction bias is also frequently seen in vendors, and occasionally researchers in the way they disclose one vulnerability multiple times, as it affects different software that bundles additional vendor’s software in it.
Measurement bias refers to potential errors in how a vulnerability is analyzed, verified, and catalogued. For example, with researchers, this bias might be in the form of failing to verify that a potential issue is actually a vulnerability, or in over-estimating the severity of the issue compared to how consumers might prioritize the issue. With vendors, measurement bias may affect how the vendor prioritizes an issue to be fixed, or in under-estimating the severity of the issue. With VDBs, measurement bias may also occur if analysts do not appropriately reflect the severity of the issue, or if inaccuracies are introduced while studying incomplete vulnerability disclosures, such as missing a version of the product that is affected by the vulnerability. It could be argued that abstraction bias is a certain type of measurement bias (since it involves using inconsistent “units of measurement”), but for the purposes of understanding vulnerability statistics, abstraction bias deserves special attention.
Measurement bias, as it affects statistics, is arguably the domain of VDBs, since most statistics are calculated using an underlying VDB instead of the original disclosures. As the primary sources of vulnerability data aggregation, several factors come into play when performing database updates.
Why Bias Matters, in Detail
These forms of bias can work together to create interesting spikes in vulnerability disclosure trends. To the VDB worker, they are typically apparent and sometimes amusing. To an outsider just using a data set to generate statistics, they can be a serious pitfall.
In August, 2008, a single researcher using rudimentary, yet effective methods for finding symlink vulnerabilities single handedly caused a significant spike in symlink vulnerability disclosures over the past 10 years. Starting in 2012 and continuing up to the publication of this paper, a pair of researchers have significantly impacted the number of disclosures in a single product. Not only has this caused a huge spike for the vulnerability count related to the product, it has led to them being ranked as two of the top vulnerability disclosers since January, 2012. Later this year, we expect there to be articles written regarding the number of supervisory control and data acquisition (SCADA) vulnerabilities disclosed from 2012 to 2013. Those articles will be based purely on vulnerability counts as determined from VDBs, likely with no mention of why the numbers are skewed. One prominent researcher who published many SCADA flaws has changed his personal disclosure policy. Instead of publicly disclosing details, he now keeps them private as part of a competitive advantage of his new business.
Another popular place for vulnerability statistics to break down is related to vulnerability severity. Researchers and journalists like to mention the raw number of vulnerabilities in two products and try to compare their relative security. They frequently overlook the severity of the vulnerabilities and may not note that while one product had twice as many disclosures, a significant percentage of them were low severity. Further, they do not understand how the industry-standard CVSSv2 scoring system works, or the bias that can creep in when using it to score vulnerabilities. Considering that a vague disclosure that has little actionable details will frequently be scored for the worst possible impact, that also drastically skews the severity ratings.
The forms of bias and how they may impact vulnerability statistics outlined in this paper are just the beginning. For each party involved, for each type of bias, there are many considerations that must be made. Accurate and meaningful vulnerability statistics are not impossible; they are just very difficult to accurately generate and disclaim.
Our 2013 BlackHat Briefings USA talk hopes to explore many of these points, outline the types of bias, and show concrete examples of misleading statistics. In addition, we will show how you can easily spot questionable statistics, and give some tips on generating and disclaiming good statistics.
Our sponsor Risk Based Security (RBS) posted an interesting blog this morning about Research In Motion (RIM), creator of the BlackBerry device. The behavior outlined in the blog, and from the original blog by Frank Rieger is shocking to say the least. In addition to the vulnerability outlined, potentially sending credentials in cleartext, this begs the question of legality. Quickly skimming the BlackBerry enterprise end-user license agreement (EULA), there doesn’t appear to be any warning that the credentials are transmitted back to RIM, or that they will authenticate to your mail server.
If the EULA does not contain explicit wording that outlines this behavior, it begs the question of the legality of RIM’s actions. Regardless of their intention, wether trying to claim that it is covered in the EULA or making it easier to use their device, this activity is inexcusable. Without permission, unauthorized possession of authentication credentials is a violation of Title 18 USC § 1030 law, section (a)(2)(C) and potentially others depending on the purpose of the computer. Since the server doing this resides in Canada, RIM may be subject to Canadian law and their activity appears to violate Section 342.1 (d). Given the U.S. government’s adoption of BlackBerry devices, if RIM is authenticating to U.S. government servers during this process, this could get really messy.
Any time a user performs an action that would result in sharing that type of information, with any third party, the device or application should give explicit warning and require the user to not only opt-in, but confirm their choice. No exceptions.
Since vulnerabilities are a cornerstone of computer crime, stats on it are of interest to us. Statistics on cybercrime have always been dodgy; more so than real-world crime statistics. When your car is broken into or stolen, you know it. More often than not, you will report it to the police. In the computer world, an un-measurable number of intrusions happen every day. The rate of malware infection, DoS attacks, and other virtual crimes are not only
difficult impossible to measure, they potentially go unreported more often than not.
Classically, the only number thrown around regarding damages from cybercrime has been this mythical one trillion dollars. Yes, with a ‘T’, not a ‘B’. That number has been challenged by many in the past, but no one has offered a better number with anything remotely factual. On July 22 the Center for Strategic and International Studies released a new study commissioned by McAfee (who previously quoted the trillion dollar figure) saying that damages are much less. From a Los Angeles Times article on the release:
Cyberattacks may be draining as much as $140 billion and half a million jobs from the U.S. economy each year, according to a new study that splashes water on a previous estimate of $1 trillion in annual losses.
“That’s our best guess,” said James Andrew Lewis, the director of the technology and public policy program at the Center for Strategic and International Studies.
James Andrew Lewis’ comment calling it a “best guess” is not assuring. The one trillion dollar figure cited for all those years was no better than a guess, as the surveys it relied on were far from a solid methodology. Regardless, the figure of $140 billion seems more rationale on the surface. Contrasting that is the claim that half a million jobs are “drained” from the U.S. economy each year. How can cybercrime conceivably lead to that? Reading on in the article:
Lewis and co-author Stewart Baker, a distinguished visiting fellow at CSIS, said that they were still working to determine cybercrime’s impact on innovation. They suggested a follow-up report might come out with a bigger number.
But preliminarily, they found U.S. losses to be somewhere between $20 billion to $140 billion, or about 1% of the nation’s GDP. They pegged job losses at 508,000.
“The effect of the net loss of jobs could be small, but if a good portion of these jobs were high-end manufacturing jobs that moved overseas because of intellectual property losses, the effect could be wide ranging,” Lewis said.
Right after the hint of a more rational number, CSIS immediately makes it a worthless number when they say it is really somewhere between $20 billion and $140 billion. In the world of sanity and statistics, that range is unreasonable. Further, Lewis goes on to say that some of the 508,000 jobs lost are due to “high-end manufacturing jobs moved overseas because of intellectual property losses”. Huh? High-end manufacturing jobs are moving overseas because of corporate budgets more than cybercrime. Such a claim should be backed up by a long list of examples showing companies losing intellectual property, and then reporting it to law enforcement or their shareholders, as well as SEC filings.
We moved from the fictional trillion number, to an overly wide range in the tens or hundreds of billions, and got an odd claim of half a million jobs lost due to cybercrime. This new study did little to clear things up.
If we’re killing one cybercrime myth, let’s kill another — one which coincidentally emerged from McAfee — namely that the wealth transfer due to hacking represents some historically-unprecedented economic disaster.
Ultimately, we also have to remember that any cybercrime statistics coming from a company like McAfee are suspect, as they directly benefit them while they sell computer security solutions.
About two weeks ago, another round of vulnerability stats got passed around. Like others before, it claims to use CVE to compare Apple iOS versus Android in an attempt to establish which is more secure based on “vulnerability counts”. The statistics put forth are basically meaningless, because like most people using a VDB to generate stats, they don’t fully understand their data source. This is one type of bias that enters the picture when generating statistics, and one of many points Steve Christey (MITRE/CVE) and I will be making next week at BlackHat (Wednesday afternoon).
As with other vulnerability statistics, I will debunk the latest by showing why the conclusions are not based on a solid understanding of vulnerabilities, or vulnerability data sources. The post is published on The Verge, written by ‘Mechanicix’. The results match last year’s Symantec Internet Security Threat Report (as mentioned in the comments), as well as the results published this year by Sourcefire in their paper titled “25 Years of Security Vulns“. In all three cases, they use the same data set (CVE), and do the same rudimentary counting to reach their results.
The gist of the finding is that Apple iOS is considerably less secure than Android, as iOS had 238 reported vulnerabilities versus the 27 reported in Android, based on CVE and illustrated through CVEdetails.com.
Total iOS Vulnerabilities 2007-2013: 238
Total Android Vulnerabilities 2009-2013: 27
Keeping in mind those numbers, if you look at the CVE entries that are included, a number of problems are obvious:
- We see that the comparison timeframes differ by two years. There are at least 3 vulnerabilities in Android SDK reported before 2009, two of which have CVEs (CVE-2008-0985 and CVE-2008-0986).
- These totals are based on CVE identifiers, which does not necessarily reflect a 1-to-1 vulnerability mapping, as they document. You absolutely cannot count CVE as a substitute for vulnerabilities, they are not the same.
- The vulnerability totals are incorrect due to using CVE, a data source that has serious gaps in coverage. For example, OSVDB has 71 documented vulnerabilities for Android, and we do not make any claims that our coverage is complete.
- The iOS results include vulnerabilities in WebKit, the framework iOS Safari uses. This is problematic for several reasons.
- First, that means Mechanicix is now comparing the Android OS to the iOS operating system and applications.
- Second, WebKit vulnerabilities account for 109 of the CVE results, almost half of the total reported.
- Third, if they did count WebKit intentionally then the numbers are way off as there were around 700 WebKit vulnerabilities reported in that time frame.
- Fourth, the default browser in Android uses WebKit, yet they weren’t counted against that platform.
- The results include 16 vulnerabilities in Safari itself (or in WebKit and just not diagnosed as such), the default browser.
- At least 4 of the 238 are vulnerabilities in Google Chrome (as opposed to WebKit) with no mention of iOS in the CVE.
- A wide variety of iOS applications are included in the list including Office Viewer, iMessage, Mail, Broadcom BCM4325 and BCM4329 Wi-Fi chips, Calendar, FreeType, libxslt, and more.
When you factor in all of the above, Android likely comes out on top for the number of vulnerabilities when comparing the operating systems. Once again, vulnerability statistics seem simple on the surface. When you consider the above, and further consider that there are likely more points that influence vulnerability counts, we see that it is anything other than simple.
The market for vulnerabilities has grown rapidly the last five years. While the market is certainly not new, going back well over ten years, more organizations are interested in acquiring 0-day / private vulnerabilities for a variety of needs. These vulnerabilities cover the gambit in applications and impacts, and range from the tens of dollars to $100,000 or more. While such transactions are sometimes public, high-end vulnerabilities that sell for large sums generally are not a matter of public record. That makes it difficult to track actual sale prices to gauge the value of such vulnerabilities.
In the vulnerability market place, the seller has the power. If they hold a 0-day vulnerability that is in demand, they can set their own price. For the few vulnerability brokers out there, the perception of vulnerability value is critical for their business. In March, 2013, a Forbes piece by Andy Greenberg covered this topic and told of the sale of an iOS vulnerability that allegedly sold for $250,000.
Even with the $250,000 payout [the Grugq] elicited for that deal, he wonders if he could have gotten more. “I think I lowballed it,” he wrote to me at one point in the dealmaking process. “The client was too happy.”
As expected, there is no validation of the claim of the sale. The price tag comes from the vulnerability broker who has an interest in making such prices public, even if they are exaggerated. Jump to July, 2013, and a New York Times article by Nicole Perlroth and David Sanger makes a vague reference to an iOS vulnerability that sold for $500,000.
Apple still has no such program, but its vulnerabilities are some of the most coveted. In one case, a zero-day exploit in Apple’s iOS operating system sold for $500,000, according to two people briefed on the sale.
Given the vague details, it is fairly safe to assume that it references the iOS vulnerability sale from a year earlier. The NY Times article sources many people regarding vulnerability value, including thegrugq on the first page. This means the vague reference to the “two people briefed on the sale” were likely people briefed by thegrugq as well. Ultimately, this means that both articles and both figures, all source to the same person who has a decided interest in publishing high numbers. Without any detail, the journalists could have contacted one or both sources via email, meaning they could have just as well been thegrugq himself.
I find it interesting that in the span of 1 year and 4 months, the price of that iOS vulnerability jumped from $250,000 to $500,000. More to the point, the original $250,000 price is way out of the league of the prices of vulnerabilities at that time, on any market. Some of us were speculating that a (truly) remote vulnerability in a default Windows installation would go for around $100,000, maybe more. Even if you double our suspected price, it wouldn’t surprise me that a nation-state with a budget would purchase for that amount. But an iOS vulnerability, even remote without user interaction, a year ago? That doesn’t make sense given the user-base and distribution.
Even more interesting, consider that 4 days after the NYTimes article, another outlet was reporting the original $250,000 price.
As I mentioned before, none of this is close to being verified. The only source on record, is someone who directly benefits from the perception that the price of that vulnerability is exceedingly high. Creating the market place value of vulnerabilities through main-stream media is brilliant on his part, if what I suspect is true. Of course, it also speaks to the state of journalism that seemingly no one tried to verify this beyond word-of-mouth.