About two weeks ago, another round of vulnerability stats got passed around. Like others before, it claims to use CVE to compare Apple iOS versus Android in an attempt to establish which is more secure based on “vulnerability counts”. The statistics put forth are basically meaningless, because like most people using a VDB to generate stats, they don’t fully understand their data source. This is one type of bias that enters the picture when generating statistics, and one of many points Steve Christey (MITRE/CVE) and I will be making next week at BlackHat (Wednesday afternoon).
As with other vulnerability statistics, I will debunk the latest by showing why the conclusions are not based on a solid understanding of vulnerabilities, or vulnerability data sources. The post is published on The Verge, written by ‘Mechanicix’. The results match last year’s Symantec Internet Security Threat Report (as mentioned in the comments), as well as the results published this year by Sourcefire in their paper titled “25 Years of Security Vulns“. In all three cases, they use the same data set (CVE), and do the same rudimentary counting to reach their results.
The gist of the finding is that Apple iOS is considerably less secure than Android, as iOS had 238 reported vulnerabilities versus the 27 reported in Android, based on CVE and illustrated through CVEdetails.com.
Total iOS Vulnerabilities 2007-2013: 238
Total Android Vulnerabilities 2009-2013: 27
Keeping in mind those numbers, if you look at the CVE entries that are included, a number of problems are obvious:
- We see that the comparison timeframes differ by two years. There are at least 3 vulnerabilities in Android SDK reported before 2009, two of which have CVEs (CVE-2008-0985 and CVE-2008-0986).
- These totals are based on CVE identifiers, which does not necessarily reflect a 1-to-1 vulnerability mapping, as they document. You absolutely cannot count CVE as a substitute for vulnerabilities, they are not the same.
- The vulnerability totals are incorrect due to using CVE, a data source that has serious gaps in coverage. For example, OSVDB has 71 documented vulnerabilities for Android, and we do not make any claims that our coverage is complete.
- The iOS results include vulnerabilities in WebKit, the framework iOS Safari uses. This is problematic for several reasons.
- First, that means Mechanicix is now comparing the Android OS to the iOS operating system and applications.
- Second, WebKit vulnerabilities account for 109 of the CVE results, almost half of the total reported.
- Third, if they did count WebKit intentionally then the numbers are way off as there were around 700 WebKit vulnerabilities reported in that time frame.
- Fourth, the default browser in Android uses WebKit, yet they weren’t counted against that platform.
- The results include 16 vulnerabilities in Safari itself (or in WebKit and just not diagnosed as such), the default browser.
- At least 4 of the 238 are vulnerabilities in Google Chrome (as opposed to WebKit) with no mention of iOS in the CVE.
- A wide variety of iOS applications are included in the list including Office Viewer, iMessage, Mail, Broadcom BCM4325 and BCM4329 Wi-Fi chips, Calendar, FreeType, libxslt, and more.
When you factor in all of the above, Android likely comes out on top for the number of vulnerabilities when comparing the operating systems. Once again, vulnerability statistics seem simple on the surface. When you consider the above, and further consider that there are likely more points that influence vulnerability counts, we see that it is anything other than simple.
A week ago, I read an interesting blog post by Jeremiah Grossman of WhiteHat Security titled: “201x: The Year of the Security Industry Breach”, which discussed that security software may be the next big target for attackers to focus on
Some great points are presented and I especially appreciate how the fact is hammered in that attackers shift focus whenever required, but as Jericho pointed out when we discussed it, “The security industry does not.” I find, however, that the blog post lacking a couple of key points, which caused me to write this follow-up (not a rebuttal – Jericho handles those).
I agree that we have for decades been offered the same solution to all our security problems: Buy more/newer/subscription security software to deal with the threats. It is also certainly installed in abundance.
Security software does present problems however, and these concerns are not new. Researchers have voiced concerns for years over security software like firewalls and especially anti-virus (AV), pointing out that businesses are adding more (potentially flawed) code to protect themselves. It’s a common rule of thumb that the more lines of code, the more vulnerabilities. Reducing attack surface by adding an even greater attack surface is a paradox.
While security software in general presents an interesting target, it does, however, not necessarily mean that we will see it receiving the same heavy abuse as Java, Internet Explorer, and Flash, which are currently getting hammered by the bad guys.
When the bad guys were exploiting vulnerabilities in Adobe Reader, they were not targeting PDF viewers overall; they did not really care about e.g. Foxit Reader, PDF-XChange, nor Sumatra PDF. When Microsoft Office was hot, other office suites like OpenOffice and Corel Office did not feel the burn. And when 0-days are being dropped in Internet Explorer, browsers like Firefox, Opera, and Chrome can relax (relatively speaking) on the side.
These other products are not necessarily going free or only receiving limited attention because they are safer – they may just not have a user base that results in a profitable ROI.
If history is permitted to serve as a pointer, the bad guys do not target a whole class of products, but generally specific products with an extremely high user base allowing widespread attacks. It would, therefore, be more likely to see attacks focusing on products from a specific security vendor. That would, however, require a strong position with the vendor’s products being installed on a very large scale and while security software is prevalent, no single vendor dominates the market – especially not with a single product.
Another option would be if the security products start using the same codebase e.g. for file parsing or other common routines. Should security software e.g. use the same third-party parsing functionality, these parsers would certainly become of greater interest.
It should also be noted that while security software has not been targeted by the bad guys in the same manner as the previously mentioned products (and currently for good reason), it has still received a fair amount of attention over the years from researchers (myself included).
Looking at the OSVDB vulnerability database, it has at the time of writing 1,919 entries covering vulnerabilities in security software – in fact almost 2% of all our entries (yes, we have a specific classification flag, making it very easy to get a historical overview). These span from 2013 all the way back to 1995, when security software started gaining a foothold. I specifically remember when Alex Wheeler in 2005 was doing some serious work on the archive parsing functionality of many security software vendors’ AV solutions.
For the past 5-6 years, Symantec has also had their worries because they decided to implement very flawed parsers from HP/Autonomy/Verity Keyview in some of their security solutions, including their Mail Security and Data Loss Prevention products. Or Microsoft who on a related note made the questionable decision of implementing Oracle Outside In parsers in Microsoft Exchange Server 2007 and 2010. A vendor with a history of vulnerabilities, that they have worked very hard to overcome, deciding to implement a more notoriously insecure vendor’s software is certainly interesting.
Does this mean we won’t see any focus on security software in the future? Certainly not. Security software exposes a broad and tantalizing attack surface and is not necessarily built to impress (as Tavis Ormandy recently demonstrated for Sophos’ security solutions). It is an interesting target class of products that will continue to receive attention – and perhaps even an increasing focus once Oracle eventually sorts out Java, though that may take a while.
When the going gets tough, attackers just change to a different target with a better ROI, and it cannot be ruled out that the target ends up being the code actually intended to protect you.
So remember that 201x is already here and that while the bad guys may not go all-in against security solutions, vulnerabilities in these have been found consistently for many years. Consider what security software you really need – and certainly which features you need enabled.
This blog entry is probably worth many pages of ranting, examining and dissecting the anatomy of a 0-day panic and the resulting fallout. Since this tends to happen more often than some of us care to stomach, I’ll touch on the major points and be liberal in pointing fingers. If you receive the “wag of my finger“, stop being part of the problem and wise up.
I blinked and missed someone disclosing that there was a dreaded 0-day vulnerability in Adobe Flash Player and that it was a big threat. Apparently Symantec noticed that evil Chinese sites were exploiting Flash and the current 220.127.116.11 could be successfully exploited. When pressed for details, Symantec backtracked and said that they were wrong and it appeared to be the same exploit as previously disclosed by Mark Dowd (CVE-2007-0071). Bad Symantec, poor research.
To make matters worse, Symantec then further claimed that even though it was an old issue, the “in-the-wild exploit was effective against stand-alone versions of Flash Player 18.104.22.168” and that not all versions had been patched correctly. Way to save face Ben Greenbaum of Symantec!! Oh wait, today he changed his mind and said that Symantec’s claims were based on erroneous conclusions and that the behavior of Flash on Linux they were observing was indeed intended by Adobe and not proof it was vulnerable. To make matters worse, Symantec researchers downloaded the “latest” Flash and found it “vulnerable”, which lead to their sky-is-falling panic. Shortly after, they realized that they didn’t download all of the security patches and had been exploiting a known vulnerable version of Flash. Oops?
Two rounds of hype-driven 0-day threat warnings, and no real new threat. Whew, hopefully Symantec raised their THREATCON to blood red or whatever is appropriate for such 0-day threats. You do monitor that don’t you?
This fiasco lead many news outlets and vendors to issue warnings about the new 0-day threat. Secunia, SecurityFocus/BID, SecurityTracker, CERT, and FrSIRT all released new warnings and created entries in their respective databases as a result. In the VDB world, this is a royal pain-in-the-ass to deal with. Secunia ‘revoked’ their entry, BID ‘retired’ their entry, SecurityTracker flaged theirs ‘duplicate entry’, FrSIRT ‘revoked’ their entry and CERT still has it listed.
Fortunately for OSVDB, we were a few hours behind the rest and noticed the discrepancies and waited for more information. Unfortunately, the rest of the world, including ALL of the VDBs and news outlets listed above (and others) failed miserably in using common sense and a government funded resource to better prevent this kind of problem. As of this posting, Secunia, BID, SecurityTracker, FrSIRT, CERT, Dancho, ComputerWorld and eWeek still don’t link to the CVE ID for the vulnerability. Only Adobe’s updated blog entry actually references CVE-2007-0071 (but doesn’t link to it). Secunia links to a previous ID that has seven CVEs associated with it. The original CVE was assigned 2007-01-04 and published around 2008-04-08, a month and a half prior to this mess.
VDBs, shame on you for adding to the confusion. Symantec, shame on you for crying 0-day when your own engineers screwed up badly. Shame on everyone for not clearing it up fully by linking to the correct CVE entry or their own previous entries.
Before any of you receiving a “wave of the finger” bitch, consider the real world impact of your actions. In this case, only 12 MILLION people ended up seeing a vague warning when they loaded their favorite game. Blizzard included the correct fix information which was the same as a month or more before, but the sudden ‘security alert’ (that is extremely rare) only prompted their customers to wonder, possibly panic and definitely kill some demons as a result.
I’m so far behind in my daily routine and missed Thomas Ptacek’s post on Vuln Research In Numbers. Fortunately, Dave Aitel referenced the blog entry which prompted me to check it out. I so desperately want Ptacek to run his numbers against a complete OSVDB data set, but alas, I know that we do not have a complete data set for 2004. Symantec’s BID database has some problems with consistancy, citing sources (aka provenance) and missing vulnerabilities (which plagues most VDBs to one degree or another). In my mind, OSVDB tends to be more complete than most VDBs and maintains a creditee field, but due to a lack of volunteers we just don’t have enough entries public for him to do the same generation of statistics. Some day maybe! Until then, this is a very neat post with a different slant.
Symantec posted a message to Bugtraq earlier this month announcing the availability of a new advisory. The advisory presumably covers a vulnerability or issue in Symantec On-Demand Protection. If you are reading this blog entry a year from now, that is all you may find on it. Yes, even in this day and age, not everything is archived in Google cache or archive.org! In December of 2000, Elias Levy (moderator of Bugtraq at the time) said that such posts were not acceptable because security company web sites had a habit of disappearing, leaving no trace of the information behind. Years later, Symantec bought SecurityFocus (who hosts/moderates the Bugtraq mail list) and we see this rule being ignored, and of course the approved post comes from their owner. Some may argue that Symantec is huge and won’t disappear like those other companies. Many said the same about @stake but shortly after they were purchased, their new owner (Symantec) opted to yank all of the old advisories off the web site making Elias Levy’s concerns reality. As Chris Wysopal said in reply, Symantec needs to post their advisories to the list just like everyone else. While Symantec may stick around, their web site may change or corporate policy may be altered, and that information may not be readily available in the future.
After flap, Symantec adjusts browser bug count
Depending on how you count flaws, either IE or Firefox could be considered less secure
News Story by Robert McMillan
MARCH 07, 2006 (IDG NEWS SERVICE) – A report issued today by Symantec Corp. seeks to satisfy users of both Mozilla Corp.’s Firefox browser and Microsoft Corp.’s Internet Explorer.
In its latest Internet Security Threat Report, covering the last six months of 2005, the company now features two different ways of counting browser bugs: one that finds that Internet Explorer has the most vulnerabilities, and a second that reveals Firefox as the bug leader.
Thank you Symantec, for generating completely useless vulnerability statistics. When you can manipulate them to support either side of an argument (and do so intentionally), what’s the point? Just define your criteria for counting a vulnerability, define your time frame, and let the results speak for themselves.
Through its Science and Technology Directorate, the department has given $1.24 million in funding to Stanford University, Coverity and Symantec to hunt for security bugs in open-source software and to improve Coverity’s commercial tool for source code analysis, representatives for the three grant recipients told CNET News.com.
The Homeland Security Department grant will be paid over a three-year period, with $841,276 going to Stanford, $297,000 to Coverity and $100,000 to Symantec, according to San Francisco-based technology provider Coverity, which plans to announce the award publicly on Wednesday.
The project, while generally welcomed, has come in for some criticism from the open-source community. The bug database should help make open-source software more secure, but in a roundabout way, said Ben Laurie, a director of the Apache Foundation who is also involved with OpenSSL. A more direct way would be to provide the code analysis tools to the open-source developers themselves, he said.
So DHS uses $1.24 million dollars to fund a university and two commercial companies. The money will be used to develop source code auditing tools that will remain private. Coverity and Symantec will use the software on open-source software (which is good), but is arguably a huge PR move to help grease the wheels of the money flow. Coverity and Symantec will also be able to use these tools for their customers, which will pay them money for this service.
Why exactly do my tax dollars pay for the commercial development of tools that are not released to the public? As Ben Laurie states, why can’t he get a copy of these tax payer funded tools to run on the code his team develops? Why must they submit their code to a commercial third party for review to get any value from this software?
Given the date of this announcement, coupled with the announcement of Stanford’s PHP-CHECKER makes me wonder when the funds started rolling. There are obviously questions to be answered regarding Stanford’s project (that I already asked). This also makes me wonder what legal and ethical questions should be asked about tax dollars being spent by the DHS, for a university to fund the development of a security tool that could potentially do great good if released for all to use.
It’s too bad there is more than a year long wait for FOIA requests made to the DHS.
OSS means slower patches
SEPTEMBER 19, 2005
This was posted to Full-Disclosure where I first replied, and ISN picked up. Articles like this do nothing positive for our industry. Jenkins should not waste his time writing fluff pieces like this, and he should do some digging or at least question other sources. Of course, this is not the first time Symantec’s vuln stats have been questioned either. Since that post, no one at Symantec has given any insight as to how they derive their statistics and what lead to their conclusions.
I haven’t had time to read the full report mirrored here, but I have a feeling it will bring more questions than answers like the previous one did.
Full text of my reply:
The obvious criticism:
“The Mozilla family of browsers had the highest number of vulnerabilities during the first six months of 2005, with 25,” the Symantec report says. “Eighteen of these, or 72 per cent, were rated as high severity. Microsoft Internet Explorer had 13 vendor confirmed vulnerabilities, of which eight, or 62 per cent, were considered high severity.”
Microsoft IE had at least 19 vulnerabilities from 2005-01-01 to 2005-06-30. Why does Symantec make the distinction of “X vulnerabilities in Mozilla” vs “MSIE had X *vendor confirmed vulnerabilities*”? This all to conveniently allows the silently patched vulnerabilities to slip through the cracks of our statistics. Does Mozilla’s honesty in acknowledging vulnerabilities come back to bite them in the ass?
Mozilla browsers had more than 25, but are 72 per cent really “high severity”? Download information spoofing x2, File extension spoofing, URL restriction bypass, DoS x2, redirect spoofing, XSS, link status bar spoofing, Dialog overlapping, URL Wrap Obfuscation.. are all of these really “high severity”? Is that theoretical, practical, or hype?
Now, the media/Symantec driven propaganda (for lack of better word?):
THE growing popularity of open-source browsers and software may be responsible for the increasing gap between the exposure of a vulnerability and the provision of patch to fix it, security software vendor Symantec has said.
Mr Sykes said the increasing popularity of open source software, such as the Mozilla Foundation’s Firefox browser, could be part of the reason for the increase in the gap between vulnerability and patch, with the open source development model itself part of the problem. “It is relying on the goodwill and best efforts of many people, and that doesn’t have the same commercial imperative,” he said. “I’m sure that is part of what is causing the blow-out in the patch window.”
The growth in Firefox vulnerability reports coincides with its increasing popularity with users. “It is very clear that Firefox is gaining acceptance and I would therefore expect to see it targeted,” Mr Sykes said. “People don’t attack browsers and systems per se, they attack the people that use them,” he said. “As soon as large banks started using Linux, Linux vulnerabilities started to get exploited.”
The premise of this article is open source software is to blame for longer vendor response times. In laymen’s terms, blame vendors like Mozilla for having vulnerabilities patched slower? Err, compared to what? This shallow article doesn’t even qualify that statement! Slower than previous vulnerabilities? Slower than non open source? Given the article directly compares Mozilla browsers to Microsoft IE, it is trivial to assume the claim is made in relation to closed source vendors such as Microsoft. So then what .. 30 days “blown out” to 54 days is some huge time gap compared
to Microsoft IE patches? What clueless *moron* really believes this crap they are shoveling? Is it Symantec or Chris Jenkins or Australian IT?
Given that Symantec won’t even quote previous statistics: “Symantec had not published previously statistics on the average time required to produce patches, but Mr Sykes said data showed the lag had previously been about 30 days.” Given that Jenkins/AusIT/Symantec won’t give us any statistics (even questionable ones) regarding MSIE patches, we’re supposed to take this at face value? It is *well documented* that Microsoft takes well over 30 days to patch vulnerabilities. It is also becoming crystal clear that Microsoft is hiding behind their “30 day patch cycle” to imply
that is the longest they go before patching a vulnerability, when it simply is not the case. Taking a look at a *single vendor*  and their experience with reporting vulnerabilities to Microsoft, we see that they give MS a 60 day window to patch vulnerabilities, and are consistently overdue. As of this mail, the worse is *ONLY* 114 days past due (we’ve seen it closer to 250 days before). So again, where are these implications coming from? Where does this statement/conclusion/observation that “OSS causes slower patches” come from exactly?