Adobe, Qualys, CVE and Math
Posted by jericho
Elinor Mills wrote an article titled Firefox, Adobe top buggiest-software list. In it, she quotes Qualys as providing vulnerability statistics for Mozilla, Adobe and others. Qualys states:
The number of vulnerabilities in Adobe programs rose from 14 last year to 45 this year, while those in Microsoft software dropped from 44 to 41, according to Qualys. Internet Explorer, Windows Media Player and Microsoft Office together had 30 vulnerabilities.
This caught my attention immediately, as I know I have mangled more than 45 Adobe entries this year.
First, the "number of vulnerabilities" game will always have wiggle room, which has been discussed before. A big factor for statistic discrepancy when using public databases is the level of abstraction. CVE tends to bunch up vulnerabilities in a single CVE, where OSVDB tends to break them out. Over the past year, X-Force and BID have started abstracting more and more as well.
Either way, Qualys cited their source, NVD, which is entirely based on CVE. How they got 45 vulns in "Adobe programs" baffles me. My count says 97 Adobe vulns, 95 of them have CVEs assigned to them (covered by a total of 93 CVEs). OSVDB abstracted the entries like CVE did for the most part, but split out CVE-2009-1872 as distinct XSS vulnerabilities. OSVDB also has two entries that do not have CVE, 55820 and 56281.
Where did Qualys get 45 if they are using the same CVE data set OSVDB does? This discrepancy has nothing to do with abstraction, so something else appears to be going on. Doing a few more searches, I believe I figured it out. Searching OSVDB for "Adobe Reader" in 2009 yields 44 entries, one off from their cited 45. That could be easily explained as OSVDB also has 9 "Adobe Multiple Products" entries that could cover Reader as well. This may in turn be a breakdown where Qualys or Mills did not specify "Adobe Software" (cumulative, all software they release) versus "Adobe Reader" or some other specific software they release.
Qualys tallied 102 vulnerabilities that were found in Firefox this year, up from 90 last year.
What is certainly a discrepancy due to abstraction, OSVDB has 74 vulnerabilities specific to Mozilla Firefox (two without CVE), 11 for "Mozilla Multiple Browsers" (Firefox, Seamonkey, etc) and 81 for "Mozilla Multiple Products" (Firefox, Thunderbird, etc). While my numbers are somewhat anecdotal, because I cannot remember every single entry, I can say that most of the 'multiple' vulnerabilities include Firefox. That means OSVDB tracked as many as, but possibly less than, 166 vulnerabilities in Firefox.
Microsoft software dropped from 44 to 41, according to Qualys. Internet Explorer, Windows Media Player and Microsoft Office together had 30 vulnerabilities.
According to my searches on OSVDB, we get the following numbers:
- 234 vulnerabilities in Microsoft, only 4 without CVE
- 50 vulnerabilities in MSIE, all with CVE
- 4 vulnerabilities in Windows Media Player, 1 without CVE
- 52 vulnerabilities in Office, all with CVE. (based on "Office" being Excel, Powerpoint, Word and Outlook.
- 92 vulnerabilities in Windows, only 2 without CVE
When dealing with vulnerability numbers and statistics, like anything else, it's all about qualifying your numbers. Saying "Adobe Software" is different than "Adobe Acrobat" or "Adobe Reader" as the software installation base is drastically different. Given the different levels of abstraction in VDBs, it is also equally important to qualify what "a vulnerability" (singular) is. Where CVE/NVD will group several vulnerabilities in one identifier, other databases may abstract and assign unique identifiers to each distinct vulnerability.
Qualys, since you provided the stats to CNet, could you clarify?
While your measure is clearly more rigorous than the nonsense Qualys came up with, it’s still pretty meaningless. Anyone who has reported issues to big vendors like Microsoft, Oracle, Adobe, etc. will tell you they can ask you to wait up to two years before fixing it.
Which year do you credit these bugs under, the years they were fixed, or the years they were reported? What if this information is not available even when the bug is published?
I believe at least Microsoft try to manipulate stats this way (This is a common “analysis” clueless journalists can make and clueless PHBs can understand), and “even out” the rough years by screwing reporters.
This is a good point. I wouldn’t argue it is meaningless, but it throws yet-another-wrench into coming up with a standard way of measuring. One neat thing about people using CVE is that if they request an ID when the bug is discovered, we get a very rough idea of when the bug was discovered, even if the researcher and vendor do not mention discovery or vendor report date.
For OSVDB, we track by public disclosure date as that is one date that every vulnerability will have, while we may have discovery date and/or vendor report date for a fraction of them. We do track these dates, and we have two things in the works to give a better picture of “time to patch” dates as well. Think eEye and iDefense, but for thousands of vulnerabilities covering a lot more vendors. This of course will only be possible where we have a few specific dates in the vulnerability timeline, but I think it will be interesting and revealing.
It is great that you brought this up, as it is definitely another factor both security analysts and journalists need to be aware of.
Jericho,
As you suspected, our (Qualys) numbers are indeed for a more narrow set of products. They are not for all of Adobe and Microsoft software, but specifically for Adobe Reader and Microsoft Office,. We have contacted the journalist and hope it can be corrected. Our overall point that we are trying to make remains the same – patching such applications is being neglected by most IT admins and attackers have increasingly shifted their attention to exploiting vulnerabilities in them. On Friday Brad Arkin from Adobe stated that Adobe Reader as a cross operating system application has a bigger installed base than Microsoft Windows, which makes it a very attractive target to attack.
Why do you think that vulnerabilities found in Adobe Reader have gone up in 2009 – did attackers first notice that there was potential, wrote exploits and then security researchers followed up or was it the other way around ?
Elinor Mills / CNet has updated the article, and Qualys has posted additional information at http://laws.qualys.com/lawsblog/2009/12/on-adobe-qualys-cve-and-math.html. Thanks for the clarification and prompt response!
As for the chicken v. egg scenario between attackers and researchers, that is likely something we will never know. I’d personally hazard a guess it was 50/50 though. Look at the companies and individuals that have been finding vulnerabilities in Adobe the past several years, and it suggests researchers were on to the “cross platform” appeal early on. However, as we keep seeing from the recurring headlines “Adobe 0day discovered in the wild”, the bad guys definitely picked up on it at some point.