This is a question OSVDB moderators, CVE staff and countless other VDB maintainers have asked. Today, Gunter Ollmann with IBM X-Force released his research trying to answer this question. Before you read on, I think this research is excellent. The relatively few criticisms I bring up are not the fault of Ollmann’s research and methodology, but the fault of his VDB of choice (and *every* other VDB) not having a complete data set.
Skimming his list, my first thought was that he was missing someone. Doing a quick search of OSVDB, I see that Lostmon Lords (aka ‘lostmon’) has close to 350 vulnerabilities published. How could the top ten list miss someone like this when his #10 only had 147? Read down to Ollmann’s caveat and there is a valid point, but sketchy wording. The data he is using relies on this information being public. As the caveat says though, “because they were disclosed on non-public lists” implies that the only source he or X-Force are using are mail lists such as Bugtraq and Full-disclosure. Back in the day, that was a pretty reliable source for a very high percentage of vulnerability information. In recent years though, a VDB must look at other sources of information to get a better picture. Web sites such as milw0rm get a steady stream of vulnerability information that is frequently not cross-posted to mail lists. In addition, many researchers (including lostmon) mail their discoveries directly to the VDBs and bypass the public mail lists. If researchers mail a few VDBs and not the rest, it creates a situation where the VDBs must start watching each other. This in turn leads to “VDB inbreeding” that Jake and I mentioned at CanSecWest 2005, which is a necessary evil if you want more data on vulnerabilities.
In May of 2008, OSVDB did the same research Ollmann did and we came up with different results. This was based on data we had available, which is still admittedly very incomplete (always need data manglers.) So who is right? Neither of us. Well, perhaps he is, perhaps we are, but unfortunately we’re both working with incomplete databases. As a matter of my opinion, I believe OSVDB has better coverage of vulnerabilities, while X-Force clearly has better consistency in their data and a fraction of the gaps we do.
Last, this data is interesting as is, but would be really fascinating if it was mixed with ‘researcher confidence’ (a big thing of Steve Christey/CVE and myself), in which we track a researcher’s track record for accuracy in disclosure. Someone that disclosed 500 vulnerabilities last year with a 10% error rate should not be above someone who found 475 with a 0% error rate. In addition, as Ollmann’s caveat says, these are pure numbers and do not factor in hundreds of XSS versus remote code execution in operating system default install services. Having a weight system that can be applied to a vulnerability (e.g., XSS = 3, SQLi = 7, remote code exec = 9) that is then factored into researcher could move beyond “who discovered the most” and perhaps start to answer “who found the most respectable vulnerabilities”.
Today just happened to be the right day where I saw the Jekyll and “Hide” of Sun though. A few days ago, |)ruid posted about a Solaris ypupdated vulnerability in which he says it corresponds to CVE-1999-0208 / OSVDB 11517. Given the original vulnerability was published in 1994, I had doubts it was truly the same vulnerability. I replied asking for confirmation, |)ruid replied and CC’d the Sun Security Coordination Team. Within 24 hours, Sun replied with a detailed analysis explaining how 11517 was different from the newly created OSVDB 43433, but very much related. This mail is a VDB maintainer’s wet dream; if only every vendor would provide this kind of detail when there is confusion over published vulnerability information. This is clearly the Dr. Jekyll locked up in a Sun complex somewhere who deserves kudos for the reply.
The Sun Microsystems “SunSolve” database is a quagmire of technical muck that is only rivaled by the IBM APAR database I believe. Tonight I find myself plowing through a grotesque changelog of Sun Java System Directory Server (SJSDS?). Sun apparently hasn’t fully mastered the idea of hyperlinking to make those annoying numbers on the left lead to somewhere with more information. So I log into the SunSolve database using my super secret ID associated with a sizable company that owns lots of Sun products. I type in a few numbers of interest off that list and away I … don’t go. Mr. Hide stops me quick, telling me that to read the bug IDs I have to be a better customer apparently.
You have selected content which is only available to registered SunSolve users with a valid Sun Service Plan. Please Login to access the restricted content of SunSolve and the Sun System Handbook if you are logged in to SunSolve and have received this message, please verify that you are associated with a valid support contract in the iSupport tool. If you have any questions about your support contract, please follow up with the Sun contract administrator contact at your company. If, however, none of the previous conditions apply, you may be trying to access a document that is no longer available. In this case please feel free to click on the SunSolve Feedback link at the bottom of the page and be sure to include the exact steps you took before you received this error message.
Wow, way to foil me via security through obscurity Sun Microsystems. Please take Mr. Hide and shove my beer bottle up his ass, sideways. Booze is the only way to adequately cope with the kind of headache born from vendors who can’t manage, organize and share information.
Really IBM, the amount of information common to all three pages is overwhelming. Do you really need a new APAR number issued for component name or level? Can’t you just list them all in one APAR and save us time? More importantly, do we need three APAR entries that say “a security issue has been fixed” and make us dig up the information?
New IBM research shows that five vendors are responsible for 12.6 percent of all disclosed vulnerabilities. Not surprising: In the first half of 2007, Microsoft was the top vendor when it came to publicly disclosed vulnerabilities. Likely surprising to some: Apple got second place. IBM Internet Security Systems’ X-Force R&D team released its 2007 report on cyber attacks on Sept. 17, revealing that the top five vulnerable vendors accounted for 12.6 of all disclosed vulnerabilities in the first half of the yearor 411 of 3,272 vulnerabilities disclosed. Here’s the order in which the top 10 vendors stacked up, by percentage of vulnerabilities publicly disclosed in the first half of the year: Microsoft, 4.2 percent Apple, 3 percent Oracle, 2 percent Cisco Systems, 1.9 percent Sun Microsystems, 1.5 percent IBM, 1.3 percent Mozilla, 1.3 percent XOOPS, 1.2 percent BEA, 1.1 percent Linux kernel, 0.9 percent
This article was posted to ISN the other day and struck a nerve. How many times are we going to see vulnerability statistics presented without qualification? Rather than really get into the details, I replied with a single simple example on why such statistics are misleading at best and incorrect at worst. The bulk of my reply follows. My hopes for Lisa or IBM/ISS clarifying this is already dwindling.
One other factor, that Lisa Vaas apparently didn’t ask about, is how ISS X-Force catalogs vulnerabilities, and if their method and standards could impact these numbers at all. Take for example, two X-Force vulnerability database entries: Oracle Critical Patch Update – July 2007 http://xforce.iss.net/xforce/xfdb/35490 18 CVE, 30+ Oracle Oracle Critical Patch Update – January 2007 http://xforce.iss.net/xforce/xfdb/31541 30 CVE, 50+ Oracle So when comparing numbers, you have 2 X-Force entries that equate to 48 CVE entries that equate to *more than 80* unique and distinct vulnerabilities according to Oracle. I’m not a math or stat guy, but I have a feeling that this could seriously skew the statistics above, especially when you consider that Microsoft and Apple both have a more distinct breakdown and separation in the X-Force database. Anyone from IBM/ISS care to clarify? Lisa, did you have more extensive notes on this aspect that didn’t make it in the article perhaps?
I saw this article the other day, IBM Scolds TippingPoint Over Hacking Contest and figured now what? But I decided it would be an interesting read.
A couple quick blurbs from the article:
IBM’s ISS division has torn into rival TippingPoint for sponsoring the hacking contest that led to the disclosure of a QuickTime vulnerability in Apple’s Safari browser. “IBM Internet Security Systems agrees with Gartner’s assessment that “public vulnerability research and ‘hacking contests’ are risky endeavors, and can run contrary to responsible disclosure practices.” It is for this reason that IBM ISS strongly adheres to its well-established responsible disclosure guidelines.”
Once I read the article it was then that I realized…. that it really wasn’t IBM, but ISS (who IBM purchased recently) that was scolding TippingPoint for sponsoring this contest. Immediately I thought about all the drama that went on when ISS disclosed their Apache Chunked Encoding Overflow back in 2002.
http://lwn.net/Articles/2756/ It all looks like a fairly normal response to security problems in the free software community, until you look a little more closely. It turns out that the Apache group was already aware of the problem and was working on a fix. The Computer Emergency Response Team (CERT) also was already involved. It also turns out that the ISS patch does not completely fix the problem. ISS, in its hurry to publicise the vulnerability, had not checked with either CERT or the Apache Software Foundation.
Does anyone remember all of this?
ISS took quite a bit of criticism for this disclosure and responded publicly to clean up any confusion and misunderstanding.
The very last portion of this posting is what I find real interesting:
ISS has made these decisions based on our mission to provide the best security to our customers and being a trusted security advisor.
For me personally.. It is kind of funny that disclosure almost always seems to come back to the argument of… we did it for the greater good… we did it for the benefit of others… we did it for the right reasons…
But you on the other hand…