Like any circle in any industry, having good professional relationships can be valuable to involved parties. In the world of security, more specifically Vulnerability Databases (VDBs), the relationships we maintain benefit the community behind the scenes. Like ogres and onions, there are layers.
Someone from CVE and someone from OSVDB run an informal list called ‘Vulnerability Information Managers’ (VIM) for discussion of vulnerabilities as relates to post-disclosure issues. New information comes up, additional research, vendor confirmations, vendor disputes and more. It’s a great resource for us to discuss the details that help each VDB fine-tune their information. (No new vulnerabilities are posted there, don’t bother)
In addition, some of the VDBs have stronger relationships that allow for great dialogue and information sharing. A few examples of these, from OSVDB’s perspective:
– A couple of the CVE guys are great for very informal chat about vulnerabilities. Despite being the dreaded “government contractors”, they are respectable, very knowledgeable and have a great sense of humor. I just sent one a mail with the subject “PROVENANCE BITCHEZ?!” challenging him on the details of a given CVE. They are so nice, I broke my rule of not taking candy from strangers and happily accepted the bag of leftover candy from their BlackHat booth. Joking aside, the ability to coordinate and share information is incredible and a testament to their integrity and desire to help the industry.
– OSVDB uses Secunia for one of our feeds to gather information. The two guys we regularly have contact with (CE & TK) lead a bright team that does an incredible amount of work behind the scenes. In case it slipped your attention, Secunia actually validates vulnerabilities before posting them. That means they take the time to install, configure and test a wide range of software based on the word of 3l1t3hax0ry0 that slapped some script tag in software you never heard of, as well as testing enterprise-level software that costs more than OSVDB makes in five years. Behind the scenes, Secunia shares information as they can with others, and there is a good chance you will never see it. If you aren’t subscribed to their service as a business, you should be. For those who asked OSVDB for years to have a ‘vulnerability alerting’ service; you can blame Secunia for us not doing it. They do it a lot better than we could ever hope to.
– The head of R&D at Tenable contributes a lot of time and information to VIM based on his research of disclosed vulnerabilities. Installing the software, configuring, testing and sometimes noticing additional vulnerabilities. He is a frequent contributor to VIM and has worked with OSVDB on sharing information to enhance the Nessus plugins as well as the OSVDB database.
– str0ke, that mysterious guy that somehow manages to run milw0rm in his spare time. What may appear to some as a website with user-posted content, is actually a horrible burden to maintain. Since the site’s inception, str0ke has not just posted the exploits sent in, but he has taken time to sanity check every single one as best he can. What you don’t see on that site are dozens (hundreds?) of exploits a month that were sent in but ended up being incorrect (or as OSVDB would label, “myth/fake”). When str0ke was overwhelmed and decided to give up the project, user demand (read: whining & complaints) lead him to change his mind and keep it going. Make sure you thank him every so often for his work and know this: milw0rm cannot be replaced as easily as you think. Not to the quality that we have seen from str0ke.
Since we have no corporate overlords, I’ll go ahead and talk about the flip side briefly:
– ISS (now IBM) runs a good database. Very thorough, keen to detail on including original source and vendor information. In 2004, the head of that group (AF) left, and until that time, we had a great dialogue and open communication. Since then, even before the IBM frenzy, we’ve mostly gotten the cold shoulder when mailing. Even when pointing out problems or negative changes on their side. LJ, bring back the old days!
– NVD. Why do you waste taxpayer money with that ‘database’? We pay $22 for Booz Allen Hamilton to “analyze” each CVE entry (thanks FOIA request!), yet they find a fraction of the typos and mistakes I do? By fraction, I mean exactly none from what I hear through the grape vine (DHS cronies are cool). If you can’t notice and report simple typos in a CVE, and you botch CVSS2 scores left and right (yes, I’ve mailed in corrections that were acted on), what exactly are you doing with our money? Are you the virtual Blackwater of VDBs?
– SecurityFocus / BID. Sorry, not going to bother with verbal fluffing. My countless mails pointing out errors and issues with your database are seemingly dumped to a black hole. Your promises of certain mail archives ‘not changing’ were pure fantasy. To this date you make erroneous assumptions about affected products, and still don’t grasp “case sensitive”. I know some of your team, you have great people there. Just lift the corporate policy that turns them into virtual shut-ins, please?
Sorry to end it on a downer. I still dream of a niche of the security industry (VDBs) where we can all play well with each other.
This is a question OSVDB moderators, CVE staff and countless other VDB maintainers have asked. Today, Gunter Ollmann with IBM X-Force released his research trying to answer this question. Before you read on, I think this research is excellent. The relatively few criticisms I bring up are not the fault of Ollmann’s research and methodology, but the fault of his VDB of choice (and *every* other VDB) not having a complete data set.
Skimming his list, my first thought was that he was missing someone. Doing a quick search of OSVDB, I see that Lostmon Lords (aka ‘lostmon’) has close to 350 vulnerabilities published. How could the top ten list miss someone like this when his #10 only had 147? Read down to Ollmann’s caveat and there is a valid point, but sketchy wording. The data he is using relies on this information being public. As the caveat says though, “because they were disclosed on non-public lists” implies that the only source he or X-Force are using are mail lists such as Bugtraq and Full-disclosure. Back in the day, that was a pretty reliable source for a very high percentage of vulnerability information. In recent years though, a VDB must look at other sources of information to get a better picture. Web sites such as milw0rm get a steady stream of vulnerability information that is frequently not cross-posted to mail lists. In addition, many researchers (including lostmon) mail their discoveries directly to the VDBs and bypass the public mail lists. If researchers mail a few VDBs and not the rest, it creates a situation where the VDBs must start watching each other. This in turn leads to “VDB inbreeding” that Jake and I mentioned at CanSecWest 2005, which is a necessary evil if you want more data on vulnerabilities.
In May of 2008, OSVDB did the same research Ollmann did and we came up with different results. This was based on data we had available, which is still admittedly very incomplete (always need data manglers.) So who is right? Neither of us. Well, perhaps he is, perhaps we are, but unfortunately we’re both working with incomplete databases. As a matter of my opinion, I believe OSVDB has better coverage of vulnerabilities, while X-Force clearly has better consistency in their data and a fraction of the gaps we do.
Last, this data is interesting as is, but would be really fascinating if it was mixed with ‘researcher confidence’ (a big thing of Steve Christey/CVE and myself), in which we track a researcher’s track record for accuracy in disclosure. Someone that disclosed 500 vulnerabilities last year with a 10% error rate should not be above someone who found 475 with a 0% error rate. In addition, as Ollmann’s caveat says, these are pure numbers and do not factor in hundreds of XSS versus remote code execution in operating system default install services. Having a weight system that can be applied to a vulnerability (e.g., XSS = 3, SQLi = 7, remote code exec = 9) that is then factored into researcher could move beyond “who discovered the most” and perhaps start to answer “who found the most respectable vulnerabilities”.
I saw this article the other day, IBM Scolds TippingPoint Over Hacking Contest and figured now what? But I decided it would be an interesting read.
A couple quick blurbs from the article:
IBM’s ISS division has torn into rival TippingPoint for sponsoring the hacking contest that led to the disclosure of a QuickTime vulnerability in Apple’s Safari browser. “IBM Internet Security Systems agrees with Gartner’s assessment that “public vulnerability research and ‘hacking contests’ are risky endeavors, and can run contrary to responsible disclosure practices.” It is for this reason that IBM ISS strongly adheres to its well-established responsible disclosure guidelines.”
Once I read the article it was then that I realized…. that it really wasn’t IBM, but ISS (who IBM purchased recently) that was scolding TippingPoint for sponsoring this contest. Immediately I thought about all the drama that went on when ISS disclosed their Apache Chunked Encoding Overflow back in 2002.
http://lwn.net/Articles/2756/ It all looks like a fairly normal response to security problems in the free software community, until you look a little more closely. It turns out that the Apache group was already aware of the problem and was working on a fix. The Computer Emergency Response Team (CERT) also was already involved. It also turns out that the ISS patch does not completely fix the problem. ISS, in its hurry to publicise the vulnerability, had not checked with either CERT or the Apache Software Foundation.
Does anyone remember all of this?
ISS took quite a bit of criticism for this disclosure and responded publicly to clean up any confusion and misunderstanding.
The very last portion of this posting is what I find real interesting:
ISS has made these decisions based on our mission to provide the best security to our customers and being a trusted security advisor.
For me personally.. It is kind of funny that disclosure almost always seems to come back to the argument of… we did it for the greater good… we did it for the benefit of others… we did it for the right reasons…
But you on the other hand…