Steve Christey w/ CVE recently posted that trying to keep up with Linux Kernel issues was getting to be a burden. Issues that may or may not be security related, even Kernel devs don’t fully know. While this is a good example of the issues VDBs face, it’s really the tip of the iceberg. Until their recent adoption of CVE identifiers, trying to distinguish Oracle vulnerabilities from each other was what you did as a gentle relief from a few hours of being water-boarded. Lately, Mozilla advisories are getting worse as they clump a dozen issues with “evidence of memory corruption” into a single advisory, that gets lumped into a single CVE. Doesn’t matter that they can be exploited separately or that some may not be exploitable at all. Reading the bugzilla entries that cover the issues is headache-inducing as their own devs frequently don’t understand the extent of the issues. Oh, if they make the bugzilla entry public. If the Linux Kernel devs and Mozilla browser wonks cannot figure out the extent of the issue, how are VDBs supposed to?
Being “open source” isn’t some get-out-of-VDB free card. You’re supposed to be better than your closed-source rivals. You’re supposed to care about your customers and be open about security issues. An advisory full of “may” and “evidence of” is nothing more than a FUD-filled excuse to blindly upgrade without understanding the real threat or exposure to the end-user.
Steve’s post is a good view of how some VDBs feel about the issue: http://marc.info/?l=oss-security&m=124061708428439&w=2
Tonight, I followed-up on his thoughts and gave more of my own (original: http://marc.info/?l=oss-security&m=124065500729868&w=2):
A question, really?
I’d like to reiterate what Steve Christey said in the last 24 hours, about the Linux Kernel vulnerabilities becoming a serious drain on CVE. Historically, OSVDB has relied on Secunia and CVE to sort out the Linux Kernel vulnerability messes. Both VDBs have full time staff that can dedicate time to figuring out such nuances as those above.
Not to pick on Eugene specifically, but I feel he makes a great example of my point. Nuances that a “Senior Security Engineer at Red Hat” who specialies in “OS and Application Security, Project Management, Vulnerability Analysis, Code-level Auditing, Penetration Testing, Red Hat Products and Services, Financial Services Technical Account Management” cannot definitely distinguish between difference in Kernel vulnerabilities. If Eugene cannot say with certainty these deserve two CVE numbers, how can Steve or his staff?
VDBs deal with thousands of vulnerabilities a year, ranging from PHP applications to Oracle to Windows services to SCADA software to cellular telephones. We’re expected to have a basic understanding of ‘vulnerabilities’, but this isn’t 1995. Software and vulnerabilities have evolved over the years. They have moved from straight-forward overflows (before buffer vs stack vs heap vs underflow) and one type of XSS to a wide variety of issues that are far from trivial to exploit. For fifteen years, it has been a balancing act for VDBs when including Denial of Service (DOS) vulnerabilities because the details are often sparse and it is not clear if an unprivileged user can reasonably affect availability. Jump to today where the software developers cannot, or will not tell the masses what the real issue is.
This isn’t just a Linux Kernel issue at all. The recent round of advisories from Mozilla contain obscure wording that allude to “memory corruption” implying arbitrary code execution. If you follow the links to the bugzilla reports, the wording becomes a quagmire of terms that not even the developers can keep up on  . That’s if they even open the bugzilla entry reference in the advisory . Again, how are people not intimately familiar with the code base supposed to understand these reports and give a reasonable definition of the vulnerability? How do we translate that mess of coder jargon into a 1 – 10 score for severity?
It is important that VDBs continue to track these issues, and it is great that we have more insight and contact with the development teams of various projects. However, this insight and contact has paved the way for a new set of problems that over-tax an already burdened effort. MITRE receives almost 5 million dollars a year from the U.S. government to fund the C*E effort, including CVE [Based on FOIA information]. If they cannot keep up with these vulnerabilities, how do their “competitors”, especially free / open source ones , have a chance?
Projects like the Linux Kernel are familiar with CVE entries. Many Linux distributions are CVE Numbering Authorities, and can assign a CVE entry to a particular vulnerability. It’s time that you (collectively) properly document and explain vulnerabilities so that VDBs don’t have to do the source code analysis, patch reversals or play 20 questions with the development team. Provide a clear understanding of what the vulnerability is so that we may properly document it, and customers can then judge the severity of issue and act on it accordingly.
I believe this is a case where over-exposure to near-proprietary technical details of a product have become the antithesis of closed-source vague disclosures like those from Microsoft or Oracle [Which are just as difficult to deal with in a totally different way.].
This is a question OSVDB moderators, CVE staff and countless other VDB maintainers have asked. Today, Gunter Ollmann with IBM X-Force released his research trying to answer this question. Before you read on, I think this research is excellent. The relatively few criticisms I bring up are not the fault of Ollmann’s research and methodology, but the fault of his VDB of choice (and *every* other VDB) not having a complete data set.
Skimming his list, my first thought was that he was missing someone. Doing a quick search of OSVDB, I see that Lostmon Lords (aka ‘lostmon’) has close to 350 vulnerabilities published. How could the top ten list miss someone like this when his #10 only had 147? Read down to Ollmann’s caveat and there is a valid point, but sketchy wording. The data he is using relies on this information being public. As the caveat says though, “because they were disclosed on non-public lists” implies that the only source he or X-Force are using are mail lists such as Bugtraq and Full-disclosure. Back in the day, that was a pretty reliable source for a very high percentage of vulnerability information. In recent years though, a VDB must look at other sources of information to get a better picture. Web sites such as milw0rm get a steady stream of vulnerability information that is frequently not cross-posted to mail lists. In addition, many researchers (including lostmon) mail their discoveries directly to the VDBs and bypass the public mail lists. If researchers mail a few VDBs and not the rest, it creates a situation where the VDBs must start watching each other. This in turn leads to “VDB inbreeding” that Jake and I mentioned at CanSecWest 2005, which is a necessary evil if you want more data on vulnerabilities.
In May of 2008, OSVDB did the same research Ollmann did and we came up with different results. This was based on data we had available, which is still admittedly very incomplete (always need data manglers.) So who is right? Neither of us. Well, perhaps he is, perhaps we are, but unfortunately we’re both working with incomplete databases. As a matter of my opinion, I believe OSVDB has better coverage of vulnerabilities, while X-Force clearly has better consistency in their data and a fraction of the gaps we do.
Last, this data is interesting as is, but would be really fascinating if it was mixed with ‘researcher confidence’ (a big thing of Steve Christey/CVE and myself), in which we track a researcher’s track record for accuracy in disclosure. Someone that disclosed 500 vulnerabilities last year with a 10% error rate should not be above someone who found 475 with a 0% error rate. In addition, as Ollmann’s caveat says, these are pure numbers and do not factor in hundreds of XSS versus remote code execution in operating system default install services. Having a weight system that can be applied to a vulnerability (e.g., XSS = 3, SQLi = 7, remote code exec = 9) that is then factored into researcher could move beyond “who discovered the most” and perhaps start to answer “who found the most respectable vulnerabilities”.
I’m big on Vulnerability Database (VDB) evolution. I tend to harp on them for not adding features, not making the data more accessible and generally doing the exact same thing they did ten years ago. While the target of my ire is typically functionality or usability, today it is about a little more.
Last night I wanted to check for details on a CVE entry that was rather vague and had a single reference to BID. This is fairly common in the VDB world as one database will add an entry and not provide a link to the source of the data (Secunia and BID primarily). As luck would have it, BID was down. Almost twelve hours later and their VDB is still down. What annoys me is that while they aren’t delivering vulnerability information, they sure are delivering advertisements. Why can’t VDBs get the same dedication and resources that ad farms get?
Next, I wanted to find out if the other VDBs created an entry for the latest OpenBSD flap yet, so I went to X-force which is a pretty reliable database. Much to my dismay, it appears that the ‘advanced’ search is now gone. While it wasn’t extremely powerful, it let you do some basic sorting that was immensely helpful in finding what you need. I have mail out to them asking for confirmation that it is indeed gone versus a web geek error. I certainly hope it is the latter…
Update: Over 24 hours later, the BID database is finally available again. ISS has not replied to at least two mails from VDB managers asking about the missing advanced search feature.
Steven Christey of CVE posted asking a question about VDBs and the inclusion of coffee makers. Yes, you read that correctly, vulnerabilities are being found in coffee makers that are network accessible. Don’t be surprised, we all knew the day was coming when every household appliance would become IP aware.
Before you laugh and spew your own coffee all over the keyboard, consider that the vulnerabilities are legitimate in the sense that a remote attacker can manipulate how the device performs and possibly do physical damage to the unit. This is really no different than SCADA devices such as air conditioners that are IP aware.
Some replies (like mine) were a bit more serious suggesting this type of vulnerability is definitely worth inclusion in OSVDB. If we can’t draw the line between coffee makers, air conditioners and other SCADA devices today, we will be able to in a year or years from now? At some point, the blur between computing device and household appliance will be too hard to distinguish. Rather than waste too much time arguing that line, why not track these few vulnerabilities now that might be a bit primitive, but will surely show historic value if nothing else.
Other replies were a bit less serious but fun, suggesting that making weak (or no) coffee would lead to disgruntled code writers that produce poor code filled with more vulnerabilities. Either way, count on us to include vulnerabilities in your favorite IP aware devices, kitchen, computing or otherwise, to this database.
Dave pushed a new set of code changes today! Here is a very brief summary of some of the highlights:
- Browse now has: Browse by Top Creditee, Browse by Creditee Name [Remember, we need more entries at 100% to make this more accurate and complete. Mangle your own vulnerabilities and fill in the missing creditee!]
- Three new dates added to schema (Screenshot) [The new date fields won’t appear on the front end yet, as more changes are required, but we now have the capability to track a more thorough history of the vulnerability]
- Menu Changes and new pages in support of that.
- More diverse “Donation” options [Come on, donate 5 bucks and skip that fourth Latte!]
- General bug fixes/tweaks
- Vendor dictionary – change e-mail addresses to stop automatic harvesting
- New template for CSRF vulnerabilities
Behind the Scenes:
CVE just announced reaching 30,000 identifiers which is a pretty scary thing. CVE staff have a good eye for catching vulnerabilities from sources away from the mainstream (e.g. bugtraq) and they have the advantage of being a very widely accepted standard for tracking vulnerabilities. As companies and researchers request CVE numbers for disclosures, they get a lot of the information handed to them on a silver platter. Of course, sometimes that platter is full of mud and confusion as vendors don’t always provide clear details to help CVE accurately track and distinguish between multiple vulnerabilities. I’ve also pointed out many times in the past that CVE is a very unique VDB that provides identifiers for vulnerability tracking. They do not provide many fields associated with other VDBs (solution, creditee, etc). As such, they may have a single entry that covers multiple distinct vulnerabilities if they are the same class (XSS, SQLi, RFI), or if there is a lack of details but they know it affects the same product (Oracle). So when we see 30,000 identifiers, we have to realize that the real count of vulnerabilities is significantly higher.
CVE is run by The MITRE Corporation, sponsored / funded by the NCSD (US-CERT) of DHS under government contract. That means our tax dollars fund this database so it should be of particular interest to U.S. taxpayers in the security industry. I know from past discussions with CVE staff and other industry veterans that on any given day, they are more likely to have more work than available staff. That means the rate of vulnerabilities that get published is greater than the resources CVE can maintain to track them. In short, the 30,000 identifiers you see only represents a percentage of the vulnerabilities actually disclosed. We could probably debate what percentage that represents all day long, and I don’t think that is really the point here other than “we know it isn’t all of them”.
Every VDB suffers from the same thing. “Commercial” VDBs like X-Force, BID and Secunia have a full time staff that maintain their databases, like CVE does. Despite having all of these teams (some of them consisting of 10 or more people) maintain VDBs, we still see countless vulnerabilities that are ‘missed’ by all of them. This is not a slight against them in any way; it is a simple manner of resources available and the amount of information out there. Even with a large team sorting disclosed vulnerabilities, some teams spend time validating the findings before adding them to the database (Secunia), which is an incredible benefit for their customers. There is also a long standing parasitic nature to VDBs, with each of them watching the others as best they can, to help ensure they are tracking all the vulnerabilities they can. For example, OSVDB keeps a close eye on Secunia and CVE specifically, and as time permits we look to X-Force, BID, SecurityTracker and others. Each VDB tends to have some researchers that exclusively disclose vulnerabilities directly to the VDB of their choice. So each one I mention above will get word of vulnerabilities that the rest really have no way of knowing about short of watching each other like this. This VDB inbreeding (I will explain the choice of word some other time) is an accepted practice and I have touched on this in the past (CanSecWest 2005).
Due to the inbreeding and OSVDB’s ability to watch other resources, it occasionally frees up our moderators to go looking for more vulnerability information that wasn’t published in the mainstream. This usually involves grueling crawls through vendor knowledge-bases, mind-numbing changelogs, searching CVS type repositories and more. That leads to the point of this lengthy post. In doing this research, we begin to see how many more vulnerabilities are out there in the software we use, that escapes the VDBs most of the time. Only now, after four years and getting an incredible developer to make many aspects of the OSVDB wish-list a reality, do we finally begin to see all of this. As I have whined about for those four years, VDBs need to evolve and move beyond this purely “mainstream reactionary” model. Meaning, we have to stop watching the half dozen usual spots for new vulnerability information, creating our entries, rinsing and repeating. There is a lot more information out there just waiting to be read and added.
In the past few weeks, largely due to the ability to free up time due to the VDB inbreeding mentioned above, we’ve been able to dig into a few products more thoroughly. These examples are not meant to pick on any product / VDB or imply anything other than what is said above. In fact, this type of research is only possible because the other VDBs are doing a good job tracking the mainstream sources, and because some vendors publish full changelogs and don’t try to hide security related fixes. Kudos to all of them.
Example: Search your favorite VDB for ”inspircd”, a popular multi-platform IRC daemon. Compare the results of BID, Secunia, X-Force, SecurityTracker, and http://osvdb.org/ref/blog/inspircd-cve.png. Compare these results to OSVDB after digging into their changelogs. Do these same searches for “xfce” (10 OSVDB, 5 max elsewhere), “safesquid” (6 OSVDB, 1 max elsewhere), “beehive forum” (27 OSVDB, 8 max elsewhere) and “jetty” (25 OSVDB, 12 max elsewhere). Let me emphasize, I did not specifically hand pick these examples to put down any VDB, these are some of the products we’ve investigated in the last few weeks.
The real point here is that no matter what vulnerability disclosure statistic you read, regardless of which VDB it uses (including OSVDB), consider that the real number of vulnerabilities disclosed is likely much higher than any of us know or have documented. As always, if you see vulnerabilities in a vendor KB or changelog, and can’t find it in your favorite VDB, let them know. We all maintain e-mail addresses for submissions and we all strive to be as complete as possible.
New IBM research shows that five vendors are responsible for 12.6 percent of all disclosed vulnerabilities. Not surprising: In the first half of 2007, Microsoft was the top vendor when it came to publicly disclosed vulnerabilities. Likely surprising to some: Apple got second place. IBM Internet Security Systems’ X-Force R&D team released its 2007 report on cyber attacks on Sept. 17, revealing that the top five vulnerable vendors accounted for 12.6 of all disclosed vulnerabilities in the first half of the yearor 411 of 3,272 vulnerabilities disclosed. Here’s the order in which the top 10 vendors stacked up, by percentage of vulnerabilities publicly disclosed in the first half of the year: Microsoft, 4.2 percent Apple, 3 percent Oracle, 2 percent Cisco Systems, 1.9 percent Sun Microsystems, 1.5 percent IBM, 1.3 percent Mozilla, 1.3 percent XOOPS, 1.2 percent BEA, 1.1 percent Linux kernel, 0.9 percent
This article was posted to ISN the other day and struck a nerve. How many times are we going to see vulnerability statistics presented without qualification? Rather than really get into the details, I replied with a single simple example on why such statistics are misleading at best and incorrect at worst. The bulk of my reply follows. My hopes for Lisa or IBM/ISS clarifying this is already dwindling.
One other factor, that Lisa Vaas apparently didn’t ask about, is how ISS X-Force catalogs vulnerabilities, and if their method and standards could impact these numbers at all. Take for example, two X-Force vulnerability database entries: Oracle Critical Patch Update – July 2007 http://xforce.iss.net/xforce/xfdb/35490 18 CVE, 30+ Oracle Oracle Critical Patch Update – January 2007 http://xforce.iss.net/xforce/xfdb/31541 30 CVE, 50+ Oracle So when comparing numbers, you have 2 X-Force entries that equate to 48 CVE entries that equate to *more than 80* unique and distinct vulnerabilities according to Oracle. I’m not a math or stat guy, but I have a feeling that this could seriously skew the statistics above, especially when you consider that Microsoft and Apple both have a more distinct breakdown and separation in the X-Force database. Anyone from IBM/ISS care to clarify? Lisa, did you have more extensive notes on this aspect that didn’t make it in the article perhaps?
Several of us working on VDBs have debated over the years how best to handle vulnerabilities that aren’t necessarily remote or local. Issues like image or archive handling vulnerabilities, where the program processing a malformed file is prone to an overflow, traversal or denial of service. While one may argue they are ‘remote’ in the sense that if I e-mail you the file, the attack is definitely remote in a sense. But, if the malformed file is loaded via a floppy disk, the attack certainly isn’t ‘local’ or ‘requires physical’ access necessarily. So we need something that covers the grey area between vectors. A while back Steven Christey at CVE began using “context-dependent attacker” to describe such vulnerabilities. OSVDB tried to come up with another term for this but after some time, we couldn’t. So, from here on out, you will start noticing the use of “context-dependent attacker” in our vulnerability descriptions more frequently, and eventually when the classification scheme is overhauled it will appear there too.
NVD announced this week that they are now going to expand and provide vulnerability information in Spanish. I found this a bit amusing since OSVDB once thought that translating the database was a critical feature that needed to be delivered back in 2002. In fact, all of the language support was in the original OSVDB database schema and the backend code was created to handle it as we truly thought this would be implemented.
However, we quickly realized there were several issues with this concept including finding people to perform the translations! Additional concerns were raised as we spoke to more people in the security industry which included many conversations with non-US based security professionals (including a long ranting conversation with FX at Defcon). The critical concern was that much of the true meaning of the vulnerabilty is lost when the information is translated. The bottom line is that it was strongly believed that the vulnerability information in OSVDB should remain only in English.
OSVDB decided that we would not proceed any further with official plans to to translate the database, however, we have been contacted from other people that have wanted to translate OSVDB and we have provided permission to do so…
Here is a copy of the NVD announcement:
The National Vulnerability Database (NVD) is expanding to provide vulnerability translations. The first translation data feed is in Spanish and is being provided in cooperation with Inteco (http://www.inteco.es/), an entity of the Spanish government’s Ministry of Industry, Tourism, and Commerce (http://www.mityc.es/). Inteco is providing the translations and is solely responsible for the translation content. NVD is providing the translation infrastructure. The result of this cooperative effort is that NVD now contains an XML feed with 7,858 Spanish translations for the Common Vulnerabilities and Exposures (CVE) dictionary of security related software flaws. This feed will be maintained with translations for all new CVE vulnerabilities and, as with the other NVD data feeds, the data can be incorporated into commercial products and services with no licensing fees or restrictions. The translations are available through translation XML feeds at http://nvd.nist.gov/download.cfm#transxml.
We would love to hear any further thoughts (good and bad) on the value of translating vulnerability information into other languages.
Steven Christey of CVE recently commented on the fact that Microsoft, Adobe, Cisco, Sun and HP all released multi-issue advisories on the same day (Feb 13). My first reaction was to come up with an amusing graphic depicting this perfect storm. Due to not having any graphic editing skills and too much cynicism, I now wonder if these are the same vendors that continually bitch about irresponsible disclosure and it “hurting their customers”?
These same customers are now being subjected to patches for at least five major vendors on the same day. In some IT shops, this is devastating and difficult to manage and recover from. If a single patch has problems it forces the entire upgrade schedule to come to a halt until the problem can be resolved. If these vendors cared for their customers like they pretend to when someone releases a critical issue w/o vendor coordination, then they would consider staggering the patches to help alleviate the burden it causes on their beloved customers.