Vulnerabilities reported ten years ago, they have no impact on your customers. If they do, then you are woefully behind and your customers are desperately hanging on to legacy products, scared to upgrade. For vendors who have kept up on security and adopted a responsible and timely manner for handling security, open up your records. Share with the world the ten or more year old vulnerabilities. Let the security community get a better picture of the real number of vulnerabilities reported to you, specifically the ones that never appeared in your advisories. This includes off-beat denial of service crashes, difficult to reproduce memory corruption, silly issues that required some level of access to begin with and everything else.
Some researchers have begun to do this, sharing more details of older disclosures that had vague details. Simple Nomad posted earlier this year about several old bugs as well as cleared up some confusion (via e-mail) regarding the old Palmetto FTP vulnerabilities.
I know this is a pipe-dream, as companies don’t want to admit to the number of vulnerabilities in their products, even ten years ago. Doesn’t matter that they fought uphill battles to win over the media and consumers with promises of how their software development life cycle matured or how they learned from their past. No way a vendor will dump hundreds of previously unpublished vulnerabilities on the world. On the rare chance a vendor will realize this can only help their reputation by sharing information and contributing to the VDB and metrics communities.. send them in! moderators[at]osvdb.org
This is a question OSVDB moderators, CVE staff and countless other VDB maintainers have asked. Today, Gunter Ollmann with IBM X-Force released his research trying to answer this question. Before you read on, I think this research is excellent. The relatively few criticisms I bring up are not the fault of Ollmann’s research and methodology, but the fault of his VDB of choice (and *every* other VDB) not having a complete data set.
Skimming his list, my first thought was that he was missing someone. Doing a quick search of OSVDB, I see that Lostmon Lords (aka ‘lostmon’) has close to 350 vulnerabilities published. How could the top ten list miss someone like this when his #10 only had 147? Read down to Ollmann’s caveat and there is a valid point, but sketchy wording. The data he is using relies on this information being public. As the caveat says though, “because they were disclosed on non-public lists” implies that the only source he or X-Force are using are mail lists such as Bugtraq and Full-disclosure. Back in the day, that was a pretty reliable source for a very high percentage of vulnerability information. In recent years though, a VDB must look at other sources of information to get a better picture. Web sites such as milw0rm get a steady stream of vulnerability information that is frequently not cross-posted to mail lists. In addition, many researchers (including lostmon) mail their discoveries directly to the VDBs and bypass the public mail lists. If researchers mail a few VDBs and not the rest, it creates a situation where the VDBs must start watching each other. This in turn leads to “VDB inbreeding” that Jake and I mentioned at CanSecWest 2005, which is a necessary evil if you want more data on vulnerabilities.
In May of 2008, OSVDB did the same research Ollmann did and we came up with different results. This was based on data we had available, which is still admittedly very incomplete (always need data manglers.) So who is right? Neither of us. Well, perhaps he is, perhaps we are, but unfortunately we’re both working with incomplete databases. As a matter of my opinion, I believe OSVDB has better coverage of vulnerabilities, while X-Force clearly has better consistency in their data and a fraction of the gaps we do.
Last, this data is interesting as is, but would be really fascinating if it was mixed with ‘researcher confidence’ (a big thing of Steve Christey/CVE and myself), in which we track a researcher’s track record for accuracy in disclosure. Someone that disclosed 500 vulnerabilities last year with a 10% error rate should not be above someone who found 475 with a 0% error rate. In addition, as Ollmann’s caveat says, these are pure numbers and do not factor in hundreds of XSS versus remote code execution in operating system default install services. Having a weight system that can be applied to a vulnerability (e.g., XSS = 3, SQLi = 7, remote code exec = 9) that is then factored into researcher could move beyond “who discovered the most” and perhaps start to answer “who found the most respectable vulnerabilities”.
Adam Penenberg wrote an article titled “The Black Market Code Industry” for FastCompany in which he details his research of two HP employees that actively sold exploit code in their spare time, at least one selling exploits in HP’s own software. According to the article, HP knew about one of the employees at the time of the article and were investigating. While a neat article and fun read, it left me with a lot more questions that I hope get answered at some point (how about a ‘Part 2’ Adam?).
- Does Rigano still work for HP now that the article has been out a week?
- Did either individual have access to source code to make their exploit writing easier? If so, did they have access to edit source code in any capacity (e.g. backdoors, adding vulnerable code)?
- Did Rigano actually sell his exploits? If so, to who and for how much? Checking the Full-Disclosure list archives, he appears to have had exploits for IIS 6.0, Firefox 2.x, MSIE 7, SAP, Apache, Microsoft Office and more.
- If Rigano did sell vulnerabilities, did he vette his buyers or could he have sold them to ‘enemy’ nations or hostile countries (relative I know)?
- Why is the FBI investigating a France based employee of HP?
- Is t0t0 a current employee of HP? If not, did he leave for his exploit selling activities? The article suggests that HP is aware of one of the two sellers. What do they have to say about this article now?
This blog entry is probably worth many pages of ranting, examining and dissecting the anatomy of a 0-day panic and the resulting fallout. Since this tends to happen more often than some of us care to stomach, I’ll touch on the major points and be liberal in pointing fingers. If you receive the “wag of my finger“, stop being part of the problem and wise up.
I blinked and missed someone disclosing that there was a dreaded 0-day vulnerability in Adobe Flash Player and that it was a big threat. Apparently Symantec noticed that evil Chinese sites were exploiting Flash and the current 220.127.116.11 could be successfully exploited. When pressed for details, Symantec backtracked and said that they were wrong and it appeared to be the same exploit as previously disclosed by Mark Dowd (CVE-2007-0071). Bad Symantec, poor research.
To make matters worse, Symantec then further claimed that even though it was an old issue, the “in-the-wild exploit was effective against stand-alone versions of Flash Player 18.104.22.168” and that not all versions had been patched correctly. Way to save face Ben Greenbaum of Symantec!! Oh wait, today he changed his mind and said that Symantec’s claims were based on erroneous conclusions and that the behavior of Flash on Linux they were observing was indeed intended by Adobe and not proof it was vulnerable. To make matters worse, Symantec researchers downloaded the “latest” Flash and found it “vulnerable”, which lead to their sky-is-falling panic. Shortly after, they realized that they didn’t download all of the security patches and had been exploiting a known vulnerable version of Flash. Oops?
Two rounds of hype-driven 0-day threat warnings, and no real new threat. Whew, hopefully Symantec raised their THREATCON to blood red or whatever is appropriate for such 0-day threats. You do monitor that don’t you?
This fiasco lead many news outlets and vendors to issue warnings about the new 0-day threat. Secunia, SecurityFocus/BID, SecurityTracker, CERT, and FrSIRT all released new warnings and created entries in their respective databases as a result. In the VDB world, this is a royal pain-in-the-ass to deal with. Secunia ‘revoked’ their entry, BID ‘retired’ their entry, SecurityTracker flaged theirs ‘duplicate entry’, FrSIRT ‘revoked’ their entry and CERT still has it listed.
Fortunately for OSVDB, we were a few hours behind the rest and noticed the discrepancies and waited for more information. Unfortunately, the rest of the world, including ALL of the VDBs and news outlets listed above (and others) failed miserably in using common sense and a government funded resource to better prevent this kind of problem. As of this posting, Secunia, BID, SecurityTracker, FrSIRT, CERT, Dancho, ComputerWorld and eWeek still don’t link to the CVE ID for the vulnerability. Only Adobe’s updated blog entry actually references CVE-2007-0071 (but doesn’t link to it). Secunia links to a previous ID that has seven CVEs associated with it. The original CVE was assigned 2007-01-04 and published around 2008-04-08, a month and a half prior to this mess.
VDBs, shame on you for adding to the confusion. Symantec, shame on you for crying 0-day when your own engineers screwed up badly. Shame on everyone for not clearing it up fully by linking to the correct CVE entry or their own previous entries.
Before any of you receiving a “wave of the finger” bitch, consider the real world impact of your actions. In this case, only 12 MILLION people ended up seeing a vague warning when they loaded their favorite game. Blizzard included the correct fix information which was the same as a month or more before, but the sudden ‘security alert’ (that is extremely rare) only prompted their customers to wonder, possibly panic and definitely kill some demons as a result.
Today just happened to be the right day where I saw the Jekyll and “Hide” of Sun though. A few days ago, |)ruid posted about a Solaris ypupdated vulnerability in which he says it corresponds to CVE-1999-0208 / OSVDB 11517. Given the original vulnerability was published in 1994, I had doubts it was truly the same vulnerability. I replied asking for confirmation, |)ruid replied and CC’d the Sun Security Coordination Team. Within 24 hours, Sun replied with a detailed analysis explaining how 11517 was different from the newly created OSVDB 43433, but very much related. This mail is a VDB maintainer’s wet dream; if only every vendor would provide this kind of detail when there is confusion over published vulnerability information. This is clearly the Dr. Jekyll locked up in a Sun complex somewhere who deserves kudos for the reply.
The Sun Microsystems “SunSolve” database is a quagmire of technical muck that is only rivaled by the IBM APAR database I believe. Tonight I find myself plowing through a grotesque changelog of Sun Java System Directory Server (SJSDS?). Sun apparently hasn’t fully mastered the idea of hyperlinking to make those annoying numbers on the left lead to somewhere with more information. So I log into the SunSolve database using my super secret ID associated with a sizable company that owns lots of Sun products. I type in a few numbers of interest off that list and away I … don’t go. Mr. Hide stops me quick, telling me that to read the bug IDs I have to be a better customer apparently.
You have selected content which is only available to registered SunSolve users with a valid Sun Service Plan. Please Login to access the restricted content of SunSolve and the Sun System Handbook if you are logged in to SunSolve and have received this message, please verify that you are associated with a valid support contract in the iSupport tool. If you have any questions about your support contract, please follow up with the Sun contract administrator contact at your company. If, however, none of the previous conditions apply, you may be trying to access a document that is no longer available. In this case please feel free to click on the SunSolve Feedback link at the bottom of the page and be sure to include the exact steps you took before you received this error message.
Wow, way to foil me via security through obscurity Sun Microsystems. Please take Mr. Hide and shove my beer bottle up his ass, sideways. Booze is the only way to adequately cope with the kind of headache born from vendors who can’t manage, organize and share information.
CVE just announced reaching 30,000 identifiers which is a pretty scary thing. CVE staff have a good eye for catching vulnerabilities from sources away from the mainstream (e.g. bugtraq) and they have the advantage of being a very widely accepted standard for tracking vulnerabilities. As companies and researchers request CVE numbers for disclosures, they get a lot of the information handed to them on a silver platter. Of course, sometimes that platter is full of mud and confusion as vendors don’t always provide clear details to help CVE accurately track and distinguish between multiple vulnerabilities. I’ve also pointed out many times in the past that CVE is a very unique VDB that provides identifiers for vulnerability tracking. They do not provide many fields associated with other VDBs (solution, creditee, etc). As such, they may have a single entry that covers multiple distinct vulnerabilities if they are the same class (XSS, SQLi, RFI), or if there is a lack of details but they know it affects the same product (Oracle). So when we see 30,000 identifiers, we have to realize that the real count of vulnerabilities is significantly higher.
CVE is run by The MITRE Corporation, sponsored / funded by the NCSD (US-CERT) of DHS under government contract. That means our tax dollars fund this database so it should be of particular interest to U.S. taxpayers in the security industry. I know from past discussions with CVE staff and other industry veterans that on any given day, they are more likely to have more work than available staff. That means the rate of vulnerabilities that get published is greater than the resources CVE can maintain to track them. In short, the 30,000 identifiers you see only represents a percentage of the vulnerabilities actually disclosed. We could probably debate what percentage that represents all day long, and I don’t think that is really the point here other than “we know it isn’t all of them”.
Every VDB suffers from the same thing. “Commercial” VDBs like X-Force, BID and Secunia have a full time staff that maintain their databases, like CVE does. Despite having all of these teams (some of them consisting of 10 or more people) maintain VDBs, we still see countless vulnerabilities that are ‘missed’ by all of them. This is not a slight against them in any way; it is a simple manner of resources available and the amount of information out there. Even with a large team sorting disclosed vulnerabilities, some teams spend time validating the findings before adding them to the database (Secunia), which is an incredible benefit for their customers. There is also a long standing parasitic nature to VDBs, with each of them watching the others as best they can, to help ensure they are tracking all the vulnerabilities they can. For example, OSVDB keeps a close eye on Secunia and CVE specifically, and as time permits we look to X-Force, BID, SecurityTracker and others. Each VDB tends to have some researchers that exclusively disclose vulnerabilities directly to the VDB of their choice. So each one I mention above will get word of vulnerabilities that the rest really have no way of knowing about short of watching each other like this. This VDB inbreeding (I will explain the choice of word some other time) is an accepted practice and I have touched on this in the past (CanSecWest 2005).
Due to the inbreeding and OSVDB’s ability to watch other resources, it occasionally frees up our moderators to go looking for more vulnerability information that wasn’t published in the mainstream. This usually involves grueling crawls through vendor knowledge-bases, mind-numbing changelogs, searching CVS type repositories and more. That leads to the point of this lengthy post. In doing this research, we begin to see how many more vulnerabilities are out there in the software we use, that escapes the VDBs most of the time. Only now, after four years and getting an incredible developer to make many aspects of the OSVDB wish-list a reality, do we finally begin to see all of this. As I have whined about for those four years, VDBs need to evolve and move beyond this purely “mainstream reactionary” model. Meaning, we have to stop watching the half dozen usual spots for new vulnerability information, creating our entries, rinsing and repeating. There is a lot more information out there just waiting to be read and added.
In the past few weeks, largely due to the ability to free up time due to the VDB inbreeding mentioned above, we’ve been able to dig into a few products more thoroughly. These examples are not meant to pick on any product / VDB or imply anything other than what is said above. In fact, this type of research is only possible because the other VDBs are doing a good job tracking the mainstream sources, and because some vendors publish full changelogs and don’t try to hide security related fixes. Kudos to all of them.
Example: Search your favorite VDB for ”inspircd”, a popular multi-platform IRC daemon. Compare the results of BID, Secunia, X-Force, SecurityTracker, and http://osvdb.org/ref/blog/inspircd-cve.png. Compare these results to OSVDB after digging into their changelogs. Do these same searches for “xfce” (10 OSVDB, 5 max elsewhere), “safesquid” (6 OSVDB, 1 max elsewhere), “beehive forum” (27 OSVDB, 8 max elsewhere) and “jetty” (25 OSVDB, 12 max elsewhere). Let me emphasize, I did not specifically hand pick these examples to put down any VDB, these are some of the products we’ve investigated in the last few weeks.
The real point here is that no matter what vulnerability disclosure statistic you read, regardless of which VDB it uses (including OSVDB), consider that the real number of vulnerabilities disclosed is likely much higher than any of us know or have documented. As always, if you see vulnerabilities in a vendor KB or changelog, and can’t find it in your favorite VDB, let them know. We all maintain e-mail addresses for submissions and we all strive to be as complete as possible.
Really IBM, the amount of information common to all three pages is overwhelming. Do you really need a new APAR number issued for component name or level? Can’t you just list them all in one APAR and save us time? More importantly, do we need three APAR entries that say “a security issue has been fixed” and make us dig up the information?
A while back, Microsoft announced they were moving to release patches on the second Tuesday of each month, lovingly called Patch Tuesday. Soon after, Oracle announced that they too would be moving to scheduled releases of patches on the Tuesday closest to the 15th day of January, April, July and October. Now, Cisco has announced they are moving to scheduled patches on the fourth Wednesday of the month in March and September of each calendar year.
In the attempt to make life easier on administrators and help avoid installing patches every few days, these scheduled releases are now causing organizations to enjoy life between monster patches.
Mar 11 – Microsoft
Mar 26 – Cisco
Apr 8 – Microsoft
Apr 15 – Oracle
May 13 – Microsoft
June 10 – Microsoft
July 8 – Microsoft
July 15 – Oracle
August 12 – Microsoft
September 9 – Microsoft
September 24 – Cisco
October 14 – Microsoft, Oracle
November 11 – Microsoft
December 9 – Microsoft
As you can see, October 14 promises to be a lot of fun for companies running Oracle products on Microsoft systems. While the scheduled dates look safe, I can’t wait until we see the ”perfect storm” of vendor patches.
In a recent discussion on the security metrics mailing list, Pete Lindstrom put forth a rough formula to throw out a number of vulnerabilities that have been discovered versus undiscovered. One of the data points that he cited lead me to his page on “undercover vulnerabilities”, his term for “0-day” in a certain context. Since the term “0-day” has been perverted to mean many things, he clearly defines his term as:
Undercover Vulnerability: A vulnerability that was generally unknown (e.g. not published on any lists, not discussed by “above ground” security folks) until it was actively exploited in the wild. The vulnerability was discovered through evidence of tampering or other means, not through the usual bugfinding ritual.
In my reply challenging some of his numbers, I specifically said that “if we consider that your number 20 is off by at least half, and I would personally guess it’s more like a small fraction, how does this change your numbers?” Pete took this in stride and offered to buy me a case of beer if I could find half a dozen that he didn’t have. Not one to pass up free booze and vulnerability research (yes, i’m weird) I spent several hours Friday doing just that. I ended up with 24 vulnerabilities that seemed to match his definition, roughly half of them in his time frame (“in the last two years”).
Pete’s page got me wondering just how many vulnerabilities classified as ‘undercover’ by his definition. Further, I thought about another question he asked on his page:
I am open to suggestions on an easy way to do this with TypePad (TypeLists, maybe?). Else, I’ll just periodically update as new vulns become available.
I cornered our lead developer Dave and said “make it so” while I mailed Pete asking if OSVDB could help in this effort. As a result, we now have a new classification that we call “Discovered In the Wild” that means the same thing as Pete’s “undercover vulnerability”. I have updated the 20 vulnerabilities listed on his page and added the flag to the ones I researched. This now shows 43 results which is good progress.
Not content with that, I asked a fellow geek who has a world more experience with IDS, NOC management and various devices that would be prone to catching such vulnerabilities “how many do you think were found this way last year”, to which she replied “at least 50”. So vulnerability researchers and OSVDB contributors, it’s up to you to help out! We’re looking for more instances of vulnerabilities being discovered “in the wild”, being exploited and subsequently disclosed (to mail list, vendor, whatever). Please cite your source as best as possible.
To see what we have so far:
- Under “Vulnerability Classification” and “Disclosure”
- Check “Discovered in the Wild”
Thanks to Pete Lindstrom and the Security Metrics mailing list for the input and great idea for a new classification!
Early in 2006, I posted about HP using multiple identifiers for the same vulnerability. Recently, Sun Microsystems has done a little overhaul to their advisory pages and I noticed that they too now use entirely too many tracking numbers.
For example, this Sun advisory has the following:
- Document ID: 200582
- Old Document ID: (formerly 103143)
- Bug ID: 6497289
- SA Document Body: PPGNRLA Internal ID use only.
Why is one tracking number so difficult?