This blog entry is probably worth many pages of ranting, examining and dissecting the anatomy of a 0-day panic and the resulting fallout. Since this tends to happen more often than some of us care to stomach, I’ll touch on the major points and be liberal in pointing fingers. If you receive the “wag of my finger“, stop being part of the problem and wise up.
I blinked and missed someone disclosing that there was a dreaded 0-day vulnerability in Adobe Flash Player and that it was a big threat. Apparently Symantec noticed that evil Chinese sites were exploiting Flash and the current 220.127.116.11 could be successfully exploited. When pressed for details, Symantec backtracked and said that they were wrong and it appeared to be the same exploit as previously disclosed by Mark Dowd (CVE-2007-0071). Bad Symantec, poor research.
To make matters worse, Symantec then further claimed that even though it was an old issue, the “in-the-wild exploit was effective against stand-alone versions of Flash Player 18.104.22.168” and that not all versions had been patched correctly. Way to save face Ben Greenbaum of Symantec!! Oh wait, today he changed his mind and said that Symantec’s claims were based on erroneous conclusions and that the behavior of Flash on Linux they were observing was indeed intended by Adobe and not proof it was vulnerable. To make matters worse, Symantec researchers downloaded the “latest” Flash and found it “vulnerable”, which lead to their sky-is-falling panic. Shortly after, they realized that they didn’t download all of the security patches and had been exploiting a known vulnerable version of Flash. Oops?
Two rounds of hype-driven 0-day threat warnings, and no real new threat. Whew, hopefully Symantec raised their THREATCON to blood red or whatever is appropriate for such 0-day threats. You do monitor that don’t you?
This fiasco lead many news outlets and vendors to issue warnings about the new 0-day threat. Secunia, SecurityFocus/BID, SecurityTracker, CERT, and FrSIRT all released new warnings and created entries in their respective databases as a result. In the VDB world, this is a royal pain-in-the-ass to deal with. Secunia ‘revoked’ their entry, BID ‘retired’ their entry, SecurityTracker flaged theirs ‘duplicate entry’, FrSIRT ‘revoked’ their entry and CERT still has it listed.
Fortunately for OSVDB, we were a few hours behind the rest and noticed the discrepancies and waited for more information. Unfortunately, the rest of the world, including ALL of the VDBs and news outlets listed above (and others) failed miserably in using common sense and a government funded resource to better prevent this kind of problem. As of this posting, Secunia, BID, SecurityTracker, FrSIRT, CERT, Dancho, ComputerWorld and eWeek still don’t link to the CVE ID for the vulnerability. Only Adobe’s updated blog entry actually references CVE-2007-0071 (but doesn’t link to it). Secunia links to a previous ID that has seven CVEs associated with it. The original CVE was assigned 2007-01-04 and published around 2008-04-08, a month and a half prior to this mess.
VDBs, shame on you for adding to the confusion. Symantec, shame on you for crying 0-day when your own engineers screwed up badly. Shame on everyone for not clearing it up fully by linking to the correct CVE entry or their own previous entries.
Before any of you receiving a “wave of the finger” bitch, consider the real world impact of your actions. In this case, only 12 MILLION people ended up seeing a vague warning when they loaded their favorite game. Blizzard included the correct fix information which was the same as a month or more before, but the sudden ‘security alert’ (that is extremely rare) only prompted their customers to wonder, possibly panic and definitely kill some demons as a result.
Today just happened to be the right day where I saw the Jekyll and “Hide” of Sun though. A few days ago, |)ruid posted about a Solaris ypupdated vulnerability in which he says it corresponds to CVE-1999-0208 / OSVDB 11517. Given the original vulnerability was published in 1994, I had doubts it was truly the same vulnerability. I replied asking for confirmation, |)ruid replied and CC’d the Sun Security Coordination Team. Within 24 hours, Sun replied with a detailed analysis explaining how 11517 was different from the newly created OSVDB 43433, but very much related. This mail is a VDB maintainer’s wet dream; if only every vendor would provide this kind of detail when there is confusion over published vulnerability information. This is clearly the Dr. Jekyll locked up in a Sun complex somewhere who deserves kudos for the reply.
The Sun Microsystems “SunSolve” database is a quagmire of technical muck that is only rivaled by the IBM APAR database I believe. Tonight I find myself plowing through a grotesque changelog of Sun Java System Directory Server (SJSDS?). Sun apparently hasn’t fully mastered the idea of hyperlinking to make those annoying numbers on the left lead to somewhere with more information. So I log into the SunSolve database using my super secret ID associated with a sizable company that owns lots of Sun products. I type in a few numbers of interest off that list and away I … don’t go. Mr. Hide stops me quick, telling me that to read the bug IDs I have to be a better customer apparently.
You have selected content which is only available to registered SunSolve users with a valid Sun Service Plan. Please Login to access the restricted content of SunSolve and the Sun System Handbook if you are logged in to SunSolve and have received this message, please verify that you are associated with a valid support contract in the iSupport tool. If you have any questions about your support contract, please follow up with the Sun contract administrator contact at your company. If, however, none of the previous conditions apply, you may be trying to access a document that is no longer available. In this case please feel free to click on the SunSolve Feedback link at the bottom of the page and be sure to include the exact steps you took before you received this error message.
Wow, way to foil me via security through obscurity Sun Microsystems. Please take Mr. Hide and shove my beer bottle up his ass, sideways. Booze is the only way to adequately cope with the kind of headache born from vendors who can’t manage, organize and share information.
CVE just announced reaching 30,000 identifiers which is a pretty scary thing. CVE staff have a good eye for catching vulnerabilities from sources away from the mainstream (e.g. bugtraq) and they have the advantage of being a very widely accepted standard for tracking vulnerabilities. As companies and researchers request CVE numbers for disclosures, they get a lot of the information handed to them on a silver platter. Of course, sometimes that platter is full of mud and confusion as vendors don’t always provide clear details to help CVE accurately track and distinguish between multiple vulnerabilities. I’ve also pointed out many times in the past that CVE is a very unique VDB that provides identifiers for vulnerability tracking. They do not provide many fields associated with other VDBs (solution, creditee, etc). As such, they may have a single entry that covers multiple distinct vulnerabilities if they are the same class (XSS, SQLi, RFI), or if there is a lack of details but they know it affects the same product (Oracle). So when we see 30,000 identifiers, we have to realize that the real count of vulnerabilities is significantly higher.
CVE is run by The MITRE Corporation, sponsored / funded by the NCSD (US-CERT) of DHS under government contract. That means our tax dollars fund this database so it should be of particular interest to U.S. taxpayers in the security industry. I know from past discussions with CVE staff and other industry veterans that on any given day, they are more likely to have more work than available staff. That means the rate of vulnerabilities that get published is greater than the resources CVE can maintain to track them. In short, the 30,000 identifiers you see only represents a percentage of the vulnerabilities actually disclosed. We could probably debate what percentage that represents all day long, and I don’t think that is really the point here other than “we know it isn’t all of them”.
Every VDB suffers from the same thing. “Commercial” VDBs like X-Force, BID and Secunia have a full time staff that maintain their databases, like CVE does. Despite having all of these teams (some of them consisting of 10 or more people) maintain VDBs, we still see countless vulnerabilities that are ‘missed’ by all of them. This is not a slight against them in any way; it is a simple manner of resources available and the amount of information out there. Even with a large team sorting disclosed vulnerabilities, some teams spend time validating the findings before adding them to the database (Secunia), which is an incredible benefit for their customers. There is also a long standing parasitic nature to VDBs, with each of them watching the others as best they can, to help ensure they are tracking all the vulnerabilities they can. For example, OSVDB keeps a close eye on Secunia and CVE specifically, and as time permits we look to X-Force, BID, SecurityTracker and others. Each VDB tends to have some researchers that exclusively disclose vulnerabilities directly to the VDB of their choice. So each one I mention above will get word of vulnerabilities that the rest really have no way of knowing about short of watching each other like this. This VDB inbreeding (I will explain the choice of word some other time) is an accepted practice and I have touched on this in the past (CanSecWest 2005).
Due to the inbreeding and OSVDB’s ability to watch other resources, it occasionally frees up our moderators to go looking for more vulnerability information that wasn’t published in the mainstream. This usually involves grueling crawls through vendor knowledge-bases, mind-numbing changelogs, searching CVS type repositories and more. That leads to the point of this lengthy post. In doing this research, we begin to see how many more vulnerabilities are out there in the software we use, that escapes the VDBs most of the time. Only now, after four years and getting an incredible developer to make many aspects of the OSVDB wish-list a reality, do we finally begin to see all of this. As I have whined about for those four years, VDBs need to evolve and move beyond this purely “mainstream reactionary” model. Meaning, we have to stop watching the half dozen usual spots for new vulnerability information, creating our entries, rinsing and repeating. There is a lot more information out there just waiting to be read and added.
In the past few weeks, largely due to the ability to free up time due to the VDB inbreeding mentioned above, we’ve been able to dig into a few products more thoroughly. These examples are not meant to pick on any product / VDB or imply anything other than what is said above. In fact, this type of research is only possible because the other VDBs are doing a good job tracking the mainstream sources, and because some vendors publish full changelogs and don’t try to hide security related fixes. Kudos to all of them.
Example: Search your favorite VDB for ”inspircd”, a popular multi-platform IRC daemon. Compare the results of BID, Secunia, X-Force, SecurityTracker, and http://osvdb.org/ref/blog/inspircd-cve.png. Compare these results to OSVDB after digging into their changelogs. Do these same searches for “xfce” (10 OSVDB, 5 max elsewhere), “safesquid” (6 OSVDB, 1 max elsewhere), “beehive forum” (27 OSVDB, 8 max elsewhere) and “jetty” (25 OSVDB, 12 max elsewhere). Let me emphasize, I did not specifically hand pick these examples to put down any VDB, these are some of the products we’ve investigated in the last few weeks.
The real point here is that no matter what vulnerability disclosure statistic you read, regardless of which VDB it uses (including OSVDB), consider that the real number of vulnerabilities disclosed is likely much higher than any of us know or have documented. As always, if you see vulnerabilities in a vendor KB or changelog, and can’t find it in your favorite VDB, let them know. We all maintain e-mail addresses for submissions and we all strive to be as complete as possible.
Really IBM, the amount of information common to all three pages is overwhelming. Do you really need a new APAR number issued for component name or level? Can’t you just list them all in one APAR and save us time? More importantly, do we need three APAR entries that say “a security issue has been fixed” and make us dig up the information?
A while back, Microsoft announced they were moving to release patches on the second Tuesday of each month, lovingly called Patch Tuesday. Soon after, Oracle announced that they too would be moving to scheduled releases of patches on the Tuesday closest to the 15th day of January, April, July and October. Now, Cisco has announced they are moving to scheduled patches on the fourth Wednesday of the month in March and September of each calendar year.
In the attempt to make life easier on administrators and help avoid installing patches every few days, these scheduled releases are now causing organizations to enjoy life between monster patches.
Mar 11 – Microsoft
Mar 26 – Cisco
Apr 8 – Microsoft
Apr 15 – Oracle
May 13 – Microsoft
June 10 – Microsoft
July 8 – Microsoft
July 15 – Oracle
August 12 – Microsoft
September 9 – Microsoft
September 24 – Cisco
October 14 – Microsoft, Oracle
November 11 – Microsoft
December 9 – Microsoft
As you can see, October 14 promises to be a lot of fun for companies running Oracle products on Microsoft systems. While the scheduled dates look safe, I can’t wait until we see the ”perfect storm” of vendor patches.
In a recent discussion on the security metrics mailing list, Pete Lindstrom put forth a rough formula to throw out a number of vulnerabilities that have been discovered versus undiscovered. One of the data points that he cited lead me to his page on “undercover vulnerabilities”, his term for “0-day” in a certain context. Since the term “0-day” has been perverted to mean many things, he clearly defines his term as:
Undercover Vulnerability: A vulnerability that was generally unknown (e.g. not published on any lists, not discussed by “above ground” security folks) until it was actively exploited in the wild. The vulnerability was discovered through evidence of tampering or other means, not through the usual bugfinding ritual.
In my reply challenging some of his numbers, I specifically said that “if we consider that your number 20 is off by at least half, and I would personally guess it’s more like a small fraction, how does this change your numbers?” Pete took this in stride and offered to buy me a case of beer if I could find half a dozen that he didn’t have. Not one to pass up free booze and vulnerability research (yes, i’m weird) I spent several hours Friday doing just that. I ended up with 24 vulnerabilities that seemed to match his definition, roughly half of them in his time frame (“in the last two years”).
Pete’s page got me wondering just how many vulnerabilities classified as ‘undercover’ by his definition. Further, I thought about another question he asked on his page:
I am open to suggestions on an easy way to do this with TypePad (TypeLists, maybe?). Else, I’ll just periodically update as new vulns become available.
I cornered our lead developer Dave and said “make it so” while I mailed Pete asking if OSVDB could help in this effort. As a result, we now have a new classification that we call “Discovered In the Wild” that means the same thing as Pete’s “undercover vulnerability”. I have updated the 20 vulnerabilities listed on his page and added the flag to the ones I researched. This now shows 43 results which is good progress.
Not content with that, I asked a fellow geek who has a world more experience with IDS, NOC management and various devices that would be prone to catching such vulnerabilities “how many do you think were found this way last year”, to which she replied “at least 50”. So vulnerability researchers and OSVDB contributors, it’s up to you to help out! We’re looking for more instances of vulnerabilities being discovered “in the wild”, being exploited and subsequently disclosed (to mail list, vendor, whatever). Please cite your source as best as possible.
To see what we have so far:
- Under “Vulnerability Classification” and “Disclosure”
- Check “Discovered in the Wild”
Thanks to Pete Lindstrom and the Security Metrics mailing list for the input and great idea for a new classification!
Early in 2006, I posted about HP using multiple identifiers for the same vulnerability. Recently, Sun Microsystems has done a little overhaul to their advisory pages and I noticed that they too now use entirely too many tracking numbers.
For example, this Sun advisory has the following:
- Document ID: 200582
- Old Document ID: (formerly 103143)
- Bug ID: 6497289
- SA Document Body: PPGNRLA Internal ID use only.
Why is one tracking number so difficult?
Nutshell What you see here is the output of the ”arfis project”, a simple perl script. It automatically downloads and extract PHP projects from sourceforge.net and checks for Remote File Inclusion vulnerabilities. It then post’s the potential (now it’s -potential-, cause the script is in an early stadium) vuln to this blog.
The idea behind this tool was joked about by several VDB managers over a year ago due to the growing trend of false vulnerability reports popping up in 2006 and 2007. The style of many posts to mail lists were becoming the same, several signatures suggesting a tool or group was involved appeared and it was speculated that many remote file inclusion (RFI) vulnerabilities were the result of a very primitive “grep and gripe” style vulnerability ‘research’. Jump to today and we have this script doing what we suspected all along. Some will proclaim “genious!” and others may be quick to download and taste the fame of being a “vulnerability researcher”. Before you plan your victory party and brush up your resume to include vulnerability research, consider that this script is blindly searching projects for specific lines that suggest an application is vulnerable to RFI. Without looking at the source code manually, there is no way to accurately determine if it is a legitimate vulnerability or a false positive. The people using this script don’t seem to fully understand that and blindly use the tool w/o consideration.
Recently, 8 or so of these arfis-found vulnerabilities were reported to milw0rm for inclusion in their database. Upon examination, 6 of the 8 were not legitimate vulnerabilities. Of the 2 that were, one had been reported two years prior. This is a good indication of how trustworthy the tool is, early release or not, and what kind of burden it places on VDBs who do their best to vet vulnerability disclosures to a limited degree.
Yes yes, yet another “Month of..” campaign. If you track the mail lists, you may have seen a post about a “Month of [something]” Bugs. Despite little follow-up, this campaign is going strong on the 17th day demonstrating a variety of vulnerabilities in lycos.com, search.myway.com, images.google.com, mamma.com, metacrawler.com, ezilon.com, ask.com, ftpsearch.rambler.ru, searcheurope.com, blogs.yandex.ru, clusty.com, autos.msn.com, shopping.msn.com, gigablast.com, hotbot.com, search.yahoo.com and meta.ua.
Definitely an interesting project to follow.