Perhaps it is the fine tequila this evening, but I really don’t get how our industry can latch on to the recent ‘Aurora’ incident and try to take Microsoft to task about it. The amount of news on this has been overwhelming, and I will try to very roughly summarize:
- News surfaces Google, Adobe and 30+ companies hit by “0-day” attack
- Google uses this for political overtones
- Originally thought to be Adobe 0-day, revealed it was MSIE 0-day
- Jan 14, confirmed it is MSIE vuln, shortly after dubbed “aurora”
- Jan 21, uproar over MS knowing about the vuln since Sept
Now, here is where we get to the whole forest, trees and some analogy about eyesight. Oh, I’ll warn (and surprise) you in advance, I am giving Microsoft the benefit of the doubt here (well, for half the blog post) and throwing this back at journalists and the security community instead. Let’s look at this from a different angle.
The big issue that is newsworthy is that Microsoft knew of this vulnerability in September, and didn’t issue a patch until late January. What is not clear, is if Microsoft knew it was being exploited. The wording of the Wired article doesn’t make it clear: “aware months ago of a critical security vulnerability well before hackers exploited it to breach Google, Adobe and other large U.S. companies” and “Microsoft confirmed it learned of the so-called ‘zero-day’ flaw months ago”. Errr, nice wording. Microsoft was aware of the vulnerability (technically), before hackers exploited it, but doesn’t specifically say if they KNEW hackers were exploiting it. Microsoft learned of the “0-day” months ago? No, bad bad bad. This is taking an over-abused term and making it even worse. If a vulnerability is found and reported to the vendor before it is exploited, is it still 0-day (tree, forest, no one there to hear it falling)?
Short of Microsoft admitting they knew it was being exploited, we can only speculate. So, for fun, let’s give them a pass on that one and assume it was like any other privately disclosed bug. They were working it like any other issue, fixing, patching, regression testing, etc. Good Microsoft!
Bad Microsoft! But, before you jump on the bandwagon, bad journalists! Bad security community!
Why do you care they sat on this one vulnerability for six months? Why is that such a big deal? Am I the only one who missed the articles pointing out that they actually sat on five code execution bugs for longer? Where was the outpour of blogs or news articles mentioning that “aurora” was one of six vulnerabilities reported to them during or before September, all in MSIE, all that allowed remote code execution (tree, forest, not seeing one for the other)?
|CVE||Reported to MS||Disclosed||Time to Patch|
|CVE-2010-0244||2009-07-14||2010-01-21||6 Months, 7 Days (191 days)|
|CVE-2010-0245||2009-07-14||2010-01-21||6 Months, 7 Days (191 days)|
|CVE-2010-0246||2009-07-16||2010-01-21||6 Months, 5 Days (189 days)|
|CVE-2010-0248||2009-08-14||2010-01-21||5 Months, 7 days (160 days)|
|CVE-2010-0247||2009-09-03||2010-01-21||4 Months, 18 days (140 days)|
|CVE-2010-0249||2009-09-??||2010-01-14||4 Months, 11 days (133 days) – approx|
|CVE-2010-0027||2009-11-15||2010-01-21||2 Months, 6 days (67 days)|
|CVE-2009-4074||2009-11-20||2009-11-21||2 Months, 1 day (62 days)|
Remind me again, why the “Aurora” conspiracy is noteworthy? If Microsoft knew of six remote code execution bugs, all from the September time-frame, why is one any more severe than the other? Is it because one was used to compromise hosts, detected and published in an extremely abnormal fashion? Are we actually trying to hold Microsoft accountable on that single vulnerability when the five others just happened not to be used to compromise Google, Adobe and others?
Going back to the Wired article, they say on the second to last paragraph: “On Thursday, meanwhile, Microsoft released a cumulative security update for Internet Explorer that fixes the flaw, as well as seven other security vulnerabilities that would allow an attacker to remotely execute code on a victim’s computer.” Really, Wired? That late in the article, you gloss over “seven other vulnerabilities” that would allow remote code execution? And worse, you don’t point out that Microsoft was informed of five of them BEFORE AURORA?
Seriously, I am the first one to hold Microsoft over the flames for bad practices, but that goes beyond my boundaries. If you are going to take them to task over all this, at least do it right. SIX CODE EXECUTION VULNERABILITIES that they KNEW ABOUT FOR SIX MONTHS. Beating them up over just one is amateur hour in this curmudgeonly world.
Earlier this evening, there was a Twitter debate regarding a proposed standard for responsible vulnerability disclosure. It referred to ISO/IEC 29147, a proposed standard for responsibly disclosing a vulnerability. @dinodaizovi brought up a fresh angle, that the “responsible disclosure” name itself completely ignored the aspect of the vendor practicing “responsible remediation”. That term should really be more in the center of our minds and discussion. The lack of “responsible remediation” is why so many researchers are fed up with dealing with vendors. That is one reason some use services like ZDI or iDefense, not just the cash.
The “responsible disclosure” debate is stale for the most part. We’ll never agree on how much time is ‘right’ for a vendor to fix a vulnerability. Some researchers think it’s days, other think weeks or months. In the paraphrased words of some female vendor on some boring responsible disclosure panel a few years back, “if i can have a kid in 9 months, i should be able to fix a vulnerability too“. Yet 9 months isn’t reasonable to some vendors like HP, who routinely break the 1,000 day mark, even for simple XSS.
@mckeay brought up another aspect to the responsible disclosure debate that was actually fresh, asking what part consumers played in the disclosure process. While I believe it is a neat aspect and something most haven’t considered, I personally believe it is quickly answered by “consumers can put financial pressure on vendors that don’t play well with others”. In reality, consumers are lazy. It takes more than a few bad acts to get us to spend time an energy finding a new vendor. Short of anally raping us with a router and pouring lemon juice in our festering wound, nine times out of ten, we will not find a new vendor.
Back to @dinodaizovi. He is right, any standard for disclosure should be equally centered on the vendor as it is for the researcher. Researchers can easily fall back on RFP’s “rfpolicy” disclosure policy and change X days to something they believe in. The framework is still perfectly valid and outlines the process, the time frames are always up for debate.
What if we carried this one step beyond? How about making the ISO standards apply to any and every vulnerability, regardless of who found it? If BigVendor finds a vulnerability during internal testing and fixes it, don’t consumers have a right to know? When BigVendor says “upgrade to Service Pack 18″ and only gives us a reason of “big stability enhancements!!”, shouldn’t we have a right to know those enhancements translate into 17 remotely exploitable vulnerabilities discovered during internal testing and QA? Wouldn’t *that* knowledge be a more significant reason to upgrade and apply the service pack?
I realize it is a pipe dream to think that most vendors would ever offer that level of transparency, even months (years?) after a given issue is fixed. In reality though, they are the proverbial large mythical flightless birds who stick their heads in the sand rather than face a difficult situation (ostriches are real and don’t bury their heads). It has been proven countless times that serious vulnerabilities in big vendors (e.g., Microsoft, Apple, Adobe) are being discovered by multiple parties. No one with an inkling of common sense and rational thinking can believe that the ‘bad guys’ aren’t also discovering some of these bugs. We’re long past the point of vendors honestly thinking that they can get away with some notion that they have a reputation for ‘security’. Add it up, and we’re to that time where the big vendors should be disclosing vulnerabilities discovered during their internal QA / SLDC process. The reputation of insecure software really can’t hurt them any more, and transparency is finally the one thing that could buy back some degree of consumer confidence.
Perhaps now is the time where ‘responsible disclosure’ should apply equally to hackers, security researchers and vendors, as well as apply to ‘responsible remediation’. Because really, some 20 years after the disclosure debate got going, do we really think we need to try to apply more guidelines to researchers giving away $250/hr consulting work or “hackers” posting vulnerabilities as a hobby? Vendors that have tried to label or apply policy to these people were simply blame-shifting from day one, while not applying that desired policy to themselves.
Vulnerabilities reported ten years ago, they have no impact on your customers. If they do, then you are woefully behind and your customers are desperately hanging on to legacy products, scared to upgrade. For vendors who have kept up on security and adopted a responsible and timely manner for handling security, open up your records. Share with the world the ten or more year old vulnerabilities. Let the security community get a better picture of the real number of vulnerabilities reported to you, specifically the ones that never appeared in your advisories. This includes off-beat denial of service crashes, difficult to reproduce memory corruption, silly issues that required some level of access to begin with and everything else.
Some researchers have begun to do this, sharing more details of older disclosures that had vague details. Simple Nomad posted earlier this year about several old bugs as well as cleared up some confusion (via e-mail) regarding the old Palmetto FTP vulnerabilities.
I know this is a pipe-dream, as companies don’t want to admit to the number of vulnerabilities in their products, even ten years ago. Doesn’t matter that they fought uphill battles to win over the media and consumers with promises of how their software development life cycle matured or how they learned from their past. No way a vendor will dump hundreds of previously unpublished vulnerabilities on the world. On the rare chance a vendor will realize this can only help their reputation by sharing information and contributing to the VDB and metrics communities.. send them in! moderators[at]osvdb.org
This is a question OSVDB moderators, CVE staff and countless other VDB maintainers have asked. Today, Gunter Ollmann with IBM X-Force released his research trying to answer this question. Before you read on, I think this research is excellent. The relatively few criticisms I bring up are not the fault of Ollmann’s research and methodology, but the fault of his VDB of choice (and *every* other VDB) not having a complete data set.
Skimming his list, my first thought was that he was missing someone. Doing a quick search of OSVDB, I see that Lostmon Lords (aka ‘lostmon’) has close to 350 vulnerabilities published. How could the top ten list miss someone like this when his #10 only had 147? Read down to Ollmann’s caveat and there is a valid point, but sketchy wording. The data he is using relies on this information being public. As the caveat says though, “because they were disclosed on non-public lists” implies that the only source he or X-Force are using are mail lists such as Bugtraq and Full-disclosure. Back in the day, that was a pretty reliable source for a very high percentage of vulnerability information. In recent years though, a VDB must look at other sources of information to get a better picture. Web sites such as milw0rm get a steady stream of vulnerability information that is frequently not cross-posted to mail lists. In addition, many researchers (including lostmon) mail their discoveries directly to the VDBs and bypass the public mail lists. If researchers mail a few VDBs and not the rest, it creates a situation where the VDBs must start watching each other. This in turn leads to “VDB inbreeding” that Jake and I mentioned at CanSecWest 2005, which is a necessary evil if you want more data on vulnerabilities.
In May of 2008, OSVDB did the same research Ollmann did and we came up with different results. This was based on data we had available, which is still admittedly very incomplete (always need data manglers.) So who is right? Neither of us. Well, perhaps he is, perhaps we are, but unfortunately we’re both working with incomplete databases. As a matter of my opinion, I believe OSVDB has better coverage of vulnerabilities, while X-Force clearly has better consistency in their data and a fraction of the gaps we do.
Last, this data is interesting as is, but would be really fascinating if it was mixed with ‘researcher confidence’ (a big thing of Steve Christey/CVE and myself), in which we track a researcher’s track record for accuracy in disclosure. Someone that disclosed 500 vulnerabilities last year with a 10% error rate should not be above someone who found 475 with a 0% error rate. In addition, as Ollmann’s caveat says, these are pure numbers and do not factor in hundreds of XSS versus remote code execution in operating system default install services. Having a weight system that can be applied to a vulnerability (e.g., XSS = 3, SQLi = 7, remote code exec = 9) that is then factored into researcher could move beyond “who discovered the most” and perhaps start to answer “who found the most respectable vulnerabilities”.
Adam Penenberg wrote an article titled “The Black Market Code Industry” for FastCompany in which he details his research of two HP employees that actively sold exploit code in their spare time, at least one selling exploits in HP’s own software. According to the article, HP knew about one of the employees at the time of the article and were investigating. While a neat article and fun read, it left me with a lot more questions that I hope get answered at some point (how about a ‘Part 2’ Adam?).
- Does Rigano still work for HP now that the article has been out a week?
- Did either individual have access to source code to make their exploit writing easier? If so, did they have access to edit source code in any capacity (e.g. backdoors, adding vulnerable code)?
- Did Rigano actually sell his exploits? If so, to who and for how much? Checking the Full-Disclosure list archives, he appears to have had exploits for IIS 6.0, Firefox 2.x, MSIE 7, SAP, Apache, Microsoft Office and more.
- If Rigano did sell vulnerabilities, did he vette his buyers or could he have sold them to ‘enemy’ nations or hostile countries (relative I know)?
- Why is the FBI investigating a France based employee of HP?
- Is t0t0 a current employee of HP? If not, did he leave for his exploit selling activities? The article suggests that HP is aware of one of the two sellers. What do they have to say about this article now?
This blog entry is probably worth many pages of ranting, examining and dissecting the anatomy of a 0-day panic and the resulting fallout. Since this tends to happen more often than some of us care to stomach, I’ll touch on the major points and be liberal in pointing fingers. If you receive the “wag of my finger“, stop being part of the problem and wise up.
I blinked and missed someone disclosing that there was a dreaded 0-day vulnerability in Adobe Flash Player and that it was a big threat. Apparently Symantec noticed that evil Chinese sites were exploiting Flash and the current 220.127.116.11 could be successfully exploited. When pressed for details, Symantec backtracked and said that they were wrong and it appeared to be the same exploit as previously disclosed by Mark Dowd (CVE-2007-0071). Bad Symantec, poor research.
To make matters worse, Symantec then further claimed that even though it was an old issue, the “in-the-wild exploit was effective against stand-alone versions of Flash Player 18.104.22.168” and that not all versions had been patched correctly. Way to save face Ben Greenbaum of Symantec!! Oh wait, today he changed his mind and said that Symantec’s claims were based on erroneous conclusions and that the behavior of Flash on Linux they were observing was indeed intended by Adobe and not proof it was vulnerable. To make matters worse, Symantec researchers downloaded the “latest” Flash and found it “vulnerable”, which lead to their sky-is-falling panic. Shortly after, they realized that they didn’t download all of the security patches and had been exploiting a known vulnerable version of Flash. Oops?
Two rounds of hype-driven 0-day threat warnings, and no real new threat. Whew, hopefully Symantec raised their THREATCON to blood red or whatever is appropriate for such 0-day threats. You do monitor that don’t you?
This fiasco lead many news outlets and vendors to issue warnings about the new 0-day threat. Secunia, SecurityFocus/BID, SecurityTracker, CERT, and FrSIRT all released new warnings and created entries in their respective databases as a result. In the VDB world, this is a royal pain-in-the-ass to deal with. Secunia ‘revoked’ their entry, BID ‘retired’ their entry, SecurityTracker flaged theirs ‘duplicate entry’, FrSIRT ‘revoked’ their entry and CERT still has it listed.
Fortunately for OSVDB, we were a few hours behind the rest and noticed the discrepancies and waited for more information. Unfortunately, the rest of the world, including ALL of the VDBs and news outlets listed above (and others) failed miserably in using common sense and a government funded resource to better prevent this kind of problem. As of this posting, Secunia, BID, SecurityTracker, FrSIRT, CERT, Dancho, ComputerWorld and eWeek still don’t link to the CVE ID for the vulnerability. Only Adobe’s updated blog entry actually references CVE-2007-0071 (but doesn’t link to it). Secunia links to a previous ID that has seven CVEs associated with it. The original CVE was assigned 2007-01-04 and published around 2008-04-08, a month and a half prior to this mess.
VDBs, shame on you for adding to the confusion. Symantec, shame on you for crying 0-day when your own engineers screwed up badly. Shame on everyone for not clearing it up fully by linking to the correct CVE entry or their own previous entries.
Before any of you receiving a “wave of the finger” bitch, consider the real world impact of your actions. In this case, only 12 MILLION people ended up seeing a vague warning when they loaded their favorite game. Blizzard included the correct fix information which was the same as a month or more before, but the sudden ‘security alert’ (that is extremely rare) only prompted their customers to wonder, possibly panic and definitely kill some demons as a result.
Today just happened to be the right day where I saw the Jekyll and “Hide” of Sun though. A few days ago, |)ruid posted about a Solaris ypupdated vulnerability in which he says it corresponds to CVE-1999-0208 / OSVDB 11517. Given the original vulnerability was published in 1994, I had doubts it was truly the same vulnerability. I replied asking for confirmation, |)ruid replied and CC’d the Sun Security Coordination Team. Within 24 hours, Sun replied with a detailed analysis explaining how 11517 was different from the newly created OSVDB 43433, but very much related. This mail is a VDB maintainer’s wet dream; if only every vendor would provide this kind of detail when there is confusion over published vulnerability information. This is clearly the Dr. Jekyll locked up in a Sun complex somewhere who deserves kudos for the reply.
The Sun Microsystems “SunSolve” database is a quagmire of technical muck that is only rivaled by the IBM APAR database I believe. Tonight I find myself plowing through a grotesque changelog of Sun Java System Directory Server (SJSDS?). Sun apparently hasn’t fully mastered the idea of hyperlinking to make those annoying numbers on the left lead to somewhere with more information. So I log into the SunSolve database using my super secret ID associated with a sizable company that owns lots of Sun products. I type in a few numbers of interest off that list and away I … don’t go. Mr. Hide stops me quick, telling me that to read the bug IDs I have to be a better customer apparently.
You have selected content which is only available to registered SunSolve users with a valid Sun Service Plan. Please Login to access the restricted content of SunSolve and the Sun System Handbook if you are logged in to SunSolve and have received this message, please verify that you are associated with a valid support contract in the iSupport tool. If you have any questions about your support contract, please follow up with the Sun contract administrator contact at your company. If, however, none of the previous conditions apply, you may be trying to access a document that is no longer available. In this case please feel free to click on the SunSolve Feedback link at the bottom of the page and be sure to include the exact steps you took before you received this error message.
Wow, way to foil me via security through obscurity Sun Microsystems. Please take Mr. Hide and shove my beer bottle up his ass, sideways. Booze is the only way to adequately cope with the kind of headache born from vendors who can’t manage, organize and share information.
CVE just announced reaching 30,000 identifiers which is a pretty scary thing. CVE staff have a good eye for catching vulnerabilities from sources away from the mainstream (e.g. bugtraq) and they have the advantage of being a very widely accepted standard for tracking vulnerabilities. As companies and researchers request CVE numbers for disclosures, they get a lot of the information handed to them on a silver platter. Of course, sometimes that platter is full of mud and confusion as vendors don’t always provide clear details to help CVE accurately track and distinguish between multiple vulnerabilities. I’ve also pointed out many times in the past that CVE is a very unique VDB that provides identifiers for vulnerability tracking. They do not provide many fields associated with other VDBs (solution, creditee, etc). As such, they may have a single entry that covers multiple distinct vulnerabilities if they are the same class (XSS, SQLi, RFI), or if there is a lack of details but they know it affects the same product (Oracle). So when we see 30,000 identifiers, we have to realize that the real count of vulnerabilities is significantly higher.
CVE is run by The MITRE Corporation, sponsored / funded by the NCSD (US-CERT) of DHS under government contract. That means our tax dollars fund this database so it should be of particular interest to U.S. taxpayers in the security industry. I know from past discussions with CVE staff and other industry veterans that on any given day, they are more likely to have more work than available staff. That means the rate of vulnerabilities that get published is greater than the resources CVE can maintain to track them. In short, the 30,000 identifiers you see only represents a percentage of the vulnerabilities actually disclosed. We could probably debate what percentage that represents all day long, and I don’t think that is really the point here other than “we know it isn’t all of them”.
Every VDB suffers from the same thing. “Commercial” VDBs like X-Force, BID and Secunia have a full time staff that maintain their databases, like CVE does. Despite having all of these teams (some of them consisting of 10 or more people) maintain VDBs, we still see countless vulnerabilities that are ‘missed’ by all of them. This is not a slight against them in any way; it is a simple manner of resources available and the amount of information out there. Even with a large team sorting disclosed vulnerabilities, some teams spend time validating the findings before adding them to the database (Secunia), which is an incredible benefit for their customers. There is also a long standing parasitic nature to VDBs, with each of them watching the others as best they can, to help ensure they are tracking all the vulnerabilities they can. For example, OSVDB keeps a close eye on Secunia and CVE specifically, and as time permits we look to X-Force, BID, SecurityTracker and others. Each VDB tends to have some researchers that exclusively disclose vulnerabilities directly to the VDB of their choice. So each one I mention above will get word of vulnerabilities that the rest really have no way of knowing about short of watching each other like this. This VDB inbreeding (I will explain the choice of word some other time) is an accepted practice and I have touched on this in the past (CanSecWest 2005).
Due to the inbreeding and OSVDB’s ability to watch other resources, it occasionally frees up our moderators to go looking for more vulnerability information that wasn’t published in the mainstream. This usually involves grueling crawls through vendor knowledge-bases, mind-numbing changelogs, searching CVS type repositories and more. That leads to the point of this lengthy post. In doing this research, we begin to see how many more vulnerabilities are out there in the software we use, that escapes the VDBs most of the time. Only now, after four years and getting an incredible developer to make many aspects of the OSVDB wish-list a reality, do we finally begin to see all of this. As I have whined about for those four years, VDBs need to evolve and move beyond this purely “mainstream reactionary” model. Meaning, we have to stop watching the half dozen usual spots for new vulnerability information, creating our entries, rinsing and repeating. There is a lot more information out there just waiting to be read and added.
In the past few weeks, largely due to the ability to free up time due to the VDB inbreeding mentioned above, we’ve been able to dig into a few products more thoroughly. These examples are not meant to pick on any product / VDB or imply anything other than what is said above. In fact, this type of research is only possible because the other VDBs are doing a good job tracking the mainstream sources, and because some vendors publish full changelogs and don’t try to hide security related fixes. Kudos to all of them.
Example: Search your favorite VDB for ”inspircd”, a popular multi-platform IRC daemon. Compare the results of BID, Secunia, X-Force, SecurityTracker, and http://osvdb.org/ref/blog/inspircd-cve.png. Compare these results to OSVDB after digging into their changelogs. Do these same searches for “xfce” (10 OSVDB, 5 max elsewhere), “safesquid” (6 OSVDB, 1 max elsewhere), “beehive forum” (27 OSVDB, 8 max elsewhere) and “jetty” (25 OSVDB, 12 max elsewhere). Let me emphasize, I did not specifically hand pick these examples to put down any VDB, these are some of the products we’ve investigated in the last few weeks.
The real point here is that no matter what vulnerability disclosure statistic you read, regardless of which VDB it uses (including OSVDB), consider that the real number of vulnerabilities disclosed is likely much higher than any of us know or have documented. As always, if you see vulnerabilities in a vendor KB or changelog, and can’t find it in your favorite VDB, let them know. We all maintain e-mail addresses for submissions and we all strive to be as complete as possible.
Really IBM, the amount of information common to all three pages is overwhelming. Do you really need a new APAR number issued for component name or level? Can’t you just list them all in one APAR and save us time? More importantly, do we need three APAR entries that say “a security issue has been fixed” and make us dig up the information?
A while back, Microsoft announced they were moving to release patches on the second Tuesday of each month, lovingly called Patch Tuesday. Soon after, Oracle announced that they too would be moving to scheduled releases of patches on the Tuesday closest to the 15th day of January, April, July and October. Now, Cisco has announced they are moving to scheduled patches on the fourth Wednesday of the month in March and September of each calendar year.
In the attempt to make life easier on administrators and help avoid installing patches every few days, these scheduled releases are now causing organizations to enjoy life between monster patches.
Mar 11 – Microsoft
Mar 26 – Cisco
Apr 8 – Microsoft
Apr 15 – Oracle
May 13 – Microsoft
June 10 – Microsoft
July 8 – Microsoft
July 15 – Oracle
August 12 – Microsoft
September 9 – Microsoft
September 24 – Cisco
October 14 – Microsoft, Oracle
November 11 – Microsoft
December 9 – Microsoft
As you can see, October 14 promises to be a lot of fun for companies running Oracle products on Microsoft systems. While the scheduled dates look safe, I can’t wait until we see the ”perfect storm” of vendor patches.