Category vulnerability-statistics, everything about vulnerability-statistics

February Update: OSVDB Winter 2010 Fundraising Goal

Posted by lyger Mon, 01 Mar 2010 13:36:00 GMT

Back in early January, I issued a challenge to donate to OSF's Winter Fundraiser for every new vulnerability pushed into OSVDB. Two of the three months have come and gone, and even though January was a little more productive than February in terms of new vulnerabilities, the moderation team is still making good progress:

2010-02-01: 13 vulns pushed, 133 vulns updated
2010-02-02: 31 vulns pushed, 79 vulns updated
2010-02-03: 25 vulns pushed, 145 vulns updated
2010-02-04: 21 vulns pushed, 31 vulns updated
2010-02-05: 25 vulns pushed, 153 vulns updated
2010-02-06: 8 vulns pushed, 76 vulns updated
2010-02-07: 3 vulns pushed, 278 vulns updated
2010-02-08: 27 vulns pushed, 64 vulns updated
2010-02-09: 47 vulns pushed, 159 vulns updated
2010-02-10: 37 vulns pushed, 160 vulns updated
2010-02-11: 16 vulns pushed, 59 vulns updated
2010-02-12: 27 vulns pushed, 128 vulns updated
2010-02-13: 10 vulns pushed, 51 vulns updated
2010-02-14: 4 vulns pushed, 112 vulns updated
2010-02-15: 12 vulns pushed, 81 vulns updated
2010-02-16: 23 vulns pushed, 181 vulns updated
2010-02-17: 28 vulns pushed, 235 vulns updated
2010-02-18: 25 vulns pushed, 119 vulns updated
2010-02-19: 43 vulns pushed, 261 vulns updated
2010-02-20: 11 vulns pushed, 126 vulns updated
2010-02-21: 2 vulns pushed, 34 vulns updated
2010-02-22: 3 vulns pushed, 64 vulns updated
2010-02-23: 41 vulns pushed, 221 vulns updated
2010-02-24: 37 vulns pushed, 112 vulns updated
2010-02-25: 15 vulns pushed, 138 vulns updated
2010-02-26: 17 vulns pushed, 146 vulns updated
2010-02-27: 9 vulns pushed, 17 vulns updated
2010-02-28: 8 vulns pushed, 24 vulns updated

With 568 new vulnerabilities pushed in February, we're now up to 1,223 new entries for 2010; personally, I'd like to see that number hit at least 2,000 by the end of March (3,000 may be out of reach, but never say never), but that will depend on the time and efforts of our moderation team and the amount of vulnerabilities uncovered by our multiple reference sources. Please remember that I will donate $0.50 to OSF for every new vulnerability pushed into the database through April 1 (and no, there will not be an April Fools announcement saying that the challenge has been called off), and we're hoping to obtain some matching offers to help offset the costs of maintaining the database. A special "thank you" goes to all parties who have offered to match the challenge so far, and we hope others who find OSVDB to be a valuable resource can jump in and help us out as well.

31 more days for the challenge... and away... we... go.

Posted in Vulnerability Statistics, OSVDB News | no comments

Adobe, Qualys, CVE and Math

Posted by jericho Sat, 19 Dec 2009 21:07:00 GMT

Elinor Mills wrote an article titled Firefox, Adobe top buggiest-software list. In it, she quotes Qualys as providing vulnerability statistics for Mozilla, Adobe and others. Qualys states:

The number of vulnerabilities in Adobe programs rose from 14 last year to 45 this year, while those in Microsoft software dropped from 44 to 41, according to Qualys. Internet Explorer, Windows Media Player and Microsoft Office together had 30 vulnerabilities.

This caught my attention immediately, as I know I have mangled more than 45 Adobe entries this year.

First, the "number of vulnerabilities" game will always have wiggle room, which has been discussed before. A big factor for statistic discrepancy when using public databases is the level of abstraction. CVE tends to bunch up vulnerabilities in a single CVE, where OSVDB tends to break them out. Over the past year, X-Force and BID have started abstracting more and more as well.

Either way, Qualys cited their source, NVD, which is entirely based on CVE. How they got 45 vulns in "Adobe programs" baffles me. My count says 97 Adobe vulns, 95 of them have CVEs assigned to them (covered by a total of 93 CVEs). OSVDB abstracted the entries like CVE did for the most part, but split out CVE-2009-1872 as distinct XSS vulnerabilities. OSVDB also has two entries that do not have CVE, 55820 and 56281.

Where did Qualys get 45 if they are using the same CVE data set OSVDB does? This discrepancy has nothing to do with abstraction, so something else appears to be going on. Doing a few more searches, I believe I figured it out. Searching OSVDB for "Adobe Reader" in 2009 yields 44 entries, one off from their cited 45. That could be easily explained as OSVDB also has 9 "Adobe Multiple Products" entries that could cover Reader as well. This may in turn be a breakdown where Qualys or Mills did not specify "Adobe Software" (cumulative, all software they release) versus "Adobe Reader" or some other specific software they release.

Qualys tallied 102 vulnerabilities that were found in Firefox this year, up from 90 last year.

What is certainly a discrepancy due to abstraction, OSVDB has 74 vulnerabilities specific to Mozilla Firefox (two without CVE), 11 for "Mozilla Multiple Browsers" (Firefox, Seamonkey, etc) and 81 for "Mozilla Multiple Products" (Firefox, Thunderbird, etc). While my numbers are somewhat anecdotal, because I cannot remember every single entry, I can say that most of the 'multiple' vulnerabilities include Firefox. That means OSVDB tracked as many as, but possibly less than, 166 vulnerabilities in Firefox.

Microsoft software dropped from 44 to 41, according to Qualys. Internet Explorer, Windows Media Player and Microsoft Office together had 30 vulnerabilities.

According to my searches on OSVDB, we get the following numbers:

234 vulnerabilities in Microsoft, only 4 without CVE
50 vulnerabilities in MSIE, all with CVE
4 vulnerabilities in Windows Media Player, 1 without CVE
52 vulnerabilities in Office, all with CVE. (based on "Office" being Excel, Powerpoint, Word and Outlook.
92 vulnerabilities in Windows, only 2 without CVE

When dealing with vulnerability numbers and statistics, like anything else, it's all about qualifying your numbers. Saying "Adobe Software" is different than "Adobe Acrobat" or "Adobe Reader" as the software installation base is drastically different. Given the different levels of abstraction in VDBs, it is also equally important to qualify what "a vulnerability" (singular) is. Where CVE/NVD will group several vulnerabilities in one identifier, other databases may abstract and assign unique identifiers to each distinct vulnerability.

Qualys, since you provided the stats to CNet, could you clarify?

Posted in Vulnerability Statistics | Tags cnet, qualys | 4 comments

Search Filters & Custom Exports

Posted by jericho Mon, 09 Nov 2009 19:59:00 GMT

Last week, OSVDB enhanced the search results capability by adding a considerable amount of filter capability, a simple "results by year" graph and export capability. Rather than draft a huge walkthrough, open a search in a new tab and title search for "microsoft windows".

As always, the results will display showing the OSVDB ID, disclosure date and OSVDB title. On the left however, are several new options. First, a summary graph will be displayed showing the number of vulnerabilities by year, based on your search results. Next, you can toggle the displayed fields to add CVE, CVSSv2 score and/or the percent complete. The percent complete refers to the status of the OSVDB entry, and how many fields have been completed. Below that are one click filters that let you further refine your search results by the following criteria:

Reference Type - only show results that contain a given type of reference
Category - show results based on the vulnerability category
Disclosure Year - refine results by limiting to a specific year
CVSS Score - only show entries that are scored in a given range
Percent Complete - filter results based on how complete the OSVDB entry is

Once you have your ideal search results, you can then export them to XML, custom RSS feed or CSV. The export will only work for the first 100 results. If you need a bigger data set to work with, we encourage you to download the database instead.

With the new search capability, you should be able to perform very detailed searches, easily manipulate the results and even import them into another application or presentation. If you have other ideas of how a VDB search can be refined to provide more flexibility and power, contact us!

Posted in Vulnerability Statistics, OSVDB News | Tags export, search | no comments

Who discovered the most vulns?

Posted by jericho Wed, 18 Feb 2009 03:31:00 GMT

This is a question OSVDB moderators, CVE staff and countless other VDB maintainers have asked. Today, Gunter Ollmann with IBM X-Force released his research trying to answer this question. Before you read on, I think this research is excellent. The relatively few criticisms I bring up are not the fault of Ollmann’s research and methodology, but the fault of his VDB of choice (and *every* other VDB) not having a complete data set.

Skimming his list, my first thought was that he was missing someone. Doing a quick search of OSVDB, I see that Lostmon Lords (aka ‘lostmon’) has close to 350 vulnerabilities published. How could the top ten list miss someone like this when his #10 only had 147? Read down to Ollmann’s caveat and there is a valid point, but sketchy wording. The data he is using relies on this information being public. As the caveat says though, "because they were disclosed on non-public lists" implies that the only source he or X-Force are using are mail lists such as Bugtraq and Full-disclosure. Back in the day, that was a pretty reliable source for a very high percentage of vulnerability information. In recent years though, a VDB must look at other sources of information to get a better picture. Web sites such as milw0rm get a steady stream of vulnerability information that is frequently not cross-posted to mail lists. In addition, many researchers (including lostmon) mail their discoveries directly to the VDBs and bypass the public mail lists. If researchers mail a few VDBs and not the rest, it creates a situation where the VDBs must start watching each other. This in turn leads to "VDB inbreeding" that Jake and I mentioned at CanSecWest 2005, which is a necessary evil if you want more data on vulnerabilities.

In May of 2008, OSVDB did the same research Ollmann did and we came up with different results. This was based on data we had available, which is still admittedly very incomplete (always need data manglers.) So who is right? Neither of us. Well, perhaps he is, perhaps we are, but unfortunately we’re both working with incomplete databases. As a matter of my opinion, I believe OSVDB has better coverage of vulnerabilities, while X-Force clearly has better consistency in their data and a fraction of the gaps we do.

Last, this data is interesting as is, but would be really fascinating if it was mixed with ‘researcher confidence’ (a big thing of Steve Christey/CVE and myself), in which we track a researcher’s track record for accuracy in disclosure. Someone that disclosed 500 vulnerabilities last year with a 10% error rate should not be above someone who found 475 with a 0% error rate. In addition, as Ollmann’s caveat says, these are pure numbers and do not factor in hundreds of XSS versus remote code execution in operating system default install services. Having a weight system that can be applied to a vulnerability (e.g., XSS = 3, SQLi = 7, remote code exec = 9) that is then factored into researcher could move beyond "who discovered the most" and perhaps start to answer "who found the most respectable vulnerabilities".

Posted in Vulnerability Disclosure, Vulnerability Statistics | no comments

Top vulnerability researcher?

Posted by jkouns Sat, 24 May 2008 20:16:53 GMT

Who is the top vulnerability researcher? Who has discovered the most computer security vulnerabilities? Which country has the most researchers and publishes the most vulnerabilities? Who has discovered the most critical vulnerabilities?

From looking at OSVDB here are the top 12 researchers in terms of volume:

Rank    Creditee        # Vulns
-----------------------------------------
1)      r0t         770
2)  Lostmon Lords       241
3)      rgod            239
4)      Aliaksandr Hartsuyeu    201
5)      Kacper          199
6)      James Bercegay      180
7)      luny            142
8)      Diabolic Crab       139
9)      Janek Vind "waraxe" 136
10)     JeiAr           117
11)     Dedi Dwianto        86
12)     M.Hasran Addahroni  79

Take a look at the other OSVDB Browse categories and note you can even click on a Creditee’s name and see all of the vulnerabilities that they have discovered here: http://osvdb.org/browse

Of course our statistics are based off of the content in OSVDB and we need your help to provide better statistics. If you are a researcher, it would help if you could take the time to create an OSVDB account and update the vulnerabilities that you have discovered!

You can signup for an OSVDB account here: https://osvdb.org/account/signup

Here is a quick overview:

-Search for your vulnerabilities at http://osvdb.org/search/advsearch -Click on your vuln, then click “Edit Vulnerability” -Click the Credits menu item, if credit is missing click “Toggle Add Author…” -You name may already be in the database, as you type it will search OSVDB to see if your information is there. If so, select and click “Add Author”. -Once you add the creditee information you can update your information or if your name is not there you can add it as a new creditee.

Rinse and repeat!

Posted in Vulnerability Statistics | 3 comments

Vulnerability counts and OSVDB advocacy

Posted by jericho Thu, 03 Apr 2008 21:42:39 GMT

CVE just announced reaching 30,000 identifiers which is a pretty scary thing. CVE staff have a good eye for catching vulnerabilities from sources away from the mainstream (e.g. bugtraq) and they have the advantage of being a very widely accepted standard for tracking vulnerabilities. As companies and researchers request CVE numbers for disclosures, they get a lot of the information handed to them on a silver platter. Of course, sometimes that platter is full of mud and confusion as vendors don’t always provide clear details to help CVE accurately track and distinguish between multiple vulnerabilities. I’ve also pointed out many times in the past that CVE is a very unique VDB that provides identifiers for vulnerability tracking. They do not provide many fields associated with other VDBs (solution, creditee, etc). As such, they may have a single entry that covers multiple distinct vulnerabilities if they are the same class (XSS, SQLi, RFI), or if there is a lack of details but they know it affects the same product (Oracle). So when we see 30,000 identifiers, we have to realize that the real count of vulnerabilities is significantly higher.

CVE is run by The MITRE Corporation, sponsored / funded by the NCSD (US-CERT) of DHS under government contract. That means our tax dollars fund this database so it should be of particular interest to U.S. taxpayers in the security industry. I know from past discussions with CVE staff and other industry veterans that on any given day, they are more likely to have more work than available staff. That means the rate of vulnerabilities that get published is greater than the resources CVE can maintain to track them. In short, the 30,000 identifiers you see only represents a percentage of the vulnerabilities actually disclosed. We could probably debate what percentage that represents all day long, and I don’t think that is really the point here other than “we know it isn’t all of them”.

Every VDB suffers from the same thing. “Commercial” VDBs like X-Force, BID and Secunia have a full time staff that maintain their databases, like CVE does. Despite having all of these teams (some of them consisting of 10 or more people) maintain VDBs, we still see countless vulnerabilities that are ‘missed’ by all of them. This is not a slight against them in any way; it is a simple manner of resources available and the amount of information out there. Even with a large team sorting disclosed vulnerabilities, some teams spend time validating the findings before adding them to the database (Secunia), which is an incredible benefit for their customers. There is also a long standing parasitic nature to VDBs, with each of them watching the others as best they can, to help ensure they are tracking all the vulnerabilities they can. For example, OSVDB keeps a close eye on Secunia and CVE specifically, and as time permits we look to X-Force, BID, SecurityTracker and others. Each VDB tends to have some researchers that exclusively disclose vulnerabilities directly to the VDB of their choice. So each one I mention above will get word of vulnerabilities that the rest really have no way of knowing about short of watching each other like this. This VDB inbreeding (I will explain the choice of word some other time) is an accepted practice and I have touched on this in the past (CanSecWest 2005).

Due to the inbreeding and OSVDB’s ability to watch other resources, it occasionally frees up our moderators to go looking for more vulnerability information that wasn’t published in the mainstream. This usually involves grueling crawls through vendor knowledge-bases, mind-numbing changelogs, searching CVS type repositories and more. That leads to the point of this lengthy post. In doing this research, we begin to see how many more vulnerabilities are out there in the software we use, that escapes the VDBs most of the time. Only now, after four years and getting an incredible developer to make many aspects of the OSVDB wish-list a reality, do we finally begin to see all of this. As I have whined about for those four years, VDBs need to evolve and move beyond this purely “mainstream reactionary” model. Meaning, we have to stop watching the half dozen usual spots for new vulnerability information, creating our entries, rinsing and repeating. There is a lot more information out there just waiting to be read and added.

In the past few weeks, largely due to the ability to free up time due to the VDB inbreeding mentioned above, we’ve been able to dig into a few products more thoroughly. These examples are not meant to pick on any product / VDB or imply anything other than what is said above. In fact, this type of research is only possible because the other VDBs are doing a good job tracking the mainstream sources, and because some vendors publish full changelogs and don’t try to hide security related fixes. Kudos to all of them.

Example: Search your favorite VDB for ”inspircd”, a popular multi-platform IRC daemon. Compare the results of BID, Secunia, X-Force, SecurityTracker and CVE. Compare these results to OSVDB after digging into their changelogs. Do these same searches for “xfce” (10 OSVDB, 5 max elsewhere), “safesquid” (6 OSVDB, 1 max elsewhere), “beehive forum” (27 OSVDB, 8 max elsewhere) and “jetty” (25 OSVDB, 12 max elsewhere). Let me emphasize, I did not specifically hand pick these examples to put down any VDB, these are some of the products we’ve investigated in the last few weeks.

The real point here is that no matter what vulnerability disclosure statistic you read, regardless of which VDB it uses (including OSVDB), consider that the real number of vulnerabilities disclosed is likely much higher than any of us know or have documented. As always, if you see vulnerabilities in a vendor KB or changelog, and can’t find it in your favorite VDB, let them know. We all maintain e-mail addresses for submissions and we all strive to be as complete as possible.

Posted in Vulnerability Disclosure, Vulnerability Statistics, Vulnerability Databases | 1 comment

2007 Top Vulnerable Vendors?

Posted by jericho Wed, 19 Sep 2007 13:45:13 GMT

http://www.eweek.com/article2/0,1895,2184206,00.asp

New IBM research shows that five vendors are responsible for 12.6 percent of all disclosed vulnerabilities. Not surprising: In the first half of 2007, Microsoft was the top vendor when it came to publicly disclosed vulnerabilities. Likely surprising to some: Apple got second place. IBM Internet Security Systems’ X-Force R&D team released its 2007 report on cyber attacks on Sept. 17, revealing that the top five vulnerable vendors accounted for 12.6 of all disclosed vulnerabilities in the first half of the yearor 411 of 3,272 vulnerabilities disclosed. Here’s the order in which the top 10 vendors stacked up, by percentage of vulnerabilities publicly disclosed in the first half of the year: Microsoft, 4.2 percent Apple, 3 percent Oracle, 2 percent Cisco Systems, 1.9 percent Sun Microsystems, 1.5 percent IBM, 1.3 percent Mozilla, 1.3 percent XOOPS, 1.2 percent BEA, 1.1 percent Linux kernel, 0.9 percent

This article was posted to ISN the other day and struck a nerve. How many times are we going to see vulnerability statistics presented without qualification? Rather than really get into the details, I replied with a single simple example on why such statistics are misleading at best and incorrect at worst. The bulk of my reply follows. My hopes for Lisa or IBM/ISS clarifying this is already dwindling.

One other factor, that Lisa Vaas apparently didn’t ask about, is how ISS X-Force catalogs vulnerabilities, and if their method and standards could impact these numbers at all. Take for example, two X-Force vulnerability database entries: Oracle Critical Patch Update - July 2007 http://xforce.iss.net/xforce/xfdb/35490 18 CVE, 30+ Oracle Oracle Critical Patch Update - January 2007 http://xforce.iss.net/xforce/xfdb/31541 30 CVE, 50+ Oracle So when comparing numbers, you have 2 X-Force entries that equate to 48 CVE entries that equate to *more than 80* unique and distinct vulnerabilities according to Oracle. I’m not a math or stat guy, but I have a feeling that this could seriously skew the statistics above, especially when you consider that Microsoft and Apple both have a more distinct breakdown and separation in the X-Force database. Anyone from IBM/ISS care to clarify? Lisa, did you have more extensive notes on this aspect that didn’t make it in the article perhaps?

Posted in Vulnerability Statistics | no comments

Scrubbing the Source Data

Posted by jericho Mon, 16 Jul 2007 05:56:40 GMT

A few months ago, Jeff Jones at CSO Online blogged about “Scrubbing the Source Data”, talking about the challenges of using vulnerability data for analysis. Part 1 examined using the National Vulnerability Database (NVD) showing how you can’t blindly rely on the data from VDBs. In his examples he shows that using the data to examine Windows is probably fairly accurate, yet examining Apple is less so and Ubuntu Linux is basically not possible. Unfortunately, there isn’t a part two to the series (yet) as implied by the title and introduction. Jones concludes the post:

Given these accuracy levels for vulnerabilities after the vendor has acknowledged it and provided a fix, it doesn’t seem like too much of a stretch to also conclude that using this data to analyze unpatched data would be equally challenging. Finally, I think this exercise helps demonstrate that anyone leveraging public data sources needs to have a good understanding of both the strengths and the weaknesses that any given data source may have, with respect to what one is trying to analyze or measure, and include steps in their methodology that accomodates accordingly.

Posted in Vulnerability Statistics | no comments

OS Security, Old Debate, New Info

Posted by jericho Thu, 29 Mar 2007 18:42:21 GMT

Check out this article/report by OmniNerd, which tested various operating systems for security. They performed a base line vulnerability scan during installation, after installation and after patches had been applied. Each installation was done to mimick as close to a ‘default install’ by clicking ‘next’ when possible. While one can argue various points of this test, they did a good job defining the operating system, configuration and resulting open ports, along with corresponding vulnerabilities. The only questions that immediately come to mind are if the Solaris install included Update 3 and why they didn’t have any charts or graphs summarizing the information.

This is hands down one of the most fair and unbiased tests I have seen in a while, based on the information in the article.

Posted in Vulnerability Statistics, General Security | no comments

« Previous 1 2 3 Next »

February Update: OSVDB Winter 2010 Fundraising Goal

Adobe, Qualys, CVE and Math

Search Filters & Custom Exports

More powerful searches, by looking at what's NOT there..

Who discovered the most vulns?

Top vulnerability researcher?

Vulnerability counts and OSVDB advocacy

2007 Top Vulnerable Vendors?

Scrubbing the Source Data

OS Security, Old Debate, New Info

Categories

Archives

Syndicate