I do not think it means what you think it means…

Sometime in the past day or so, CVE-2016-10001 was publicly disclosed, and possibly a duplicate. Regardless, CVE-2016-10002 is also now public and legitimate. Tonight, I Tweeted that the presence of those IDs doesn’t mean what many will think it means. I say that based on the past experience, both historical and more recent. Even 17 years later, many people believe that CVE assignments are sequential and that a given ID means that is the number of vulnerabilities aggregated by MITRE that year. That isn’t how it works and it never has.

As of the December 18 dump available from MITRE, there are 10,137 identifiers in the dump. However, 44 of them are REJECTED and 4,760 are in RESERVED status. That means there are 5,333 live CVE identifiers at this time that correspond to vulnerabilities. Since a single CVE ID can include multiple similar vulnerabilities, that number is also misleading. If you take their data and abstract it on a per-vulnerability basis, they cover 8,058 issues as aggregated by Risk Based Security’s VulnDB. So, to be very clear:

CVE has not cataloged 10,000 vulnerabilities in 2016 based on CVE IDs.

Additionally, to be very clear again:

CVE has not cataloged 10,000 vulnerabilities in 2016 based on their actual aggregated vulnerability data.

Meanwhile, VulnDB has currently cataloged 14,485 vulnerabilities, compared to the CVE 8,058 actual number. Hopefully your organization uses more than just CVE data. That means within your security products that scan for vulnerabilities, your tools that collect the data, and ultimately the reporting that guides your security team in making decisions.

All of that said… taking bets if we see Tweets, blogs, or news articles claiming the “10,000 vulns in 2016” notion.

Advertisements

3 responses

  1. I know you’ve answered questions in the past on this very topic and the (basically) misleading nature of it all, but what’s the point? What is the point of cataloguing this stuff without clarity? Seems like a recipe for disaster and, quite frankly, counter productive.

    Is the act of cataloguing still evolving? Or is it a lack of care/interest to do it right? From what I understand (again, from listening to your past comments on the topic) this is really just scratching the surface of the problems.

    As we know, a huge component of security is awareness. And if the layman’s awareness comes from the media, movies, “experienced” journalists, etc., no wonder there’s such a misunderstood and misapplied understanding of security. At that point, it’s practically cultural in nature, which is much more difficult to course correct or change.

    And don’t get me started on the manufacturers and the link between lawsuits versus proper engineering (cheaper to pay out than make a solid product). Frustration abounds.

    1. That is one of the biggest points I have tried to make, is that any statistics around vulnerabilities must be disclaimed or explained. Since many companies love doing third-party analysis of the CVE/NVD data set, many tend to forget that while they are ‘security experts’, they are not ‘VDB experts’. Running a VDB is simple on the surface, but so is any other task or job. There are an incredible amount of nuances to it.

      Personally, I believe it is still evolving for at least a couple people (Carsten Eiram and myself), and we tend to have some pretty pedantic, but interesting talks about VDB refinement, constantly revisiting standards for inclusion, debating one-off disclosures, and a world of other minutiae. I like to believe that OSVDB evolved the discipline for years. I know that a few of us are still pushing MITRE heavily to do better, but that often seems to fall on deaf ears (more blogs on that in the future, promise). Most of the other commercial VDBs are actually devolving. In 2014, Secunia announce they would be covering less sources, and that statement was buried in their yearly report with dreadful ‘stats’. Symantec/SecurityFocus/BID doesn’t seem to have evolved in 15 years and haven’t really improved their coverage. There are a couple other private VDBs out there that I haven’t seen personally, but ex-customers say they aren’t as advertised and they left in disappointment.

      Regarding awareness, that is why I tend to shout in the ether about vuln stats. All it takes is some random company that doesn’t truly understand the data set to produce really bad analysis, and worse, make speculations on future trends, to prompt a more mainstream news article. And if enough security journalists cover the bad analysis, it may get noticed by increasingly bigger publications with more readership. Next think you know, absurd claims with absolutely no foundation are being passed around and accepted as facts. I’ve said it many times; if you don’t personally aggregate and maintain a data set of vulnerabilities, you likely aren’t qualified to speak on the matter. It isn’t just about parsing a CSV and crunching some basic numbers. The fact that so many don’t understand what RESERVED means in CVE is telling, yet it completely skews a lot of ‘statistics’ and perception of the data.

      1. It’s a lot like health studies. When you really get into it, you see how many people skew results for one fad or the next to make money. People jump on the train, websites spring up left and right, paid for services, gurus offer advice, and so forth.

        It’s an age old problem that plagues many industries and areas of life. So, given that it’s not unique to InfoSec, I struggle to see how we can fix it. We can be industry police and call out those who mishandle the data…but you’d need a full time army to keep up with all the “experts” out there.

        We can hold one another accountable, which will help, but it won’t stop people from misinterpreting data to fit their needs/goals. It’s such a fundamental problem, but it has huge implications. Fascinating topic.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: