WhiteSource on ‘Open Source Vulnerability Databases’ – Errata

On September 8, 2016, Jason Levy of WhiteSource Software published a blog titled “Open Source Vulnerability Database”. Almost two years later it came across my radar and I asked via Twitter if WhiteSource was interested in getting feedback on the blog, since it contained errata. They never replied and it fell off my radar. Last week, someone asked me about WhiteSource’s claims about maintaining the largest vulnerability database. While I have not seen it, I expressed doubt and dug into my notes. Until then, I had forgotten that I created notes to write a blog listing the errata. Since their name came up again, I figured it is worth revisiting this, especially since they never replied to me.

First, I was a long-term contributor to OSVDB, an officer in OSF (the 501(c)(3) that managed the project), a founder at Risk Based Security, and one of the maintainers of VulnDB since day one. Don’t worry, all of that is explained in more detail below. Here are quotes from their blog and my thoughts on the errata:

The open source community has been living up to this statement recently, with the accelerated rate of discoveries of open source vulnerabilities reported by such sources as the NVD open source vulnerability database.

CVE/NVD have dismal coverage of open source libraries in the big picture. Their coverage is primarily through a couple dozen projects that seek out CVE IDs, researchers that request CVE IDs, and the OSS-Sec mail list. To date, there are still many developers, projects, and even security researchers that don’t know what CVE is.

The NVD is by far the main source for researching vulnerabilities.

This sentence does not make sense. As you note below, NVD does not research vulnerabilities; they don’t even aggregate vulnerabilities themselves. While many consume NVD’s data over CVE, due to the format they make the data available, more “analysis” is done via CVEDetails.com than NVD in my experience.

Their analysis contains information like how the vulnerability operates, its impact rating, CVSS score, and links to any available patches/fixes.

The main purpose of NVD, aside from a burden on taxpayer’s dollars, is to provide CPE and CVSS scores for vulnerabilities via CVE. They rarely, if ever, do additional analysis of the vulnerability unless required to determine a CVSS score. If they do, they do not “show their work” so to speak, instead using the same CVE description. As best I am aware, they don’t dig up additional references either, so the only solution links provided are via CVE. That said, I haven’t audited NVD in several years to see if that still holds true, but I would be surprised if anything has changed.

The OSVDB (open source vulnerability database) was launched in 2004 by Jake Kouhns, the founder and current CISO of Risk Based Security – the company which now operates OSVDB’s commercial version, the VulnDB.

Jake Kouns, one of the founders of Risk Based Security (RBS) did not launch OSVDB. And that is the Open Sourced Vulnerability Database. OSVDB was created and launched by H.D. Moore. More history is available via Wikipedia. While OSVDB was a basis for the historical data in VulnDB, Risk Based Security funded OSVDB entirely for over two years and it was their data that was shared publicly via OSVDB before the project shut down. When OSVDB shut down, RBS continued to maintain VulnDB which contains over eight years of vulnerability information not found in OSVDB. With considerable resources and a team maintaining VulnDB, it has become something OSVDB wished it could, but never came close.

This commercial version continues to be maintained by Risk Based Security, but without the strong community that maintained the OSVDB.

There was no “strong community” that maintained OSVDB. For years, it limped along on the backs of a handful of volunteers, with very little support from the community; financial or labor. At times, there were only two of us maintaining the project while most volunteers signed up, worked on the database for 20 minutes, and never returned to help again.

The OSVDB/VulnDB is perceived by many in the community as redundant, for the majority of vulnerabilities are also reported in the NVD.

This is WhiteSource’s ignorance or bias at play. OSVDB was known for having tens of thousands of vulnerabilities not found in CVE/NVD. One of VulnDBs strong points is that it contains almost 69,000 vulnerabilities not found in CVE/NVD. Since WhiteSource claims to have a comprehensive database, you should have disclaimed this while spreading false information about VulnDB. Since VulnDB is not public, the notion that is perceived by “many” in such a light is just your marketing. Anyone reading VulnDBs quarterly or year-end reports can see the incredible gap between them and CVE/NVD. Also curious is that the link off “monitoring of multiple vulnerability databases including the CVE/NVD” just links to your blog on the topic. Other than CVE/NVD, which vulnerability databases do you monitor exactly?

I’d also happily sign a non-disclosure agreement to do a quick audit of your database, to verify or refute your claim that you have the “the largest coverage of vulnerability listings“. I’d only ask that in return, I can publish a simple ‘yes’ or ‘no’ answer to that claim.

Security advisories are usually the first place that security professionals and developers look when they have security issues within a specific scope.

If this is WhiteSource’s methodology for maintaining a vulnerability database, it does not bode well for your customers. Formal security advisories are quickly becoming, or have already become, a minority in the disclosure game. OSVDB and VulnDB didn’t stay far ahead of the curve using this antiquated mindset. We knew to look elsewhere all along.

WhiteSource tracks almost a dozen security advisories (like RubyOnRails, RubySec and Node Security), to ensure our customers are alerted on new vulnerabilities the minute they’re reported.

A whole dozen? The VulnDB team currently monitors 3,100 sources every week. Some are monitored ~ hourly, some a few times a day, some once a week depending on various criteria such as publication frequency. If WhiteSource is only looking at those sources, and that type of source, it really doesn’t bode well for your claims of the largest vulnerability listing.

The majority of vulnerabilities are published with a remediation suggestion, like a new patch, new version, system configuration suggestion etc.

While technically true, per VulnDB data, it is better to qualify that. For example, in Q1 2019, 60.8% of disclosed vulnerabilities they aggregated had a documented solution. But, a couple caveats. First, that was down 13.5% over Q1 2018! Second, not all solutions are created equal. Just because there is a patch committed against the ‘master’ branch doesn’t mean an organization can actually use it. Their policy or available resources may not allow for one-off code patching like that.

And here at WhiteSource, this is exactly what we do.

Unfortunately, based on the way you describe what you do, you are not much better than CVE/NVD I’m afraid.

Advertisements

3 responses

  1. This might be a bit tangential, but it’s something I’ve been thinking about lately: what’s your take on Tenable’s VPR approach (https://www.tenable.com/sc-dashboards/vulnerability-priority-rating-vpr-summary)?

    I had management wanting to jump on board, both feet, eyes closed almost immediately.

    For me, being the lead of a vulnerability management group, I was much more hesitant, even reserved. Some of it has to do with the state of CVEs and vuln tracking/disclosures already, but there was also the ever present problem of context; VPR, like CVSS, assumes the worst and does not provide the context of your environment, deployment, vendor factors, etc.

    The biggest difference for me was that at least CVSS is open in showing how it’s scores are calculated. Therefore, if you pull those levers and provide evidence, you can contextual the scoring. VPR? Nada. They’ll just tell you they monitor thousands of sources, include CVSS scoring, and have a bunch of “really smart people” in a room working on this.

    It seems like this could lead to another industry wide problem (if other vendors follow suit, assuming they’re not already) by controlling the narrative.

    There’s also the very large problem of VPR vs CVSS2 vs CVSS3 scoring in that not every plugin (ala Tenable), and therefore CVE, has consistent scoring across these 3 systems. We had to define our own logic to cover all the various cases of whether a plugin had this, but not that, or this and that, or nothing, etc.

    1. Disclaimer: I (jericho) worked at Tenable for eight years, but left a couple years before they released this new VPR.

      You key in on the biggest issue in my eyes too; the fact that the scoring isn’t open and you can’t see the calculations. While CVSS (both v2 and v3) have serious shortcomings, the ability to use Temporal and Environmental scoring, as well as the ability to re-cast the score (which SecurityCenter has allowed for ages) seems like a better approach. If management came to me asking why I did or did not prioritize a given vulnerability, and all I can point to is a pretty dashboard and arbitrary number, that doesn’t work for me.

      Tenable plugins, unless their process has changed, get their CVSS scoring from the Plugin Writing team which should use a consistent approach to scoring. While I was there, plugins would also go through at least two sets of eyes, and toward the end of my time, three sets of eyes. That would typically help iron out any CVSS mistakes giving the plugins a fairly uniform basis. That said, the Nessus plugins only cover a small subset of CVE, and an even smaller subset of VulnDB, so their scoring is skewed toward the vulnerabilities they can write plugins for.

      1. Yup. You can recast, which is nice. We have a justification process to ask questions, gather evidence, then store in a Jira ticket. The results of that ticket are then recast back into SecurityCenter/Tenable.sc as appropriate, and it helps weed out a lot of noise.

        The issue is, when we tried to incorporate VPR into our scoring system, I literally had every case present (we also recently switched to CVSS3 for compliance requirements – I think FedRAMP or PCI specifically, but I can’t remember).

        Anyway, we had:
        CVSS2, no CVSS3, no VPR
        CVSS2, CVSS3, no VPR
        CVSS2, CVSS3, VPR
        no CVSS2, CVSS3, VPR
        no CVSS2, no CVSS3, VPR
        no CVSS2, no CVSS3, no VPR

        I mean, all over the map. I did a break down by percentage of each to show my boss and he just stared blankly for a few seconds. So, we wrote in logic to prefer CVSS3 over v2, and if VPR present, to weight towards VPR, but still consider CVSS. That was the compromise.

        We have to do better than this, but I honestly don’t know how. It’s like…we need to go back to the beginning, gut EVERYTHING, then have some really honest, direct conversations to fix this…we’re drowning.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: