Mail List Archives 101 (or why SF hates VDBs)
Posted by jericho
Running a mail list archive is a straight forward task. Collect, organize and make mail list posts available via the web. You can see such archives at seclists.org or the Neohapsis arhive. Most folks that use archives like this have their favorites for various reasons. Speed, the lists they archive, or the organization usually. Most archives use a system where the URL to the post is logical and somewhat informative. Looking at the URL to the latest Bugtraq post archived at Neohapsis: http://archives.neohapsis.com/archives/bugtraq/2005-10/0259.html. We see the mail list name, the year and month, and a unique number for the post. Bugtraq, 2005, October, 259th post. Simple and easy!
SecurityFocus maintains an archive of the mail lists they run. Until a couple months ago, they used a scheme that wasn’t very informative. A sample URL http://www.securityfocus.com/archive/1/245152 shows it doesn’t help us discern anything about the post. Annoying, but oh well, most people could live with it. A month or two ago, SecurityFocus decided to revamp their system which would also impact their entire archive. They made assurances that the changes would be transparent and that old URLs would work. As I predicted to one SF employee, it didn’t work out so smooth, and many of the old URLs did not translate properly. At first he doubted me, then asked for examples. After providing half a dozen he saw that it wasn’t a fluke and that something had gone wrong. Unfortunately, whoever he shared that information with didn’t act on it. What is more annoying, and more damning, is that SF implemented a new scheme that is just as bad as the old one. Look at an example of their new scheme: http://www.securityfocus.com/archive/1/414100/30/0/threaded. This URL doesn’t tell us anything about the mail list or post either.
Why does this matter? There are hundreds of Nessus plugins that reference these old URLs, and in some cases only reference mail list posts, via the SecurityFocus archive. Clicking these now leads to .. no information. There are also countless CVE entries that reference the old URL scheme. If you want to see the original point of disclosure, you are forced to visit another database (that competes with SecurityFocus) such as ISS X-Force or OSVDB to see a valid link, as they choose to reference mail list archives that are more friendly to users.
In short, if you maintain a security product or database, please do not reference SecurityFocus or any other archive that uses an obscured scheme, or has intentions of changing their scheme.
Shortly after posting this, Conrad Schilbe from SecurityFocus contacted me. I appreciate the fast response and consideration of my comments.
From: Conrad Schilbe Date: Wed, 26 Oct 2005 14:17:56 -0600 Subject: SecurityFocus mailing lists
Jericho,
I’d like to thank you for the information provided in your recent blog entry discussing the SecurityFocus mailing lists. While we strived to ensure availability of all data through the wide-spread changes implemented earlier this year, it appears some mailing list archive data did not survive the trip. I assure you that the standard schema for linking has remained the same. Where it differs is only in regards to additional formatting functionality.
I will consider your suggestions for a more useful linking schema, while preserving the existing standard, and will investigate the incidents of missing entries.
Please feel free to contact me with any further questions or comments.
Regards,
Conrad Schilbe Software Engineer SecurityFocus
Unfortunately you’re probably screwed no matter what. Mass reorgs seem to happen on a regular basis, and few archives seem to be immune - think about all the URLs that suddenly went away when Full-Disclosure was picked up by Secunia. URL portability is probably doable with the message ID, but of course it’s not overly usable.
While working through the NetBSD Advisory Archive, I noticed that six advisories referenced SecurityFocus archive URLs (2001-004,2002-003,2003-008,2003-015,2003-017,2004-001). Of those, two (2001-004,2002-003) of the SF archive links did not work, and the other four did.
Working through the VMWare knowledge base (full of references to published vulnerabilities), I found that they too reference some of the old SF archive URLs.
http://www.vmware.com/support/kb/enduser/stdadp.php?psid=dsxk*BWh&plva=&pfaqid=787
While using SPI Dynamics’ WebInspect for an audit, I noticed that their vulnerability ID 3457 links to http://online.securityfocus.com/archive/1/281199 which no longer displays information. Google has now cached the blank page, further making it difficult to find.
Almost 2.5 years later, and we have run into countless SF archive URLs that have broken. Many Nessus plugins from 2002 contain references to the SF archive and no other VDB ref. Since posts were made to Bugtraq, NTBugtraq, F-D, Vulnwatch and Vulndev, it’s a guessing game as to which list had the material being referenced.
SF has been told about dozens of these URLs, and none have been fixed.