Via Twitter, blogs, or talking with our people, you may have heard us mention the ‘scraping’ problem we have. In short, individuals and companies are using automated methods to harvest (or ‘scrape’) our data. They do it via a wide variety of methods but most boil down to a couple methods involving a stupid amount of requests made to our web server.
This is bad for everyone, including you. First, it grinds our poor server to a stand-still at times, even after several upgrades to larger hosting plans with more resources. Second, it violates our license as many of these people scraping our data are using it in a commercial capacity without returning anything to the project. Third, it forces us to remove functionality that you liked and may have been using in an acceptable manner. Over the years we’ve had to limit the API, restrict the information / tools you see unauthenticated (e.g. RSS feed, ‘browse’, ‘advanced search’), and implement additional protections to stop the scraping.
So just how bad is it? We enabled some CloudFlare protection mechanisms a few weeks back and then looked at the logs.
- The attacks against OSVDB.org were so numerous, the logs being generated by CloudFlare were too big to be managed by their customer dashboard application. They quickly fixed that problem, which is great. Apparently they hadn’t run into this before, even for the HUGE sites getting DDoS’d. Think about it.
- We were hit by requests with no user agent (a sign of someone scraping us via automated means) 1,060,599 times in a matter of days…
- We got hit by 1,843,180 SQL injection attack attempts, trying to dump our entire database in a matter of weeks…
- We got hit by ‘generic’ web app attacks only 688,803 times in a matter of weeks….
- In the two-hour period of us chatting about the new protection mechanisms and looking at logs, we had an additional ~ 130,000 requests with no user-agent.
To put that in perspective, DatalossDB was hit only 218 times in the same time period by requests with no user agent. We want to be open and want to help everyone with security information. But we also need for them to play by the rules.
Researcher Security Advisory Writing Guidelines
Open Security Foundation / OSVDB.org
moderators at osvdb.org
This document has been prepared by the Open Security Foundation (OSF) to assist security researchers in working with vendors and creating advisories. Security advisories help convey important information to the community, regardless of your goals or intentions. While you may have an intended audience in mind as you write an advisory, they will not be the only ones to read it. There is a lot of information that can be included in a properly written advisory, and leaving any out makes your advisory something less than it could be.
The OSF encourages researchers to use this document as a guideline for writing security advisories. We will focus on the content of the advisory, not the style. While there is a logical order of presentation, what ultimately matters is including the necessary information, though some things are most beneficial at the start of an advisory. Remember; more information is better, and including information for other parties ultimately helps more people.
How you disclose a vulnerability is your choice. The debate about “responsible” or “coordinated” disclosure has raged for over two decades. There is no universal accord on what is an appropriate period of time for a vendor to reply to a vulnerability report, or fix the issue, though it is generally agreed that it is at the least more than a day and less than a year. Researchers, we fully encourage you to work with vendors and coordinate disclosure if possible; your goal is to improve security after all, right? The following material will give you additional information and considerations for this process.
Brian Martin & Daniel Moeller
At a glance, it may appear as if the OSVDB project has fallen by the wayside. Some of our public facing pages have not been updated in several years, the last string of blog posts was over a year ago, and a recent update caused a few functions to fail (e.g., data exports). On the other hand, anyone paying attention to the data has noticed we are certainly present and moving forward. We have had one person working full time on OSVDB for over a year now. He is responsible for the daily push of new vulnerabilities and is scouring additional sources for vulnerabilities that didn’t appear through the normal channels. Given the nature of the project, we place data completeness and integrity as the top priority.
The OSVDB project is coming up on its tenth year anniversary. The last ten years have seen some big changes, as well as many things that have not changed one bit. The biggest thing that hasn’t changed is the lack of support we receive from the community. The top ten all time contributors are the core members of OSF, the handful of longstanding dedicated volunteers we have had over the years, or some people we have been able to pay to help work on the project. Beyond those ten people, the volunteer support we lobbied for years never materialized. We still enjoy a couple dozen volunteers that primarily mangle their own disclosures, or add CVE references, which we appreciate greatly. Unfortunately, the rate of vulnerability disclosures demands a lot more time and attention. In addition to the lack of volunteers, community support in the form of sponsorship and donations has been minimal at best. Tenable Network Security and Layered Technologies have been with us for many years and have largely been responsible for our ability to keep up with the incoming data.
Other than those two generous companies, we have had a few other sponsors/donations over the years but nothing consistent. In the last year, we have spent most of our time trying to convince companies that are using our data in violation of our posted license to come clean and support our project. In a few cases, these companies have have built full products and services that are entirely based on our data. In other cases, companies use our data for presentations, marketing, customer reports, and more while trying to sell their products and services. Regardless, the one thing they aren’t doing is supporting the project by helping to update data, properly licensing the data or at least throwing us a few bucks as an apology. In short, several security companies, both new and well established, that sell integrity in one form or another, appear to have little integrity of their own. After a recent server upgrade broke our data export functionality, it was amazing to see the number of companies that came out of the woodwork complaining about the lack of exports. Some of them were presumptuous and demanding, as if it is a Constitutional right to have unfettered access to our data. Because of these mails, and because none of these companies want to license our data, we are in no hurry to fix the data exports. In short, they don’t get to profit heavily off the work of our small group of volunteers, many of whom are no longer with us.
Even as an officer of OSF and data manager of OSVDB, I honestly couldn’t tell you how we have survived this long as a project. I can tell you that it involved a lot of personal time, limping along, and the hardcore dedication of less than a dozen individuals over ten years that made it happen. With almost no income and no swarm of volunteers, the project simply isn’t sustainable moving forward, while still maintaining our high standards for data quality. We gave the community ten years to adopt us, and many did. Unfortunately, they largely did it in a completely self serving manner that did not contribute back to the project. That will be ending shortly. In the coming months, there will be big changes to the project as we are forced to shift to a model that allows us to not only make the project sustainable, but push for the evolution we have been preaching about for years. This will involve making the project less open in some aspects, such as our data exports, and has required us to seek a partnership to financially support our efforts.
For ten years we have had a passion for making OSVDB work in an open and free manner. Unfortunately, the rest of the community did not have the same passion and these changes have become a necessity. The upside to all of this is that our recent partnership has allowed us to develop and we will be offering a subscription data feed that has better vulnerability coverage than other solutions, at a considerably better price point. That said, the data will remain open via HTTP and for a 99% of our users this is all that is required. When exports are fixed, we will offer a free export to support the community, but approval will be required and it will contain a limited set of fields for each entry. We are still working out the details and considering a variety of ideas to better support a wide range of interest in the project, but doing so in a sustainable manner. In the end, our new model will help us greatly improve the data we make available, free or otherwise and ensure OSVDB is around for the next 10 years.
As security vulnerabilities and data loss incidents become a regular occurrence, the Open Security Foundation has grown from supporting a single project in 2004 to a leading provider of filtering through security information and providing notifications and aggregation for data for data loss and cloud security incidents.
The Open Security Foundation has evolved into one of the most utilized resources in providing security information, and as a 501c3 non-profit organization relies heavily on public contributions, volunteer effort and corporate sponsorships.
The growing demand for information to provide proper risk management has led to additional projects and now the introduction of an advisory board consisting of industry professionals to lend their expertise in areas to keep OSF moving in a positive direction and to be the first line of access to all that require their service.
Open Security Foundation CEO and founder Jake Kouns stated, “This is a very important step in shaping the future of the Open Security Foundation.” “OSF has reached a point in growth that requires a strategic move to provide longevity and sustainability. It has always been a goal of this organization to provide our work to the broadest audience and the introduction of the advisory board will contribute to that objective. I am extremely proud to be part of such an amazing organization that has built a reputation of excellence and serves a very important function,” adds Kouns. “We put out a call for qualified individuals that could provide guidance and insight to keep OSF a leader in the security information arena. The results of our search far exceeded our highest expectations; it’s not only provides us with confidence in our direction, but the impact OSF has had on the industry.”
The new advisory board members comprises of an array of specific industries that understand the importance of OSF resources. Each member was chosen for a specific contribution to ultimately achieve the objective and mission of this foundation and capable of providing broad based perspective on information security, business management and fundraising.
Tom Srail, Senior VP Willis Group provides 19 years of experience in the insurance industry with an expertise in risk consulting, professional liabilities, network security risks, intellectual property and technology professional risks.
Shawn Andreas, VP Marketing Guard Dog Inc. (GRDO.PK) will contribute his 20 years of experience in marketing and brand awareness to remake OSF to be more consumer and market friendly focusing on fundraising and sponsorships opportunities. His expertise in marketing spans over diverse markets and includes opportunities working with some of the country’s top companies including GM, Apple, Viacom and more.
Jim Hietala VP, Security for a leading IT standards organization, manages all security and risk management programs. Mr. Hietala is a frequent speaker at industry conferences. In addition he has published numerous articles on information security, risk management and compliance topics.
Daniel E. Geer, Jr. Sc.D. Chief Information security officer In-Q-Tel Washington. Mr. Geer has a list of accomplishments including participation in government advisory roles for the Federal Trade Commission, the Departments of Justice and Treasury, the National Academy of Sciences, the National Science Foundation, the US Secret Service, the Department of Homeland Security, and the Commonwealth of Massachusetts.
Andrew Lewman, Executive Director The Tor Project, Inc. Andrew Lewman is the Executive Director of The Tor Project, a non-profit organization. Mr. Lewman worked on projects with the National Science Foundation, Internews Network, Freedom House, Google, Broadcasting Board of Governors, National Network to End Domestic Violence, and the US State Department.
In addition to the advisory board, OSF also announces new leadership positions with the organization. We are pleased to announce that Becky Chickering and Corey Quinn are now curators for the DataLossDB project. We want to thank everyone that contacted OSF to volunteer their time and skills for the advisory board and flexibility as we went through this process. During our conversations with potential members we spoke with several passionate individuals that have a great deal to offer OSF. We plan to continue to expand our leadership team and are always looking for volunteers to help the organization.
The Open Security Foundation, providing independent, accurate, detailed, current, and unbiased security information to professionals around the world, announced today that it has launched Cloutage (cloutage.org) that will bring enhanced visibility and transparency to Cloud security. The name Cloutage comes from a play on two words, Cloud and Outage, that combine to describe what the new website offers: a destination for organizations to learn about cloud security issues as well as a complete list of any problems around the globe among cloud service providers.
The new website is aimed at empowering organizations by providing cloud security knowledge and resources so that they may properly assess information security risks related to the cloud. Cloutage documents known and reported incidents with cloud services while also providing a one-stop shop for cloud security news and resources.
“When speaking with individuals about the cloud, to this point it has been a very emotional conversation. People either love or hate the cloud,” says Jake Kouns, Chairman, Open Security Foundation. “Our goal with Cloutage is to bring grounded data and facts to the conversation so we can have more meaningful discussions about the risks and how to improve cloud security controls.”
Cloutage captures data about incidents affecting cloud services in several forms including vulnerabilities that affect the confidentiality and integrity of customer data, automatic update failures, data loss, hacks and outages that impact service availability. Data is acquired from verifiable media resources and is also open for community participation based on anonymous user submissions. Cloud solution providers are listed on the website and the community can provide comments and ratings based on their experiences. Cloutage also features an extensive news service, mailing lists and links to organizations focused on the secure advancement of cloud computing.
“The nebulous world of cloud computing and the security concerns associated with it confuses many people, even IT and security professionals,” says Patrick McDonald, a volunteer on the Cloutage project. “We want a clearinghouse of information that provides a clear picture of the cloud security issues.”
The Open Security Foundation (OSF) is an internationally recognized 501(c)(3) non-profit public organization seeking senior leaders capable of providing broad-based perspective on information security, business management and fundraising to volunteer for an Advisory Board. The Advisory Board will provide insight and guidance when developing future plans, an open forum for reviewing community feedback and a broader view when prioritizing potential new services.
OSF was founded in 2004 and has been operated by information security enthusiasts since its inception. We exist to empower all types of organizations by providing knowledge and resources so that they may properly protect, detect and mitigate information security risks. We believe that security information and services should be easily accessible for all who have the need for such information. We promote open collaboration between companies and individuals, provide unbiased information to uphold educated decision-making, and attempt to eliminate the need for redundant works while striving to improve organizations’ overall security posture.
Prospective Advisory Board members should show an ability and willingness to:
- Participate actively in all meetings of the Advisory Board (2 times per year and as otherwise needed)
- Represent OSF and its mission to organizations and the general public
- Review and provide feedback for proposed OSF plans
- Chair and serve as members of committees
- Assist in locating and developing funding sources for OSF
If you are interested in volunteering please email us at firstname.lastname@example.org and provide the following information:
Area of Expertise:
If you know someone with senior leader experience who you believe could act in an advisory position please contact us at officers @ opensecurityfoundation.org.
The call for Advisory Board volunteers will be open until March 19, 2010. We will review all submissions by March 31, 2010.
The Open Security Foundation (OSF) has grown from a humble beginning in 2004 to an internationally recognized 501(c)(3) non-profit public organization. Through the work of a small team of dedicated information security enthusiasts, the Open Source Vulnerability Database (OSVDB) and DataLossDB projects have provided organizations of all sizes with the knowledge and resources to accurately detect, protect and mitigate information security risks. OSF research is often cited throughout the security industry and the organization was honored by being named winner of the SC Magazine’s Editors Choice award for 2009.
To ensure the highest quality information that has become the trademark of OSF, a tremendous amount of effort is expended on a daily basis by OSF volunteers to process an ever increasing amount of data loss and vulnerability reports. Over the years, many volunteers have been involved in the projects, but for the most part the the heavy lifting has been the work of only a few very dedicated volunteers. The “open source” approach to resourcing the projects has been successful to date but is now proving to be an unsustainable model. With long-term sustainability and increased services as our goal, we have initiated a comprehensive review of our current operations, our existing approach to project funding and the creation of potential new services for the security community.
As a start, we plan to do a better job of sharing our view on the state of the information security industry and creating a mechanism to gain community feedback to better establish our vision for the OSVDB and DataLossDB projects.
To that end I want to take a moment to share our initial plans for 2010.
The OSF officers and project leads have been dedicated to the daily operations required to make OSVDB and DataLossDB the recognized leader in vulnerability and data loss tracking. This focused dedication has left little time to take the pulse of the industry as it relates to our projects or to establish a clear long-term vision for the projects. To address this need, OSF will be creating an Advisory Board. The board will consist of three to five senior leaders capable of providing broad based perspective on information security, business management and fundraising. It is our hope that this will provide a sounding board when developing future plans, an open forum when reviewing community feedback and a broader view when prioritizing potential new services. Additional information along with an official call for Advisory Board nominations is planned for 2/12/2010.
Direct unfiltered feedback from both the security community as well as the organizations that benefit from our projects is critical. Over the next few weeks, we plan to post a public survey asking for feedback that will help shape our long-term vision and establish our near-term plans for OSVDB and DataLossDB. Those of you who value the work that the OSF provides and/or consider yourselves friends and supporters of OSF are asked to help spread the word to maximize the feedback provided.
Feedback from the survey will be the foundation for the OSF vision and 2010 plan. Our goal is to present a draft of both the vision and the 2010 plan to the newly formed Advisory Board by mid-April 2010. Once finalized, both documents will be shared with the information security community.
OSF has been recognized for providing a critical service to the information security community but our potential is much greater. We look forward to hearing your ideas on how OSF can further improve the state of security while building a stronger organization to deliver even higher quality research and additional services.
We appreciate your support and if you are interested in working with OSF please contact us at moderators @ osvdb.org or curators @ datalossdb.org.
Chairman, Open Security Foundation
OSVDB has just announced its Winter 2010 Fundraising Goal, which currently hopes to raise $9,000 before April 1, 2010. Looking back over the last couple of years of advances in the project, it’s easy to see not only how the project has evolved, but also how operational costs have increased to cover software development, content development, server hosting costs, and other assorted expenses to help keep OSVDB interesting, timely, and functional.
On an average, OSVDB has promoted 10,000 to 12,000 vulnerabilites per year for the last the last few years. Breaking that down to about 1,000 per month, the vulnerabilities in the database are gathered from a variety of sources, such as CVE, Secunia and various vendor changelogs and advisories. Keeping up a pace of about 1,000 newly listed vulerabilities per month hasn’t always been easy… but it’s about to get interesting.
I recently resigned my position as Chief Communications Officer with Open Security Foundation to focus more on the “content” aspect of OSVDB and DataLossDB. The extra time gained from giving up administrative duties will hopefully help the sites keep content fresh and accurate. Jericho, CJI, and I are going to keep working on new vulnerabilities as we can and keep the ball rolling.
With that said, I’m issuing a challenge: For every new vulnerability issued an OSVDB ID from January 1, 2010 through April 1, 2010, I will donate $0.50 (fiddy cents) of my own money to the OSVDB fundraiser. I challenge anyone who feels that OSVDB is a valuable resource to the security community to match my donation.
To make a few points clear:
- I am no longer an OSF officer. My donation comes out of my own pocket, not the OSF coffers, and I will accept no compensation from OSF for this offer. If I have to sell a kidney, I hear you only need one anyway.
- Since Jericho, CJI, and I are the ones who generally push new vulnerabilities to “live” status, there will be no slacking to save my bank account. If anything, I’ll be more motivated to push the potential donations higher and they’ll be motivated to watch me suffer on April 2. That’s how we roll.
- At an average of 1,000 vulnerabilities a month, over three months I expect to donate $1,500. It may be less, it may be more. There will be a maximum cap of $2,500 donated by myself and anyone who matches it. If we can push 5,000 vulns in three months, something is either very wrong or very great. YMMV.
- If five other people and/or groups take me up on the challenge and we meet our average, OSF will meet its goal. We still hope everyone else will contribute not only time but *effort* to help the project.
- This is not a gimmick. It’s not smoke and mirrors. You can see what OSVDB pushes on a daily basis on our Twitter page and on our contributors page. We will push all legitimate vulnerabilities just as we have been doing for years. If we’re slow for a few days, don’t worry. We’ll catch up.
So, that’s the challenge. If anyone wants to play and match my offer, please contact us at moderators[at]osvdb.org. I’m going back to work now.
Welcoming in 2009
OSVDB would like to wish everyone a happy and hopefully prosperous new year! 2008 was pretty cool for us as far as enhancements and support of OSVDB 2.0 go, and we were very happy to add over 11,000 new vulnerabilities to the database in the last year. We currently have over 51,000 vulnerabilities in the database to start the new year, and would like to invite everyone to please consider adding to this resource, whether you have a user account or not. We can use (and will gladly accept) as much help and input as we can get, so if you’re lacking a new year resolution, maybe consider an hour a week to assist the security industry gather and share knowledge about vulnerabilities.
If you have any questions, comments, or ideas, please contact us at email@example.com
General information can be found at Opensecurityfoundation.org
Happy new year, everyone!
The Open Security Foundation (OSF) is pleased to announce that the DataLossDB (also known as the Data Loss Database – Open Source (DLDOS) currently run by Attrition.org) will be formally maintained as an ongoing project under the OSF umbrella organization as of July 15, 2008.
Attrition.org’s Data Loss project, which was originally conceptualized in 2001 and has been maintained since July 2005, introduced DLDOS to the public in September of 2006. The project’s core mission is to track the loss or theft of personally identifying information not just from the United States, but across the world. As of June 4, 2008, DataLossDB contains information on over 1,000 breaches of personal identifying information covering over 330 million records.
DataLossDB has become a recognized leader in the categorization of dataloss incidents over the past several years. In an effort to build off the current success and further enhance the project, the new relationship with OSF provides opportunities for growth, an improved data set, and expanded community involvement. “We’ve worked hard to research, gather, and make this data open to the public,” says Kelly Todd, one of the project leaders for DataLossDB. “Hopefully, the migration to OSF will lead to more community participation, public awareness, and consumer advocacy by providing an open forum for submitting information.”
The Open Security Foundation’s DataLossDB will be free for download and use in non-profit work and research. The new website launch (http://www.datalossdb.org/) builds off of the current data set and provides an extensive list of new features. DataLossDB has attained rapid success due to a core group of volunteers who have populated and maintained the database. However, the new system will provide an open framework that allows the community to get involved and enhance the project. “For a data set as dynamic as this, it made sense to build it into a more user-driven format.”, states David Shettler, the lead developer for the Open Security Foundation. “With the release of this new site, the project can now be fed by anyone, from data loss victims to researchers”.
The DataLossDB’s mail list will continue to be available to over 1,500 current subscribers and will accept new subscriptions under the Attrition.org banner until a migration to OSF has been completed. RSS feeds will also be available under the OSF banner for timely alerts about new and updated data loss events. We expect this transition to be completed in the coming months without impact to current subscribers.
Open Security Foundation’s DataLossDB is an open source community project that strives to provide a clear understanding of data loss issues and needs your support. Assistance can be provided through database updates, project leadership, word-of-mouth promotion, financial donations, and sponsorship to assist with the ongoing maintenance of the project. “The DataLossDB project provides a critical service that enables detailed analysis on the true impact of data loss.”, says Jake Kouns. “The Open Security Foundation is in a perfect position to support the expansion of the DataLossDB project.” Any entities interested in licensing the database for commercial ventures are encouraged to contact OSF.