Tag Archives: SCADA

2013 Superdome Outage a Hack? The Value of Post-Incident Investigations.

As we approach the pinnacle of U.S. sportsball, I am reminded of the complete scandal from a past Superbowl. No, not the obviously-setup wardrobe malfunction scandal. No, not the one where we might have been subjected to a pre-recorded half-time show. The one in 2013 where hackers terrorism who-knows-why caused the stadium lights to go out for 34 minutes. That day, and the days after, everyone sure seemed to ‘know’ what happened. Since many were throwing around claims of ‘hacking’ or ‘cyber terrorism’ at the time, this incident caught my attention.

Here’s what we know, with selected highlights:

  • February 3, 2013: Superbowl happened.
  • February 3, 2013: Anonymous takes credit for the blackout.
  • February 3, 2013: Because theories of hacking or terrorism aren’t enough, Mashable comes up with 13 more things that may have caused it.
  • February 4, 2013: A day later, we’re once again reminded that “inside sources” are often full of it. Baltimore Sun initial report claimed a “power-intensive” halftime show might have been a factor.
  • February 4, 2013: The FBI makes a statement saying that terrorism was not a factor.
  • February 4, 2013: We learn that such a failure may have been predicted in 2012.
  • February 4, 2013: Of course the outage doesn’t really matter. A little game delay, and it is a “boon for super bowl ratings“, the most critical thing to the corrupt NFL.
  • February 4, 2013: By this point, people are pretty sure hackers didn’t do it. They probably didn’t, but they could have!
  • February 4, 2013: Oh sorry, it could still be hackers. The Christian Science monitor actually covers the likely reason, yet that isn’t sexy. Chinese hacker ploy seems more reasonable to cover…
  • February 4, 2013: Not only Anonymous, but ‘Rustle League’ claimed to hack the super bowl. A day later we learn that notorious Rustle League trolls were … wait for it … trolling.
  • February 5, 2013: Officials at Entergy, who provide power for that property clearly state “There was no Internet or remote computer access to the piece of equipment inside the stadium that sensed an abnormality in the electrical system and partially cut power to the Superdome…”
  • February 6, 2013: While the Superdome was not hacked on Sunday, the U.S. Federal Reserve was.
  • February 8, 2013: Multiple sources begin covering the real reason for the Superdome outage.
  • February 8, 2013: We now have a good idea what caused it, but let the blame game begin. Manufacturer error, or user error?
  • March 21, 2013: The official Entergy report is released (PDF), giving a very technical analysis and summary of what happened. Everyone but conspiracy theorists can sleep well.

The reason for this blog is that Chris Sistrunk, a noted SCADA security researcher, pinged me the other day about the report. We were curious if the failure described could be considered a vulnerability by OSVDB standards. After reading the report and several questions for him, this seems like a simple case of device malfunction / failure. Quoting relevant bits from the report:

During the testing, behavior of the relay was not entirely consistent with the function described in the instruction manual. Under some circumstances, when the current exceeded the trip
setting and then decreased below the trip setting even after the timer had expired, the relay did not operate.

This instability was observed on all of the relays tested (during testing by this engineer, ENOI, and others in coordination with S&C at Vault 24 on March 1, 2013), including the subject
(Bay 8) relay and two identical (exemplar) relays. Behavior of the device in a manner contrary to the published functionality of the device constitutes a design defect.

Interesting read and glimpse into the world of SCADA / ICS. While the notion that the outage was due to hackers, the reality is far more mundane. We could certainly learn from this case, along with thousands of others… but who am I kidding. News covering the mundane and real doesn’t sell.

The problem with SCADA goes deeper…

We know SCADA is virtual swiss cheese, ready to be owned if someone can reach a device. We have preached airgaps for decades, even before we knew how bad the software was. Back then it was just, “this is so critical, it has to be separate!”

The last five years have proven how bad it is, with the rise of SCADA vulnerabilities. Sure, we can overlook the bad coding, proprietary protocols, no evidence of a SDLC, and the incredible amount of time it can take to patch. For some silly reason we put up with “forever-day bugs” because something is so critical it can’t be rebooted (forgetting how absurd that design choice is). But, what if we go a step beyond that?

An ICS-CERT 14-084-01 advisory released yesterday on vulnerabilities in Festo products is a good reminder of just how bad the problem is, and how much deeper it goes. First, the product has a backdoor in the FTP service allowing unauthenticated access (CVSSv2 9.3). This can allow a remote attacker to crash the device or execute arbitrary code. Second, the device is vulnerable due to bundling the 3S CoDeSys Runtime Toolkit which does not require authentication for admin functions (CVSSv2 10.0), and a traversal flaw that allows file manipulation leading to code execution (CVSSv2 10.0). Those two issues were reported in January of 2013, making this report as relates to Festo products over a year late.

So we have a vendor backdoor, unauthenticated administrator access, and a way to bypass authentication if it was there to gain privileges. So realistically, what type of organizations does this potentially impact? From the ICS-CERT advisory:

This product is used industrywide as a programmable logic controller with inclusion of a multiaxis controller for automated assembly and automated manufacturing. Identified customers are in solar cell manufacturing, automobile assembly, general assembly and parts control, and airframe manufacturing where tolerances are particularly critical to end product operations.

Now to dig the hole deeper. Under the “Mitigation” section, we see how serious Festo considers these vulnerabilities. Paraphrased from two lines in the advisory:

Festo has decided not to resolve these vulnerabilities, placing critical infrastructure asset owners using this product at risk … because of compatibility reasons with existing engineering tools.

The two 3S CoDeSys vulnerabilities have a fix available and just need to be integrated into the Festo products. What does “compatibility with existing engineering tools” really mean in the context of software? The ICS-CERT advisory also says:

According to the Festo product web page, other products are using newer versions of CoDeSys software and may not be vulnerable to the CoDeSys vulnerability, but this has not been evaluated by the researcher.

The researcher already spent time finding the issues, reporting them to a coordinating body, and following coordinated disclosure practices. Expecting them to also evaluate which products are not vulnerable is ridiculous. This is a case of the vendor just being lazy and irresponsible.

A company that makes vulnerable critical components that affect our infrastructure and directly impact our safety, but refuses to fix them. Why is this allowed to exist in our society?


Ferreting Out Unique Vulnerability Data in OSVDB

In previous blog posts and on Twitter, I have shown and mentioned various methods for searching OSVDB to find interesting data. However, there is no written guide to the ins-and-outs of the data. The search interface is simple enough, but it can be used in a manner that allows for some complicated and useful searches that are not immediately obvious. This blog post will show several examples and highlight some of the interesting data we have available, along with an explanation to the method of our madness.

The OSVDB classification system allows for a variety of one-click searches. Using the search interface and selecting any of the classifications (single, or multiple) will let you quickly search for denial of service, exploit public, security software, and a lot more. Note that our data set is not complete, and not all of our entries have classification data. Do not rely on this type of search for complete results. Over time as the data set is completed, it will provide powerful one-click searches that will make for interesting metrics.

While our classification system is robust, it has been a struggle for us to determine if we want to add classes of issues as a new classification option, or use specific keywords that can be searched for. While a classification box is convenient, it can quickly become bloated if there are hundreds to choose from. We have “security software” as a classification because of the irony in software designed to protect you from threats adding to your vulnerability footprint. In the coming year, we may expand the ‘OSVDB’ classification box to allow for additional searches, where that box can be hidden entirely if desired. Until then, there are several fun keyword-based searches you can do:

  1. SCADA, the hot topic lately. Using the “vulnerability text” field, input “SCADA” and select “All Text” (defaults to “Titles only”). This will bring back all vulnerabilities related to SCADA products.
  2. Another field that has been interesting to us for several years, that will likely gain more attention this year in the wake of recent election problems, is Electronic Voting Machines. We’ve all read articles about the insecurity of Diebold for example. But have you looked at just how bad it is, and how bad the other vendors are? Do a “vulnerability text”, “all text” search for “electronic voting machine”. Prepare to be scared for the coming elections.
  3. There has been an increasing interest in vulnerabilities in embedded computers found in cars. While “car hacking” has been going on for many years, a big part of that field is based on modding and enhancing a car, not so much exploiting vulnerabilities in it. OSVDB has only delved into this topic a little bit so far, but it has been on our radar for some time. Doing the same “all text” search for the word “automobile” will bring up what we have. There are dozens of research papers and sites on our list to check out as time permits.
  4. We have spent a lot of time digging into the history of encryption algorithms, noting when they were effectively compromised or proven vulnerable to varying degrees of practical attacks. Having these in the database makes for an interesting history, great reference, and potentially helpful to pen-testers that find applications using insecure algorithms. Even if you don’t have time to leverage the weakness during the test, you can provide a standardized reference in the report. To find these, do a “vulnerability text”, “title only” search for the word “algorithm”.
  5. Using specific keywords in our standardized titles, quick searches can be performed for other interesting sets of vulnerabilities. For example, the word “hardcoded” is used to denote when a vendor uses an account name, password, community string, or other piece of identifying / security information in a manner that does not allow the user to change it. It is scary to see that hardcoded accounts and credentials are still being used in 2012, by security vendors no less. In a similar vein, the word “persistent” is used to denote other conditions where some form of weakness will continue to be present, regardless of administrative action.

Other interesting search tips:

  • “all text” word searches; botnet shows the increasing vulnerabilities found in botnet software
  • Want to find vulnerabilities in Drupal, but not all those third-party modules? Title search “drupal -module -theme” to see the ‘core’ software issues.
  • Similarly, title search for “wordpress” and “wordpress -plugin” to get a feel for the disparity in vulnerabilities between the core software and third-party plugins.

These represent just a few examples of the types of searches you can perform using OSVDB to ferret out interesting data and vulnerabilities that tend not to make it in the other VDBs.

Coffee makers are SCADA, right?!

Steven Christey of CVE posted asking a question about VDBs and the inclusion of coffee makers. Yes, you read that correctly, vulnerabilities are being found in coffee makers that are network accessible. Don’t be surprised, we all knew the day was coming when every household appliance would become IP aware.

Before you laugh and spew your own coffee all over the keyboard, consider that the vulnerabilities are legitimate in the sense that a remote attacker can manipulate how the device performs and possibly do physical damage to the unit. This is really no different than SCADA devices such as air conditioners that are IP aware.

Some replies (like mine) were a bit more serious suggesting this type of vulnerability is definitely worth inclusion in OSVDB. If we can’t draw the line between coffee makers, air conditioners and other SCADA devices today, we will be able to in a year or years from now? At some point, the blur between computing device and household appliance will be too hard to distinguish. Rather than waste too much time arguing that line, why not track these few vulnerabilities now that might be a bit primitive, but will surely show historic value if nothing else.

Other replies were a bit less serious but fun, suggesting that making weak (or no) coffee would lead to disgruntled code writers that produce poor code filled with more vulnerabilities. Either way, count on us to include vulnerabilities in your favorite IP aware devices, kitchen, computing or otherwise, to this database.


Get every new post delivered to your Inbox.

Join 6,043 other followers