2013 Superdome Outage a Hack? The Value of Post-Incident Investigations.

As we approach the pinnacle of U.S. sportsball, I am reminded of the complete scandal from a past Superbowl. No, not the obviously-setup wardrobe malfunction scandal. No, not the one where we might have been subjected to a pre-recorded half-time show. The one in 2013 where hackers terrorism who-knows-why caused the stadium lights to go out for 34 minutes. That day, and the days after, everyone sure seemed to ‘know’ what happened. Since many were throwing around claims of ‘hacking’ or ‘cyber terrorism’ at the time, this incident caught my attention.

Here’s what we know, with selected highlights:

  • February 3, 2013: Superbowl happened.
  • February 3, 2013: Anonymous takes credit for the blackout.
  • February 3, 2013: Because theories of hacking or terrorism aren’t enough, Mashable comes up with 13 more things that may have caused it.
  • February 4, 2013: A day later, we’re once again reminded that “inside sources” are often full of it. Baltimore Sun initial report claimed a “power-intensive” halftime show might have been a factor.
  • February 4, 2013: The FBI makes a statement saying that terrorism was not a factor.
  • February 4, 2013: We learn that such a failure may have been predicted in 2012.
  • February 4, 2013: Of course the outage doesn’t really matter. A little game delay, and it is a “boon for super bowl ratings“, the most critical thing to the corrupt NFL.
  • February 4, 2013: By this point, people are pretty sure hackers didn’t do it. They probably didn’t, but they could have!
  • February 4, 2013: Oh sorry, it could still be hackers. The Christian Science monitor actually covers the likely reason, yet that isn’t sexy. Chinese hacker ploy seems more reasonable to cover…
  • February 4, 2013: Not only Anonymous, but ‘Rustle League’ claimed to hack the super bowl. A day later we learn that notorious Rustle League trolls were … wait for it … trolling.
  • February 5, 2013: Officials at Entergy, who provide power for that property clearly state “There was no Internet or remote computer access to the piece of equipment inside the stadium that sensed an abnormality in the electrical system and partially cut power to the Superdome…”
  • February 6, 2013: While the Superdome was not hacked on Sunday, the U.S. Federal Reserve was.
  • February 8, 2013: Multiple sources begin covering the real reason for the Superdome outage.
  • February 8, 2013: We now have a good idea what caused it, but let the blame game begin. Manufacturer error, or user error?
  • March 21, 2013: The official Entergy report is released (PDF), giving a very technical analysis and summary of what happened. Everyone but conspiracy theorists can sleep well.

The reason for this blog is that Chris Sistrunk, a noted SCADA security researcher, pinged me the other day about the report. We were curious if the failure described could be considered a vulnerability by OSVDB standards. After reading the report and several questions for him, this seems like a simple case of device malfunction / failure. Quoting relevant bits from the report:

During the testing, behavior of the relay was not entirely consistent with the function described in the instruction manual. Under some circumstances, when the current exceeded the trip
setting and then decreased below the trip setting even after the timer had expired, the relay did not operate.

This instability was observed on all of the relays tested (during testing by this engineer, ENOI, and others in coordination with S&C at Vault 24 on March 1, 2013), including the subject
(Bay 8) relay and two identical (exemplar) relays. Behavior of the device in a manner contrary to the published functionality of the device constitutes a design defect.

Interesting read and glimpse into the world of SCADA / ICS. While the notion that the outage was due to hackers, the reality is far more mundane. We could certainly learn from this case, along with thousands of others… but who am I kidding. News covering the mundane and real doesn’t sell.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: