The problem with SCADA goes deeper…

We know SCADA is virtual swiss cheese, ready to be owned if someone can reach a device. We have preached airgaps for decades, even before we knew how bad the software was. Back then it was just, “this is so critical, it has to be separate!”

The last five years have proven how bad it is, with the rise of SCADA vulnerabilities. Sure, we can overlook the bad coding, proprietary protocols, no evidence of a SDLC, and the incredible amount of time it can take to patch. For some silly reason we put up with “forever-day bugs” because something is so critical it can’t be rebooted (forgetting how absurd that design choice is). But, what if we go a step beyond that?

An ICS-CERT 14-084-01 advisory released yesterday on vulnerabilities in Festo products is a good reminder of just how bad the problem is, and how much deeper it goes. First, the product has a backdoor in the FTP service allowing unauthenticated access (CVSSv2 9.3). This can allow a remote attacker to crash the device or execute arbitrary code. Second, the device is vulnerable due to bundling the 3S CoDeSys Runtime Toolkit which does not require authentication for admin functions (CVSSv2 10.0), and a traversal flaw that allows file manipulation leading to code execution (CVSSv2 10.0). Those two issues were reported in January of 2013, making this report as relates to Festo products over a year late.

So we have a vendor backdoor, unauthenticated administrator access, and a way to bypass authentication if it was there to gain privileges. So realistically, what type of organizations does this potentially impact? From the ICS-CERT advisory:

This product is used industrywide as a programmable logic controller with inclusion of a multiaxis controller for automated assembly and automated manufacturing. Identified customers are in solar cell manufacturing, automobile assembly, general assembly and parts control, and airframe manufacturing where tolerances are particularly critical to end product operations.

Now to dig the hole deeper. Under the “Mitigation” section, we see how serious Festo considers these vulnerabilities. Paraphrased from two lines in the advisory:

Festo has decided not to resolve these vulnerabilities, placing critical infrastructure asset owners using this product at risk … because of compatibility reasons with existing engineering tools.

The two 3S CoDeSys vulnerabilities have a fix available and just need to be integrated into the Festo products. What does “compatibility with existing engineering tools” really mean in the context of software? The ICS-CERT advisory also says:

According to the Festo product web page, other products are using newer versions of CoDeSys software and may not be vulnerable to the CoDeSys vulnerability, but this has not been evaluated by the researcher.

The researcher already spent time finding the issues, reporting them to a coordinating body, and following coordinated disclosure practices. Expecting them to also evaluate which products are not vulnerable is ridiculous. This is a case of the vendor just being lazy and irresponsible.

A company that makes vulnerable critical components that affect our infrastructure and directly impact our safety, but refuses to fix them. Why is this allowed to exist in our society?


5 responses

  1. When was the last time you patched the firmware in your washing machine or dryer? How can you be certain that the new firmware for your clothes dryer will never cause a house fire? How about the firmware in a gas boiler? What about the various computer systems in your car? These are critical elements to your regular daily life. Honda and Toyota are being sued over engine control and stability control firmware. YOU SHOULD PATCH THEM REGULARLY! –But you don’t, do you?

    Patching an embedded system is no picnic. You never do it all at once in a critical system and one can never quite be certain that the patch will be reasonable under all the specific conditions in your system. The testing regimen is expensive and not trivial. Do you have the guts to stand next to that 4160 volt three phase Variable Frequency Drive and start it up for the first time after that patch you just uploaded? Be sure you’re wearing arc-flash protection because if it goes up in smoke you could easily go with it.

    It is easy to sneer at the situation until you look more closely at it. This is not about some office computer with a peripheral attached to it. It is a machine with an embedded processor hanging off of it. Patching these things is never as simple as it looks to someone who only understands computers.

    Rants like this are simply going to keep people from attaching networks to things. They will embed their code deeper than ever and you’ll be left wondering what the hell a machine is doing and why.

    I personally find Festo’s behavior unsettling, but the situation itself is far larger than just poor behavior by one or two companies. The entire, project oriented design process and the market itself is open to question here. There is much more to this than just opening the door so that self-proclaimed experts can teach us earthy engineers about computer magic. Government will probably perceive a need and issue regulations, standards, and practices that will make software a much more bureaucratic and less performance oriented endeavor than it has ever been. We will soon see licensed embedded software engineers who will have to certify that their design will do exactly some goal, no more and no less.

    That is where engineers are today. We engineers have a stamp used on our designs which has strong legal implications and obligations toward our clients. I foresee a future where embedded software developers could be held criminally responsible for what they write and no weasel words in a contract will be able to save them.

    A future embedded system patch will probably involve a product recall, just like we’re doing today with Toyota ECUs.

    Be careful what you ask for…

    1. Are you sure you read this blog? Your analogy is absolutely absurd. Comparing a clothes dryer to a SCADA installation suggests you don’t really know anything about the topic. Was that your intention? (I doubt it, but that is exactly how you come across here…)

      I understand patching SCADA is no picnic, I know it can’t be done on a whim. I am more understanding of patches taking longer to develop, and longer to implement than other systems. Your second analogy is also absurd. If a software patch can cause a “4160 volt three phase Variable Frequency Drive” to go up in flames, then I would hope anyone familiar with it know not to stand next to it during a reboot, no matter what. If you stand next to it, even “wearing arc-flash protection”, you sound like you aren’t qualified to work wherever that is. In two paragraphs, you have demonstrated you aren’t an expert on SCADA any more than I am.

      I have looked closely at it, as much as can be done from someone that doesn’t work with the gear every day. I understand this isn’t a simple “install patches, reboot a Windows machine”. I have tested a few pieces of SCADA gear and disclosed vulnerabilities, I have gone out of my way to attend SCADA conferences, I have cultivated SCADA security experts in my field to discuss this stuff with so that I understand the topic before I write about it. While I am not an expert, I have gone out of my way to make sure I am surrounded by experts to give me feedback and information that is realistic to ensure I present a reasonable view and argument.

      The point of this post really goes in the direction you do; if the SCADA companies don’t begin to regulate themselves, and find a way to address security concerns, the government will step in. Like you, I don’t think that is a good thing necessarily. I’d rather have these companies understand the impact and want to give security a priority, even if they didn’t historically. And you know what? Many of them are doing just that. We’re seeing it a lot more often now, where a SCADA vuln is patched in *months*, when we would have accepted *years* as an improvement. They understand what is risk, they understand the attack scenarios, and most importantly, they understand that the actual deployments are not air-gapped and difficult to reach. So when a company like Festo comes along, we call them out. We do it so our industry knows they are a bad actor in this game, and we put pressure on them to fix things. Combined with government issued advisories calling them out for refusing to fix vulnerabilities, that is a good thing.

      The days of EULAs and shrink-wrap licenses protecting vendors are probably closing. Microsoft, Apple, and even Oracle don’t have to face the serious consequences that a SCADA vendor does, if that happens. So again, the point of the post is to remind vendors that security is important to everyone. Worse, when they *advertise* security and put in a backdoor, they simply cannot get away with it. The analogy of a software developer being compared to a bridge engineer has been thrown around for a decade, and it should scare software devs everywhere. If they are held to the same standards as building or bridge engineers, life will get very interesting very fast.

      The only thing I am asking for is for vendors, especially in the SCADA arena, is to put some care and effort into security. Backdoors, unauthenticated access, and flat out REFUSING to patch a vulnerability is ridiculous. In the last years, we have seen SCADA vendors do a full 180 and begin patching vulnerabilities quickly. They clearly understand what is at stake, and they are doing the right thing. Companies like Festo are not only potentially endangering lives, they are putting a black mark on the rest of their industry who actually cares.

      I get that you don’t like the general SCADA hate. That said, I recommend you actually read more before you jump in with these mostly insipid comments. You aren’t helping your argument, and you aren’t making SCADA engineers look good to everyone else. While that doesn’t matter to me, it does to your clients and the government who has a vested interest in stepping in to make things right, even if they don’t know what that entails. I am actually on your side; I understand the balance you face between upgrades and functionality, while understanding the real impact of the vulnerabilities, and the real-world implementations of both. Attacking me doesn’t help you, doesn’t help me, and only serves to help everything you are against. In short, you are a dumbass.

  2. Yes. I confess. I am a dumbass. I am one of those dumbasses who has been perpetrating industrial control systems and SCADA systems for most of 30 years. We dumbasses are so dumb that we regularly invite people over to tell us what they think we should do to build a better systems. Just last week, we had someone come by and show us some new chemical resistant fiber optic innerduct. We routinely experiment with new equipment in our lab before we deploy it to the field. I have radios, routers, switches, RTUs, PLCs, VM hardware, 48 volt systems, 24 volt systems, fiber-optic systems, NAS hardware, firewalls…

    Yeah, we’re in-house integrators. We have to live with our creations. So we design it for dumbasses like me.

    So why should you listen to what I have to say?

    SCADA systems do not stop at the control room. They connect to real assets in the field. The RTU may need to be patched too. And that VFD that hooks to the RTU? Yeah, they get patches. People act as if these patches are nearly free. But they’re not. I have to travel to the site. I have to patch each thing, and carefully test it to ensure that I haven’t broken anything obvious. These checks take lots of time and they also frequently require downtime scheduled or even the distribution system itself to be configured for a test. In other words, I have to coordinate with our operations control center to do these patches. There are times, sometimes entire seasons, when they legitimately have to tell me no, I can’t patch.

    I can show you a water pumping station with a PowerFlex 755 VFD attached to a small 75 HP motor. Yeah, 75 HP is small. I’ll show you the many safety features (including arc-flash stickers and zones painted on the floor). I can also show you the really large stuff.
    We usually power them with medium voltage switchgear. I can show you how we deal with the risks of software, hardware, and infrastructure that we just can’t trust.

    And yes, you can hear stories about how a 200 HP motor nearly jackhammered itself off the reinforced concrete floor while testing a soft-starter that had a problem with one of the phases.

    I can show you the resiliency we have built in to the system and the private microwave networks we have built to support our SCADA system.

    But I’m just a dumbass. You shouldn’t listen to what I think, unless you have just a slight doubt that you might be wrong. If that’s the case, and you can get to Laurel, Maryland, let me know. I’ll show you one of the most subtle, clever, and dumbass industrial control systems you’ve ever seen. Look me up on LinkedIn.

    1. I get it, you are a hero and the bestest expert. One that still seems to have missed the entire point of the blog.

  3. “It is easy to sneer at the situation until you look more closely at it.” But the vendors are literally retarded. It’s not even remotely complex or nuanced to avoid introducing these types of vulnerabilities in the first place.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: