I’m here today to write about one particularly thorny area of operational technology (OT) and security that I run into somewhat routinely. Given my own particular interests as an incorrigible vulnerability-gazer, and my professional role as vice president of security research at runZero, I deal with OT security issues more often than the average bear. I’ve noticed that there’s definitely a vibe of, “IT be like this, but OT be like that” going on in the wider world of vulnerability management. The process of discovering, documenting, and disclosing vulnerabilities all have their own little quirks here in OT-land, so let’s jump into it!
Here, everything is legacy
At DEF CON, the ICS Village is one of the more popular places to hang out. It works well for folks who are either new to the field of infosec and cybersecurity, or old-hands in the industry, for the same reason: once you get close enough to a piece of OT technology with your modern IT vulnerability-hunting tooling and instincts, it often feels like you’re hacking like it’s 1999, all over again. Until very recently, OT, as a class, hasn’t been much concerned with prompting for passwords or validating user-supplied inputs; the assumption was that the local network is trusted. The software itself is typically run as compiled objects with limited hardware resources, so there’s not much room for fancy 21st Century defenses like ASLR and DEP (Address Space Layout Randomization and Data Execution Protection, respectively). Therefore, it’s an ideal platform species to practice, and kind of nostalgic for the more, shall we say, life-experienced.
Denial of service is catastrophic
Many classic OT attacks, though, aren’t really even about remote code execution (RCE) or local privilege escalations (LPE), which are the usual prize bugs for an IT-based attacker. Instead, the value of a denial of service (DoS) effect is of paramount importance in OT. A legitimate one-packet killer that bricks a wildly expensive piece of equipment (which, to be fair, would itself raise eyebrows in the IT world), a mere “temporary” condition like a sustained flow of garbage traffic that stops the device from doing its OT thing, or a safety-control tripping sequence (which intentionally causes a fail-safe condition) all end up in the same place: Actuators stop, robots freeze, and the whole purpose of the OT buildout is interrupted, off-schedule, and sometimes with human life and limb hanging in the balance.
This is a huge deal for OT operators, much more so than in IT environments where graceful failure is normal, handled, and usually solvable in a pinch by throwing cheap bandwidth and CPUs at the problem. Almost nobody has a spare backup factory.
See something, say something
So, if you find yourself with a new (to you) vulnerability in OT gear, what do you do? In IT, the normal route typically comes down to notifying the software producer, grabbing a CVE, maybe getting a fix in the works (or maybe not), then publishing findings. Rarely, some IT bugs warrant more care and scrutiny on the pre-disclosure side, but there’s nearly always some kind of mitigation or configuration that can blunt the bug even without a patch. Occasionally, there are bugs in core libraries (or entire software supply chains) that warrant some extra level of coordination. It’s bad, but manageable.
OT bugs are different; unlike in IT, in OT, it’s routine to describe the effects of a theoretical attack ending with “and that’s how you poison an entire community” or “once power delivery is interrupted, hospitals go offline,” or something equally apocalyptic. Even discussing the presence of the vulnerability is fraught, with media outlets and the government freaking out. And don’t get me started on releasing a full proof-of-concept bit of code.
The icing on this terrible, terrible cake is the fact that actually patching many OT devices is difficult to impossible, even when you know about the bug. The vulnerable code may live on a board that’s hundreds of miles away on an oil field, or subject to very strict regulations for unscheduled updates. It may not be reprogrammable at all, and instead require a costly forklift update to mitigate, and it’s anyone’s guess if the new hardware is even compatible with your suddenly-legacy control plane. Oftentimes, the best you can do is segment the vulnerable devices off to their own naughty corner of the network, and tightly control access to that network, living with the fact that anyone who touches it, physically or virtually, has the power to destroy it.
When worlds collide
The fact of the matter is, most of the defensive posture of OT is precisely network segmentation, and network segmentation alone. But, that’s changing, and like it or not, the OT/IT convergence is upon us, which means that OT is increasingly becoming IT’s problem. We cannot keep these bugs bottled up forever. So, what do you do with your fresh OT 0-day? I would argue that you cannot just sit on it. In a world of frontier AI versus soft targets, it’s only a matter of time when your easy finding becomes someone else’s critical infrastructure weapon. I’d urge you to swing by https://cisa.gov/report, and hit the “Report a software or ICS vulnerability” button. It’s the first step in what’s often a confusing and scary process, but in recent years, the major OT vendors have been taking these reports seriously and have good relationships with both CISA in the U.S. and the regional CERT/CC’s where they operate.
While “see something, say something” may not have been popular in the past, the reality of OT/IT convergence demands that we evolve our thinking and tooling to address the fundamental cybersecurity challenges facing OT before AI-assisted attackers turn off the lights or worse.
