Building an ICS Firing Range – Part 1 (Defcon 29 ICS Village)

An Incident in a Water Treatment Plant

Beginning of this year, the supposed hack of a Water Treatment plant in Florida made some waves. While we often read about news-worthy hacks, this one stuck out due to the apparent simplicity of the compromise and the severe consequences it could have had. So, what had happened?

As so often, a lot of assumptions are made during investigations of the incident which must be taken with a grain of salt. What is confirmed though is the fact that an individual directly connected to an internet-facing operator workstation via TeamViewer. This enabled them to remotely control the system and dramatically increase the levels of sodium hydroxide in the water with severe consequences to the health of the consumers. Luckily, an operator spotted this and could revert any changes. It must also be noted that reportedly other controls existed that could have prevented sodium hydroxide from actually leaking into the drinking water.

Why does this incident not come as a big surprise? More often than not in the world of Operational Technology (OT), you find personnel with a degree in Engineering. They do great work in their field of expertise but might lack the insight into common security topics relevant to Information Technology (IT). After all, their priority is to make things work and keep them running. Therefore, such an engineer might approach problems in a more pragmatic way as opposed to someone who’s spent their fair share in IT and faced the various security issues.

Security Requirements: OT vs IT

The major differences between the two worlds, OT and IT, becomes evident when you look at the specific security requirements. While in Information Technology, confidentiality is paramount in most scenarios before integrity and availability, it is the other way around in OT. If a critical system in a water treatment plant becomes unavailable, things can go down-hill fast.

Needless to say, for a plant operator directly accessing the workstation via TeamViewer is a lot more convenient and efficient than having to jump through hoops of security controls.

While the incident in Florida is not a security flaw in OT infrastructure per se, it has to do with pragmatism contrasting with security awareness.

The incident in Florida is not an isolated case. Be it the incidents of the Ukrainian power grid, ransomware attacks in Hospitals or Transportation around the globe or the sophisticated attack against a power plant in Saudi Arabia, it begs the question: Did nobody test the security controls of the critical infrastructure?

Security Testing in OT

First and foremost, most OT environments are years old and were not designed with security in mind. Any security controls therefore need to be built on top. Therefore, one of the possible answers to the question as to why security controls have not been tested or implemented is: Availability. Implementing a security control will restrict access in some form. To ensure that the control works as intended and cannot easily be circumvented, testing is required. Testing of such security controls is generally an intrusive act and can sometimes be destructive even against IT infrastructure. Let’s not forget, during most security tests, you’d want to provoke an action of a system by supplying data it did not expect, or, in order to find out more about a service are sending possible triggers. If the system cannot properly handle this, it might fault. That is why for critical IT systems you’d want to have any active testing be conducted out of business hours or against a test environment. But this is not feasible to do in a factory or plant that works 24/7.

Naturally, if you decided to conduct a security test of your OT infrastructure anyway, you’d want to hire a team of security professionals who also bag the required awareness and sensitiveness of the OT world. And this is where OT security testing becomes a hen and egg problem.

Training of ICS Security Experts

A lot of those security testers, pentesters and red teamers grew up in the IT world. They have not had the chance to actively test the infrastructure in a steel producing factory or audit segmentation firewalls in a water treatment plant. And let’s be honest, would you want to be the first one to give them the chance?

A solution to all this is to give them the tools and means to acquire the skills and awareness well before they are on the factory floor.

This is where we began our journey of creating a training environment. The primary motivation of our firing range, comprised of virtualized IT infrastructures and real OT components, was to internally train the Red Team on how to safely approach OT level infrastructure. Secondly, it allowed our Blue Team to detect and investigate those attacks. Lastly, having a working OT infrastructure for testing purposes opened up all sorts of possibilities for isolated OT component testing and further research into tooling, vulnerabilities and methodologies.

Our lab is a 120 by 80 cm wide, 3D-printed bascule bridge which is controlled by several Siemens PLCs and HMIs with entire virtualized enterprise and SCADA networks. It is mobile and thus can be easily transported between sites. It is designed to cover different training scenarios, depending on the individual needs.

Recently, we could present out Firing Range at the Defcon 29 ICS Village. Check out the video below.

In this series of blog posts, we’ll discuss our development process of the firing range in more detail, present the rationale behind decisions made and present the challenges we faced along the way.

This series of posts is continued in part 2!