Blog

When Florida law enforcement announced a cyberattack in the Oldsmar water district, we were startled to learn that cyberattacks can actually reach inside our homes, right down to the water we drink and use.

It was alarming to learn that a cyber attacker had leveraged the TeamViewer app to remotely access the city’s water treatment system and increase sodium hydroxide levels by more than 100 times. Fortunately, the problem was fixed without damage, but the incident should be a wake-up call for water districts everywhere.

A Low Profile, Underdeveloped Cyber Attack

An analysis by the Nozomi Networks Labs team cites the attack’s lack of sophistication, which raises a red flag. If Oldsmar is so easy to access, what about thousands of other water production facilities around the world? If a low-skilled attacker can start the process of mass poisoning the population with a few mouse clicks, what can a medium or high-skilled attacker do? When it comes to protecting our home, the bar can never be set high enough. However, in this case, the level seems too low. How did we get here? What went wrong?

Based on the information currently available, this attack appears to lack any complexity that could trigger deeper responses. The fact that the perpetrator did not conceal his access to the system to personnel monitoring the water treatment operation is the first sign of the relatively low sophistication of the attack. Moreover, according to reports of the incident, the attacker increased sodium hydroxide levels by a significant amount, typically monitored by automated systems, which likely means that the threat actor did not have any specific background knowledge about the water treatment process. 

Nevertheless, this event is important because it reflects the situation of too many Control System installations, especially smaller sized ones with smaller budgets and where security is often overlooked. Remote access, especially when not designed with security in mind, is often the battering ram used by remote attackers to infiltrate an OT network. In this very case, Oldsmar’s water treatment plant uses a TeamViewer instance, apparently accessible from the internet. While it is unknown at this stage how the attackers obtained the necessary credentials, this incident, like many others we have documented in recent years, did not appear to rely on sophisticated zero-day exploits for its execution.

TeamViewer and Other Remote Access Tools

The impact of COVID and the resulting mass rollout of remote access technologies to the detriment of cybersecurity has been discussed in great detail in the cybersecurity space. In many water utilities – as in other utilities and industries – the need to keep systems up and running in the midst of the viral pandemic has prompted many operators to face hard realities.

To be able to work from home, production control networks need to be accessible from home. Enter TeamViewer and many other remote access applications like it. Before COVID-19, using these applications to access critical process controls could be described as negligent or used without regard for safety. But today, we have legitimate reasons to need remote access and monitoring during the pandemic.

Reducing Risk by Prioritizing the Process

Realizing that we will have to live with such remote access and supply chain risks for the unforeseeable future, how can we reduce the impact of risks as much as possible?

First, we need to remember the whole point of the (risk assessment) exercise. Why try to protect one thing and not something else? If the whole point of a business is to produce water, then start there. How is water produced? What specific overtime processes are involved? And what are the risks of these processes? If we lead the process, we may find that some devices are less worthy of our attention than others. Should I secure my iPhone or the digital dashboard in the operations center?

How Can This Help Mitigate Attacks Like Oldsmar?

Water utilities investing in cybersecurity can answer many of these questions quite easily by looking at state-of-the-art asset inventory software that tracks every aspect of critical equipment used in the process. Unfortunately, many municipalities do not have the budget for state-of-the-art cybersecurity.

Most of the utilities with asset inventory are using IT products that do not do a good job of handling operations technology equipment that is highly proprietary. In fact, many of these water districts are purpose-built for exactly where they are located.

Only a subset of facilities that monitor their networks run network anomaly detection, a technology many rely on to hunt down previously unknown threats. However, in the Oldsmar attack, no network anomalies (from behind the firewall) would have been generated due to the attacker leveraging an officially approved remote access path (using TeamViewer to set OT parameters on the PLC/DCS from the HMI).

There is a possibility that the attacker was from outside the region, which might have shown an anomaly in terms of the firewall, but if the attacker was in the region, the connection could have looked like a district employee working at home. Or the attacker could have breached a completely unrelated device, such as a personal tablet plugged into the same Wi-Fi network used by a telecommuting engineer responsible for managing the facility. The attacker(s) then stole TeamViewer’s credentials and then used those credentials to log into the facility and attempt to cause damage.

Due to the Human Machine Interface (HMI) on the same machine as TeamViewer access, there would be no network anomalies. In this scenario, the attacker(s) would already have the necessary access to the process. There was no need for reconnaissance activities or lateral movement. It is as if the front door of the bank opened directly to the vault, with nothing or no one in between, including security. As unfortunate as it is, network anomaly detection technologies are ill-suited to mitigate this particular use case.

Another common effort is to harden the system by identifying and patching vulnerabilities. However, in the Oldsmar case, there was no need for vulnerabilities (with the possible exception of TeamViewer, which is still not explained how access was gained). While exploiting vulnerabilities is a crucial part of a robust cybersecurity program and its importance should not be diminished, it could not have prevented this attack.

The best of the best Protection?

Few water utilities have a detailed asset inventory, or monitor their networks for attacks and have network anomaly detection solutions up and running, like other critical infrastructure facilities. This is the highest level of protection. This approach is based on using artificial intelligence to run anomaly detection against real parameters used to control the industrial process.

Consider a pump, for example. If it is set to spin at 100 revolutions per minute (RPM), it would not be safe to run higher. The engineers programming the HMI must not allow dangerous conditions to exist and in many cases will prevent operators from entering invalid inputs or introducing unsafe conditions, such as sending a parameter value to the pump to spin at 150 RPM.

Protecting the Process

The Oldsmar attack took place within the data stream used to monitor and control the process. The attacker used a legitimate HMI to send a legitimate packet with a legitimate payload that increased the amount of sodium hydroxide in the water. Had anomaly detection been applied to the OT parameters at the Oldsmar water plant, a successful attack elsewhere within the enterprise could have attempted to impact the industrial process using a variety of methods.

Perhaps today it was this particular HMI, perhaps yesterday they injected packets directly into the network. For all we know, the attacker(s) could have moved sideways for weeks and are still there today, even using a different device in the industrial process, sending commands to disable an automated safety system that is commonly referred to as being there to prevent toxic water from reaching the population. The only way to know for sure is to monitor the actual industrial process tags and values to look for anomalies in it. Not only the Oldsmar attack, but many other risks will be addressed, including negligent or risky insiders.

Oldsmar and many other OT-specific attacks have one thing in common: they all impacted the process and used approved technology infrastructure to target the process. Attacks are more easily detected when infrastructure is the target, but when infrastructure is used to attack the process, things become more complex and more advanced tools are required to mitigate threats.

Fortunately, many of our customers are very well equipped to deal with these scenarios. By leveraging Nozomi Networks solutions, they can go from virtually invisibility directly to the most important stage of industrial cybersecurity monitoring. The entire journey can take place in as little as a few hours.

Nozomi Networks customers leverage comprehensive asset inventory, network anomaly detection, vulnerability management, investigative graphical tools, forensics, reporting, real-time dashboards and partner integrations to not only quickly identify when an attack like Oldsmar has occurred, but also to automate and orchestrate the appropriate response to such a catastrophic intrusion of their facilities. In many cases, the value is instantaneous because these capabilities are available out of the box and require little or no configuration.

Today it’s TeamViewer. Yesterday it was SolarWinds. A week ago, Ransomware #5,001. Let’s face it, the risks are pervasive and persistent. Full end-to-end monitoring of critical industrial processes should be the goal and the norm, not a nice-to-have.

Source: Nozomi Networks Labs