By Gregory Hale
There was an incident Aug. 4 that shut down a critical infrastructure organization in the Middle East where an attacker deployed malware designed to manipulate industrial safety systems along with controlling the distributed control system, researchers said.
The targeted safety system, a Triconex safety system from Schneider Electric, hit by the malware did what it was supposed to do and shut down the facility.
Mandiant and FireEye researchers said they have moderate confidence the attacker was developing the capability to cause physical damage and inadvertently shutdown operations. Dragos, which also released a report on the incident, said “the malware leverages no inherent vulnerability in Schneider Electric products. However, this capability, methodology, and tradecraft in this very specific event may now be replicated by other adversaries and thus represents an addition to industrial asset owner and operators’ threat models.”
“We got a call after the asset owner experienced a shutdown of the facility and the controllers failed safe and the asset owner investigated and they noticed there were some suspicious things going on and they noticed the malware and they called us to investigate,” said Dan Scali, senior manager, industrial control systems security consulting at Mandiant.
“The SIS engineering workstation was compromised and had the Triton malware deployed on it and also that the DCS was compromised, so you can envision an attack where the attacker has the ability to manipulate the DCS while reprogramming the SIS controllers.”
“The DCS was compromised and we think that is important and relevant because by manipulating the SIS creates the potential for the process to go into an unsafe state, but if you also have control and compromise the DCS, with control of the process and enough knowledge and the ability to reprogram the SIS, you have the potential to create more significant consequences,” Scali said. “We believe the attacker intended to cause a physical consequence at some point in the future. We don’t believe the attacker intended to cause that at the time the shutdown actually occurred.”
“The Tricons behaved like they were supposed to behave,” said Andrew Kling, director of cyber security and architecture at Schneider Electric. “Something happened in memory and the Tricons found a mis-compare and it said ‘something is wrong go to safe state’ and the plant went to safe state.”
This malware, which Mandiant researchers are calling TRITON and Dragos is calling TRISIS, is an attack framework built to interact with Triconex Safety Instrumented System (SIS) controllers.
If the controllers did not trip, the malware could have sat on the system until it got the code to attack.
If the shut did not occur, the malware “could have sat there for years,” Kling said.
Nation State Consistency
Researchers did not attribute the attack to any person, group or country, but did say they believe the activity is consistent with a nation state preparing for an attack.
They also feel the attackers’ long-term objective was to develop the capability to cause a physical consequence.
“In the context of the shutdown, there is an important nuance,” said Blake Johnson, industrial control systems security consultant at Mandiant. “The shutdown the controllers experienced we do not believe to have been the intent of the malware executed. The evidence we have to support that is only a subset of the controllers targeted successfully entered the fault state. The fault state they encountered was due to a diagnostic failure associated with the attacker logic replicated amongst the main processer blades of the Triconex controller chassis. The malware itself did not intend in that moment to cause a fault or exception. It in fact included error handling logic to attempt to recover the controllers to a running state in case it did cause a fault or exception. Ultimately, some of the targeted controllers in the environment did continue to run after being successfully targeted. The two that went into fail safe because of the MP diagnostic failure, those two controllers going into fail safe was sufficient to bring down the process because of site specifics.”
“The point is attackers are taking a new attack vector, which is a new way of looking at this,” Kling said. “They are getting bolder and more sophisticated. This is not a disgruntled employee, it was geo political.”
In the incident, the attacker gained remote access to an SIS engineering workstation and deployed the TRITON attack framework to reprogram the SIS controllers. During the incident, some SIS controllers entered a fail safe state, which automatically shut down the industrial process and prompted the user to initiate an investigation.
The Mandiant researchers are continuing to look into why some controllers were successfully attacked with others were not.
“We believe it to be a combination of existing conditions on those two specific controllers plus the attackers’ actions that caused those to fail and not the others,” Johnson said.
It does appear as if the attacker was going after the end user which happened to be using the Triconex system.
“The malware implemented the Triconex communication protocols and would only work on Triconex,” Scali said. “Obviously, this type of attack is relevant to all SIS’. You can rewrite malware to implement different communications protocols to attack other SIS controllers. In this particular situation, the malware had a framework for interaction with Triconex. It is relevant the SIS was targeted and Triconex happens to be the SIS deployed at the facility that was targeted. This is just what happened to be there in the targeted facility.”
“We just happened to be the system in place at the customer,” said David Smith, media relations director for Schneider Electric North America.
In the incident, the system shut down, and Smith said “there was nothing beyond that.”
Attackers Messed Up
Mandiant researchers said they feel the attacker inadvertently shutdown operations while developing the ability to cause physical damage for the following reasons:
• Modifying the SIS could prevent it from functioning correctly, increasing the likelihood of a failure that would result in physical consequences
• TRITON was used to modify application memory on SIS controllers in the environment, which could have led to a failed validation check
• The failure occurred during the time period when TRITON was used
• It is not likely that existing or external conditions, in isolation, caused a fault during the time of the incident
“It is important to know there was reconnaissance capability within the framework they developed and they did leverage that reconnaissance capability, but they also leveraged the payload capability and to me that promotes what occurred beyond just reconnaissance because they did deliver a payload,” Johnson said. “The indication was shutdown was not their goal. It would have been much simpler to shut down the system without developing the fully functional logic payload they developed. Reconnaissance was their mission and shutdown was not their immediate goal.”
In other words, the attackers messed up their mission.
“The way to think about it is the attacker wanted to be in place and wanted to have the SIS compromised and the DCs compromised so that at a time in the future they could cause a disruption, but in the process of trying to get in place to do something like that there was this shutdown that was inadvertently triggered,” Scali said. “If they just wanted to shut it down, there would have been far easier ways to do that.”
FireEye feels the attack was from a nation state. The targeting of critical infrastructure as well as the attacker’s persistence, lack of any clear monetary goal and the technical resources necessary to create the attack framework suggest a well-resourced nation state attacker.
Once on the SIS network, the attacker used their pre-built TRITON attack framework to interact with the SIS controllers using the TriStation protocol, researchers said. The attacker could have caused a process shutdown by issuing a halt command or intentionally uploading flawed code to the SIS controller to cause it to fail. Instead, the attacker made several attempts over a period of time to develop and deliver functioning control logic for the SIS controllers in this target environment. While these attempts appear to have failed due one of the attack scripts’ conditional checks, the attacker persisted with their efforts. This suggests the attacker was intent on causing a specific outcome beyond a process shutdown.
The TRITON attack tool was built with a number of features, including the ability to read and write programs, read and write individual functions and query the state of the SIS controller, the researchers said. However, only some of these capabilities were leveraged in the trilog.exe sample.
The TRITON malware contained the capability to communicate with Triconex SIS controllers and remotely reprogram them with an attacker-defined payload. The TRITON sample Mandiant analyzed added an attacker-provided program to the execution table of the Triconex controller. This sample left legitimate programs in place, expecting the controller to continue operating without a fault or exception. If the controller failed, TRITON would attempt to return it to a running state. If the controller did not recover within a defined time window, this sample would overwrite the malicious program with invalid data to cover its tracks.
Schneider Electric released a statement saying:
“Schneider Electric is aware of a directed incident targeting a single customer’s Triconex Tricon safety shutdown system.
“We are working closely with our customer, independent cybersecurity organizations and ICS-CERT to investigate and mitigate the risks of this type of attack. While evidence suggests this was an isolated incident and not due to a vulnerability in the Triconex system or its program code, we continue to investigate whether there are additional attack vectors. It is important to note that in this instance, the Triconex system responded appropriately, safely shutting down plant operations. No harm was incurred by the customer or the environment.
“Triconex user documentation contains detailed security guidelines and recommendations on how to protect Triconex systems from attack. We strongly encourage all our customers to follow these recommendations regarding product use and security, as well as apply and follow industry-recognized cybersecurity best practices at all times to protect their installations:
• Ensure the cybersecurity features in Triconex solutions are always enabled
• Never leave the front panel key position in the “Program” mode when not actively configuring the controller
• Ensure all TriStation terminals, safety controllers and the safety network are isolated from the rest of the plant communication channels
“Also, review and assess your site’s cyber preparedness. Schneider Electric is a proponent of the NIST Cyber Security Framework and is ready to assist should this be necessary.
“The Schneider Electric Product Security Office continues to work with ICS-CERT and will update this advisory as more information becomes available.”
FireEye said users looking to defend against an attack of this nature should consider these controls:
• Where technically feasible, segregate safety system networks from process control and information system networks. Engineering workstations capable of programming SIS controllers should not be dual-homed to any other DCS process control or information system network.
• Leverage hardware features that provide for physical control of the ability to program safety controllers. These usually take the form of switches controlled by a physical key. On Triconex controllers, keys should not be left in the PROGRAM mode other than during scheduled programming events.
• Implement change management procedures for changes to key position. Audit current key state regularly.
• Use a unidirectional gateway rather than bidirectional network connections for any applications that depend on the data provided by the SIS.
• Implement strict access control and application whitelisting on any server or workstation endpoints that can reach the SIS system over TCP/IP.
• Monitor ICS network traffic for unexpected communication flows and other anomalous activity.
Separate or Integrated
“One of the recommendations we made is maintaining separation between control and safety and between DCS and SIS,” Scali said. “We think the incident really highlights that risk of integration and the lack of independence of the SIS.”
For end users, one of the messages moving forward is constant vigilance in monitoring all systems.
“Even in the presence of strong segmentation or an air gap, this really makes the case for more extensive monitoring of SIS environments both on the end point side at the engineering workstation as well as the network side that can detect some of this reconnaissance activity,” Johnson said. “Asset owners that think they are safe because they have that level of segmentation they should really spend time to understand the attackers’ tools, techniques and practices and ask how they would detect those actions in their SIS environment.”
Safety systems sometimes can end up being a crutch, but this incident proves that should not be the case.
“One argument would be ‘we don’t have to worry so much about cybersecurity because we have a SIS, a safety system,’ ” Scali said. “So, somebody compromises the control system and messes around that is not a problem (because there is a safety system). Cybersecurity can also be a single point of failure on the control side and the safety side.”