An attack on the electric grid in the western U.S. earlier this year leveraged a known vulnerability in a firewall where the operator experienced brief outages that controlled communications between the control center and multiple remote generation sites and between equipment on these sites.
The outages were five minutes or less and had no impact on generation. The affected firewalls were all perimeter devices that served as the outer security layer, according to the report from the North American Electric Reliability Corporation (NERC) in its “lessons learned” report.
An investigation into the March incident found the communications outages were due to reboots of the firewalls at each of the sites. The entity’s system monitoring tools also provided notification of the firewall reboots. These records show the firewall reboots occurred over a 10-hour time period with each firewall showing offline status for less than five minutes.
After an initial internal investigation, the entity decided, in order to fully characterize the nature of the reboots and the potential causes, the firewall manufacturer should review logs. Subsequent analysis determined the reboots were initiated by an external entity exploiting a known firewall vulnerability, according to the report. After receiving this notification, the entity initiated their event reporting procedure as dictated by their cybersecurity incident response plan. Along with identifying the cause of the reboots, the firewall manufacturer offered a firmware update that would address the vulnerability.
The entity assessed the update details and determined it was appropriate to deploy immediately, according to the report. It first deployed the firmware patch on a firewall within a non-critical environment at the entity’s control center that would not impact operational assets and monitored the changes for any adverse effects. After seeing no adverse effects, the entity deployed the firmware patch at an operational generation site that night. After monitoring traffic in the production environment overnight and early the following morning, the entity deployed the update to all remaining assets that had common hardware with the firmware vulnerability.
After completing mitigation efforts to address the immediate risk posed by the firmware vulnerability, the entity performed an internal assessment to identify internal process improvements to reduce the likelihood of an event with a similar cause from happening again, according to the NERC report. Given that a firmware update to address the exploited vulnerability had been released prior to the event, the entity’s process for assessing and implementing firmware updates was reviewed.
Based on this review, the entity decided to implement a more formal and more frequent review of vendor firmware updates that would be tracked within internal compliance tracking software. It should be noted that the entity was already working to develop internal procedures to support this process; however, these were not completed or being practiced at the time of the event. Additionally, the entity now utilizes firewall rules that restrict allowable traffic to the minimum required to operate the assets.
“This attack highlights the vulnerabilities of the IT-OT convergence that is happening throughout the industry and the need for better oversight and protection,” said Barak Perelman, chief executive of network security provider, Indegy. “The fact is that malware can penetrate through either network and traverse across to impact all industrial networks, devices and IoT. NERC offers excellent advice to industrial operators, but we suggest that this is only the starting point, the low watermark and not the end goal. IT tools don’t speak OT, given their mostly undocumented and proprietary protocols, and OT monitoring tools don’t help against cyber threats, as they are designed to monitor industrial processes and not to detect cyber threats. In addition, we have known for over a year that nation-states and rogue factions have gained ‘red button’ functionality to disable critical infrastructure. Organizations that wait for incidents to happen before adding ICS cybersecurity may get caught, either directly or in the cross-hairs, and they will require more effort and higher costs to manage the incident after it occurs.”
Even in cases involving low-Impact assets, an entity should strive for good cyber security policies and procedures, like:
• Follow good industry practices for vulnerability and patch management.
• Reduce and control your attack surface.
• Use virtual private networks.
• se access control lists (ACLs) to filter inbound traffic prior to handling by the firewall; minimize the traffic through a denial by default configuration with whitelisting for the allowed and expected IP addresses. Limit outbound traffic similarly for information security purposes.
• Layer defenses. It is harder to penetrate a screening router, a virtual private network terminator, and a firewall in series than just a firewall (assuming the ACLs and other configurations are appropriate).
• Segment your network. Restrict lateral communication to necessary and expected traffic to reduce the impact of a breach.
• Know your exploitable vulnerabilities so you can pursue fixes.
• Monitor your network.
• Employ redundant solutions to provide resilience and on-line maintenance capabilities