These Vital Disciplines Need to Secure Plant from Cyber Incidents
“No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as if a manor of thy friend’s or of thine own were: any man’s death diminishes me, because I am involved in mankind, and therefore never send to know for whom the bells tolls; it tolls for thee.” — John Donne (1572-1631)
By Gregory Hale
In the wee hours early one morning not too long ago a firewall ceased operating at a major oil company refinery, and while there were no dramatic explosions or oil spills, the end result was five hours of lost production and a cyber incident that caused unplanned downtime.
The shutdown was not the result of a malicious attack, but rather a simple miscommunication between IT and the process engineering folks.
That was the type of “minor” incident that was never reported publicly, but after analyzing the costs associated with those lost five hours, that small, simple incident ended up costing the oil company $1 million. Labor costs, lost production, remediation, and meetings, it all adds up. Time truly is money.
For a major oil company $1 million may appear to be peanuts, but that was just one refinery, on one day, during one month of the year. That one small accidental cyber incident can occur anytime and the costs do add up.
Stuxnet has captured the interest of those in and around the manufacturing automation industry these days and deservedly so, it was a targeted attack at an industrial control system. What doesn’t get reported on and what occurs more frequently are the small cyber incidents that occur daily that didn’t have to happen. At some point there is usually a breakdown in some form of communication and the manufacturer ends up paying the price in lost production and revenue.
With the manufacturing automation industry losing at least $20 billion a year due to safety and security incidents, one way to clamp down on revenue eluding a manufacturer’s grasp — and the bottom line -– is to ensure a tight fit and cohesive working relationship between engineers on the plant floor and IT specialists. In an incredibly dynamic environment, these two vital areas need to secure the process from any kind of cyber incident — whether accidental or an attack — and keep it chugging along, while being able to hop between a menagerie of disparate systems and meeting regulatory compliance.
In this era of open systems connected across the enterprise allowing a digital view into the plant floor, all departments need to work together to help a manufacturer squeeze as much out of a process as humanly possible. No singularity. Just solid working relationships that ensure a secure plant floor. While process engineers and IT may seem to be working in opposite directions, now more than ever they need to be able to work together or the results could be disastrous.
“Deploying an open system allows plant personnel to utilize a variety of business software applications that typically fall within the domain of business IT,” said Shawn Gold, global solutions leader of open system services for Honeywell Process Solutions. “However, business IT personnel are focused on enabling users to perform their business activities while maintaining the LAN and protecting the company’s intellectual property, where process control regards human and plant safety as a primary concern.”
Just take a look at one of the Big 3 automakers, where one control engineer lost his entire test control network in the middle of a multi-day test.
When the network first started acting odd and crashing, the engineer called IT and asked if they were doing anything to cause this. The quick, curt reply was “no way.”
But the problems persisted. His network was still sluggish and unstable so he did a little troubleshooting of his own to see if he could figure out what was going on.
He found huge amounts of traffic hitting all of his devices. He called IT back and asked again if they were doing anything odd or unusual. Again, the response was negative.
He then gave them the address that was the source of the bulk of the traffic. After hemming and hawing, IT said as it turned out one of their guys was running a utility to check who and what was out on the network.
It was essentially ping-type traffic. Since there was no firewall or even a router (which doesn’t forward broadcast traffic) between the office network and the control network, the PLCs, and HMIs, among others got overwhelmed and started locking up.
Once again, IT doing its job and control engineers doing theirs. No communication and goodbye test control network. A cyber incident from a friendly source.
Benefits Outweigh Threats
There is no doubt process control engineers profit from the economies of commodity hardware and open system software platforms.
With these benefits, though, comes the exposure to the threats of hackers, worms and viruses. There are safeguards that neutralize these threats. That is why in these turbulent days of living in fear of the next Stuxnet-type of attack, control engineers and IT specialists need to work together to keep processes up and running and free from any kind of cyber incident. That, however, has not always been an easy relationship.
A quick glance at IT’s goals and it is easy to see why the two disciplines come to loggerheads. IT wants to keep confidentiality by preventing unauthorized disclosure of information, integrity of data by preventing unauthorized modification, and availability by preventing unauthorized destruction or denial of services. It is the CIA (Confidentiality, Integrity and Availability) approach.
While control engineers focus mainly on availability and reliability. They need the system to stay up so they don’t lose view of the process. They are not as concerned about confidentiality.
“With IT, it is the CIA mentality, however on the plant floor, it is CIA upside down,” said Rick Kaun, director of network security solutions for Matrikon.
“They both have a different focus and ultimately that is where the head butting comes in,” Gold said. “Back in the days of the mainframe, there was some arrogance from IT saying they knew best and what they said goes.”
True Service Partner
Those days are gone as the two groups are starting to understand the mission at hand is to eliminate cyber risks, however each often tug at opposite ends of the rope.
That is where a third party integrator can come into play. By employing best practices, they are able to understand the situation in a dispassionate way to make sure the solution remains secure.
“In today’s control world, the control group will sometimes use some back door methods around IT to get support,” Gold said. “When it comes to security, we don’t want that. We are quite adamant in working with the IT group. The controls group really should; it is to their advantage to do so.”
The idea of thinking through security like they would a safety solution just is not the first line of thought for manufacturers. The concept of thinking security all the way through has not quite sunk in yet.
“In the safety culture most companies now have a near miss type of culture, where there is any kind of near miss that occurs there is an understanding if you ignore the incident, it could lead to more major types of incidents, said Kevin Staggs, engineering fellow ACS Advanced Technology at Honeywell. “Security has a near miss environment as well and if you can identify those near misses early and get to the root cause and figure out what they are, you can certainly get yourself in a situation where you might be able to avoid a major security incident in the future.”
“Just look at the emergence of safety cultures,” Kaun said. “It takes time to get everybody to accept changes. Look at safety belts and smoking. Until security becomes a part of the culture on a day to day basis, it will continue to be a daily battle.”
Making a Checklist
While it may be more time consuming at first, working with each other and understanding the needs, priorities and the complexity of the process control environment is all a part of that daily battle.
Indeed, according to survey by Check Point and the Ponemon Institute of over 2,400 IT security administrators around the world, managing complex security environments is the most significant challenge facing organizations today, with over 55% of companies using more than seven different vendors to secure their network.
“Open systems is just the tip of the iceberg,” Kaun said. “You have the complexity of the plants with an increase in technology to get more out of the industry gray hairs that know more about all the individual devices than anyone else, then you have plants running at 110 percent capacity, where do you think changing a password fits in that environment? It is explosive.”
That being said, Kaun said today “the relationship is much more cooperative. For the first time, I have seen engineering and IT working together.”
The two areas have to understand each other’s strengths. After all, IT is not the bad guy. Their security expertise will help the control group stay up and running.
“IT becomes more aware of measures to take before the control people do,” Gold said. “That is where they need to work together. They should give the control engineers guidance on what the IT community is comfortable using.”
Kaun related a story about when he came in as a third party to oversee a security solution.
“We were once going into a corporate policy review and the corporate IT people felt they didn’t need us,” he said. “After a few choice words, they said we have all the right policies in place and they felt we were impinging on their turf. I looked at the plant manager and he said ‘yes, we do have polices, but we are not using them.’ At that point we were able to work together.”
That just goes to show that once a communication barrier breaks down, it allows the plant floor and IT to move forward and secure platforms throughout a plant in addition to meeting industry standards and regulations like NERC-CIP.
Plan in Place
Part of securing the plant floor includes a cyber security plan that requires: Risk and vulnerability assessments, hierarchical networks with access restrictions at each level, a high-security model deployed on personal computers and servers, physically separated process control and enterprise networks with limited access points, security hotfix and an antivirus deployment strategy, and disaster recovery.
In addition to properly managing cyber security, it remains vital to understand there could be a potential for a disaster that could impact critical data. Put aside the sophisticated Stuxnet worm, even a simple hardware failure could seriously jeopardize critical data, and therefore data recovery plans must be in place that allow for rapid recovery through an automated backup and restore application.
With open technology platforms, control system networks in facilities such as power plants face exposure to the same security threats facing corporate networks, but with the added safety considerations associated with the process industries. These threats include:
- Indiscriminant, potentially destructive intrusions, such as viruses and worms
- Network spoofing and denial of service attacks that have performance impacts and potential safety issues
- Eavesdropping and password cracking that are threats to confidentiality
- Malicious threats, such as data tampering, impersonation and packet modification
Corporate IT groups manage these issues and challenges for enterprise networks. Along those lines, IT and engineers need to communicate and understand each other’s specialized skills to secure process control management processes, services and tools.
It ends up being an exercise in communication that has to occur on a daily basis so plants or refineries don’t end up having to endure five hours of down time for a very preventable cyber incident.
“When it comes to applying technology to delicate legacy plant environment,” Kaun said, “you have to let the IT folks use their knowledge and work hand in hand to maximize uptime for the facility.”
Gregory Hale is the editor and founder of Industrial Safety and Security Source (ISSSource.com). You can reach him at firstname.lastname@example.org.