Sometimes a Contentious Path, But There is a Solution
By Gregory Hale
Oftentimes there is a misconception on just what the true mission is for a manufacturer.
In the manufacturing automation sector the simple fact is the goal is for the company to safely and securely produce their product in a productive and profitable manner. The route it takes to get there is often very complicated, but the manufacturer needs to generate revenue and keep turning a profit. To keep adding positive results to the bottom line, manufacturers need to ensure everyone is working from the same play book to ensure avoidable errors do not occur.
With the industry losing over $20 billion a year in revenues from preventable safety and security incidents, it goes back to truly knowing, and understanding, what that focus is all about.
“One of the things we are trying to do with our systems is we are trying to mitigate those losses. We are doing that by gathering more information and more data into the systems so they can make more of the right decisions,” said Kevin Staggs, engineering fellow ACS Advanced Technology at Honeywell. “Because of that, it is actually crucial control engineering and IT come together. I mean our whole goal in the industry is for automation to help reduce these incidents and the cost of these incidents and downtime. A well implemented system can do that.”
That means the well implemented system will need to have process engineers and IT specialists working closely to avoid a cyber incident, whether it is accidental or malicious, forcing a system to come tumbling down. In today’s manufacturing automation environment where an unplanned downtime costing millions of dollars can occur as easily as a worm slithering its way into a system, the level of communication needs to be smooth and seamless. Just how can a manufacturer get to that point?
Starting with Standards
Staggs feels one area that can help clarify communications is through standards. He said existing IT standards cover at least 80 percent of the security issues on the plant floor and remaining 20 percent is part of an ongoing manufacturing automation industry standards process.
“In the IT space most companies work with the ISO 27000 series of governance standards that exist,” he said. “There needs to be an understanding those standards can not completely be applied to an industrial automation control system. There are things that are done on IT systems that will cause failures to occur in an industrial automation control system (IACS).”
Those differences, though, end up covered in the ISA99 standard, which is still under development.
“They are not quite done yet, but there is enough out in ISA99 that I believe the two sides can use the information, especially in the ISA99.02.01. It is entitled ‘Setting up a security management system for your industrial automation and control system.’ Because the title says ‘setting up,’ it actually talks about all the different things you need to take into consideration that are specific to an IACS.”
That is also where working together comes into play as “those things actually require the skill set that is available from the IT organization so it is important to understand there is the ISA99 standard,” Staggs said. “You need to understand that standard and then you need to understand a lot of the skill set to implement that standard comes from IT.”
Most of everything process control and IT need to know is a working tool on the IT side.
“If you look at what is in ISA99 and compare it to what you already have in your governance procedures that you set up in your company through the 27000 series, you find you have between 80 to 90 percent are what you are doing on the IT side,” Staggs said.
Another positive aspect of ISA99 and with NERC-CIP is the standards spell out there must be buy in from the top.
“I think that is why you see that in NERC-CIP and in ISA99.02.01 you must have executive management sponsorship and you must define who that is and who is responsible and what the reporting structure is before you can get on a path to success,” Staggs said.
Shift in Thinking
For process control engineers to understand they don’t have to reinvent the wheel to secure the plant floor, is easier said than done though. There are companies just now coming to the realization the two sides have more in common than they think.
“I think where there is maturity in the organization they actually do believe that. I have seen organizations that have gone through the exercise of mapping what they already do and what their IT organization’s procedures are and where they need to create new procedures,” Staggs said.
“We have done that at Honeywell ourselves in our chemical plants. What we found in our chemicals plants is we needed to write an IT exception spec for the things we need to do differently on our control systems. Those specs are not that large. It basically says here are the things we need to do that are unique for our control systems and the reason we need to do this is because in the control system world our focus is on uptime, availability and/or integrity,” he added.
Having the two disciplines work together is one thing, but they will need to speak the same language and have a set of criteria to act as a basis for a solution.
The process side speaks the language of HMI’s, PLCs, DCS and SCADA, while the IT folks’ lingo deals with Active Directory, email, ERP, among other things. The difference between the two languages may be like French and Spanish – they are similar in some cases but very different in others.
To keep everyone speaking the same language, there are a series of security practices that focus on the preservation of confidentiality, integrity of data and availability of the network. These services offer key elements to aid in securing a process control network including assessment, design, implementation and network security management.
These security practices for a manufacturer include:
- The need to establish a baseline of the current security processes, procedures and safeguards used to protect a process control network from external threats. The baseline allows for the creation of a documented set of recommendations that outlines procedures and changes to the environment that will mitigate identified vulnerabilities.
- A detailed design of the security infrastructure connecting a process control network to a company’s plant information network. This includes a list of all hardware and software needed to implement the design, along with detailed descriptions on how to configure and use the security infrastructure. Initial steps in the design process include the collection of needs, analysis of requirements, including adherence to policies and regulations.
- Implementing the design, which should result in a secure control network infrastructure.
For systems in the process control industry it always comes down to availability and integrity.
“Availability and integrity depends on what business you are in,” Staggs said. “In the chemical industry, I would say integrity comes first because if things go wrong, things can really go wrong in a very bad way. Availability is a very, very close follower because you want your uptime as close as 100 percent as you can get it.”
This is where the conflict comes in because the IT side’s main focus falls in the confidentiality aspect.
More than Talk
Just how can the two sides get together? Staggs and Rick Kaun, director of network security solutions for Matrikon agree communication is the key, but sometimes it needs to go beyond that.
“Engineers need to come to understand and respect the mentality IT folks have for maintaining the IT assets, which are the computers, the routers and servers and firewalls and utilize the expertise where it can be utilized,” Staggs said.
“Once the communication starts, the beginning of an understanding of where skills can be utilized starts to occur. But it is usually not friendly at first and it can be a very combative environment when it first starts,” he said. “But if you can get by that you can usually build a relationship. I have seen relationships grow in some of the companies I have worked with and that is good.”
“In the end it is all about communicating,” Kaun said. “You need to have a forward thinking company.”
“You need someone that is responsible and someone that is accountable,” he added. “For instance when it comes to patching, you need to make one guy responsible and another that is accountable. This also forces teamwork.”
Another approach Kaun talks about is creating a matrix that is a more formal way to ensure communications. That is where RASCI comes into play. RASCI stands for Responsible, Accountable, Supportive, Consulted, Informed.
“RASCI identifies and documents roles, responsibilities, and reporting relationships within an organization or for a particular project,” Kaun said.
“By creating a RASCI chart, organizations are then able to successfully complete tasks by clearly outlining the individuals within the organization that will be involved in enforcing, completing, supporting, or approving each task. In the case of combing critical production assets and traditional IT controls a RASCI chart creates the best possible sharing of key skill sets as they apply to unique applications and environments. This sharing thus maximizes security while simultaneously leveraging existing technologies and skills thereby saving the organization duplication of effort and reduces costs,” he said.
“It is a bit of work,” Kaun said, “but it can point out who should be doing what.”
Another way Staggs suggests is to cross mingle the departments.
“One thing that has worked in a couple of companies is they actually assigned some of their IT folks to the process engineering department and they actually had a process engineer assigned to the IT environment,” he said. “It’s a way to help the two organizations gain some respect for each other and learn what each other does. I have seen that done quite successfully in a couple of companies.”
“There are activities driving the two groups closer together,” said Shawn Gold, global solutions leader of open system services for Honeywell Process Solutions. “It all depends on the attitudes of both teams. If the IT group comes across to the controls group and is willing to talk and listen, then that is a step in the right direction.”
Gold agrees on exchanging personnel between the two departments.
“What works best is if someone in IT is seconded to the controls group so they get to learn what is important (from the controls perspective),” Gold said. “They appreciate what is important (to engineering) and after about six months they learn to appreciate the controls side. (When an employee is seconded the employee temporarily changes job roles within the same company or transfers to another organization for an agreed period of time.) Hopefully, when someone that was seconded rotates back into the IT group, they will be able to share their experiences and help enlighten them with what controls is all about.”
Hear the Same Story
Kaun also pointed out making sure they are all in the same room promotes more harmony. “I know of a fellow that runs the process control group (where he works) and now he has IT going to all the same meetings.” This way they are all hearing the same thing at the same time. No hidden agendas.
“One of the major differences is in the IT world the focus is on intellectual property and intellectual property protection and in the control world the focus is really on physical equipment protection, safety, protection of the environment and in many cases there is an education process that needs to occur and it is a bidirectional education process that the it folks need to learn about the equipment and the priorities of maintaining and managing the physical equipment,” Staggs said.
“Both sides have to understand the demands are different and immediate,” Gold said. “They both have to want to make it work.”
Whether a person works in IT or in process engineering, they all have a story about hwo the two groups get along. Some positive and some not so positive. There is no getting around it, today’s open systems are ripe for cyber incidents and the two sides need to play off of each other’s strengths to succeed in the company’s basic principle of producing more product and boosting the bottom line.
“A proper working relationship between the two organizations is the only way to go,” Kaun said. “I don’t think any one person can be knowledgeable in both areas. Security specialists need to work side by side with the production specialists. They need to share information and then you can live happily ever after.”
Gregory Hale is the editor and founder of Industrial Safety and Security Source (ISSSource.com). You can reach him at firstname.lastname@example.org.