What Leaders Need to Know and Ask to Ensure a Strong Security Profile
By Marc Ayala and Jeff Jensen
Among phrases sure to catch the attention of most all oil and gas executives: Enhanced asset utilization, production optimization, accelerated resource recovery and capital efficiency. Keep these moving in the right direction and greater profitability and market capitalization will surely grow. But one phrase that might escape their concern could endanger any initiative: Network security.
In fact, executives could be doing a grave disservice to their shareholders and their own fortunes if they choose to ignore this threat or to delegate their understanding of how it can undermine the safety of people, production and property at the core of a thriving oil and gas enterprise. What they need is the knowledge to evaluate the nature of this risk and to ask informed questions about their companies’ defenses against it.
Oil and gas industry executives must stay informed of cyber security threats for two reasons: The energy sector is by far hackers’ top target and a cyber attack on their own facilities can potentially have serious impacts on operations and profitability as well as grave consequences for the life safety of personnel and nearby communities.
How important is network security? Consider this: Of the top 16 security targets designated as critical by the U.S. Department of Homeland Security (DHS), cyber attacks on the energy sector in 2013 were 59 percent of 256 total attacks deemed serious enough for its Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) to investigate. That was three times the number of attacks on critical manufacturing facilities, the runner-up, and 30 times the number of attacks on government facilities.
And how frequent are those attacks? With hackers automating their network assaults, one can occur every few minutes until a penetration occurs. During one session on network security led by a Siemens expert, he prefaced his presentation by opening a new, working web server connected to the Internet with its Modbus TCP/IP port 502 exposed. At the end of his remarks, he checked the web server’s security monitoring software and found 35 attacks had occurred from all over the world – all in just one hour.
OT vs. IT Security
IT professionals have plenty to worry about in defending against cyber attacks on their companies’ enterprise networks. These are what connect people with each other, via email, web collaboration tools and even voice communications, and also with information, via various company databases, customer relationship management (CRM) tools and so forth. After all, malware, data theft and corrupted data or devices can disrupt user productivity and even a company’s transactional capabilities.
But in that environment, no one ever suffers an injury or worse.
This is one of the biggest differences between enterprise network security and industrial network security. If a hacker, whether a deliberate saboteur or a teenage malcontent, penetrates an industrial network and disrupts critical processes or controls, especially automated life safety protections, someone could get seriously or even mortally hurt. That’s why DHS set up ICS-CERT to reduce risks associated with control systems-related incidents and mitigation measures.
Industrial Network Realities
Aside from the critical life-safety security distinction of industrial networks, they differ from non-industrial enterprise networks. First, industrial control systems (ICSs), which include supervisory control and data acquisition (SCADA) systems, are by definition connected to networks. These ICS and SCADA networks are often linked to enterprise networks, which have external-facing vulnerabilities that can open doors for hackers. Wireless SCADA systems, often operating from remote locations using public IP addresses, are also vulnerable to attack, accessible via their wireless media, including cellular, 900MHz radio, satellite and microwave.
Industrial networks must often operate 24×7, in real or near-real time and require 99.9 percent uptime or better (99.99 or 99.999 percent in the case of public communication networks). In contrast, enterprise IT networks typically must operate on a best-effort basis (so a break in one part of the network forces routers to send data packets down alternate paths) and be available during “business hours.” Point is, the disruption risks of a security breach in an ICS or SCADA network can be much greater than for an enterprise IT network.
In the past 20 years, industrial automation and control systems have become more vulnerable to cyber security intrusions, primary among them are:
• The increasing mobility of workers, which has created greater demands for 24×7 remote network access for engineering, operations and technical support personnel, sometimes leading to less secure network connections and security practices.
• Growing use and integration of commercial and open source technologies, such as Windows and Linux operating systems, SQL databases and Ethernet protocols, all of which a hacker can exploit by opening back doors for the same malware that can infect enterprise IT systems.
• Proliferation of “how-to” documentation and actual code on the Internet, which has lowered the bar for the technical competencies needed to hack industrial control systems.
• Integration of a company’s legacy plant systems with its enterprise systems by interconnecting industrial and corporate networks – and external third parties via the public Internet. Not only does external connectivity create vulnerabilities, but the integration also introduces ambiguity within companies as to which group – enterprise IT or process engineering – owns responsibility for overall cyber security.
Another set of security issues with industrial networks involves their evolution from early patchworks of electrical relays or antiquated microprocessor controllers and manually monitored indicator lights, trips and breakers. While those legacy systems might work well enough to operate relatively simple processes even today, they likely lack proper security controls. Nonetheless, they most likely end up connected to modern distributed control systems (DCSs) that feature the latest programmable logic controllers (PLCs), which are micro-computers using Windows or Linux which connect over industrial Ethernet to human-machine interfaces (HMIs). In turn, these HMIs are often accessible anywhere in the world via PCs or touchscreen tablets and smartphones – by legitimate DCS operators or by hackers exploiting the vulnerabilities in the connections between old and new systems.
With modern ICS, SCADA and DCS networks, infiltrations can occur from any of three sources:
1. Top-down from the corporate and data zones (Zones 4, 3 and 2)
2. Bottom-up from the field and safety/control zones (Zones 0 and 1); and
3. “Sideways” from external sources, either via the Internet, remote operations and facilities or remote
Lower Security Risks
Companies can find plenty of information to help guide their efforts to harden and secure their industrial control systems. Three internationally recognized ICS security standards, which can provide excellent starting points and guidance, are IEC 62443 / ISA99, NIST 800-82, and NERC-CIP.
These standards boil down to three steps: A current state assessment; hardening the environment, physical and logical; and ongoing vigilance.
They incorporate what’s known in security as the “defense-in-depth” model. This involves dividing a security deployment strategy into layers, with the most critical systems protected by multiple levels of security.
Every security risk mitigation effort for an industrial control system must start by evaluating the current state of its security by conducting an assessment. Here are some questions to consider:
Does a network’s borders correspond to its physical borders? They should. For example, if the user locked down the SCADA server and its software in an effort to prevent tampering with its configurations and data, is the server itself securely located to prevent unauthorized access to its network ports, removable media drives, keyboard and mouse?
Where are the network’s security zones and conduits? An industrial control system should have distinct functional zones that separate the field device control layer from the SCADA remote monitoring layer. In turn, these should end up separated from the DCS control layer – and more importantly, separated from any layer of safety-critical systems. Finally, the DCS and safety-critical system layers must end up separated from the enterprise IT layer. All those layers should communicate with each other only via carefully prescribed and secure conduit connections. And all those layers need to be separate from all external connections, each of which should also end up carefully prescribed and secured.
What and where is each connection within the industrial control network? This step helps identify what’s known as the network attack interface. Look for internal local area network (LAN) connections and wide area network (WAN) connections; remote connections with distant sensors and operating facilities; internal wireless connections, including Internet connections; modem or dial-up connections (yes, they do still exist); and external connections to third-parties, such as business partners, vendors and regulatory agencies. All connections should end up catalogued in detail and their current security measures noted, especially their firewall protection and update status.
What devices and software applications have connections, and what are their functions? This step helps identify what’s known as the software attack interface. Similar to the step above, all hardware devices – HMIs, PCs, servers, wireless access points, phones, even printers and video surveillance cameras – must end up catalogued along with all their operating system versions, software applications and the port numbers that each device uses to communicate. All current security measures should end up noted as well as their status regarding updates and patches.
Who is in charge of securing the industrial control network? For quite a few companies this might not be clear – yet it’s critically important. ICS, SCADA, DCS and safety systems typically evolved with industrial and process engineering teams in charge. During that time frame, enterprise IT teams had their hands full with rationalizing the corporate IT landscape. That left a large gray area of unclear responsibilities and sometimes adversarial relationships between the two groups. It can be a classic human story of in-fighting going on while the barbarians are tearing down the city gates. Executives – especially CEOs, CIOs and CISOs (chief information security officers) – need to recognize this phenomenon and put one qualified company person or team in charge of securing the industrial control system, in concert with enterprise IT and plant or production management. This person or individuals should have clear cyber security roles, responsibilities and authority to formulate and enforce well-defined security governance policies for managers, system administrators and end-users.
How vulnerable are the network and software attack fabrics? After identifying all the elements subject to cyber attack, the next step is to conduct penetration testing, to determine each one’s vulnerability. This can be a time-consuming, tedious task for large systems comprising hundreds of connections and components or more, but it’s necessary to fully assess the strengths and weaknesses of ICS, SCADA, DCS and safety networks, which are only as strong as their weakest component.
Due to the nature of these critical, real-time production systems, it’s vitally important that any penetration testing occur in a lab environment and not on the production system itself. With extreme care, caution and coordination, production, operations and process safety management will need to conduct a risk analysis and develop contingency plans – with executive management sign-off – before doing any penetrating testing or modification of a live control system. Failure to do so could have grave consequences not only for the personnel and property of a plant or production site, but also for the people and property in surrounding communities. This is why any third-parties selected to help with ISC, SCADA, DCS and safety system security testing or modification must be exceptionally well qualified and experienced in the engineering and workings of your system(s).
Hardening the Environment
A thorough assessment will reveal all existing and potential security holes and everything that needs strengthening. In effect, the list of all a system’s security shortcomings will become its punch list for action. Depending on how long that list is, levels of prioritization can come into play to close the worst vulnerabilities as soon as possible.
Assigning Security Access Levels (SALs) to each element can help with prioritization. Next steps in this stage would include:
Remove, disable or disconnect anything not needed. An assessment will probably uncover elements never needed but ended up installed as part of bigger installation or became unnecessary over time. If you find any unnecessary connections, disconnect them. If any unnecessary software applications or default network services end up discovered, remove or disable them.
Establish a security strategy based on a layered “defense-in-depth” model. After eliminating unnecessary connections, and software, what’s left needs protection. Ensure physical and logical security coincide, with strict access privileges for all users, providing access only to what they need to do their jobs. Logs should be kept for all accesses and video surveillance placed on the locked-down physical confines of network elements – HMIs, servers, routers and switches. All firewalls should be up-to-date. Full security features should be turned on in all hardware devices, operating systems, software and hardware devices.
Document, document, document. The catalog of a system’s network and software attack surfaces should be the start of a full documentation of its security. This should include “as-built” system architecture diagrams showing all elements, their locations, their functions, their governance and their connections with other elements.
Add to that written policies and procedures for: establishing, updating and terminating user accounts; upgrade and patch management policies, procedures and assigned responsibilities for all firewalls, devices and software applications; and scope, frequency and procedures for conducting security audits and penetration testing. All this documentation itself should have version and access controls, plus always be backed up to an offsite location, so it’s available by alternative means if the system goes down due to a cyber attack or some unrelated disaster.
Communicate, communicate, communicate. During the hardening stage, many employees and other stakeholders will become aware of what’s going on, so it’s important to communicate with them the reasons for doing so, let them know who is in charge of the effort, advise them of any changes in their day-to-day work as a result, and set proper expectations for their roles in supporting the effort.
After hardening a company’s ICS, SCADA, DCS and safety networks, the heightened protection will begin degrading over time without ongoing efforts to maintain security levels. To watch for and respond to apparent and actual attacks, and to conduct periodic security audits and tests, a user should:
Establish response teams to identify and evaluate potential attack scenarios. The designated person or team in charge of industrial network security should identify potential attack scenarios and then convene the core stakeholders into a rapid response team. Each team member needs to imagine, describe and document the potential impact on his or her function should a security attack succeed, as well as what mitigation measures to take. Roles and responsibilities need to be assigned and contact information shared in a central place. The team should meet at least annually to reacquaint themselves with each other and with their risk and mitigation scenarios. It’s a good idea to conduct exercises that assume the worst-case scenario has occurred, which can provide the team with practice.
Conduct periodic audits and penetration testing. The frequency of audits and penetration testing depends on how critical an industrial control system is to a company’s functioning or the life-safety of personnel and surrounding communities. Obviously a nuclear plant would require much more frequent audits and systems testing than a dairy products plant. Any industrial facility, however, should conduct an audit and systems testing no less frequently than once a year. Notably, audits often overlook evaluating the currency and relevancy of existing documentation. That’s why it’s important to review and update documentation. If production lines are frequently reconfigured, with consequent changes made to their control systems, then mini-audits should then end up conducted to avoid introducing any unintended system vulnerabilities.
The ultimate goal of securing industrial control systems and networks against cyber attacks is to ensure their reliable and safe operation.
Oil and gas industry executives can make tremendous progress in reaching this goal by initiating a thorough systems assessment and needed hardening, then putting in place a formal watchdog process governed by designated, well-qualified people with the knowledge and authority to create and enforce policies and procedures.
Doing so will cost money and time, but it will be one of the most important investments that oil and gas operators can make in the safety and well-being of their people, production and property.
Marc Ayala is the former senior technical advisor at system integrator, Cimation and Jeff Jensen is an application engineer at Siemens Industry, Inc.