By Gregory Hale
It is a given the supply chain could be a security issue for end users, but talking about it and doing something about are two different things.
While the issue can be a major problem, it is also manageable.
“The more astute users are thinking about their supply chain and making sure they remain secure,” said John Cusimano, director of industrial cybersecurity at safety and security integrator, aeSolutions. “There are standards in IEC 62443 that address supply chain in a couple of ways. The one that would address third party suppliers like system integrators making sure they are properly securing the information they may have about your control system is covered in the standard 2-4 which talks about suppliers’ security practices. While it is not perfect, it identifies the requirements asset owners should put on their suppliers to make sure they are at least following best security practices regarding the information they are being entrusted with. If people are looking for guidance on what to do about securing vendors or suppliers that would provide a good starting place.”
“In addition, there are multiple things users can do from a procedural point like making sure their suppliers are properly trained in security and they are following best security practices. There are also technical things like requiring if they are storing information offsite, they are storing it securely so it is encrypted. There are other things where a lot of companies will not allow a vendor to come in and plug in their laptop. That is a very common practice for a supplier to come out and plug in their equipment. This integrator can go from one company to another and then to another and then plug in at home. There is no good way to know if that machine is compromised or not. There are a few options around that. If you are going to allow someone to plug in their laptop, you have them plug in behind a security device that will not only inspect traffic, but inspect the laptop and perform some type of network access control. Or the facility can supply a workstation with all the necessary engineering tools and forbid the third party from using their own laptop. Other things I have seen is the use of quarantine stations, which would work more with USB sticks suppliers use to upload software updates. They would first scan it on a quarantine machines because they are not connected to the process control network and they would verify it is free from malware.”
In the end it is all about knowing who and what is on the system.
“I have seen several cases where sites did a root cause analysis and determined they got infected by a third party service provider,” Cusimano said. “In more than one case, I have seen systems delivered from factory acceptance testing (FAT) to the site where it already had a virus preinstalled, not deliberately of course. When you do a FAT there is a lot of activity on those systems so security controls tend to be turned off to facilitate a check out. There hasn’t been a regular practice where systems are checked out for security before they are shipped. That is a growing requirement we are starting to see where there is a specification where the user wants a cyber acceptance test following a FAT so they can be assured a system will be clean before it gets delivered to them.”
Allowing integrators or suppliers into the system just boosts the threat level.
“The fact a lot of suppliers demand a connection to the industrial environment in order to supply their support and services also increases the risk factor,” said Yoni Shohet, chief executive SCADAfence. “I think the problem with the supply chain and even the user’s own personnel is they are not aware of the potential problems they could cause. Therefore, the industrial environment and operators need to ensure where there are human operations inside the environment they need to make sure all operations are according to their own protocol. No one can perform an operation first of all without the user knowing and second without them being able to moderate and validate what is occurring inside the environment, this also goes for their own personnel.
“Second level suppliers need to build a better system to allow the end user to validate certain devices they are getting are using basic security measures to make sure they were not manipulated.
“You need to instill some kind of internal strategy that would allow external contractors to connect into the environment,” Shohet said. “I am not only talking about a secure VPN connection but also once they have connected to the environment, what type of operation are they allowed to do? Are you able to monitor an external contractor or technician that is performing an upgrade on a PLC. If you not able to monitor these types of activities, then how can you monitor the potential consequences of these environments? So you have to have the control and understanding of the operations being performed inside you network. This way you can minimize the unchecked solutions and devices that are being connected inside the network without the proper measurement and proper procedure.”
Attackers today are looking for the easiest access to systems and that includes anyone in the supply chain.
Fix Easy Stuff First
“My first advice to the ICS owner operator is to go after the low hanging fruit, which is securing and validating the software patch process. After all, the bad guys will likely be going after the same low-hanging fruit in their attack plans,” said security controls expert, Eric Byres.
“Securing that part of the supply chain starts by demanding that suppliers create a proper software distribution process with signed software. For starters, never download software from a supplier’s insecure HTTP site — software should always be distributed via a secured HTTPS site so you know it is the authentic site, not some fake site and so the package isn’t tampered with in transit. And all software must be digitally signed by the vendor. But for many products, the validation will be a manual process, but one that needs to be done.
“Once the new software is received, it needs to be validated both for the correctness of its digital signature and for its impact on the process. This is nothing new — a decade ago at ISA Expo 2006, an engineer from the pharmaceutical company Astra-Zeneca gave a presentation on how patches should be staged over a period of months. Astra-Zeneca started by testing new patches ‘Central Test Platforms’ which are isolated and sacrificial systems to see how the patch impacts those machines. Company Training systems or active spares are a good option for such platforms. Once the patches had been installed and have operated without any ‘issues’ for a number of days (4 days in the case of Astra-Zeneca) they would be distributed to the next system in line. Only after a month of successful installations, would the patches be installed in the truly mission critical systems.
“Patch issues could be something obvious, such has the patch crashes the control system, or it could be more subtle, such as the control system starts trying to access external web sites. Interestingly, systems infected with some versions of Havex immediately tried to connect to the website ‘sinfulcelebs.freesexycomics.com’ but no one noticed this inappropriate behavior for almost 14 months. An active patch management strategy like the one proposed by Astra-Zeneca would have detected the Dragonfly-infected software in the first few days,” Byres said.
In the end ICS owners and vendors need to agree on processes that will build a verifiable trust relationship between the end-user and the supplier.
“Industry already does this for requirements like safety or reliability,” Byres said. “If I purchase a PLC with a claimed -40C to +70C temperature rating or a 40 year MTBF, I don’t immediately retest the PLC to make sure it meets those specifications. I trust that it will because I trust the vendor. And I trust the vendor because I have the ability to confirm the vendor has followed a recognized method, such as MIL-HDBK-217 or Telcordia for MTBF. If I don’t completely trust the supplier, I can even ask to see the test reports.
The key here is for vendors to create a verifiable process to demonstrate their product is secure. As part of this overall process they should have processes that demand their suppliers provide them with secure software and so on down the line. Typically, these process are called Security Development Lifecycle (SDL) and companies like Honeywell and Schneider Electric already have them in place. End users should not just take vendors at their word — they should ask to see the validation reports that document the SDL process being used to ensure product security.”