Editor’s Note: This is an excerpt from the Practical SCADA Security blog at Tofino Security.
By Eric Byres
It is time to take a look at the good, the bad and the ugly details of patching as a means to secure SCADA and ICS systems. And to begin, let’s suppose you could install patches without shutting down the process (for example, through the staged patching of redundant controllers).
In a landmark study of the patches for post-release bugs in OS software, between 14.8 percent and 24.4 percent of all fixes are incorrect and directly impact the end user. And if that’s not bad enough, 43 percent of these faulty ‘fixes’ resulted in crashes, hangs, data corruption or additional security problems.
What’s more, patches don’t always solve the security issues they were designed to address. Kevin Hemsley, a member of the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT), said in 2011 ICS-CERT saw a 60 percent failure rate in patches fixing the reported vulnerability in control system products.
Good Patches Can Cause Issues
Most patches require the shutdown and restart of the manufacturing process. Some can also break or remove functionality previously relied on by the control system. For example, one of the vulnerabilities the Stuxnet worm exploited was a hardcoded password in Siemens’ WinCC SQL database.
At the time, Siemens was widely criticized for not quickly releasing a patch to remove the password. However, customers who took it upon themselves to manually change the password soon discovered that many critical control functions depended on this password to access accounts. In this case, the “cure” was even worse than the disease.
Patching Often Requires Experts
Another ugly truth about patching is that the process itself often requires staff with special skills to be present.
For example, the vulnerability exploited by the Slammer worm in January 2003 actually did have a patch (MS02-039) released in 2002. Unfortunately, this didn’t help an oil company with numerous production platforms in the Gulf of Mexico. The company started rolling out the patch in the summer of 2002, but issues with server restarts required Windows experts to be present during patching. Since very few of these experts were safety certified for platform access, most platforms were still not patched when Slammer hit six months later.
When There Are No Patches
Of course, you can only use patches to fix vulnerabilities if the vendor has created a patch. Unfortunately, this is the exception rather than the rule. At the SCADA Security Scientific Symposium (S4) in January 2012, Sean McBride noted less than half of the 364 public vulnerabilities recorded at ICS-CERT had patches available at that time.
Some accuse the vendors of indifference or laziness, but there are many factors that prevent the quick release of a patch.
In 2010, a major ICS vendor said internal testing of a mission critical product had revealed security issues. Unfortunately, these vulnerabilities were part of an embedded OS supplied by a 3rd party. The OS supplier refused to address the vulnerabilities, and so the ICS vendor (and its customers) faced a situation where patching was not possible.
In a 2011 case involving another ICS vendor, vulnerable backdoors were found in a PLC by an independent security researcher, who publically exposed them. The vendor designed a patch to remove backdoors, but then learned these backdoors were widely used by troubleshooting teams for customer support. To complicate matters, the company’s quality assurance (QA) process for product changes required four months to complete. This meant that even if customers were willing to sacrifice support for security, they were faced with a four month window of exposure while they waited for the proper testing of patches to be completed.
Users Choose Not to Patch
My last example highlights a core problem with a patch-based strategy for control system security. Many customers simply don’t want to run the risk of degrading service and increasing downtime. The vendor noted in the previous example privately told me they have a 10 percent patch download rate for released patches.
My own experience with an ICS security product confirms the reality of low patch acceptance in the field.
In September 2010, we released Tofino Industrial Security System version 1.6. This upgrade addressed a number of security and performance issues and was offered to all registered users at no charge — if downloaded within 30 days. Initial acceptance was low, so we repeated the offer for an additional 30 days. After two months, only 30 percent of users had downloaded the free upgrade. And that doesn’t necessarily mean they all installed it.
Planned Patching is Good
Let’s be clear — patching bugs is an important process for any control system. And patching for vulnerabilities is critical for good security. But the IT strategy of reactive, continuous patching on a monthly or weekly basis just won’t work for SCADA and ICS systems. Patching in a hurry is just plain dangerous.
SCADA/ICS vendors face multiple issues when trying to create “quick” patches — they have to consider safety and QA requirements that often delay patch releases. In other cases, a reasonable and safe patch just isn’t possible.
SCADA/ICS customers face similar concerns. And quite frankly, who can blame them for not wanting to increase downtime or expose their critical controller or server systems to safety risks?
Patch support for legacy products is also an issue – many expect a control product to operate for 20 years, putting it well outside the typical IT support window. Finally, as noted in the Slammer worm example, patches can require significant staff resources to install safely.
So create a patching plan that works for your process environment. Make sure that it includes processes for proper tests and change management controls.
Just don’t expect patches to be a quick fix for your control system’s security problems. If you do, you may discover new problems that are worse than the bugs the patches cure.
Eric Byres is vice president and chief technology officer at Tofino Security. Click here to read the full version of the Practical SCADA Security blog.