By Andrew Ginter
How much does your plant firewall cost? How secure is it? These are two questions for which most security practitioners do not have answers.
The cost to purchase a firewall is clear. Operating costs though, tend to be dominated by difficult-to-track labor costs. As for the security of the firewall, there are a few answers, but one will become clearer at the ICSJWG 2012 Fall Meeting in October and take in my session “13 Ways Through a Firewall,” which will include live demonstrations.
To complicate matters on the cost side, firewall costs increase as you work to make the devices more secure. Take an extreme example: Say you configure your firewall to accept all incoming connections and all outgoing connections so it effectively becomes a router. You no longer need to manage remote access accounts, because you need no such accounts, everyone who wants to get through the firewall can do so already without an account. You no longer need to manage rules, because you need only the two rules which allow all traffic, and so on. The firewall costs you nothing, because as you’ve configured it, the device provides no security either.
So what does a real firewall cost? One which provides at least some degree of perimeter security? Let’s take the example of a plant firewall separating operations and control systems networks from business networks at a large facility — a refinery or a chemicals plant.
To understand capital expenses of firewalls, you need to know how many and what class of equipment you are purchasing. A conservative estimate for firewall equipment sized for a large plant is $10,000 per unit. At a large facility, high-availability (HA) operation is a common requirement, so you are buying at least two units. High-end security advice in the ISA99 and NIST 800-82 documents recommends two layers of firewalls with a DMZ between them, separating business networks from control system networks. Couple this with the HA requirement and you would need to purchase four firewalls, two pair from two different firewall vendors at a cost of about $40,000.
Not all large sites follow this advice though, so let’s assume you have branched your DMZ out of a single HA pair of firewalls separating your plant network from your business network at a capital cost of $20,000. This set-up means you have only one layer of firewall between your business and control networks, but it is what many large sites deploy.
Firewalls most commonly recommended for the business/plant interface are not just firewalls, but encompass an array of other functionality, including virtual private networking (VPNs), inline network intrusion detection (IDS) and intrusion prevention (IPS) systems, inline anti-virus (AV) scanning, and a host of other functions. The inline scanning functions are all signature-based. For the firewalls to remain as effective as the day you purchased the firewalls, you must also purchase a signature subscription. A bundle of hardware support, software support and signature updates generally costs 60%-80% of the purchase price of the firewall equipment annually. That’s right, you pay most of the cost of purchasing the firewalls every year in support and signature subscription costs.
To prevent these costly artifacts from becoming mere routers takes labor. You must create a set of rules which permits business-essential information to flow between control system components and your plant DMZ, and between that DMZ and your business network, without allowing other kinds of connections to form and information to flow. You must provide access to those DMZ systems for only the control system and business computers and users which require such access. And as business needs change, you must update your lists of users, list of machines, rules, and other configurations.
You may be tempted to create a set of rules which allows every last piece of infrequently-needed data to move through your firewalls, but you must resist that temptation. The more paths you configure through your firewall, the less secure your perimeter becomes. Every path through a firewall is a potential attack vector.
For maximum security, you should configure only the paths you need through the firewall today. If you need something different tomorrow, then tomorrow you can remove the paths you no longer require tomorrow and add in the paths you need then. For maximum security you should be deeply suspicious of any path you set up through the firewall which is intended to facilitate remote control of control system components. The physical process should be controlled from within the secure physical perimeter of the physical process, not from insecure locations distant from the process.
If this sounds like a lot of work, you are right, it is. Keeping plant firewalls reasonably secure is labor-intensive. How much labor? Most organizations do not track their costs accurately enough to be able to answer the question. One quick way to estimate your costs is to ask a third party to give you a quote for managing your firewalls for you. Credible industrial security vendors will quote you between $1500 and $5000 per firewall per month. For the two firewalls in our scenario, that adds up to somewhere between $2,500 and $10,000 per month, depending on the model of firewall, depending on complexity of your firewall configurations, depending on the management vendor, depending on the amount of documentation your firewall management policies require, and depending on whether you can negotiate a discount for managing an HA pair vs. two separate and independent devices.
Now, if that seems like a lot of money, you may be asking “what do I get for these monthly payments?” Industrial firewall management services generally include:
• Emergency repair – coordinating the replacement and re-configuration of failed firewalls,
• Keeping firewall firmware and signatures up to date, though the latter can sometimes be automated, depending on whether a firewall management server or the open Internet is routable from the affected firewalls,
• Tracking announced firewall Zero Day vulnerabilities and implementing compensating measures until updated firmware is available,
• Processing routine changes, according to a defined change-management /workflow system, including remote access accounts and passwords, and VPN accounts and passwords,
• Responding to support requests for problems in accessing equipment protected by the firewalls,
• Designing, reviewing and testing more complex changes, including rule set changes,
• 24×7 monitoring of firewall logs, investigating anomalies in those logs, escalating those anomalies to the site and supporting site personnel in real-time incident response as well as forensic investigations, and
• Retention and back-ups of firewall logs and configurations per corporate policy and regulatory requirements.
Fundamentally, firewalls are software artifacts, even though they look like hardware. Incorrectly configured, firewalls are not secure at all. To keep firewalls configured correctly, and to monitor those configurations and their behavior for security anomalies is labor intensive, especially at a large site. An internal team carrying out these tasks as effectively as an outsourced team of specialists is going to incur labor and overhead costs very similar to the outsourced solution.
For example: the security people in many of the large facilities I talk to no longer have a clear idea of what data passes through their firewalls, or why that data passes. Yes, you can look at the firewall configurations and your records of why you set up those configurations as you did, but those configurations typically contain hundreds of rules providing data to thousands of users — a real “spaghetti code” of connectivity. Trying to keep such systems secure when you are no longer confident of the consequences of changing individual rules is a daunting task.
Adding It Up
Back to costs – what is the tally thus far?
• CapEx: A cminimum of $20,000 every few years, to purchase a pair of plant-class HA firewall pairs, and you get to purchase them from two different vendors to minimize the risk of a single vendor’s vulnerabilities allowing an attacker through both layers of firewalls. So you get to learn two different vendors’ configurations and rules as well.
• OpEx: A minimum of $12,000 annually in hardware, software and signature support costs per year. A minimum of $2,500 per month in firewall managements costs, or $30,000 annually, for a total of $42,000 annually. And this really is a minimum — typical costs are substantially higher.
To be fair though, when you purchase the firewall hardware new, the first year’s support costs tend to be included in the purchase. What other costs might there be? Well arguably, we should also count:
• Costs associated with periodic audits, vulnerability assessments, risk assessments, and penetration tests, all of which tend to spend a disproportionate fraction of time and effort on firewall perimeters,
• Costs associated with managing remote access — keeping laptops secure, keeping personnel with remote access capabilities trained as to how to keep their laptops and workstations secure, and so on,
• The cost of security incidents which could have been prevented, had a more secure perimeter protection mechanism been used, and
• Costs associated with compensating controls — some best-practice guidance, such as the draft NERC-CIP v5, is recommending stand-alone Network Intrusion Detection Systems as a separate layer of defense behind perimeter firewalls, because of the security problems with all firewalls, and these systems incur their own capital and operating expenses.
These costs typically add up to be at least as large as the firewall capital and operating costs we have already examined. Credible security from plant firewalls at large plants is not cheap.
There is little sense in dwelling on the costs of firewalls unless there is an alternative — without an alternative, the cost of firewalls is simply the price you pay to achieve some degree of perimeter protection for your plant networks.
One alternative: There are some security experts recommending industrial server replication via hardware-enforced unidirectional gateways. When at least one layer of perimeter protection between your control systems and the Internet consists of unidirectional replication, cyber risks to control system components drop measurably. The hardware-enforced gateways protect absolutely against business network insider attacks, Internet-based attacks, and even errors and omissions on the part of corporate IT security.
What does a unidirectional project cost? All-in CapEx costs for unidirectional server replication projects start at around $50,000 for a small project and range up to $250,000 for large installations comparable to the refinery or chemical plant example above. This includes up-front unidirectional replication system costs, systems integration services charges and all other up-front project costs. Hardware and software support costs are typically 10-20% of CapEx costs annually. On-going labor costs are negligible — there is almost nothing to adjust on unidirectional server replication technologies. Better yet, no adjustment to the replication software can introduce vulnerabilities, because it is the hardware which prevents attacks through the network perimeter.
The Bottom Line
Most sites report that operating cost savings recoup the costs of unidirectional server replication projects within 12-18 months of deployment. After that payback period, these sites not only enjoy the benefits of stronger perimeter protection, they save money as well.
So ask yourself again — are you really getting what you pay for from what you spend every year on plant firewalls?
For a more detailed treatment of this topic, join us for the upcoming Waterfall webinar “Stronger Than Firewalls, and Cheaper Too.” Details of the webinar are at the Waterfall website.
Andrew Ginter is the Director of Industrial Security at Waterfall Security Solutions, the makers of hardware-enforced unidirectional gateways.