By Ron Brash
Let’s face it, with all the talk about the SolarWinds attack, supply chains have suffered compromises several times in the past.
GitHub ended up exploited to push malicious changes. Attackers stole certificates, which ended used to sign malware, integration software development kits (SDKs)/machines were lost, and development systems are among the dirtiest of them all.
Being optimistic, SolarWinds was an unfortunate cybersecurity incident, but it may lead to positive change at an increased rate and awareness. Although we still do not have complete information, the dust has settled enough in the SolarWinds attack to draw some implications and recommendations.
The SolarWinds Orion platform product (affected versions are 2019.4 through 2020.2.1 HF1) ended up compromised via the vendor’s software development process. Attackers were able to insert signed malicious components into the product and which distributed out unknowingly through authorized distribution channels. The tampered product ended up installed as a standard business process by customers who had invested in SolarWinds and trusted their product’s integrity.
Similarly, around that time, other organizations such as FireEye or Microsoft suffered compromises as part of a related and coordinated campaign.
As far as the incident goes, this is what is most important to know:
- The software development cycle of one of the most prevalent software companies (SolarWinds) in the United States suffered compromise to insert malicious code into a signed version of the software
- Some organizations ended up compromised and acted upon and others remained sitting in a compromised state
- The attackers demonstrated a determined attack can get through relatively sophisticated processes
Challenges
This incident pointed out challenges that could easily extend to other vendors such as Cisco (e.g., when they were rumored to have devices tampered with in transit), Microsoft, or even the FOSS community (e.g., targeting NPMs), where:
Companies developing products must utilize a secure development lifecycle (SDLC) to securely engineer, develop, and maintain a product.
Even if a vendor may use an SDLC, it is never completely foolproof. A determined attacker is always a risk.
The easiest companies to target are the ones with the most ROI for attackers. Supply chain is merely another word for third-party risk; this should be on nearly every company’s conscious assuming a reasonably developed risk register. And the same goes for insider risks.
When you are talking about risk, here are four elements for asset owners to address supply chain risks:
- Assume any software installation can end up used as an attack vector with the goal of managing that supply chain with multiple layers to validate and verify security. You cannot trust it implicitly.
- Focus on identifying and protecting your most valuable assets and ensure they have extra protection.
- Build a programmatic cybersecurity program that includes multiple levels of security and robust incident response to a supply chain incident, assuming a software product may suffer compromise.
- Add multiple layers of security in supply chain management.
Code Signing
Code signing is a mechanism where you imprint your organization’s digital signature via cryptographic primitives to state a piece of software came from your organization. In theory, code signing to the end customer means the code is exactly as a vendor built it, and trustworthy (assuming no one stole the signing keys, or compromised the process). However, bugs may still exist, and potentially disgruntled employees or malicious persons may have installed backdoors if they built the artifact insecurely, or it contained weaknesses before it ended up signed.
Code signing is an important part of any supply chain security program. However, even if another SolarWinds type event occurred, code signing while being part of a security solution, is not infallible. In other words, security takes getting a lot of things right, and even then, it might not be right even if it includes:
- People-Process-Technology (PPT)
- Technological/architectural diversity
- Secure development pipelines
- Organizational security and training
- Incident and sudden teardown readiness when a product/component is compromised
Code signing is a useful security tool that has vast benefits, but it was not the cause of the SolarWinds incident. The vendor didn’t have control of their development pipeline, and as with any technology or solution, there is always a case for why you use it and take active measures to implement it properly.
Dealing with Issues
For a product development company, there are issues it will have to deal with.
- To date, malicious code made its way into product development pipeline for distribution, and peer review missed it. The catch is, tight control over a supply chain is expected by customers. It’s a shift left philosophy, but code signing is just part of a security strategy; it is not something to rely solely upon.
- Always identify and protect the “crown jewels” of your development infrastructure: Pragmatically, this incident implies if you have “crown jewels,” you should protect them adequately with respect to their scope and power. If your organization uses management platforms to control and monitor the security of your organization, you should not expect them to be a silver bullet, but as part of a holistic security program. Software or hardware in any shape or form can be a risk unless consciously managed.
- Build a programmatic security effort: Train developers to have awareness and implement security, leverage best practices and code enhancement strategies, take care of keys/credentials (even those of your developers), maintain and secure the systems used in the development process (including workstations or virtual machines), and be aware your customers may make you a target.
Build Comprehensive Security Program
The SolarWinds software incident is evidence that even large, well-funded companies are at risk of compromise. As a result, asset owners need to build a comprehensive program of security including configuration management, patch management, network segmentation, incident response, etc. to limit any potential compromise.
In the case of the SolarWinds attack, there is nothing the asset owner could have done to prevent the compromise. The asset owner could reduce the likelihood of a successful compromise and impact by:
- Ensuring security processes end up followed and have evidence of sufficient certification and legal documentation
- Helping mitigate the circumstances by thinking critically of solutions deployed or by monitoring for anomalous conditions
- Performing security validation testing not just at factory or site levels, but continuously and periodically
- Choosing solutions that end up secured and designed by functional purpose (e.g., the management zone should never have communicated across zonal boundaries in this case)
- Preparing security and management teams to act on any anomalous conditions vs. implicitly trusting (e.g., no management server should be beaconing to random DNS domains)
- Having the means, training, and processes to quickly recover or nip an issue in the bud
Incident Response Vital
One of the key elements of a robust cyber security program is incident response. One of the biggest fallacies of the remediation notes by SolarWinds, Microsoft, and CISA is an “update” may remediate this type of issue had a compromised binary been found. What I mean by this is the compromised software is often a delivery mechanism for more “bad things” (e.g., it’s a method for dropping additional malware), so no software patch could truly know what the original malware may have left or added to the affected system, but it would at least stop the original compromised binaries from adding bad things through the same vector.
Therefore, a patch in this case, even if it’s the most impressively engineered solution, cannot ensure the validity and integrity of your industrial environment because the original attackers (or copycats) may have added other malicious “treats” to your systems. This means no update will absolve you of that risk.
This does not mean you shouldn’t patch the problem, nor is it an excuse to avoid updates in the future, but it means systems affected by a supply chain compromise require more than a patch, and they need to be:
- Isolated
- Archived for forensic purposes
- Accounts/credentials reset for all affected assets (and or controlled by compromised software)
- Rebuilt from a backup before the compromise (including adjacent ones)
- Treat the event as an incident – it is vulnerability, credential, response, and recovery management all in one
- Chase down legal, procurement, and legal for post-mortem steps
Unfortunately, this tough love is for all parties involved, and I do not wish this on any vendor or asset owner. However, there were lessons learned here, and I think that is the silver lining.
Ron Brash is director of cybersecurity insights for Verve Industrial. This was an excerpt from Ron Brash’s blog post.

