If you didn't know what a software supply chain was - let alone a software supply chain attack - you do now. As someone who's been researching, studying and talking about this attack vector for the past seven years, the malicious December 2020 attack on SolarWinds' Orion leading to public and private sector breaches has been fascinating - but not unheard of. Yet industry attention switched swiftly to this attack vector as the latest "what happened" story and "how do we not end up like SolarWinds" curiosity.
Let's look first to "what happened." In this particular case, malicious code was inserted somewhere in the build process that SolarWinds has for its Orion product. My colleague Ax Sharma detailed a bit more about this last Monday. Keep in mind, more has unfolded since then - but the general story has remained the same. According to Microsoft:
The addition of a few benign-looking lines of code into a single DLL file spelled a serious threat to organizations using the affected product, a widely used IT administration software used across verticals, including government and the security industry. The discreet malicious code inserted into the DLL called a backdoor composed of almost 4,000 lines of code that allowed the threat actor behind the attack to operate unfettered in compromised networks.
The fact that the compromised file is digitally signed suggests the attackers were able to access the company's software development or distribution pipeline. Evidence suggests that as early as October 2019, these attackers have been testing their ability to insert code by adding empty classes. Therefore, insertion of malicious code into the SolarWinds.Orion.Core.BusinessLayer.dll likely occurred at an early stage, before the final stages of the software build, which would include digitally signing the compiled code. As a result, the DLL containing the malicious code is also digitally signed, which enhances its ability to run privileged actions — and keep a low profile.
Ax continues:
"The SolarWinds Orion attack started with attackers intruding through malicious code that was implanted into SolarWinds Orion instances via trojanized updates. These updates delivered a backdoor known as SUNBURST and Solorigate, which were deployed on systems running Orion platform versions. The impact? Roughly, 18,000 customers automatically pulled these malicious updates."
It seems the malicious code was injected to the Orion build by the perpetrators gaining access to Solarwinds' build systems, as evidenced by the certified status of the release. The attackers found a way in, and compromised SolarWinds' build infrastructure to spread the malware further downstream the supply chain.
Once we understand a little bit about what's happening, the next topic folks in our industry pursue is "How do I avoid becoming the next SolarWinds?"
While every marketing department within a cybersecurity vendor is focused on this topic, the remedy is only partly delivered by InfoSec groups. I would argue that software supply chain hygiene, governance, and security have more to do with Development than InfoSec teams.
Here are a few tips for software development teams to know about software supply chains. Knowing them, might help you (and your InfoSec teams) minimize the risk from a software supply chain attack similar to what we saw at SolarWinds.
As SolarWinds shows, a software supply chain attack can either be aimed at you executing tainted third-party code, or having the tainted code run in your customer environments. In the SolarWinds case, the latter was the aim. To begin to defend against these mediums, it is important to know what is in your software.
Just like DevOps teams map out their value delivery chains, awareness starts by mapping out your software supply chains. As the old adage goes, "you can't manage what you can't see."
Look beyond the code you write to the code you don't write. There are a whole host of open source packages and containers that your development teams are pulling down from the internet to accelerate innovation. It's impossible to keep a running list in your head. For example, the average Java development organization relies on software from over 3,500 open source projects, including 14,000 unique component releases. For the average JavaScript developer, 90,000 npm packages packages are downloaded annually. Organizations need to map out where they are getting their OSS and third-party packaged code, containers, and infrastructure as code. Keeping track of the suppliers, supply lines, and code packages is the first step to managing software supply chains.
This is where Sonatype Platform can help. Organizations can store, manage and track all of the open source software components, containers and build artifacts used by their developers with our Sonatype Nexus Repository solution. Access to a central inventory of artifacts used in development can provide immediate insight as to whether or not newly discovered vulnerable artifacts were ever used in the organization. But just knowing that artifacts are present in your repositories (the local warehouses of software supply chains) is not enough.
Open source software components and containers are constantly flowing through development pipelines, and not all of them are created equal. We've long reported that 1 in 10 open source components being downloaded contain known security vulnerabilities that can represent breach points for adversaries. Just as quality is inspected across supply chains of physical goods, Sonatype Lifecycle inspects and tracks the quality and security of components and containers flowing through software supply chains. Directly integrated with the tools developers use to build, integrated, test, and deploy application code, Sonatype Lifecycle automatically alerts developers to troublesome components and offers guidance on updating them to safer and higher quality alternatives. Developers, governance, and security teams are no longer tasked with manual review processes, but aided by automated reviews and tracking of artifacts to ensure proper security and quality standards are enforced with little to no manual effort.
Next, you need to map out your build and release platforms. What's in your DevOps pipeline tech stack and who has access to it? Is your infrastructure publicly accessible? How do you manage updates? The machinery and the code it produces should be tamper-free, meaning they're not erroneously injecting malicious code somewhere along the way.
SolarWinds isn't the only company to experience a software supply chain attack that looks to be focused on the build process and perhaps even the build tools themselves. Before SolarWinds, in May 2020, there was Octopus Scanner which was caught by GitHub for having IDEs injecting malicious code as part of the build process. Similarly, Gitpaste-12 (which should now be Gitpaste-30) leveraged trustworthy sites like GitHub and Pastebin to host itself and maliciously infect users. Finally, publicly accessible build infrastructure has been exploited for several years for several purposes, like the Jenkins Cryptomining campaign.
While attacks through the build process may not seem that common, it's clear, they're becoming more and more of a prime vector. Why? Because as investments shift from protecting networks, to applications, to open source code, to containers, adversaries shift tactics to the next easiest and lucrative targets.
In short, define your entire software supply chain (where do you source open source software packages, containers, IaC, software updates, build and release pipelines). Without this step, you can't see, let alone prevent, attacks on your supply chain.
In our sixth annual State of the Software Supply Chain Report, we documented a 430% increase in software supply chain related attacks. There were almost 900 attacks that we documented between June 2019 and June 2020. The increase is truly astonishing and it's clear that they’re only going to get worse.
In a "normal" breach pattern, time between a vulnerability disclosure and a breach is about three days, when it comes to open source software packages. This is when a vulnerability is discovered, appropriate processes are taken so project owners can remedy the issue and the known vulnerability is then shared publicly along with a fixed version of the code.
In a case where adversaries are injecting malicious code into containers or open source packages, those breaches can occur as soon as the code is deployed into production and into your customers environment. Adversaries know what the malicious code or backdoor is, can spot that malicious code being distributed and can then initiate their attack path.
Over the past two years, I have worked with Gene Kim and Dr. Stephen Magill to understand how high performance software development teams improve security outcomes. A few things are clear from this research.
High performance teams demonstrate:
15x more frequent deployments
26x faster detection and remediation of vulnerable OSS components
51% more likely to keep an inventory of all OSS and third-party software components
77% more likely to automate the analysis and approval of OSS dependencies
High performance software development teams have mapped their software supply chains, maintain automated checks on the quality of software components and packages moving through them, and update the components to the latest releases on a regular basis. We also know that teams that update their code more often, generally stay more secure. BONUS: high performance development teams also have happier developers with greater job satisfaction.
What does that mean in practice? It means that awareness of software supply chains, the infrastructure supporting them and the components moving through them, can lead to better management of them. Better managed software supply chains result in lower risk, better performance and happier developers.
This is just a start, but by understanding basic best practices like the above, you can protect your organization from software supply chain attacks. To learn more about my research into software supply chains and what the best software development teams are doing to better understand and mitigate risk within their supply chains, you can download our sixth annual State of the Software Supply Chain report here. The report specifically details the use of open source containers, adversary attacks, government policies and regulations when it comes to software supply chains - and most importantly, what best practices development teams are pursuing to try and minimize the risk around software supply chain attacks.