Last week, software testing firm Codecov disclosed a noteworthy security incident that gained the attention of the U.S. federal government investigators.
Codecov has over 29,000 enterprise customers, including reputed names like GoDaddy, Washington Post, Royal Bank of Canada, and Procter & Gamble.
On April 1, 2021, Codecov became aware that threat actors had gained unauthorized access to their Bash Uploader script and altered it without raising any obvious red flags.
The issue occurred due to an error in Codecov's Docker image creation process that enabled the actors to extract sensitive credentials and modify the Bash Uploader script.
Consequently, this allowed the actors to potentially exfiltrate sensitive information from Codecov customers' continuous integration (CI) environments, such as environment variables containing keys, credentials, and tokens, outside of Codecov's infrastructure.
The system breach remained undetected for over two months, which is when a Codecov customer alerted the company of a discrepancy between the shashum (hash or "file fingerprint") of the Bash Uploader script present on the website and the (correct) shasum listed on Codecov's GitHub.
Essentially, the attackers had replaced the IP address of Codecov's servers with their own in the Bash Uploader script:
This meant data such as a Codecov user's system environment variables, meant to be used for legitimate operations, instead got uploaded to the attacker's IP address, shown here.
Environment variables can contain a plethora of information about a system. From simple information such as the PATH variable, current username, and working directory, to sensitive API keys, tokens, and credentials/passwords used by applications.
Image: Example Apple environment variable configuration file with key information redacted in gray
In what is now being compared to the SolarWinds software supply chain attack, Codecov incident is yet another testament to developers and development tools being targeted by adversaries to conduct sinister "upstream" attacks on infrastructure. This allows the perpetrators to distribute malicious code downstream to a large number of users.
The news comes shortly after Sonatype had discovered a novel Linux and macOS malware hidden inside a counterfeit npm component named after Browserify, also used by a large number of developers.
Codecov's integration with GitHub makes it attractive to open source developers on projects, as it can seamlessly leverage the tool's software testing functionality for their applications.
Given the vast adoption of open source tools, the popularity and reputation of any particular tool (not just Codecov) also makes it an attractive target for adversaries. Attackers find value in either directly attacking and altering the tool itself — as in this case, or engaging in typosquatting or brandjacking attempts to target the tool's users (in this case, the developers).
The GitHub infrastructure has itself been abused in cryptomining attacks due to how GitHub Actions work.
With an increasing number of incidents targeting the tooling, infrastructure, and environments where developers live and work, there's a lot of work ahead on everyone's part to help curb these attacks. But, as my colleague Derek Weeks pointed out around the SolarWinds incident, software supply chain hygiene, governance, and security might have more to do with development than InfoSec teams.
Derek shared a few tips for developers to keep in mind, that I wanted to highlight again.
A software supply chain attack can either be aimed at you executing tainted third party code, or having the tainted code run in your customer environments. To begin to defend against these mediums, it is important to know what’s in your software. As the old adage goes, "you can't manage what you can't see."
For starters, look beyond the code you write to the code you don't write. There are a whole host of open source packages and containers that your development teams are downloading from the Internet to accelerate innovation. And there are far too many to keep a running list in your head.
For example, the average Java development organization relies on software from over 3,500 open source projects, including 14,000 unique component releases. For the average JavaScript developer, 90,000 npm packages are downloaded annually. Organizations need to map out where they are getting their open source software (OSS) and third-party packaged code, containers, and Infrastructure as Code (IaC). Keeping track of the suppliers, supply lines, and code packages is the first step to managing software supply chains.
This is where the Sonatype Platform can help. Organizations can store, manage, and track all of the open source software components, containers, and build artifacts used by their developers with our Sonatype Nexus Repository solution. Access to a central inventory of artifacts used in development can provide immediate insight if newly discovered vulnerable artifacts were ever used in the organization. But just knowing that artifacts are present in your repositories (the local warehouses of software supply chains) is not enough.
OSS components and containers are constantly flowing through development pipelines, and not all of them are created equal. We've long reported that 1 in 10 open source components being downloaded contain known security vulnerabilities that can represent breach points for adversaries. Just as quality is inspected across supply chains of physical goods, Sonatype Lifecycle inspects and tracks the quality and security of components and containers flowing through software supply chains.
Sonatype Lifecycle is directly integrated with the tools developers use to build, integrate, test, and deploy application code, and automatically alerts developers to troublesome components. When problems are found, the software offers guidance on updating them to safer and higher quality alternatives. Developers, governance, and security teams are no longer tasked with a manual review process, but aided by automated reviews and artifact tracking to ensure proper security and quality standards are enforced with little to no manual effort.
Next, you need to map out your build and release platforms. What's in your DevOps pipeline tech stack and who has access to it? Is your infrastructure publicly accessible? How do you manage updates? The machinery and the code it produces should be tamper-free, meaning, they're not erroneously injecting malicious code somewhere along the way.
While attacks through the build process may not seem that common, it's clear they're becoming an increasingly prime vector for abuse. Why? Because as investments shift from protecting networks, to applications, to open source code, to containers, adversaries must shift tactics to the next easiest and lucrative targets.
In short, define your entire software supply chain (where do you source open source software packages, containers, IaC, software updates, and build and release pipelines). Without this step, you can't see, let alone prevent, attacks on your supply chain.
Sonatype's 2020 State of the Software Supply Chain states that next-generation upstream software supply chain attacks are far more sinister because bad actors no longer wait for public vulnerability disclosures. Instead, they are taking the initiative to contribute code to open source projects and then - unbeknownst to the other OSS project maintainers - inject malicious code. Those code changes then make their way into open source projects that feed the software supply chains of developers around the world.
And, this is happening at a rapidly increased rate. In fact, there was a 430% increase in upstream software supply chain attacks over the past year. Keeping this in mind, it is virtually impossible to manually chase and keep track of such components.
Sonatype's world-class security research data, combined with our deep binary scanning technology and AI/ML-powered automated malware detection safeguards your developers, customers, and software supply chain from attacks.