Modernizing Open Source Dependency Management

 

In the last three State of the Software Supply Chain reports, we’ve delved into the intricate aspects of managing open source dependencies. We firmly believe that understanding the intricacies of both macro and micro decisions when selecting open source components fosters robust, streamlined, and secure software supply chains.

Each year, our cumulative analyses bring us closer to uncovering the key to optimal dependency management and ensuring safer, more efficient software supply chains. However, the incremental improvements we've observed in open source dependency management over the past three years, while noteworthy, fall short of our desired progress.

Open source consumers are not paying attention

In last year's report, we posed this critical question: Who is primarily responsible for creating the greater share of risk in open source ecosystems, open source maintainers, or consumers? The answer was overwhelmingly open source consumers. We wanted to see if this had changed, so re-ran some of the numbers for 2023. We saw no change in the consumption of vulnerable downloads within the Maven Central ecosystem.

This means, we once again saw that 96% of all known-vulnerable downloads were avoidable.

The size of the “consumer” problem is concerning 

This year, we again analyzed how the world consumes open source from Maven Central across 400+ billion downloads over the year. We compared consumers downloading vulnerable dependencies without a fixed version to vulnerable dependencies where a fixed version was available but not chosen.

How common are vulnerable downloads?

From the average 37.8 billion monthly downloads from Maven Central, 3.97 billion vulnerable components were consumed.

 

How common are fixed vulnerabilities?

From there, we found that 1.8 billion or 3.8% of the vulnerable dependencies were due to a vulnerable version with no available fix – meaning 96.2% of known-vulnerable downloads had a non-vulnerable option available. That means 2.1 billion avoidable vulnerable dependencies are being consumed each month.

Consumers opted for these no-fix-available projects due to lack of alternatives. Interestingly, even though there are only a few vulnerable versions without alternative fixes, a significant portion of vulnerable versions with available solutions are still being downloaded.

This sentiment is echoed in the Consumption Manifesto recently published by the Open Source Security Foundation (OpenSSF). The manifesto emphasizes the need for organizations to shoulder the responsibility for the open source software they employ, their methods of consumption, and their strategies for handling the associated risks.

Why is making good choices so hard? 

Developers face a multitude of challenges and responsibilities in their work, often leading to overwhelming and inefficient experiences when dealing with dependency management. This struggle has been humorously dubbed "Dependency Hell" in the development community. This bears repeating. 

Organizations expect developers to make informed choices regarding open source components for their software projects. However,  a significant portion (approximately 85%) of projects hosted on Maven Central are considered inactive, with fewer than 500 monthly downloads. This proliferation of inactive projects further complicates the already challenging task of selecting the most suitable components for a given project.

Consider for a moment the average Java application now boasts a whopping 148 dependencies, with around 10 releases occurring annually. Developers not only contend with the initial selection and management of around 150 dependencies but are also tasked with tracking an average of 1,500 dependency changes per year per application, possessing security and legal expertise to choose the safest versions, understanding ecosystem nuances, and sifting through thousands of projects to pick the best one. Now, imagine the scale of these decisions for enterprises with tens of thousands of developers and thousands of applications.

Developers not only contend with the initial selection and management of around 150 dependencies but are also tasked with:

Modern dependency management practices

While not surprising, the problem of dependency management didn’t magically go away in the past year. So, as we continue on the quest to understand how we could potentially fix it, we honed in on the aspects that reflect modern dependency management or a potential answer to solving the 96% problem. 

Our emphasis lies on understanding the nuances of behavior within this domain and how it impacts the way we work. This includes: 

• Defining the optimal component. What actually makes a "good open source component"
• Dissecting the optimal time to upgrade an open source dependency
• Reflecting on current upgrade behavior
• Analyzing global patterns of download behavior 

What makes an optimal open source component? 

In the realm of dependency management, understanding the inherent risks and benefits of individual software component versions is paramount. To address this challenge, we conducted a comprehensive assessment of software artifacts and have developed a robust scoring system that evaluates components across five key dimensions.

Our scoring system is not arbitrary. Built on years of analyzing dependency management, it seeks to create a holistic representation of a component's risk and value, facilitating data-driven decisions in software supply chain management. By examining these dimensions, we can assess dependency management, prioritized upgrades, and optimized software development practices.

 

Understanding component upgrade urgency

In the fast-paced realm of software development, staying current with component versions is paramount to ensure security, performance, and reliability. However, merely having access to better versions of components does not always guarantee that an upgrade is immediately necessary. Compare this to the way we buy and "upgrade" cars. You could get the newest model and upgrade every year. You have access to the new models (versions), and each year, there are improvements made that make it better. But, would that be effective? Usually, no. It would be even more time-intensive, some of those new features may have bugs that need to be worked out, and you may simply not need those features. However, there are definitely advantages to upgrading when there are new safety features, bugs have been worked out, and if there is a recall.

At first, the comparison seems unwarranted. Buying a car is considerably more time-intensive than upgrading a component. However, consider everything that needs to go into updating a dependency. For example, if your organization follows best practices, you will create a ticket for the work. Then, you may need discussion and approval to bring that work into the current iteration. Then, the work needs to be completed, a pull request and review from other members of the team, which could be immediate but often is not. And this list continues through every step of your development process. In the best case, there are no issues, but it has still touched multiple team members such that a five-minute change turns into an hour (or more) of work. For enterprises at scale, a common component with frequent updates could introduce significant interaction and waste associated with developer time.

To better understand this cost, we looked into the concept of "upgrade urgency" to shed light on the need for a data-driven solution. Our approach is based on a component scoring algorithm, which not only scores a component but also categorizes its versions into distinct zones. These zones range from the best/optimal versions to reactive (worst) versions.

Proactively managing dependencies is of utmost importance because, as the saying goes, software ages like milk not wine. Remaining in a reactive state is not only suboptimal from a security perspective, but it also puts development teams at an immediate disadvantage and penalizes them when issues arise. Analyzing the observed patterns reveals that teams that proactively make upgrade decisions are in a significantly better position to make the most informed choices.

The upgrade urgency algorithm assesses how urgently a component should be upgraded by comparing its version score to the best available. It categorizes urgency as follows:

• 0 - Optimal version(s): Component scores in the top 10% of the best version’s score
• 1 - Proactive: Scores between the top 10% and 1 standard deviation below the max
• 2 - Borderline proactive: Scores 1 standard deviation below the max but less than 2 standard deviations
• 3 - Reactive: Scores 2 or more standard deviations below the best version for that component

Unlike component scoring, upgrade urgency considers a version's distance from the best available, whereas being on the best version eliminates the need for urgent upgrading.

We will apply these algorithms to explore download patterns across different urgency zones, providing valuable insights into how software development industry professionals are making their component selections and upgrade decisions.

Downloads by upgrade urgency

As you know, Maven Central is the de facto repository for Java-based open source libraries. It acts as a centralized hub, enabling developers to effortlessly discover, access, and integrate dependencies into their projects. Its widespread adoption and comprehensive collection of artifacts have made it an essential resource for Java developers worldwide. 

In our analysis, we used Maven Central to examine the download patterns of component versions. We looked at a typical month of data and determined the upgrade urgency of each download. In a perfect world, developers would only be downloading optimal versions of their dependencies in order to minimize risk and future upgrade effort.

What we found is that a significant number of downloads (80%) are of the best available versions of the components being downloaded. At first glance this seems commendable, but it's worth noting that being on the best version doesn't necessarily mean it's a high-quality component. You might be on the best version of a really poor component. Let’s go back to our car analogy from above. If your brakes only worked 80% of the time, that would be unacceptable. That would mean, a 1 in 5 opportunity your brakes stopped working while driving around town. To further underscore this point, think about it in terms of food preparation. Maintaining strict hygiene standards for 80% of food products in a restaurant also means the majority of ingredients are safe to consume, but the remaining 20% could lead to food poisoning or other health risks.

With 2.6 billion downloads being classified as “borderline” and a further 1.1 billion falling into the worst (reactive) category, there is some cause for concern. After all, we live in a world where it only takes just one vulnerability to "set the internet on fire."

Moreover, a comprehensive analysis of the component scores for all downloaded versions reveals a striking pattern: during the time of download, each component, on average, had 10 superior versions available. This finding emphasizes that many developers not only miss out on the best versions but even when selecting versions that are considered "good enough," relatively speaking, there are better alternatives they could have chosen. This reveals that the selection process is likely driven by heuristics such as just picking the latest version. However, we see that even that strategy has its pitfalls. 

A ton of wasted time

In prior years, we reported on how efficient timing for upgrades can yield significant cost savings. Opting for safer versions and prolonging their use can significantly reduce upgrade expenses. In a medium-sized enterprise with 20 development teams, the potential gain equates to two extra development weeks per application each year.

This year, we’re continuing to dissect this cost-savings coefficient further. In addition to the above, which takes into account optimal upgrade decisions, we looked at what the time savings would be when you consider the quality and accuracy of your security data in making these decisions.

In answering this question, we found that when teams use better security data that reduce false positive findings by 25% in combination with making optimal upgrade decisions, each team saved a total 1.5 months of time per application, per year. This equates to a 2X boost in time saved over just the optimal upgrade process we described in our 8th annual State of the Software Supply Chain report last year. In other words, if you improve the security data you use and reduce false positives in tandem with upgrade efficiency, you double your gain.

In past years, we’ve looked at "optimal" as being a single component or a single version. There was a right and wrong version. This year, we’ve established that optimal is really a range. There is still a “right” or a “wrong” but on a spectrum. To further this, we examined the Maven Central Repository to quantify wasted developer time. In this scenario, we define wasted time as the download of an optimal version of a component when another optimal version of that same component has already been downloaded by that user instance. To put this into perspective, let’s again go back to our car analogy. Would you buy a car every time a new model year comes out? The optimal buying period might be when the model refreshes say every five years, rather than incremental changes each year. In our analysis, we consistently observed this phenomenon, which is all too frequent and raises concerns. The gains of such frequent upgrades are minimal, and from a lead/management perspective prioritization can also be optimized/aligned based on things that are in the optimal range versus just getting the new component.

However, we also noted great variability in this practice. To highlight this drawback we call attention to anonymized data from several companies in two different industries. We analyzed two organizations in the energy industry with a similar number of Maven Central downloads. Out of each company's total downloads, we found there was approximately 1.8 times more wasted downloads from one organization compared to the other.

We also examined two significant players in the online information industry. One organization had 5.4% wasted time, whereas the other enterprise had 10.2% wasted time. That’s an 89% increase in unnecessary downloads! It is evident that mature enterprises can reduce this variability and wasted time. By having a centralized platform for storing, managing, and distributing quality software components, this is a very solvable issue.

The value of an artifact repository manager 

In the scenario above, we know much of this wasted time could be solved by simply using a good repository manager, like Sonatype Nexus Repository Manager. A centralized, binary repository allows you to proxy, collect, and manage your dependencies so that you are not constantly juggling all of these various versions. Or, in this case, wasting time in downloading optimal versions, when another optimal version is already in your instance. It becomes the single source of truth for an organization’s software components and applications.

Global patterns of open source consumption behavior 

In today's interconnected global landscape, various regional groupings and trade agreements play a pivotal role in shaping international relations, trade dynamics, and economic cooperation. We explored whether these groupings also inform software development in the form of component consumption patterns.

With the component score, we can consider the quality of the component. With upgrade urgency, we can assess how good the downloaded version is. Bringing it all together, we examined global download patterns. We gave equal weight to component scoring and upgrade urgency. We gave a little additional weight to the sheer quantity of downloads, because picking the right components and staying on the best versions is surely harder at scale. Below are some observed global trends.

Within the Group of Seven (G7) nations, renowned for hosting some of the world's most advanced economies, the United States emerged as the frontrunner boasting the highest cumulative score. In contrast, its northern counterpart, Canada, found itself positioned at the bottom of the list. Germany had the highest proportion of optimal versions downloaded. Italy had the highest proportion of reactive downloads among the G7 countries at over 8%, which also made it one of the lowest-ranked countries across the globe on that metric, only beating out China, Hong Kong, Macedonia, and San Marino. Some of this isn’t surprising. While we talk later in this report about the sheer volume of regulations taking place around securing software supply chains, the United States has by far taken this issue the most seriously, at least in terms of focused regulation and national policy conversations.