Maven Central and the tragedy of the commons

The tragedy of the commons is a concept in economics and ecology that describes a situation where individuals, acting in their own self-interest, collectively deplete a shared resource. In simpler terms, it's the idea that when a resource is available to everyone without restriction, some individuals tend to overuse it, leading to its eventual depletion and harming everyone in the long run. In the case of Maven Central, we are experiencing an unwitting tyranny by the few.

Maven Central is a vital resource for the Java community, providing a central repository for artifacts and dependencies.

Last year we again saw significant growth and over 1 trillion downloads:

Recently, I have been thinking about the long-term sustainability of open source software (OSS) in general as this has been a topic in many communities. Many OSS projects and ecosystems depend upon services donated by organizations large and small, and Central is no different in that regard.

I did some analysis of consumption behaviors and uncovered a shocking statistic:

• 83% of the total bandwidth of Maven Central is being consumed by just 1% of the IP addresses. Further, many of those IPs originate from some of the world's largest companies.

This situation perfectly exemplifies the tragedy of the commons, and it would be reckless to ignore.

For nearly two decades it has been a well-understood best practice to leverage a repository manager as a caching proxy. Not only does this reduce the load on the central repositories, but it improves your own build performance and reliability by caching the components needed by your builds instead of fetching them from the internet 10,000 times a day. This can also dramatically reduce your ingress/egress bandwidth costs if you are operating within a cloud infrastructure.

In today's internet, it can be very hard to identify the actual organization behind a given IP… as an example, 75% of the total traffic to Central originates from hyperscale cloud customers, and another decent chunk maps back to telecom providers worldwide. This means it is largely impossible to reach out to these heavy users proactively in most cases.

In the coming weeks, we will start to work with our providers to implement throttling mechanisms aimed at the extremely heavy consumers, which are effectively abusing a community resource. We are making every attempt to do this in a way that minimizes disruption to those builds such as slowing their actual download speeds, but in some circumstances it may lead to 429 error codes.

If your organization suspects it is being throttled or blocked, you have a few options:

1. Installing or enforcing use of existing repository managers: Implementing caching proxies like Sonatype Nexus Repository can significantly reduce the load on Maven Central by serving frequently accessed artifacts locally. Modern repository managers provide this caching capability for every component type such as npm, Docker, NuGet, PyPI, Ruby, etc. This approach allows heavy users to maintain their download speeds while minimizing the impact on all of the central repositories when organizations deploy them.

2. Contacting us for additional options: The Maven Central team may offer alternative solutions or support for heavy users with specific needs. This could involve exploring different download strategies or discussing custom arrangements. Start a conversation with us at mavencentral@sonatype.com.

The situation with Maven Central highlights the importance of responsible resource management. By acknowledging the tragedy of the commons and implementing solutions like throttling and caching, we can ensure the long-term sustainability of this valuable resource for the entire Java community.