Recently, I chatted with developers from a customer in a heavily regulated industry. They were manually updating their open source dependencies and wanted to find a better solution to save time. Keeping their dependencies up-to-date was very time-consuming but something they identified as crucial for their business.
They have since had great success in automating much of this process, which is why I wanted to share their success and help inspire others to achieve similar outcomes.
Today, development teams have a huge array of responsibilities. Once upon a time, a developer simply had to write code. But in this new open source world with so many different tools, clouds, architectures, processes, and deployment options, more is being asked of development teams than ever before.
All of this can be overwhelming, so no wonder any new process and technology is met with resistance from development teams. This leads me to the next responsibility coming the developer's way: open source dependency management.
What is dependency management?
Before we all started sharing open source code over the internet from repositories like Maven Central, dependency management was not such a big issue, as all the code in our applications was built in-house.
Nowadays, most development teams leverage this huge resource of open source code. Up to 90% of the code in new applications is open source. While that has allowed development teams to innovate at a very fast pace, we need to make sure that the code we didn't write in our applications is not going to cause issues later down the road. If that does happen, it can be catastrophic for a development organization and company.
Dependency management is important, and while I could write a whole article on why that is, the main reasons for keeping your dependencies up-to-date is to:
- fix security vulnerabilities;
- improve performance; and
- enhance collaboration by keeping teams on the same dependency versions.
If you want to know more about dependency management, check out this other great article.
Before automation
The developers we interviewed already knew the importance of dependency management, but they were manually having to do this for around 30 different repositories.
Before they automated part of this process, they would spend days of development time every two sprints (four weeks) keeping their applications up-to-date in a very tedious process.
The manual process they followed was similar to the following:
- Identify what dependencies are out-of-date in all 30 repos.
- Determine if the updates contain breaking changes via release logs.
- Manually update the version of each dependency in each repo.
- Create a pull request with the updates and wait for any tests to pass.
- Have another member of the development team review and merge the pull request.
They previously tested another tool before implementing their latest automation. This tool would find the latest versions but merge dependency updates together in a single pull request. This tool actually had negative impacts on their ability to keep dependencies up-to-date, to the point it was taking them just as much time as manually updating due to merge conflicts with their development processes.
While their current solution does not automate all of the above, it does drastically reduce the amount of time spent on updating dependencies.
After automation
They have now automated some of the above processes, which has translated to two developers spending around 15 minutes of development time per sprint (two weeks) identifying and applying dependency updates.
Compare that to when they were spending days of development time every four weeks! Not only are they now saving huge amounts of developer time on manual tasks, but they are keeping their applications more up-to-date, which also improves their application security among many other things.
From their old process, they have managed to automate the following:
- Identify what dependencies are out-of-date in all 30 repos.
- Manually update the version of each dependency in each repo.
- Create a pull request with the updates and wait for tests to pass.
While two steps are still manual, the above three steps are by far the most time-consuming in keeping dependencies up-to-date.
One thing to consider is that auto-merging a pull request is always difficult, as you need to have great confidence in your testing. If you trust your tests, these processes can and should be automated. Some critical builds will most likely always require a human set of eyes to make sure you are not going to break critical business applications. This development team stated their patch and minor version builds could be automated with the right testing in place.
Another thing mentioned was that breaking changes are one of the hardest things to detect in updates, especially when they are not included in release logs. Breaking API changes are a good example of this.
I personally believe solutions will emerge to help automate this step or at least reduce the time spent on it. At Sonatype, we have already invested in solutions that can predict dependency end-of-life and analysis in breaking API changes/incompatibility, such as in products like our Sonatype Lifecycle offering with the end goal being complete automation for our customers.
Takeaways
The approach this development team has implemented has profoundly reduced the amount of time spent on keeping their dependencies up-to-date. This will have improved security, application performance, reduced bugs, and among other things, saved a huge amount of development time, allowing them to focus more on innovation.
Some of the main takeaways I learned from talking to this customer are:
- Before automating dependency management, you need to make sure you have great testing in place as breaking changes can be hard to detect.
- You must scale any automation you implement out slowly to make sure you have confidence in the tool.
- They evaluated different tools and identified issues quickly. Even though manual processes were being automated by some tools, some actually caused more issues such as merge conflicts that required just as much effort as manually updating.
- While most importantly, automating this task can save huge amounts of time and headache for development teams. This has now allowed them to focus more on innovation and less on boring mundane manual tasks.
Hopefully this article has inspired you to start thinking about how you can automate dependency management as this is becoming more the development team's responsibility. Getting ahead of this new ask of developers can help you and your team focus more on the things you care about such as innovation and business value by harnessing the power of automation.
