DevOps and Opportunities in Software Supply Chain Governance

Governance has been an evil word for software developers but new approaches unlock massive gains in productivity, reductions in cost, and improvements in quality.

These last few years have been a fascinating journey. Concepts that we at Sonatype have been evangelizing for years – component based software development, the proliferation of open source, the inevitability of DevOps – are now becoming widely understood if not accepted. The simple fact is that “software is eating the world” and innovation is driving competitive differentiation in almost every industry, in every market.  These advances were inevitable.

Unfortunately, this tectonic shift is not without a downside. Unmanaged use of open source components, for example, can lead to a massive surface area that becomes impossible to maintain (robbing resources from innovation) and high risk from both cyber and operational perspectives. 

Leading organizations in verticals such as Banking and Insurance internalized these challenges very early and established centralized and highly structured open source governance programs. Typically, these programs consisted of processes by which developers requested permission to use components. Those requests would then be evaluated by domain experts comprised of Security, Legal, Architecture and others who worked toward, most typically, a whitelist of components deemed acceptable for use, often stored in a “golden repository” to be accessed by developers.

Until recently, these approaches represented best practice. Today, these programs are incongruent with modern development practices for four simple reasons:

• Time: the typical request to approval time in most organizations that we have talked to ranges from 6 to 12 weeks. Sometimes longer for very large projects, shorter for version increments to approved projects.

• Conflicting demands: lengthy approval cycles are not compatible with business demands for innovation and developers often use creative methods for working around incompatible mandates. One example is that of a development team that was using the components that they felt they needed, but renamed them to appear as though they were on the whitelist.

• Deterioration: software ages like milk, not wine. What is perfectly acceptable one day may become enormously risky the next as new flaws are discovered and new exploit methods are invented. Unfortunately, very few organizations revisit their governance determinations absent meaningful disruption (e.g. Heartbleed, Commons Collection).

• Volume: more than 7,000 new projects and 70,000 open source components (versions) are released every week! The interdependencies between these components are nearly incalculable and the volumes of usage are remarkable – in 2016, we serviced 52 billion component download requests! The simple truth is that the modern software development ecosystem (ecosystems, to be more precise) defies any process that relies on human interpretation as the front line of governance engagement.

So now what? It is becoming increasingly obvious that as we embrace DevOps transformation we must also embrace governance transformation. Governance transformation should naturally embrace the same concepts that enable successful DevOps initiatives: 

• Scale: the governance function must operate in the real time flow of the DevOps pipeline. Of course this is possible only through automation. In order to automate, organizations must be able to define policies – the attributes of acceptability – that otherwise reflect the considerations applied in manual review. Automation also demands extremely precise metadata and the ability to exactly identify a given component; not just what the name indicates it should be, but what the internals prove it to be.

• Early: problem discovery, at scale, is relatively valueless in and of itself. In order for a DevOps governance program to realize its full potential, problems must be identified as early as possible with remediation details that are crisp, detailed, and easily consumed by developers inside the tools that they already use. The reduction or elimination of rework is an enormous value driver.

• Everywhere: integrations with automated governance should be in place for the entire DevOps tool chain. There are any number of opportunities for components to be introduced into a given piece of software. Also, the actions taken related to a given policy violation could, and arguably should, be different as software applications get closer to production. Once in production, software inventories should by continually monitored in the context of constantly evolving knowledge about the software supply chain.

Ironically, the organizations that were among the first to understand the challenges of modern software development now have the most entrenched, traditional (manual) governance programs. As challenging as reshaping these programs will undoubtedly be, human-led processes are simply incompatible with DevOps and the goals of continuous delivery. Waterfall concepts and capabilities will still play a role, but not in the native patterns of DevOps processes.

The good news is that for organizations that have the will and resources to embrace DevOps and governance transformations, what Gartner has termed DevSecOps, the gains can be enormous. We have started to see customer studies of their own transformation initiatives and the results are stunning. The most recent that I have seen include three industry leading organizations in the Banking, Finance, and Insurance sectors. Their findings: 

• The elimination of 54,000 hours spent in manual processes (in one organization alone!).
• 28% (or more) developer productivity gains, mostly driven by faster access to technology and rework reduction.
• 30% reduction in overall program costs.
• Quality improvements of between 29% and 48%. 

Competition has never been more intense for nearly every business in nearly every segment. As challenging as some transitions can be, the rewards for innovation delivery are increasingly obvious and quantitatively measurable. So while historically, governance has carried negative connotations (especially for developers), modern software supply chain governance aligns interests, encourages innovation, and enables DevOps to realize its full potential.