Brett Lesczynski, Ryan Hejnosz and Adam Arihart, security analysts and administrators from U.S. Steel Corporation, spoke at the Elevate 2021 conference about how they upgraded their security practices at one of the largest and oldest companies in the world. Legacy practices are common at organizations with their age and scale, which can sometimes make change happen quite slowly. In this case, the efforts to bring a modern DevSecOps process into the environment has to start where the work is done (with support from leadership, of course).
Starting at the Beginning
Change in this environment is usually burdened with many error-prone and slow manual processes. This can cause significant delays in keeping your tech stack up to snuff. Even commonly-used technology that enables good processes – such as source version control –might not be used uniformly across all technology departments that own code.
As with any process re-engineering project, Brett and Ryan started by mapping the process. You can see the general change process mapped below as an example of how you can start:
Caption: Example change process
Managing Priorities
While environments vary, determining priorities is a constant and crucial in this process. Generally, people are limited to three things in any given time frame. Increasing that number results in diminishing returns. A monthly planning cycle can help avoid the tendency to plan for 10 improvements and only getting one completed.
In April 2019, they were determined to keep it to address three goals: project inertia, system maintenance, and addressing a low user base.
Caption: Example of three selected goals.
Priority management means setting a few key objectives and keeping them in focus. Brett and Ryan decided to focus on a Git program before moving forward.
Solving Open Source Security Concerns
A very common source of concern in any software engineering environment is the use of freely available, open source code. Issues such as dependency management, security, duplication, and version management are ubiquitous. If your environment is built around manual processes, tackling these issues can be daunting and complex.
Brett and Ryan considered several aspects of not only the technical implementation but also the impact each decision would have on people doing the work. Not wanting to bog down their developers with additional barriers, they considered using an air-gapped, or fully disconnected, build system. In the end, they introduced the Nexus Repository and Nexus Lifecycle into their environment so that OSS libraries could be scanned and managed effectively.
Automating Processes
Technology is great for automating processes, but you typically can’t go from nil to full automation overnight. Like the team at U.S. Steel, you probably want to start slow and introduce some steps that will move away from purely manual and lead into full automation. Often described as the “crawl, walk, run” approach, iterations in this process aid adoption. You might have a vision of what a fully automated build-test-deploy pipeline looks like, but it won’t materialize overnight. Its more important to establish a vision and communicate with your team.
Caption: Example staged improvements towards improved automation
Breaking up the process into stages also lets you create a team focus on continuous improvements, so you’re always getting better.
U.S. Steel’s stages for continuous improvement:
- Introduce DevOps and Git SCM.
- Create build pipelines to automate deployment.
- Investigate security solutions.
- Integrate security solutions in the processes and pipelines.
- Review OSS using Sonatype Nexus, etc.
- Improve developer skills and awareness of security issues (ongoing).
- Continue to improve and automate (ongoing).
Face Failure Head-On
Adam Arihart shared his experience of facing change in this process, explaining how pushing tools and processes on developers was ineffective. Instead, working with the software team directly and approaching their frame of reference. Additionally, confronting setbacks in the process lets you learn from them and part of your team’s learning goals, in addition to new goals that arise.
Take it Away
It’s possible to modernize the build-test-release process, even at a large company with legacy manual processes, antiquated tools, and dispersed teams. Like Brett and Ryan, the way forward is to understand the current process, establish a clear vision, share your ideas and benefits with others, and then map out how to get there.
View the Talk
A video of the Elevate 2021 full discussion is available.
