Editor's Note: This post came from an energetic session at All Day DevOps. Don't miss the upcoming All Day DevOps | Spring Break, a free event on April 17. New television series spoilers might be available, there, too -- you'll need to register here to find out.
Today Sean Davis (@seanasaservice) will guide us through DevSecOps from a holistic view, using the story of Breaking Bad as the basis for our exploration. So let’s keep an open mind while we kick things off. And get ready for some Breaking Bad spoilers!
Breaking Bad Characters
Let’s start off with Walter. He’s a seasoned pro. He’s seen trends come and go, so he’s a bit skeptical. He wants to transform DevOpsSec to DevSecOps.
Jesse is our code slinger and has seen it all. He loves a challenge and has street cred.
Todd represents security. He’s not the smartest guy in the room, but makes up for it with ingenuity and creativity. He’s never met a website he can’t hack.
Then Lydia is operations, and she’s always on. She’s about the uptime. And she’s constantly in planning mode.
Finally, Saul is the voice of the businessman: clever, charismatic, and able to sell his way into and out of anything.
A Word on DevSecOps and the Periodic Table of DevOps
Many people say that if you’re doing DevOps, security should be built in. It should, but security didn’t have a seat at the table when DevOps came about. However, now all three have a seat at the table. All three have been integrated and aligned at each step to deliver value.
DevSecOps is more like DevOps 2.0. But it doesn’t matter what you call it. It matters what you do with it.
When you look at everything in play in the Periodic Table of DevOps, it’s a huge table of tools.
XebiaLab's Periodic Table of DevOps, shared in the “Breaking Bad: DevOpsSec to DevSecOps” presentation.
But now let’s look at the security part of the table. It’s off in the corner by itself, small and segregated. What do you think that means from a perception standpoint? How do companies view security and how it fits into the DevOps world?
Next let’s look at DevOpsSec formula for mayhem.
Sean Davis’s Formula for Mayhem from his “Breaking Bad: DevOpsSec to DevSecOps” presentation.
We’ve got tools added in, with CD and monitoring, JIRA, and all the other hot new tools.
Security is again a smaller piece of the picture, and it’s off to the side. There we’ve got GRC, risk, and other tools. But just because it’s off to the side doesn’t mean we shouldn’t worry about it.
The Story Continues
So let’s continue with Walter. They’ve had a recent breach and security has been made the focus. However, security once again was just bolted on at the end. Walter realizes that we can’t have that. It can’t just be a handoff between steps in the bigger picture.
Walter knows he has things that need to be done. He sees that we have operations, security, development, and business. So what does he do? Whatever is necessary.
To start, he talks to the developers. They know their stuff and don’t want people encroaching on their turf. When they’re not aligned with others, everyone is going to lose. They make a lot of quick decisions, and security isn’t always top of mind.
So Walter works with Todd to figure out how to make their development operation move faster. He then embeds Todd with the developers. They can share the challenges each team is working through and open the lines of communication. Things start moving in the right direction.
Unfortunately, the business comes in and throws unreasonable deadlines at the teams. And everything seems to get in the way of delivering value. Security is again stepping in to slow things down.
Next, Walter reaches out to operations. He has to partner with Lydia here. With the goal of determining how operations and security interact, he finds that operations loves data. And they’re also incentivized differently. They focus on automation, stability, monitoring, and alerting. Walter realizes that ops and devs need to find a common incentive that doesn’t cause friction between the teams. And then when you pull in security, it adds additional friction.
So Walter and Todd are making progress. They’re starting discussions with developers earlier and teaching them threat modeling. They’re showing the devs what operations and security need to be successful. Communication is improving between all these teams.
But Then, Trouble
Then disaster strikes. Security is blamed; operations takes a hit. Everyone is confused and no one understands who’s responsible for what.
So we realize that security being left out causes a lot of issues. They start tightening the screws and reinforcing once again.
But developers try to find workarounds so they can still go fast. And in turn, security pushes more demands and red tape onto the dev team. Everyone becomes frustrated. Walter realizes the problem—security is always lagging behind.
It’s not security’s fault. They’re forced to react. So what can we do?
Saul to the Rescue...?
Walter is on the hunt to find Saul. And when he finally finds Saul, Saul throws money at the problem.
Walter brings everyone together to build a new plan. A reorganization develops, bringing together a tiger team. With shiny new tools and equipment, energy is high.
Everyone is working to make this a success. Alliances form, and some teams find success. However, security is still not involved. Some things are better, but other things are still off.
Devs appreciate the automated testing from security, but then they’re inundated with security tasks in the backlog. Operations and SRE start a turf war over how to support the product. Things are slowing down again.
Saul becomes frustrated with Lydia. He supplied the funds. Where are the benefits?
And Lydia focuses again on stability, slowing releases down and preventing new features from getting out. The cycle begins again.
Customers are irate. No one understands what’s gone wrong. We’re still not getting the difference that we want.
What Were the Mistakes?
Let’s look at Walter’s mistakes and how the outcome could be changed.
First, Walter missed team alignment with business goals and security needs. Everyone needs to be aware of what needs to be fixed first.
Next, Walter missed a key fact: blameless operations could have prevented outages. Operations could have come in and worked with developers to understand how each team works and what each team should focus on.
And then, when Walter identified that devs were working around security to deliver value, he still missed key opportunities to build alignment in all the teams.
When the business brought in big funds to solve the problem, Walter needed to the team to not just buy in. He needed to keep them engaged. And he needed to train the other teams on security concerns so it wasn’t just security’s torch to carry.
And finally, Walter missed a key focus on the culture shift. He needed to work to bring all the teams into alignment and improve communication.
So don’t be like Walter. Though he made a lot of strides in getting everyone involved, it could have been different, and the tragic ending could have been prevented.
