How to Design Teams to Bridge the Security Gap

Even if you work for a large organization, chances are that you could memorize the names of all the security folks working in IT. That's because the ratio of developers to security is 100:1, according to a recent survey (the same study indicates a 10:1 ratio of devs to ops). Previous studies have reported ratios from 100:3 to 100:6, so there's some progress but not fast enough.

That doesn't bode well for these organizations' ability to tackle upcoming data privacy protection regulations like the European Union's GDPR and stop the data spilling that has been on the news on a regular basis over the past few years. There aren't enough skilled security folks out there today to fill in the gap. Successfully adopting secure development and operations practices requires not only deep technical understanding, but also, critically, possessing the communication, persuasion and balancing skills to achieve buy in from development teams and (more often than not) senior management too.

We're still inside a vicious cycle where security specialization is not attractive enough because, historically, companies have not given security enough priority. Those same companies are now facing a growing need to up their security game, but they lack enough people with the right skills.

So what options do we have to bridge this security gap?

Let's first clarify that automating basic security checks (such as third party vulnerabilities) and coding guidelines and integrating security in the delivery pipeline should be a no brainer for any product development team today. The toolset in DevSecOps keeps expanding and both community and enterprise-grade requirements are well served.

So where's the gap? It's in the "hacker mindset", a constant search for new potential attack vectors in our applications (the "unknown unknowns"). And it's also in the hard task of meaningful risk analysis and threat modeling and its translation into actions and priorities. These require deep security thinking and background, it's not something product development teams can be asked to take on as added cognitive load on top of their existing responsibilities.

The work that Matthew Skelton and I have done researching, cataloging and explaining how different DevOps Topologies can play an important role in progressing or blocking DevOps adoption fits nicely with the problem at hand.

Two distinct (and possibly complementary) team topologies come to mind:

 A) Fully Shared Security Responsibilities

 B) Security as an Enabling Team

What are the differences, the pros and the cons of each approach?

Fully Shared Security Responsibilities means that each product development team integrates at least one security-focused T-shape team member. This approach is adequate when the organization strives for cross-functional autonomous product development teams. The security person needs to be able to translate complex security threats into actionable security stories and acceptance criteria that the team can understand and implement from a technical standpoint.

They also must be able to communicate risks and potential costs for the organization (compliance, revenue and reputation) in a way that business owners can understand and prioritize. On the other hand, they should be ready to perform non-security related tasks when needed. Over time, ownership of security analysis and implementation should spread in the team while the T-shaped security person might maintain expertise in terms of staying on top of security best practices, new methods and tools, facilitating security-related workshops and leading product security strategy.

The goal is not for the security person to be allocated all the security work, otherwise we are just moving a silo at a macro level (isolated security team) to a micro-level (isolated team member).

Each product team - depicted as a yellow circle - with 7 to 9 team members, where at least one is a T-shaped security person - depicted inside a green circle - but other people also participate in security work.

Of course, there is still an important place for a centralized view of security across multiple products and business areas in the organization. Sharing experiences, approaches and results is critical. But this can take the form of acommunity of practice or a guild (in Spotify parlance) of motivated individuals gathering on a regular basis, rather than a dedicated team.

Pros

• Team security ownership will rise, leading to faster (non-trivial) security incident resolution and, perhaps more importantly, learning from them to avoid repeating the pain in the future.
• As teams have a natural size limitation (8-9 people), the ratio of IT staff to security (T-shaped) staff should move much closer to 10:1 over time with this topology. The diversity in technical background alone will bring benefits in terms of approach to problems and priorities.
• The move to this topology will send out an undeniable signal to everyone in the organization that security is now a first class citizen in our products.

Cons

• The cost of finding and recruiting (which might require generous packages) and ramping up those extra 9 security persons (in an org with 100 developers) will be a serious issue (as per the intro to this post). Expect this transition to take a considerable amount of time, during which you need to have a mix of models in place (some autonomous product teams and some still depending on a centralized team), and think about a strategy for selecting which teams go first. For example, start by assigning the more experienced security folks to the teams working on the riskier products in your organization.
• High-performing autonomous teams are based on stability and knowing each other's strong and weak points. Any change to the team composition, even a single person, will inevitably cause some disruption. The team might feel it is delivering less and becoming less productive while taking in the new security angle. It is crucial that everyone understands this is normal and accepted.
• Lack of a centralized contact point for security concerns. Senior managers, in particular, might feel there is "no one at the wheel of security" in a decentralized structure. Making sure there are communication channels and people able to direct requests for security information to the right teams (or to a community of practice) can help here (although that could work as well, if you have the means).

Heuristics

• Does your organization already have cross-functional teams in place, integrating business owners and owning responsibility for QA and monitoring and running the applications they develop?
• Do products (or services) display very different risk profiles?
• Do products (or services) run on heterogenous tech stacks and infrastructure?

If the answer to one more more of the above questions is affirmative, then this topology might prove suitable to address the security gap.

Security as an Enabling Team means a dedicated security team provides support and guidance (for example, producing secure development guidelines) to product development teams.

Crucially, this centralized team does not perform the actual security analysis and implementation work, instead promotes and guides it where necessary.

It's equally critical that this team gets involved from the very start of the project or release in order to help the product team consider the security implications, approaches and practices at each stage of the lifecycle.

A transversal security team - depicted vertically in grey - supports three product teams - depicted horizontally in orange - resulting in shared responsibility for security - depicted as a light green circle

Many organizations have taken this approach, including Spotify and Sportradar. It requires a much less drastic structural change from the traditional "security team silo", as we're essentially shifting this team's responsibilities (guidance and support rather than implementation). But that might make it harder for the rest of the organization to fully realize that product teams are expected to take greater ownership of the security of their systems.

Pablo Jensen, CTO at Sportradar, recently talked about their dedicated information security team's role of promoting security guidelines and policies in close collaboration with product teams, listening to their feedback and iterating over time. One of their project lifecycle gates is a "fitness to start development" which requires approval by the security team that enough thought has been given to security implications and work planned accordingly. This team reports directly to CEO and has authority to stop product launches if necessary.

Role of a centralized security team providing guidance rather than doing the product security work Credit: Pablo Jensen, CTO at Sportradar (slide from his QCon London 2018 presentation)

Pros

• Shorter timeframe to move to an enablement model from a traditional isolated security team doing the bulk of the security work.
• Will not require a large number of new hires, thus avoiding all the associated effort and potential team disruption. The focus here should be on making sure the enabling team as a whole has all the soft skills required, on top of technical skills.

Cons

• This team needs to master effective communication, which in itself is a good thing. This is a skill that needs to be honed over time, thus initially investing in external or internal help to define communication strategies and curate content can be required.
• Moving to this model sends out a weak signal about the change in security focus. Signal needs to be amplified, and might require repeating for a long time until it sinks in as part of the org culture.

• Introducing new responsibilities, practices and tools can be challenging for product development teams already at the peak of their capacity. The centralized enablement team might get overwhelmed with requests for help and be put under pressure to help with product security implementation, which would defeat the entire purpose of the model.
• Over time the security focus inside product teams might fade away without a security person participating in the team's release/sprint planning and daily standups. Some kind of governance will need to be put in place to avoid this effect.

Heuristics

• Is there a small number of product development teams (allowing for closer collaboration with the security team)?
• Do the product team members display interest in security topics and appetite to learn (for example attending technical conferences or meetups)?
• Does the resolution of security related incidents in production naturally involve both security and development people (as opposed to a blame game where each side shies away from responsibiity)?

If the answer to one more more of the above questions is affirmative, then this topology might prove suitable to address the security gap.

Conclusion

DevSecOps has raised the profile of security in IT and the regular stream of serious data breaches has exposed the security gap in most organizations. In this post I presented a couple of possible approaches to bridge that gap (fully shared security responsibility vs security as an enabling team), along with their pros and cons, and heuristics based on context.

Remember team design should not be static. Organizations, people and processes evolve naturally over time and what today's best fit might be tomorrow's obstacle to faster and safer software. It's quite possibly an organization would first move from a traditional security silo approach to a "security as enabling team" model at first and over time transition to a "fully shared security responsibilities" structure, as security awareness and expertise spreads across the product teams.

What other team topologies and security strategies have you seen in your organizations or clients? Please email me at: me at manuelpais dot net.

Finally, I encourage you to read this year’s full set of responses from the 2018 DevSecOps Community Survey here.  The results are fascinating.