Empowering Developers: Security Self Serve and Automated Time-Based Waivers

Tyro is an Australian financial institution that specializes in merchant credit and debit solutions. The company helps its customers improve and grow their businesses.

Founded in 2003 and based in Sydney, Tyro has over 400 employees and 20,000 customers. The company also has about 150 engineers, with several cross-functional teams working together to release innovative products.

Tyro recently empowered their developers to build more secure software by instituting time-based waivers. Here’s how they made it happen.

What Tyro Is Currently Working With

Before we cover what they’re doing to automate security, let’s talk about Tyro’s architecture and deployment cycle.

Tyro uses a microservices architecture, with more than 200 microservices running in production. The team favors pushing small incremental events instead of massive updates.

Like most leading DevOps teams, Tyro employs continuous deployment and continuous release techniques. More specifically, Tyro is able to deploy updates weekly, and they were able to able to simplify the deployment process further with on-demand deployment via a single click. For cloud applications, they deploy continuously.

The Nexus Lifecycle Journey

The company started trying to figure out how to build even more secure software in 2015. This was precisely when they embarked on their application security journey. At the time, Tyro discovered a lot of security issues and had to address them, grandfathering in existing CVEs so their builds wouldn’t break.

It took around two years before the company was able to reduce its nearly 600 security issues to just a handful.

One of the challenges they faced when addressing CVEs was that new CVEs kept popping up. Fix one thing and along comes another. Tyro quickly found out that addressing security issues was a continuous process. When builds would break, they’d have to explain to the team what happened and why it wasn’t an issue until the waiver was removed.

Automated Time-Based Waivers

Prior to making a change, Tyro noticed that, on Lifecycle scans, servers took too long to build. In fact, they experienced a fourfold increase in build time.

Needless to say, the developers—who are focused on delivering transformative digital experiences to their customers—didn’t like that.

To fix this problem, the team created a Java-based custom Maven plugin that only triggers a scan when changes are being made. This plugin runs a dependency tree against the outputs to see what changed in each new build. This approach brought the build time down considerably, and the team was happy.

However, it didn’t address shared components upgrades. Normally, when a shared component gets updated, you have to update a number of additional applications as well. So the addition of the 55 seconds used as an example below would be done on every single application that’s updating. The complaints started rolling back in.

Tyro’s engineers knew that when they conducted a security scan, the bulk of the time was spent waiting for a response from the Sonatype servers.

To speed up the process, the team added a caching service that would only forward a scan request when a component was unknown.

Here’s what their new service looked like:

The benefit of this approach is that if all components are known, you get a much faster response.

The results?

• The scan results were faster (most of the time)
• It prevents the introduction of components with known security issues

However, this doesn’t help with security issues discovered in previously “good” components. So the company decided to use automated time-based waivers.

As a result, builds don’t fail when CVEs are discovered on existing components. If the security issue is critical, the team fixes it immediately. If it’s a high security issue, the team has two weeks to respond because they can always plan for the next iteration. Medium and low issues are given more leeway.

When a security issue is discovered, a webhook call is made. Then the team applies a policy waiver to the lifecycle management. That way, the build won’t break.

The team reviews waivers every day to see whether a waiver is past an SLA or they access this information via:

• Email notifications—send emails out during continuous monitoring processes
• A real-time dashboard—every development delivery team has multiple monitors where they display data they care about (e.g., health checks)

• Confluence—displays a table that lists waivers and keeps track of how many days are left in each of them

• Direct feedback—the Maven plugin feedback that renders directly in the code; when builds break, the code turns red, making it easy to identify issues

Tyro Moves to Security Self Service

Security self-service is a mechanism that allows teams to apply for extensions for security waivers. By doing that, they get more time to fix issues and allow builds to pass again.

Why did the team enact this process?

Simple: because of business priorities, like time to market and needing a bit more time to fix security issues. Sometimes, these issues require a lot of changes to be made. In some cases, there’s no fixed version of a component.

Getting started with security self-service requires two different logins, along with a reason for the request for an extension. The team’s Confluence page is then updated alongside it, and developers can see a list of which issues need a waiver extension. It’s a pretty simple service.

But, of course, there are some challenges that come with it:

• This tool only works with Java and Maven (pom.xml)
• Tyro is playing with unsupported Nexus Lifecycle APIs. That means the team has to be careful when they upgrade to the next version to make sure they haven’t broken any functionality.
• There are syncing issues between their source of truth (Lifecycle) and their service.
• There are issues related to “accidental” upgrades to minor versions.

Overall, this isn’t perfect because it’s limited to Java. But developers love the new system because they’re in control of the security issues.

Security issues are unplanned, but developers can manage them within the way they work. They can also address issues based on the company’s risk at the time.

Add it all up, and it’s an easy way to empower developers to fix security issues at their most convenient times, on schedule, and right in their regular workflows. As a result, developers are more engaged with their work and software solutions are more secure.