DevSecOps: Secrets in the Cloud

You know that feeling when you tell someone a secret and then wonder if it is going to get out? (I mean, I have heard some people worry about that.)

Well, system operators managing security often (maybe always) worry about secrets used to access their networks and resources will be get out or be setup incorrectly, exposing their systems.

If this describes someone you know, you can ease their worries with system hardening. Akash Mahajan (@makash) told his personal journey with system hardening in his talk, The Secrets in Our Clouds.  Perhaps his experiences could help you implement or improve system hardening for you own system.

Akash is with Appsecco, which is a relatively small company with typical production and staging requirements:

1. They run multiple web applications with integrated authentication
2. They run multiple websites with authentication and authorization
3. In most cases they terminate SSL/TLS at Nginx and reverse proxy to internal apps
4. They generate TLS certificates using Let’s Encrypt
5. Whenever they add authentication as a middleware they rely on a SSO provider
6. Their apps which are running need access to certain secrets such as passwords and authentication tokens

Akash said he started his system hardening journey with, “what I know best: installing and configuring software”. He chose Vault and Goldfish because of its well-written installation instructions and most of the complexity is hidden. Additionally, it seemed easy enough to expand.

Once your software is installed, you have to answer the question of where to store the secrets and what secrets do we need to worry about. Examples of secrets include:

• Certificates
• SSH keys
• API tokens
• Passwords (database, system, user, etc.)
• Anything else you need to store safely, such as audit logs that may contain sensitive details and TOTP generation keys

After this was setup for Akash, he still had nagging doubts and gnawing fears in the back of his mind:

• Is it the best way to manage secrets?
• Should I be using something which is cloud specific?
• How resilient is this?
• How will I automate to be able to make it reproducible?

• What if I make a mistake in configuration and it leaks data out?
• What if my backups fail?

What did Akash do? He switched hats from operations to security and didn’t panic. He took a step back and thought of, “abstractions which are simple enough for me to understand.” Some concepts that helped Akash include, plaintext keys, secure storage, secure access, and the ability to choose which users see which secret.

Some attributes that helped Akash:

• If a digital secret is stolen, you still have your copy
• The best way to store something that is important to you is in a form which is encrypted and not in plaintext
• If we encrypt something, there exists a key which is in plaintext
• This plaintext key needs to be protected
• This plaintext key should have secure storage and we should be able to prove that we are authorized to access it when we want it
• Our plaintext key can be stored inside the Vault behind keys which are with multiple people.
• A cloud is someone else’s computer. You can’t store it in plaintext, but you can be rest assured AWS, Azure, etc. have done their security homework.

If you plan to store secrets in your cloud, Akash offered up a few things to think about:

• Storage security - Can you expect to have better security than the cloud provider?
• Access security - Can your internal processes do better security than the cloud provider?
• Automation possibilities - If you do go ahead, what are the parts that will not be automated?
• Cooking required - If you plan to have a multi-cloud strategy, do you need to write your applications, automation scripts to have wrapper functions to ensure that you can plug-in/plug-out cloud provider specific code?
• Compliance - Do you have a specific compliance requirement which require you to think differently about all of this?

Akash wrapped up his talk with a recommendation to start simple with two approaches for managing secrets: one for humans and another for systems. For humans:

• Setup shared Keepass databases synced with each other
• Setup a Vault server behind Nginx and create entries for SSH keys and database passwords
• Use a provider which can match company directory to make it easy to revoke access to an employee if required

For systems:

• Use the same cloud provider where your servers are based
• If possible deploy a serverless app which exposes secrets both from the Cloud KMS and the Vault server.

If you want to hear Akash’s full talk, you can watch it here.

If you are craving more on DevOps in Modern Infrastructure?  Binge watch any of the 20 DevOps in Modern Infrastructure sessions, free of charge, from All Day DevOps.