Almost everything in an infrastructure needs to know one or more secrets (passwords, some kind of private keys, access tokens, confidential data, etc.):
- to communicate with other services: to allow or to be allowed by others
- or to expose them
A simple use-case: a Wordpress website needs to connect to a database to find its content.
A more complex use-case: a shared secret between the nodes of a cluster to encrypt communications.
When you need to deploy multiple instances of such services:
- you usually use a tool like Ansible to handle the packages and the configuration files
- you are versioning your configuration files (like git) and also your secrets
- you have some machines dedicated to manage the workflow of the deployment (like Jenkins)
The question is: how to manage the secrets ? and have at least some basic features like:
Ansible Vault ?
Ansible ships with Ansible Vault.
It is very useful as:
- it can turn any YAML files into an password encrypted file: so it can be versioned
- it can decrypt them on-the-fly when running a playbook
- really easy to use by everyone
But maintaining one or multiple secrets files can become very tricky as:
- your team is growing
- your code base is growing and you want to break it in multiple pieces
- you need to share the same secrets between different applications/projects/environments
- you want add some roles/ACLs to identify/limit the access to the secrets
- you want to keep your files structure simple
And of course, all your secrets are pushed to a repository: they are encrypted but it’s probably not a good idea.
The classical use-case :
- you are working on a merge-request that implies to modify an encrypted file containing a secret
- a team-mate is also working on a merge-request that will modify the same encrypted file (perhaps not for the same secret)
You will end up with a merge conflict because git can’t know how to handle the encrypted file.
It’s not a big deal: you can decrypt, merge, rencrypt and push a new merge-request. But you’ll lose a bit of time and sometime you could even lose some secrets (typo during the merge) and this is bad.
To limit this kind of problem, you can find different strategies for your file structures: but, in my experience, you will always have problems.
Consul + Vault from Hashicorp!
A solution is to store your secrets in a secrets manager running inside your infrastructure.
The main advantages are:
- available everywhere and for everyone
- no secrets in your code repository (even encrypted, I don’t feel comfortable to leave them there)
Vault from Hashicorp handle this and adds:
- high availability (if you use Consul or other is used as a storage engine)
- lots of features you always need in an infrastructure: certificates creation, access to MySQL, access to OpenSSH, etc.
- advanced security features
- nice documentation
An demo of a working infrastructure: the big infra