How to store and use secrets securely

Secrets usually mean sensitive data such as passwords, tokens, and access keys that are needed for an application to work correctly but must be hidden from prying eyes.

So, for example, developers shouldn't see the passwords for databases, including the staging and production ones. They typically work locally, and that's enough for them. There are exceptions, of course.

There are also infrastructure secrets, that is, those that aren't required by the application to run but are used to create and operate the infrastructure itself, whether closely related to the application or not.

It makes sense to separate secrets by these types because the approaches for injecting them can differ. For example, secrets are often added to an application during the deploy process, while they're stored in specialized vaults like Vault.

When it comes to infrastructure secrets, there are different approaches:

1. You can use integration with the same Vault for the tools used for deployment: helm, helmfile, terraform. Here the secrets themselves are stored as a path: pass: ref+vault://databases/postgres/gitlab#DB_PASSWORD

2. Another solution can be sops (https://github.com/getsops/sops), that is, storing them in encrypted form.

It doesn't much matter which approach you choose; what's important to remember is that under no circumstances should secrets end up in the code, and certainly not in public git repositories.

We had a case where SMTP credentials (AWS SES) ended up publicly available on github. We reacted quickly — no more than half an hour had passed — but it was already too late: the daily email quota (and that's 500K) had been successfully used up by spammers.