G
GuideDevOps
Lesson 9 of 13

Secrets Management

Part of the Security & DevSecOps tutorial series.

The Secret Sprawl Problem

Modern applications require an excessive amount of "Secrets" to function:

  • Database passwords.
  • Third-party API keys (Stripe, Twilio, Datadog).
  • SSL/TLS Private Certificates.
  • SSH keys.

As organizations grow, a dangerous phenomenon known as Secret Sprawl occurs.

Developers need these secrets to deploy applications. Because finding a secure way to share them is irritating, secrets end up scattered everywhere:

  • Hardcoded randomly inside config.js source code files.
  • Sent securely (but permanently) over Slack or Microsoft Teams messages.
  • Written on Post-it notes attached to monitors.
  • Stored in a Google Doc called "Dev_Passwords_DO_NOT_SHARE".

If an attacker breaches a developer's laptop, or discovers a public GitHub repository with an accidentally committed .env file, they have the master keys to the entire production environment.

What is a Secrets Manager?

To eliminate secret sprawl, organizations deploy a Secrets Management System (also known as a Vault).

A Secrets Manager is a heavily-fortified, centralized server or SaaS application whose sole purpose is to securely store, encrypt, and tightly control access to passwords and certificates.

The Workflow:

  1. A developer needs the production MongoDB password.
  2. They do not write it in the code.
  3. Instead, the application makes an API call to the Secrets Manager at boot time: "Hello Vault, I am the Node.js application. Please give me the MongoDB password."
  4. The Vault verifies the identity of the application (using complex IAM roles) and delivers the password securely into the application's RAM.
  5. If the vault determines the requester is unauthorized, it rejects the attempt and logs to the security team.

Dynamic Secrets (The Holy Grail)

Advanced Vaults don't just store static passwords; they create Dynamic Secrets.

If a web server needs to talk to the database for 30 minutes, the Vault reaches out to the database, actively generates a brand new username and password on the fly, hands it to the web server, and then automatically destroys the password 30 minutes later.

If a hacker intercepts that password, it is completely useless 30 minutes later. This completely eliminates the need to manually rotate passwords!

Popular Secrets Managers

HashiCorp Vault

The absolute gold standard in the DevOps / Cloud-Native world. It is open-source (though complex to self-host), extremely powerful, platform agnostic, and masters the art of Dynamic Secrets.

Cloud Native Options

If you are heavily embedded within a specific cloud provider, their native options are usually the path of least resistance:

  • AWS Secrets Manager / AWS Systems Manager Parameter Store
  • Azure Key Vault
  • Google Cloud Secret Manager

Handling Secrets in Kubernetes

Kubernetes provides a native object simply called Secret.

apiVersion: v1
kind: Secret
metadata:
  name: my-db-secret
type: Opaque
data:
  db_pass: cGFzc3dvcmQxMjM=

WARNING: Native Kubernetes Secrets are NOT ENCRYPTED at rest by default. Notice the data field in the YAML above. That isn't military-grade encryption; it is just Base64 encoding. Anyone who simply reverse-encodes that string gets your password in plain text (password123).

Because you cannot safely commit native Kubernetes Secret YAML files into Git (violating GitOps principles), the DevSecOps ecosystem created solutions:

1. Bitnami Sealed Secrets

Sealed Secrets allows you to commit encrypted YAML files directly into Git. You use a CLI tool (kubeseal) on your laptop to heavily encrypt your Secret using a public key. You commit this locked file to GitHub. Nobody on GitHub can read it. When the locked file is deployed to Kubernetes, the SealedSecrets Controller (which holds the private decryption key) decrypts it inside the cluster seamlessly.

2. External Secrets Operator (ESO)

ESO is a Kubernetes controller that actively reaches out to an external API (like AWS Secrets Manager or HashiCorp Vault), reads the secret in real-time, and automatically injects it into the native Kubernetes Secret store. This keeps Git completely clean of any secret files.