System Design HLD-16.03
Encryption, keys and secrets
Encryption in transit and at rest, envelope encryption with a key management service, secret managers instead of secrets in code, and key rotation.
The big idea
Data leaks in boring ways: a backup copied to the wrong bucket, a stolen laptop with a database dump, an API key committed to a public repository, traffic sniffed on an internal network that everyone assumed was safe. Encryption makes stolen bytes useless, but only if the keys are protected better than the data. Encrypting a database and storing the key in the same server's config file is a locked door with the key taped to it.
So there are three jobs. Encrypt data in transit with TLS and at rest on disks, backups and object storage. Keep encryption keys in a key management service (KMS)Key management service: a managed system, usually backed by hardware security modules, that stores master keys and performs encryption operations with them so the keys never leave it., using envelope encryption so the master keys never leave secure hardware. And keep all other secrets, such as database passwords and API keys, in a secret managerA service that stores credentials such as passwords, API keys and certificates, controls who can read them, audits access and supports rotation., never in code, images or chat.
Think of a bank's safe deposit room. Each box has its own key (the data key). The box keys are stored inside a vault that only opens with the bank's master combination, held by a guard who never hands it over but will unlock a box key for authorized people (the KMS).
How it works
Step 1: In transit and at rest
In transit: TLS on every connection, including internal service-to-service traffic, not just at the public edge ( TLS and HTTPS ). A service mesh can make internal mutual TLS automatic ( Service mesh ).
At rest: disks, databases, backups, queues and object storage encrypted, usually with AES-256. Cloud providers can do this transparently for storage services, which protects against stolen disks and mishandled backups. It does not protect against someone who can query the running database; for highly sensitive fields, add application-level encryption so only the services that need the plaintext can decrypt it.
Step 2: Envelope encryption
Envelope encryptionEncrypting data with a data key and then encrypting that data key with a master key held in a key management service, storing only the encrypted data and the wrapped data key. combines speed and safety:
- Generate a fresh data key for each object or table.
- Encrypt the data locally with it: fast, and works for any size.
- Ask the KMS to encrypt (wrap) the data key with a master key that never leaves the KMS.
- Store the ciphertext with the wrapped key and throw the plaintext data key away.
To read, send the wrapped key to the KMS, which checks your permission, logs the request and returns the data key. Access to data now requires access to the KMS, which is centrally controlled and audited.
import hashlib, random
rnd = random.Random(7) # seeded so the demo output is repeatable; real keys come from a CSPRNG
def toy_cipher(key, data):
"""TOY stream cipher (SHA-256 in counter mode) to show the flow. Real systems use AES-GCM."""
stream = b"".join(hashlib.sha256(key + i.to_bytes(4, "big")).digest() for i in range(len(data) // 32 + 1))
return bytes(a ^ b for a, b in zip(data, stream))
class KMS:
"""The master key never leaves this object, like a key held in an HSM."""
def __init__(self):
self._master = {"v1": rnd.randbytes(32)}
self.current = "v1"
def wrap(self, data_key):
return self.current, toy_cipher(self._master[self.current], data_key)
def unwrap(self, version, wrapped):
return toy_cipher(self._master[version], wrapped)
def rotate(self):
self.current = f"v{len(self._master) + 1}"
self._master[self.current] = rnd.randbytes(32)
kms = KMS()
record = b"card ending 4242, exp 09/29"
data_key = rnd.randbytes(32) # 1. fresh data key for this object
ciphertext = toy_cipher(data_key, record) # 2. encrypt the data locally (fast, any size)
version, wrapped_key = kms.wrap(data_key) # 3. KMS encrypts ("wraps") the data key
del data_key # 4. store ciphertext + wrapped key; drop plaintext key
print("stored:", ciphertext.hex()[:24] + "...", "| wrapped key under", version)
key = kms.unwrap(version, wrapped_key) # read: ask KMS to unwrap, decrypt locally
print("decrypted:", toy_cipher(key, ciphertext).decode())
kms.rotate() # rotation: re-wrap the small data key only
version, wrapped_key = kms.wrap(kms.unwrap(version, wrapped_key))
print("after rotation, key wrapped under", version, "and data still reads:",
toy_cipher(kms.unwrap(version, wrapped_key), ciphertext).decode())Output
stored: 74ffe1594b2acf274cf9642f... | wrapped key under v1
decrypted: card ending 4242, exp 09/29
after rotation, key wrapped under v2 and data still reads: card ending 4242, exp 09/29 Rotation shows the payoff: rotating the master key only re-wraps the small data keys; the gigabytes of data encrypted under them are untouched.
Step 3: Secrets management
Database passwords, API keys, signing keys and certificates belong in a secret manager, not in source code, container images, environment files committed to git, or tickets. Applications fetch secrets at startup or runtime using their own workload identity, so no bootstrap password is needed. Benefits: one place to control and audit access, and rotation without redeploying code. Some systems issue dynamic secrets, short-lived database credentials created per service on demand, so there is nothing long-lived to steal. Add secret scanning to repositories and CI to catch leaks early.
Step 4: Rotation
Rotate keys and secrets on a schedule and immediately after any suspected exposure. Design for it from the start: support two valid versions at once (old and new) during the switch, tag ciphertext with the key version that encrypted it, and automate the process, because manual rotation tends not to happen.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Provider-managed encryption at rest | Baseline protection for disks, backups and storage with no code | No protection from anyone who can query the running service |
| Application-level encryption of sensitive fields | Card data, health data, secrets inside records | Cannot query or index encrypted fields easily; key management in the app |
| KMS with envelope encryption | Central control, audit and rotation of master keys | KMS calls add latency and a dependency; cache data keys briefly |
| Secret manager with dynamic credentials | No long-lived passwords to leak | More moving parts and a hard dependency at startup |
Common mistakes
- Secrets committed to git, baked into images or pasted into tickets.
- Encrypting data and storing the key next to it.
- Using TLS at the edge but plaintext between internal services.
- One key for everything, forever, with no rotation plan.
- Writing your own cryptography instead of standard algorithms and vetted libraries.
In the wild
Cloud providers offer key management services (AWS KMS, Google Cloud KMS, Azure Key Vault) backed by hardware security modules and designed for envelope encryption, and storage services use them to encrypt data at rest. HashiCorp Vault and the cloud secret managers store application secrets and can issue short-lived database credentials, and code hosts such as GitHub scan repositories for accidentally committed secrets.
Check yourself
-
Why is the master key in envelope encryption never sent to the application?
Show answer
B. It stays inside the KMS's secure hardware; the app only gets data keys for objects it is authorized to read Keeping the master key in the KMS centralizes control and audit.
-
What does rotating the master key require in envelope encryption?
Show answer
B. Re-wrapping the small data keys with the new master key Data stays encrypted under its data keys; only the wrappers change.
-
Disk encryption at rest is enabled. Does it stop an attacker who steals database credentials?
Show answer
B. No: the database decrypts transparently for anyone who can query it At-rest encryption protects media; sensitive fields need application-level encryption and access control.
-
Where should a service's database password live?
Show answer
B. In a secret manager, fetched at runtime with the service's own identity Secret managers centralize access control, auditing and rotation.
Recap
- Encrypt in transit with TLS everywhere and at rest on disks, backups and storage.
- Envelope encryption: data keys encrypt data, a KMS master key wraps the data keys.
- Master keys stay in hardware-backed KMS; every unwrap is authorized and audited.
- Secrets live in a secret manager, never in code or images; prefer short-lived dynamic credentials.
- Plan and automate rotation, supporting old and new key versions during the switch.