System Design HLD-17.03
Infrastructure as code and environments
Describing servers, networks and databases in versioned files, immutable infrastructure, and keeping development, staging and production alike.
The big idea
Someone once set up the production database by clicking through a cloud console. Two years later nobody knows its exact settings, the staging copy is subtly different, and rebuilding it after a disaster would be guesswork. Infrastructure as codeDescribing infrastructure such as networks, servers, databases and permissions in versioned text files that a tool applies, instead of configuring it by hand. (IaC) prevents this: networks, servers, databases, load balancers and permissions are described in text files, stored in version control, reviewed like any other code, and applied by a tool such as Terraform, OpenTofu, Pulumi or CloudFormation.
The payoff is reproducibility. A new environment is one command. Every change has an author, a review and a history. A region lost in a disaster can be rebuilt from the same files ( Disaster recovery: RPO and RTO ). And because the files describe desired state, the tool can show you exactly what will change before it changes anything.
Think of a recipe versus a dish someone improvised. Anyone can cook the recipe again and get the same result; nobody can reproduce the improvised dish.
How it works
Step 1: Declare, plan, apply
The tool compares what the files declare with what actually exists and produces a plan: what it will create, change and destroy.
# A toy "terraform plan": compare the declared infrastructure with what exists, list the changes.
declared = {
"vpc/main": {"cidr": "10.0.0.0/16"},
"db/orders": {"engine": "postgres", "size": "large", "replicas": 2},
"app/api": {"image": "api:1.8.2", "instances": 4},
"bucket/uploads": {"versioning": True},
}
actual = {
"vpc/main": {"cidr": "10.0.0.0/16"},
"db/orders": {"engine": "postgres", "size": "medium", "replicas": 2},
"app/api": {"image": "api:1.8.1", "instances": 4},
"app/legacy-cron": {"image": "cron:0.3", "instances": 1}, # created by hand, never declared
}
def plan(declared, actual):
for name in sorted(declared.keys() | actual.keys()):
want, have = declared.get(name), actual.get(name)
if have is None:
yield f"+ create {name} {want}"
elif want is None:
yield f"- destroy {name} (not in code)"
elif want != have:
diff = {k: f"{have.get(k)} -> {v}" for k, v in want.items() if have.get(k) != v}
yield f"~ update {name} {diff}"
changes = list(plan(declared, actual))
print("\n".join(changes))
print(f"Plan: {sum(c[0] == '+' for c in changes)} to add, "
f"{sum(c[0] == '~' for c in changes)} to change, {sum(c[0] == '-' for c in changes)} to destroy.")Output
~ update app/api {'image': 'api:1.8.1 -> api:1.8.2'}
- destroy app/legacy-cron (not in code)
+ create bucket/uploads {'versioning': True}
~ update db/orders {'size': 'medium -> large'}
Plan: 1 to add, 2 to change, 1 to destroy. The plan catches two kinds of drift at once: the declared upgrade of the database size and app image, and a legacy-cron server someone created by hand that is not in the code, which apply would destroy. Reading the plan before applying is the most important habit in IaC: a one-line edit can, through dependencies, replace a database.
Step 2: Immutable infrastructure
Instead of logging in to patch running servers, build a new image with the change and replace the old servers with new ones. Immutable infrastructureNever modifying running servers; every change ships as a new image or instance that replaces the old one. means every server matches a known, tested image, rollback is redeploying the previous image, and there is no slowly accumulating hand-made difference between machines. Containers and Kubernetes ( Containers and Kubernetes for system designers ) work exactly this way.
Step 3: Environments that match
Development, staging and production should be built from the same code with different parameters (size, count, domain names), not maintained separately. The closer staging is to production, in versions, configuration and topology, the more problems it catches before users do. Differences that cannot be avoided, such as data volume, should be known and written down.
Going deeper
State and secrets
Tools like Terraform keep a state file recording what they manage. It must be stored remotely, locked during applies so two people cannot apply at once, and protected, because it can contain sensitive values. Secrets themselves belong in a secret manager, referenced by the code, never written into it ( Encryption, keys and secrets ).
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Infrastructure as code | Anything that must be reproducible, reviewed and rebuilt: production, staging, disaster recovery | Speed for one-off experiments, and a learning curve |
| Click-ops (console changes) | Quick exploration in a throwaway sandbox | Reproducibility, review and history |
| Immutable replacement | Consistent fleets and easy rollback | Image build time for every small change |
Common mistakes
- Applying without reading the plan, and destroying something by accident.
- Making "quick fixes" in the console that drift from the code and are wiped by the next apply.
- Committing secrets or the state file to the repository.
- Staging built by hand and drifting until it no longer predicts production.
Check yourself
-
What does a plan show?
Show answer
B. The changes apply would make: create, update and destroy It compares declared and actual state so changes can be reviewed before they happen.
-
A server was created by hand and is not in the code. What will apply do?
Show answer
B. Propose to destroy it, since the code does not declare it Desired-state tools converge on what the files declare.
-
What is immutable infrastructure?
Show answer
B. Replacing servers with new images instead of patching them in place Every change ships as a new image, which keeps fleets consistent and rollbacks simple.
-
Why build staging from the same code as production?
Show answer
B. So it matches production closely enough to catch problems first Hand-maintained staging drifts and stops predicting production behavior.
Recap
- Infrastructure as code describes infrastructure in versioned, reviewed files.
- Plan shows the changes before apply makes them; always read it.
- Immutable infrastructure replaces servers instead of patching them.
- Build every environment from the same code with different parameters.
- Keep state locked and protected, and secrets in a secret manager.