System Design HLD-13.04
Deployment strategies
Rolling, blue-green, canary and shadow deployments, feature flags that separate deploy from release, and automated rollback on bad metrics.
The big idea
Most outages are self-inflicted: a new release with a bug, a bad configuration value, a migration that locks a table. You cannot stop shipping, and you cannot test every interaction with real traffic before release. What you can control is how many users a bad release reaches before you notice, and how fast you can undo it.
Deployment strategiesThe method used to replace a running version of software with a new one, such as rolling, blue-green or canary, which determines how much traffic a bad release can affect and how fast it can be undone. answer exactly that. Instead of switching everyone to the new version at once, release it to a small slice first, compare its metrics with the old version, and only then widen it. Keep the old version ready so rollback takes seconds. And separate deploying code from releasing features, so new behavior can be turned on gradually and off instantly without another deploy.
Think of a restaurant changing a recipe. Rather than switching the whole menu overnight, the chef serves the new dish to a few tables, watches their faces, and keeps the old recipe ready in case it goes badly.
How it works
Step 1: Rolling deployments
A rolling deploymentReplacing instances of the old version with the new one a few at a time, so capacity is maintained throughout the deployment. replaces servers a few at a time: take 10 percent out of the load balancer, update them, health-check them, put them back, repeat. It needs no extra capacity and is the default in orchestrators such as Kubernetes. But during the rollout both versions serve traffic, so they must be compatible, and rolling back means rolling forward through every server again.
Step 2: Blue-green deployments
- Deploy green All traffic is on blue (v1). Green (v2) is deployed alongside it and tested without users.
- Canary The router sends 5 percent of traffic to green. Error rate and latency on green are compared with blue.
- Switch Metrics look good, so all traffic moves to green. Blue stays up for a while: rollback is an instant traffic switch.
Blue-greenRunning two complete environments, switching all traffic from the current one (blue) to the new one (green) at once, and keeping blue available for instant rollback. runs two full environments. Deploy and test the new version in green while blue serves users, then switch the router. Rollback is switching back, which takes seconds. The costs are double capacity during the switch and care with shared state: both versions talk to the same database, so schema changes must work with both (expand first, migrate, then contract later, Zero-downtime schema and data migrations ).
Step 3: Canary releases
A canary releaseReleasing a new version to a small percentage of traffic first, comparing its metrics with the current version, and widening the rollout only if it is healthy. sends a small share of traffic, say 1 percent, to the new version and compares its error rate and latency with the old version side by side. Healthy? Go to 5, 25, 50, 100 percent, pausing at each step. Worse? Roll back automatically. A bug that would have hit everyone hits 1 percent of users for a few minutes. Canary analysis needs enough traffic at each step to be statistically meaningful, and fair comparisons (canary versus a baseline of the old version at the same time, not versus yesterday).
Shadow traffic goes further for risky rewrites: copy real requests to the new version, discard its responses, and compare them with the old version's. Users are never affected, but writes must be suppressed or redirected so shadows do not double-charge or double-send.
Try it
Try this:
- With a 4 percent error rate on the new version, the canary is rolled back while only 1 percent of traffic sees it. Note the number of extra failed requests.
- Switch to all at once with the same bug. Everyone gets the new version, the alert fires, and 15 ticks pass before a human rolls back. Compare the extra failures: roughly two orders of magnitude more.
- Lower the new version's error rate to 1 percent and raise the threshold to 1.5 points. The canary passes every stage: a small regression can slip through a loose threshold. What threshold would you choose for payments?
Going deeper
Feature flags: deploy is not release
A feature flagA runtime switch that turns a feature on or off, or on for a subset of users, without deploying new code. wraps new behavior in a runtime switch. The code ships dark, turned off, through the normal deploy pipeline. Then the feature is released separately: to employees, to 1 percent of users, to one country, to everyone, and back off instantly if something goes wrong. Flags also enable kill switches for incidents ( Load shedding and graceful degradation ). Their cost is accumulated complexity, so remove flags once a feature is fully released.
Automated rollback
The rollback decision should not wait for a human to notice a dashboard. Rollout tooling watches the canary's error rate, latency and key business metrics and rolls back on its own when they degrade beyond a threshold, then alerts the team. Combined with deploying to one cell or region at a time ( Cell-based architecture and shuffle sharding ), this keeps most bad releases to a small blast radius.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| All at once | Tiny internal tools, or emergency fixes | A bad release hits everyone; slow rollback |
| Rolling | The default for stateless services; no extra capacity | Mixed versions during rollout; rollback takes a full roll |
| Blue-green | Instant switch and instant rollback | Double capacity; shared database must support both versions |
| Canary with automated analysis | Catching regressions while they affect a small share of users | Needs enough traffic and good metrics; slower full rollout |
| Feature flags | Separating release from deploy; gradual and targeted rollout | Flag debt and combinatorial testing if not cleaned up |
Common mistakes
- Releasing to 100 percent at once because "it passed staging".
- Schema changes that break the old version, making rollback impossible.
- Judging a canary against yesterday's metrics instead of the current baseline.
- Canary stages too short or too small to collect meaningful data.
- Leaving hundreds of stale feature flags in the code.
In the wild
Kubernetes performs rolling updates by default, and tools such as Argo Rollouts and Flagger add blue-green and canary strategies with automated metric analysis and rollback. Large web companies commonly release first to employees, then to small percentages of users, and manage features with flag systems such as LaunchDarkly or in-house equivalents. Deploying region by region or cell by cell is standard practice at cloud providers.
Check yourself
-
What makes blue-green rollback fast?
Show answer
B. The old environment is still running, so rollback is just switching traffic back No redeploy is needed; the router simply points at blue again.
-
A canary at 1 percent shows 3 percent errors versus 0.5 percent on the old version. What should happen?
Show answer
B. Roll back automatically A clear regression at small exposure is exactly what canaries are meant to catch cheaply.
-
Why must database schema changes support both versions during a deploy?
Show answer
B. Old and new versions run side by side and rollback must remain possible Expand-migrate-contract keeps both versions working until the old one is gone.
-
What does a feature flag separate?
Show answer
B. Deploying code from releasing a feature to users Code ships dark; the feature is turned on gradually and off instantly at runtime.
Recap
- Deployment strategy decides how many users a bad release reaches and how fast you can undo it.
- Rolling replaces servers gradually; blue-green switches between two full environments with instant rollback.
- Canaries expose a small share of traffic first, compare against a live baseline and roll back automatically.
- Shadow traffic tests risky rewrites with real requests and no user impact.
- Feature flags separate deploy from release; keep both versions compatible and clean up old flags.