Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Reliability and Resilience

System Design HLD-12.09

Chaos engineering

Injecting failures on purpose to prove a system survives them: hypotheses about steady state, small blast radius, abort conditions and game days.

Advanced 8 min Last reviewed

The big idea

This module has added timeouts, retries, circuit breakers, fallbacks, redundancy, failover and disaster recovery. Each one is a claim: "if the recommendations service dies, product pages still load"; "if a zone fails, the other two absorb its traffic". Until those claims are tested in the real system, with real traffic, real configuration and real monitoring, they are hopes. Many outages turn out to be a resilience mechanism that did not work the one time it was needed: a fallback that was never wired up, a timeout set to the library default, a failover nobody had run in a year.

Chaos engineeringThe practice of deliberately injecting failures into a system, under controlled conditions, to verify that it keeps behaving acceptably and to find weaknesses before they cause real outages. tests those claims by causing the failures on purpose, carefully: kill an instance, add latency to a dependency, block a network path, fill a disk. You state what you expect to happen, run the experiment on a small slice of the system with a way to stop it instantly, and watch. Either your confidence grows, or you found a weakness on your own schedule instead of at 3 a.m.

Think of a fire drill. Nobody sets the building on fire, but everyone practices leaving as if it were burning, and you find out that one exit is locked while the stakes are low.

How it works

The chaos engineering experiment loop A ring of five stages: define steady state with a business metric, form a hypothesis that it will hold during a failure, inject the failure with a small blast radius, observe and abort if the metric breaks, then fix weaknesses and expand the blast radius, looping back to steady state. Define steady state e.g. orders per minute Hypothesis it holds when X fails Inject failure small blast radius Observe or abort stop on metric breach Fix, then expand more traffic, more scope experiment small, measured, reversible Figure 1. Each experiment starts from a measurable steady state and a hypothesis, injects a failure into a small slice of the system with automatic abort conditions, and ends with either a fix or more confidence and a larger blast radius.

Step 1: Define steady state

Pick a metric that captures whether the system is doing its job for users: successful checkouts per minute, video starts per second, p99 latency of the login page. Not CPU, not "no errors in logs". Business-level metrics catch failures that show up anywhere in the stack.

Step 2: State a hypothesis

"If we terminate one of the three instances of the cart service, checkout success stays within 1 percent of normal." "If the recommendations service adds 2 seconds of latency, product page p99 stays under 800 ms because of the timeout and fallback." A hypothesis makes the result unambiguous: it held, or it did not.

Step 3: Inject, with a small blast radius

Start small: one instance, one percent of traffic, one cell ( Cell-based architecture and shuffle sharding ), a staging environment if production is too risky at first. Define abort conditions in advance, such as checkout success dropping more than 2 percent, and make stopping automatic and instant. Common failures to inject:

  • Terminate instances, containers or whole availability zones.
  • Add latency or errors to calls to a dependency.
  • Block network traffic between services, or to a database.
  • Exhaust resources: CPU, memory, disk, file descriptors.
  • Expire certificates or credentials, skew clocks.

Step 4: Learn, fix, expand

If the hypothesis failed, you found a real bug: a missing timeout, a retry storm, an alert that did not fire, a runbook that was wrong. Fix it and rerun. If it held, expand: more traffic, more instances, a larger failure, running continuously. Mature teams automate the experiments that pass so regressions are caught as the system changes.

Going deeper

Game days

A game day is a scheduled, team-wide exercise: simulate a large failure, such as losing a region or a core database, and practice the whole response, including detection, paging, decision-making, runbooks and communication. Game days test people and processes as much as software, and they are how disaster recovery plans get verified ( Disaster recovery: RPO and RTO ).

Prerequisites

Chaos engineering is not a first step. You need good monitoring to see the effect, the ability to stop an experiment instantly, and basic resilience already in place. Breaking a system that has no redundancy just causes an outage and teaches nothing new.

Trade-offs

OptionWins whenLoses when
No failure testing Nothing; resilience stays unverified Discovering broken failover during real incidents
Fault injection in staging Early learning with zero user risk Staging differs from production in traffic, data and config
Controlled production experiments Verifying the real system with small blast radius and abort conditions Some risk to a small slice of users; needs mature monitoring
Game days Testing people, runbooks and large-scale failover Time and coordination for the whole team

Common mistakes

  • Running experiments without a clear steady-state metric, so nobody can tell whether it worked.
  • No abort conditions or no way to stop the experiment immediately.
  • Starting with a large blast radius in production.
  • Treating chaos as random destruction rather than hypothesis-driven experiments.
  • Finding a weakness and not fixing it before the next experiment.

In the wild

Netflix

Chaos Monkey (opens in a new tab)

Netflix's open source Chaos Monkey randomly terminates instances in production so that engineers build their services to be resilient to instance failures. Applications can customize how Chaos Monkey treats them through the Spinnaker deployment platform. It is the tool that popularized the practice of injecting failures into production on purpose.

Check yourself

  1. What makes a good steady-state metric for a chaos experiment?

    Show answer

    B. A business-level signal such as successful checkouts per minute User-visible outcomes capture failures anywhere in the stack.

  2. Why start with a small blast radius?

    Show answer

    B. If the hypothesis is wrong, only a small slice of users is affected while you learn Small, reversible experiments limit the damage of the weakness you might find.

  3. An experiment adds latency to a dependency and checkout success drops 5 percent, past the abort threshold. What next?

    Show answer

    B. Abort automatically, then find and fix the missing timeout or fallback before rerunning Abort conditions protect users; the failure is the finding to fix.

  4. What does a game day test that automated experiments often do not?

    Show answer

    B. People and processes: detection, paging, runbooks, decisions and communication Large simulated incidents exercise the human response as well as the system.

Recap

  • Resilience mechanisms are claims until they are tested in the real system.
  • Chaos engineering injects failures deliberately to verify those claims.
  • Define a business-level steady state and a hypothesis before injecting anything.
  • Start with a small blast radius and automatic abort conditions; expand as confidence grows.
  • Fix what you find, automate passing experiments, and run game days for people and processes.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy