Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Distributed Systems Theory

System Design HLD-08.09

Testing distributed systems

Why normal tests miss distributed bugs, and how fault injection, deterministic simulation and linearizability checkers find them, Jepsen style.

Advanced 8 min Last reviewed

The big idea

Every lesson in this module has described a failure that happens rarely: a partition at the wrong moment, a lost acknowledgement, a 15 second pause, a clock jump. A unit test runs on one machine with a perfect network and a fixed ordering, so it exercises none of them. A distributed bug might need three specific events in a specific order, and it may show up once in a million runs. Then, in production, at scale, it shows up every day.

Testing distributed systems means deliberately causing those rare events, recording what the system did, and checking the result against what it promised. You do not assert "x equals 5". You assert "everything clients saw is consistent with the guarantee in the documentation".

Think of crash-testing a car. Driving it around the block proves nothing about a collision. Engineers build a rig that causes collisions on purpose, repeatably, with sensors recording everything, then compare the readings against the safety standard.

How it works

Step 1: Fault injection

Fault injectionDeliberately introducing failures (killed processes, network partitions, delays, clock skew, disk errors) to see how a system behaves under them. causes failures on purpose while a workload runs. The usual menu:

  • Process faults: kill a node, pause it with SIGSTOP (a perfect stand-in for a long GC pause), restart it.
  • Network faults: partition the cluster, isolate the leader, drop or delay packets, make a link one-way.
  • Clock faults: skew one node's clock forward or backward, or make it jump.
  • Disk faults: slow fsync, full disk, corrupted or lost writes that were never flushed.

The point is to target the moments the design depends on: the leader during a commit, a lock holder mid-write ( Distributed locks and fencing tokens ), a node during an election.

Step 2: Record a history and check it

A Jepsen-style test harness Several clients send reads, writes and compare-and-set operations to a five-node cluster under test. A nemesis process injects faults into the cluster, such as partitions, kills and clock skew. Every invocation and its result is recorded with timestamps into a history. A checker reads the history and decides whether it is consistent with the promised model, such as linearizability, and reports valid or the anomaly it found. ops faults results Clients read, write, CAS Cluster under test 5 nodes Nemesis partition, kill, skew History invoke + ok/fail/info Checker linearizable? Figure 1. Clients run a workload while the nemesis injects faults. Every operation's start, end and result goes into a history, and a checker decides whether any single-copy order could explain it. A red verdict comes with the exact operations that broke the rule.

The test records a historyThe log of every operation clients issued in a test: when it started, when it finished, and what it returned (ok, failed, or unknown).: for each operation, when it was invoked, when it returned, and with what result. Some operations end as unknown: the request timed out, so it may or may not have happened. Those matter most, because "unknown" is where systems quietly lose or duplicate writes.

A linearizability checkerA program that decides whether a recorded history could have been produced by a single copy of the data executing operations one at a time, each taking effect between its start and end. then searches for any order of operations that obeys two rules: each operation takes effect at one instant between its start and end, and the result matches a single copy of the data. Example: client 1 writes x = 1 and gets ok at time 10. Client 2 starts a read at time 12 and gets x = 0. No ordering explains that, since the write finished before the read began, so the checker reports a stale read. If the read had overlapped the write, either 0 or 1 would be fine.

Checking linearizability is expensive in general (the search can blow up with many concurrent operations), so checkers keep histories short or concurrency bounded, and specialized checkers look for specific anomalies such as lost updates or transaction cycles.

Step 3: Deterministic simulation

Fault injection on real machines finds bugs, but you often cannot reproduce them: the timing never lines up the same way twice. Deterministic simulationRunning a whole distributed system inside one process where the network, disks, clocks and scheduling are simulated and driven by a single seeded random generator, so any run can be replayed exactly. fixes that. The entire cluster runs in one process. Network, disk and time are fakes, and every decision about which message arrives next, which packet drops and when a node crashes comes from one random number generator with a seed.

simulation.pyruns with python3 (standard library)
import random

def run(seed, steps=40):
    """Simulate two replicas of a counter with a bug: a retried increment is applied twice
    if the ack is lost. Faults come from the seeded RNG, so any failure is reproducible."""
    rnd = random.Random(seed)
    replicas, expected = [0, 0], 0
    for step in range(steps):
        expected += 1
        for r in range(2):
            replicas[r] += 1
            ack_lost = rnd.random() < 0.001           # injected fault: a lost acknowledgement
            if ack_lost and r == 1:
                replicas[r] += 1                      # BUG: the retry is not idempotent
        if replicas[0] != replicas[1]:
            return f"seed {seed}: replicas diverged at step {step}: {replicas}, expected {expected}"
    return None

failures = [msg for seed in range(200) if (msg := run(seed))]
print(f"ran 200 seeded simulations, {len(failures)} found the bug")
print("first failure:", failures[0])
print("replaying the same seed reproduces it exactly:", run(int(failures[0].split()[1][:-1])) == failures[0])

Output

ran 200 seeded simulations, 1 found the bug
first failure: seed 32: replicas diverged at step 6: [7, 8], expected 7
replaying the same seed reproduces it exactly: True

The bug here (a retried increment applied twice when an ack is lost) fires with a 0.1 percent chance per operation. A handful of manual tests would miss it. Two hundred seeded runs find it, and replaying the failing seed reproduces the exact same divergence at the exact same step. That is the superpower: a one-in-a-million interleaving becomes a one-line bug report, "seed 32", that any engineer can replay in a debugger. Because simulated time does not wait for real clocks, one machine can run thousands of simulated hours overnight.

Going deeper

What deterministic simulation demands

Everything must go through the simulator: no direct system calls for time, randomness, threads or sockets. That usually has to be designed in from the start, which is why it is most common in systems built with it as a goal, such as FoundationDB and TigerBeetle. Retrofitting it onto an existing codebase is a large project, though libraries and hypervisor-level approaches exist.

Simulation is not reality

A simulator only explores faults you modeled. Real disks, kernels and network cards misbehave in ways your model may not include. Teams combine both: simulation for depth and reproducibility, fault injection on real clusters for realism, and chaos experiments in production to verify that the whole system, including monitoring and operators, copes.

Trade-offs

OptionWins whenLoses when
Unit and integration tests Logic in one process; fast feedback on every commit Anything that depends on timing, partitions or partial failure
Fault injection on a real cluster (Jepsen style) Finding real consistency violations in real binaries, with real networks and disks Reproducibility: failures can be hard to replay; runs are slow
Deterministic simulation Exploring huge numbers of fault schedules and replaying any failure exactly Must be designed in; only covers faults the simulator models
Chaos experiments in production Verifying the whole system, including alerts and runbooks, survives real failures Risk to users; needs strong safeguards and a small blast radius

Common mistakes

  • Treating timeouts as failures in the history; an unknown result may still have taken effect.
  • Asserting on final state only, which misses stale reads and lost updates that happened along the way.
  • Running fault tests without a seed or a recorded schedule, so the one failure you saw can never be reproduced.
  • Testing only clean crashes, never pauses, clock jumps or one-way partitions.
  • Believing a database's consistency claims without checking whether they hold under faults.

In the wild

Jepsen

Jepsen analyses (opens in a new tab)

Jepsen publishes in-depth analyses of how real systems behave under faults. Its index says that since 2013 it has analyzed over two dozen databases, coordination services and queues, finding replica divergence, data loss, stale reads, read skew, lock conflicts and more. The list runs from Cassandra and Aerospike in the early years to recent analyses such as Amazon RDS for PostgreSQL 17.4 in 2025. Reading a few reports is the best way to see the techniques in this lesson applied to real products.

Check yourself

  1. Why do ordinary unit tests rarely catch distributed bugs?

    Show answer

    B. They run with a perfect network and fixed ordering, so the rare faults and interleavings never happen Distributed bugs need specific failures at specific moments; tests must cause them on purpose.

  2. Client 1's write of x = 1 returns ok at t = 10. Client 2's read starts at t = 12 and returns 0. Is this linearizable?

    Show answer

    B. No, the write finished before the read began, so the read must see it Linearizability requires that once an operation completes, every later operation sees its effect.

  3. A write in the history timed out. How should the checker treat it?

    Show answer

    B. As unknown: it may or may not have taken effect, and both possibilities must be considered Treating timeouts as failures hides lost or duplicated writes, which is exactly what you are hunting for.

  4. What is the main advantage of deterministic simulation over fault injection on real machines?

    Show answer

    B. Every run is driven by a seed, so any failure can be replayed exactly and many schedules explored quickly Reproducibility and speed are the wins; the cost is designing the system to run under a simulator.

Recap

  • Distributed bugs hide in rare faults and interleavings, so you must cause them deliberately.
  • Fault injection kills, pauses, partitions and skews nodes while a workload runs.
  • Record a history of every operation, including unknown outcomes, and check it against the promised model.
  • Linearizability checkers search for a single-copy order that explains what clients saw.
  • Deterministic simulation makes any failing schedule replayable from a seed.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy