Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Scaling Data

System Design HLD-07.01

Replication: leader-follower

One leader takes writes and streams its log to followers that serve reads and stand by for failover: sync versus async, promotion, and split brain.

Intermediate 12 min Last reviewed

The big idea

A single database server is a single point of failure and a ceiling on read traffic. ReplicationKeeping copies of the same data on several machines, so the data survives failures and reads can be spread across copies. keeps copies of the data on several machines. The most common arrangement is leader-followerA replication setup where one node, the leader, accepts all writes and streams changes to follower nodes, which apply them in the same order. (also called primary-replica): one node, the leader, accepts every write; it streams its change log to followers, which apply the same changes in the same order. Followers serve read traffic and stand ready to take over if the leader dies.

This one idea buys a lot: read capacity scales with the number of followers, a follower in another zone or region keeps the data safe from a local disaster, and backups and analytics can run on a follower without touching the leader. The price is a set of new questions: how far behind can followers be, what happens to writes in flight when the leader dies, and how do we make sure only one node ever thinks it is the leader?

Think of a teacher dictating notes to a class. One person speaks (the leader); everyone else writes the same words in the same order (followers). Students can answer questions from their notes, but a student who fell behind may give an answer that is out of date.

How it works

Step 1: The log stream

A leader streaming its log to followers Clients send writes to the leader, which appends them to its log and streams the log entries to three followers. Followers apply the entries in order and serve read requests from other clients. log log log Writes Reads Leader accepts writes Follower 1 same zone Follower 2 other zone Follower 3 other region Figure 1. All writes go to the leader. It streams log entries (glowing) to followers, which replay them in order. Reads can go to any follower, which spreads load but may return slightly old data.

Databases replicate either the physical write-ahead log (byte-level page changes, as in PostgreSQL streaming replication) or a logical log of row changes (as in MySQL's row-based binlog). Logical logs are also what change data capture reads ( Transactional outbox and change data capture ).

Step 2: Synchronous or asynchronous

  • Synchronous: the leader waits for a follower to confirm it has the write before acknowledging the client. If the leader dies, that follower has every acknowledged write. But each write pays a network round trip, and if the synchronous follower is down, writes stall.
  • Asynchronous: the leader acknowledges immediately and ships the change in the background. Writes are fast and unaffected by followers, but followers lag behind, and if the leader dies, its last unshipped writes are lost on failover.
  • Semi-synchronous: wait for one follower out of several. A common compromise: one guaranteed copy without stalling on the slowest.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. With 1 second of lag, read 0.3 seconds after your write: your write is missing from the follower.
  2. Wait longer before reading, or reduce the lag, until the read sees it.
  3. Switch to synchronous: no lag, but read the output line about the cost. The next lesson, Replication lag and read consistency guarantees , shows ways to keep async speed without surprising users.

Step 3: Failover

When the leader fails, a follower must be promoted:

  1. Detect the failure, usually by missed heartbeats over some timeout. Too short and a slow leader is wrongly declared dead; too long and writes are unavailable for longer.
  2. Choose the most up-to-date follower and promote it.
  3. Redirect clients and the remaining followers to the new leader.

The great danger is split brainTwo nodes both believing they are the leader and accepting conflicting writes, usually after a network partition or a false failure detection.: the old leader was only slow or cut off by a network partition, comes back, and still believes it is the leader. Now two nodes accept writes that conflict. Systems prevent it with fencing (the old leader is cut off from storage or shut down) and with consensus to agree on exactly one leader per term ( Consensus: Raft ).

Going deeper

Adding a follower without downtime

A new follower starts from a consistent snapshot of the leader, noting the log position it corresponds to, then streams the log from that position until it catches up. No locking of the leader is needed.

Replication is not a backup

A mistaken DELETE FROM orders is replicated to every follower within milliseconds. Followers protect against hardware failure, not against bad writes. You still need backups and point-in-time recovery ( Disaster recovery: RPO and RTO ).

Trade-offs

OptionWins whenLoses when
Asynchronous replication Fast writes and read scaling where a small window of loss on failover is acceptable Recent writes on failover, and stale reads from lagging followers
Synchronous replication No acknowledged write is lost on failover Write latency, and availability if the sync follower is down
Semi-synchronous (one of N) Most production setups: one guaranteed extra copy without waiting for the slowest A little write latency, and some complexity in configuration

Common mistakes

  • Reading from followers for data the user just wrote, and showing them stale results.
  • Automatic failover with aggressive timeouts that promote a follower during a brief network blip.
  • Treating replicas as backups.
  • Sending writes to whichever node a load balancer picks, including followers.

Check yourself

  1. In leader-follower replication, who accepts writes?

    Show answer

    B. Only the leader One leader orders all writes; followers replay its log.

  2. With asynchronous replication, what can happen when the leader crashes?

    Show answer

    B. The most recent writes not yet shipped to followers are lost Acknowledged writes that had not reached a follower are gone after failover.

  3. What is split brain?

    Show answer

    B. Two nodes both acting as leader and accepting conflicting writes Usually after a partition or false failure detection; prevented by fencing and consensus.

  4. Why is a replica not a backup?

    Show answer

    B. Mistaken writes and deletes are replicated too Replication copies everything, including mistakes, almost instantly.

Recap

  • One leader accepts writes and streams its log to followers, which apply it in order.
  • Followers serve reads, provide redundancy and can be promoted on failure.
  • Synchronous replication avoids losing acknowledged writes; asynchronous is faster but lags.
  • Failover must detect failure carefully and prevent split brain with fencing or consensus.
  • Replicas protect against hardware loss, not bad writes: keep backups.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up