Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Scaling Data

System Design HLD-07.02

Replication lag and read consistency guarantees

Why your own write can vanish when reading from replicas, and how read-your-writes, monotonic reads and consistent prefix reads fix it.

Advanced 12 min Last reviewed

The big idea

Asynchronous replicas are usually a fraction of a second behind the leader, sometimes seconds or minutes under load. That is invisible for most reads: a product page a second old is fine. But some sequences of reads look broken to a human. You edit your profile, the page reloads from a lagging replica, and your edit is gone. You refresh a comment thread twice and a comment appears, disappears, then reappears. You see an answer before the question it replies to.

Full strong consistency would fix all of this, at the cost of reading everything from the leader. A cheaper approach is to give users a few specific guarantees that cover exactly the anomalies people notice: read-your-writesA guarantee that after a user writes something, that same user's later reads will see the write., monotonic readsA guarantee that a user never sees data go backwards in time: once they have seen a value, later reads do not return an older one. and consistent prefix readsA guarantee that readers see writes in the order they happened, never an effect before its cause..

Think of reading the news from several printed copies that arrive at different times. You do not care that the copy is an hour old. You do care if your own letter to the editor is missing right after you were told it was printed, or if today's paper is followed by yesterday's.

How it works

Step 1: The anomaly

Read-your-writes violated by a lagging replica, then fixed A user updates their bio on the leader and gets ok. Reloading routes the read to a replica two seconds behind, which returns the old bio, shown in red. The fix routes this user's reads to the leader for a few seconds after a write, which returns the new bio, shown in green. User Leader Lagging replica 2 s behind UPDATE profile SET bio = 'new' ok reload profile (read routed to replica) bio = 'old' -- my edit vanished! fix: read from leader for a few seconds after writing bio = 'new' Figure 1. The user's own write is missing from the lagging replica (red). Routing their reads to the leader for a short time after they write restores read-your-writes (green).

Step 2: Read-your-writes

Techniques, from simplest to most precise:

  • Read your own data from the leader. Anything the user can edit themselves (their profile, settings, drafts) is always read from the leader; everything else from replicas.
  • Leader for a while after a write. Remember the time of the user's last write (in the session or a cookie) and route their reads to the leader for, say, 10 seconds afterwards.
  • Track log positions. On each write the leader returns its log position (PostgreSQL's LSN, MySQL's GTID). The client sends it with later reads, and the system uses a replica only if it has replayed at least that position, otherwise it waits or goes to the leader. This is precise and works across devices if the token is stored server-side.

Step 3: Monotonic reads and consistent prefix

Monotonic reads: if two consecutive reads hit different replicas with different lag, the second can be older than the first and data appears to go back in time. Fix: pin each user to one replica (for example by hashing the user id), or apply the log-position technique to every read.

Consistent prefix: in partitioned systems, different partitions replicate independently, so a reader can see a reply before the message it answers. Fix: keep causally related data in the same partition, or track dependencies so a write is only shown after its causes.

Going deeper

How much lag is normal?

In a healthy cluster, lag is typically milliseconds. Under heavy write load, during long-running queries on a replica, or across regions, it can grow to seconds or more, and occasionally a replica falls far behind. Monitor lag per replica, alert on it, and take replicas out of read rotation when they exceed a threshold, so they cannot serve badly stale data.

Connection to consistency models

These guarantees are session-level consistency models. They sit between eventual consistency (replicas converge eventually, no promises meanwhile) and linearizability (every read sees the latest write). The full spectrum is in PACELC and consistency models .

Trade-offs

OptionWins whenLoses when
Read own data from the leader Simple apps where users mostly edit their own data Leader load if most reads are of the user's own data
Log-position tokens Precise read-your-writes across replicas and devices Plumbing tokens through clients and services
Pin users to one replica Monotonic reads with no extra coordination Rebalancing when that replica fails or lags

Common mistakes

  • Sending all reads to replicas behind a load balancer and ignoring lag entirely.
  • Using a fixed "leader for 1 second after write" window while replicas sometimes lag for 10.
  • Fixing read-your-writes for the web app but not for the mobile app that reads from another path.
  • Not alerting on replica lag until users report vanishing data.

Check yourself

  1. A user saves a setting and the next page shows the old value. Which guarantee was violated?

    Show answer

    B. Read-your-writes The user's own write was not visible to their subsequent read.

  2. Two refreshes return a new comment and then an older state without it. Which guarantee fixes this?

    Show answer

    B. Monotonic reads Monotonic reads ensure data never appears to go backwards for a user, e.g. by pinning them to one replica.

  3. How does log-position tracking give read-your-writes?

    Show answer

    B. Reads carry the position of the user's last write and only use replicas that have replayed it A replica is used only once it has caught up to the user's write.

  4. Why monitor replica lag?

    Show answer

    B. So badly lagging replicas can be removed from read rotation before they serve very stale data Lag varies; a replica far behind should stop serving reads.

Recap

  • Asynchronous replicas lag; some read sequences then look broken to users.
  • Read-your-writes: users always see their own changes.
  • Monotonic reads: data never appears to go backwards for a user.
  • Consistent prefix: effects never appear before their causes.
  • Use leader reads after writes, log-position tokens, replica pinning, and lag monitoring.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up