Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Unique IDs and Coordination Primitives

System Design HLD-11.02

Coordination services (ZooKeeper, etcd)

What ZooKeeper and etcd store (config, leaders, locks, membership), how watches and leases work, and why they are small and strongly consistent.

Advanced 10 min Last reviewed

The big idea

Earlier lessons kept saying "use etcd or ZooKeeper for that": for leader election ( Quorums and leader election ), for distributed locks with fencing tokens ( Distributed locks and fencing tokens ), for assigning Snowflake machine ids ( Generating unique IDs at scale ). Each of those needs a small amount of data that every node agrees on exactly, even while machines fail and the network splits. Building that correctly is hard, so it is built once and shared.

A coordination serviceA small, strongly consistent, replicated key-value store, such as ZooKeeper or etcd, used by other systems for configuration, leader election, locks and membership. is that shared building block: a tiny, replicated key-value store that runs consensus, gives linearizable reads and writes, and adds three features ordinary databases lack or do poorly: watches that push changes to clients, leases that make keys vanish when their owner dies, and compare-and-set transactions with ever-increasing revision numbers.

Think of the noticeboard in a ship's control room. Only a few things are pinned there: who is on watch, the current heading, which hatches are locked. Everyone trusts it completely, a crew member's name comes off the board the moment they leave the bridge, and a bell rings whenever anything changes.

How it works

Step 1: A small, consistent key-value store

ZooKeeper stores data in a tree of nodes called znodes, like a file system (/services/payments/leader). etcd stores a flat, sorted key space where prefixes act like directories (/config/feature-x). Both replicate every write through a consensus protocol, Raft for etcd and ZAB for ZooKeeper, across typically 3 or 5 servers ( Consensus: Raft ). A write succeeds only once a majority has it, so clients never see two different values for the same key, and the service survives the loss of a minority of servers.

etcd gives every change a cluster-wide, ever-increasing revision number. That number is exactly what a fencing token needs ( Distributed locks and fencing tokens ), and it lets clients say "watch everything after revision 4,812" without missing events.

Step 2: What systems keep there

  • Configuration and feature flags that every instance must agree on.
  • Leader election: whoever creates /leader/scheduler first is leader ( Quorums and leader election ).
  • Distributed locks with fencing tokens taken from revisions.
  • Membership and service discovery: each live instance registers a key tied to its session; when it dies, the key disappears.
  • Metadata: Kubernetes keeps all of its cluster state (pods, deployments, config) in etcd, and Kafka kept its cluster metadata in ZooKeeper for years before moving to its own Raft quorum, KRaft.

Step 3: Watches

Services watching keys in an etcd cluster A three-node etcd cluster in the center, with one leader and two followers replicating via Raft. An admin updates the key config/feature-x. The change is committed by a majority and immediately pushed as watch events to three API service instances, which glow as they receive it. A scheduler instance watches the leader key and is not notified by this change. etcd cluster (Raft) event Admin PUT feature-x=on etcd 1 leader etcd 2 etcd 3 api-1 watch feature-x api-2 watch feature-x api-3 watch feature-x scheduler watch /leader Figure 1. An admin writes /config/feature-x. Once a majority of the etcd cluster commits it, every client watching that key is pushed the change at once. Clients watching other keys hear nothing.

Polling a key every second wastes work and still reacts late. A watchA subscription to a key or prefix in a coordination service; the service pushes an event to the client whenever the key changes. asks the service to push an event whenever a key, or any key under a prefix, changes. The moment a flag flips or a leader disappears, every interested client knows. etcd watches stream every change from a given revision onward; ZooKeeper's classic watches fire once and must be re-registered, and newer versions add persistent watches.

Step 4: Leases and ephemeral keys

A leaseA time-limited grant that a client keeps alive by sending heartbeats; keys attached to it are deleted automatically when it expires. ties keys to a client's liveness. In etcd, a client creates a lease with a TTL, say 10 seconds, attaches keys to it, and keeps it alive with heartbeats. In ZooKeeper the same idea is an ephemeral znode, tied to the client's session. If the client crashes or is cut off, the heartbeats stop, the lease expires and its keys are deleted, and watchers are told.

Combine compare-and-set, leases and watches and you get leader election in three moves:

coord.pyruns with python3 (standard library)
class Coord:
    """A toy etcd: versioned keys, leases that expire, watches that fire on change."""
    def __init__(self):
        self.rev, self.data, self.watches = 0, {}, {}
    def put(self, key, value, lease=None, if_absent=False):
        if if_absent and key in self.data:              # compare-and-set: only if missing
            return False
        self.rev += 1
        self.data[key] = (value, lease, self.rev)
        self._notify(key, f"PUT {key}={value} @rev {self.rev}")
        return True
    def expire(self, lease):                            # lease holder stopped heartbeating
        for key in [k for k, (_, l, _) in self.data.items() if l == lease]:
            self.rev += 1
            del self.data[key]
            self._notify(key, f"DELETE {key} (lease {lease} expired) @rev {self.rev}")
    def watch(self, key, who):
        self.watches.setdefault(key, []).append(who)
    def _notify(self, key, event):
        for who in self.watches.get(key, []):
            print(f"  -> {who} notified: {event}")

etcd = Coord()
etcd.watch("/config/feature-x", "api-1")
etcd.watch("/leader/scheduler", "node-b")
print("config change:");        etcd.put("/config/feature-x", "on")
print("leader election:")
print("  node-a wins:", etcd.put("/leader/scheduler", "node-a", lease=17, if_absent=True))
print("  node-b wins:", etcd.put("/leader/scheduler", "node-b", lease=42, if_absent=True))
print("node-a crashes:");       etcd.expire(17)
print("  node-b wins:", etcd.put("/leader/scheduler", "node-b", lease=42, if_absent=True))

Output

config change:
  -> api-1 notified: PUT /config/feature-x=on @rev 1
leader election:
  -> node-b notified: PUT /leader/scheduler=node-a @rev 2
  node-a wins: True
  node-b wins: False
node-a crashes:
  -> node-b notified: DELETE /leader/scheduler (lease 17 expired) @rev 3
  -> node-b notified: PUT /leader/scheduler=node-b @rev 4
  node-b wins: True

node-a creates the leader key only if it is absent, attached to its lease, and wins. node-b's attempt fails, so it watches the key instead. When node-a crashes, its lease expires, the key is deleted, node-b is notified at once and takes over. Real recipes add one step: the new leader uses the key's revision as a fencing token, because node-a may only be paused, not dead ( Distributed locks and fencing tokens ).

Step 5: Why they are kept small

Every write goes through consensus on every member and the whole dataset is kept in memory (ZooKeeper) or in a bounded store (etcd has a default storage quota of 2 GiB). Throughput is limited by one leader and a majority round trip. That is a perfect fit for a few megabytes of critical metadata that changes occasionally, and a terrible one for user data, logs or anything high-volume. Treat the coordination service as a dependency for the control plane, not the data plane: if every request reads it, it becomes the bottleneck and its outage becomes yours.

Going deeper

Session timeouts are the hard part

A lease or session timeout is a failure detector ( Membership, gossip and failure detection ). Too short, and a garbage collection pause expires a healthy leader's lease, causing needless failovers. Too long, and a dead leader is not replaced for that whole time. And between the lease expiring on the server and the old client noticing, two nodes can believe they lead, which is why fencing tokens matter.

Herd effects

If 1,000 clients all watch one lock key and it is released, all 1,000 wake up and race to grab it. ZooKeeper's lock recipe avoids this with sequential nodes: each waiter watches only the node just before it in the queue, so a release wakes exactly one client.

Trade-offs

OptionWins whenLoses when
etcd Kubernetes-style metadata, config and election with revisions and streaming watches Limited data size and write throughput; another critical cluster to run
ZooKeeper Mature recipes for locks, queues and election in the JVM ecosystem Separate ensemble to operate; one-shot watches in older versions
Embedded consensus (a Raft library, KRaft) Systems that want no external dependency for their own metadata You own the correctness and operations of consensus
Regular database row with locks Simple leases and config at small scale where a DB already exists Watches, ephemeral keys and clean failover semantics

Common mistakes

  • Storing application data or large blobs in etcd or ZooKeeper.
  • Reading from the coordination service on every user request instead of watching and caching.
  • Session timeouts shorter than worst-case GC pauses, causing flapping leadership.
  • Electing a leader without fencing, trusting the lease alone.
  • Running an even number of members, or all members in one failure domain.

In the wild

Kubernetes stores its entire cluster state in etcd, and its controllers use leases there for leader election. Apache Kafka depended on ZooKeeper for cluster metadata for years before replacing it with KRaft. Hadoop, HBase and many JVM systems use ZooKeeper for election and configuration, and HashiCorp Consul provides a similar Raft-backed key-value store with service discovery.

Check yourself

  1. Why is a coordination service a poor place to store user profiles?

    Show answer

    B. Every write goes through consensus on every member and the data size is deliberately small, so it cannot handle high-volume data Coordination services trade throughput and capacity for strong consistency on small, critical data.

  2. A leader holds an etcd lease and crashes. What happens to its leader key?

    Show answer

    B. Its lease stops being renewed, expires, and the key is deleted; watchers are notified Keys attached to a lease disappear with it, which is how failover is triggered.

  3. What do watches replace?

    Show answer

    B. Polling a key repeatedly to see whether it changed Watches push changes as they happen, avoiding wasted polls and slow reactions.

  4. Why is a coordination service usually 3 or 5 nodes?

    Show answer

    B. It uses majority consensus; odd sizes tolerate the most failures per node, and more nodes slow every write 3 tolerates 1 failure and 5 tolerates 2; each extra member adds to every write's quorum.

Recap

  • ZooKeeper and etcd are small, consensus-replicated key-value stores with linearizable operations.
  • They hold config, leader keys, locks and membership, not application data.
  • Watches push changes to clients; leases (ephemeral nodes) delete keys when their owner dies.
  • Compare-and-set plus leases plus watches gives leader election; revisions give fencing tokens.
  • Keep them off the request path, tune session timeouts carefully, and run 3 or 5 members.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy