System Design HLD-11.02
Coordination services (ZooKeeper, etcd)
What ZooKeeper and etcd store (config, leaders, locks, membership), how watches and leases work, and why they are small and strongly consistent.
The big idea
Earlier lessons kept saying "use etcd or ZooKeeper for that": for leader election ( Quorums and leader election ), for distributed locks with fencing tokens ( Distributed locks and fencing tokens ), for assigning Snowflake machine ids ( Generating unique IDs at scale ). Each of those needs a small amount of data that every node agrees on exactly, even while machines fail and the network splits. Building that correctly is hard, so it is built once and shared.
A coordination serviceA small, strongly consistent, replicated key-value store, such as ZooKeeper or etcd, used by other systems for configuration, leader election, locks and membership. is that shared building block: a tiny, replicated key-value store that runs consensus, gives linearizable reads and writes, and adds three features ordinary databases lack or do poorly: watches that push changes to clients, leases that make keys vanish when their owner dies, and compare-and-set transactions with ever-increasing revision numbers.
Think of the noticeboard in a ship's control room. Only a few things are pinned there: who is on watch, the current heading, which hatches are locked. Everyone trusts it completely, a crew member's name comes off the board the moment they leave the bridge, and a bell rings whenever anything changes.
How it works
Step 1: A small, consistent key-value store
ZooKeeper stores data in a tree of nodes called znodes, like a file system (/services/payments/leader). etcd stores a flat, sorted key space where prefixes act like directories (/config/feature-x). Both replicate every write through a consensus protocol, Raft for etcd and ZAB for ZooKeeper, across typically 3 or 5 servers ( Consensus: Raft ). A write succeeds only once a majority has it, so clients never see two different values for the same key, and the service survives the loss of a minority of servers.
etcd gives every change a cluster-wide, ever-increasing revision number. That number is exactly what a fencing token needs ( Distributed locks and fencing tokens ), and it lets clients say "watch everything after revision 4,812" without missing events.
Step 2: What systems keep there
- Configuration and feature flags that every instance must agree on.
- Leader election: whoever creates
/leader/schedulerfirst is leader ( Quorums and leader election ). - Distributed locks with fencing tokens taken from revisions.
- Membership and service discovery: each live instance registers a key tied to its session; when it dies, the key disappears.
- Metadata: Kubernetes keeps all of its cluster state (pods, deployments, config) in etcd, and Kafka kept its cluster metadata in ZooKeeper for years before moving to its own Raft quorum, KRaft.
Step 3: Watches
Polling a key every second wastes work and still reacts late. A watchA subscription to a key or prefix in a coordination service; the service pushes an event to the client whenever the key changes. asks the service to push an event whenever a key, or any key under a prefix, changes. The moment a flag flips or a leader disappears, every interested client knows. etcd watches stream every change from a given revision onward; ZooKeeper's classic watches fire once and must be re-registered, and newer versions add persistent watches.
Step 4: Leases and ephemeral keys
A leaseA time-limited grant that a client keeps alive by sending heartbeats; keys attached to it are deleted automatically when it expires. ties keys to a client's liveness. In etcd, a client creates a lease with a TTL, say 10 seconds, attaches keys to it, and keeps it alive with heartbeats. In ZooKeeper the same idea is an ephemeral znode, tied to the client's session. If the client crashes or is cut off, the heartbeats stop, the lease expires and its keys are deleted, and watchers are told.
Combine compare-and-set, leases and watches and you get leader election in three moves:
class Coord:
"""A toy etcd: versioned keys, leases that expire, watches that fire on change."""
def __init__(self):
self.rev, self.data, self.watches = 0, {}, {}
def put(self, key, value, lease=None, if_absent=False):
if if_absent and key in self.data: # compare-and-set: only if missing
return False
self.rev += 1
self.data[key] = (value, lease, self.rev)
self._notify(key, f"PUT {key}={value} @rev {self.rev}")
return True
def expire(self, lease): # lease holder stopped heartbeating
for key in [k for k, (_, l, _) in self.data.items() if l == lease]:
self.rev += 1
del self.data[key]
self._notify(key, f"DELETE {key} (lease {lease} expired) @rev {self.rev}")
def watch(self, key, who):
self.watches.setdefault(key, []).append(who)
def _notify(self, key, event):
for who in self.watches.get(key, []):
print(f" -> {who} notified: {event}")
etcd = Coord()
etcd.watch("/config/feature-x", "api-1")
etcd.watch("/leader/scheduler", "node-b")
print("config change:"); etcd.put("/config/feature-x", "on")
print("leader election:")
print(" node-a wins:", etcd.put("/leader/scheduler", "node-a", lease=17, if_absent=True))
print(" node-b wins:", etcd.put("/leader/scheduler", "node-b", lease=42, if_absent=True))
print("node-a crashes:"); etcd.expire(17)
print(" node-b wins:", etcd.put("/leader/scheduler", "node-b", lease=42, if_absent=True))Output
config change:
-> api-1 notified: PUT /config/feature-x=on @rev 1
leader election:
-> node-b notified: PUT /leader/scheduler=node-a @rev 2
node-a wins: True
node-b wins: False
node-a crashes:
-> node-b notified: DELETE /leader/scheduler (lease 17 expired) @rev 3
-> node-b notified: PUT /leader/scheduler=node-b @rev 4
node-b wins: True node-a creates the leader key only if it is absent, attached to its lease, and wins. node-b's attempt fails, so it watches the key instead. When node-a crashes, its lease expires, the key is deleted, node-b is notified at once and takes over. Real recipes add one step: the new leader uses the key's revision as a fencing token, because node-a may only be paused, not dead ( Distributed locks and fencing tokens ).
Step 5: Why they are kept small
Every write goes through consensus on every member and the whole dataset is kept in memory (ZooKeeper) or in a bounded store (etcd has a default storage quota of 2 GiB). Throughput is limited by one leader and a majority round trip. That is a perfect fit for a few megabytes of critical metadata that changes occasionally, and a terrible one for user data, logs or anything high-volume. Treat the coordination service as a dependency for the control plane, not the data plane: if every request reads it, it becomes the bottleneck and its outage becomes yours.
Going deeper
Session timeouts are the hard part
A lease or session timeout is a failure detector ( Membership, gossip and failure detection ). Too short, and a garbage collection pause expires a healthy leader's lease, causing needless failovers. Too long, and a dead leader is not replaced for that whole time. And between the lease expiring on the server and the old client noticing, two nodes can believe they lead, which is why fencing tokens matter.
Herd effects
If 1,000 clients all watch one lock key and it is released, all 1,000 wake up and race to grab it. ZooKeeper's lock recipe avoids this with sequential nodes: each waiter watches only the node just before it in the queue, so a release wakes exactly one client.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| etcd | Kubernetes-style metadata, config and election with revisions and streaming watches | Limited data size and write throughput; another critical cluster to run |
| ZooKeeper | Mature recipes for locks, queues and election in the JVM ecosystem | Separate ensemble to operate; one-shot watches in older versions |
| Embedded consensus (a Raft library, KRaft) | Systems that want no external dependency for their own metadata | You own the correctness and operations of consensus |
| Regular database row with locks | Simple leases and config at small scale where a DB already exists | Watches, ephemeral keys and clean failover semantics |
Common mistakes
- Storing application data or large blobs in etcd or ZooKeeper.
- Reading from the coordination service on every user request instead of watching and caching.
- Session timeouts shorter than worst-case GC pauses, causing flapping leadership.
- Electing a leader without fencing, trusting the lease alone.
- Running an even number of members, or all members in one failure domain.
In the wild
Kubernetes stores its entire cluster state in etcd, and its controllers use leases there for leader election. Apache Kafka depended on ZooKeeper for cluster metadata for years before replacing it with KRaft. Hadoop, HBase and many JVM systems use ZooKeeper for election and configuration, and HashiCorp Consul provides a similar Raft-backed key-value store with service discovery.
Check yourself
-
Why is a coordination service a poor place to store user profiles?
Show answer
B. Every write goes through consensus on every member and the data size is deliberately small, so it cannot handle high-volume data Coordination services trade throughput and capacity for strong consistency on small, critical data.
-
A leader holds an etcd lease and crashes. What happens to its leader key?
Show answer
B. Its lease stops being renewed, expires, and the key is deleted; watchers are notified Keys attached to a lease disappear with it, which is how failover is triggered.
-
What do watches replace?
Show answer
B. Polling a key repeatedly to see whether it changed Watches push changes as they happen, avoiding wasted polls and slow reactions.
-
Why is a coordination service usually 3 or 5 nodes?
Show answer
B. It uses majority consensus; odd sizes tolerate the most failures per node, and more nodes slow every write 3 tolerates 1 failure and 5 tolerates 2; each extra member adds to every write's quorum.
Recap
- ZooKeeper and etcd are small, consensus-replicated key-value stores with linearizable operations.
- They hold config, leader keys, locks and membership, not application data.
- Watches push changes to clients; leases (ephemeral nodes) delete keys when their owner dies.
- Compare-and-set plus leases plus watches gives leader election; revisions give fencing tokens.
- Keep them off the request path, tune session timeouts carefully, and run 3 or 5 members.