System Design HLD-07.06
Hot partitions and the celebrity problem
When one key is far hotter than the rest: key salting, splitting hot partitions, sharded counters, caching hot reads, and detecting hot keys early.
The big idea
A perfect shard key spreads keys evenly. It cannot spread traffic evenly when the traffic itself is skewed. A celebrity with 100 million followers posts, and every like, comment and view of that one post goes to the one shard that owns it. A flash sale hammers one product's inventory row. A viral video's view counter takes thousands of increments per second. The cluster as a whole is mostly idle while one shard melts: a hot partitionA partition that receives a disproportionate share of reads or writes, overloading its node while others sit idle..
Adding more shards does not help, because the hot key still lives on exactly one of them. The fixes all change how the hot key itself is stored or served: split its writes across several keys, split its partition, absorb its reads in caches, or isolate it on dedicated capacity.
Think of a bank with ten tellers where one customer brings a thousand separate deposits. Adding an eleventh teller does nothing for the queue behind that customer; splitting the deposits across several tellers does.
How it works
Step 1: See the hotspot
Detecting hot keys is half the battle. Track request rates per key or per partition (sampled, to keep it cheap), alert when one partition's load is far above the median, and log the top keys. Many outages blamed on "the database" are one key.
Step 2: Salt hot keys
Append a random suffix to a hot key on write, so likes:post123 becomes likes:post123#0 to #15, spread across shards. Reads sum the sub-keys:
import hashlib, random
from collections import Counter
SHARDS = 8
def shard_of(key: str) -> int:
return int(hashlib.md5(key.encode()).hexdigest(), 16) % SHARDS
random.seed(4)
# 100,000 like events: 40% are for one viral post, the rest spread over 10,000 posts
events = ["post:viral" if random.random() < 0.4 else f"post:{random.randrange(10_000)}"
for _ in range(100_000)]
plain = Counter(shard_of(f"likes:{p}") for p in events)
SALTS = 16 # the hot key's counter is split into 16 sub-counters
def write_key(p):
return f"likes:{p}#{random.randrange(SALTS)}" if p == "post:viral" else f"likes:{p}"
salted = Counter(shard_of(write_key(p)) for p in events)
def show(name, c):
loads = [c[s] for s in range(SHARDS)]
print(f"{name:8} busiest shard {max(loads):6,} writes = {max(loads) / (len(events) / SHARDS):.1f}x fair share")
show("plain", plain)
show("salted", salted)
print(f"reading the viral count now sums {SALTS} keys: likes:post:viral#0 .. #{SALTS - 1}")Output
plain busiest shard 47,853 writes = 3.8x fair share
salted busiest shard 17,765 writes = 1.4x fair share
reading the viral count now sums 16 keys: likes:post:viral#0 .. #15 With 40 percent of all writes going to one post, the shard holding it carries 3.8 times its fair share. Salting that one key over 16 sub-keys brings the busiest shard down to 1.4 times. The cost moves to reads, which must now fetch and add 16 values, so salt only the keys that need it, and cache the summed result.
Step 3: Other fixes
- Sharded counters: the same idea for counters: many sub-counters incremented at random, summed on read or periodically rolled up.
- Split hot partitions: in range-partitioned systems, split a hot range into smaller ranges on separate nodes; some databases do this automatically.
- Cache hot reads: serve the celebrity's profile or the viral post from caches and replicas, possibly a local in-process cache, so reads never reach the shard ( Thundering herd, cache stampede and hot keys ).
- Isolate: give known huge tenants or celebrities dedicated shards or capacity, using a directory-based mapping.
- Absorb writes: buffer increments in memory or a queue and write aggregated batches.
Going deeper
Fan-out for celebrities
Social feeds usually push a new post into each follower's precomputed timeline (fan-out on write). For an account with 100 million followers that is 100 million writes per post, so feed systems switch to fan-out on read for celebrities: their posts are merged into a follower's feed when it is loaded. You will design exactly this in the news feed case study ( Design a news feed (Twitter/Instagram) ).
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Key salting / sharded counters | Write-hot keys such as counters and likes | Read cost (sum N keys) and code complexity |
| Caching hot reads | Read-hot keys such as celebrity profiles and viral posts | Freshness; does nothing for write-hot keys |
| Dedicated capacity for huge tenants | A few known, very large customers | Operational overhead and a directory to maintain |
Common mistakes
- Adding shards to fix a single hot key.
- Salting every key "just in case", making all reads expensive.
- Not measuring per-key load, so hot keys are found only during an outage.
- Forgetting that a hot key also concentrates load on its cache node.
Check yourself
-
Why doesn't adding shards fix a hot key?
Show answer
B. The hot key still lives on exactly one shard Sharding spreads keys, but a single key's traffic stays on one shard.
-
What does key salting do?
Show answer
B. Splits one hot key into several sub-keys spread across shards, combined on read Writes are spread by a random suffix; reads aggregate the sub-keys.
-
A celebrity profile is read 200,000 times per second but rarely changes. What helps most?
Show answer
B. Caching the profile, including locally in app servers It is read-hot, not write-hot; caching absorbs the reads.
-
What is the cost of salting a counter into 16 sub-counters?
Show answer
B. Reading the total requires fetching and summing 16 values Write load spreads, but reads do 16 times the work unless the sum is cached.
Recap
- Sharding spreads keys, not traffic: one skewed key can overload one shard.
- Measure per-key and per-partition load to find hot keys early.
- Salt write-hot keys and shard counters; sum on read.
- Cache read-hot keys, split hot ranges, and isolate huge tenants.
- For celebrities in feeds, switch from fan-out on write to fan-out on read.
Next up
- Crack the system design round learning path