Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Scaling Data

System Design HLD-07.06

Hot partitions and the celebrity problem

When one key is far hotter than the rest: key salting, splitting hot partitions, sharded counters, caching hot reads, and detecting hot keys early.

Advanced 10 min Last reviewed

The big idea

A perfect shard key spreads keys evenly. It cannot spread traffic evenly when the traffic itself is skewed. A celebrity with 100 million followers posts, and every like, comment and view of that one post goes to the one shard that owns it. A flash sale hammers one product's inventory row. A viral video's view counter takes thousands of increments per second. The cluster as a whole is mostly idle while one shard melts: a hot partitionA partition that receives a disproportionate share of reads or writes, overloading its node while others sit idle..

Adding more shards does not help, because the hot key still lives on exactly one of them. The fixes all change how the hot key itself is stored or served: split its writes across several keys, split its partition, absorb its reads in caches, or isolate it on dedicated capacity.

Think of a bank with ten tellers where one customer brings a thousand separate deposits. Adding an eleventh teller does nothing for the queue behind that customer; splitting the deposits across several tellers does.

How it works

Step 1: See the hotspot

A hot shard split into cooler pieces Left: six shards, one glowing red with most of the traffic because it holds a celebrity's post. Right: the hot key's writes are salted across four sub-keys that land on four different shards, so every shard is moderately loaded. Before: one hot shard After: salted across shards S1 8% S2 7% S3 61% hot S4 8% S5 8% S6 8% S1 17% S2 16% S3 18% S4 17% S5 16% S6 16% Figure 1. Left, one shard holds the celebrity's key and takes most of the traffic (red). Right, salting splits that key into sub-keys spread across shards, so the load evens out.

Detecting hot keys is half the battle. Track request rates per key or per partition (sampled, to keep it cheap), alert when one partition's load is far above the median, and log the top keys. Many outages blamed on "the database" are one key.

Step 2: Salt hot keys

Append a random suffix to a hot key on write, so likes:post123 becomes likes:post123#0 to #15, spread across shards. Reads sum the sub-keys:

salting.pyruns with python3 (standard library)
import hashlib, random
from collections import Counter

SHARDS = 8
def shard_of(key: str) -> int:
    return int(hashlib.md5(key.encode()).hexdigest(), 16) % SHARDS

random.seed(4)
# 100,000 like events: 40% are for one viral post, the rest spread over 10,000 posts
events = ["post:viral" if random.random() < 0.4 else f"post:{random.randrange(10_000)}"
          for _ in range(100_000)]

plain = Counter(shard_of(f"likes:{p}") for p in events)

SALTS = 16   # the hot key's counter is split into 16 sub-counters
def write_key(p):
    return f"likes:{p}#{random.randrange(SALTS)}" if p == "post:viral" else f"likes:{p}"
salted = Counter(shard_of(write_key(p)) for p in events)

def show(name, c):
    loads = [c[s] for s in range(SHARDS)]
    print(f"{name:8} busiest shard {max(loads):6,} writes = {max(loads) / (len(events) / SHARDS):.1f}x fair share")

show("plain", plain)
show("salted", salted)
print(f"reading the viral count now sums {SALTS} keys: likes:post:viral#0 .. #{SALTS - 1}")

Output

plain    busiest shard 47,853 writes = 3.8x fair share
salted   busiest shard 17,765 writes = 1.4x fair share
reading the viral count now sums 16 keys: likes:post:viral#0 .. #15

With 40 percent of all writes going to one post, the shard holding it carries 3.8 times its fair share. Salting that one key over 16 sub-keys brings the busiest shard down to 1.4 times. The cost moves to reads, which must now fetch and add 16 values, so salt only the keys that need it, and cache the summed result.

Step 3: Other fixes

  • Sharded counters: the same idea for counters: many sub-counters incremented at random, summed on read or periodically rolled up.
  • Split hot partitions: in range-partitioned systems, split a hot range into smaller ranges on separate nodes; some databases do this automatically.
  • Cache hot reads: serve the celebrity's profile or the viral post from caches and replicas, possibly a local in-process cache, so reads never reach the shard ( Thundering herd, cache stampede and hot keys ).
  • Isolate: give known huge tenants or celebrities dedicated shards or capacity, using a directory-based mapping.
  • Absorb writes: buffer increments in memory or a queue and write aggregated batches.

Going deeper

Fan-out for celebrities

Social feeds usually push a new post into each follower's precomputed timeline (fan-out on write). For an account with 100 million followers that is 100 million writes per post, so feed systems switch to fan-out on read for celebrities: their posts are merged into a follower's feed when it is loaded. You will design exactly this in the news feed case study ( Design a news feed (Twitter/Instagram) ).

Trade-offs

OptionWins whenLoses when
Key salting / sharded counters Write-hot keys such as counters and likes Read cost (sum N keys) and code complexity
Caching hot reads Read-hot keys such as celebrity profiles and viral posts Freshness; does nothing for write-hot keys
Dedicated capacity for huge tenants A few known, very large customers Operational overhead and a directory to maintain

Common mistakes

  • Adding shards to fix a single hot key.
  • Salting every key "just in case", making all reads expensive.
  • Not measuring per-key load, so hot keys are found only during an outage.
  • Forgetting that a hot key also concentrates load on its cache node.

Check yourself

  1. Why doesn't adding shards fix a hot key?

    Show answer

    B. The hot key still lives on exactly one shard Sharding spreads keys, but a single key's traffic stays on one shard.

  2. What does key salting do?

    Show answer

    B. Splits one hot key into several sub-keys spread across shards, combined on read Writes are spread by a random suffix; reads aggregate the sub-keys.

  3. A celebrity profile is read 200,000 times per second but rarely changes. What helps most?

    Show answer

    B. Caching the profile, including locally in app servers It is read-hot, not write-hot; caching absorbs the reads.

  4. What is the cost of salting a counter into 16 sub-counters?

    Show answer

    B. Reading the total requires fetching and summing 16 values Write load spreads, but reads do 16 times the work unless the sum is cached.

Recap

  • Sharding spreads keys, not traffic: one skewed key can overload one shard.
  • Measure per-key and per-partition load to find hot keys early.
  • Salt write-hot keys and shard counters; sum on read.
  • Cache read-hot keys, split hot ranges, and isolate huge tenants.
  • For celebrities in feeds, switch from fan-out on write to fan-out on read.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up