Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Reliability and Resilience

System Design HLD-12.07

Cell-based architecture and shuffle sharding

Limiting blast radius: independent cells behind a thin router, and shuffle sharding so one bad customer can hurt only a sliver of others.

Advanced 12 min Last reviewed

The big idea

Redundancy protects you against a machine or a zone dying. It does not protect you against the failures that hit every copy at once: a bad deploy, a configuration mistake, a query of death that crashes any server it reaches, or one customer whose traffic overwhelms whatever it touches. In a single big shared system, those failures reach everyone, and the blast radiusThe share of users, customers or functionality affected when something fails. is 100 percent.

The answer is to deliberately divide the system so a failure can only reach part of it. Cell-based architectureAn architecture that splits a system into many independent, complete copies called cells, each serving a subset of customers, with a thin routing layer that sends each customer to its cell. splits it into many complete, independent copies, each serving a fixed subset of customers. A bad deploy or a poison request takes down one cell, and the rest never notice. Shuffle shardingAssigning each customer a random combination of a few workers out of many, so that two customers rarely share exactly the same set and one bad customer can fully disable only a tiny fraction of others. goes further by giving each customer its own random combination of resources, so even customers that share some resources almost never share all of them.

Think of a ship with watertight compartments. A hole floods one compartment, and the ship stays afloat. Shuffle sharding is like giving every passenger two lifeboats chosen at random from a large set: if a pair of boats sinks, only the few passengers assigned to exactly that pair are in trouble.

How it works

Step 1: Cells

Customers routed to independent cells, one cell failing A thin cell router receives all requests and sends each customer to its assigned cell. There are four cells, each a complete stack with app servers and its own database. Cell 3 has received a bad deploy and glows red; its customers are affected. Cells 1, 2 and 4 stay green and their customers are unaffected. Customers Cell router maps customer to cell Cell 1 app + DB, customers A-F Cell 2 app + DB, customers G-M Cell 3 bad deploy: customers N-S down Cell 4 app + DB, customers T-Z Figure 1. The router maps each customer to one of four complete, independent cells. A bad deploy or poison request in cell 3 affects only its quarter of customers; the other cells are untouched.

A cellA complete, independent copy of a service stack, with its own compute and data, serving a fixed subset of customers. is a full copy of the stack (app servers, queues, databases), sized to a fixed maximum, serving a subset of customers. Cells share nothing on the request path: no common database, no cross-cell calls. A thin cell router in front looks up each request's customer (or tenant) and forwards it to the right cell. The router must be as simple and robust as possible, because it is the one shared component: often just a lookup table cached everywhere.

Cells change how you operate:

  • Blast radius: with 20 cells, any cell-local failure affects about 5 percent of customers.
  • Safer deploys: roll out to one cell, watch it, then the next few, then the rest. A bad build is caught while it affects one cell.
  • Scaling by adding cells: when cells are full, add more instead of growing one giant system. Each cell stays at a size you have tested.
  • Noisy customers stay inside their cell, and a very large customer can get a cell of its own.

The costs: some duplication and idle capacity, a routing layer to build, tooling to move customers between cells, and features that span customers (global search, analytics) need a separate path.

Step 2: Shuffle sharding

Plain sharding with 8 workers in 4 shards of 2 means a customer that sends a request of death, crashing every worker it touches, takes down its shard and everyone on it: 25 percent of customers. Shuffle sharding instead gives each customer a random pair from all 8 workers. There are 28 possible pairs, so a crashed pair fully disables only customers assigned to exactly that pair, about 1 in 28. Customers that share one of the two workers still have the other, and with client-side retry to a different worker in their shard, they keep working.

shuffle.pyruns with python3 (standard library)
import random
from itertools import combinations
from math import comb

WORKERS, K, CUSTOMERS = 8, 2, 1000
rnd = random.Random(7)

# Plain sharding: 4 fixed shards of 2 workers; each customer lives on one shard.
plain = {c: (c % 4) * 2 for c in range(CUSTOMERS)}
# Shuffle sharding: each customer gets a random combination of 2 of the 8 workers.
shards = list(combinations(range(WORKERS), K))
shuffle = {c: set(rnd.choice(shards)) for c in range(CUSTOMERS)}

bad = 0                                   # customer 0 sends a poison request that kills its workers
dead_shuffle = shuffle[bad]
down_plain = sum(1 for c in range(1, CUSTOMERS) if plain[c] == plain[bad])
down_shuffle = sum(1 for c in range(1, CUSTOMERS) if shuffle[c] <= dead_shuffle)
print(f"possible shuffle shards of {K} from {WORKERS} workers: {comb(WORKERS, K)}")
print(f"plain sharding  : {down_plain} of {CUSTOMERS - 1} other customers fully down ({down_plain / (CUSTOMERS - 1):.0%})")
print(f"shuffle sharding: {down_shuffle} of {CUSTOMERS - 1} other customers fully down ({down_shuffle / (CUSTOMERS - 1):.1%})")
print(f"100 workers, shards of 5: {comb(100, 5):,} combinations")

Output

possible shuffle shards of 2 from 8 workers: 28
plain sharding  : 249 of 999 other customers fully down (25%)
shuffle sharding: 32 of 999 other customers fully down (3.2%)
100 workers, shards of 5: 75,287,520 combinations

The poison customer knocks out a quarter of customers under plain sharding but only about 3 percent with shuffle sharding, with exactly the same 8 workers. The numbers improve fast with scale: with 100 workers and shards of 5, there are over 75 million combinations, so it is extremely unlikely that two customers share a whole shard.

Step 3: Making shuffle sharding work

Shuffle sharding needs clients (or a front end) that retry on another member of their shard when one fails, otherwise sharing one worker with a bad customer still hurts. It also assumes the bad workload is confined to the resources it is assigned to, which means the workers must fail independently and not share a hidden dependency. It applies to anything assignable: servers, queues, name servers, cells themselves.

Going deeper

Cells versus availability zones

Zones protect against physical failures; cells protect against logical ones such as deploys, configuration and poison inputs. They combine: each cell typically spans several zones for its own redundancy, and deploys move cell by cell rather than zone by zone.

Choosing cell size

Smaller cells mean smaller blast radius and faster per-cell operations, but more cells to manage and more fragmentation of capacity. A cell must still be big enough for your largest customer, or that customer needs special handling.

Trade-offs

OptionWins whenLoses when
One shared system Small scale, simple operations Every logical failure affects everyone
Plain sharding Scaling data and load A bad shard takes down all of its customers
Cell-based architecture Large multi-tenant services needing small blast radius and safe deploys Duplication, a routing layer, cross-cell features and customer migration tooling
Shuffle sharding Isolating noisy or malicious customers with shared workers Needs retries across shard members and independent failures

Common mistakes

  • Building cells that still share a database, cache or configuration service on the request path.
  • A complex cell router that becomes the new single point of failure.
  • Deploying to every cell at once, giving up the main benefit of cells.
  • Shuffle sharding without client retries on other shard members.
  • Letting cells grow without limit instead of adding new cells.

In the wild

Amazon

Workload isolation using shuffle-sharding (opens in a new tab)

This Builders' Library article starts from a fleet of eight workers: with no sharding, a poison request or attack affects everyone; with four shards of two, the impact drops to 25 percent. With shuffle sharding there are 28 unique pairs of workers, so a problem customer affects only 1/28th of customers, which the article calls 7 times better than regular sharding. Amazon Route 53 applies the idea to DNS, assigning each hosted domain a shuffle shard of name servers out of 2,048 virtual name servers, giving about 730 billion possible shuffle shards.

Shopify

A pods architecture to allow Shopify to scale (opens in a new tab)

Shopify's 2018 post explains that sharding its database in 2015 brought scale but cost resilience, since code that touched every shard failed if any shard was down. It reorganized into pods: a pod is a set of shops on a fully isolated set of datastores, and shared resources such as app servers and job workers may talk to only one pod at a time. A component in the load balancers called Sorting Hat matches each request to its pod, and a Pod Mover tool can move a pod to its recovery data center in about a minute without dropping requests.

Check yourself

  1. A bad configuration is pushed to one of 20 cells. Roughly what share of customers is affected?

    Show answer

    B. About 5 percent Cells share nothing on the request path, so a cell-local failure reaches only its customers.

  2. With 8 workers and shuffle shards of 2, how many distinct shards are possible?

    Show answer

    C. 28 8 choose 2 is 28, so a full-shard collision affects about 1 in 28 customers.

  3. Why does shuffle sharding need clients to retry on another worker in their shard?

    Show answer

    B. Customers sharing one worker with a bad customer still have a healthy one, but only if they move to it Partial overlap is survivable only if requests can avoid the failed member.

  4. What makes the cell router a critical design point?

    Show answer

    B. It is the one shared component, so it must be extremely simple and robust Everything else is partitioned; a fragile router would reintroduce a global blast radius.

Recap

  • Blast radius is the share of customers a failure can reach; shared systems have 100 percent.
  • Cells are complete, independent stacks for subsets of customers behind a thin router.
  • Cells contain bad deploys, poison requests and noisy customers, and enable cell-by-cell rollouts.
  • Shuffle sharding gives each customer a random combination of workers, so full overlap is rare.
  • It needs retries across shard members and truly independent resources.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy