System Design HLD-12.04
Load shedding and graceful degradation
Serving some requests well beats serving all badly: priority-based load shedding, brownouts, serving stale data and feature kill switches.
The big idea
A flash sale starts and traffic doubles in a minute. Autoscaling will add capacity, but not for several minutes. Until then the service has more work than it can do, and it has to choose. If it tries to serve everyone, every request queues, latency climbs past every client's timeout, and the service ends up doing lots of work whose answers nobody receives. Throughput of useful work collapses to near zero, and the checkout flow fails along with everything else.
The better choice is deliberate: serve some requests well rather than all requests badly. Load sheddingRejecting some incoming work, early and cheaply, when a system is beyond its capacity, so that the work it accepts completes in time. rejects the least important work first, as early and cheaply as possible. Graceful degradationKeeping core functionality working under stress by turning off or simplifying expensive, non-essential features. keeps core flows alive by switching off expensive extras, such as recommendations, live counters or high-resolution images, until the pressure passes.
Think of a hospital emergency room on a bad night. Triage decides who is seen first: heart attacks before sprained ankles. Some people wait or are sent elsewhere, but the critical cases get full care instead of everyone getting a rushed, dangerous minimum.
How it works
Step 1: Prioritize traffic
- Normal load Everything fits within capacity, so every tier is served.
- Load over capacity The sheddable tier (recommendations, prefetch) is rejected or replaced by a fallback; critical and normal traffic are unaffected.
- Severe overload Part of the normal tier is shed as well. Critical flows, checkout and login, are still fully served.
Shedding only works if requests carry a priority. Tag them at the edge: by endpoint (checkout beats recommendations), by caller (user-facing beats batch jobs and prefetches), by customer tier, or by retry status (a first attempt beats a third retry). Then, under pressure, reject from the bottom up:
CAPACITY = 1000 # requests per second the service can serve well
TIERS = [("critical: checkout, login", 300), # (tier, requests per second offered)
("normal: browse, search", 500),
("sheddable: recommendations, prefetch", 400)]
def admit(load_factor, shed):
"""Returns how much of each tier is served at a given multiple of normal traffic."""
offered = [(name, int(rps * load_factor)) for name, rps in TIERS]
total = sum(r for _, r in offered)
if not shed: # no priorities: everyone gets the same share,
share = min(1, CAPACITY / total) # and past capacity, latency explodes for all
return [(n, int(r * share)) for n, r in offered]
left, served = CAPACITY, []
for name, rps in offered: # serve tiers in priority order
take = min(rps, left)
served.append((name, take))
left -= take
return served
for load in (0.8, 1.2, 2.0):
print(f"traffic x{load}:")
for shed in (False, True):
row = ", ".join(f"{n.split(':')[0]} {s}/{int(r * load)}" for (n, s), (_, r) in zip(admit(load, shed), TIERS))
print(f" {'priority shedding' if shed else 'no priorities '}: {row}")Output
traffic x0.8:
no priorities : critical 240/240, normal 400/400, sheddable 320/320
priority shedding: critical 240/240, normal 400/400, sheddable 320/320
traffic x1.2:
no priorities : critical 250/360, normal 416/600, sheddable 333/480
priority shedding: critical 360/360, normal 600/600, sheddable 40/480
traffic x2.0:
no priorities : critical 250/600, normal 416/1000, sheddable 333/800
priority shedding: critical 600/600, normal 400/1000, sheddable 0/800 At 1.2 times normal traffic, giving every request an equal share means a third of checkouts fail. With priority shedding, all checkouts and all browsing succeed, and only recommendations are cut. At 2 times, checkout is still fully served. In reality, the unprioritized case is worse than the table shows, because an overloaded service also slows down for everyone.
Step 2: Shed early and cheaply
A rejected request should cost almost nothing: reject at the load balancer or in the first middleware, before authentication, database calls or rendering. Shedding a request after doing 90 percent of its work saves nothing. Good signals for "we are overloaded" are concurrency (requests in flight), queue length, and queueing time, more than CPU. A request that has already waited longer than its client's timeout should be dropped, not processed ( Backpressure and flow control ). Reply quickly with 503 or 429 and a Retry-After hint so clients back off rather than retrying at once ( Timeouts, retries, backoff and jitter ).
Step 3: Degrade instead of failing
Many features can get cheaper instead of disappearing:
- Serve stale data: return cached product pages, prices or profiles that are a few minutes old instead of recomputing ( Thundering herd, cache stampede and hot keys ). For most reads, slightly stale beats unavailable.
- Brownouts: temporarily turn off expensive optional features: personalized recommendations become a static best-seller list, search drops spelling correction, images are served at lower resolution, infinite scroll stops prefetching.
- Simplify writes: accept the order and queue the confirmation email, the loyalty points and the analytics for later.
Step 4: Kill switches
All of this must be quick to apply during an incident. Expensive and risky features should sit behind kill switchesA feature flag that lets operators instantly disable a feature in production without deploying code.: feature flags that operators can flip in seconds, without a deploy. Better still, some switches flip automatically when load or latency crosses a threshold, and every switch is exercised regularly so it is known to work when it is needed.
Going deeper
Shedding versus rate limiting
Rate limiting caps each client's share whether or not the system is busy ( Rate limiting algorithms ). Load shedding looks at the system's own health and drops work only when it is overloaded, regardless of who sent it. You want both: rate limits keep one client from hogging capacity, shedding protects the service when the total is too much.
Avoid shedding the same users repeatedly
Random shedding can hit the same user on every page load, which is worse than a consistent experience. Some systems shed by user hash so a small set of users is affected for a short time, or prefer to shed new sessions over users halfway through a purchase.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| No shedding | Nothing beyond small, predictable workloads | Everything at once when overloaded, including the most important flows |
| Priority-based shedding | Protecting critical flows during spikes and partial outages | Needs request classification and honest priorities |
| Serving stale data | Read-heavy pages where freshness of seconds or minutes is acceptable | Correctness for data that must be current, such as balances |
| Brownouts and kill switches | Expensive optional features during incidents | Reduced functionality; switches must be tested to be trusted |
Common mistakes
- Treating all traffic as equal, so an overload fails checkout as often as prefetching.
- Shedding late, after authentication, database work and rendering are already done.
- Using CPU as the only overload signal while requests pile up waiting on I/O.
- Kill switches that were never tested and do not work during the incident.
- Returning slow errors instead of fast 503s, so clients time out and retry harder.
In the wild
Large sites routinely degrade under load: shopping sites hide recommendations and reviews to keep checkout fast, streaming services lower video quality, and social apps show cached feeds when ranking is overloaded. Rate limiting and load shedding are often combined, as in the Stripe case study in Rate limiting algorithms , which describes load shedders that reserve capacity for the most important API requests.
Check yourself
-
Traffic is 1.5 times capacity. Why is serving everyone equally worse than shedding by priority?
Show answer
B. Every request slows down and many time out, so even critical flows fail; shedding keeps critical flows fully served Overload hurts everyone; deliberate shedding chooses who is protected.
-
Where should a request be shed?
Show answer
B. As early as possible, ideally at the edge, before expensive work Shedding saves capacity only if the rejected request cost almost nothing.
-
Which is a graceful degradation?
Show answer
B. Replacing personalized recommendations with a cached best-seller list during overload The page still works; only an expensive optional feature is simplified.
-
What makes a kill switch useful in an incident?
Show answer
B. It can disable a feature in seconds without a deploy, and it has been tested Speed and confidence matter most mid-incident.
Recap
- When demand exceeds capacity, serve some requests well rather than all badly.
- Classify requests by priority and shed the lowest tiers first.
- Shed early and cheaply, using concurrency and queueing time as signals, with fast 503 or 429 replies.
- Degrade gracefully: stale data, brownouts of optional features, deferred work.
- Put expensive features behind tested kill switches that operators can flip instantly.