Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Reliability and Resilience

System Design HLD-12.04

Load shedding and graceful degradation

Serving some requests well beats serving all badly: priority-based load shedding, brownouts, serving stale data and feature kill switches.

Advanced 10 min Last reviewed

The big idea

A flash sale starts and traffic doubles in a minute. Autoscaling will add capacity, but not for several minutes. Until then the service has more work than it can do, and it has to choose. If it tries to serve everyone, every request queues, latency climbs past every client's timeout, and the service ends up doing lots of work whose answers nobody receives. Throughput of useful work collapses to near zero, and the checkout flow fails along with everything else.

The better choice is deliberate: serve some requests well rather than all requests badly. Load sheddingRejecting some incoming work, early and cheaply, when a system is beyond its capacity, so that the work it accepts completes in time. rejects the least important work first, as early and cheaply as possible. Graceful degradationKeeping core functionality working under stress by turning off or simplifying expensive, non-essential features. keeps core flows alive by switching off expensive extras, such as recommendations, live counters or high-resolution images, until the pressure passes.

Think of a hospital emergency room on a bad night. Triage decides who is seen first: heart attacks before sprained ankles. Some people wait or are sent elsewhere, but the critical cases get full care instead of everyone getting a rushed, dangerous minimum.

How it works

Step 1: Prioritize traffic

Traffic tiers dimming as load rises Three tiers of traffic feed a service: critical (checkout and login), normal (browse and search) and sheddable (recommendations and prefetch). Step 1: normal load, all three tiers glow and are served. Step 2: load passes capacity, the sheddable tier dims and is rejected. Step 3: severe overload, the normal tier is partly shed too, and only critical traffic is fully served. part Critical checkout, login Normal browse, search Sheddable recs, prefetch Service capacity 1,000 rps Shed early fast 503 or fallback Figure 1. As load rises, the lowest tier is shed first, then part of the next, so checkout and login keep working at full quality while extras are switched off. Step through rising load.
  1. Normal load Everything fits within capacity, so every tier is served.
  2. Load over capacity The sheddable tier (recommendations, prefetch) is rejected or replaced by a fallback; critical and normal traffic are unaffected.
  3. Severe overload Part of the normal tier is shed as well. Critical flows, checkout and login, are still fully served.

Shedding only works if requests carry a priority. Tag them at the edge: by endpoint (checkout beats recommendations), by caller (user-facing beats batch jobs and prefetches), by customer tier, or by retry status (a first attempt beats a third retry). Then, under pressure, reject from the bottom up:

shedding.pyruns with python3 (standard library)
CAPACITY = 1000                                   # requests per second the service can serve well
TIERS = [("critical: checkout, login", 300),       # (tier, requests per second offered)
         ("normal: browse, search", 500),
         ("sheddable: recommendations, prefetch", 400)]

def admit(load_factor, shed):
    """Returns how much of each tier is served at a given multiple of normal traffic."""
    offered = [(name, int(rps * load_factor)) for name, rps in TIERS]
    total = sum(r for _, r in offered)
    if not shed:                                   # no priorities: everyone gets the same share,
        share = min(1, CAPACITY / total)           # and past capacity, latency explodes for all
        return [(n, int(r * share)) for n, r in offered]
    left, served = CAPACITY, []
    for name, rps in offered:                      # serve tiers in priority order
        take = min(rps, left)
        served.append((name, take))
        left -= take
    return served

for load in (0.8, 1.2, 2.0):
    print(f"traffic x{load}:")
    for shed in (False, True):
        row = ", ".join(f"{n.split(':')[0]} {s}/{int(r * load)}" for (n, s), (_, r) in zip(admit(load, shed), TIERS))
        print(f"  {'priority shedding' if shed else 'no priorities     '}: {row}")

Output

traffic x0.8:
  no priorities     : critical 240/240, normal 400/400, sheddable 320/320
  priority shedding: critical 240/240, normal 400/400, sheddable 320/320
traffic x1.2:
  no priorities     : critical 250/360, normal 416/600, sheddable 333/480
  priority shedding: critical 360/360, normal 600/600, sheddable 40/480
traffic x2.0:
  no priorities     : critical 250/600, normal 416/1000, sheddable 333/800
  priority shedding: critical 600/600, normal 400/1000, sheddable 0/800

At 1.2 times normal traffic, giving every request an equal share means a third of checkouts fail. With priority shedding, all checkouts and all browsing succeed, and only recommendations are cut. At 2 times, checkout is still fully served. In reality, the unprioritized case is worse than the table shows, because an overloaded service also slows down for everyone.

Step 2: Shed early and cheaply

A rejected request should cost almost nothing: reject at the load balancer or in the first middleware, before authentication, database calls or rendering. Shedding a request after doing 90 percent of its work saves nothing. Good signals for "we are overloaded" are concurrency (requests in flight), queue length, and queueing time, more than CPU. A request that has already waited longer than its client's timeout should be dropped, not processed ( Backpressure and flow control ). Reply quickly with 503 or 429 and a Retry-After hint so clients back off rather than retrying at once ( Timeouts, retries, backoff and jitter ).

Step 3: Degrade instead of failing

Many features can get cheaper instead of disappearing:

  • Serve stale data: return cached product pages, prices or profiles that are a few minutes old instead of recomputing ( Thundering herd, cache stampede and hot keys ). For most reads, slightly stale beats unavailable.
  • Brownouts: temporarily turn off expensive optional features: personalized recommendations become a static best-seller list, search drops spelling correction, images are served at lower resolution, infinite scroll stops prefetching.
  • Simplify writes: accept the order and queue the confirmation email, the loyalty points and the analytics for later.

Step 4: Kill switches

All of this must be quick to apply during an incident. Expensive and risky features should sit behind kill switchesA feature flag that lets operators instantly disable a feature in production without deploying code.: feature flags that operators can flip in seconds, without a deploy. Better still, some switches flip automatically when load or latency crosses a threshold, and every switch is exercised regularly so it is known to work when it is needed.

Going deeper

Shedding versus rate limiting

Rate limiting caps each client's share whether or not the system is busy ( Rate limiting algorithms ). Load shedding looks at the system's own health and drops work only when it is overloaded, regardless of who sent it. You want both: rate limits keep one client from hogging capacity, shedding protects the service when the total is too much.

Avoid shedding the same users repeatedly

Random shedding can hit the same user on every page load, which is worse than a consistent experience. Some systems shed by user hash so a small set of users is affected for a short time, or prefer to shed new sessions over users halfway through a purchase.

Trade-offs

OptionWins whenLoses when
No shedding Nothing beyond small, predictable workloads Everything at once when overloaded, including the most important flows
Priority-based shedding Protecting critical flows during spikes and partial outages Needs request classification and honest priorities
Serving stale data Read-heavy pages where freshness of seconds or minutes is acceptable Correctness for data that must be current, such as balances
Brownouts and kill switches Expensive optional features during incidents Reduced functionality; switches must be tested to be trusted

Common mistakes

  • Treating all traffic as equal, so an overload fails checkout as often as prefetching.
  • Shedding late, after authentication, database work and rendering are already done.
  • Using CPU as the only overload signal while requests pile up waiting on I/O.
  • Kill switches that were never tested and do not work during the incident.
  • Returning slow errors instead of fast 503s, so clients time out and retry harder.

In the wild

Large sites routinely degrade under load: shopping sites hide recommendations and reviews to keep checkout fast, streaming services lower video quality, and social apps show cached feeds when ranking is overloaded. Rate limiting and load shedding are often combined, as in the Stripe case study in Rate limiting algorithms , which describes load shedders that reserve capacity for the most important API requests.

Check yourself

  1. Traffic is 1.5 times capacity. Why is serving everyone equally worse than shedding by priority?

    Show answer

    B. Every request slows down and many time out, so even critical flows fail; shedding keeps critical flows fully served Overload hurts everyone; deliberate shedding chooses who is protected.

  2. Where should a request be shed?

    Show answer

    B. As early as possible, ideally at the edge, before expensive work Shedding saves capacity only if the rejected request cost almost nothing.

  3. Which is a graceful degradation?

    Show answer

    B. Replacing personalized recommendations with a cached best-seller list during overload The page still works; only an expensive optional feature is simplified.

  4. What makes a kill switch useful in an incident?

    Show answer

    B. It can disable a feature in seconds without a deploy, and it has been tested Speed and confidence matter most mid-incident.

Recap

  • When demand exceeds capacity, serve some requests well rather than all badly.
  • Classify requests by priority and shed the lowest tiers first.
  • Shed early and cheaply, using concurrency and queueing time as signals, with fast 503 or 429 replies.
  • Degrade gracefully: stale data, brownouts of optional features, deferred work.
  • Put expensive features behind tested kill switches that operators can flip instantly.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy