Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Reliability and Resilience

System Design HLD-12.01

Timeouts, retries, backoff and jitter

Why every remote call needs a timeout, how naive retries cause retry storms, and how exponential backoff, jitter and retry budgets fix them.

Intermediate 12 min Last reviewed

The big idea

Your checkout service calls a pricing service. One day the pricing service does not fail; it just gets slow, taking 30 seconds instead of 30 milliseconds. Checkout's threads all end up waiting on it. New checkout requests queue behind them, checkout stops responding, and now everything that depends on checkout fails too. A dependency you barely think about took down your main flow, not by crashing, but by being slow.

Two tools prevent this, and used naively the second makes things worse. A timeout caps how long you wait. A retry tries again after a failure, which hides brief glitches. But when a dependency is overloaded, every client retrying at once multiplies the load just as it tries to recover. The fix is to retry politely: wait longer after each failure (backoff), spread the retries randomly in time (jitter), and cap how many retries you send at all (budgets).

Think of a crowd outside a shop that briefly closed its doors. If everyone pushes on the door every second, the staff can never get it open. If each person waits a random, growing amount of time before trying again, the door opens and people trickle in.

How it works

Step 1: Every remote call needs a timeout

Without a timeout, a call can wait forever: a dead connection may not be noticed for minutes. Every network call, database query and lock acquisition should have one. A useful way to pick it: decide what fraction of calls you are willing to cut off by mistake, say 0.1 percent, and set the timeout near that latency percentile of the dependency (here p99.9), plus a margin.

Timeouts must also fit inside each other. If a user request has a 2-second budget and calls three services in sequence, they cannot each have a 2-second timeout. Pass a deadline along with the request so every layer knows how much time is left and gives up when there is none.

Step 2: Retries, and how they turn into storms

Many failures are transient: a dropped connection, a node restarting, a brief overload. Retrying once or twice converts them into successes the user never sees. Retry only what is safe to repeat: reads, and writes made idempotent with a key ( Idempotency and safe retries ). A timeout does not mean the operation did not happen.

The danger is amplification. If a struggling dependency fails, every client retries, and the dependency now receives the original load plus all the retries. If retries are immediate, the load never drops, and the service cannot recover even after the original cause is gone. Worse, retries multiply across layers: if five layers each try three times, the bottom one can receive 3^5 = 243 times the original request volume. Retry at one layer, usually the one closest to the user or the one that knows the operation best.

Step 3: Exponential backoff and jitter

Synchronized retries versus jittered retries Two charts with the same number of failed clients. Top: with exponential backoff but no jitter, all clients retry at the same moments, 1, 2, 4, 8 and 16 ticks later, creating tall red spikes far above the capacity line. Bottom: with full jitter, the same retries are spread randomly across time, forming a low, smooth band that stays under capacity. exponential backoff, no jitter backoff + full jitter Figure 1. Backoff alone keeps clients in lockstep: every retry wave lands at the same instant and overshoots capacity (green line). Full jitter spreads the same retries out so the load stays under capacity and the service recovers.

Exponential backoffWaiting longer after each failed attempt, typically doubling the delay each time up to a cap. doubles the wait after each failure: 100 ms, 200 ms, 400 ms, 800 ms, up to a cap. It gives a struggling dependency room to recover. But on its own it is not enough: if a thousand clients failed at the same moment, they all wait exactly 100 ms and all retry at the same moment again. The retries arrive as synchronized spikes.

JitterRandomness added to retry delays so that clients that failed together do not retry together. breaks the lockstep. With full jitter, each client waits a random time between zero and the backoff value: sleep = random(0, min(cap, base * 2^attempt)). The same number of retries now arrives spread over the whole interval instead of in one burst.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. With retry immediately, the bars stay far above capacity. The overloaded server serves less than it could, and every client sends about 14 requests before it gets through.
  2. Switch to exponential backoff. Far fewer requests are sent, but they arrive in synchronized red spikes. Each wave overshoots capacity, so after 60 ticks most clients are still failing.
  3. Switch to backoff + full jitter. The waves dissolve into a band near capacity, everyone succeeds, and the total is about a third of the immediate-retry traffic. Then lower the capacity and see which policy degrades most gracefully.

Going deeper

Retry budgets

Backoff limits how fast one client retries; a retry budgetA cap on retries as a fraction of normal traffic or as tokens in a bucket, so retries cannot multiply load when a dependency is failing. limits how much retry traffic a whole client sends. A common form is a token bucket: each retry spends a token, tokens refill slowly, and when the bucket is empty, the client fails fast instead of retrying. Another is a ratio: retries may not exceed, say, 10 percent of requests. When a dependency is healthy, almost no retries happen and the budget is never touched. When it is failing, the budget caps the extra load.

Don't forget the server side

Clients behaving well is only half of it. Servers should reject work they cannot finish before its deadline, shed load early ( Load shedding and graceful degradation ), and return clear "try later" responses such as 429 or 503 with a Retry-After header. When a dependency is clearly down, a circuit breaker ( Circuit breakers and bulkheads ) stops calls entirely instead of retrying them.

Trade-offs

OptionWins whenLoses when
No retries Non-idempotent operations; callers that will retry themselves Transient glitches become user-visible errors
Immediate retries Never, except a single quick retry for a dropped connection Retry storms that keep a struggling dependency down
Exponential backoff Giving dependencies time to recover Clients stay synchronized without jitter
Backoff + full jitter + budget The default for almost all remote calls Slightly longer average wait for an individual retry

Common mistakes

  • Calling remote services with no timeout, or with library defaults measured in minutes.
  • Setting each layer's timeout independently so inner calls outlive the outer request.
  • Retrying at every layer of the stack, multiplying load exponentially.
  • Retrying non-idempotent operations, such as charges, without an idempotency key.
  • Backoff without jitter, so retries still arrive in synchronized waves.

In the wild

Amazon

Timeouts, retries and backoff with jitter (opens in a new tab)

This Amazon Builders' Library article describes how Amazon picks timeouts from the downstream service's latency: choose an acceptable rate of false timeouts, such as 0.1 percent, and use the matching percentile (p99.9). It warns that retries multiply when every layer retries independently, increasing load on a database by 243 times in its five-layer example, so the practice is to retry at a single point in the stack and to limit retries locally with a token bucket. It recommends exponential backoff with jitter to spread retries in time, and stresses that APIs with side effects are not safe to retry unless they are idempotent.

Check yourself

  1. A dependency becomes very slow but does not fail. Without timeouts, what happens to the caller?

    Show answer

    B. Its threads or connections fill up waiting, and it stops serving its own requests Slow dependencies exhaust callers' resources; timeouts bound how long each call can hold them.

  2. Five layers each retry a failing call up to 3 times. How much load can reach the bottom layer?

    Show answer

    C. 243x Retries multiply per layer: 3 to the 5th power is 243. Retry at one layer only.

  3. With exponential backoff but no jitter, why do retries still overload a recovering service?

    Show answer

    B. Clients that failed together all wait the same time and retry together Identical delays keep clients synchronized; jitter randomizes them.

  4. What does a retry budget protect against?

    Show answer

    B. Retries multiplying total load when a dependency is failing Capping retries as tokens or a ratio keeps extra load bounded during outages.

Recap

  • Every remote call needs a timeout, chosen from the dependency's latency and nested within a deadline.
  • Retry only safe (idempotent) operations, and only at one layer of the stack.
  • Immediate retries create storms that keep a struggling service down.
  • Exponential backoff spaces retries out; full jitter desynchronizes clients.
  • Retry budgets cap total retry load; servers should shed load and say when to retry.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up