Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Reliability and Resilience

System Design HLD-12.02

Circuit breakers and bulkheads

Circuit breakers fail fast when a dependency is broken (closed, open, half-open), fallbacks keep users served, and bulkheads isolate resources.

Intermediate 12 min Last reviewed

The big idea

Timeouts stop one call from waiting forever. But if the recommendations service is down, every product page still spends a full timeout, say 2 seconds, discovering that again, thousands of times a minute. Users wait, threads pile up, and the broken service is hammered with requests it cannot serve, which makes recovery harder.

A circuit breakerA wrapper around calls to a dependency that tracks failures and, once they pass a threshold, stops sending calls for a while and fails immediately instead. notices the pattern and stops trying. After enough failures it "opens" and fails every call instantly, usually returning a fallback, such as an empty recommendations list. Every so often it lets a single test call through to see whether the dependency has recovered. A bulkheadIsolating resources such as thread pools or connection pools per dependency, so that one failing dependency cannot consume the resources needed by others. is the companion idea: give each dependency its own limited pool of threads or connections, so one slow dependency can exhaust only its own pool, not the whole service.

Think of the circuit breaker in your house. When a faulty appliance draws too much current, the breaker trips and cuts that circuit instantly, protecting the wiring and every other room. You fix the fault, then flip it back. Bulkheads are named after the walls that divide a ship's hull into compartments, so one breach floods one compartment, not the ship.

How it works

Step 1: Three states

Circuit breaker state machine Three states. Closed, glowing as the current state: calls pass through and failures are counted. When the error rate over a window exceeds the threshold with enough volume, it moves to Open: calls fail fast with a fallback. After the sleep window it moves to Half-open: one trial call is allowed. If the trial succeeds, it returns to Closed; if it fails, it goes back to Open. error rate over threshold sleep window passed trial succeeds CLOSED calls pass, failures counted OPEN fail fast, return fallback HALF-OPEN one trial call; if it fails, reopen Figure 1. Closed counts failures. Too many trips it Open, where calls fail fast. After a sleep window it goes Half-open and sends one trial call: success closes the breaker, failure reopens it.
  • Closed (normal): calls go through. The breaker records successes, failures and timeouts over a rolling window, such as the last 10 seconds.
  • Open: once the window has enough requests (so 2 failures out of 3 do not trip it) and the error percentage passes a threshold, the breaker opens. Every call now fails immediately, without touching the network, and the caller runs its fallback.
  • Half-open: after a sleep window, the breaker lets a trial call through. If it succeeds, the dependency seems healthy and the breaker closes. If it fails, it opens again for another sleep window.

Failing fast does two things at once: callers stop wasting time and threads on a broken dependency, and the dependency gets breathing room to recover instead of being flooded.

Step 2: Fallbacks

An open breaker still has to answer the caller. Good fallbacks degrade gracefully:

  • Default or empty value: no recommendations, a generic banner.
  • Cached or stale data: the last known price or profile ( Load shedding and graceful degradation ).
  • Queue for later: accept the order and process the payment when the dependency returns ( Why async? Queues explained ).
  • Fail clearly: for things with no safe substitute, such as payment authorization, a fast, clear error beats a slow timeout.

A fallback should not depend on the network itself, or it can fail in the same outage.

Step 3: Bulkheads

A web server with 200 worker threads calls five dependencies. If one dependency becomes slow, requests that need it hold threads for the full timeout, and soon all 200 threads are waiting on it. Requests that never touch that dependency now cannot get a thread either.

Bulkheads cap each dependency's share: for example, at most 30 concurrent calls to recommendations, via a dedicated thread pool or a semaphore. When the pool is full, further calls fail fast (and can use the fallback) rather than queuing. A slow dependency then degrades only the features that use it. The same idea applies to connection pools per database, separate worker pools per queue, and, at a larger scale, cells ( Cell-based architecture and shuffle sharding ).

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. With the defaults, watch the breaker open shortly after the incident starts. Every sleep window it goes half-open (amber) for a trial call, which fails and reopens it, until the dependency recovers at 70 s and a trial succeeds.
  2. Lower the dependency error rate to 30 percent. The breaker never opens: every failing call still goes through. Is 50 percent the right threshold for a payment dependency? For recommendations?
  3. Change the sleep window. A short window probes more often and recovers faster; a long one protects the dependency longer but keeps users on the fallback after it has healed.

Going deeper

Tuning

Thresholds come from normal behavior: if a dependency normally fails 1 percent of calls, opening at 50 percent is very conservative and 10 percent may be reasonable. A minimum request volume prevents tripping on tiny samples at night. Slow calls can count as failures too, since a dependency taking 5 seconds is as harmful as one returning errors.

Where breakers live

Breakers can live in application libraries (Netflix popularized this with Hystrix; Resilience4j is a common modern Java library) or in the network layer: service meshes and proxies such as Envoy implement outlier detection, ejecting hosts that return too many errors, and connection limits per upstream cluster. Breakers per host are useful when only some instances are bad; breakers per dependency protect against a dependency failing everywhere.

Trade-offs

OptionWins whenLoses when
Timeouts only Simple services with few, reliable dependencies Every call still pays the full timeout during an outage
Circuit breaker + fallback Non-critical dependencies with a sensible degraded answer Tuning effort; a mis-tuned breaker can open on noise or stay open too long
Bulkheads (pools per dependency) Services with several dependencies of varying reliability Idle reserved capacity; pool sizes must be tuned
Breakers in a service mesh Consistent policy across many services and languages Less knowledge of business-level fallbacks

Common mistakes

  • Breakers with no minimum volume, tripping on two failures at 3 a.m.
  • Fallbacks that call another network service and fail in the same outage.
  • One shared thread pool for every dependency, so one slow call path starves the rest.
  • Using a fallback where none is safe, such as pretending a payment succeeded.
  • Not alerting when breakers open, so degraded mode goes unnoticed for hours.

In the wild

Netflix

Hystrix: How it works (opens in a new tab)

Netflix's Hystrix library wraps each call to a dependency in a command that checks a response cache, checks whether the circuit is open, checks whether its thread pool or semaphore is full, and only then runs the call with a timeout, falling back on any failure. Its circuit opens when request volume over a window meets a threshold and the error percentage exceeds a configured limit; after a sleep window, a single request is let through in a half-open state, and its success closes the circuit. Hystrix implements bulkheads with per-dependency thread pools, or semaphores where the threading overhead is not worth it.

Check yourself

  1. What does an open circuit breaker do with new calls?

    Show answer

    B. Fails them immediately, usually with a fallback, without calling the dependency Failing fast saves the caller's resources and gives the dependency room to recover.

  2. Why is there a half-open state?

    Show answer

    B. To test with a trial call whether the dependency has recovered before letting full traffic back A single probe avoids slamming a still-broken dependency with full traffic.

  3. A service has one 200-thread pool and calls five dependencies. One becomes very slow. What does a bulkhead change?

    Show answer

    B. Only that dependency's own pool fills up; calls to the other four keep their threads Isolating resources per dependency contains the damage to the features that need it.

  4. Which is the worst fallback?

    Show answer

    C. Reporting a payment as successful when the payment service is down Fallbacks must never invent results for critical operations; fail clearly instead.

Recap

  • A circuit breaker opens when failures pass a threshold and fails calls fast.
  • After a sleep window, half-open sends a trial call: success closes, failure reopens.
  • Fallbacks (defaults, stale data, queuing, clear errors) keep users served without the dependency.
  • Bulkheads isolate threads and connections per dependency so one failure cannot starve the rest.
  • Tune thresholds from normal error rates, require minimum volume, and alert when breakers open.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up