Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Reliability and Resilience

System Design HLD-12.05

Redundancy, failover and multi-AZ

Spare capacity so a failure is not an outage: active-passive vs active-active, availability zones, health-checked failover and correlated failures.

Intermediate 10 min Last reviewed

The big idea

Any single machine, disk, power supply, network switch or building will fail eventually. If your service runs on exactly one of something, that thing is a single point of failureA component whose failure alone brings down the whole system., and its outage is your outage. RedundancyRunning more copies of a component than strictly needed, so that the failure of one does not take the service down. means running spares, so a failure becomes a non-event: the spare takes over, and users never notice.

Redundancy only works if the copies do not fail together. Two servers on the same power strip, or two database replicas in the same building, will both go dark when that strip or building does. So redundancy is as much about where you place copies as how many you run. Cloud providers make this practical with availability zones (AZs)One or more data centers in a cloud region with independent power, cooling and networking, close enough to other zones in the region for fast, synchronous communication.: physically separate data centers in one region, close enough for fast replication but far enough apart not to share a fire, a flood or a power feed.

Think of a twin-engine airplane. It can fly on one engine, so a single engine failure is an inconvenience, not a crash. But both engines drink from the same fuel system, so the designers also make sure no single fault can starve both at once.

How it works

Step 1: Active-passive and active-active

  • Active-passive: one copy serves traffic; a standby stays in sync and takes over if the primary fails. Common for databases with a single writer. It is simple, but the standby is idle capacity, and failover takes time: detecting the failure, promoting the standby, redirecting clients.
  • Active-active: every copy serves traffic all the time, and a load balancer spreads requests across them ( What a load balancer does ). Losing a copy just means the others take its share. Typical for stateless application servers. It uses all capacity and fails over instantly, but each copy needs headroom, and stateful systems need care to avoid conflicting writes.

Step 2: Spread across availability zones

A service spread across two availability zones Users reach a load balancer that spans two availability zones. Each zone has two app servers. A primary database in zone A replicates synchronously to a standby in zone B. Step 1: both zones serve traffic. Step 2: zone A fails and flickers red; the load balancer's health checks stop sending traffic there, all traffic glows over to zone B, and the standby database is promoted to primary. Availability zone A Availability zone B sync replication Load balancer app A1 app A2 app B1 app B2 DB primary DB standby promoted on failure ZONE DOWN Figure 1. Normally both zones serve traffic and the database replicates across them. When zone A fails, health checks pull it out of rotation, zone B takes all traffic, and its standby database is promoted. Step through the failure.
  1. Both zones healthy The load balancer spreads traffic over app servers in both zones. The primary database in zone A replicates synchronously to a standby in zone B.
  2. Zone A fails Health checks fail for zone A, so the load balancer sends everything to zone B. The standby database is promoted to primary. Zone B must have enough capacity for all traffic.

Place copies in at least two zones, often three: app servers behind a load balancer that spans zones, and database replicas in different zones. When a zone has a power or network failure, the rest keep running. Most managed databases offer this as a switch, keeping a synchronous standby in another zone and promoting it automatically.

Step 3: Health checks drive failover

Failover is only as good as failure detection ( Membership, gossip and failure detection ). Load balancers send health checksA periodic request, such as GET /health, that a load balancer or orchestrator uses to decide whether an instance should receive traffic. to every instance and stop routing to ones that fail several in a row. Make them meaningful: a check that returns 200 while the database connection is broken keeps a dead instance in rotation. But not too deep either: if every instance's health check calls a shared dependency, that dependency's hiccup marks the whole fleet unhealthy at once.

For databases, failover must also avoid split brain: the old primary must stop accepting writes before the new one starts, which is what leader election and fencing are for ( Quorums and leader election , Distributed locks and fencing tokens ).

Step 4: The math, and its catch

redundancy.pyruns with python3 (standard library)
def parallel(a, n):
    """Availability of n independent copies where any one is enough."""
    return 1 - (1 - a) ** n

def nines(a):
    minutes = (1 - a) * 365 * 24 * 60
    return f"{a * 100:.4f}% (~{minutes:,.0f} min down per year)"

a = 0.99                                            # one server or one zone, alone
for n in (1, 2, 3):
    print(f"{n} independent cop{'y' if n == 1 else 'ies'}: {nines(parallel(a, n))}")

# The catch: copies sharing a failure domain fail together. If 1% of the time the shared
# power, network or deploy takes everything down, that caps the result.
shared = 0.99
print("2 copies in one building:", nines(parallel(a, 2) * shared))

# Capacity: to survive losing one zone, the others must absorb its traffic.
for zones in (2, 3, 4):
    print(f"{zones} zones: run each at most {(zones - 1) / zones:.0%} busy to survive losing one")

Output

1 independent copy: 99.0000% (~5,256 min down per year)
2 independent copies: 99.9900% (~53 min down per year)
3 independent copies: 99.9999% (~1 min down per year)
2 copies in one building: 98.9901% (~5,308 min down per year)
2 zones: run each at most 50% busy to survive losing one
3 zones: run each at most 67% busy to survive losing one
4 zones: run each at most 75% busy to survive losing one

Two independent 99 percent components, either of which is enough, give 99.99 percent: downtime drops from days a year to under an hour. But the formula assumes independent failures. Put both copies in one building that itself fails 1 percent of the time and you are back below 99 percent. The last lines show the other cost: with two zones, each must be able to carry all traffic alone, so it runs at most half busy. With three zones, each runs up to about two-thirds busy, which is one reason three zones are a popular choice.

Going deeper

Correlated failures hide everywhere

Not just buildings. A bad config or deploy pushed everywhere at once, an expired certificate, a shared DNS provider, a dependency that every zone calls, a bug triggered by a leap second: each can take down every redundant copy at the same moment. Defenses include staged rollouts zone by zone, avoiding shared dependencies on the critical path, and cells ( Cell-based architecture and shuffle sharding ).

Static stability

A design is statically stable if it keeps working through a zone failure without having to do anything: no new instances launched, no config pushed, no control plane called. Provisioning enough capacity in each zone ahead of time, rather than relying on autoscaling during the incident, means the failure recovery does not depend on systems that may be impaired by the same event.

Trade-offs

OptionWins whenLoses when
Single instance Development, internal tools where downtime is acceptable Any failure is an outage
Active-passive Single-writer databases and stateful systems Idle standby capacity; failover takes seconds to minutes
Active-active Stateless services behind load balancers Each copy needs headroom; stateful active-active needs conflict handling
Multi-AZ (2 or 3 zones) The default for production in the cloud Cross-zone traffic cost and latency; spare capacity per zone

Common mistakes

  • Running redundant copies in the same rack, building or zone.
  • Health checks that return OK while the instance cannot do its job, or that all depend on one shared service.
  • Not provisioning enough capacity in the surviving zones to absorb a failed one.
  • Never testing failover, then discovering during an outage that it does not work.
  • Deploying a change to all zones at once, turning a bad deploy into a correlated failure.

In the wild

Major cloud providers organize regions into multiple availability zones and recommend spreading production workloads across them. Managed databases such as Amazon RDS Multi-AZ, Google Cloud SQL high availability and Azure zone-redundant databases keep a standby in another zone and fail over automatically, while load balancers spread traffic across zones and remove unhealthy targets using health checks.

Check yourself

  1. Two copies of a service are each 99.9 percent available and fail independently. Roughly what is the availability of the pair?

    Show answer

    C. 99.9999 percent Both must fail at once: 0.001 x 0.001 = 0.000001 downtime, so about 99.9999 percent.

  2. Why does putting both copies in one data center undermine redundancy?

    Show answer

    B. They share failure causes such as power and network, so they fail together The availability formula only holds for independent failures.

  3. A service runs in 2 zones at 70 percent utilization each. One zone fails. What happens?

    Show answer

    B. The surviving zone would need 140 percent, so it overloads With two zones, each must run at no more than 50 percent to absorb the other's load.

  4. What is wrong with a health check that always returns 200?

    Show answer

    B. Broken instances stay in rotation because the check does not test real functionality Health checks must reflect whether the instance can actually serve requests.

Recap

  • Redundancy runs spare copies so one failure is not an outage; avoid single points of failure.
  • Active-passive keeps a standby ready; active-active serves from every copy.
  • Spread copies across availability zones so they do not share power, network or buildings.
  • Health checks drive failover; make them meaningful but not dependent on shared services.
  • Redundancy math assumes independence: watch for correlated failures and keep enough headroom in each zone.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy