System Design HLD-04.01
What a load balancer does
Spreading requests across servers, health checks that remove failed ones, and making the load balancer itself redundant with failover pairs and virtual IPs.
The big idea
Ten identical servers are useless if every user connects to the first one. A load balancerA component that receives incoming connections or requests and distributes them across a pool of servers, skipping servers that are unhealthy. sits in front of a pool of servers and hands each incoming request to one of them, so the load is shared and no single server is overwhelmed. Need more capacity? Add a server to the pool. Clients never notice, because they only ever talk to the load balancer's address.
Its second job matters just as much: noticing failure. Servers crash, hang and get deployed. The load balancer constantly checks each one and quietly stops sending traffic to any that stop answering. That turns "a server died" from an outage into a non-event, as long as the remaining servers have enough spare capacity.
Think of the host at a busy restaurant. Diners do not pick a waiter; the host seats each party with a waiter who has room, and if a waiter goes home sick, the host simply stops seating people in that section.
How it works
Step 1: Spread the load
The simplest rule is round robin: A, B, C, A, B, C. Smarter algorithms account for servers of different sizes or with different amounts of work in progress; Load balancing algorithms compares them.
Step 2: Health checks
The load balancer probes each server on a schedule, for example GET /healthz every 2 seconds, expecting a 200 within a short timeout. After a few consecutive failures it marks the server unhealthy and removes it from rotation; after a few consecutive successes it adds it back. Requiring several results in a row avoids flapping on a single slow response.
Two kinds of check:
- Liveness: is the process up and answering? Cheap and fast.
- Readiness: can it actually serve? It might check that the server has warmed up and can reach its database. A server that is alive but not ready, for example during startup, should receive no traffic yet.
Detection is never instant. With checks every 2 seconds and 2 failures needed, a dead server can receive traffic for up to about 4 seconds, and those requests fail. Many load balancers also watch real traffic (passive health checksDetecting failing servers by observing errors and timeouts on real requests, rather than only by separate health probes.) and retry a failed idempotent request on another server, hiding most of those errors.
Try it
Try this:
- Kill server B. A few requests turn red and fail, then the load balancer marks B down and all traffic flows to A and C.
- Raise the health check interval to 5 seconds and kill another server. The window of failed requests grows: fast detection matters.
- Revive the servers. They rejoin after passing a check. Then kill all three: no healthy server remains, and every request fails, which is why capacity planning assumes some servers will be down.
Step 3: The load balancer must not be a single point of failure
Every request now passes through the load balancer, so if it dies, everything dies. Load balancers are therefore run redundantly:
- Active-passive with a virtual IP. Two load balancers share a virtual IPAn IP address not tied to one machine; it can move between machines, so clients keep using the same address when the machine behind it changes.. The active one answers on it; the standby monitors the active one's heartbeats and claims the VIP if they stop.
- Active-active. Several load balancers all serve traffic, with DNS or an anycast address ( Global traffic: GeoDNS, anycast and GSLB ) spreading clients across them.
- Managed cloud load balancers. Cloud providers run load balancing as a distributed service spread across zones, so you never manage the pair yourself.
Going deeper
Draining before you remove a server
When you deploy, do not just kill a server: tell the load balancer to stop sending it new requests, let in-flight requests finish (connection draining, often 10 to 60 seconds), then stop it. Rolling deploys repeat this server by server, so capacity never drops much and no request is cut off.
Capacity for failure
If you run 3 servers at 80 percent load each and one dies, the other two need 120 percent: they fall over too, a cascading failure. Plan so the pool survives losing at least one server, or one zone, with headroom: N+1 or N+2 capacity. More in Redundancy, failover and multi-AZ .
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Frequent, strict health checks | Fast detection: few requests go to dead servers | Extra probe load, and false alarms that flap healthy servers in and out |
| Infrequent, lenient health checks | Stability and less probe traffic | Longer windows where users hit a dead server |
| Managed cloud load balancer | Most teams: redundancy, scaling and TLS handled for you | Fine control, some protocols, and cost at very large scale |
Common mistakes
- Running a single load balancer instance in front of a redundant server pool.
- Health checks that only test "process is up" while the server cannot reach its database.
- Health checks that call every dependency, so one slow dependency marks every server unhealthy at once.
- Removing servers without draining, cutting off in-flight requests on every deploy.
Check yourself
-
What are the two main jobs of a load balancer?
Show answer
B. Spreading requests across servers and routing around unhealthy ones It distributes traffic and removes failed servers from rotation using health checks.
-
Checks run every 3 seconds and 3 failures mark a server down. Roughly how long can a dead server keep receiving traffic?
Show answer
B. About 9 seconds Three failed checks, 3 seconds apart, means up to about 9 seconds before it is removed.
-
Why use readiness checks in addition to liveness?
Show answer
B. A process can be running but not yet able to serve, for example during startup Readiness keeps traffic away until the server can actually handle requests.
-
How is a load balancer itself made highly available?
Show answer
B. By running a redundant pair or cluster, for example active-passive sharing a virtual IP A standby takes over the virtual IP, or several balancers share traffic, so one failure does not stop service.
Recap
- A load balancer gives clients one address and spreads their requests over a pool of servers.
- Health checks remove failed servers and add recovered ones, after several results in a row.
- Detection takes time; passive checks and retries hide most of the gap.
- The load balancer must be redundant itself: active-passive with a VIP, active-active, or managed.
- Drain servers before removing them, and keep spare capacity for failures.
Next up
- I know nothing yet learning path
- Crack the system design round learning path