System Design HLD-08.01
Why distributed systems are hard
Partial failure, unreliable networks and clocks, the eight fallacies of distributed computing, and the two generals problem.
The big idea
On one computer, a function call either returns or the whole program crashes. Things fail together, which makes them simple to reason about. A distributed system is different: it is many computers cooperating over a network, and its parts fail independently. One server is down while the others run; a message is lost while the next one arrives; a disk is slow while the network is fine. This is partial failureWhen some components of a system fail while others keep working, so the system as a whole is neither fully up nor fully down., and it is the root of almost everything hard in this field.
The cruelest consequence: when you send a request and hear nothing back, you cannot know why. Maybe the request was lost. Maybe the server crashed before doing the work, or after doing it. Maybe it is just slow, and the answer will arrive in a second. All of these look identical from the outside, and your design has to be correct in every one of them.
Think of texting a friend "meet at 7?" and getting no reply. Did they not see it, see it and forget, reply into a dead zone, or are they typing right now? You cannot tell, and you have to decide whether to go anyway.
How it works
Step 1: The eight fallacies
In the 1990s, engineers at Sun Microsystems listed the false assumptions newcomers make about networks. Every one of them has caused outages:
- The network is reliable.
- Latency is zero.
- Bandwidth is infinite.
- The network is secure.
- Topology does not change.
- There is one administrator.
- Transport cost is zero.
- The network is homogeneous.
Designs that quietly rely on any of these break in production. Retries, timeouts, idempotency, encryption and capacity planning are all responses to one fallacy or another.
Step 2: Unreliable networks and timeouts
The only tool for detecting failure is a timeoutA limit on how long to wait for a response before treating the request as failed. It cannot distinguish a dead node from a slow one.: wait some time, then assume the worst. A short timeout declares healthy-but-slow nodes dead, triggering needless failovers and retries that add load; a long timeout leaves users waiting. No value is correct for all situations, which is why failure detection is a whole topic ( Membership, gossip and failure detection ). And a timeout never tells you whether the work was done, which is why operations must be idempotent ( Idempotency and safe retries ).
Step 3: Unreliable clocks
Each machine has its own clock, and they drift apart by milliseconds or more, jump when synchronized, and can even go backwards. So "this event has an earlier timestamp" does not reliably mean "this event happened first" across machines. Ordering events needs logical clocks instead ( Time and clocks ).
Step 4: The two generals problem
Two armies on opposite hills must attack at the same time to win, and can communicate only by messengers who cross a valley where they may be captured:
No finite number of messages can make both generals certain, because whoever sends the last message never knows if it arrived. It is a proof that over an unreliable network you cannot get guaranteed agreement, only probable or eventual agreement. Real protocols accept this and design around it, with retries, idempotency and majority voting.
Try it
Going deeper
Byzantine versus crash failures
Most systems assume nodes fail by crashing or going silent, not by lying. Tolerating nodes that send wrong or malicious messages, Byzantine faults, is much more expensive and is mainly needed in settings like blockchains or safety-critical systems where nodes are not trusted. Everything in this module assumes crash failures.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Short timeouts | Fast detection of truly dead nodes and quick retries | False alarms, needless failovers and retry storms on slow days |
| Long timeouts | Tolerance of temporary slowness | Users wait longer when something really is down |
| One machine | Simplicity: things fail together | Scale and surviving that machine's failure |
Common mistakes
- Treating a timeout as proof the operation did not happen.
- Comparing timestamps from different machines to decide which event came first.
- Assuming a call to another service is as reliable and fast as a local function call.
- Designing for the happy path and adding failure handling "later".
Check yourself
-
What is partial failure?
Show answer
B. Some components fail while others keep working Parts of a distributed system fail independently, leaving it neither fully up nor down.
-
A request times out. What do you know?
Show answer
C. Nothing certain: it may have been lost, done, or still running A timeout cannot distinguish lost, slow and completed-but-unacknowledged.
-
What does the two generals problem show?
Show answer
B. Guaranteed agreement over a lossy channel is impossible with finite messages The sender of the last message can never be sure it arrived.
-
Why not order events across machines by wall-clock timestamps?
Show answer
B. Clocks drift and jump, so an earlier timestamp may belong to a later event Physical clocks disagree across machines; logical clocks give a reliable order.
Recap
- Distributed systems fail partially: parts break independently.
- A missing response cannot be told apart from a slow or completed one.
- The eight fallacies are the assumptions that break real systems.
- Clocks drift, so timestamps cannot reliably order events across machines.
- The two generals problem: certain agreement over lossy links is impossible; design for it.