Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Distributed Systems Theory

System Design HLD-08.01

Why distributed systems are hard

Partial failure, unreliable networks and clocks, the eight fallacies of distributed computing, and the two generals problem.

Intermediate 10 min Last reviewed

The big idea

On one computer, a function call either returns or the whole program crashes. Things fail together, which makes them simple to reason about. A distributed system is different: it is many computers cooperating over a network, and its parts fail independently. One server is down while the others run; a message is lost while the next one arrives; a disk is slow while the network is fine. This is partial failureWhen some components of a system fail while others keep working, so the system as a whole is neither fully up nor fully down., and it is the root of almost everything hard in this field.

The cruelest consequence: when you send a request and hear nothing back, you cannot know why. Maybe the request was lost. Maybe the server crashed before doing the work, or after doing it. Maybe it is just slow, and the answer will arrive in a second. All of these look identical from the outside, and your design has to be correct in every one of them.

Think of texting a friend "meet at 7?" and getting no reply. Did they not see it, see it and forget, reply into a dead zone, or are they typing right now? You cannot tell, and you have to decide whether to go anyway.

How it works

Step 1: The eight fallacies

In the 1990s, engineers at Sun Microsystems listed the false assumptions newcomers make about networks. Every one of them has caused outages:

  1. The network is reliable.
  2. Latency is zero.
  3. Bandwidth is infinite.
  4. The network is secure.
  5. Topology does not change.
  6. There is one administrator.
  7. Transport cost is zero.
  8. The network is homogeneous.

Designs that quietly rely on any of these break in production. Retries, timeouts, idempotency, encryption and capacity planning are all responses to one fallacy or another.

Step 2: Unreliable networks and timeouts

The only tool for detecting failure is a timeoutA limit on how long to wait for a response before treating the request as failed. It cannot distinguish a dead node from a slow one.: wait some time, then assume the worst. A short timeout declares healthy-but-slow nodes dead, triggering needless failovers and retries that add load; a long timeout leaves users waiting. No value is correct for all situations, which is why failure detection is a whole topic ( Membership, gossip and failure detection ). And a timeout never tells you whether the work was done, which is why operations must be idempotent ( Idempotency and safe retries ).

Step 3: Unreliable clocks

Each machine has its own clock, and they drift apart by milliseconds or more, jump when synchronized, and can even go backwards. So "this event has an earlier timestamp" does not reliably mean "this event happened first" across machines. Ordering events needs logical clocks instead ( Time and clocks ).

Step 4: The two generals problem

Two armies on opposite hills must attack at the same time to win, and can communicate only by messengers who cross a valley where they may be captured:

The two generals problem General A proposes attacking at dawn. General B agrees but cannot know whether A received the agreement. A confirms, but cannot know whether B received the confirmation. Each acknowledgement needs another acknowledgement, forever. General A Valley messengers may be caught General B attack at dawn? agreed (did A get this?) got your agreement (did B get this?) got your confirmation (did A get this?) ... the last messenger is never confirmed Figure 1. Every acknowledgement needs its own acknowledgement. Whoever sends the last message can never know it arrived, so certain agreement over a lossy channel is impossible.

No finite number of messages can make both generals certain, because whoever sends the last message never knows if it arrived. It is a proof that over an unreliable network you cannot get guaranteed agreement, only probable or eventual agreement. Real protocols accept this and design around it, with retries, idempotency and majority voting.

Try it

Which fallacy does each incident come from?

Show answers
  • A payment was charged twice because the client retried after a lost response: The network is reliable. The design assumed a response always arrives; it needed idempotency.
  • A page makes 40 sequential service calls and takes two seconds: Latency is zero. Each hop costs real time; chatty designs add it up.
  • After a server was replaced, clients kept connecting to its old IP: Topology does not change. Addresses and routes change; use service discovery and DNS with sane TTLs.
  • Internal service traffic was read by an attacker who got onto the network: The network is secure. Encrypt internal traffic too, for example with mutual TLS.

Going deeper

Byzantine versus crash failures

Most systems assume nodes fail by crashing or going silent, not by lying. Tolerating nodes that send wrong or malicious messages, Byzantine faults, is much more expensive and is mainly needed in settings like blockchains or safety-critical systems where nodes are not trusted. Everything in this module assumes crash failures.

Trade-offs

OptionWins whenLoses when
Short timeouts Fast detection of truly dead nodes and quick retries False alarms, needless failovers and retry storms on slow days
Long timeouts Tolerance of temporary slowness Users wait longer when something really is down
One machine Simplicity: things fail together Scale and surviving that machine's failure

Common mistakes

  • Treating a timeout as proof the operation did not happen.
  • Comparing timestamps from different machines to decide which event came first.
  • Assuming a call to another service is as reliable and fast as a local function call.
  • Designing for the happy path and adding failure handling "later".

Check yourself

  1. What is partial failure?

    Show answer

    B. Some components fail while others keep working Parts of a distributed system fail independently, leaving it neither fully up nor down.

  2. A request times out. What do you know?

    Show answer

    C. Nothing certain: it may have been lost, done, or still running A timeout cannot distinguish lost, slow and completed-but-unacknowledged.

  3. What does the two generals problem show?

    Show answer

    B. Guaranteed agreement over a lossy channel is impossible with finite messages The sender of the last message can never be sure it arrived.

  4. Why not order events across machines by wall-clock timestamps?

    Show answer

    B. Clocks drift and jump, so an earlier timestamp may belong to a later event Physical clocks disagree across machines; logical clocks give a reliable order.

Recap

  • Distributed systems fail partially: parts break independently.
  • A missing response cannot be told apart from a slow or completed one.
  • The eight fallacies are the assumptions that break real systems.
  • Clocks drift, so timestamps cannot reliably order events across machines.
  • The two generals problem: certain agreement over lossy links is impossible; design for it.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy