System Design HLD-01.01
Latency numbers every engineer should know
From a nanosecond in CPU cache to 150 ms across an ocean: the orders of magnitude that decide where data should live and when to cache.
The big idea
A computer can fetch a number from its CPU cache in about a nanosecond, a billionth of a second. Fetching the same number from a server on another continent takes about 150 milliseconds. That is a difference of over 100 million times, and almost every design decision you will ever make is about which side of that gap your data sits on.
You do not need exact numbers. Hardware changes and real values vary by a factor of two or more. What you need are the orders of magnitudePowers of ten. Two values differ by an order of magnitude when one is about ten times the other.: memory is about a thousand times faster than an SSD, a round trip inside a data center is about a millisecond, and crossing an ocean is about a hundred milliseconds. With those in your head you can judge a design in seconds.
Picture fetching something from your desk drawer, from the kitchen, from the shop down the road and from another country. Nobody needs a stopwatch to know which trip to avoid. Latency numbers give you the same feel for computers.
How it works
Step 1: The ladder
Here are the numbers that matter most, on a log scaleA scale where each equal step multiplies the value by ten, so values from 1 to 100 million fit on one chart. so they fit on one chart. The right column imagines a slowed-down world where one nanosecond lasts one second.
Step 2: Read the gaps, not the numbers
Three gaps explain most of system design:
- Memory versus disk: about 1,000 times. Reading from RAM takes about 100 ns; an SSD random read about 100 microseconds. This is why databases keep hot data in memory and why caches like Redis exist.
- Local versus network: about 5,000 times. Even a fast round trip inside a data center (about 0.5 ms) is thousands of times slower than reading memory. Every extra service call in a request adds up, which is why chatty designs are slow.
- Near versus far: about 300 times. A round trip across an ocean (about 150 ms) costs hundreds of data center round trips. Physics sets this: light in fiber covers roughly 200 km per millisecond, so no software can fix it. Only moving data closer to users can.
Step 3: Turn numbers into decisions
The ladder becomes a set of rules of thumb:
- Keep hot data in memory. If a value is read on every request, a memory lookup at 100 ns beats a disk read at 100 microseconds by a factor of 1,000. At the extreme, a stock exchange's order matching engine design keeps every order book in memory, because one network call per order would cost more than the matching itself.
- Avoid unnecessary network hops. Ten sequential calls to other services at 1 ms each cost 10 ms before any work is done.
- Batch remote calls. Fetching 100 items in one request costs one round trip; fetching them one by one costs 100.
- Put data near users. A CDN edge 20 ms away instead of an origin 150 ms away saves more than any code optimization can.
- Read sequentially from disk when you can. A disk is much faster at reading one long run of data than many random small pieces, which is why logs and some databases append instead of updating in place.
Try it
Try this:
- Switch to the human scale. A memory read takes under two minutes; a disk seek takes four months. Does that change how you feel about a "small" disk read?
- Count how many data center round trips fit into one cross-continent round trip.
- Pick a page on a site you use. Guess how many network round trips it needs, then multiply. Is that most of the load time?
Going deeper
Bandwidth is a different number
Latency is how long the first byte takes; bandwidthHow many bytes per second a link or device can move once data is flowing. is how many bytes per second flow after that. A satellite link can have high bandwidth and terrible latency. Sending 1 GB across a 1 Gbit per second link takes about 8 seconds no matter how close the server is, and sending a tiny message across an ocean takes 150 ms no matter how fat the pipe is. Large transfers are limited by bandwidth; small, chatty ones by latency.
These numbers drift, the ratios less so
SSDs got much faster in the last decade and networks inside data centers did too. The exact values in the chart will age. The shape will not: caches beat memory, memory beats storage, local beats remote, near beats far. Estimation in Back-of-the-envelope estimation uses the same idea: be right to within a factor of ten, not to the decimal place.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Keep data in memory | Hot data read on many requests: about 1,000 times faster than SSD | Large or cold data: memory costs much more per gigabyte and is lost on restart |
| Keep data on SSD | Durable storage of large data sets with good random reads | Paths that need sub-millisecond reads on every request |
| Copy data near users | Global users and read-mostly content: saves 100 ms or more per round trip | Data that changes constantly or must be strongly consistent everywhere |
Common mistakes
- Memorizing exact values and arguing over them. The point is the ratio between rungs.
- Ignoring sequential calls. Five services called one after another cost five round trips, even if each is "fast".
- Optimizing CPU code while a request makes a cross-region call. The network dominates by orders of magnitude.
- Confusing latency with bandwidth. A faster link does not shorten the speed-of-light delay.
Check yourself
-
Roughly how much slower is an SSD random read than a main memory read?
Show answer
B. About 1,000 times Memory is about 100 ns and an SSD random read about 100 microseconds: a factor of about 1,000.
-
A request makes 20 sequential calls to services in the same data center. About how much time do the round trips alone add?
Show answer
B. About 10 ms Each round trip is around 0.5 ms, so 20 of them add about 10 ms.
-
Users in Asia complain the site is slow; the servers are in Europe. What helps most?
Show answer
B. Serving content from locations close to Asian users Cross-continent round trips cost over 100 ms each and are set by physics; moving data closer is the main fix.
-
Why is batching 100 reads into one request faster than 100 separate requests?
Show answer
B. It pays the network round trip once instead of 100 times Each separate request pays the full round-trip latency; a batch pays it once.
Recap
- CPU cache is about 1 ns, memory about 100 ns, SSD about 100 microseconds.
- A round trip in a data center is about 0.5 ms; across continents about 150 ms.
- The gaps matter more than the values: memory beats disk, local beats remote, near beats far.
- Keep hot data in memory, avoid and batch network hops, and put data near users.
- Latency is time to the first byte; bandwidth is bytes per second after that.
Next up
- I know nothing yet learning path