System Design HLD-02.01
Latency vs throughput, and why percentiles matter
Latency, throughput and bandwidth, why p99 beats the average, how fan-out multiplies tail latency, and Little's Law.
The big idea
Two numbers describe how a system performs. LatencyHow long one request takes from start to finish, usually measured in milliseconds. is how long one request takes. ThroughputHow many requests or bytes a system completes per unit of time, such as requests per second. is how many requests the system finishes per second. A highway shows the difference: latency is how long your drive takes, throughput is how many cars pass per hour. Adding lanes raises throughput without making anyone's drive shorter.
The trap is how latency gets summarized. "Average response time: 60 ms" sounds fine, but it can hide the fact that one request in a hundred takes over 200 ms, and a heavy user making a hundred requests a day hits that every day. So engineers describe latency with percentilesThe value below which a given share of measurements fall. p99 is the latency 99 percent of requests beat., and they worry most about the slow end, the tail.
How it works
Step 1: Latency is a distribution
Real request latencies are not bunched around one value. Most requests are fast, and a long tail is slow because of cache misses, garbage collection pauses, retries or a busy disk. Here is a typical shape:
- p50 (the median): half of requests are faster. The typical experience.
- p95, p99: 95 or 99 percent are faster. The experience of your unlucky but not rare users.
- p99.9: matters at large scale. At 10,000 requests per second, 0.1 percent is still 10 slow requests every second.
The mean sits between and describes nobody. One request that takes 30 seconds can drag the mean of a thousand fast ones far up while the median does not move.
Step 2: Fan-out multiplies the tail
Modern requests rarely hit one server. Loading a home feed might call 20 services or 100 shards in parallel and wait for all of them. The response is only as fast as the slowest call:
If each backend is slow 1 percent of the time, the chance that at least one of independent calls is slow is
With 20 calls that is about 18 percent; with 100 calls, about 63 percent. A rare event per server becomes the normal experience for users. This is why large companies obsess over p99 and p99.9 of every internal service.
import numpy as np
rng = np.random.default_rng(7)
# 10,000 request latencies in ms: most are fast, a few are very slow
lat = rng.lognormal(mean=np.log(50), sigma=0.69, size=10_000)
print(f"mean {lat.mean():6.1f} ms")
for p in (50, 95, 99, 99.9):
print(f"p{p:<4} {np.percentile(lat, p):6.1f} ms")
# Fan-out: a page waits for the slowest of 20 backend calls
pages = lat[:10_000 - 10_000 % 20].reshape(-1, 20).max(axis=1)
print(f"page p50 with 20 calls: {np.percentile(pages, 50):6.1f} ms")
print(f"share of pages slower than one call's p99: "
f"{(pages > np.percentile(lat, 99)).mean():.0%}")Output
mean 62.8 ms
p50 49.5 ms
p95 153.9 ms
p99 243.4 ms
p99.9 419.7 ms
page p50 with 20 calls: 176.5 ms
share of pages slower than one call's p99: 18% The simulation agrees with the formula: with 20 calls per page, 18 percent of pages are slower than a single call's p99, and the median page is slower than a single call's p95.
Step 3: Little's Law connects them
How many requests are inside your system at once? Little's LawAverage number of items in a system equals arrival rate times average time each spends there: L = lambda x W. answers it for any stable system:
is requests in flight, the arrival rate and the time each spends inside. At 2,000 requests per second and 50 ms each, about requests are in progress at any moment. That tells you how many threads, connections or memory slots you need. If latency doubles to 100 ms with the same traffic, you need 200, which is why a slow database can exhaust a connection pool upstream.
Try it
Try this:
- Set the slow chance to 1 percent and the backends to 1. One percent of requests are slow, as expected.
- Raise backends to 20, then 100. Watch the curve climb fast at first.
- With 100 backends, how low must a single backend's slow chance be to keep slow pages under 10 percent? (About 0.1 percent: that is why teams set p99.9 targets on internal services.)
Going deeper
Taming the tail
Since you cannot make every server perfect, systems work around slow ones. Hedged requestsSending a second copy of a request to another replica if the first has not answered within, say, the p95 latency, and using whichever answers first. resend a slow call to another replica and take the first answer, at the cost of a little extra load. Timeouts with partial results return the page without the slow piece. Reducing fan-out, by caching or by combining calls, attacks the multiplier itself.
Measure percentiles correctly
You cannot average percentiles: the p99 of two servers is not the mean of their p99s. Collect latency histograms and merge those, then compute percentiles from the merged data. Monitoring tools that store histograms do this for you; dashboards that average p99s across hosts are quietly wrong. This comes back in Logs, metrics and traces .
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Optimize median latency | Typical experience and simple dashboards | Fan-out systems and heavy users, who live in the tail |
| Optimize p99 and p99.9 | Large fan-out and high-volume services, where tails hit most users | Engineering effort and sometimes extra load, such as hedged requests |
| Maximize throughput (batching) | Offline jobs and pipelines where total work per second matters | Interactive requests: batching adds waiting time to each one |
Common mistakes
- Reporting only the average. It hides the tail and moves with a single outlier.
- Averaging percentiles across servers or time windows. Merge histograms instead.
- Testing one service's latency and forgetting the request fans out to many.
- Confusing throughput with speed. A system can handle 100,000 requests per second while each takes two seconds.
Check yourself
-
A service reports p99 = 300 ms. What does that mean?
Show answer
B. 99 percent of requests finish within 300 ms p99 is the value 99 percent of requests beat; 1 percent are slower.
-
Each of 50 parallel backend calls is slow 1 percent of the time. Roughly what share of pages is slow?
Show answer
C. About 40 percent 1 - 0.99^50 is about 0.39, so roughly 40 percent of pages hit at least one slow call.
-
A service gets 500 requests per second and each takes 200 ms. On average, how many are in flight?
Show answer
B. 100 Little's Law: L = 500 x 0.2 = 100 requests in progress.
-
Why is the mean a poor summary of latency?
Show answer
B. A few very slow requests pull it up while describing nobody's experience Latency is skewed; the mean mixes the bulk and the tail, and is moved by outliers.
Recap
- Latency is time per request; throughput is requests per second. They are different goals.
- Describe latency with percentiles: p50 for typical, p99 and p99.9 for the tail.
- With fan-out, the slowest call decides: 1 - (1 - p)^n of pages are slow.
- Little's Law: requests in flight = arrival rate x time in system.
- Merge histograms to compute percentiles; never average them.
Next up
- Crack the system design round learning path