System Design HLD-13.06
Capacity planning and load testing
Forecasting demand, keeping headroom, and using load, stress and soak tests to find the bottleneck and the knee where latency takes off.
The big idea
Black Friday arrives. Traffic triples, as it does every year, and the site falls over at 9:02. Nobody had measured how much the system could actually handle, and the database that tipped over was never the thing anyone was worried about. Capacity problems are among the most predictable outages there are: demand grows on a curve you can see coming, and the limits of a system can be measured in advance.
Capacity planningForecasting future demand and making sure enough resources are in place ahead of it, with headroom for spikes and failures. forecasts demand and provisions ahead of it. Load testingGenerating synthetic traffic against a system to measure how its latency, errors and resource use change as load increases. measures what the system can really take, and where it breaks. Together they answer: how much traffic can we serve today, which component gives out first, and when do we need more?
Think of a highway engineer. They count cars over years to forecast traffic, and they know a road flows well up to a point, then a few extra cars turn smooth traffic into a jam. They widen the road before the jam, not after.
How it works
Step 1: The knee of the curve
Latency stays almost flat as load rises, because requests rarely wait for each other. As the busiest resource (CPU, a database's connections, a lock, a disk) approaches full, requests start queueing behind each other and latency climbs steeply:
SERVICE_MS = 20 # time one request takes with no queueing
CAPACITY = 1000 # requests per second the fleet can process
# Simple queueing model (M/M/1): latency = service time / (1 - utilization).
print("load utilization avg latency")
for rps in (200, 500, 700, 800, 900, 950, 990):
u = rps / CAPACITY
print(f"{rps:>4}/s {u:>9.0%} {SERVICE_MS / (1 - u):>8.0f} ms")
# Forecast: peak traffic today 600/s, growing 8% a month. Keep peak at or under 70% of capacity
# so there is headroom for spikes and for losing one of three zones.
peak, month = 600, 0
while peak <= 0.7 * CAPACITY:
peak *= 1.08
month += 1
print(f"peak passes 70% of capacity in month {month} (about {peak:.0f}/s): add capacity before then")Output
load utilization avg latency
200/s 20% 25 ms
500/s 50% 40 ms
700/s 70% 67 ms
800/s 80% 100 ms
900/s 90% 200 ms
950/s 95% 400 ms
990/s 99% 2000 ms
peak passes 70% of capacity in month 3 (about 756/s): add capacity before then In this simple queueing model, latency doubles between 50 and 75 percent utilization, doubles again by 90 percent, and explodes near 100 percent. That is why the usable capacity of a system is the knee of the curve, well before the point where it stops responding, and why teams plan to run at around 60 to 70 percent at peak.
Step 2: Forecast and keep headroom
Capacity planning starts from data: peak traffic per day and week, growth rate, seasonality, and known events such as launches or holidays. Convert business forecasts into resources: requests per second, storage per month, database IOPS. Then add headroom for:
- Spikes and traffic you did not forecast.
- Losing a zone or a cell: with three zones, each must absorb a third more ( Redundancy, failover and multi-AZ ).
- Lead time: new hardware, quotas or database migrations can take weeks.
The second half of the code projects 8 percent monthly growth: today's peak of 600 requests per second crosses the 70 percent planning line in about three months, so the capacity work must start now.
Step 3: Kinds of load tests
- Load test: ramp traffic to the expected peak (and a margin above it) and check that latency and errors stay within SLOs.
- Stress test: keep increasing load past the expected peak to find the breaking point and the first bottleneck, and to see how it fails: gracefully with shedding ( Load shedding and graceful degradation ), or by falling over.
- Soak test: hold a realistic load for hours or days to reveal slow problems: memory leaks, connection pool exhaustion, disks filling, caches degrading.
- Spike test: jump suddenly from normal to many times normal, to check autoscaling and queues.
Step 4: Make tests realistic
A load test is only as good as its traffic. Replay or model the real mix of endpoints, payload sizes and user behavior; use production-like data volumes, since a query that is fast on 1,000 rows may crawl on 100 million; include cache misses, not just a warm cache; and generate load from enough machines that the load generator itself is not the bottleneck. Watch every tier during the test to find the first resource to saturate: that is the bottleneck, and fixing it moves the knee.
Going deeper
Testing in production
Staging environments are rarely the same size or shape as production. Many teams load test production itself, carefully: in off-peak hours, with synthetic accounts, ramping slowly with automatic abort on SLO breach. Another technique is to shift real traffic so a subset of servers carries more than its share, measuring the real limit per server with real requests.
Autoscaling is not capacity planning
Autoscaling adds instances as load grows, but it has limits: it reacts with a delay of minutes, it cannot scale databases or third-party quotas, and it may hit cloud capacity or account limits during a regional surge. Plan for the stateful tiers and for the first minutes of a spike, and test that autoscaling actually works under a spike test.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Over-provisioning | Predictable latency and safety during spikes and failures | Paying for idle capacity |
| Running hot (high utilization) | Lower cost per request | Operating near the knee: latency spikes and no room for failures |
| Autoscaling | Stateless tiers with gradual load changes | Reaction time; does not cover databases or sudden spikes |
| Regular load and soak tests | Knowing real limits and catching leaks before customers do | Test infrastructure and realistic data to maintain |
Common mistakes
- Planning by average traffic instead of peak.
- Treating the breaking point as capacity instead of the knee where latency degrades.
- Load testing with a single endpoint, tiny data sets or a fully warm cache.
- Forgetting headroom for losing a zone or a deploy taking servers out of rotation.
- Assuming autoscaling covers databases, third-party limits and sudden spikes.
In the wild
Retailers and ticketing sites run large load tests before predictable peaks such as holiday sales, and streaming and sports platforms prepare for big live events the same way. Open source tools such as k6, Locust, Gatling and JMeter generate load, and cloud providers publish quotas and capacity guidance that capacity plans must account for.
Check yourself
-
A service's latency is flat up to 600 requests per second and doubles by 800. Where is its usable capacity?
Show answer
B. Around the knee, before latency climbs steeply Beyond the knee, small load increases cause large latency increases; operate below it.
-
Which test finds a memory leak that appears after 6 hours?
Show answer
B. Soak test Soak tests hold realistic load for long periods to expose slow resource exhaustion.
-
Why plan to run at about 60 to 70 percent at peak with three zones?
Show answer
B. To stay below the knee and still absorb a lost zone's traffic Headroom covers both latency near saturation and failover capacity.
-
A load test runs against a staging database with 10,000 rows. Production has 200 million. What is the risk?
Show answer
B. Queries that are fast on small data may be slow at production scale, so the test overstates capacity Data volume changes index sizes, cache hit rates and query plans.
Recap
- Capacity problems are predictable: forecast from real peaks, growth and events.
- Latency stays flat until a resource saturates, then climbs steeply: the knee is real capacity.
- Keep headroom for spikes, lost zones and lead time; plan to run around 60 to 70 percent at peak.
- Load, stress, soak and spike tests answer different questions; make traffic and data realistic.
- Find the first bottleneck and fix it to move the knee; autoscaling does not replace planning.