System Design HLD-17.02
Autoscaling
Adding capacity as load rises and removing it as load falls: metrics to scale on, fast scale-up and cautious scale-down, cooldowns and warm pools.
The big idea
Traffic is never flat. It rises in the morning, peaks in the evening, spikes when a newsletter goes out, and falls at night. A fleet sized for the peak wastes money most of the day; a fleet sized for the average falls over at the peak. AutoscalingAutomatically changing the number or size of instances in response to load or another metric. adjusts capacity to follow demand.
The hard part is time. Starting a new instance takes minutes: boot, pull the image, warm caches. Load can rise faster than that. So an autoscaler is a control loop with a delay, and like any such loop it can lag behind, overshoot, or oscillate. Good autoscaling is about choosing the right signal and tuning how eagerly it reacts in each direction.
Think of a supermarket opening checkouts. Seeing a queue form, the manager calls staff from the back; they take a few minutes to arrive. Close checkouts the moment the queue shrinks and the next wave finds too few open.
How it works
Step 1: What to scale on
- CPU utilization: simple and common; works when CPU is the bottleneck.
- Request rate or concurrency: better for I/O-bound services where CPU stays low while latency climbs.
- Queue length or consumer lag: the right signal for workers draining a queue ( Why async? Queues explained ).
- Custom or predictive: latency, or forecasts from yesterday's curve to scale up before a known peak.
Target utilization is usually well below 100 percent, often around 60 to 70 percent, to leave headroom for the minutes it takes to add capacity ( Capacity planning and load testing ).
Step 2: Fast up, slow down
Scale-up should be fast and generous: being short of capacity drops requests. Scale-down should be slow and cautious, with a cooldownA minimum time between scaling actions in one direction, so the fleet does not shrink and grow repeatedly in response to brief changes., because removing instances you will need again soon costs a full startup delay later. Getting this wrong causes flapping: the fleet shrinks in a lull and is caught short by the next burst, over and over.
import math
# Requests per second, one value per minute: traffic that comes in bursts every 8 minutes.
load = ([400] * 4 + [1600] * 4) * 8
PER_INSTANCE = 250 # requests per second one instance handles at the 70% target
STARTUP = 3 # minutes before a newly added instance can take traffic
def run(cooldown_down, warm_pool=0):
ready, pending, last_down = 2, [], -99
dropped = instance_minutes = 0
for t, rps in enumerate(load):
ready += sum(1 for s in pending if s == t)
pending = [s for s in pending if s > t]
want = max(2, math.ceil(rps / PER_INSTANCE))
coming = ready + len(pending)
if want > coming: # scale up as soon as needed
warm = min(warm_pool, want - coming) # warm instances start instantly
ready += warm
pending += [t + STARTUP] * (want - coming - warm)
elif want < ready and t - last_down >= cooldown_down: # scale down one at a time
ready, last_down = ready - 1, t
capacity = ready * PER_INSTANCE / 0.7 # most it can take before overload
dropped += max(0, rps - capacity) * 60
instance_minutes += ready + warm_pool
return dropped, instance_minutes
for label, cd, warm in [("cooldown 1 min", 1, 0), ("cooldown 10 min", 10, 0), ("cooldown 1 min + 4 warm", 1, 4)]:
d, im = run(cd, warm)
print(f"{label:24} dropped {d:>9,.0f} requests cost {im:>4} instance-minutes")Output
cooldown 1 min dropped 825,429 requests cost 259 instance-minutes
cooldown 10 min dropped 159,429 requests cost 382 instance-minutes
cooldown 1 min + 4 warm dropped 0 requests cost 611 instance-minutes With bursty traffic and a 3-minute startup, scaling down after a 1-minute cooldown is cheapest but drops over 800,000 requests, because each burst finds the fleet shrunk. A 10-minute cooldown drops a fifth as many at about 50 percent more cost. A warm poolA set of pre-started, idle instances that can take traffic immediately, avoiding the startup delay. of four pre-started instances drops nothing, at more than double the cost. There is no free setting; you are buying reliability with idle capacity.
Try it
Try this:
- Bursts every 8 minutes, cooldown 1: red keeps poking above blue at the start of every burst. Raise the cooldown and watch the drops fall and the cost rise.
- Set startup to 0: even a twitchy autoscaler copes. Startup time is the root of the problem.
- Sudden spike with no warm pool: the first minutes of the spike are always dropped. Add a warm pool and they disappear.
Going deeper
Horizontal versus vertical
Horizontal autoscaling changes the number of instances and suits stateless services. Vertical autoscaling changes an instance's CPU and memory, which helps workloads that cannot be split but usually requires a restart. Databases are rarely autoscaled quickly in either direction; plan their capacity ahead ( Capacity planning and load testing ).
Scale the bottleneck
Adding app servers does nothing if the database is saturated; it can make things worse by opening more connections. Autoscale the tier that limits throughput, and protect the tiers that cannot scale with limits and queues.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Aggressive scale-down | Smooth, slowly changing traffic where idle cost matters | Bursty traffic: flapping and dropped requests |
| Long cooldowns | Bursty or unpredictable traffic | Cost: instances sit idle after peaks |
| Warm pools or predictive scaling | Known spikes and slow-starting services | Paying for capacity that is often idle |
Common mistakes
- Scaling on CPU for an I/O-bound service whose latency climbs while CPU stays low.
- Symmetric up and down policies, causing flapping.
- Ignoring startup time, so capacity always arrives after the spike has passed.
- Autoscaling app servers in front of a database that cannot scale with them.
Check yourself
-
Why should scale-down be slower than scale-up?
Show answer
B. Removing capacity you need again soon costs a full startup delay and dropped requests Being short of capacity is worse than briefly paying for idle instances.
-
Which metric fits a pool of queue workers best?
Show answer
B. Queue length or consumer lag Lag directly measures whether workers keep up with incoming messages.
-
What does a warm pool trade?
Show answer
B. Extra idle cost for instant capacity without startup delay Pre-started instances serve immediately, but you pay for them while idle.
-
The app tier autoscales but the database is saturated. What happens when load rises?
Show answer
B. More app servers add connections to an overloaded database, which can make things worse Scaling the wrong tier does not help the bottleneck.
Recap
- Autoscaling matches capacity to load so you do not pay for the peak all day.
- Choose the metric that reflects the real bottleneck: CPU, requests, queue lag or custom.
- Scale up fast and down slowly, with cooldowns, to avoid flapping.
- Startup time is the core difficulty; warm pools and prediction buy speed with idle cost.
- Autoscale the bottleneck tier and protect the ones that cannot scale.