Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Infrastructure Essentials

System Design HLD-17.02

Autoscaling

Adding capacity as load rises and removing it as load falls: metrics to scale on, fast scale-up and cautious scale-down, cooldowns and warm pools.

Intermediate 10 min Last reviewed

The big idea

Traffic is never flat. It rises in the morning, peaks in the evening, spikes when a newsletter goes out, and falls at night. A fleet sized for the peak wastes money most of the day; a fleet sized for the average falls over at the peak. AutoscalingAutomatically changing the number or size of instances in response to load or another metric. adjusts capacity to follow demand.

The hard part is time. Starting a new instance takes minutes: boot, pull the image, warm caches. Load can rise faster than that. So an autoscaler is a control loop with a delay, and like any such loop it can lag behind, overshoot, or oscillate. Good autoscaling is about choosing the right signal and tuning how eagerly it reacts in each direction.

Think of a supermarket opening checkouts. Seeing a queue form, the manager calls staff from the back; they take a few minutes to arrive. Close checkouts the moment the queue shrinks and the next wave finds too few open.

How it works

Step 1: What to scale on

  • CPU utilization: simple and common; works when CPU is the bottleneck.
  • Request rate or concurrency: better for I/O-bound services where CPU stays low while latency climbs.
  • Queue length or consumer lag: the right signal for workers draining a queue ( Why async? Queues explained ).
  • Custom or predictive: latency, or forecasts from yesterday's curve to scale up before a known peak.

Target utilization is usually well below 100 percent, often around 60 to 70 percent, to leave headroom for the minutes it takes to add capacity ( Capacity planning and load testing ).

Step 2: Fast up, slow down

Autoscaled capacity following load with a startup lag and a cooldown Load rises steeply around minute 20 and falls around minute 42. Capacity follows in steps, a few minutes late on the way up because new instances take time to start, and stays high for several minutes after load falls because of the scale-down cooldown. 01020304050600500100015002000time (minutes)requests per second load capacity (steps) short of capacity while instances start held during cooldown Figure 1. Capacity follows load in steps. On the way up it lags by the instance startup time; on the way down it is held deliberately by the cooldown, so a returning burst does not find the fleet already shrunk.

Scale-up should be fast and generous: being short of capacity drops requests. Scale-down should be slow and cautious, with a cooldownA minimum time between scaling actions in one direction, so the fleet does not shrink and grow repeatedly in response to brief changes., because removing instances you will need again soon costs a full startup delay later. Getting this wrong causes flapping: the fleet shrinks in a lull and is caught short by the next burst, over and over.

autoscale.pyruns with python3 (standard library)
import math

# Requests per second, one value per minute: traffic that comes in bursts every 8 minutes.
load = ([400] * 4 + [1600] * 4) * 8
PER_INSTANCE = 250       # requests per second one instance handles at the 70% target
STARTUP = 3              # minutes before a newly added instance can take traffic

def run(cooldown_down, warm_pool=0):
    ready, pending, last_down = 2, [], -99
    dropped = instance_minutes = 0
    for t, rps in enumerate(load):
        ready += sum(1 for s in pending if s == t)
        pending = [s for s in pending if s > t]
        want = max(2, math.ceil(rps / PER_INSTANCE))
        coming = ready + len(pending)
        if want > coming:                                         # scale up as soon as needed
            warm = min(warm_pool, want - coming)                  # warm instances start instantly
            ready += warm
            pending += [t + STARTUP] * (want - coming - warm)
        elif want < ready and t - last_down >= cooldown_down:     # scale down one at a time
            ready, last_down = ready - 1, t
        capacity = ready * PER_INSTANCE / 0.7                     # most it can take before overload
        dropped += max(0, rps - capacity) * 60
        instance_minutes += ready + warm_pool
    return dropped, instance_minutes

for label, cd, warm in [("cooldown 1 min", 1, 0), ("cooldown 10 min", 10, 0), ("cooldown 1 min + 4 warm", 1, 4)]:
    d, im = run(cd, warm)
    print(f"{label:24} dropped {d:>9,.0f} requests   cost {im:>4} instance-minutes")

Output

cooldown 1 min           dropped   825,429 requests   cost  259 instance-minutes
cooldown 10 min          dropped   159,429 requests   cost  382 instance-minutes
cooldown 1 min + 4 warm  dropped         0 requests   cost  611 instance-minutes

With bursty traffic and a 3-minute startup, scaling down after a 1-minute cooldown is cheapest but drops over 800,000 requests, because each burst finds the fleet shrunk. A 10-minute cooldown drops a fifth as many at about 50 percent more cost. A warm poolA set of pre-started, idle instances that can take traffic immediately, avoiding the startup delay. of four pre-started instances drops nothing, at more than double the cost. There is no free setting; you are buying reliability with idle capacity.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Bursts every 8 minutes, cooldown 1: red keeps poking above blue at the start of every burst. Raise the cooldown and watch the drops fall and the cost rise.
  2. Set startup to 0: even a twitchy autoscaler copes. Startup time is the root of the problem.
  3. Sudden spike with no warm pool: the first minutes of the spike are always dropped. Add a warm pool and they disappear.

Going deeper

Horizontal versus vertical

Horizontal autoscaling changes the number of instances and suits stateless services. Vertical autoscaling changes an instance's CPU and memory, which helps workloads that cannot be split but usually requires a restart. Databases are rarely autoscaled quickly in either direction; plan their capacity ahead ( Capacity planning and load testing ).

Scale the bottleneck

Adding app servers does nothing if the database is saturated; it can make things worse by opening more connections. Autoscale the tier that limits throughput, and protect the tiers that cannot scale with limits and queues.

Trade-offs

OptionWins whenLoses when
Aggressive scale-down Smooth, slowly changing traffic where idle cost matters Bursty traffic: flapping and dropped requests
Long cooldowns Bursty or unpredictable traffic Cost: instances sit idle after peaks
Warm pools or predictive scaling Known spikes and slow-starting services Paying for capacity that is often idle

Common mistakes

  • Scaling on CPU for an I/O-bound service whose latency climbs while CPU stays low.
  • Symmetric up and down policies, causing flapping.
  • Ignoring startup time, so capacity always arrives after the spike has passed.
  • Autoscaling app servers in front of a database that cannot scale with them.

Check yourself

  1. Why should scale-down be slower than scale-up?

    Show answer

    B. Removing capacity you need again soon costs a full startup delay and dropped requests Being short of capacity is worse than briefly paying for idle instances.

  2. Which metric fits a pool of queue workers best?

    Show answer

    B. Queue length or consumer lag Lag directly measures whether workers keep up with incoming messages.

  3. What does a warm pool trade?

    Show answer

    B. Extra idle cost for instant capacity without startup delay Pre-started instances serve immediately, but you pay for them while idle.

  4. The app tier autoscales but the database is saturated. What happens when load rises?

    Show answer

    B. More app servers add connections to an overloaded database, which can make things worse Scaling the wrong tier does not help the bottleneck.

Recap

  • Autoscaling matches capacity to load so you do not pay for the peak all day.
  • Choose the metric that reflects the real bottleneck: CPU, requests, queue lag or custom.
  • Scale up fast and down slowly, with cooldowns, to avoid flapping.
  • Startup time is the core difficulty; warm pools and prediction buy speed with idle cost.
  • Autoscale the bottleneck tier and protect the ones that cannot scale.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy