System Design HLD-09.07
Backpressure and flow control
When consumers are slower than producers: bounded queues, pushing back, load shedding, pull-based consumption and autoscaling on lag keep pipelines stable.
The big idea
A queue absorbs a temporary burst. It cannot absorb a permanent mismatch. If a producer sends 100 messages a second and the consumer handles 60, the buffer between them grows by 40 every second, forever. Memory fills, the process crashes, and before that every message waits longer and longer, so results arrive minutes late and are useless by the time they appear.
Something has to give, and you get to choose what. BackpressureA mechanism by which a slow downstream stage signals upstream stages to slow down, so work is not produced faster than it can be processed. pushes the slowness back to the producer: "stop, I am full". Load shedding drops or rejects the excess. Scaling adds consumer capacity. What you must not do is pretend the mismatch is not there by making the buffer unbounded.
Think of a motorway on-ramp with traffic lights. When the motorway is congested, the lights hold cars on the ramp, so the motorway keeps flowing instead of jamming completely. The queue moves upstream to where it is cheap and visible.
How it works
Step 1: Unbounded buffers hide the problem
The tempting fix for a slow consumer is a bigger buffer. That converts a throughput problem into a latency problem and then a memory problem:
PRODUCE, CONSUME, SECONDS = 100, 60, 60 # msgs/s offered, msgs/s the consumer can handle
def run(policy, bound=200):
buf = accepted = shed = 0
for _ in range(SECONDS):
offered = PRODUCE
if policy == "block": # backpressure: producer may only send
offered = min(PRODUCE, bound - buf + CONSUME) # what fits after this second's drain
room = float("inf") if policy == "unbounded" else bound - buf + CONSUME
take = min(offered, room)
shed += offered - take if policy == "shed" else 0
accepted += take
buf = max(0, buf + take - CONSUME)
wait = buf / CONSUME # seconds a new message waits in the buffer
held = PRODUCE * SECONDS - accepted - shed # work the producer had to hold back
return f"{policy:<9} buffer={buf:>5} wait={wait:5.1f}s accepted={accepted} shed={shed} held={held}"
for policy in ("unbounded", "block", "shed"):
print(run(policy))Output
unbounded buffer= 2400 wait= 40.0s accepted=6000 shed=0 held=0
block buffer= 200 wait= 3.3s accepted=3800 shed=0 held=2200
shed buffer= 200 wait= 3.3s accepted=3800 shed=2200 held=0 After one minute the unbounded buffer holds 2,400 messages, and a new message waits 40 seconds, growing by about 0.7 seconds every second. Nothing has failed yet, which is exactly why it is dangerous: dashboards look green until the process runs out of memory. Both bounded versions keep the wait near 3 seconds; the difference is where the excess went. With blocking, the producer held back 2,200 messages, slowing to the consumer's pace. With shedding, 2,200 messages were dropped or rejected.
Little's law gives the arithmetic: average wait = items in the system / throughput. 2,400 queued messages at 60 per second is 40 seconds.
Step 2: Bounded queues
Every buffer in a pipeline should have a limit: in-memory channels, thread pool queues, broker queues, connection pools. When the limit is hit, the system must make an explicit decision rather than silently growing. The size should come from latency, not memory: if a message older than 5 seconds is useless and the consumer does 60 per second, a buffer of about 300 is the most that can be useful.
Step 3: Push back on the producer
When the buffer is full, the producer has to wait. Inside a process this is a blocking put on a bounded queue. Across processes it is a signal: TCP does it automatically with its receive window, HTTP/2 and gRPC use per-stream flow-control windows, and an HTTP service can answer 429 Too Many Requests or 503 with a Retry-After header so well-behaved clients slow down.
Pushing back only works if the producer can wait. A batch job, a database change stream or an upstream service with its own queue can. A user tapping a button usually cannot wait 40 seconds, which leads to the next option.
Step 4: Shed load
Load sheddingDeliberately rejecting or dropping some work when a system is overloaded, so the work it does accept is completed in time. rejects work you cannot do in time, as early and as cheaply as possible, so the work you accept is still served well. Shed by priority where you can: drop analytics events before checkout events, and reject new requests before abandoning ones already half done. A fast, clear rejection is kinder to users and to retrying clients than a slow timeout ( Load shedding and graceful degradation ).
Step 5: Pull instead of push
In a push design, the producer decides the rate and the consumer must keep up. In a pull-basedA design where consumers request work when they have capacity, so the rate of delivery is set by the consumer rather than the producer. design, the consumer asks for more only when it has capacity, so it can never be overwhelmed: backpressure is built in. Kafka consumers poll for batches. The Reactive Streams specification, implemented by libraries such as Project Reactor, RxJava and Akka Streams, makes demand explicit: a subscriber calls request(n) and the publisher may send at most n more items.
In a pull-based log, the pressure becomes consumer lag ( Kafka internals: the distributed log ): the broker retains messages durably on disk, and lag grows instead of memory.
Step 6: Autoscale consumers on lag
If the mismatch lasts, add capacity. Scale the consumer pool on the signal that matters: queue depth, consumer lag, or the age of the oldest message, rather than CPU, which may look fine while lag climbs because consumers are waiting on a slow database. Tools such as KEDA on Kubernetes scale deployments on queue length and Kafka lag. Two limits remain: a consumer group cannot usefully exceed the partition count, and scaling consumers does not help when the real bottleneck is a shared downstream database, which more consumers will only overload further.
Going deeper
Backpressure must propagate end to end
A pipeline is only as stable as its weakest unbounded link. If stage C pushes back on stage B, but B buffers without limit, the pressure stops at B and B runs out of memory instead. Every stage must either propagate the signal upstream or shed at its own boundary.
Timeouts and abandoned work
Under overload, clients time out and retry while the server is still processing their original request. The server does the work, nobody reads the answer, and the retry adds more load. Pass deadlines along with requests and drop queued work whose deadline has already passed.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Unbounded buffer | Never: it only delays the failure | Memory, latency and eventually the whole process |
| Bounded queue + blocking (backpressure) | Producers that can wait: batch jobs, change streams, internal services | Upstream latency; can spread a slowdown across the whole system |
| Load shedding | Interactive traffic where a fast 'try later' beats a slow answer | The shed work, unless clients retry later |
| Pull-based consumption | Consumers that must never be overwhelmed; durable logs | Lag grows instead, which still needs monitoring and capacity |
| Autoscale on lag | Sustained load changes with a scalable bottleneck | Slow to react to sudden spikes; capped by partitions and downstream limits |
Common mistakes
- Using unbounded queues or channels "to be safe".
- Scaling consumers on CPU while the real signal, lag or message age, is climbing.
- Adding consumers when the bottleneck is a shared database, making it worse.
- Shedding late, after expensive work has already been done for requests that will be dropped.
- Applying backpressure in one stage while an earlier stage buffers without limit.
In the wild
Backpressure is built into the protocols under you: TCP's receive window and the flow-control windows in HTTP/2 and gRPC stop a fast sender from overrunning a slow receiver. Kafka consumers pull at their own pace, and Kubernetes clusters commonly scale consumers on lag or queue depth with KEDA. Reactive libraries on the JVM, such as Project Reactor and Akka Streams, implement the Reactive Streams demand protocol.
Check yourself
-
A consumer handles 60 msg/s and the producer sends 100 msg/s. You make the queue unbounded. What happens?
Show answer
B. The backlog grows 40 msg/s forever: latency climbs, then memory runs out A buffer cannot fix a permanent rate mismatch; it only turns it into growing delay and memory use.
-
A queue holds 3,000 messages and the consumers process 100 per second. Roughly how long does a new message wait?
Show answer
B. 30 s Little's law: wait = queue length / throughput = 3,000 / 100 = 30 seconds.
-
Which situation suits load shedding better than blocking the producer?
Show answer
B. Interactive API requests during an overload Users cannot wait indefinitely; a fast rejection lets them retry and protects the requests you do accept.
-
Lag keeps rising, CPU on consumers is at 20 percent, and every consumer waits on the same slow database. Should you add consumers?
Show answer
B. No: the bottleneck is the database, and more consumers will overload it further Scaling the wrong tier moves no extra work; fix or scale the actual bottleneck.
Recap
- A queue absorbs bursts but not a permanent rate mismatch; unbounded buffers only delay the failure.
- Bound every buffer, sized by how long work stays useful.
- When full, push back on producers that can wait, or shed load early for those that cannot.
- Pull-based consumers set their own pace, turning overload into lag rather than memory growth.
- Autoscale consumers on lag or message age, but only when they are the real bottleneck.
Next up
- Crack the system design round learning path