System Design HLD-16.05
DDoS protection and abuse prevention
Volumetric versus application-layer attacks, absorbing floods at the edge, WAFs, bot detection, challenges and rate limits, and shrinking the attack surface.
The big idea
A distributed denial of serviceDistributed denial of service: many machines sending traffic at once to exhaust a target's network, servers or application so real users cannot get through. (DDoS) attack tries to make a service unavailable by sending it more work than it can handle, from many machines at once, usually a botnet of hijacked computers, routers and cameras. Because the traffic comes from thousands of addresses, blocking "the attacker" is not an option, and because much of it can look like normal requests, telling it apart from real users is hard.
Defending against it is mostly a design problem, not a firewall rule. You absorb floods with capacity you do not own, filter as far from your servers as possible, make expensive operations cheap or gated, and accept that some attacks will only be degraded, not stopped. The same toolkit also handles everyday abuse: scrapers, credential stuffing, fake sign-ups and spam.
Think of a shop on a busy street. A crowd blocking the entrance is a volumetric attack: you need a wider street. A crowd of people each asking the one expert to explain the most complicated product is an application attack: you need a queue, a receptionist and a leaflet that answers the common questions.
How it works
Step 1: Two kinds of attack
- Volumetric attacks (network and transport layer) flood bandwidth or connection tables: UDP floods, SYN floods, and reflection attacks that bounce small requests off open servers to send large replies to the victim. They are measured in bits or packets per second and can exceed any single data center's capacity.
- Application-layer attacks (layer 7) send real-looking HTTP requests that are expensive to serve: searches, logins, uncached pages, large exports. A modest number of requests per second can overwhelm the database behind them.
Step 2: Defense in layers
- Absorb at the edge. CDNs and DDoS protection services run huge anycast networks ( Global traffic: GeoDNS, anycast and GSLB ), so a flood is spread across hundreds of locations instead of hitting one. Your origin should accept traffic only from the edge, so attackers cannot bypass it.
- Filter at the edge. A web application firewallWeb application firewall: a filter in front of an application that inspects HTTP requests and blocks those matching attack patterns or custom rules. blocks known attack patterns and lets you add rules quickly during an incident: block a path, a country, a header pattern.
- Detect bots. Signals such as request timing, browser fingerprints and reputation separate scripts from people; suspicious clients get a challenge (a CAPTCHA or an invisible proof-of-work check) instead of a hard block.
- Rate limit per IP, per account and per API key ( Rate limiting algorithms ), with tighter limits on expensive endpoints like login and search.
Step 3: Per-IP limits are not enough
A botnet spreads its traffic so each address stays under any sensible per-IP limit:
import random
rnd = random.Random(5)
LIMIT = 100 # per-IP limit: requests per minute
SEARCH_COST_MS = 40 # CPU per uncached search request
CPU_BUDGET_MS = 60_000 * 8 # 8 cores for one minute
def minute(attackers, per_attacker, real_users=2_000, per_user=3):
"""One minute of traffic to an expensive /search endpoint."""
counts = {}
traffic = [(f"bot{i}", per_attacker) for i in range(attackers)] + \
[(f"user{i}", per_user) for i in range(real_users)]
served = blocked = 0
for ip, n in traffic:
ok = min(n, LIMIT) # the per-IP rate limiter
served += ok
blocked += n - ok
cpu = served * SEARCH_COST_MS
return served, blocked, cpu / CPU_BUDGET_MS
for label, attackers, per in [("one noisy IP", 1, 50_000),
("botnet of 5,000 IPs", 5_000, 90),
("same botnet, search cached", 5_000, 90)]:
served, blocked, load = minute(attackers, per)
if "cached" in label:
load *= 0.01 # 99% of the bots' repeated searches answered from cache
verdict = "overloaded" if load > 1 else "fine"
print(f"{label:28} served {served:>7,} blocked {blocked:>7,} CPU load {load:5.0%} {verdict}")Output
one noisy IP served 6,100 blocked 49,900 CPU load 51% fine
botnet of 5,000 IPs served 456,000 blocked 0 CPU load 3800% overloaded
same botnet, search cached served 456,000 blocked 0 CPU load 38% fine The per-IP limit stops one noisy client completely, but a botnet of 5,000 addresses sending 90 requests a minute each passes straight through and pushes CPU to 38 times capacity. What saves the service is making the expensive work cheap: once the repeated searches are answered from cache, the same attack costs 38 percent of capacity. Shrinking the cost of each request often matters more than catching every bot.
Step 4: Shrink the attack surface
- Cache public pages and API responses at the edge, so floods of identical requests never reach you.
- Require authentication for expensive operations, and limit per account as well as per IP.
- Put hard limits on work per request: page sizes, query complexity ( GraphQL ), upload sizes, timeouts.
- Shed lower-priority load first when overloaded ( Load shedding and graceful degradation ).
Going deeper
Protocol-level attacks
Some of the largest attacks abuse a protocol feature rather than raw bandwidth. They are fixed by patching servers and proxies, which is one more reason to terminate traffic at a well-maintained edge rather than exposing your own servers directly.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Managed edge DDoS protection | Any public service: absorbs floods far beyond your own capacity | Dependence on a provider, and cost at very high traffic |
| Strict rate limits and challenges | Login, sign-up, search and other expensive or abusable endpoints | Friction for real users, and false positives behind shared IPs |
| Caching and gating expensive work | Making attacks cheap to absorb, whoever sends them | Freshness, and design effort on endpoints that cannot be cached |
Common mistakes
- Leaving the origin's IP address reachable directly, so attackers bypass the edge.
- Relying only on per-IP limits against distributed attacks.
- Expensive endpoints, such as unbounded search or export, available without login or limits.
- No incident runbook: who turns on stricter rules, and how, at 3 a.m.
In the wild
Cloudflare
HTTP/2 Rapid Reset: deconstructing the record-breaking attack (opens in a new tab)Lucas Pardue's October 2023 post explains an application-layer attack that peaked just above 201 million requests per second, generated by a botnet of only about 20,000 machines. It abused HTTP/2: a client can cancel a stream and immediately open another, so attackers opened and reset streams as fast as possible, making servers start work they never finished. The flaw was assigned CVE-2023-44487. Cloudflare mitigated it by monitoring client-sent reset frames, closing connections that abused them, and extending its IP jail so that offending addresses were blocked from using HTTP/2 across its whole network for a time.
Check yourself
-
What distinguishes an application-layer attack from a volumetric one?
Show answer
B. It sends realistic but expensive requests rather than raw floods of traffic Layer 7 attacks exhaust the application, such as the database behind search, with legitimate-looking requests.
-
Why are per-IP rate limits weak against a botnet?
Show answer
B. Each of thousands of addresses stays under the limit Spreading traffic across many addresses keeps each one under any reasonable per-IP threshold.
-
Why must the origin accept traffic only from the edge network?
Show answer
B. Otherwise attackers can bypass the edge's protection by hitting the origin directly Protection only works if all traffic passes through it.
-
Which change most reduces the impact of a flood of identical search requests?
Show answer
B. Caching search results Answering repeats from cache makes each attack request nearly free.
Recap
- DDoS attacks overload a service from many machines; blocking one attacker is not an option.
- Volumetric floods are absorbed by large anycast edge networks.
- Application attacks are fought with WAF rules, bot detection, challenges and rate limits.
- Make expensive work cheap or gated: caching, authentication and per-request limits.
- Hide the origin behind the edge and keep an incident runbook ready.