System Design HLD-13.03
Monitoring and alerting
Watch what users feel: golden signals, RED and USE, alerting on symptoms rather than causes, SLO burn-rate alerts, and avoiding alert fatigue.
The big idea
A new on-call engineer gets 60 alerts in a week: CPU above 80 percent, disk at 75 percent, one pod restarted, a queue briefly backed up. Almost none needed action. By week three they mute the channel, and on week four the alert that mattered, checkout failing for 5 percent of users, is lost in the noise. Too many alerts are as dangerous as too few.
Good monitoring starts from the user: is the service doing what users need, fast enough? Watch a few signals that capture that, and page a human only when users are being hurt, or soon will be, in a way that needs a person to fix. Everything else goes on dashboards for investigation, or into tickets for working hours.
Think of a car dashboard. It shows speed and fuel all the time, but the bright warning lights are reserved for the few things that need you to act now, like low oil pressure. If every light blinked constantly, you would stop looking at any of them.
How it works
Step 1: The signals worth watching
For any user-facing service, the four golden signals cover most of what matters:
- Latency: how long requests take, watched as percentiles (p50, p99), and separately for successes and errors, since fast errors can make latency look good.
- Traffic: demand, such as requests per second.
- Errors: the rate of failed requests, including wrong answers and responses slower than promised.
- Saturation: how full the most constrained resource is: CPU, memory, connections, queue depth.
Two shorthand methods help apply them. RED for services: Rate, Errors, Duration. USE for resources such as disks, CPUs and connection pools: Utilization, Saturation, Errors.
Step 2: Alert on symptoms, not causes
A symptom is what users experience: "checkout errors at 3 percent", "search p99 at 4 seconds". A cause is a reason it might happen: "CPU at 95 percent", "a replica is lagging". Causes are poor alerts: high CPU often hurts nobody, and many real outages have causes no one thought to alert on. Alert on symptoms, and use cause metrics on dashboards to find out why once a symptom fires. Exceptions are causes that will very soon become user pain with certainty, such as a disk that will be full in an hour.
Every page should be actionable, urgent and need human judgment. If the response is always the same mechanical step, automate it instead of paging.
Step 3: SLO burn-rate alerts
An SLO turns "is it bad enough to wake someone" into arithmetic ( SLIs, SLOs, SLAs and error budgets ). With a 99.9 percent success SLO over 30 days, the error budget is 0.1 percent of requests. The burn rateHow fast the error budget is being consumed relative to the rate that would use exactly all of it by the end of the SLO window; a burn rate of 1 is exactly on budget. is how many times faster than "exactly on budget" you are spending it. A common policy: page if the last hour burned 2 percent of the month's budget (a 14.4x burn rate), and open a ticket if the last six hours burned 5 percent (6x).
SLO = 0.999 # 99.9% of requests succeed over 30 days
BUDGET = 1 - SLO # 0.1% of requests may fail
HOURS = 30 * 24
def burn_rate(error_rate):
"""How many times faster than 'exactly on budget' we are spending the error budget."""
return error_rate / BUDGET
# Page when the last hour burned >= 2% of the monthly budget: a 14.4x burn rate.
PAGE = 0.02 * HOURS / 1 # = 14.4
# Open a ticket when the last 6 hours burned >= 5%: a 6x burn rate.
TICKET = 0.05 * HOURS / 6 # = 6.0
print(f"page at burn rate {PAGE:.1f}x, ticket at {TICKET:.1f}x")
for label, err in [("normal (0.05% errors)", 0.0005), ("CPU spike, users fine", 0.0005),
("bad deploy (2% errors)", 0.02), ("slow leak (0.7% errors)", 0.007)]:
b = burn_rate(err)
action = "PAGE" if b >= PAGE else "ticket" if b >= TICKET else "no alert"
hours_left = HOURS / b if b else float("inf")
print(f"{label:<26} burn {b:5.1f}x budget gone in {hours_left:6.0f} h -> {action}")Output
page at burn rate 14.4x, ticket at 6.0x
normal (0.05% errors) burn 0.5x budget gone in 1440 h -> no alert
CPU spike, users fine burn 0.5x budget gone in 1440 h -> no alert
bad deploy (2% errors) burn 20.0x budget gone in 36 h -> PAGE
slow leak (0.7% errors) burn 7.0x budget gone in 103 h -> ticket A CPU spike with users unaffected burns nothing extra and stays quiet. A bad deploy at 2 percent errors burns 20 times too fast and would exhaust the month's budget in a day and a half, so it pages. A slow leak at 0.7 percent is real but not urgent, so it becomes a ticket for working hours. Production setups pair each window with a short one (for example 1 hour and 5 minutes) so the alert also stops quickly after the problem is fixed.
Step 4: Keep alerts healthy
Review alerts regularly: every page that needed no action is a candidate to delete, demote to a ticket or fix at the source. Track pages per on-call shift, and give every alert a runbook link explaining what it means and what to check first.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Threshold alerts on causes (CPU, disk) | Imminent, certain problems such as a disk about to fill | Noise: most cause thresholds fire when users are fine |
| Symptom alerts (error rate, latency) | Paging only when users are affected | May fire later than a cause would; still need tuning |
| SLO burn-rate alerts | Urgency proportional to user impact; fewer, better pages | Need well-defined SLIs and SLOs first |
| Black-box probes (synthetic checks) | Catching total outages and external problems like DNS | Only test the paths they script |
Common mistakes
- Paging on causes such as CPU or memory when users are unaffected.
- Alerting on average latency instead of percentiles.
- Alerts with no runbook, no owner and no clear action.
- Never pruning alerts, until on-call ignores all of them.
- Measuring only from inside the service and missing failures users see first, such as DNS or the CDN.
In the wild
This chapter of Google's SRE book introduces the four golden signals: latency, traffic, errors and saturation. It argues that the distinction between what is broken (the symptom) and why (the cause) is one of the most important in writing monitoring with high signal and low noise, and that effort should go into catching symptoms. It sets a high bar for paging: every page should be actionable and require intelligence, and a page that merits only a robotic response should not be a page. It also warns that averages hide slow tails, and recommends latency histograms.
Check yourself
-
Which is a symptom-based alert?
Show answer
B. Checkout error rate above 2 percent It describes what users experience; the others are possible causes that often hurt nobody.
-
With a 99.9 percent SLO, the current error rate is 1.44 percent. What is the burn rate?
Show answer
B. 14.4x 1.44 percent divided by the 0.1 percent budget is 14.4: the budget is being spent 14.4 times too fast.
-
What does RED stand for?
Show answer
B. Rate, Errors, Duration RED is a quick checklist for request-driven services.
-
An alert fires every night and the fix is always restarting one process. What should you do?
Show answer
B. Automate the restart (and fix the root cause), so it no longer needs a human Pages should need human judgment; mechanical responses should be automated.
Recap
- Monitor what users feel: latency, traffic, errors and saturation (RED for services, USE for resources).
- Page on symptoms that need a human now; use cause metrics for diagnosis.
- SLO burn-rate alerts scale urgency to how fast the error budget is being spent.
- Every page must be actionable, with a runbook; automate the mechanical ones.
- Prune alerts regularly to avoid fatigue.