System Design HLD-19.18
Design a metrics and monitoring system
Design monitoring: push vs pull collection, a time-series store with compression and downsampling, alert evaluation, and cardinality limits.
The problem
Two thousand servers each report hundreds of numbers every ten seconds: requests per second, error counts, latency histograms, memory, queue depth. Engineers want dashboards that load in a second, and they want to be woken up within a minute or two when something breaks, and only then. A monitoring system collects those numbers, stores them cheaply for months, answers queries over them, and evaluates alert rules continuously.
It has an unusual shape. Writes are enormous, constant and append-only; reads are rare by comparison but scan many series at once; recent data is precious and old data is mostly needed in summary. And the monitoring system has a special availability requirement: it must work when everything else is broken, because that is exactly when people need it.
Requirements
Functional
- Collect metrics (counters, gauges, histograms) with labels from services, hosts and short-lived jobs.
- Store them; query by metric name and labels with aggregation over time ranges.
- Dashboards with graphs; alert rules ("error ratio above 5 percent for 5 minutes") that notify on-call people.
- Keep recent data at full resolution and older data in summary.
Non-functional
- Scale: 2,000 instances, 6 million active time series, a sample every 10 seconds.
- Alert latency: a rule fires within about a minute of the condition starting.
- Query latency: a dashboard over the last few hours loads in about a second.
- Availability: alerting keeps working during outages of the systems it watches, and of parts of itself.
Out of scope
- Logs and traces (separate stores), anomaly detection, and the on-call scheduling product.
Estimates
The sample below works out series, ingestion rate and storage with downsampling:
# How many time series, samples and bytes does a metrics system hold?
# Every unique combination of label values is its own series.
DAY = 86_400
instances = 50 * 40 # 50 services x 40 instances each
per_instance = 30 * 5 * 20 # endpoints x status classes x metric names
series = instances * per_instance
print(f"series: {series:,}")
SCRAPE = 10 # seconds between samples
rate = series / SCRAPE
print(f"samples per second: {rate:,.0f}")
BYTES = 2 # per sample after compression (assumed)
tiers = [("raw 10 s", 10, 15), # (name, step in s, days kept)
("5 min rollup", 300, 90),
("1 h rollup", 3600, 730)]
total = 0
for name, step, days in tiers:
points = series * days * DAY / step
values = 1 if step == SCRAPE else 4 # rollups: min, max, sum, count
size = points * BYTES * values
total += size
print(f"{name:13} kept {days:3} days: {size / 1e12:5.2f} TB")
print(f"total {total / 1e12:.2f} TB; raw for 2 years would be "
f"{series * 730 * DAY / 10 * BYTES / 1e12:.0f} TB")
print(f"add a user_id label with 1,000,000 values: up to "
f"{series * 1_000_000:,} series")Output
series: 6,000,000
samples per second: 600,000
raw 10 s kept 15 days: 1.56 TB
5 min rollup kept 90 days: 1.24 TB
1 h rollup kept 730 days: 0.84 TB
total 3.64 TB; raw for 2 years would be 76 TB
add a user_id label with 1,000,000 values: up to 6,000,000,000,000 series - Ingestion: 600,000 samples per second, constant, day and night. Each sample is a timestamp and a value, which compress extremely well because consecutive timestamps are evenly spaced and consecutive values are close. The sample assumes 2 bytes per sample after compression.
- Storage: about 3.6 TB with three retention tiers, against 76 TB if raw data were kept for two years. Downsampling, not disks, is what makes long retention affordable.
- Cardinality: the last line is the danger. One label with a million values (a user id) multiplies the series count. Series, not samples, are what exhaust memory and indexes.
API
# collection (pull): every target exposes its current values
GET /metrics
http_requests_total{service="cart",endpoint="/add",status="5xx"} 1027
http_request_duration_seconds_bucket{service="cart",le="0.25"} 99120
# collection (push), for short-lived jobs and edge devices
POST /v1/write [ { series labels, [ (timestamp, value) ... ] } ]
# query
GET /v1/query_range?q=sum by (service) (rate(http_requests_total{status="5xx"}[5m]))&start=..&end=..&step=30s
# alert rules
PUT /v1/rules/{name} { expr, for: "5m", severity, route }
Data model
- Series: identified by metric name plus a sorted set of label pairs, mapped to a numeric series id.
- Inverted index:
label=value -> list of series ids, so a query forservice="cart", status="5xx"intersects two posting lists ( Inverted indexes and full-text search ). - Chunks: per series, blocks of compressed
(timestamp, value)pairs covering, for example, two hours. Recent chunks live in memory and a write-ahead log; full ones are written to disk or object storage as immutable blocks ( Time-series data ). - Rollups: per series, per 5 minutes or 1 hour:
min, max, sum, count, enough to compute averages and extremes later.
High-level design
- Collect path Collectors discover targets and scrape each one's metrics endpoint every 10 seconds. Short-lived batch jobs push their final values to a gateway, which is scraped like any target. Samples stream to ingesters, sharded by series.
- Store path Ingesters append samples to in-memory chunks with a write-ahead log and flush compressed blocks to object storage every couple of hours. A compactor merges blocks and writes 5-minute and 1-hour rollups. Queries read recent data from ingesters and history from storage.
- Alert path Rule evaluators, run as a redundant pair, query recent data every 30 seconds. When a condition has held for its 'for' duration, the alert fires; the notifier deduplicates the pair's copies, groups related alerts and routes them to the right on-call person.
Deep dives
Deep dive 1: Push or pull
- Pull (the collector scrapes targets): the monitoring system controls the rate, so a misbehaving service cannot flood it; a failed scrape is itself a signal ("target down"); and you can curl any endpoint to see what it reports. It needs service discovery to know what to scrape ( Service discovery ), and it struggles with jobs that finish before the next scrape and with targets behind firewalls.
- Push (agents send to the system): works for short-lived jobs, edge devices and networks where the collector cannot reach in. But a silent target is ambiguous (dead, or just quiet?), and a buggy client can send a flood, so ingestion needs per-tenant rate limits.
Many systems use both: pull as the default, with a push gateway or agent for the exceptions. Either way, ingestion is sharded by series id so each ingester owns a slice of series and their in-memory chunks.
Deep dive 2: Storing time series
Time-series storage exploits the shape of the data. Timestamps arrive at regular intervals, so storing the difference of differences usually gives zero, which encodes in a bit or two. Values change slowly, so XOR with the previous value leaves mostly zero bits. Together, a sample shrinks from 16 bytes to a couple of bytes, which is the assumption in the estimate.
Writes go to an in-memory head block per series, protected by a write-ahead log. Every couple of hours the head is cut into an immutable, compressed block. A compactor later merges small blocks into larger ones, and produces rollups. Queries over the last few hours hit memory; queries over months read the rollups. This is the LSM idea ( Storage engines: B-tree vs LSM-tree ) specialised for append-only, time-ordered data.
Downsampling keeps min, max, sum, count per interval rather than just the average, so a dashboard over a year can still show that latency spiked, and averages can be recombined correctly over longer windows.
Deep dive 3: Alert evaluation
An alert rule is a query plus a duration: "the 5xx ratio for a service is above 5 percent for 5 minutes". The evaluator runs every rule every 30 seconds against recent data in memory; the duration suppresses blips. Firing alerts go to a notifier that:
- deduplicates copies (evaluators run as a redundant pair, so each alert arrives twice),
- groups related alerts (fifty instances of one service failing is one page, not fifty),
- routes by labels to the owning team, with silences during maintenance and inhibition ("do not page for every service if the whole region is down").
The alerting path must be more available than what it watches. Run it as a separate, replicated deployment, ideally in another failure domain, reading recent data directly from ingesters rather than through the long-term query path. Add a "dead man's switch": an alert that always fires, and an external service that pages if it ever stops arriving, so a broken monitoring system cannot fail silently ( Monitoring and alerting ).
Deep dive 4: Cardinality limits
Every unique combination of label values is a new series with its own memory, index entries and chunks. Labels such as service, endpoint and status class are bounded and safe. Labels such as user id, request id, full URL with ids in it, or error message text are unbounded: a single deploy that adds one can multiply the series count, run ingesters out of memory and take monitoring down at the worst moment.
Defences: per-tenant and per-metric limits on active series that reject new series beyond the limit; relabelling rules that drop or rewrite dangerous labels at collection time; and dashboards that show which metrics are growing. Per-user and per-request detail belongs in logs and traces ( Logs, metrics and traces ), which are built for high-cardinality data.
Try it
Try this:
- Sort all six, then Check.
- A team adds a
customer_idlabel to a request counter for 50,000 customers across 30 endpoints. How many new series could that create? - Why does the alerting path read from ingesters directly instead of through the long-term query path?
Bottlenecks and failure modes
| Failure | Impact | Mitigation |
|---|---|---|
| Cardinality explosion | Ingesters run out of memory | Series limits per tenant and metric; relabel at collection |
| Monitoring down during an incident | Blind exactly when needed | Separate, replicated alert path; dead man's switch |
| Alert storm | On-call flooded with pages | Grouping, inhibition, deduplication |
| Ingester crash | Recent samples lost | Write-ahead log; replicate samples to two ingesters |
| Expensive dashboard query | Slow queries for everyone | Query limits, rollups for long ranges, recording rules |
| Flapping alert | Pages on and off | A 'for' duration and hysteresis |
Trade-offs we made
| Option | Wins when | Loses when |
|---|---|---|
| Pull by default (chosen) | Control over load; failed scrape is a signal | Needs discovery; awkward for short jobs |
| Push | Works through firewalls and for short jobs | Silence is ambiguous; clients can flood the system |
| Downsample old data (chosen) | Years of retention for a few TB | Fine detail lost after the raw window |
| Separate alert path (chosen) | Alerts survive query and storage problems | Two deployments to run and keep in sync |
Extensions
- Recording rules that precompute expensive queries every minute.
- Multi-tenant monitoring as a service with per-tenant quotas.
- SLO burn-rate alerts instead of raw thresholds ( SLIs, SLOs, SLAs and error budgets ).
- Exemplars linking a metric spike to example traces.
In the wild
Where you will see this: open-source monitoring stacks built around pull-based collectors and a time-series database, hosted metrics services, and the internal monitoring platforms of large companies. They share the same core decisions: pull or push, compressed chunks in memory and object storage, downsampling for retention, an alerting path that outlives what it watches, and hard limits on cardinality.
Check yourself
-
What usually exhausts a metrics system first?
Show answer
B. The number of distinct series, driven by label cardinality Each series costs memory and index space; unbounded labels multiply them.
-
Why keep min, max, sum and count when downsampling, not just the average?
Show answer
B. So spikes stay visible and averages can be recombined correctly over longer windows An average of averages hides spikes and is wrong when counts differ.
-
Why does an alert rule have a 'for' duration?
Show answer
B. To ignore brief blips so only sustained problems page someone A short spike that resolves itself should not wake anyone.
-
What is a dead man's switch in monitoring?
Show answer
B. An always-firing alert whose absence, noticed externally, means monitoring itself is broken It makes a failure of the alerting pipeline visible.
Recap
- Every unique label combination is a series; limit cardinality and keep ids in logs and traces.
- Pull is the default for control and health signals; push covers short jobs and unreachable targets.
- Compressed chunks in memory for recent data, immutable blocks in object storage for history.
- Downsample with min, max, sum and count to keep years of data cheaply.
- Run alerting as a separate, replicated path with grouping, dedupe and a dead man's switch.
Next up
- Crack the system design round learning path