Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Design Case Studies

System Design HLD-19.18

Design a metrics and monitoring system

Design monitoring: push vs pull collection, a time-series store with compression and downsampling, alert evaluation, and cardinality limits.

Advanced 20 min Last reviewed

The problem

Two thousand servers each report hundreds of numbers every ten seconds: requests per second, error counts, latency histograms, memory, queue depth. Engineers want dashboards that load in a second, and they want to be woken up within a minute or two when something breaks, and only then. A monitoring system collects those numbers, stores them cheaply for months, answers queries over them, and evaluates alert rules continuously.

It has an unusual shape. Writes are enormous, constant and append-only; reads are rare by comparison but scan many series at once; recent data is precious and old data is mostly needed in summary. And the monitoring system has a special availability requirement: it must work when everything else is broken, because that is exactly when people need it.

Requirements

Functional

  • Collect metrics (counters, gauges, histograms) with labels from services, hosts and short-lived jobs.
  • Store them; query by metric name and labels with aggregation over time ranges.
  • Dashboards with graphs; alert rules ("error ratio above 5 percent for 5 minutes") that notify on-call people.
  • Keep recent data at full resolution and older data in summary.

Non-functional

  • Scale: 2,000 instances, 6 million active time series, a sample every 10 seconds.
  • Alert latency: a rule fires within about a minute of the condition starting.
  • Query latency: a dashboard over the last few hours loads in about a second.
  • Availability: alerting keeps working during outages of the systems it watches, and of parts of itself.

Out of scope

  • Logs and traces (separate stores), anomaly detection, and the on-call scheduling product.

Estimates

The sample below works out series, ingestion rate and storage with downsampling:

tsdb.pyruns with python3 (standard library)
# How many time series, samples and bytes does a metrics system hold?
# Every unique combination of label values is its own series.
DAY = 86_400
instances = 50 * 40           # 50 services x 40 instances each
per_instance = 30 * 5 * 20    # endpoints x status classes x metric names
series = instances * per_instance
print(f"series: {series:,}")
SCRAPE = 10                   # seconds between samples
rate = series / SCRAPE
print(f"samples per second: {rate:,.0f}")

BYTES = 2                     # per sample after compression (assumed)
tiers = [("raw 10 s", 10, 15),          # (name, step in s, days kept)
         ("5 min rollup", 300, 90),
         ("1 h rollup", 3600, 730)]
total = 0
for name, step, days in tiers:
    points = series * days * DAY / step
    values = 1 if step == SCRAPE else 4  # rollups: min, max, sum, count
    size = points * BYTES * values
    total += size
    print(f"{name:13} kept {days:3} days: {size / 1e12:5.2f} TB")
print(f"total {total / 1e12:.2f} TB; raw for 2 years would be "
      f"{series * 730 * DAY / 10 * BYTES / 1e12:.0f} TB")

print(f"add a user_id label with 1,000,000 values: up to "
      f"{series * 1_000_000:,} series")

Output

series: 6,000,000
samples per second: 600,000
raw 10 s      kept  15 days:  1.56 TB
5 min rollup  kept  90 days:  1.24 TB
1 h rollup    kept 730 days:  0.84 TB
total 3.64 TB; raw for 2 years would be 76 TB
add a user_id label with 1,000,000 values: up to 6,000,000,000,000 series
  • Ingestion: 600,000 samples per second, constant, day and night. Each sample is a timestamp and a value, which compress extremely well because consecutive timestamps are evenly spaced and consecutive values are close. The sample assumes 2 bytes per sample after compression.
  • Storage: about 3.6 TB with three retention tiers, against 76 TB if raw data were kept for two years. Downsampling, not disks, is what makes long retention affordable.
  • Cardinality: the last line is the danger. One label with a million values (a user id) multiplies the series count. Series, not samples, are what exhaust memory and indexes.

API

# collection (pull): every target exposes its current values
GET /metrics
  http_requests_total{service="cart",endpoint="/add",status="5xx"} 1027
  http_request_duration_seconds_bucket{service="cart",le="0.25"} 99120
# collection (push), for short-lived jobs and edge devices
POST /v1/write   [ { series labels, [ (timestamp, value) ... ] } ]
# query
GET  /v1/query_range?q=sum by (service) (rate(http_requests_total{status="5xx"}[5m]))&start=..&end=..&step=30s
# alert rules
PUT  /v1/rules/{name}   { expr, for: "5m", severity, route }

Data model

  • Series: identified by metric name plus a sorted set of label pairs, mapped to a numeric series id.
  • Inverted index: label=value -> list of series ids, so a query for service="cart", status="5xx" intersects two posting lists ( Inverted indexes and full-text search ).
  • Chunks: per series, blocks of compressed (timestamp, value) pairs covering, for example, two hours. Recent chunks live in memory and a write-ahead log; full ones are written to disk or object storage as immutable blocks ( Time-series data ).
  • Rollups: per series, per 5 minutes or 1 hour: min, max, sum, count, enough to compute averages and extremes later.

High-level design

Monitoring: collect, store and alert paths Collect path: collectors scrape metrics endpoints of discovered targets every 10 seconds, while short-lived jobs push to a gateway that collectors also scrape; samples are written to ingesters. Store path: ingesters keep recent data in memory with a write-ahead log, flush compressed blocks to object storage, and a compactor downsamples older blocks; the query service reads both recent and historical data for dashboards. Alert path: a separate, replicated rule evaluator queries recent data every 30 seconds and sends firing alerts to a notifier that deduplicates, groups and routes them to on-call people. scrape samples flush recent history every 30 s firing page Services /metrics endpoints Push gateway batch jobs Collectors scrape every 10 s Ingesters memory + WAL Object storage compressed blocks Compactor downsample Query service Dashboards Rule evaluator replicated Notifier dedupe, group, route On-call Figure 1. Collection, storage and alerting are separate paths. Recent data lives in memory for fast queries and alerts; older data is compacted, downsampled and kept in cheap object storage.
  1. Collect path Collectors discover targets and scrape each one's metrics endpoint every 10 seconds. Short-lived batch jobs push their final values to a gateway, which is scraped like any target. Samples stream to ingesters, sharded by series.
  2. Store path Ingesters append samples to in-memory chunks with a write-ahead log and flush compressed blocks to object storage every couple of hours. A compactor merges blocks and writes 5-minute and 1-hour rollups. Queries read recent data from ingesters and history from storage.
  3. Alert path Rule evaluators, run as a redundant pair, query recent data every 30 seconds. When a condition has held for its 'for' duration, the alert fires; the notifier deduplicates the pair's copies, groups related alerts and routes them to the right on-call person.

Deep dives

Deep dive 1: Push or pull

  • Pull (the collector scrapes targets): the monitoring system controls the rate, so a misbehaving service cannot flood it; a failed scrape is itself a signal ("target down"); and you can curl any endpoint to see what it reports. It needs service discovery to know what to scrape ( Service discovery ), and it struggles with jobs that finish before the next scrape and with targets behind firewalls.
  • Push (agents send to the system): works for short-lived jobs, edge devices and networks where the collector cannot reach in. But a silent target is ambiguous (dead, or just quiet?), and a buggy client can send a flood, so ingestion needs per-tenant rate limits.

Many systems use both: pull as the default, with a push gateway or agent for the exceptions. Either way, ingestion is sharded by series id so each ingester owns a slice of series and their in-memory chunks.

Deep dive 2: Storing time series

Time-series storage exploits the shape of the data. Timestamps arrive at regular intervals, so storing the difference of differences usually gives zero, which encodes in a bit or two. Values change slowly, so XOR with the previous value leaves mostly zero bits. Together, a sample shrinks from 16 bytes to a couple of bytes, which is the assumption in the estimate.

Writes go to an in-memory head block per series, protected by a write-ahead log. Every couple of hours the head is cut into an immutable, compressed block. A compactor later merges small blocks into larger ones, and produces rollups. Queries over the last few hours hit memory; queries over months read the rollups. This is the LSM idea ( Storage engines: B-tree vs LSM-tree ) specialised for append-only, time-ordered data.

Downsampling keeps min, max, sum, count per interval rather than just the average, so a dashboard over a year can still show that latency spiked, and averages can be recombined correctly over longer windows.

Deep dive 3: Alert evaluation

An alert rule is a query plus a duration: "the 5xx ratio for a service is above 5 percent for 5 minutes". The evaluator runs every rule every 30 seconds against recent data in memory; the duration suppresses blips. Firing alerts go to a notifier that:

  • deduplicates copies (evaluators run as a redundant pair, so each alert arrives twice),
  • groups related alerts (fifty instances of one service failing is one page, not fifty),
  • routes by labels to the owning team, with silences during maintenance and inhibition ("do not page for every service if the whole region is down").

The alerting path must be more available than what it watches. Run it as a separate, replicated deployment, ideally in another failure domain, reading recent data directly from ingesters rather than through the long-term query path. Add a "dead man's switch": an alert that always fires, and an external service that pages if it ever stops arriving, so a broken monitoring system cannot fail silently ( Monitoring and alerting ).

Deep dive 4: Cardinality limits

Every unique combination of label values is a new series with its own memory, index entries and chunks. Labels such as service, endpoint and status class are bounded and safe. Labels such as user id, request id, full URL with ids in it, or error message text are unbounded: a single deploy that adds one can multiply the series count, run ingesters out of memory and take monitoring down at the worst moment.

Defences: per-tenant and per-metric limits on active series that reject new series beyond the limit; relabelling rules that drop or rewrite dangerous labels at collection time; and dashboards that show which metrics are growing. Per-user and per-request detail belongs in logs and traces ( Logs, metrics and traces ), which are built for high-cardinality data.

Try it

Where should each piece of information go?

Show answers
  • HTTP status class (2xx, 4xx, 5xx): Metric label. A handful of values, useful for error ratios and alerts.
  • The user id who made the request: Log or trace field. Millions of values would create millions of series; search for a user in logs instead.
  • Service name: Metric label. Bounded and essential for grouping and routing alerts.
  • The full request URL, including order ids: Log or trace field. Every id makes a new series. Use the route template (/orders/:id) as the label instead.
  • Region or availability zone: Metric label. A few values, and alerts often group by them.
  • The text of an exception message: Log or trace field. Free text is unbounded; count errors by type in metrics and keep the message in logs.

Try this:

  1. Sort all six, then Check.
  2. A team adds a customer_id label to a request counter for 50,000 customers across 30 endpoints. How many new series could that create?
  3. Why does the alerting path read from ingesters directly instead of through the long-term query path?

Bottlenecks and failure modes

FailureImpactMitigation
Cardinality explosionIngesters run out of memorySeries limits per tenant and metric; relabel at collection
Monitoring down during an incidentBlind exactly when neededSeparate, replicated alert path; dead man's switch
Alert stormOn-call flooded with pagesGrouping, inhibition, deduplication
Ingester crashRecent samples lostWrite-ahead log; replicate samples to two ingesters
Expensive dashboard querySlow queries for everyoneQuery limits, rollups for long ranges, recording rules
Flapping alertPages on and offA 'for' duration and hysteresis

Trade-offs we made

OptionWins whenLoses when
Pull by default (chosen) Control over load; failed scrape is a signal Needs discovery; awkward for short jobs
Push Works through firewalls and for short jobs Silence is ambiguous; clients can flood the system
Downsample old data (chosen) Years of retention for a few TB Fine detail lost after the raw window
Separate alert path (chosen) Alerts survive query and storage problems Two deployments to run and keep in sync

Extensions

  • Recording rules that precompute expensive queries every minute.
  • Multi-tenant monitoring as a service with per-tenant quotas.
  • SLO burn-rate alerts instead of raw thresholds ( SLIs, SLOs, SLAs and error budgets ).
  • Exemplars linking a metric spike to example traces.

In the wild

Where you will see this: open-source monitoring stacks built around pull-based collectors and a time-series database, hosted metrics services, and the internal monitoring platforms of large companies. They share the same core decisions: pull or push, compressed chunks in memory and object storage, downsampling for retention, an alerting path that outlives what it watches, and hard limits on cardinality.

Check yourself

  1. What usually exhausts a metrics system first?

    Show answer

    B. The number of distinct series, driven by label cardinality Each series costs memory and index space; unbounded labels multiply them.

  2. Why keep min, max, sum and count when downsampling, not just the average?

    Show answer

    B. So spikes stay visible and averages can be recombined correctly over longer windows An average of averages hides spikes and is wrong when counts differ.

  3. Why does an alert rule have a 'for' duration?

    Show answer

    B. To ignore brief blips so only sustained problems page someone A short spike that resolves itself should not wake anyone.

  4. What is a dead man's switch in monitoring?

    Show answer

    B. An always-firing alert whose absence, noticed externally, means monitoring itself is broken It makes a failure of the alerting pipeline visible.

Recap

  • Every unique label combination is a series; limit cardinality and keep ids in logs and traces.
  • Pull is the default for control and health signals; push covers short jobs and unreachable targets.
  • Compressed chunks in memory for recent data, immutable blocks in object storage for history.
  • Downsample with min, max, sum and count to keep years of data cheaply.
  • Run alerting as a separate, replicated path with grouping, dedupe and a dead man's switch.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up