System Design HLD-15.05
Time-series data
Storing metrics and events over time: append-heavy writes, delta and XOR compression, downsampling into rollups, retention, and the cardinality trap.
The big idea
A fleet of 2,000 servers each reports 500 metrics every 10 seconds: 100,000 new data points per second, every second, forever. Nobody updates a past measurement, nearly every query asks for a time range ("CPU for the last hour"), and the older the data, the less detail anyone needs. A general-purpose database can store this, but wastes most of its effort on features this workload never uses.
Time-series dataData points that are each a timestamp and a value for a named series, written mostly in time order and queried by time range and aggregation. has a very particular shape: append-only, ordered by time, queried in ranges, aggregated, and less valuable as it ages. Specialized time-series databases exploit every part of that: they compress consecutive points to a byte or two each, keep recent data at full resolution and old data as summaries, and drop data past a retention period automatically.
Think of a weather station's records. The last day is kept minute by minute; last year survives as daily highs and lows; the last century as monthly averages. Nobody needs the temperature at 14:03 on a Tuesday in 1987, and keeping it would fill the archive.
How it works
Step 1: The data model
A series is identified by a metric name plus labels, such as http_requests_total{service="cart", status="500", region="eu"}. Each series is a list of (timestamp, value) points. Writes arrive in roughly time order and almost never update the past, so storage engines append to an in-memory buffer and periodically write immutable, compressed blocks for a time window, much like an LSM-tree ( Storage engines: B-tree vs LSM-tree ). Queries find the matching series by label, then read only the blocks for the requested window.
Step 2: Compression
import random, struct
rnd = random.Random(5)
# One metric scraped every 10 s for a day: timestamps are regular, values change slowly.
ts = [1_700_000_000 + 10 * i for i in range(8640)]
vals, v = [], 50.0
for _ in ts:
v = max(0.0, v + rnd.choice([0.0, 0.0, 0.0, 0.5, -0.5]))
vals.append(v)
raw = len(ts) * (8 + 8) # 8-byte timestamp + 8-byte float
# Delta-of-delta timestamps: regular intervals make almost every entry 0 (about 1 bit).
dod = [b - a - (a - z) for z, a, b in zip(ts, ts[1:], ts[2:])]
ts_bits = 64 + 64 + sum(1 if d == 0 else 32 for d in dod)
# XOR values with the previous one: unchanged values XOR to 0 (about 1 bit), others ~ 20 bits.
as_int = [struct.unpack(">Q", struct.pack(">d", x))[0] for x in vals]
val_bits = 64 + sum(1 if a == b else 20 for a, b in zip(as_int, as_int[1:]))
compressed = (ts_bits + val_bits) / 8
print(f"raw {raw:,} bytes, compressed ~{compressed:,.0f} bytes ({raw / compressed:.0f}x smaller)")
# Downsampling: keep 10 s points for recent data, 5-minute rollups (min/avg/max) for older data.
rollups = [(min(c), sum(c) / len(c), max(c)) for c in (vals[i:i + 30] for i in range(0, len(vals), 30))]
print(f"10 s points per day: {len(vals):,}; 5 min rollups per day: {len(rollups)}")
print("first rollup (min, avg, max):", tuple(round(x, 2) for x in rollups[0]))Output
raw 138,240 bytes, compressed ~10,349 bytes (13x smaller)
10 s points per day: 8,640; 5 min rollups per day: 288
first rollup (min, avg, max): (49.0, 49.78, 50.5) Two tricks do most of the work, popularized by Facebook's Gorilla in-memory database:
- Delta-of-delta timestamps: points arrive at regular intervals, so the difference between consecutive differences is almost always zero, which can be stored in about one bit.
- XOR of consecutive values: a value that did not change XORs to zero (one bit); one that changed slightly shares most of its bits with the previous value, so only the differing middle bits are stored.
On this sample day of 10-second points, that shrinks 16 bytes per point to a little over 1 byte, about 13 times smaller. Prometheus's storage engine uses this style of compression.
Step 3: Downsampling and retention
DownsamplingReplacing high-resolution data points with aggregates over larger time buckets, such as 5-minute minimum, maximum, sum and count, to save storage for older data. replaces old raw points with aggregates over larger buckets: for example, 10-second points for 15 days, then 5-minute rollups for a year, then hourly rollups beyond. Store several aggregates (min, max, sum, count) rather than only the average, so later queries can still compute correct averages and see spikes. Retention policies then delete raw data, and eventually rollups, past their age limit, typically by dropping whole time-window blocks, which is far cheaper than deleting rows.
Step 4: The cardinality trap
The number of series, not the number of points, is what usually breaks time-series systems. Every unique combination of label values is a separate series with its own index entries and buffers ( Logs, metrics and traces ). Add a user_id label to a request metric with a million users and you create a million series overnight, memory explodes and queries crawl. Keep labels to values with a small, bounded set (status code, region, endpoint), and send per-user detail to logs or traces instead.
Going deeper
Choosing a store
Purpose-built engines (Prometheus for metrics, InfluxDB, VictoriaMetrics) optimize for this exact workload. Relational extensions such as TimescaleDB keep SQL and joins while partitioning tables by time. Columnar analytics databases handle huge event volumes with flexible queries. For long-term metrics, many teams send data from short-retention local stores to a scalable, horizontally sharded long-term backend.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| General relational table | Small volumes, joins with business data | Storage efficiency, ingest rate and cheap retention at scale |
| Purpose-built time-series database | High ingest, heavy compression, fast range aggregation | Limited query flexibility; sensitive to cardinality |
| Time-partitioned SQL (e.g. TimescaleDB) | Time-series scale with SQL and joins | Usually less compression than specialized engines |
| Keeping raw data forever | Audit or research needs for full detail | Storage cost and slow long-range queries |
Common mistakes
- Adding unbounded labels such as user id, request id or full URL to metrics.
- Storing only averages in rollups, losing spikes and making re-aggregation wrong.
- Deleting old data row by row instead of dropping whole time blocks.
- One retention policy for everything, keeping debug metrics as long as billing data.
- Querying months of raw data where rollups would answer instantly.
In the wild
Prometheus is the most widely used open source metrics system, with a local time-series database using Gorilla-style compression and per-block retention. Facebook described the delta-of-delta and XOR compression techniques in its Gorilla paper. InfluxDB, TimescaleDB and VictoriaMetrics are popular stores, and IoT platforms and financial tick databases rely on the same principles.
Check yourself
-
Why do delta-of-delta timestamps compress so well?
Show answer
B. Points arrive at regular intervals, so the change in the interval is usually zero Regular scrapes make second differences almost always zero, about one bit each.
-
Why store min, max, sum and count in a rollup instead of only the average?
Show answer
B. So spikes stay visible and averages can be re-aggregated correctly over bigger windows An average of averages is wrong when buckets have different counts, and averages hide peaks.
-
A team adds a request_id label to a latency metric. What happens?
Show answer
B. Every request creates a new series, exploding cardinality, memory and query time Series count, not point count, is the scaling limit; unbounded labels break the system.
-
What is the cheapest way to apply a 30-day retention policy?
Show answer
B. Drop whole time-window blocks once they are older than 30 days Time-partitioned storage makes expiry a file or partition drop.
Recap
- Time-series data is append-heavy, ordered by time, queried by range and aggregated.
- Delta-of-delta timestamps and XOR-encoded values compress points to about a byte or two.
- Downsample old data into rollups (min, max, sum, count) and expire it with retention policies.
- Time-partitioned blocks make writes, range scans and deletion cheap.
- Cardinality, the number of unique label combinations, is the most common way these systems fail.