Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Observability and Operations

System Design HLD-13.01

Logs, metrics and traces

The three pillars of observability: structured logs, metric types and cardinality, traces, and the shared trace id that ties them together.

Beginner 12 min Last reviewed

The big idea

A customer says "checkout was really slow at 2 p.m.". Your service runs on 40 machines and calls six other services. Where do you even start? You cannot attach a debugger to production or reproduce the exact moment. All you have is what the system recorded while it ran. ObservabilityHow well you can understand what a system is doing, and why, from the data it emits, without changing its code. is the practice of recording the right things so you can answer questions like that after the fact, including questions you did not think of in advance.

Three kinds of data do most of the work. Metrics are numbers over time, cheap to keep and great for noticing that something is wrong ("p99 latency jumped at 2 p.m."). Logs are records of individual events with details ("order 812 failed: card declined"). Traces follow one request across every service it touches and show where its time went. Each answers a different question, and they become powerful when they share a common trace idA unique identifier attached to a request at the edge and passed to every service it touches, so logs, metrics and traces from that request can be linked. so you can jump from one to another.

Think of investigating a delayed parcel. A dashboard showing the share of parcels delivered late each day (metrics) tells you Tuesday was bad. The tracking history of one parcel through each depot (a trace) shows where it sat for six hours. The depot's incident notes (logs) say why: the sorting machine jammed.

How it works

Step 1: One request, three signals

One request emitting a log line, a metric and trace spans with the same trace id A request to the checkout service carries trace id b7c2. It emits three kinds of telemetry, all glowing with the same id: a structured JSON log line with status 504 and the trace id; a metric update incrementing the request counter for status 504 and recording 1225 ms in the latency histogram; and two trace spans, db.query taking 1200 ms and render taking 25 ms. An arrow shows jumping from the latency alert to the trace and from the trace to the log. alert to example trace span to its logs GET /cart trace_id = b7c2 Metric requests{status=504} +1, latency 1225 ms Trace b7c2 db.query 1200 ms | render 25 ms Log level=error status=504 trace_id=b7c2 Figure 1. A single request emits all three signals, tagged with the same trace id. An alert on the latency metric leads you to example traces, the trace shows the database query took 1,200 ms, and the log line for that trace id tells you the exact error.
signals.pyruns with python3 (standard library)
import json
from collections import Counter

requests_total = Counter()                         # metric: counter (only goes up)
in_flight = 0                                      # metric: gauge (goes up and down)
BUCKETS = [50, 100, 250, 500, 1000]                # metric: latency histogram buckets (ms)
latency_hist = Counter()
spans = []                                         # trace: one span per operation

def handle(trace_id, path, db_ms, render_ms):
    global in_flight
    in_flight += 1
    spans.append({"trace_id": trace_id, "name": "db.query", "ms": db_ms})
    spans.append({"trace_id": trace_id, "name": "render", "ms": render_ms})
    total = db_ms + render_ms
    status = 200 if total < 1000 else 504
    requests_total[(path, status)] += 1
    latency_hist[next((b for b in BUCKETS if total <= b), "+Inf")] += 1
    in_flight -= 1
    # structured log: one JSON object per event, with the same trace id
    print(json.dumps({"level": "info" if status == 200 else "error", "msg": "request done",
                      "path": path, "status": status, "ms": total, "trace_id": trace_id}))

handle("a1f3", "/cart", db_ms=40, render_ms=30)
handle("b7c2", "/cart", db_ms=1200, render_ms=25)  # the slow one
print("counter  :", dict(requests_total))
print("histogram:", dict(latency_hist), "| gauge in_flight:", in_flight)
print("trace b7c2:", [s for s in spans if s["trace_id"] == "b7c2"])

Output

{"level": "info", "msg": "request done", "path": "/cart", "status": 200, "ms": 70, "trace_id": "a1f3"}
{"level": "error", "msg": "request done", "path": "/cart", "status": 504, "ms": 1225, "trace_id": "b7c2"}
counter  : {('/cart', 200): 1, ('/cart', 504): 1}
histogram: {100: 1, '+Inf': 1} | gauge in_flight: 0
trace b7c2: [{'trace_id': 'b7c2', 'name': 'db.query', 'ms': 1200}, {'trace_id': 'b7c2', 'name': 'render', 'ms': 25}]

The second request is slow. The metrics show a 504 and a request in the slowest histogram bucket, which is what an alert would fire on. The trace for b7c2 shows the database query took 1,200 of its 1,225 milliseconds. The log line, findable by the same trace id, has the full context.

Step 2: Logs

A logA timestamped record of a discrete event emitted by a service, ideally structured as key-value fields rather than free text. records an event. Plain text like Payment failed for user 42 is easy to write and hard to search across billions of lines. Structured logs emit each event as fields, usually JSON: {"msg": "payment failed", "user_id": 42, "reason": "card_declined", "trace_id": "b7c2"}. Now you can filter by any field, count by reason, and join to traces.

Logs are the most detailed signal and the most expensive at volume. Log levels (debug, info, warn, error) and sampling of noisy info logs keep costs down, and personal data such as passwords, tokens and card numbers must never be logged.

Step 3: Metrics

A metricA numeric measurement aggregated over time, such as requests per second or 99th percentile latency, stored as a time series. is a number recorded over time, aggregated so it costs the same whether you serve 10 or 10 million requests. Three basic types:

  • Counter: only goes up, such as total requests or total errors. Dashboards show its rate per second.
  • Gauge: goes up and down, such as requests in flight, queue depth or memory used.
  • Histogram: counts observations into buckets (under 50 ms, under 100 ms, ...), from which you compute percentiles such as p99. Averages hide the slow tail; histograms show it.

Metrics carry labels (tags) such as path, status and region. Every unique combination of label values is a separate time series. Label by something with millions of values, such as user_id, and you create millions of series, the cardinalityThe number of distinct time series a metric produces, determined by the number of unique combinations of its label values. problem that makes metrics systems slow and expensive. High-cardinality details belong in logs and traces.

Step 4: Traces

A trace follows one request through every service it touches, recorded as a tree of spans, each one an operation with a start time and duration ( Distributed tracing ). Traces answer "where did the time go?" for a single request, which neither metrics nor logs can show on their own across many services.

Step 5: Connect them with a trace id

Generate an id at the edge (the load balancer or first service), pass it on every downstream call in a header, and include it in every log line and span. Many metrics systems can also attach example trace ids (exemplars) to histogram buckets. Then the workflow becomes: an alert fires on a metric, you open an example trace from the bad time window, find the slow span, and read the logs for that trace id.

Trade-offs

OptionWins whenLoses when
Metrics Dashboards, alerting and trends; cheap at any traffic volume Per-request detail; high-cardinality labels
Logs Detailed context for specific events and errors Cost and noise at high volume; hard to see request paths across services
Traces Where time goes for one request across services Cost, so usually sampled; needs instrumentation in every service

Common mistakes

  • Free-text logs that cannot be filtered or aggregated.
  • Putting user ids, request ids or URLs with ids into metric labels, exploding cardinality.
  • Alerting and dashboarding on average latency, hiding the slow tail.
  • Not propagating a trace id, so logs from different services cannot be connected.
  • Logging secrets or personal data.

In the wild

Most production stacks combine a metrics system such as Prometheus, a log pipeline such as Elasticsearch or Loki, and a tracing backend such as Jaeger or Tempo, or a commercial product that does all three. OpenTelemetry has become the common, vendor-neutral way to instrument applications for all three signals and to propagate trace context between services.

Check yourself

  1. Which signal is best for alerting that p99 latency has doubled?

    Show answer

    B. Metrics Metrics are cheap aggregated time series, ideal for thresholds and trends.

  2. Why should you not label a request metric with user_id?

    Show answer

    B. Each unique value creates a new time series, exploding cardinality and cost Millions of users mean millions of series; put such details in logs and traces.

  3. Queue depth that rises and falls is which metric type?

    Show answer

    B. Gauge Gauges represent a current value that can go up or down.

  4. What lets you jump from a slow trace to the exact log lines for that request?

    Show answer

    B. A shared trace id included in every log line and span Propagating and logging the trace id links the signals for one request.

Recap

  • Observability means recording enough to answer new questions about production after the fact.
  • Metrics are cheap time series for alerting; logs are detailed events; traces show one request's path.
  • Use structured logs, the right metric types (counter, gauge, histogram) and percentiles, not averages.
  • Keep metric label cardinality low; put high-cardinality detail in logs and traces.
  • Propagate a trace id everywhere so you can move from alert to trace to logs.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy