Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Observability and Operations

System Design HLD-13.02

Distributed tracing

Spans, trace context propagation, sampling strategies and OpenTelemetry, and how to read a trace waterfall to find the critical path.

Intermediate 12 min Last reviewed

The big idea

A checkout request takes 1.2 seconds. It passed through a gateway, the checkout service, cart, pricing, payments and two databases. Each service's metrics look roughly normal, and each one's logs show only its own part. Which call made this request slow? Without a way to follow the request across services, you are guessing.

Distributed tracingRecording the path of a single request through all the services it touches, as a tree of timed operations, so you can see where its time was spent. follows the request. Every service records the operations it performs for that request as spansOne timed operation within a trace, such as an HTTP call or database query, with a start time, duration, parent span and attributes., all tagged with the same trace id and linked parent to child. A tracing backend stitches the spans into a tree and draws it as a waterfall: one bar per operation, on a shared timeline. The slow part usually jumps out immediately.

Think of a relay race filmed by a camera at every handoff. Each camera only sees its own leg, but if every clip is stamped with the race number and the time, you can lay them end to end and see exactly which runner lost the time.

How it works

Step 1: Spans and the waterfall

A trace waterfall with the slowest span glowing A trace drawn as horizontal bars on a shared timeline from 0 to 1,000 ms. The root span POST /checkout covers the whole request. Under it, checkout.place_order covers nearly all of it. Its children are cart (short, near the start), pricing (short, in parallel with cart), payments.charge (900 ms, glowing red), and a short INSERT at the end. POST /checkout 1000 ms checkout.place_order cart pricing payments.charge 900 ms INSERT Figure 1. Each bar is a span on a shared timeline; indentation shows parent and child. Cart and pricing run in parallel and are quick. The payments.charge span takes 900 of the 1,000 ms, so that is where to look.

Each span records its name, start time, duration, status, its parent span and attributes such as http.status_code=200 or db.statement. The first span, the root span, covers the whole request; its children are the calls it made, and so on. Gaps between child spans reveal time spent inside a service doing its own work, such as serialization or waiting for a lock.

Step 2: Context propagation

For spans from different services to join one trace, the trace id and the current span id must travel with every call. This is context propagationPassing trace identifiers along with a request, usually in headers, so that downstream services can attach their spans to the same trace.. Over HTTP, the W3C traceparent header carries them, for example traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01, and messaging systems carry the same values in message headers ( Message queues vs pub/sub vs event streams ). If any hop drops the header, the trace breaks into disconnected pieces.

Step 3: Finding the critical path

The slowest span is not always the one to fix. If two calls run in parallel, the faster one can be made instant without changing the total. The critical pathThe chain of operations in a trace that determined the total latency; speeding up anything off this path does not make the request faster. is the chain of spans that actually determined when the request finished: work on it reduces latency, and work elsewhere does not.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. In the normal checkout, highlight the critical path. Cart and pricing run in parallel: only the slower one, pricing, is on the path. Making cart faster would change nothing.
  2. Switch to slow payment provider. One span dominates the critical path; this is the classic slow-dependency trace.
  3. Switch to N+1 queries in pricing. No single query is slow, but 12 sequential queries make pricing the bottleneck. The waterfall's staircase shape is the giveaway, and a batched query is the fix.

Going deeper

Sampling

Recording every span of every request is expensive at scale, so tracing samples:

  • Head-based sampling decides at the start of a request, for example keep 1 percent, and propagates the decision so the whole trace is kept or dropped together. Cheap and simple, but it throws away most of the interesting slow and failed requests along with the boring ones.
  • Tail-based sampling buffers all spans briefly and decides after the request finishes: keep every error, every request slower than a threshold, and a small share of normal ones. Much better traces, at the cost of a collector that holds spans in memory.

OpenTelemetry

OpenTelemetry is the vendor-neutral standard for instrumentation: APIs and SDKs for many languages, automatic instrumentation for common frameworks and clients, the propagation formats, and a collector that receives, samples and exports telemetry to whatever backend you use. Instrumenting with it once avoids rewriting instrumentation when you change vendors.

Trade-offs

OptionWins whenLoses when
No tracing Monoliths and very simple systems Debugging latency across services by guesswork
Head-based sampling Low overhead, simple setup Most rare errors and slow requests are not captured
Tail-based sampling Keeping all errors and slow requests with modest storage A stateful collector tier and added complexity
Tracing everything Low-traffic or critical paths where every request matters Cost and overhead at scale

Common mistakes

  • Not propagating context through queues, thread pools or async callbacks, breaking traces into fragments.
  • Optimizing the slowest span when it is not on the critical path.
  • Head-based sampling at 1 percent and then finding no trace of the rare failure you are debugging.
  • Putting sensitive data such as tokens or personal data into span attributes.
  • Instrumenting only some services, leaving unexplained gaps in the waterfall.

In the wild

OpenTelemetry

Observability primer (opens in a new tab)

The OpenTelemetry primer defines observability as being able to understand a system from the outside by asking questions about it without knowing its inner workings, including "unknown unknowns". It describes telemetry as traces, metrics and logs; a span as a single unit of work with a name, timing, structured log messages and attributes; and a distributed trace as the path of one request through multiple services, made of spans where the root span represents the request from start to finish. It also frames reliability in terms of SLIs measured from the user's perspective.

Check yourself

  1. What links spans emitted by different services into one trace?

    Show answer

    B. A propagated trace context (trace id and parent span id), typically in a header such as traceparent Each service reads the incoming context and attaches its spans to the same trace and parent.

  2. Cart (30 ms) and pricing (45 ms) run in parallel. You cut cart to 5 ms. How much faster is the request?

    Show answer

    B. Not at all; pricing still determines when both are done Only spans on the critical path affect total latency.

  3. Why might head-based 1 percent sampling miss the bug you are hunting?

    Show answer

    B. It decides before the request runs, so 99 percent of rare slow or failed requests are discarded Tail-based sampling can keep all errors and slow requests because it decides afterwards.

  4. A waterfall shows a staircase of 12 short, sequential database spans inside one service. What does it suggest?

    Show answer

    B. An N+1 query pattern that should be batched Many sequential small queries add up; one batched query removes most of the time.

Recap

  • A trace is a tree of spans for one request; each span records an operation's timing and attributes.
  • Context propagation (for example the traceparent header) stitches spans from many services together.
  • Read the waterfall for the critical path; speeding up spans off it does not help.
  • Head sampling is cheap but misses rare problems; tail sampling keeps errors and slow requests.
  • OpenTelemetry is the standard way to instrument, propagate context and export traces.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy