System Design HLD-13.02
Distributed tracing
Spans, trace context propagation, sampling strategies and OpenTelemetry, and how to read a trace waterfall to find the critical path.
The big idea
A checkout request takes 1.2 seconds. It passed through a gateway, the checkout service, cart, pricing, payments and two databases. Each service's metrics look roughly normal, and each one's logs show only its own part. Which call made this request slow? Without a way to follow the request across services, you are guessing.
Distributed tracingRecording the path of a single request through all the services it touches, as a tree of timed operations, so you can see where its time was spent. follows the request. Every service records the operations it performs for that request as spansOne timed operation within a trace, such as an HTTP call or database query, with a start time, duration, parent span and attributes., all tagged with the same trace id and linked parent to child. A tracing backend stitches the spans into a tree and draws it as a waterfall: one bar per operation, on a shared timeline. The slow part usually jumps out immediately.
Think of a relay race filmed by a camera at every handoff. Each camera only sees its own leg, but if every clip is stamped with the race number and the time, you can lay them end to end and see exactly which runner lost the time.
How it works
Step 1: Spans and the waterfall
Each span records its name, start time, duration, status, its parent span and attributes such as http.status_code=200 or db.statement. The first span, the root span, covers the whole request; its children are the calls it made, and so on. Gaps between child spans reveal time spent inside a service doing its own work, such as serialization or waiting for a lock.
Step 2: Context propagation
For spans from different services to join one trace, the trace id and the current span id must travel with every call. This is context propagationPassing trace identifiers along with a request, usually in headers, so that downstream services can attach their spans to the same trace.. Over HTTP, the W3C traceparent header carries them, for example traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01, and messaging systems carry the same values in message headers ( Message queues vs pub/sub vs event streams ). If any hop drops the header, the trace breaks into disconnected pieces.
Step 3: Finding the critical path
The slowest span is not always the one to fix. If two calls run in parallel, the faster one can be made instant without changing the total. The critical pathThe chain of operations in a trace that determined the total latency; speeding up anything off this path does not make the request faster. is the chain of spans that actually determined when the request finished: work on it reduces latency, and work elsewhere does not.
Try it
Try this:
- In the normal checkout, highlight the critical path. Cart and pricing run in parallel: only the slower one, pricing, is on the path. Making cart faster would change nothing.
- Switch to slow payment provider. One span dominates the critical path; this is the classic slow-dependency trace.
- Switch to N+1 queries in pricing. No single query is slow, but 12 sequential queries make pricing the bottleneck. The waterfall's staircase shape is the giveaway, and a batched query is the fix.
Going deeper
Sampling
Recording every span of every request is expensive at scale, so tracing samples:
- Head-based sampling decides at the start of a request, for example keep 1 percent, and propagates the decision so the whole trace is kept or dropped together. Cheap and simple, but it throws away most of the interesting slow and failed requests along with the boring ones.
- Tail-based sampling buffers all spans briefly and decides after the request finishes: keep every error, every request slower than a threshold, and a small share of normal ones. Much better traces, at the cost of a collector that holds spans in memory.
OpenTelemetry
OpenTelemetry is the vendor-neutral standard for instrumentation: APIs and SDKs for many languages, automatic instrumentation for common frameworks and clients, the propagation formats, and a collector that receives, samples and exports telemetry to whatever backend you use. Instrumenting with it once avoids rewriting instrumentation when you change vendors.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| No tracing | Monoliths and very simple systems | Debugging latency across services by guesswork |
| Head-based sampling | Low overhead, simple setup | Most rare errors and slow requests are not captured |
| Tail-based sampling | Keeping all errors and slow requests with modest storage | A stateful collector tier and added complexity |
| Tracing everything | Low-traffic or critical paths where every request matters | Cost and overhead at scale |
Common mistakes
- Not propagating context through queues, thread pools or async callbacks, breaking traces into fragments.
- Optimizing the slowest span when it is not on the critical path.
- Head-based sampling at 1 percent and then finding no trace of the rare failure you are debugging.
- Putting sensitive data such as tokens or personal data into span attributes.
- Instrumenting only some services, leaving unexplained gaps in the waterfall.
In the wild
OpenTelemetry
Observability primer (opens in a new tab)The OpenTelemetry primer defines observability as being able to understand a system from the outside by asking questions about it without knowing its inner workings, including "unknown unknowns". It describes telemetry as traces, metrics and logs; a span as a single unit of work with a name, timing, structured log messages and attributes; and a distributed trace as the path of one request through multiple services, made of spans where the root span represents the request from start to finish. It also frames reliability in terms of SLIs measured from the user's perspective.
Check yourself
-
What links spans emitted by different services into one trace?
Show answer
B. A propagated trace context (trace id and parent span id), typically in a header such as traceparent Each service reads the incoming context and attaches its spans to the same trace and parent.
-
Cart (30 ms) and pricing (45 ms) run in parallel. You cut cart to 5 ms. How much faster is the request?
Show answer
B. Not at all; pricing still determines when both are done Only spans on the critical path affect total latency.
-
Why might head-based 1 percent sampling miss the bug you are hunting?
Show answer
B. It decides before the request runs, so 99 percent of rare slow or failed requests are discarded Tail-based sampling can keep all errors and slow requests because it decides afterwards.
-
A waterfall shows a staircase of 12 short, sequential database spans inside one service. What does it suggest?
Show answer
B. An N+1 query pattern that should be batched Many sequential small queries add up; one batched query removes most of the time.
Recap
- A trace is a tree of spans for one request; each span records an operation's timing and attributes.
- Context propagation (for example the traceparent header) stitches spans from many services together.
- Read the waterfall for the critical path; speeding up spans off it does not help.
- Head sampling is cheap but misses rare problems; tail sampling keeps errors and slow requests.
- OpenTelemetry is the standard way to instrument, propagate context and export traces.