LLM observability and tracing

The big idea

A user writes in: "Your assistant told me I cannot return my jacket." Your server logs say 200 OK, 3.7 s. That is all. Was the right policy page retrieved? Did the model ignore it? Did a tool fail and the model guess? Without a record of what actually went into and came out of each step, you cannot tell, and you cannot fix it.

Classic monitoring answers "is it up and how fast". LLM apps also need "what did it say and why", because their failures are mostly wrong content, not crashes. So you record, for every request, the exact prompt, the retrieved passages, each tool call and result, the output, and the tokens, latency and cost of every step, linked together so you can replay the whole story.

It is a flight recorder. Planes rarely crash, but when something odd happens, investigators do not guess: they read the recorder, which captured every control input and instrument reading in order. A trace is the flight recorder of one request.

How it works

Spans and traces

A spanThe record of one unit of work, such as a model call or tool call: its name, start time, duration, parent and attributes like tokens and errors. is one step. Spans nest: the request is the root, and its children are the query rewrite, retrieval, each model call and each tool call. All spans of one request share a trace id, and together they form a traceThe tree of linked spans that records everything that happened while serving one request.. If the request crosses services, such as a separate retrieval service, the trace id travels in the request headers so the tree stays whole, exactly as in distributed tracing for microservices ( Distributed tracing ).

What to log on each span

For LLM apps, record content as well as numbers:

• Model calls: model name and version, parameters (temperature, max tokens), the full input messages including the system prompt and tool definitions, the output, input and output token counts, cached tokens, latency and time to first token, and the stop reason.
• Retrieval: the query actually sent, the ids, scores and text of the passages returned.
• Tool calls: tool name, arguments, result or error, duration, and retries.
• The request: user or tenant id (pseudonymous), session id, prompt version, feature name, and any user feedback that arrives later.

The prompt version matters more than it looks. When quality drops, the first question is "what changed", and a trace that says prompt=refunds-v14 answers it ( LLMOps: shipping and iterating ).

Tokens and cost

Every model response reports its token usage. Multiply by the price for that model and you have the cost of the span; sum the spans and you have the cost of the request; group by feature or customer and you know where the money goes. Token counts also explain latency: a prompt that grew from 3,000 to 28,000 tokens is slower and costs roughly nine times as much in input tokens per call, and the trace shows exactly which retrieval step put the extra text there. Budgets built on these numbers are the subject of Latency and cost budgets .

Trace viewers

A trace viewerA tool that displays a trace as a timeline of nested spans, with each span's inputs, outputs and attributes available on click. shows a request as a waterfall: one bar per span, nested and placed on a time axis, with the prompt and output one click away. Open-source and hosted LLM observability tools, and general tracing systems, all provide one. Most also let you search traces (all requests where a tool errored, or where the output contained "I cannot"), compare two versions of a prompt on the same inputs, and attach a human label or a score to a trace.

From traces to evals

Traces are the raw material of evaluation. Error analysis ( Error analysis ) means sampling traces, reading them, writing down what went wrong, and counting. The categories you find become eval cases and graders, and the eval loop from Why evals are the core skill starts. Teams that skip tracing end up writing evals for problems they imagine rather than the ones users actually hit.

Try it

1. Start with trace 1 (healthy). Which spans take most of the time? Which cost money?
2. Trace 2: the user says it was slow. Find the span responsible without opening the diagnosis. Would a better prompt help?
3. Trace 5: time and cost look normal. Click through the spans until you find where the wrong answer came from. What would a log of only latency and status codes have shown?

The same idea in code: a tracer with nested spans, a fake clock and a text waterfall.

# A minimal tracer: nested spans with time, tokens and cost, plus a text trace viewer.
PRICE = {"big": (3.00, 15.00), "small": (0.25, 1.25)}  # illustrative dollars per 1M tokens (in, out)
SPANS, STACK, CLOCK = [], [], [0]   # a fake clock (ms) keeps the output deterministic

class span:
    def __init__(self, name, **attrs):
        self.name, self.attrs = name, attrs
    def __enter__(self):
        self.depth, self.start = len(STACK), CLOCK[0]
        STACK.append(self); SPANS.append(self)
        return self
    def __exit__(self, *exc):
        self.end = CLOCK[0]; STACK.pop()

def llm(model, tokens_in, tokens_out, ms):
    with span(f"chat {model}", tokens_in=tokens_in, tokens_out=tokens_out) as s:
        CLOCK[0] += ms
        p_in, p_out = PRICE[model]
        s.attrs["cost"] = (tokens_in * p_in + tokens_out * p_out) / 1e6

def tool(name, ms, error=None):
    with span(f"execute_tool {name}", error=error):
        CLOCK[0] += ms

with span("handle_request", user="u_381"):      # one user message
    llm("small", 400, 20, 300)                     # rewrite the search query
    with span("retrieval help_center", hits=5):
        CLOCK[0] += 180
    llm("big", 3200, 40, 1100)                     # model asks for a tool
    tool("order_status", 2400, error="timeout")    # slow, failed call
    tool("order_status", 260)                      # retry succeeds
    llm("big", 3350, 180, 1900)                    # final answer

total = SPANS[0].end
for s in SPANS:                                    # the waterfall view
    a, b = int(40 * s.start / total), max(1, int(40 * (s.end - s.start) / total))
    bar = " " * a + "#" * b
    note = ", ".join(f"{k}={v:.5f}" if k == "cost" else f"{k}={v}" for k, v in s.attrs.items() if v)
    label = "  " * s.depth + s.name
    print(f"{label:<28}{s.end - s.start:>6} ms |{bar:<40}| {note}".rstrip())
cost = sum(s.attrs.get("cost", 0) for s in SPANS)
slow = max(SPANS[1:], key=lambda s: s.end - s.start)
print(f"total {total} ms, cost {cost:.4f} dollars, slowest step: {slow.name} ({slow.end - slow.start} ms)")

handle_request                6140 ms |########################################| user=u_381
  chat small                   300 ms |#                                       | tokens_in=400, tokens_out=20, cost=0.00013
  retrieval help_center        180 ms | #                                      | hits=5
  chat big                    1100 ms |   #######                              | tokens_in=3200, tokens_out=40, cost=0.01020
  execute_tool order_status   2400 ms |          ###############               | error=timeout
  execute_tool order_status    260 ms |                         #              |
  chat big                    1900 ms |                           ############ | tokens_in=3350, tokens_out=180, cost=0.01275
total 6140 ms, cost 0.0231 dollars, slowest step: execute_tool order_status (2400 ms)

The timed-out tool call is 39 percent of the request's time and costs nothing in tokens, which is why cost dashboards alone would never find it. The two big-model calls are almost all of the cost.

Going deeper

Use a standard schema

OpenTelemetry, the open standard for traces, metrics and logs, has semantic conventions for generative AI: standard span names such as chat {model} and execute_tool {tool name}, and attributes such as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Capturing message content (gen_ai.input.messages, gen_ai.output.messages) is opt-in, because it may contain sensitive data. Using the conventions lets you switch trace backends without re-instrumenting your code.

Privacy and retention

Prompts contain whatever users type: names, addresses, health details, secrets pasted by accident. Logging everything forever is a liability. Decide per field what you keep: redact obvious personal data and secrets before export, keep full content for a short window and only aggregates after that, restrict who can open raw traces, and honor deletion requests. Some teams keep content only for sampled traces, or only for internal users. These choices tie into PII handling ( Privacy and PII handling ).

Sampling and overhead

Tracing every span of every request is cheap compared with the model call itself, but storing full prompts at scale is not. Common practice: keep metadata (tokens, latency, errors) for 100 percent of requests, keep full content for a sample, and always keep full content for requests that errored, got negative feedback, or were flagged by an online check. Export spans asynchronously so a slow trace backend never slows a user request.

Trade-offs

Common mistakes

• Logging only the user message and final answer, not the full prompt with retrieved context and tool results.
• Not recording the prompt and model version, so a quality drop cannot be tied to a change.
• Tracking cost from the monthly invoice instead of per request, so nobody knows which feature is expensive.
• Storing raw prompts with personal data indefinitely and open to everyone on the team.
• Collecting traces but never reading them, so the data never turns into fixes or eval cases.

In the wild

Check yourself

Recap