Error analysis

The big idea

Your assistant's answers are "not great", and the team has five theories: the model is too small, the prompt is too long, temperature is too high, users ask bad questions, the documents are outdated. Everyone argues for their favourite. Nobody has read a hundred real conversations and counted.

Error analysisReading real outputs and traces, noting what went wrong in each, grouping the notes into failure categories and counting them to decide what to fix. settles the argument with data. You read traces, write down what went wrong in plain words, group those notes into categories, and count. The counts tell you which problem is most common, and very often the biggest one is not in the prompt at all, but in retrieval, a tool or the data.

A teacher marking 40 essays works the same way. After a dozen, patterns appear: most lost marks come not from grammar but from not answering the question. The fix is a lesson on reading the question, and the teacher only knows that because they read the essays and kept a tally.

How it works

1. Gather traces worth reading

Start with roughly 100 diverse traces: real traffic sampled across topics, plus any you already know failed ( Building an eval set ). If you have no users yet, generate synthetic inputs and run them through the full system. Use a viewer that shows everything on one screen: the user's message, retrieved documents, tool calls and the final answer. Clicking between three tools to understand one trace means you will read ten traces instead of a hundred.

2. Open coding: notes, not categories

Read each trace and write a short note on what went wrong, in your own words. This is open codingWriting free-form notes about each example without a predefined list of categories; a method borrowed from qualitative research.. Be specific: "quoted the 2023 refund window, the current policy is 30 days" is useful; "bad answer" is not. Mark traces that were fine as fine.

Two rules make this go faster. Note the first upstream failure: if retrieval fetched the wrong document and the answer then invented a policy, the root cause is retrieval, and the invention is a symptom. And let one person who knows the domain make the call on quality, rather than a committee that has to agree on every trace.

3. Axial coding: build a failure taxonomy

After 30 to 50 notes, patterns appear. Axial codingGrouping open-coding notes into a small set of named categories, so they can be counted. groups similar notes into categories, such as wrong document retrieved, wrong tool arguments, invented facts, wrong tone, missed handoff to a human. A model can propose a first grouping of your notes, but review it yourself: the categories should be specific enough that you know what to do about each one.

4. Count, then decide

Count traces per category and sort. Failures usually follow a long tail: a few categories cover most of the problems. Priority is frequency, adjusted for severity and cost of the fix: a rare failure that leaks private data outranks a common one about tone.

Then decide what each category needs. Many failures are cheap fixes: the prompt never said answers should be short, or a tool description was ambiguous. Fix those directly. Build automated evals for failures that persist after the obvious fixes, starting with code checks, and keep model judges ( LLM-as-judge ) for subjective failures you will iterate on many times.

5. Repeat until nothing new appears

Keep reading fresh traces until new ones stop revealing new kinds of failure, which qualitative researchers call theoretical saturationThe point in reviewing examples where new ones no longer reveal new categories or change existing ones.. Repeat the exercise after significant changes: a new feature, a model switch, a large prompt rewrite.

Try it

Here are eight open-coding notes from a travel booking assistant. Do the axial coding: put each note in the category that names its first upstream failure.

1. Which note describes a wrong answer that is tempting to file as "invented facts", but really started earlier in the pipeline?
2. For each category, what is the cheapest fix you could try before writing an eval?

Counting categories over 40 annotated traces, once counting every failure and once counting only the first upstream failure:

# Error analysis: count failure categories (axial codes) in 40 annotated traces.
from collections import Counter

# Each trace lists its failures in the order they happened; "" means it was fine.
# The codes came from grouping open-coding notes such as
# "searched for the 2023 refund policy, not the current one" -> retrieval.
TRACES = (["retrieval,hallucination"] * 6 + ["retrieval,hallucination,tone"]
          + ["hallucination"] * 2 + ["tool_args"] * 4 + ["tone"] * 3
          + ["missed_handoff"] * 2 + [""] * 22)

failing = [t.split(",") for t in TRACES if t]
print(f"{len(TRACES)} traces read, {len(failing)} with at least one failure\n")

def pareto(title, counter):
    print(title)
    cum = 0
    for code, n in counter.most_common():
        cum += n
        print(f"  {code:<15}{n:>3}  {n / len(failing):5.0%} of failing traces"
              f"  (cumulative {cum / sum(counter.values()):4.0%})")
    print()

pareto("Every failure counted:", Counter(c for t in failing for c in t))
pareto("First upstream failure only:", Counter(t[0] for t in failing))

40 traces read, 18 with at least one failure

Every failure counted:
  hallucination    9    50% of failing traces  (cumulative  35%)
  retrieval        7    39% of failing traces  (cumulative  62%)
  tone             4    22% of failing traces  (cumulative  77%)
  tool_args        4    22% of failing traces  (cumulative  92%)
  missed_handoff   2    11% of failing traces  (cumulative 100%)

First upstream failure only:
  retrieval        7    39% of failing traces  (cumulative  39%)
  tool_args        4    22% of failing traces  (cumulative  61%)
  tone             3    17% of failing traces  (cumulative  78%)
  hallucination    2    11% of failing traces  (cumulative  89%)
  missed_handoff   2    11% of failing traces  (cumulative 100%)

Counted naively, invented facts ("hallucination") look like the top problem, and the team would spend a week on prompt wording. Counted by first upstream failure, retrieval is the root cause in 7 of the 18 failing traces, and most of the invented answers disappear once the right document is found.

Going deeper

Where in the pipeline?

For multi-step systems, the most useful question per trace is where it first went wrong. In RAG, check whether the right document was retrieved before judging the answer: if it was not, generation never had a chance ( Evaluating RAG ). For agents, read the user-visible outcome first, then walk back through tool calls to the first bad step, since errors compound downstream ( Making agents reliable ).

Why not start with generic metrics?

It is tempting to skip reading and adopt off-the-shelf scores for "helpfulness" or text similarity. Those measure something, but not necessarily your failures. Error analysis tells you what your system actually gets wrong, and those categories become the specific, binary evals worth automating. A metric invented before you read the data often measures a problem you do not have.

Making it a habit

Error analysis is not a one-off audit. Teams that improve fastest make it routine: a short session reading fresh traces every week or two, a shared spreadsheet of notes and categories, and a chart of each category's rate going down after its fix. That chart is also the most convincing way to show progress to people who do not read traces.

Trade-offs

Common mistakes

• Debating causes in meetings instead of reading 100 traces and counting.
• Writing category names before reading anything, so the notes get forced into the wrong boxes.
• Vague notes like "bad answer" that cannot be grouped or acted on.
• Counting downstream symptoms as separate problems, so a retrieval bug looks like a prompt problem.
• Building an LLM judge for a failure that one sentence in the prompt would fix.

In the wild

Check yourself

Recap