Building an eval set

The big idea

An eval score is only as meaningful as the cases behind it. If your support bot's eval set has 90 questions about order status and none in Spanish, a 95 percent score tells you nothing about the Spanish speakers who make up part of your traffic. The set is a sample of the world your system will face, and every gap in the sample is a place where you are flying blind.

Think of a driving test. A good test route is not the straight road outside the test centre. It includes a busy junction, a roundabout, parallel parking and, ideally, some rain, because those are where drivers fail. It still mostly looks like normal driving, because that is what drivers do all day. An eval set needs the same mix: mostly real, everyday traffic, plus deliberate visits to the places where systems break.

How it works

1. Start from real traffic

Real user inputs beat anything you imagine. Users phrase things in ways you never would, paste in strange formatting, ask two questions at once and misspell product names. Pull a sample of production tracesThe full record of one request or session: the user input, any retrieved context and tool calls, and the final output., with personal data removed ( Privacy and PII handling ).

How you sample matters. A random sample mirrors traffic, so its overall score is honest, but rare topics barely appear. Stratified samplingSampling separately within each group, such as each topic or customer type, so every group gets enough cases, instead of sampling the whole population at once. fixes that: group traces by topic, language or customer type and take a minimum number from each group. Other useful picks are traces with negative feedback, unusually long or slow ones, and ones a classifier flags as likely failures. Each of those signals has blind spots, so keep some purely random traces in every batch.

Before your product has users, write the first cases yourself and ask colleagues to use the system the way customers would.

2. Add known failures and edge cases

Every bug report, complaint and failure you find in error analysis ( Error analysis ) becomes a case, so it can never silently come back. Then add edge casesUnusual inputs at the boundaries of normal use, where systems are most likely to break. on purpose. Anthropic's evaluation guide suggests irrelevant or missing input, overly long input, poor, harmful or irrelevant user messages, and ambiguous cases where even humans would disagree. For agents and RAG, add questions the documents cannot answer, conflicting sources and attempts at prompt injection ( Prompt injection and data exfiltration ).

3. Fill gaps with synthetic cases

When real data is thin, for a new feature or a rare failure, generate cases with a model. The naive prompt, "give me 50 test questions", produces generic, repetitive questions. A structured approach works much better:

1. Define dimensions: independent ways inputs vary, such as issue type, customer mood, new issue or follow-up, writing style.
2. Write about 20 tuples by hand, each picking one value per dimension: (refund status, angry, follow-up, typos). This forces you to think about the problem space.
3. Generate more tuples, either as the full cross product filtered for realism, which guarantees coverage of rare combinations, or by asking a model for realistic tuples.
4. In a separate prompt, turn each tuple into a natural message. Keeping the two steps apart avoids every case sounding the same.

Synthetic data has limits. It cannot tell you how common a failure is, and it is often unrealistic for low-resource languages, high-stakes domains such as medicine or law, and under-represented user groups. Replace it with real examples as soon as you have them.

4. Label every case

A case without a way to grade it is just an input. Each case needs one of:

• a reference answer for exact or near-exact checks (the right SQL result, the right category), or
• a rubricA written list of criteria that defines what a passing output must and must not do, used by human or model graders.: specific criteria the output must meet.

Good rubrics are specific and binary. "Is the answer helpful? Rate 1 to 5" invites every grader to invent their own scale, and people drift toward the middle. "Mentions the order number. States the 30-day refund window. Does not promise a refund for orders over 30 days old" can be checked the same way by anyone, and each line is its own pass or fail. Rubrics should come from someone who knows the domain, and two people labelling the same 20 cases will quickly show you which criteria are vague.

Try it

1. With a random sample of 100, which topics are blind spots? Push the size to 400. Is non-English covered yet?
2. Switch to stratified at 100. What did you give up on order status, and what did you gain?
3. With stratified or top-up selection, why does the text warn you about reporting one overall number?

Generating synthetic cases from dimensions, with a template standing in for the model that writes the final messages:

# Synthetic eval cases from dimensions: cross product, filter, cover, phrase.
import itertools, random

DIMS = {
    "issue":   ["late delivery", "damaged item", "refund status", "change address"],
    "mood":    ["calm", "angry"],
    "context": ["new issue", "follow-up"],
    "style":   ["plain", "typos", "very long"],
}

def realistic(t):   # stands in for an LLM or human filter
    # nobody follows up on an address change they have not made yet
    return not (t["issue"] == "change address" and t["context"] == "follow-up")

tuples = [dict(zip(DIMS, v)) for v in itertools.product(*DIMS.values())]
valid = [t for t in tuples if realistic(t)]
print(f"{len(tuples)} combinations, {len(valid)} realistic")

# Pick a small set that still uses every value of every dimension at least once.
rng = random.Random(3)
rng.shuffle(valid)
picked, unseen = [], {(d, v) for d in DIMS for v in DIMS[d]}
for t in valid:
    gain = {(d, t[d]) for d in DIMS} & unseen
    if gain:
        picked.append(t)
        unseen -= gain
print(f"{len(picked)} cases cover all {sum(map(len, DIMS.values()))} values\n")

OPEN = {"calm": "Hi.", "angry": "This is ridiculous."}
ASK = {"late delivery": "My order 5521 is late", "damaged item": "Order 1182 arrived broken",
       "refund status": "Where is my refund for order 2240", "change address": "I need to change the address on order 3307"}
for t in picked:   # a second LLM prompt would phrase these naturally
    q = f'{OPEN[t["mood"]]} {ASK[t["issue"]]}.'
    if t["context"] == "follow-up": q += " I already asked yesterday."
    if t["style"] == "typos": q = q.replace("rder", "rdr").replace("the", "teh")
    if t["style"] == "very long": q += " (plus 300 words about their week)"
    print(" | ".join(t.values()))
    print("   ->", q)

48 combinations, 42 realistic
6 cases cover all 11 values

damaged item | calm | new issue | typos
   -> Hi. Ordr 1182 arrived broken.
refund status | angry | new issue | typos
   -> This is ridiculous. Where is my refund for ordr 2240.
late delivery | calm | new issue | typos
   -> Hi. My ordr 5521 is late.
refund status | angry | follow-up | plain
   -> This is ridiculous. Where is my refund for order 2240. I already asked yesterday.
late delivery | calm | follow-up | very long
   -> Hi. My order 5521 is late. I already asked yesterday. (plus 300 words about their week)
change address | calm | new issue | very long
   -> Hi. I need to change the address on order 3307. (plus 300 words about their week)

Four dimensions give 48 combinations, and a realism filter drops 6. Six cases are enough to use every value of every dimension at least once, which is the point: a small set that still touches every kind of variation, instead of 50 variations of the same polite question.

Going deeper

Weighting a stratified set

A stratified set over-represents rare topics on purpose, so its plain average is not your production pass rate. Weight each topic by its share of traffic:

Say order status is 90 percent of traffic and passes 90 percent of cases, and refunds are 10 percent and pass 50 percent. With 30 cases each, the plain average is 70 percent, but real users see $0.9\times 0.9+0.1\times 0.5=0.86$, or 86 percent. Report the weighted number for "how are we doing" and the per-topic numbers for "what do we fix".

Development and held-out sets

If you tune prompts against the same cases over and over, you will eventually fit the quirks of those cases, the same overfitting you meet in classical ML ( Train, validation and test splits ). Keep a held-out slice you do not look at while iterating, and use it to confirm that an improvement is real before shipping.

How big, and how to keep it healthy

Start with what you can label well: fifty real cases with clear criteria beat a thousand vague ones. Practitioners report purpose-built sets often grow past 100 cases as every important workflow and known failure gets represented. Grow the set every time you find a new failure, and keep it cheap enough to run on every change. Version it like code, record which version produced each score, remove duplicates, and review it when the product changes, because cases about features that no longer exist only add noise.

Trade-offs

Common mistakes

• Writing every case yourself from imagination, so the set tests how you talk, not how users talk.
• Asking a model for "50 test questions" with no dimensions, and getting 50 versions of the same polite question.
• Grading with vague 1 to 5 scales instead of specific pass or fail criteria.
• Averaging a stratified set as if it were a random sample of traffic.
• Tuning on every case you have, with no held-out slice to check you have not overfit.

In the wild

Check yourself

Recap