Why evals are the core skill

The big idea

Users complain that your support assistant gives useless answers about refunds. You add a paragraph about the refund policy to the prompt, try three refund questions, and the answers look great. You ship. A week later, account questions start going wrong. Nobody tried an account question, because nobody was complaining about those.

That is a vibe checkJudging an AI system by trying a few examples by hand and deciding whether the outputs feel right.: useful for a first look, but it cannot tell you whether a change helped overall, because it tests a handful of cases you picked while thinking about one problem. An evalA set of test inputs plus a way to judge each output, run automatically so you get a score you can compare across versions. replaces the guess with a measurement: many test inputs, a clear way to grade each output, run automatically on every change.

Think of a restaurant changing a recipe. The chef tasting one spoonful knows whether the new sauce is good. They do not know whether the change made the soup too salty, which only shows up if someone tastes every dish on the menu. Evals are tasting the whole menu, every time.

How it works

What an eval is made of

Every eval has three parts:

• Cases: inputs that look like what real users send, including the awkward ones. A support bot's cases are real customer messages; a SQL generator's cases are real questions about your data.
• A graderThe code, model or person that decides whether one output passes, or how good it is. for each case: something that decides pass or fail. The cheapest graders are plain code: does the reply mention the order number, is the JSON valid, does it leak an internal ID. Harder qualities, like tone or faithfulness, need a model as judge ( LLM-as-judge ) or a person.
• A runner that feeds every case through your system and reports a score, overall and broken down by topic, so you can compare two versions side by side.

Why a handful of examples cannot tell you

Two things go wrong with vibe checks. First, they are too small. Model outputs vary from run to run, and a pass rate measured on a few cases is mostly noise. If your system truly passes 80 percent of cases, the 95 percent margin of error on $n$ cases is about

which is plus or minus 25 points on 10 cases, 8 points on 100 and 4 points on 400. A change that really gains 5 points is invisible on 10 cases.

Second, they test the wrong cases. You check the thing you were fixing, so you miss regressionsSomething that used to work and breaks after a change. elsewhere. Prompts are shared by every request: a sentence added for refunds is read on every account question too.

The eval-driven loop

With an eval set, development becomes a loop (Figure 1). Look at real outputs to find failures, add them as cases, change one thing, run everything, and ship only if the score goes up without breaking a topic that used to work. This is the same habit as running unit tests before merging code, applied to a system whose behaviour you cannot read off the source. Classical machine learning has the same split between offline test sets and online experiments ( Offline vs online evaluation and A/B tests ).

You do not need much to start. Twenty to fifty real cases and a few code graders in a script beat any amount of debating examples in a meeting. Building an eval set covers building the set properly.

Try it

1. Start as most teams do: only refunds, 3 cases. B passes everything and A fails everything. Would you ship B? What does the verdict say, and why?
2. Switch to every topic at 30 cases each, then 100. At which size does the problem with account questions show up? Is the overall average enough to see it?
3. At 10 cases per topic, press "Draw new test cases" a few times and watch the numbers move. Then show the true rates to see what you were trying to measure.

The same idea in a few lines of Python: seven cases, three code graders and two versions of a bot. The fake bots stand in for a model with prompt v1 and v2.

# A tiny eval: test cases, code graders, and two versions of a support bot.
import re

CASES = [  # (topic, user message, order id the reply must mention)
    ("refund",   "Refund for order 1182? It arrived broken.", "1182"),
    ("refund",   "I want my money back for order 2240.", "2240"),
    ("refund",   "Can I return order 2391 for a refund?", "2391"),
    ("account",  "Change the email on order 3307 please.", "3307"),
    ("account",  "Update the phone number for order 4410.", "4410"),
    ("shipping", "Where is order 5521?", "5521"),
    ("shipping", "Order 6650 has not arrived yet.", "6650"),
]
UUID = "3f2a9c1e-77b0-4d2e-9a51-0c6e1d2b8f40"     # internal ticket id

def bot_v1(topic, oid):   # stand-in for model + prompt v1 (msg unused)
    if topic == "refund":
        return f"Sorry about order {oid}. Please contact support."
    return f"Order {oid}: done. Anything else?"

def bot_v2(topic, oid):   # prompt v2 adds the refund policy and "cite the ticket"
    if topic == "refund":
        return f"Order {oid} can be refunded within 30 days. Started it."
    if topic == "account":
        return f"Order {oid}: updated (ticket {UUID})."
    return f"Order {oid}: done. Anything else?"

GRADERS = {
    "mentions_order": lambda t, oid, r: oid in r,
    "no_uuid": lambda t, oid, r: not re.search(r"[0-9a-f]{8}-[0-9a-f]{4}-", r),
    "states_policy": lambda t, oid, r: t != "refund" or "30 days" in r,
}

for name, bot in [("v1", bot_v1), ("v2", bot_v2)]:
    passed, fails = 0, []
    for topic, msg, oid in CASES:
        reply = bot(topic, oid)
        bad = [g for g, check in GRADERS.items() if not check(topic, oid, reply)]
        passed += not bad
        fails += [f"{topic}:{oid}:{g}" for g in bad]
    print(f"{name}: {passed}/{len(CASES)} passed. Failures: {', '.join(fails) or 'none'}")

v1: 4/7 passed. Failures: refund:1182:states_policy, refund:2240:states_policy, refund:2391:states_policy
v2: 5/7 passed. Failures: account:3307:no_uuid, account:4410:no_uuid

Version 2 wins overall, 5 of 7 against 4 of 7, and it fixes every refund case. It also leaks the internal ticket ID on every account reply, which a quick look at refund answers would never have caught. The no_uuid grader is the kind of cheap check that pays for itself many times over.

Trade-offs

Common mistakes

• Shipping a prompt change after trying the three examples that motivated it.
• Reporting one overall number, so a gain on one topic hides a regression on another.
• Building the eval set from imagined questions instead of real user inputs and real failures.
• Waiting for a perfect eval framework before writing the first twenty cases.
• Never adding new failures to the set, so the same bug can quietly come back.

In the wild

Check yourself

Recap