Online evaluation and monitoring

The big idea

Your offline eval set passed, so you ship the new prompt. A week later, support tickets mention answers that "sound confident but skip my question". None of your test cases looked like that, because your users started asking things you never collected. Offline evals test the questions you thought of; production is where you meet the ones you did not.

Online evaluationMeasuring an AI system's quality on live traffic, using user feedback, behaviour signals and automatic grading of sampled outputs. closes that gap. You watch three things on real traffic: what users tell you, what users do, and what your graders say about a sample of outputs. You compare versions with controlled experiments, and you watch the inputs themselves for change.

Think of a restaurant. Tasting in the kitchen (offline evals) catches most problems before a dish goes out. But you also watch what comes back on the plates, read the comment cards, and notice when the neighbourhood starts ordering dishes that are not on the menu.

How it works

User feedback signals

Explicit feedbackFeedback a user gives on purpose, such as a thumbs up or down, a star rating or a written comment. is the obvious one: a thumbs up or down, a rating, a comment box. It is cheap to collect and easy to read, but it is sparse (most users never click), skewed towards people who are very pleased or very annoyed, and it rewards what users like in the moment. People tend to like answers that agree with them, which is not the same as answers that are right.

Implicit signalsFeedback inferred from what users do, such as accepting a suggestion, retrying, editing the output or asking for a human. come from behaviour, so you get them for every session:

• Acceptance: the user copies the answer, accepts the code suggestion or sends the drafted email.
• Rework: the user regenerates, rephrases the same question or heavily edits the output.
• Abandonment and escalation: the session ends without an answer, or the user asks for a human.
• Task outcome: the ticket is closed, the order goes through, the query returns rows.

None of these is a quality score on its own. A user may copy a wrong answer, or leave because the first answer was perfect. Use them as trends: a sudden rise in regenerations after a release is a strong hint, even if no single session proves anything.

Sampled grading in production

You cannot read every trace, but you can grade a sample. Run your cheap code checks on everything (valid format, no leaked internal IDs, tool calls that succeed) and your validated judges ( LLM-as-judge ) on a random sample, for example a few hundred traces a day per version. Track guardrail metrics alongside quality: latency, cost per request, error rate and refusal rate. A change that improves answers but doubles latency may still be a loss.

A/B tests for prompts and models

To know whether prompt B is better than prompt A, randomly assign users to one or the other and compare a metric chosen in advance. This is an A/B testA controlled experiment that randomly splits live traffic between two versions and compares a pre-chosen metric, so the difference can be attributed to the change.. Randomisation is what makes it work: both groups face the same mix of users, topics and days, so a difference in outcome is caused by the change, up to chance.

The practical rules:

• Randomise by user, not by request, so each person gets a consistent experience and one heavy user cannot land in both groups.
• Pick one primary metric and a few guardrails before you start, such as task success as primary, and latency, cost and complaint rate as guardrails.
• Decide the sample size up front and run until you reach it. Small effects need far more users than intuition suggests (see Going deeper).
• Ramp up gradually. A safer path is to run the new version in shadow mode first (it answers a copy of the traffic, users never see its output, you grade it offline), then a small canary, then the full test.

Drift in inputs

Your eval set is a snapshot of past traffic. Over time the inputs change: a new product launches, a policy changes, a marketing campaign brings a new kind of user, or people start pasting longer documents. This is input driftA change over time in the kind of inputs a system receives, so the data it was evaluated on no longer represents real use., and a system tuned on last quarter's questions can quietly degrade on this quarter's.

Watch the input distribution, not just the output quality. Classify incoming requests into topics (with keywords, clustered embeddings ( From symbols to vectors ), or a small classifier), track each topic's share week by week, and pay special attention to the share that fits no known topic. When something new grows, read those traces, run error analysis on them ( Error analysis ) and add representative cases to your eval set.

There is a second kind of drift if you call a hosted model through a name that the provider can update: the model itself may change under you. Pin model versions where the provider allows it, and re-run your eval set whenever a version changes.

Try it

1. Keep the defaults (B is truly 2 points better, 500 users per day, 21 days). The band clears zero long before the planned size marker. Compare the early estimates with the true 2 points. Why would stopping early overstate the gain?
2. Set the true effect to 0 and press "Rerun with new users" several times. How often do some daily dots look significant even though B is no better?
3. Set the true effect to plus 1 point. How many days would you need at 500 users per day, and at 10,000?

A simple drift check over two weeks of messages to a support assistant:

# Input drift check: bucket each week's user messages into known topics,
# compare topic shares, and watch the bucket nothing was designed for.
from collections import Counter

TOPICS = {
    "billing": ["invoice", "refund", "charge", "payment"],
    "shipping": ["delivery", "parcel", "tracking", "shipped"],
    "account": ["password", "login", "email", "account"],
}

def topic(msg):
    words = msg.lower().split()
    for name, keys in TOPICS.items():
        if any(k in words for k in keys):
            return name
    return "unknown"

week_1 = (["where is my parcel"] * 30 + ["refund for a double charge"] * 40
          + ["reset my password"] * 25 + ["do you sell gift cards"] * 5)
week_4 = (["where is my parcel"] * 28 + ["refund for a double charge"] * 30
          + ["reset my password"] * 20 + ["cancel my subscription now"] * 14
          + ["subscription renewal price went up"] * 8)

def shares(msgs):
    counts = Counter(topic(m) for m in msgs)
    return {t: counts[t] / len(msgs) for t in [*TOPICS, "unknown"]}

before, after = shares(week_1), shares(week_4)
print(f"{'topic':<10}{'week 1':>8}{'week 4':>8}{'change':>9}")
for t in before:
    flag = "  <- investigate" if after[t] - before[t] > 0.05 else ""
    print(f"{t:<10}{before[t]:>8.0%}{after[t]:>8.0%}{after[t] - before[t]:>+9.0%}{flag}")

unknown = [m for m in week_4 if topic(m) == "unknown"]
print("\nmost common unknown messages:")
for msg, n in Counter(unknown).most_common(3):
    print(f"  {n:>3}  {msg}")

topic       week 1  week 4   change
billing        40%     30%     -10%
shipping       30%     28%      -2%
account        25%     20%      -5%
unknown         5%     22%     +17%  <- investigate

most common unknown messages:
   14  cancel my subscription now
    8  subscription renewal price went up

The known topics all shrank a little, while messages that fit no topic grew from 5 to 22 percent. Reading them shows why: subscription cancellations, a request type the assistant was never designed or evaluated for. That is the cue to run error analysis on those traces and add subscription cases to the eval set.

Going deeper

How many users does an A/B test need?

To detect a difference $\delta$ between success rates $p_A$ and $p_B$, with a 5 percent false positive rate and 80 percent power (the chance of detecting a real effect of that size), each version needs roughly

users. With $p_A=0.60$ and $p_B=0.62$: the numerator is $7.84\times (0.24+0.2356)\approx 3.73$, and ${\delta }^2=0.0004$, so $n\approx 9,300$ users per version. Halving the effect to 1 point multiplies the sample by four, to about 37,500. Small improvements to a prompt are real but expensive to prove.

Do not stop at the first good-looking day

If you check the result every day and stop as soon as it looks significant, you raise your false positive rate well above 5 percent, because you give chance many tries. In the lab's simulation with no real effect, 500 users per version per day and 14 days, about 5 percent of runs end significant on day 14, but about 22 percent look significant on at least one daily check. Fix the sample size in advance, or use a sequential testing method designed for repeated looks.

Other traps

• Novelty effects: users may react to a change because it is new. Run long enough to see the effect settle.
• Too many metrics: compare 20 metrics and one will look significant by chance. Name the primary metric before the test.
• Mismatched units: if you randomise by user but compute the metric per request, heavy users dominate and the confidence interval is too narrow.

Trade-offs

Common mistakes

• Treating the thumbs-up rate as the quality metric and optimising for it alone.
• Stopping an A/B test the first day it looks significant, or running it with a few hundred users to detect a 1 point change.
• Randomising per request, so the same user bounces between versions.
• Monitoring output quality but never looking at how the inputs are changing.
• Calling a model name the provider can update without pinning a version or re-running evals when it changes.

In the wild

Check yourself

Recap