Offline vs online evaluation and A/B tests

The big idea

Your new recommendation model beats the old one offline: higher accuracy on last month's clicks, cross-validated, no leakage. You ship it, and revenue does not move. Or it drops. What happened?

Offline evaluationEvaluation on a fixed, historical dataset, without showing the model's output to real users. answers "does the model predict the past well?" The business cares about "does showing this model's output to people change what they do?" Those are different questions. Users react to what you show them, the logged data was produced by the old system, and the metric you optimised (clicks) is only a proxy for what you want (satisfied customers who come back).

Online evaluationEvaluation with real users in production, usually by comparing randomly assigned groups that see different versions. closes the gap. The standard tool is the A/B testA randomized controlled experiment: users are randomly split into a control group that gets the current version and a treatment group that gets the new one, and their outcomes are compared., which is a clinical trial for software: flip a fair coin for each user, give heads the new model and tails the old one, and compare what they actually do. Because the coin decides, the only systematic difference between the groups is the model.

How it works

Step 1: Know why offline and online disagree

• Proxy metrics: offline you measure clicks or log loss; the business needs retention, revenue or satisfaction. A model can win the proxy and lose the goal, for example by promoting clickbait.
• Feedback loops: historical data only contains reactions to what the old system showed. Items it never showed have no clicks, so offline metrics are biased towards the old model's behaviour.
• System effects: latency, freshness of features and interactions with other components only show up live ( Feature stores and training-serving skew covers training-serving skew).

Offline evaluation is still essential. It is cheap, fast and safe, so it filters out bad ideas before they reach users. Think of it as the gate to an online test, not a replacement for one.

Step 2: Pick one primary metric and decide the size of change you care about

Before the test, write down the primary metricThe single metric that decides whether the treatment wins, chosen before the experiment starts. (say, conversion rate) and the minimum detectable effectThe smallest true effect worth detecting; it sets the required sample size.: the smallest lift that would justify shipping. Then compute how many users you need.

Two error rates drive the sample size. The significance level alpha (usually 5 percent) is the chance of calling a winner when there is no real difference. PowerThe probability that an experiment detects a true effect of a given size; 1 minus the false-negative rate. Usually set to 80 percent. (usually 80 percent) is the chance of detecting a real effect of the chosen size. For a conversion rate p and a treatment rate p2, the users needed per group are roughly

With alpha 5 percent and power 80 percent, the z values are 1.96 and 0.84, so the numerator constant is about 7.85. Worked example: p = 0.05 and a 10 percent relative lift, so p2 = 0.055. The variance term is 0.0475 + 0.0520 = 0.0995, the squared difference is 0.000025, so n is about 7.85 x 0.0995 / 0.000025, roughly 31,000 per group.

This is why small companies cannot detect 1 percent improvements: they do not have the traffic. Either test bolder changes or pick a more sensitive metric.

Step 3: Add guardrail metrics

A model that lifts clicks by recommending sensational content can quietly raise complaints or unsubscribes. Guardrail metricsMetrics that must not get significantly worse in an experiment, such as latency, errors, crashes or unsubscribes, even if the primary metric improves. catch that: latency, error rates, crash rates, complaint rates, revenue per user. The decision rule becomes "ship if the primary metric improves and no guardrail degrades significantly."

Step 4: Run for the planned duration, then decide once

Run until you reach the planned sample size and at least one or two full weekly cycles, because behaviour on Mondays differs from Saturdays. Then compute the result once. In the code below, a test with 31,000 users per group gives control 5.08 percent, treatment 5.53 percent, z = 2.47, p-value 0.0134 and a 95 percent interval for the difference of +0.09 to +0.80 percentage points. The interval excludes zero, so the lift is significant, and its width tells you how uncertain the size of the lift still is.

Step 5: Watch for novelty and other time effects

A novelty effectA temporary change in behaviour caused by something being new, which fades as users get used to it. inflates early results: users click a redesigned carousel because it is different, not better. The opposite, a primacy effectA temporary drop in a metric because users must adjust to a change, which recovers as they learn it., hurts early results when users must unlearn habits. Plot the daily lift: if it shrinks steadily, wait for it to settle, or compare only new users, who have no old habits.

Going deeper

Code: sample size, one test and the cost of peeking

import numpy as np
from scipy.stats import norm

# Sample size per group to detect a lift in a conversion rate (two-sided z-test)
def n_per_group(p, rel_lift, alpha=0.05, power=0.8):
    p2 = p * (1 + rel_lift)
    z = norm.ppf(1 - alpha / 2) + norm.ppf(power)
    return int(np.ceil(z ** 2 * (p * (1 - p) + p2 * (1 - p2)) / (p2 - p) ** 2))

print("baseline  lift   users per group")
for p, lift in [(0.05, 0.10), (0.05, 0.05), (0.05, 0.02), (0.20, 0.05)]:
    print(f"  {p:.2f}    {lift:4.0%}   {n_per_group(p, lift):>10,}")

# One finished test: 31,000 users per group
rng = np.random.default_rng(1)
n = 31_000
a = rng.random(n) < 0.050            # control converts at 5.0 percent
b = rng.random(n) < 0.055            # treatment truly converts at 5.5 percent
pa, pb = a.mean(), b.mean()
se = np.sqrt(pa * (1 - pa) / n + pb * (1 - pb) / n)
z = (pb - pa) / se
print(f"\ncontrol {pa:.4f}  treatment {pb:.4f}  lift {pb / pa - 1:+.1%}")
print(f"z = {z:.2f}, p-value = {2 * (1 - norm.cdf(abs(z))):.4f}")
print(f"95% interval for the difference: [{pb - pa - 1.96 * se:+.4f}, {pb - pa + 1.96 * se:+.4f}]")

# Peeking: 2,000 A/A tests (no real difference), checked after every 1,000 users per group
false_fixed, false_peek = 0, 0
for _ in range(2000):
    xa = rng.random(20_000) < 0.05
    xb = rng.random(20_000) < 0.05
    stopped = False
    for k in range(1000, 20_001, 1000):
        qa, qb = xa[:k].mean(), xb[:k].mean()
        s = np.sqrt((qa * (1 - qa) + qb * (1 - qb)) / k)
        if abs(qb - qa) / s > 1.96:
            stopped = True
            break
    false_peek += stopped
    qa, qb = xa.mean(), xb.mean()
    false_fixed += abs(qb - qa) / np.sqrt((qa * (1 - qa) + qb * (1 - qb)) / 20_000) > 1.96
print(f"\nA/A tests declared 'significant': fixed horizon {false_fixed / 2000:.1%}, "
      f"peeking 20 times {false_peek / 2000:.1%}")

baseline  lift   users per group
  0.05     10%       31,231
  0.05      5%      122,121
  0.05      2%      752,700
  0.20      5%       25,580

control 0.0508  treatment 0.0553  lift +8.8%
z = 2.47, p-value = 0.0134
95% interval for the difference: [+0.0009, +0.0080]

A/A tests declared 'significant': fixed horizon 5.1%, peeking 20 times 25.4%

The last block simulates 2,000 A/A testsAn experiment where both groups get the identical version; any 'significant' difference is a false positive. Used to validate the testing system., where nothing differs between groups. Checking once at the end flags 5.1 percent as significant, as alpha promises. Checking after every 1,000 users and stopping at the first significant result flags 25.4 percent.

If you must monitor continuously, use methods designed for it, such as sequential tests or always-valid p-values, which widen the threshold to pay for the extra looks.

Other online methods

• Interleaving (for ranking): merge results from both rankers into one list for the same user and see which ranker's items get clicked. It needs far fewer users than an A/B test, but only measures preference, not long-term outcomes.
• Multi-armed bandits shift traffic towards the better variant during the test ( Multi-armed bandits ). They lose less to the worse arm but give a weaker estimate of the effect size.
• Shadow mode and canaries: run the new model on live traffic without showing its output, or show it to 1 percent first, to catch crashes and latency before any statistics.

Trade-offs

Common mistakes

• Shipping on an offline win without an online test, then blaming "the market" when nothing improves.
• Peeking daily and stopping the moment the p-value dips under 0.05.
• Choosing the metric after seeing the results, or testing twenty metrics and reporting the one that moved.
• Running for three days and calling it, so weekly cycles and novelty effects decide the outcome.
• Randomizing by request instead of by user, so the same person sees both versions and the groups contaminate each other.

In the wild

Check yourself

Recap