Benchmarks and their limits

The big idea

Every new model arrives with a table of numbers: a knowledge test, a maths test, a coding test, a ranking on a leaderboard. These benchmarksA public, fixed set of test questions with automatic scoring, used to compare models on the same task. are useful. They let you compare models on the same questions, and they track how fast the field moves. But a benchmark answers "how good is this model at these questions?", and your question is "how good is it at my product's job?"

Think of a standardised exam for school leavers. It is a fair way to compare thousands of students on the same material. It says little about whether one particular student will be good at the job you are hiring for. And if the questions leaked before the exam, the scores stop meaning anything at all.

How it works

What public benchmarks measure

Most benchmarks are a fixed set of questions with an automatic check. A few you will see often:

• Knowledge: MMLU asks multiple-choice questions across 57 subjects, from elementary mathematics to law, to test broad world knowledge.
• Maths: GSM8K has 8,500 grade school word problems that need several steps of reasoning.
• Code: HumanEval asks the model to write Python functions from docstrings and runs unit tests to check functional correctness.
• Agents: SWE-bench ( Coding agents and computer-use agents ) and tau-bench ( Making agents reliable ) check whether an agent resolves real issues or completes customer-service tasks.
• Human preference: Chatbot Arena shows people two anonymous answers and asks which they prefer, turning many votes into a ranking.

Each one measures a narrow, well-defined skill. That is what makes them comparable, and also what limits them.

Why the numbers mislead

Contamination. Benchmark questions are public, so copies end up on forums, blogs and code repositories, and those end up in training data. A model that has seen the test can score well by memory. This is called contaminationWhen test questions or close copies of them appear in a model's training data, inflating its score.. One study rewrote a fresh set of grade school maths problems matching GSM8K's style and difficulty, and found accuracy drops of up to 8 percent on the new set, with signs of systematic overfitting in some model families.

Saturation. When top models all score near the maximum, a benchmark stops telling them apart, and differences of a point or two are mostly noise. The field then moves to harder benchmarks, which go through the same cycle.

Errors in the benchmark. Benchmarks are built by people and contain mistakes. A re-annotation of MMLU estimated that 6.49 percent of its questions contain errors, and in one subject, virology, more than half of the questions analysed had problems.

Selection and setup. If a provider tests many private variants and publishes only the best score, the published number is the luckiest draw, not the typical one. Prompt format, the number of examples in the prompt and sampling settings also move scores, so numbers are only comparable when measured the same way.

Goodhart's law, in its popular form: "When a measure becomes a target, it ceases to be a good measure." Once a benchmark is what everyone optimises for, its score drifts away from the ability it was meant to reflect.

Why your own evals matter more

Even a perfectly measured benchmark measures someone else's task. Your support bot does not need to pass a law exam; it needs to read your policy documents, follow your tone guide and call your tools correctly. Two models that tie on a public leaderboard can differ widely on your traffic, and the cheaper or faster one may be just as good for your job.

So use benchmarks to build a shortlist, then decide with your own eval set ( Building an eval set ), measuring cost and latency alongside quality.

Try it

1. With 1 variant and no leakage, compare 100 questions with 10,000. How far can an honest score land from 70 percent?
2. At 100 questions, raise the variants to 30. How much does the published score rise, though no variant is any better?
3. Add 10 percent leakage. Which points would disappear on a fresh set of similar questions?

A simple way to check for contamination: count how many of a test question's 8-word sequences already appear in training text.

# A simple contamination check: how many of each test question's 8-word
# sequences already appear in the training text?
import re

TRAINING_TEXT = """
Forum post: Priya sold 48 cups of lemonade on Saturday, and then she sold
half as many cups on Sunday. How many cups did Priya sell over the whole
weekend? Answer: 72. Also, here is my favourite cookie recipe with oats and honey.
A train leaves the station at 3pm travelling at 60 miles per hour toward the coast.
"""

TEST_QUESTIONS = [
    "Priya sold 48 cups of lemonade on Saturday, and then she sold half as many cups on Sunday. How many cups did Priya sell over the whole weekend?",
    "A train leaves the station at 3pm travelling at 60 miles per hour. A second train leaves at 4pm at 80 miles per hour. When does it catch up?",
    "Omar bakes 3 trays of 12 cookies and gives away a quarter of them. How many are left?",
]

def ngrams(text, n=8):
    words = re.findall(r"[a-z0-9]+", text.lower())
    return {tuple(words[i:i + n]) for i in range(len(words) - n + 1)}

seen = ngrams(TRAINING_TEXT)
for q in TEST_QUESTIONS:
    grams = ngrams(q)
    overlap = len(grams & seen) / len(grams)
    flag = "LEAKED" if overlap > 0.5 else "partial" if overlap > 0 else "clean"
    print(f"{overlap:5.0%} of 8-grams seen  {flag:<8} {q[:48]}...")

 100% of 8-grams seen  LEAKED   Priya sold 48 cups of lemonade on Saturday, and ...
  27% of 8-grams seen  partial  A train leaves the station at 3pm travelling at ...
   0% of 8-grams seen  clean    Omar bakes 3 trays of 12 cookies and gives away ...

The first question appears word for word in the training text, so a correct answer proves nothing. The second shares some phrasing but asks something new. Real checks run the same idea over billions of documents, and they still miss paraphrased or translated copies, which is why fresh, private test sets remain the strongest defence.

Trade-offs

Common mistakes

• Picking the model at the top of a leaderboard without testing it on your own traffic.
• Treating a 1 or 2 point difference on a saturated benchmark as meaningful.
• Comparing scores measured with different prompts, settings or benchmark versions.
• Publishing your own eval questions, so future models train on them.
• Ignoring cost and latency because the benchmark only reports accuracy.

In the wild

Check yourself

Recap