LLM-as-judge

The big idea

Some checks are easy for code: valid JSON, an order number present, no internal IDs. Others are not: did the answer actually address the question, is every claim supported by the retrieved documents, is the tone right for an upset customer? Humans can judge those, but not 5,000 outputs after every prompt change.

An LLM judgeA language model prompted to grade another model's outputs against a rubric, either scoring one answer or choosing between two. fills the gap: a model prompted with the input, the output and a rubric, asked to return a verdict. It is fast, cheap compared with people, and explains its reasoning. It is also a model, with its own biases, so it is only useful once you have measured how often it agrees with the humans whose judgment it replaces.

Think of a new teaching assistant marking exams. You hand them the marking scheme, mark 50 scripts yourself, and compare. Where they disagree with you, you fix the scheme or coach them. Only when they agree on the scripts you checked do you let them mark the rest, and you still spot-check a few every week.

How it works

Writing the judge prompt

A judge prompt has the same parts as any prompt ( Prompting fundamentals ): the role, the material to grade, and above all a rubric. The rules that matter most:

• Judge one thing. One judge per failure category from your error analysis, such as "unsupported claims" or "ignores the user's actual question", each returning pass or fail. A single "overall quality, 1 to 10" judge mixes everything and is hard to validate.
• Be specific. Spell out what passes and what fails, with a short example of each. "Fail if any number in the answer does not appear in the retrieved documents" is gradable; "is the answer accurate?" is not.
• Reason first, then decide. Ask for a short explanation before the verdict. This tends to improve judgments on hard cases and gives you something to read when the judge is wrong.
• Give a reference when one exists. For questions with a correct answer, include it: grading against a reference is much more reliable than asking the judge to solve the problem itself.
• Ask for a fixed output format, such as a JSON object with reasoning and verdict, so your code can parse it ( Structured outputs ).

Single-answer vs pairwise

There are three common setups. In single-answer gradingGrading one output on its own against a rubric, returning a score or pass or fail. the judge looks at one output and returns a verdict. In pairwise comparisonShowing a judge two outputs for the same input and asking which is better, or whether they tie. it sees two outputs and picks the better one. Reference-guided grading adds a known good answer to either.

Pairwise suits "is the new prompt better than the old one?", because relative judgments are easier and more stable than absolute scores. But the number of pairs grows quickly with the number of versions, and a pairwise win does not tell you whether either answer was acceptable. Single-answer pass or fail scales to any number of versions and maps directly onto your failure categories, which is why it is the usual default for product evals.

Judge biases

The MT-Bench paper (see the case study below) documented several biases in model judges:

• Position bias: preferring an answer because of where it appears. With two similar answers, most judges they tested favoured the first position. The standard fix is to run both orders and only declare a winner when both agree, calling it a tie otherwise.
• Verbosity bias: preferring longer answers. In their "repetitive list" test, padding an answer with rephrased copies of its own points fooled judges into preferring it.
• Self-enhancement bias: possibly favouring outputs from the same model. Their data could not settle whether it exists, so test it on your own data rather than assume either way.
• Limited reasoning: judges grading maths or logic can repeat the very mistake they are grading. Giving the judge a reference answer cut the failure rate sharply in their tests.

Try it

1. Press "Judge that always says pass". What is its raw agreement with the humans? Why is agreement a misleading way to validate a judge?
2. Reset, set the true failure rate to 20 percent and note what the judge reports. Then go back to 5 percent with the same judge. Why does the reported rate now overshoot the truth?
3. Why can a judge with known error rates still give you an accurate failure rate, even though it gets individual cases wrong?

A judge with position bias, simulated, and the swap-both-orders fix:

# A pairwise judge with position bias, and the swap-both-orders fix.
import random

rng = random.Random(42)
# 300 pairs of answers with hidden quality: A better, B better, or about equal.
pairs = [rng.choice(["A", "B", "tie", "tie"]) for _ in range(300)]

def judge(truth_first_vs_second, bias=0.35):
    """Stand-in judge: 35% of the time it just picks whichever answer it saw first."""
    if rng.random() < bias:
        return "first"
    if truth_first_vs_second == "tie":
        return rng.choice(["first", "second"])   # cannot tell, guesses
    return truth_first_vs_second

def one_order(truth):            # A always shown first
    v = judge({"A": "first", "B": "second", "tie": "tie"}[truth])
    return "A" if v == "first" else "B"

def both_orders(truth):          # show A first, then B first; win only if consistent
    v1 = one_order(truth)
    v2 = judge({"A": "second", "B": "first", "tie": "tie"}[truth])
    v2 = "B" if v2 == "first" else "A"
    return v1 if v1 == v2 else "tie"

for name, method in [("one order", one_order), ("swap both orders", both_orders)]:
    votes = [method(t) for t in pairs]
    a, b, t = (votes.count(x) for x in ("A", "B", "tie"))
    wrong = sum(v != p for v, p in zip(votes, pairs))
    print(f"{name:<17} A wins {a:3}  B wins {b:3}  ties {t:3}  differ from truth {wrong:3}")
print(f"{'truth':<17} A better {pairs.count('A'):3}  B better {pairs.count('B'):3}  equal {pairs.count('tie'):3}")

one order         A wins 202  B wins  98  ties   0  differ from truth 162
swap both orders  A wins  90  B wins  81  ties 129  differ from truth 123
truth             A better  87  B better  81  equal 132

Shown in one order, the biased judge gives A 202 wins when A was truly better in only 87 pairs, and it never calls a tie. Running both orders and keeping only consistent verdicts brings A's wins down to 90 and turns most of the inconsistent cases into ties, which is the honest answer for answers of equal quality.

Going deeper

Validating a judge

Treat the judge like a classifier and measure it against labels from a trusted domain expert. Practitioners suggest about 100 to 200 labelled examples per failure mode, with plenty of both passes and failures, split into a small set of examples you may show in the prompt, a development set you tune the prompt against, and a test set you use once at the end. Report two numbers, not one:

• True positive rateOf the outputs a human labelled as failures, the share the judge also flags. Also called recall. (TPR): of the real failures, how many the judge catches.
• True negative rateOf the outputs a human labelled as good, the share the judge also passes. (TNR): of the good outputs, how many it passes.

Raw agreement hides problems when failures are rare. If 5 percent of outputs fail, a judge that always says "pass" agrees with humans 95 percent of the time and catches nothing.

Correcting the failure rate

A judge that is wrong on individual cases can still measure a rate accurately, if you know its TPR and TNR. If the judge flags a fraction $f$ of outputs, the true failure rate $\theta$ satisfies $f=\text{TPR}\theta +(1-\text{TNR})(1-\theta )$, so

With TPR 0.8, TNR 0.95 and the judge flagging 8.7 percent of outputs, $\theta =(0.087-0.05)/0.75\approx 0.049$: about 4.9 percent, even though the raw number was almost double. This only works when $\text{TPR}+\text{TNR}$ is well above 1; a judge near 1 is guessing.

Keeping judges honest over time

Judges drift when you change their model or prompt, and your product's outputs drift too. Re-check agreement on a fresh labelled sample regularly, keep the judge prompt and model version in the eval record, and read some of the judge's reasoning, especially on its failures. A judge is a small eval system of its own, and needs the same care.

Trade-offs

Common mistakes

• Trusting a judge's scores without ever comparing them with human labels.
• Validating with raw agreement when failures are rare, so a judge that catches nothing looks 95 percent accurate.
• One vague "rate the quality from 1 to 10" judge instead of specific pass or fail judges per failure category.
• Running pairwise comparisons in only one order.
• Tuning the judge prompt on the same examples you use to report its accuracy.

In the wild

Check yourself

Recap