Hallucinations

The big idea

Ask a chatbot for the birthday of someone only slightly famous and it may answer at once with a specific date. Ask again and you may get a different date, just as confident. Neither is from a source; both are guesses that look like facts. That is a hallucinationA fluent, plausible statement produced by a language model that is false or not supported by its sources, usually delivered with the same confidence as true statements..

A language model is trained to produce likely text, not verified truth. For facts that appear often in its training data, likely and true are usually the same thing. For rare facts, such as a birthday mentioned once on one web page, there is no pattern to learn, and the most fluent continuation is a guess.

Think of a student in an exam where blank answers score zero and wrong answers also score zero. The rational strategy is to guess on every question. A model trained and graded the same way learns the same habit. If you want "I don't know" when it does not know, you have to make that answer possible, and worth more than a confident wrong one.

How it works

Why models confabulate

• Rare facts cannot be learned from patterns. Spelling and grammar follow patterns, so errors there disappear with scale. A fact that appears once in the training data has no pattern behind it. Researchers have shown that for such "arbitrary" facts, a base model's hallucination rate is at least roughly the share of facts that appeared only once: if 20 percent of birthday facts appear exactly once, expect at least about 20 percent wrong birthdays.
• Grading rewards guessing. Most benchmarks score accuracy only. A guess sometimes scores; "I don't know" never does. Models tuned to climb those leaderboards learn to guess.
• Pressure from the prompt. A question that assumes something false ("Why did the company close in 2019?") or asks for a quote, a citation or a number invites the model to produce one.
• Missing or ignored context. If retrieval returns nothing useful, or the answer sits in a part of a long context the model attends to poorly, the model falls back on what sounds right.

The fixes, in order of strength

1. Ground the answer. Give the model the relevant documents ( Why RAG? ) and tell it to answer only from them. This moves the job from recalling facts to reading them, which models do far more reliably.
2. Allow and encourage abstention. Say explicitly that "I don't know" or "the documents do not say" is an acceptable answer. Anthropic's guide to reducing hallucinations lists this first and says it can drastically reduce false information.
3. Ask for quotes and citations. For long documents, have the model extract word-for-word quotes first and reason from them. Then each claim carries evidence a person, or code, can check.
4. Verify. Check that cited quotes really exist in the source; ask the model to re-check each claim and retract any it cannot support; for high-stakes answers, compare several samples, since disagreement between them is a warning sign.
5. Use tools for what tools do better. Dates, arithmetic, prices and stock levels should come from a calculator, a database or an API call, not from the model's memory.

None of these drives the rate to zero. They lower it and, as important, make the remaining errors visible.

Try it

1. Keep "accuracy only" and move the threshold up from 0. The wrong answers drop, but what happens to the score? This is why accuracy-only leaderboards reward guessing.
2. Switch to "wrong = -3". Where is the best threshold now? Compare it with 3 / (1 + 3) = 0.75.
3. Set "wrong = -9", the kind of cost a wrong medical or legal answer carries. How many questions should the model now decline, and is that a failure or the right behaviour?

Measuring it in Python: grade answers as correct, wrong or abstained, then check citations:

# Measure hallucination the way short-answer factuality evals do: each answer is
# correct, incorrect, or not attempted. Then check that every cited quote really
# appears in the source, a cheap guard for grounded (RAG) answers.
# Answers and sources are made up for illustration.

gold = {"q1": "1889", "q2": "Canberra", "q3": "8", "q4": "Lisbon", "q5": "1969"}
model_a = {"q1": "1889", "q2": "Sydney", "q3": "8", "q4": "Porto", "q5": "1969"}
model_b = {"q1": "1889", "q2": "Canberra", "q3": None, "q4": None, "q5": "1969"}  # None = "I don't know"

def grade(answers):
    c = sum(answers[q] == gold[q] for q in gold)
    na = sum(answers[q] is None for q in gold)
    w = len(gold) - c - na
    return c, w, na

for name, ans in [("A (always answers)", model_a), ("B (abstains)", model_b)]:
    c, w, na = grade(ans)
    print(f"{name:19} correct {c}/5  wrong {w}/5  abstained {na}/5  "
          f"wrong when it answered {w / (c + w):.0%}")

source = ("The refund window is 30 days from delivery. Opened software "
          "cannot be returned. Refunds go back to the original payment method.")
claims = [
    ("You have 30 days to return it.", "The refund window is 30 days from delivery."),
    ("Refunds can be paid as store credit.", "Refunds go back as store credit."),
]
for claim, quote in claims:
    ok = quote in source
    print(f"{'supported  ' if ok else 'UNSUPPORTED'} {claim}")

A (always answers)  correct 3/5  wrong 2/5  abstained 0/5  wrong when it answered 40%
B (abstains)        correct 3/5  wrong 0/5  abstained 2/5  wrong when it answered 0%
supported   You have 30 days to return it.
UNSUPPORTED Refunds can be paid as store credit.

Both models have the same accuracy, 3 out of 5. Model A is wrong 40 percent of the time it answers; model B never is. An accuracy-only score cannot tell them apart. The quote check catches the second claim because its "quote" is not in the source at all.

Going deeper

Measuring hallucination rates

Report three numbers, not one: correct, incorrect and not attempted. OpenAI's SimpleQA benchmark grades short factual answers exactly this way. Then track the error rate on attempted answers alongside accuracy, so a model that guesses less is not punished for it. For long answers, split the text into small factual claims and check each against a trusted source; the FActScore method does this, and found that one widely used chat model's biographies had only 58 percent of their atomic facts supported. For your own product, build the eval from your own questions and documents ( Building an eval set ) and include questions the documents cannot answer, where the right response is to abstain.

Scoring that rewards honesty

Kalai and colleagues propose stating a confidence target in the instructions: answer only if you are more than t confident, because wrong answers cost t / (1 - t) points, right ones earn 1 and "I don't know" earns 0. With t = 0.75 a wrong answer costs 3. A model whose confidence is calibrated then maximizes its score by answering exactly when it is more than 75 percent sure, which is what the lab shows.

Trade-offs

Common mistakes

• Treating a fluent, detailed answer as evidence it is true. Specific details such as dates, names and citations are exactly where guesses hide.
• Never telling the model it may say "I don't know", then counting refusals as failures in your eval.
• Measuring only accuracy, so a model that guesses more looks better than one that abstains appropriately.
• Asking the model for citations and never checking that they exist.
• Using the model for arithmetic, prices or live data that a tool or database should supply.

In the wild

Check yourself

Recap