Why RAG?

The big idea

Ask a language model "What is our refund policy for annual plans?" and one of two things happens. Either it says it does not know, or, worse, it writes a confident, plausible policy that your company never had. The model was trained on public text up to some date. It has never seen your help center, your contracts, or last week's pricing change.

Retrieval augmented generationRetrieval augmented generation: fetching relevant text from your own data at question time and giving it to the model as context, so its answer is grounded in that text. (RAG) fixes this with an open-book exam. Before the model answers, your system looks up the most relevant passages from your own documents and pastes them into the prompt with an instruction: answer using only this material, and cite it. The model does what it is good at (reading and writing), and your retrieval system does what the model cannot (knowing your private, current facts).

How it works

RAG has two halves that run at different times.

Step 1, offline: ingest your documents

1. Load sources: PDFs, web pages, wiki pages, tickets, database rows.
2. Chunk them into passages of a few hundred tokens. A whole manual is too big to retrieve usefully; a single sentence loses its context. Chunking is covered in depth in Document ingestion and chunking .
3. Embed each chunk into a vector with an embedding model, so passages with similar meaning end up close together.
4. Index the vectors in a vector indexA store built for fast nearest-neighbour search over vectors, usually with approximate indexes such as HNSW., along with metadata such as source URL, title, date and who is allowed to see it.

This runs once up front and then again whenever documents change.

Step 2, online: answer a question

1. Embed the question with the same embedding model.
2. Retrieve the top k nearest chunks, often 20 to 50, usually combining vector similarity with keyword search (BM25) so exact terms like error codes are not missed.
3. Rerank those candidates with a slower, more precise model and keep the best 3 to 8.
4. Generate: build a prompt with the instructions, the retrieved chunks (each labelled with its source) and the question. Ask for an answer that cites sources and says "I don't know" when the chunks do not contain the answer.

Step 3: what the prompt actually looks like

You answer questions about Acme's products using ONLY the documents below.
If the answer is not in the documents, say you don't know.
Cite sources as [1], [2].

[1] (help/refunds.md) Annual plans can be refunded in full within 14 days of purchase...
[2] (help/billing-faq.md) After 14 days, annual plans are refunded pro rata for unused months...

Question: What is the refund policy for annual plans?

Nothing magical: the model is reading two paragraphs and summarizing them. That simplicity is why RAG is usually the first thing to try when a model needs your knowledge.

Try it

Try this:

1. Step through once and note which component decides whether the right paragraph is even visible to the model.
2. Imagine the billing FAQ was never ingested. Which step fails, and what should the model say?
3. Imagine retrieval returns an outdated 2023 refunds page. How could metadata (dates) help?

The same pipeline in miniature, with word overlap standing in for embeddings:

# The smallest useful RAG loop: chunk, index, retrieve by word overlap, build a cited prompt.
# Real systems use embeddings and a vector index; the shape of the pipeline is the same.
import re

DOCS = {
    "help/refunds.md": "Annual plans can be refunded in full within 14 days of purchase. "
                       "Monthly plans are not refundable. Contact billing to start a refund.",
    "help/billing-faq.md": "After 14 days, annual plans are refunded pro rata for unused months. "
                           "Refunds go back to the original payment method within 10 days.",
    "help/export.md": "You can export notes as Markdown or PDF from the File menu. "
                      "Exports include attachments up to 50 MB.",
}

def chunks(path, text, size=2):                    # offline: split into 2-sentence chunks
    sents = re.split(r"(?<=\.) ", text)
    for i in range(0, len(sents), size):
        yield {"src": path, "text": " ".join(sents[i:i + size])}

STOP = {"the", "a", "of", "can", "i", "my", "to", "in", "an", "on", "after"}
STEM = False                                         # crude stemming: refunds/refunded -> refund
def words(s):
    w = set(re.findall(r"[a-z0-9]+", s.lower())) - STOP
    return {re.sub(r"(ed|s)$", "", x) for x in w} if STEM else w
INDEX = [c for p, t in DOCS.items() for c in chunks(p, t)]

def retrieve(question, k=2):                         # online: score every chunk, keep the top k
    q = words(question)
    scored = sorted(INDEX, key=lambda c: len(q & words(c["text"])), reverse=True)
    return [c for c in scored[:k] if q & words(c["text"])]

def build_prompt(question):
    ctx = "\n".join(f"[{i}] ({c['src']}) {c['text']}" for i, c in enumerate(retrieve(question), 1))
    return ("Answer using ONLY the documents below. If the answer is not there, say you don't know.\n"
            f"Cite sources as [1], [2].\n\n{ctx}\n\nQuestion: {question}")

Q = "Can I get a refund on my annual plan after 3 weeks?"
print(f"{len(INDEX)} chunks indexed")
print("exact words:", [c["src"] for c in retrieve(Q)])   # misses the FAQ: "plans", "refunded"
STEM = True
print("with stems: ", [c["src"] for c in retrieve(Q)], "\n")
print(build_prompt(Q))

4 chunks indexed
exact words: ['help/refunds.md', 'help/refunds.md']
with stems:  ['help/refunds.md', 'help/billing-faq.md'] 

Answer using ONLY the documents below. If the answer is not there, say you don't know.
Cite sources as [1], [2].

[1] (help/refunds.md) Annual plans can be refunded in full within 14 days of purchase. Monthly plans are not refundable.
[2] (help/billing-faq.md) After 14 days, annual plans are refunded pro rata for unused months. Refunds go back to the original payment method within 10 days.

Question: Can I get a refund on my annual plan after 3 weeks?

Matching exact words, retrieval returns two chunks of the refunds page and misses the billing FAQ, the one passage that answers the question, because it says "plans" and "refunded" instead of "plan" and "refund". Crude stemming fixes it. Real systems use embeddings and hybrid search for the same reason ( Hybrid search and reranking ), and the lesson holds at any scale: if retrieval misses, the model never sees the answer.

Going deeper

Where RAG breaks

Most RAG failures are retrieval failures, not model failures. The model can only use what retrieval hands it.

• The answer is split across chunks, and only half was retrieved.
• Vocabulary mismatch: the user says "money back", the docs say "reimbursement". Hybrid search and query rewriting help ( Contextual retrieval and query transformation ).
• Lost context: a chunk says "this limit is 10 per day" but no longer says which product "this" is. Contextual retrieval adds a short document summary to each chunk before embedding.
• Stale or conflicting sources: two versions of the same policy. Use dates, versioning and deletion in the ingestion pipeline.
• Permission leaks: retrieval returns a document the user should not see. Filter by access control at query time, never after generation.

RAG vs fine-tuning vs long context

RAG gives a model knowledge. Fine-tuning mostly changes behaviour, style and format. Long context windows let you paste in more material, but cost and latency grow with every token, and models attend less reliably to the middle of very long inputs. A common production pattern combines them: RAG for facts, a light fine-tune or strong prompt for tone and format. See the decision guide in Prompting vs RAG vs fine-tuning: choosing .

Measuring it

Evaluate the two halves separately ( Evaluating RAG ): retrieval with recall at k ("was the right chunk in the top k?") on a set of questions with known source passages, and generation with faithfulness ("is every claim supported by the retrieved text?") and answer relevance.

Trade-offs

Common mistakes

• Skipping evaluation and judging quality by trying five questions by hand.
• Tuning the prompt when the real problem is that retrieval never returned the right chunk.
• Chunking by a fixed character count that splits tables and code in half.
• Embedding queries and documents with different embedding models.
• Not telling the model what to do when the context does not contain the answer.

In the wild

Check yourself

Recap