Production RAG architecture

The big idea

A RAG demo takes an afternoon: load some PDFs, embed them, ask a question. A RAG product is a different system. Documents change every day and some get deleted. Different users may see different documents. The same questions arrive thousands of times. The model sometimes answers from its imagination instead of the sources. And nobody notices quality slipping unless something measures it.

So production RAG is two pipelines and a loop. An offline pipeline keeps the index an accurate, permission-aware mirror of the sources. An online pipeline answers each question: check a cache, retrieve only what this user may see, rerank, generate with citations, and check the answer before showing it. And a feedback loop logs everything, evaluates it, and turns failures into fixes, most of which end up being fixes to data and retrieval, not to the model.

Think of a public library rather than a bookshelf at home. New books are catalogued daily and withdrawn books are pulled from the shelves. Some rooms need a staff card. Librarians keep answers to the most common questions at the desk. They point to the exact book and page. And someone tracks which questions went unanswered.

How it works

Ingestion that stays fresh

Connect to each source system (wiki, ticketing, file storage, databases) and sync on a schedule or on change events. Compare a content hash per document to re-process only what changed, and delete the chunks of documents that were removed or moved, otherwise the system keeps citing a policy that no longer exists. Record a version and timestamp on every chunk, and prefer newer versions when two conflict. Monitor the pipeline itself: a connector that silently failed last Tuesday means a week of stale answers.

Access control at retrieval time

Store each document's permissions (groups, roles, tenant) with its chunks, and apply the asking user's permissions as a filter inside the search ( Vector databases and ANN indexes ). Never retrieve first and filter later, and never rely on the prompt to keep secrets: once a restricted chunk is in the context, the model can repeat it. Permissions change too, so the sync pipeline must pick up permission changes as well as content changes. For multi-tenant products, separate indexes or partitions per tenant are the simplest strong guarantee.

Caching

Many users ask the same questions. Cache final answers keyed by the normalised question plus the user's permission scope, so one user's answer is never served to someone who could not see its sources. A semantic cacheA cache that returns a stored answer when a new question is similar in meaning to a cached one, not only when the text is identical, usually by comparing embeddings. also matches paraphrases by embedding similarity, with a strict threshold, since "can I cancel?" and "can I cancel my order?" may need different answers. Expire cached answers when their source documents change. Prompt caching cuts the cost of the rest ( Prompt caching and cost control ).

Citations in the interface

Ask the model to cite chunk IDs inline, map them to titles and links, and show them next to the answer. Check that each cited ID was actually in the context, and drop or flag citations that were not. Citations let users verify answers, build trust, and give you a signal: answers people click through and still mark wrong point at bad source documents.

Guardrails against hallucination

Before showing an answer, check it is grounded in the retrieved text ( Guardrails ). A cheap first check compares the answer with the retrieved chunks by similarity; answers that fail go to an LLM judge that checks each claim, or are replaced with a safe fallback such as "I could not find this in our help centre" and a link to a person. When retrieval finds nothing relevant (low reranker scores), say so rather than letting the model improvise.

Monitoring and the feedback loop

Log, for every request: the question, the rewritten query, the retrieved chunk IDs and scores, the prompt version, the answer, latency, cost, guardrail results and user feedback. From these, track retrieval quality, groundedness, answer ratings and the share of "I don't know" answers over time, sliced by topic. Feed failures into the golden set ( Evaluating RAG ), and gate every prompt, model or retrieval change on it.

Most improvements come from the data side: rewriting or adding knowledge base articles that answers kept missing, fixing parsing, tuning chunking and retrieval. Swapping to a bigger model is rarely the highest-leverage change.

Try it

Use Prev and Next on Figure 1 to walk the four paths. As you go, answer these:

1. A help article is deleted on Monday. In step 1, what must happen so it stops being cited, and what in step 2 could still serve it? (The cached answers that used it.)
2. An HR document is indexed with group "hr". In step 2, where exactly must the "support" user's permissions apply, and why is filtering the model's answer afterwards too late?
3. In step 4, a cluster of thumbs-down answers all cite one outdated article. Which box does the fix go back to?

The code below shows the first two chores, sync and permission filtering, in miniature.

# Two production chores: keep the index in sync with the sources (upsert changed, delete removed),
# and filter by permissions inside retrieval, before anything reaches the model.
import hashlib

def h(text):
    return hashlib.sha256(text.encode()).hexdigest()[:8]

index = {  # doc_id -> what the index currently holds
    "refunds": {"hash": h("Refunds within 14 days."), "groups": {"everyone"}, "text": "Refunds within 14 days."},
    "pricing-2025": {"hash": h("Pro costs 8 dollars."), "groups": {"everyone"}, "text": "Pro costs 8 dollars."},
    "salaries": {"hash": h("Band 4: ..."), "groups": {"hr"}, "text": "Band 4: ..."},
}
source = {  # what the source systems hold now
    "refunds": ("Refunds within 30 days.", {"everyone"}),          # edited
    "salaries": ("Band 4: ...", {"hr"}),                            # unchanged
    "pricing-2026": ("Pro costs 10 dollars.", {"everyone"}),        # new; 2025 page deleted
}

def sync(index, source):
    for doc_id in list(index):
        if doc_id not in source:
            del index[doc_id]; print(f"delete  {doc_id}")
    for doc_id, (text, groups) in source.items():
        if doc_id not in index or index[doc_id]["hash"] != h(text):
            print(f"{'update ' if doc_id in index else 'insert '} {doc_id}  (re-chunk, re-embed)")
            index[doc_id] = {"hash": h(text), "groups": groups, "text": text}
        else:
            print(f"skip    {doc_id}  (unchanged)")

def retrieve(query_words, user_groups, k=3):
    allowed = [d for d, v in index.items() if v["groups"] & user_groups]   # filter first
    scored = sorted(allowed, key=lambda d: -sum(w in index[d]["text"].lower() for w in query_words))
    return scored[:k]

sync(index, source)
print("\nsupport agent sees:", retrieve({"band", "refunds", "pro"}, {"everyone"}))
print("HR sees:           ", retrieve({"band", "refunds", "pro"}, {"everyone", "hr"}))

delete  pricing-2025
update  refunds  (re-chunk, re-embed)
skip    salaries  (unchanged)
insert  pricing-2026  (re-chunk, re-embed)

support agent sees: ['refunds', 'pricing-2026']
HR sees:            ['refunds', 'salaries', 'pricing-2026']

The 2025 pricing page is deleted from the index rather than left to compete with the 2026 one, the edited refunds page is re-processed, and the unchanged page is skipped. The support agent and HR ask the same question, but salary data is filtered out for support before ranking even starts.

Going deeper

Latency budget

A production RAG request chains several steps: cache lookup, query rewrite, two searches, reranking, generation and a guardrail. Each adds latency. Run independent steps in parallel, use small fast models for rewriting and guardrails, stream the answer while it generates ( Streaming responses to users ), and budget each step ( Latency and cost budgets ).

Fallbacks

Every dependency fails sometimes: the vector store times out, the model returns an error, the guardrail rejects the answer. Decide the behaviour for each: keyword-only search, a smaller model, a link to search results, or a hand-off to a person. A graceful "here are the most relevant articles" beats an error page.

Structured output for actions

When the assistant can also act, such as opening a ticket or issuing a refund, structured outputs and validation matter as much as retrieval ( Structured outputs ), and actions need their own permission checks.

Trade-offs

Common mistakes

• Never deleting chunks of removed or replaced documents.
• Filtering permissions after retrieval, or trusting the prompt to keep restricted text secret.
• Caching answers without the user's permission scope in the key.
• Showing citations without checking they were actually in the retrieved context.
• Logging only the final answer, so failures cannot be traced to retrieval, prompt or data.

In the wild

Check yourself

Recap