GraphRAG and structured retrieval

The big idea

Chunk retrieval answers questions whose answer sits in one or a few passages. Many real questions do not look like that. "Who is on call for the services Checkout depends on?" needs facts from four documents chained together. "What are the main causes of incidents across all postmortems?" needs a view of the whole corpus, not the top 10 passages. "Which service had the most incidents in March?" is a count over rows in a database, and no passage contains the answer at all.

For these, retrieve from structure. A GraphRAGRetrieval over a knowledge graph built from your documents: entities and their relations are extracted, so questions can be answered by following relations or from summaries of groups of related entities. system extracts entities and relations from documents into a knowledge graph, so multi-hop questions become walks along edges, and corpus-wide questions can be answered from summaries of related groups of entities. Text-to-SQLHaving a language model write a database query from a natural-language question and the database schema, running it, and using the results to ground the answer. treats a database as the knowledge source: the model writes the query, the database does the counting.

Think of a company directory. To find "who to call about Checkout", you do not search for a page that says it; you look up what Checkout depends on, who owns that, and who is on call this week. And to know how many incidents happened in March, you do not read reports; you ask the database.

How it works

Building a knowledge graph from documents

Offline, a language model reads each chunk and extracts entities (services, teams, people, products) and relations ("payments-api is owned by Team Payments"). The same entity mentioned in different documents must be merged into one node, which is the hard part: "Payments team", "Team Payments" and "payments squad" should be one node; two different people named Ana should not. The graph can be stored in a graph database or as simple tables of nodes and edges.

Multi-hop questions

With a graph, "who is on call for the services Checkout depends on?" becomes a traversal: Checkout, depends on, payments-api, owned by, Team Payments, on call, Ana, and onward through fraud-check to Team Risk. The model can be given the traversal results, or the subgraph around the entities in the question, as context. Chunk retrieval would need every link to appear in the top k, and usually it does not.

Corpus-wide questions

"What are the main themes?" is not a retrieval question at all: no top-k set of chunks covers a whole corpus. The GraphRAG approach groups closely related entities into communities with a graph clustering algorithm and has a model write a summary of each community ahead of time. At question time, each community summary produces a partial answer, and the partial answers are combined. It is expensive to build, but it makes sensemaking questions over large collections possible.

Text-to-SQL

When the answer lives in a database, give the model the schema (tables, columns, types, example values and short descriptions), ask it to write a query, run the query, and give the rows to the model to phrase the answer. At scale, the hard part is choosing which few tables and columns to show out of thousands, which is itself a retrieval problem: search over table descriptions and example queries. Safety is essential: run as a read-only user, allow only single SELECT statements on approved tables, add row limits and timeouts, and apply the same access controls as the user's own permissions.

Routing

Production systems often combine sources and route each question: rules ("contains a number of days and 'how many'"), a classifier, or a model choosing a tool. An agent can also call several retrievers in sequence ( Function calling and tool use ).

Try it

Try this:

1. Pick "who is on call for checkout?". The path lights up across four hops and two branches. Ask yourself which single chunk of text could contain all of that: none, which is why chunk retrieval struggles.
2. Pick "what does fraud-check do?". A single passage answers it; a graph adds nothing. Most questions are like this, which is why chunk retrieval stays the default.
3. Pick "most incidents in March?" and then "main causes across postmortems?". One is a count, best done by the database; the other is a whole-corpus summary, done from community summaries.

Going deeper

Text-to-SQL with guardrails, in code

# Text-to-SQL as retrieval: give the model the schema, let it write a query, check the query,
# run it read-only, and hand the rows to the model as grounding. `fake_model` stands in for it.
import re
import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE incidents (id INTEGER, service TEXT, month TEXT, severity TEXT);
INSERT INTO incidents VALUES
 (1,'payments-api','2026-03','S2'),(2,'payments-api','2026-03','S3'),(3,'fraud-check','2026-03','S2'),
 (4,'payments-api','2026-03','S1'),(5,'ledger-db','2026-02','S2'),(6,'fraud-check','2026-03','S3'),
 (7,'payments-api','2026-02','S3'),(8,'ledger-db','2026-03','S3');
""")
SCHEMA = "incidents(id INTEGER, service TEXT, month TEXT 'YYYY-MM', severity TEXT 'S1'..'S4')"

def fake_model(question, schema):
    return ("SELECT service, COUNT(*) AS n FROM incidents WHERE month = '2026-03' "
            "GROUP BY service ORDER BY n DESC")

def check(sql, allowed_tables={"incidents"}):
    s = sql.strip().rstrip(";")
    if not re.match(r"(?is)^select\b", s) or ";" in s:
        return "only a single SELECT is allowed"
    if re.search(r"(?i)\b(insert|update|delete|drop|alter|attach|pragma)\b", s):
        return "write or admin keyword found"
    tables = set(re.findall(r"(?i)\b(?:from|join)\s+(\w+)", s))
    if not tables <= allowed_tables:
        return f"unknown tables: {tables - allowed_tables}"
    return None

question = "Which service had the most incidents in March 2026?"
sql = fake_model(question, SCHEMA)
print("SQL:", sql)
print("check:", check(sql) or "ok")
rows = db.execute(sql).fetchall()
print("rows:", rows)
print("grounding for the model:", f"Question: {question}\nQuery result: {rows}")
print("\nbad query check:", check("DROP TABLE incidents"))

SQL: SELECT service, COUNT(*) AS n FROM incidents WHERE month = '2026-03' GROUP BY service ORDER BY n DESC
check: ok
rows: [('payments-api', 3), ('fraud-check', 2), ('ledger-db', 1)]
grounding for the model: Question: Which service had the most incidents in March 2026?
Query result: [('payments-api', 3), ('fraud-check', 2), ('ledger-db', 1)]

bad query check: only a single SELECT is allowed

The model writes the query; the code decides whether it may run. A schema-aware check (single SELECT, approved tables, no write keywords) and a read-only database user are two independent layers. The rows, not the model's memory, ground the answer.

Costs and pitfalls of graphs

Graph extraction calls a model over every chunk, then again for community summaries, so building a GraphRAG index costs far more than embedding. Extraction errors and failed entity merges create wrong or missing edges, and the graph must be updated when documents change. Start with chunk retrieval, collect the questions it fails on, and add a graph only if many of them are multi-hop or corpus-wide ( Evaluating RAG ).

Lighter alternatives

Before building a full graph, try metadata: tagging chunks with the services, teams or products they mention and filtering on those tags handles many "related entity" questions. For one or two hops, an agent that searches, reads and searches again can also chain facts ( The agent loop (ReAct) ).

Trade-offs

Common mistakes

• Expecting top-k chunks to answer "how many" or "which is largest" questions that need aggregation.
• Building a knowledge graph before checking whether failed questions are actually multi-hop.
• Merging entities carelessly, so two different things share a node or one thing is split across several.
• Running model-written SQL with write permissions or without table allowlists and row limits.
• Showing the model the entire database schema instead of the few relevant tables and columns.

In the wild

Check yourself

Recap