Document ingestion and chunking

The big idea

Retrieval does not search documents; it searches chunksA passage cut from a document, typically a few hundred tokens long, that is embedded, indexed and retrieved as one unit in a RAG system.. Before anything can be retrieved, every PDF, web page and wiki article has to be turned into clean text, cut into passages, and labelled with where each passage came from. This is ingestion, and the quality of everything downstream is capped by it. A table cut in half, a heading separated from its paragraph, or a PDF parsed into scrambled columns cannot be fixed by a better model later.

Chunk size is a balancing act. Too large, and a chunk mixes several topics, its embedding becomes a blurry average, and it wastes context when retrieved. Too small, and a chunk loses the context that gives it meaning: "the limit resets at midnight" does not say which limit. The best splits follow the document's own structure: sections, paragraphs, tables and code blocks.

Think of cutting a cookbook into index cards. Cut every 300 characters and you get cards that start halfway through one recipe's ingredients and end in another's method. Cut one card per recipe, with the recipe's name at the top, and every card is useful on its own.

How it works

Parsing

Each format has its traps. HTML is full of navigation menus, cookie banners and footers that should not become chunks; extract the main content and keep the heading tags. PDFs store characters at positions on a page, not paragraphs: multi-column layouts, headers and footers, hyphenation and tables all need reconstruction, and scanned PDFs need OCR. Layout-aware parsers or vision-capable models handle hard PDFs far better than plain text extraction. Whatever the format, convert to a common representation, often Markdown, that keeps headings, lists, tables and code.

Always look at the parsed output for a sample of documents. Parsing errors are invisible in retrieval metrics until a user asks about the table that came out as a jumble of numbers.

Chunk size and overlap

Chunk size is usually set in tokens, commonly a few hundred, with documents and questions deciding the right value. Short factual answers favour smaller chunks; questions that need explanation favour larger ones. Test a few sizes on your evaluation set ( Evaluating RAG ) rather than trusting a default.

OverlapRepeating the end of one chunk at the start of the next, so a sentence or idea that falls on a boundary appears whole in at least one chunk. repeats a little of the previous chunk at the start of the next, so an idea that straddles a boundary survives whole in at least one chunk. It costs storage and can produce near-duplicate results, so keep it modest, and it is less needed when splits already follow natural boundaries.

Structure-aware chunking

Instead of counting characters, split where the document says a new idea starts:

• Headings start a new chunk; a long section is split further by paragraphs.
• Paragraphs and sentences are the next boundaries when a section is too long.
• Tables and code blocks stay whole: half a table or half a function is useless and misleading.
• Small neighbouring pieces under the same heading are packed together up to the size limit.

A recursive splitter that tries paragraph breaks first, then line breaks, then sentence ends, is a simple version of this that works for plain text.

Metadata

Every chunk should carry where it came from: source URL or file, document title, the heading path ("Exporting notes > Limits"), date or version, and who may see it. Metadata powers citations ( Why RAG? ), filtering by date or product, freshness checks, and access control at query time ( Production RAG architecture ). Prepending the title and heading path to the chunk text before embedding also restores context the chunk lost when it was cut out ( Contextual retrieval and query transformation ).

Try it

Try this:

1. At the defaults (fixed, 300 characters, no overlap), there are 3 chunks. The table is cut in half and two chunks end in the middle of a word or sentence.
2. Switch to recursive at 300: 4 chunks and nothing is broken, because cuts land on paragraph and sentence ends. Now drop the size to 200: the code block is split, since a size limit alone knows nothing about code.
3. Switch to structure-aware at 200: 5 chunks, nothing broken, each inside one section. Then add overlap of 60. One chunk now starts in the middle of the table: overlap repeats text blindly, so with good boundaries you often need little or none.

Going deeper

A structure-aware chunker in code

# Structure-aware chunking for Markdown: split by headings, keep code blocks and tables whole,
# pack paragraphs up to a size limit, and attach source and heading path as metadata.
import re

DOC = """# Exporting notes
You can export any note as Markdown, PDF or HTML. Attachments over 50 MB are skipped.

## Export from the command line
Use the CLI to export a whole notebook. It needs an API token from Settings.

```
notes export --notebook Work --format pdf
notes export --all --format md --out ./backup
```

Large notebooks can take several minutes.

## Limits
Free plans can export 20 notes per day. Paid plans have no daily limit."""

def blocks(md):
    """Yield (heading_path, block) pairs; a fenced code block is one block."""
    path, buf = [], []
    for part in re.split(r"(```[\s\S]*?```)", md):          # protect code fences
        if part.startswith("```"):
            yield list(path), part
            continue
        for para in re.split(r"\n\s*\n", part):
            for line in para.strip().splitlines():
                if m := re.match(r"(#+) (.*)", line):
                    path = path[: len(m[1]) - 1] + [m[2]]   # update the heading path
                else:
                    buf.append(line)
            if buf:
                yield list(path), " ".join(buf)
                buf = []

def chunk(md, source, max_chars=200):
    out = []
    for path, block in blocks(md):
        last = out[-1] if out else None
        if last and last["headings"] == path and len(last["text"]) + len(block) < max_chars:
            last["text"] += "\n" + block                      # pack small blocks together
        else:
            out.append({"source": source, "headings": path, "text": block})
    return out

for i, c in enumerate(chunk(DOC, "help/export.md")):
    print(f"chunk {i}  [{' > '.join(c['headings'])}]  {len(c['text'])} chars")
    first = c["text"].split("\n")[0][:60]
    print("   ", first + ("  + whole code block" if "```" in c["text"] else ""))

chunk 0  [Exporting notes]  85 chars
    You can export any note as Markdown, PDF or HTML. Attachment
chunk 1  [Exporting notes > Export from the command line]  172 chars
    Use the CLI to export a whole notebook. It needs an API toke  + whole code block
chunk 2  [Exporting notes > Export from the command line]  41 chars
    Large notebooks can take several minutes.
chunk 3  [Exporting notes > Limits]  71 chars
    Free plans can export 20 notes per day. Paid plans have no d

The code block stays whole inside chunk 1, packed with the paragraph that introduces it, and every chunk records its heading path. A code block or table longer than the limit stays a single oversized chunk by design; for very large tables, split by rows and repeat the header row in every piece.

Parent-child chunks

You can retrieve on small chunks for precision but give the model a larger window for context: index sentences or short passages, and when one matches, return its whole parent section. This decouples "what is easy to find" from "what is enough to answer".

Keeping ingestion fresh

Documents change. Track a content hash per document, re-ingest only what changed, and delete the chunks of removed documents, or the index will happily serve last year's policy. Production pipelines run this continuously ( Production RAG architecture ).

Trade-offs

Common mistakes

• Never looking at the parsed text, so scrambled PDF tables and boilerplate go straight into the index.
• Fixed character windows that cut tables, code and sentences in half.
• Chunks with no metadata, so answers cannot cite sources or be filtered by permission or date.
• Choosing chunk size by habit instead of testing a few sizes on real questions.
• Re-ingesting changed documents without deleting their old chunks.

In the wild

Where you will see this: RAG frameworks ship recursive and Markdown or HTML header-based splitters as their standard chunking tools; document-parsing services and open-source libraries specialise in turning PDFs, slides and scanned forms into structured text; and hosted retrieval products typically expose chunk size and overlap as the first settings to tune, with metadata filters alongside.

Check yourself

Recap