Embedding and indexing

The big idea

After chunking, every chunk goes through an embedding modelA model that maps a piece of text to a fixed-length vector so that texts with similar meaning get vectors that are close together., which turns it into a vector of a few hundred to a few thousand numbers. The vector, the chunk text and its metadata are stored together, so that a query vector can find the nearest chunks and the system can show their text and cite their source.

Three decisions here are hard to undo. The embedding model decides how well meaning is captured for your language and domain. Its vector size and storage precision decide how much memory and disk the index needs. And because every model has its own vector space, the documents and the queries must always be embedded by the same model: switching models means re-embedding the entire corpus.

Think of a library catalogue where every book gets a shelf code. The codes only work with one classification system. You can pick the system, and how many digits each code has, but if you switch systems you must relabel every book before anyone can find anything.

How it works

Choosing an embedding model

Things that matter, roughly in order:

• Retrieval quality on your data. Public leaderboards such as MTEB are a starting point ( Sentence and document embeddings ), but test the top few candidates on your own questions and documents. Domain language, such as legal, medical or code, can change the ranking.
• Languages. If users ask in one language about documents in another, you need a multilingual model.
• Input length. Chunks longer than the model's maximum input are silently truncated.
• Vector size, cost and hosting. Bigger vectors mean bigger indexes. A hosted API is simple; an open-weights model you run yourself keeps data in-house and avoids per-token fees.

Some models are trained to embed queries and documents differently, and expect a prefix or a separate mode for each. Read the model's instructions; getting this wrong quietly lowers recall.

Batching

Embedding one chunk per request wastes most of the time on network round trips and leaves the GPU idle. Send batches, often dozens to hundreds of chunks per call, run several batches in parallel within rate limits, and retry failures. For millions of chunks, ingestion is a batch job that should be resumable: record which chunks are done so a crash does not restart from zero.

Storing vectors and metadata

Store, for every chunk: an ID, the vector, the chunk text, and metadata such as source, section, date and permissions, plus the embedding model name and version. Normalising vectors to length 1 lets a dot product stand in for cosine similarity.

Storage adds up quickly. A 1,024-dimension vector in 32-bit floats is 4 KB, so 10 million chunks need about 41 GB for vectors alone. Lower precision, 16-bit, 8-bit integers or even 1 bit per number, cuts that by 2, 4 or 32 times at some cost in accuracy, often recovered by re-scoring the top candidates with full-precision vectors. Search indexes for fast lookup come in Vector databases and ANN indexes .

Re-embedding on model change

Vectors from different models live in different spaces. A query embedded with model B compared against documents embedded with model A gives meaningless distances. So a model upgrade is a migration: embed the whole corpus with the new model into a new index, evaluate it, switch queries over at once, then delete the old index. Recording the model version on every vector makes mistakes visible and lets you run old and new side by side.

Try it

Graph-link overhead assumes an HNSW-style index with 16 neighbours per vector; throughput and prices are assumptions to replace with yours.

Try this:

1. At the defaults (10 million chunks, 1,024 dimensions, float32), vectors take about 41 GB, more than the chunk text itself. Re-embedding the corpus takes about 2.8 hours at 1,000 chunks per second and costs about 150 dollars at the assumed price.
2. Switch precision to int8, then binary. Vector storage falls to about 10 GB, then 1.3 GB. This is why large indexes use quantised vectors with a full-precision re-check of the top results.
3. Set chunks to 1 billion. Even at int8, vectors need about a terabyte, and re-embedding takes over 11 days at the same throughput. At that scale the choice of embedding model is close to permanent.

Going deeper

Same model, or nothing

Two toy embedding models, each a different random projection of word counts, show the vector-space problem directly.

# Two toy "embedding models" (different random projections of word counts), an index of
# normalised vectors with metadata, and what happens if queries use the wrong model.
import numpy as np

rng = np.random.default_rng(0)
VOCAB = "refund annual plan monthly export pdf markdown cli token limit daily free paid".split()
CHUNKS = [("help/refunds.md", "refund annual plan"), ("help/refunds.md", "monthly plan refund"),
          ("help/export.md", "export pdf markdown"), ("help/cli.md", "cli export token"),
          ("help/limits.md", "daily limit free plan"), ("help/limits.md", "paid plan limit")]
QUERIES = [("refund for annual plan", 0), ("export as pdf", 2), ("cli token", 3), ("free daily limit", 4)]

def make_model(seed, dim=64):
    P = np.random.default_rng(seed).normal(size=(len(VOCAB), dim))
    def embed(texts):                                   # one batch in, one matrix out
        counts = np.array([[t.split().count(w) for w in VOCAB] for t in texts], float)
        V = counts @ P
        return V / np.linalg.norm(V, axis=1, keepdims=True)   # normalise: dot = cosine
    return embed

model_a, model_b = make_model(1), make_model(2)
index = {"vectors": model_a([t for _, t in CHUNKS]).astype(np.float32),   # store vectors...
         "meta": [{"id": i, "source": s, "model": "toy-a"} for i, (s, _) in enumerate(CHUNKS)]}

def hits(embed, vectors):
    q = embed([q for q, _ in QUERIES])
    return sum(int(np.argmax(vectors @ v) == want) for v, (_, want) in zip(q, QUERIES))

print("index:", index["vectors"].shape, "model:", index["meta"][0]["model"])
print("queries with model A (same as index):", hits(model_a, index["vectors"]), "of", len(QUERIES))
print("queries with model B (wrong model):  ", hits(model_b, index["vectors"]), "of", len(QUERIES))

q8 = np.round(index["vectors"] * 127).astype(np.int8)   # int8: 4x smaller than float32
print("int8 index:", hits(model_a, q8.astype(float) / 127), "of", len(QUERIES),
      f"| bytes float32 {index['vectors'].nbytes}, int8 {q8.nbytes}")

index: (6, 64) model: toy-a
queries with model A (same as index): 4 of 4
queries with model B (wrong model):   0 of 4
int8 index: 4 of 4 | bytes float32 1536, int8 384

Queries embedded with the index's own model find the right chunk 4 times out of 4. The same queries embedded with the other model find none: the numbers look like vectors, but they live in a different space. The int8 copy of the index still gets all 4 right at a quarter of the size, which is typical when vectors are well spread out.

Hybrid indexes

Most production systems keep a keyword index (BM25) next to the vector index, built from the same chunks, so exact terms like error codes and product names are not missed ( Hybrid search and reranking ). Both indexes must be updated and deleted together.

Trade-offs

Common mistakes

• Embedding queries with a different model, or a different model version, than the documents.
• Ignoring a model's query and document prefixes, quietly lowering recall.
• Chunks longer than the model's input limit, silently truncated.
• Embedding one chunk per request, with no batching, parallelism or resumable progress.
• Not storing the model version, so nobody knows which vectors need re-embedding after an upgrade.

In the wild

Where you will see this: vector databases and search engines store vectors next to text and metadata and offer quantised vector types; databases such as PostgreSQL gained vector columns through extensions so embeddings can live beside the rest of an application's data ( Vector databases and ANN indexes ); and teams migrating to a new embedding model typically build a second index, compare retrieval quality on an evaluation set, then switch traffic over.

Check yourself

Recap