Skip to content
learn
  1. Learn
  2. High Level System Design
  3. Search, Geo and Specialized Data Structures

System Design HLD-15.06

Vector search in system design

Embeddings as data, approximate nearest neighbour indexes at a glance, and when a system needs a vector database: memory, build time and filtering.

Intermediate 10 min Last reviewed

The big idea

A customer searches a help center for "my parcel never showed up". The best article is titled "Missing delivery". Keyword search finds nothing in common: no shared words. What the system needs is to match meaning, not spelling. An embeddingA list of numbers (a vector) produced by a model to represent an item such as a sentence, image or user, placed so that similar items have nearby vectors. model turns each article, query, image or user into a vector of a few hundred numbers, placed so that things with similar meaning land close together. "Parcel never showed up" and "missing delivery" end up neighbours.

From a system designer's point of view, embeddings are just a new kind of data with a new kind of query: "give me the k vectors nearest to this one". Doing that exactly means comparing against every stored vector, which is too slow for millions of items. Approximate nearest neighbour (ANN) indexes answer it in milliseconds by accepting that a few true neighbours may occasionally be missed. A vector databaseA database or index specialized in storing embedding vectors and answering approximate nearest-neighbour queries, usually with metadata filtering. packages such an index with storage, updates and filters. How documents become those vectors in the first place, in batches and with metadata attached, is covered in embedding and indexing for RAG .

Think of a library arranged by topic rather than alphabetically. To find books like the one in your hand, you walk to its shelf and look at the neighbours, instead of checking every book in the building. Occasionally a relevant book sits on a different shelf, and that is the approximation.

How it works

Step 1: Embeddings as data

Embeddings in two dimensions with a query's nearest neighbours Points on a plane represent help articles embedded as vectors; articles on the same topic form clusters. A query point for 'my parcel never showed up' sits inside the delivery cluster, and its three nearest neighbours, missing delivery, late parcel and tracking stuck, glow inside a dashed circle. Billing and account security clusters are far away. missing delivery late parcel tracking stuck query billing articles account security Figure 1. Real embeddings have hundreds of dimensions, but the idea fits in two: similar meanings cluster together, and a query's nearest neighbours (inside the circle) are the most relevant items even with no words in common.

Each stored item gets a vector (commonly a few hundred to a few thousand dimensions, 4 bytes each) plus metadata such as id, language, price or tenant. Similarity is usually cosine similarity or dot product. The vectors come from a model, which has consequences for the system: change or upgrade the model and every stored vector must be recomputed, because vectors from different models are not comparable.

Step 2: Approximate nearest neighbour indexes

Exact search compares the query with every vector: fine for tens of thousands, too slow for hundreds of millions. ANN indexes look at a small, promising fraction instead:

  • Clustering (IVF): group vectors around centroids; at query time scan only the few closest groups.
  • Graphs (HNSW): link each vector to near neighbours in a layered graph and walk greedily toward the query.
  • Quantization: compress vectors (for example to bytes per dimension, or product quantization) so more fit in memory and comparisons are cheaper, at some cost in accuracy.
ann.pyruns with python3 (standard library)
import random, math

rnd = random.Random(11)
DIM, N, K = 16, 4000, 10
def unit(v):
    n = math.sqrt(sum(x * x for x in v)); return [x / n for x in v]
# Embeddings cluster by topic, so build 40 topic centers and scatter items around them.
topics = [unit([rnd.gauss(0, 1) for _ in range(DIM)]) for _ in range(40)]
items = [unit([c + rnd.gauss(0, 0.35) for c in rnd.choice(topics)]) for _ in range(N)]
dot = lambda a, b: sum(x * y for x, y in zip(a, b))           # cosine similarity (unit vectors)

# Approximate index (IVF): group items under 40 centroids; search only the nprobe closest groups.
cents = items[:40]
groups = {i: [] for i in range(40)}
for j, v in enumerate(items):
    groups[max(range(40), key=lambda i: dot(v, cents[i]))].append(j)

queries = [unit([c + rnd.gauss(0, 0.35) for c in rnd.choice(topics)]) for _ in range(20)]
for nprobe in (1, 3, 8):
    recall = scanned = 0
    for q in queries:
        exact = set(sorted(range(N), key=lambda j: -dot(q, items[j]))[:K])
        probe = sorted(range(40), key=lambda i: -dot(q, cents[i]))[:nprobe]
        cand = [j for i in probe for j in groups[i]]
        approx = set(sorted(cand, key=lambda j: -dot(q, items[j]))[:K])
        recall += len(exact & approx) / K
        scanned += len(cand) / N
    print(f"nprobe {nprobe}: scanned {scanned / 20:5.1%} of vectors, recall@{K} {recall / 20:.0%}")

Output

nprobe 1: scanned  2.8% of vectors, recall@10 26%
nprobe 3: scanned  7.8% of vectors, recall@10 58%
nprobe 8: scanned 20.3% of vectors, recall@10 84%

The quality metric is recall@k: what fraction of the true top k the index returns. Probing more groups scans more vectors and raises recall; every ANN index has knobs like this that trade latency and cost against recall. (This toy index uses crude centroids; real IVF and HNSW indexes reach high recall while scanning far less.)

Step 3: When a system needs a vector database

Common uses: semantic search, "more like this" recommendations, deduplication, and retrieval for LLM applications that feed relevant documents into a model's prompt ( Why RAG? ). Practical questions decide the design:

  • Scale and memory: 100 million 768-dimensional float vectors are about 300 GB raw before index overhead. Quantization, disk-based indexes or sharding become necessary ( Partitioning and sharding ).
  • Filtering: real queries combine similarity with conditions ("in stock", "this tenant only", "language = de"). Filtering after the ANN search can return too few results; good systems filter during the search.
  • Freshness: inserts and deletes must be handled without rebuilding the whole index, which can take hours at scale.
  • Hybrid ranking: combining BM25 keyword scores with vector similarity often beats either alone, especially for exact terms like product codes ( Inverted indexes and full-text search ).

Options range from a vector extension in an existing database (such as pgvector for PostgreSQL), to vector support in search engines (Elasticsearch, OpenSearch), to dedicated vector databases. If you already run one of the first two and have millions rather than billions of vectors, start there.

Trade-offs

OptionWins whenLoses when
Exact (brute-force) search Up to tens of thousands of vectors, or when perfect recall matters Latency and cost grow linearly with data
Vector extension in an existing database Moderate scale, transactional data and filters in one place Fewer tuning options at very large scale
Search engine with vector support Hybrid keyword plus semantic search Memory-heavy indexes; operational tuning
Dedicated vector database Very large collections and high query rates Another system to run, sync and secure

Common mistakes

  • Mixing vectors from different embedding models, or upgrading the model without re-embedding everything.
  • Filtering after the ANN search and returning too few or no results.
  • Ignoring memory: raw float vectors at scale need hundreds of gigabytes.
  • Assuming vector search replaces keyword search; exact identifiers and rare terms still need BM25.
  • Never measuring recall, so index settings silently trade away quality.

In the wild

Semantic search, recommendations and retrieval-augmented generation for LLM applications are the main drivers of vector search. PostgreSQL's pgvector extension, Elasticsearch and OpenSearch vector fields, and dedicated systems such as Milvus, Qdrant, Weaviate and Pinecone all implement ANN indexes such as HNSW and IVF, typically alongside metadata filtering.

Check yourself

  1. Why can vector search match 'my parcel never showed up' to 'missing delivery'?

    Show answer

    B. Both texts' embeddings are close together because the model captures similar meaning Embeddings place semantically similar items near each other regardless of wording.

  2. What does recall@10 measure for an ANN index?

    Show answer

    B. The fraction of the true 10 nearest neighbours that the index returned ANN trades some recall for speed; recall shows how much.

  3. You switch to a new embedding model. What must happen to stored vectors?

    Show answer

    B. Recompute them all, because vectors from different models are not comparable Each model defines its own space; mixing them breaks similarity.

  4. A product search must match exact model numbers like 'XR-2200' and also natural language questions. What fits?

    Show answer

    B. Hybrid ranking combining BM25 keyword scores with vector similarity Keywords catch exact identifiers; vectors catch meaning.

Recap

  • Embeddings turn items into vectors where similar meanings are close together.
  • Nearest-neighbour search finds related items; exact search does not scale, so ANN indexes trade a little recall for speed.
  • IVF, HNSW and quantization are the main ANN techniques; tune them against measured recall.
  • Plan for memory, filtering during search, index updates and re-embedding when the model changes.
  • Combine with keyword search for hybrid ranking, and start with vector support in systems you already run.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy