AI AI-04.03
Semantic search basics
Keyword versus semantic search: embed a corpus, find nearest neighbours for a query, and see when exact keywords still win and why hybrid search exists.
The big idea
A customer types "I want my money back" into your help centre. The right article is titled "Refund policy" and says plans "can be refunded within 14 days". A classic search box finds nothing: none of the words "want", "money" or "back" appear in the article. The customer opens a support ticket, or leaves.
Keyword searchSearch that finds documents by matching the words in the query, typically ranking them with a scoring function such as TF-IDF or BM25. asks "which documents contain these words?". Semantic searchSearch that embeds the query and the documents and returns the documents whose embeddings are nearest to the query's, so it matches meaning rather than exact words. asks "which documents mean something close to this?". It embeds the query, then returns the documents whose embeddings are nearest. "Money back" and "refund" use different words but land in the same neighbourhood, so the refund article comes first.
Think of asking for a book in two shops. In the first, a clerk searches the catalogue for the exact words you said. In the second, a well-read bookseller listens to what you are after and walks you to the right shelf, even if you misremembered the title. The second is usually more helpful, until you ask for the book with ISBN 978-0-13-468599-1 and the bookseller hands you something "similar".
How it works
Semantic search runs in two phases. It is also the first half of every retrieval augmented generation system ( Why RAG? ).
Step 1: Embed the corpus
Run every document (in practice every chunk of a document, see Document ingestion and chunking ) through the embedding model and store the vectors, normalised to length 1, together with an ID that points back to the text. This happens once, and again only for documents that change.
Step 2: Embed the query
When a query arrives, embed it with the same model. A different model would put it in a different space where distances are meaningless.
Step 3: Find the nearest neighbours
Compute the similarity between the query vector and the stored vectors and return the top . This is k-nearest-neighbour searchFinding the k stored items whose vectors are closest to a query vector, by cosine similarity or distance.. The simplest way is brute force: one dot product per document, then keep the highest. With 100,000 documents of 384 dimensions that is 38.4 million multiply-adds per query, which a laptop does in milliseconds. At tens of millions of documents it gets slow, and approximate indexes such as HNSW take over, trading a little accuracy for a huge speed-up ( Vector databases and ANN indexes ).
Step 4: Return the documents
Map the IDs back to text and show them, or, in a RAG system, paste them into the model's prompt ( Why RAG? ).
Try it
The semantic column uses a toy embedding model, a small table of related words, so it runs in your browser. Real models learn far richer relationships, but they fail in the same places.
Try this:
- Pick I want my money back. Keyword search returns nothing: no query word appears in any article. Semantic search finds Refund policy. This is the vocabulary mismatch semantic search was invented for.
- Pick E1042. Keyword search goes straight to the right article, because the code appears in it word for word. Semantic search still puts it first, but only just: other articles, including the one for a different error code, E1024, score close behind. To an embedding, a rare code is a few meaningless fragments, so E1042 and E1024 look almost alike.
- Back to where is my package. Keyword search matches "package" and returns Track your parcel. Semantic search prefers Shipping times, which is about the same topic but answers a different question. The hybrid column fuses the two lists and puts Track your parcel first. Now type your own queries, such as
refundorhow long does delivery take.
The core of semantic search is a few lines of numpy:
# Semantic search in its simplest form: embed the corpus once, then for each
# query compute similarity to every vector and keep the top k.
import numpy as np
rng = np.random.default_rng(42)
n_docs, dims, k = 100_000, 384, 5
# Stand-ins for document embeddings, normalised to length 1 (offline, once).
docs = rng.normal(size=(n_docs, dims)).astype(np.float32)
docs /= np.linalg.norm(docs, axis=1, keepdims=True)
# The query is a noisy copy of document 1234, so we know the answer.
query = docs[1234] + 0.05 * rng.normal(size=dims).astype(np.float32)
query /= np.linalg.norm(query)
scores = docs @ query # one dot product per document
top = np.argpartition(-scores, k)[:k] # the k best, unordered
top = top[np.argsort(-scores[top])] # order just those k
for rank, i in enumerate(top, 1):
print(f"{rank}. doc {i:>6} cosine {scores[i]:.3f}")
work = n_docs * dims
print(f"\nwork per query: {n_docs:,} x {dims} = {work:,} multiply-adds")
print(f"index memory: {docs.nbytes / 1e6:.1f} MB")Output
1. doc 1234 cosine 0.727
2. doc 5597 cosine 0.266
3. doc 6214 cosine 0.218
4. doc 19843 cosine 0.208
5. doc 19352 cosine 0.197
work per query: 100,000 x 384 = 38,400,000 multiply-adds
index memory: 153.6 MB The query was built as a noisy copy of document 1234, so the right answer is known, and it comes first with a cosine of 0.727 while the best wrong document scores 0.266. argpartition finds the top without sorting all 100,000 scores.
Going deeper
When keywords still win
Keyword search is not obsolete. It is better whenever exact strings carry the meaning:
- identifiers: error codes, SKUs, order numbers, ticket IDs, legal clause numbers;
- names and rare terms the embedding model never learned well;
- precise phrases, such as quoted text or a function name in code;
- negation and specificity: "laptops without a touch screen" can retrieve touch-screen laptops semantically.
It is also cheap, needs no model, and explains itself: you can see which words matched. The classic ranking function is BM25, which rewards documents that contain rare query terms several times without letting long documents win by sheer length. Inverted indexes and full-text search covers how search engines build its inverted index.
Hybrid search
Because the two methods fail in different places, production systems run both and merge the lists. A simple and robust merge is reciprocal rank fusionA way to merge ranked lists: each document scores the sum of 1 / (c + rank) over the lists it appears in, with c usually 60. It needs no score calibration. (RRF): each document gets over the lists it appears in. It uses ranks, not scores, so it does not care that BM25 scores and cosine similarities live on different scales. A reranker then often reorders the top few ( Hybrid search and reranking ).
Evaluating search
Do not judge search by trying five queries by hand. Collect real queries, mark which documents are correct, and measure recall at k (did the right document appear in the top k?) for keyword, semantic and hybrid. The answer is often surprising, and it differs per domain. Metrics are covered in Evaluating RAG .
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Keyword (BM25) | Exact identifiers, names, rare terms; cheap; explainable | Synonyms, paraphrases, typos in natural language, questions in other words |
| Semantic (embeddings) | Paraphrases, natural-language questions, cross-language with a multilingual model | Exact codes and names, negation; needs an embedding model and a vector index |
| Hybrid (both + fusion) | General-purpose search over mixed queries; the usual production choice | Two indexes to build and keep in sync; more moving parts to tune |
Common mistakes
- Replacing keyword search entirely and then discovering that product codes and names no longer work.
- Embedding queries and documents with different models, or with a model version that changed since indexing.
- Embedding whole long documents as one vector instead of chunks, so the right paragraph is drowned in an average.
- Merging BM25 scores and cosine scores by adding them directly, although they are on different scales. Use rank fusion or calibrate.
- Assuming the nearest document is relevant. Nearest neighbour search always returns something, even for queries your corpus cannot answer.
In the wild
Where you will see this: help-centre and documentation search that understands questions, e-commerce search ("comfortable shoes for standing all day"), code search across a repository, finding duplicate support tickets or bug reports, "related articles" boxes, and the retrieval step of every RAG chatbot. Most web-scale and enterprise search engines now combine a keyword index with a vector index, then rerank.
Check yourself
-
The query 'how do I get my money back' finds nothing in keyword search but works in semantic search. Why?
Show answer
B. The relevant article uses different words such as 'refund', and only semantic search matches meaning This is vocabulary mismatch. Keyword search needs shared words; semantic search compares meanings in embedding space.
-
Users search for exact order numbers like 'ORD-77812'. Which approach should handle these?
Show answer
B. Keyword (exact) search, alone or inside a hybrid Identifiers carry meaning only as exact strings. Embeddings blur 'ORD-77812' and 'ORD-77821'; keyword matching gets it right.
-
Brute-force search over 1,000,000 documents with 768-dimensional embeddings needs how many multiply-adds per query?
Show answer
C. 768 million 1,000,000 x 768 = 768,000,000. That is why very large collections use approximate nearest-neighbour indexes.
-
Why is reciprocal rank fusion a safe way to merge BM25 and vector results?
Show answer
B. It combines ranks, so the two methods' different score scales do not matter BM25 scores and cosine similarities are not comparable. RRF adds 1 / (60 + rank) from each list, which needs no calibration.
Recap
- Keyword search matches words; semantic search matches meaning by comparing embeddings.
- Embed the corpus once, embed each query with the same model, return the k nearest neighbours.
- Brute force is one dot product per document; very large collections need approximate indexes.
- Keywords still win for identifiers, names, rare terms and negation.
- Production search is usually hybrid: run both, fuse the rankings, then rerank.
Next up
- Build production LLM apps learning path