Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Representations: Embeddings and Vector Spaces

AI AI-04.03

Semantic search basics

Keyword versus semantic search: embed a corpus, find nearest neighbours for a query, and see when exact keywords still win and why hybrid search exists.

Beginner 10 min Last reviewed

The big idea

A customer types "I want my money back" into your help centre. The right article is titled "Refund policy" and says plans "can be refunded within 14 days". A classic search box finds nothing: none of the words "want", "money" or "back" appear in the article. The customer opens a support ticket, or leaves.

Keyword searchSearch that finds documents by matching the words in the query, typically ranking them with a scoring function such as TF-IDF or BM25. asks "which documents contain these words?". Semantic searchSearch that embeds the query and the documents and returns the documents whose embeddings are nearest to the query's, so it matches meaning rather than exact words. asks "which documents mean something close to this?". It embeds the query, then returns the documents whose embeddings are nearest. "Money back" and "refund" use different words but land in the same neighbourhood, so the refund article comes first.

Think of asking for a book in two shops. In the first, a clerk searches the catalogue for the exact words you said. In the second, a well-read bookseller listens to what you are after and walks you to the right shelf, even if you misremembered the title. The second is usually more helpful, until you ask for the book with ISBN 978-0-13-468599-1 and the bookseller hands you something "similar".

How it works

Semantic search runs in two phases. It is also the first half of every retrieval augmented generation system ( Why RAG? ).

Semantic search: embed the corpus offline, embed the query online, return the nearest neighbours Top row, offline: documents go through the embedding model and the resulting vectors are stored in an index. Bottom row, online: the query goes through the same embedding model, its vector is compared with every stored vector, and the top k nearest documents are returned. The online path glows. compare documents embedding model vector index one vector per document query "money back" same model top k nearest Refund policy, ... offline, once online, per query Figure 1. Offline (top), every document is embedded once and stored. Online (bottom, glowing), the query is embedded with the same model and the nearest stored vectors come back as results.

Step 1: Embed the corpus

Run every document (in practice every chunk of a document, see Document ingestion and chunking ) through the embedding model and store the vectors, normalised to length 1, together with an ID that points back to the text. This happens once, and again only for documents that change.

Step 2: Embed the query

When a query arrives, embed it with the same model. A different model would put it in a different space where distances are meaningless.

Step 3: Find the nearest neighbours

Compute the similarity between the query vector and the stored vectors and return the top . This is k-nearest-neighbour searchFinding the k stored items whose vectors are closest to a query vector, by cosine similarity or distance.. The simplest way is brute force: one dot product per document, then keep the highest. With 100,000 documents of 384 dimensions that is 38.4 million multiply-adds per query, which a laptop does in milliseconds. At tens of millions of documents it gets slow, and approximate indexes such as HNSW take over, trading a little accuracy for a huge speed-up ( Vector databases and ANN indexes ).

Step 4: Return the documents

Map the IDs back to text and show them, or, in a RAG system, paste them into the model's prompt ( Why RAG? ).

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

The semantic column uses a toy embedding model, a small table of related words, so it runs in your browser. Real models learn far richer relationships, but they fail in the same places.

Try this:

  1. Pick I want my money back. Keyword search returns nothing: no query word appears in any article. Semantic search finds Refund policy. This is the vocabulary mismatch semantic search was invented for.
  2. Pick E1042. Keyword search goes straight to the right article, because the code appears in it word for word. Semantic search still puts it first, but only just: other articles, including the one for a different error code, E1024, score close behind. To an embedding, a rare code is a few meaningless fragments, so E1042 and E1024 look almost alike.
  3. Back to where is my package. Keyword search matches "package" and returns Track your parcel. Semantic search prefers Shipping times, which is about the same topic but answers a different question. The hybrid column fuses the two lists and puts Track your parcel first. Now type your own queries, such as refund or how long does delivery take.

The core of semantic search is a few lines of numpy:

brute-force-knn.pyruns with python3 + numpy
# Semantic search in its simplest form: embed the corpus once, then for each
# query compute similarity to every vector and keep the top k.
import numpy as np

rng = np.random.default_rng(42)
n_docs, dims, k = 100_000, 384, 5

# Stand-ins for document embeddings, normalised to length 1 (offline, once).
docs = rng.normal(size=(n_docs, dims)).astype(np.float32)
docs /= np.linalg.norm(docs, axis=1, keepdims=True)

# The query is a noisy copy of document 1234, so we know the answer.
query = docs[1234] + 0.05 * rng.normal(size=dims).astype(np.float32)
query /= np.linalg.norm(query)

scores = docs @ query                  # one dot product per document
top = np.argpartition(-scores, k)[:k]  # the k best, unordered
top = top[np.argsort(-scores[top])]    # order just those k

for rank, i in enumerate(top, 1):
    print(f"{rank}. doc {i:>6}  cosine {scores[i]:.3f}")

work = n_docs * dims
print(f"\nwork per query: {n_docs:,} x {dims} = {work:,} multiply-adds")
print(f"index memory: {docs.nbytes / 1e6:.1f} MB")

Output

1. doc   1234  cosine 0.727
2. doc   5597  cosine 0.266
3. doc   6214  cosine 0.218
4. doc  19843  cosine 0.208
5. doc  19352  cosine 0.197

work per query: 100,000 x 384 = 38,400,000 multiply-adds
index memory: 153.6 MB

The query was built as a noisy copy of document 1234, so the right answer is known, and it comes first with a cosine of 0.727 while the best wrong document scores 0.266. argpartition finds the top without sorting all 100,000 scores.

Going deeper

When keywords still win

Keyword search is not obsolete. It is better whenever exact strings carry the meaning:

  • identifiers: error codes, SKUs, order numbers, ticket IDs, legal clause numbers;
  • names and rare terms the embedding model never learned well;
  • precise phrases, such as quoted text or a function name in code;
  • negation and specificity: "laptops without a touch screen" can retrieve touch-screen laptops semantically.

It is also cheap, needs no model, and explains itself: you can see which words matched. The classic ranking function is BM25, which rewards documents that contain rare query terms several times without letting long documents win by sheer length. Inverted indexes and full-text search covers how search engines build its inverted index.

Because the two methods fail in different places, production systems run both and merge the lists. A simple and robust merge is reciprocal rank fusionA way to merge ranked lists: each document scores the sum of 1 / (c + rank) over the lists it appears in, with c usually 60. It needs no score calibration. (RRF): each document gets over the lists it appears in. It uses ranks, not scores, so it does not care that BM25 scores and cosine similarities live on different scales. A reranker then often reorders the top few ( Hybrid search and reranking ).

Do not judge search by trying five queries by hand. Collect real queries, mark which documents are correct, and measure recall at k (did the right document appear in the top k?) for keyword, semantic and hybrid. The answer is often surprising, and it differs per domain. Metrics are covered in Evaluating RAG .

Trade-offs

OptionWins whenLoses when
Keyword (BM25) Exact identifiers, names, rare terms; cheap; explainable Synonyms, paraphrases, typos in natural language, questions in other words
Semantic (embeddings) Paraphrases, natural-language questions, cross-language with a multilingual model Exact codes and names, negation; needs an embedding model and a vector index
Hybrid (both + fusion) General-purpose search over mixed queries; the usual production choice Two indexes to build and keep in sync; more moving parts to tune

Common mistakes

  • Replacing keyword search entirely and then discovering that product codes and names no longer work.
  • Embedding queries and documents with different models, or with a model version that changed since indexing.
  • Embedding whole long documents as one vector instead of chunks, so the right paragraph is drowned in an average.
  • Merging BM25 scores and cosine scores by adding them directly, although they are on different scales. Use rank fusion or calibrate.
  • Assuming the nearest document is relevant. Nearest neighbour search always returns something, even for queries your corpus cannot answer.

In the wild

Where you will see this: help-centre and documentation search that understands questions, e-commerce search ("comfortable shoes for standing all day"), code search across a repository, finding duplicate support tickets or bug reports, "related articles" boxes, and the retrieval step of every RAG chatbot. Most web-scale and enterprise search engines now combine a keyword index with a vector index, then rerank.

Check yourself

  1. The query 'how do I get my money back' finds nothing in keyword search but works in semantic search. Why?

    Show answer

    B. The relevant article uses different words such as 'refund', and only semantic search matches meaning This is vocabulary mismatch. Keyword search needs shared words; semantic search compares meanings in embedding space.

  2. Users search for exact order numbers like 'ORD-77812'. Which approach should handle these?

    Show answer

    B. Keyword (exact) search, alone or inside a hybrid Identifiers carry meaning only as exact strings. Embeddings blur 'ORD-77812' and 'ORD-77821'; keyword matching gets it right.

  3. Brute-force search over 1,000,000 documents with 768-dimensional embeddings needs how many multiply-adds per query?

    Show answer

    C. 768 million 1,000,000 x 768 = 768,000,000. That is why very large collections use approximate nearest-neighbour indexes.

  4. Why is reciprocal rank fusion a safe way to merge BM25 and vector results?

    Show answer

    B. It combines ranks, so the two methods' different score scales do not matter BM25 scores and cosine similarities are not comparable. RRF adds 1 / (60 + rank) from each list, which needs no calibration.

Recap

  • Keyword search matches words; semantic search matches meaning by comparing embeddings.
  • Embed the corpus once, embed each query with the same model, return the k nearest neighbours.
  • Brute force is one dot product per document; very large collections need approximate indexes.
  • Keywords still win for identifiers, names, rare terms and negation.
  • Production search is usually hybrid: run both, fuse the rankings, then rerank.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up