Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Representations: Embeddings and Vector Spaces

AI AI-04.02

Sentence and document embeddings

How embedding models turn a whole sentence into one vector, why you normalise, what dimensions cost, and how to choose a model with MTEB and your own data.

Intermediate 12 min Last reviewed

The big idea

Word vectors tell you that password is close to login. But users do not search with single words. They type "I forgot my login credentials", and the help article is titled "How do I reset my password?". Not one word in common, yet they mean the same thing. To match them you need one vector for the whole sentence, placed so that sentences with the same meaning land close together whatever words they use.

That is what a text embedding modelA model that maps a whole piece of text (sentence, paragraph, document) to a single fixed-length vector so that texts with similar meaning have high cosine similarity. does. It reads the full text, takes word order and context into account, and outputs a single fixed-length vector. Compare two vectors with cosine similarity and you get a meaning score between texts.

Think of a librarian who has read every book and gives each one a spot on a vast map, placed by what it is about. You hand over your question, they place it on the same map, and you walk to the nearest shelves. The map does not care whether you said "money back" or "refund".

How it works

Step 1: From tokens to one vector

An embedding model is usually a transformer encoder (the architecture is unpacked in Encoder, decoder and decoder-only models ). Text goes in as tokens, and every layer refines a vector per token using the tokens around it, so "bank" in "river bank" ends up different from "bank" in "bank account". At the end there are as many vectors as tokens, and you need one. A pooling step combines them, most often by averaging (mean pooling) or by taking a special summary token's vector. Finally the vector is usually normalised to length 1.

The path from a sentence to a single normalised embedding A sentence is split into tokens, each token gets a context-aware vector from the transformer encoder, the token vectors are pooled into one vector by averaging, and the result is normalised to length 1. The main path glows. sentence "I forgot my login" tokens I | forgot | my | login encoder a vector per token pooling mean of token vectors normalise length 1, e.g. 768 numbers Figure 1. A text embedding model: tokens become context-aware vectors, pooling averages them into one, and normalising scales it to length 1 so that a plain dot product equals cosine similarity.

How does the model learn to put paraphrases together? A pretrained language model gives decent token vectors but poor sentence vectors on its own. Embedding models are then trained with contrastive learningTraining that pulls the embeddings of matching pairs (a question and its answer, two paraphrases) together and pushes non-matching pairs apart. on large collections of pairs that should match: a question and its answer, a title and its article, two paraphrases. Each step pulls the matching pairs together and pushes the other texts in the batch apart. The geometry you want is literally what the training optimises.

Step 2: Cosine similarity and normalisation

Similarity between two embeddings and is almost always cosine similarityThe cosine of the angle between two vectors: their dot product divided by the product of their lengths. 1 means same direction, 0 unrelated, -1 opposite.:

It ignores length and compares direction only. If every vector is normalised to length 1 in advance, the denominator is 1 and cosine is just the dot product, which is the cheapest operation a vector database can run. For unit vectors, Euclidean distance carries the same information too:

So once you normalise, "highest cosine", "highest dot product" and "smallest distance" give the same ranking. Without normalisation, a raw dot product favours long vectors regardless of meaning, which is a classic source of strange results. The code below checks both claims.

Scores are relative, not absolute. One model might give unrelated sentences 0.1 and paraphrases 0.9; another might squeeze everything into 0.7 to 0.95. A threshold such as "0.8 means relevant" only makes sense for the model you calibrated it on.

Step 3: Bi-encoders and cross-encoders

An embedding model encodes each text separately, so it is called a bi-encoderA model that embeds each text independently into a vector; texts are compared by vector similarity, so documents can be embedded once in advance.. Its superpower is that documents can be embedded once, ahead of time, and a query costs one embedding plus fast vector comparisons. A cross-encoderA model that reads a query and a document together and outputs a relevance score. More accurate than a bi-encoder but must run once per pair. instead reads the query and a document together and outputs a score. Seeing both texts at once makes it more accurate, but it must run once for every pair, so it is too slow to scan a large collection. Production systems use both: a bi-encoder to fetch candidates, a cross-encoder to rerank the best few ( Hybrid search and reranking ).

Bi-encoder versus cross-encoder Left: the query and the document go through the encoder separately, each producing a vector, and the vectors are compared with cosine similarity; document vectors can be precomputed. Right: the query and document are fed together into one model that outputs a relevance score directly, which is more accurate but must run for every pair. query document embedded in advance encoder encoder cosine fast query + document read together cross-encoder relevance score slow, one run per pair Figure 2. Bi-encoder (left, glowing): embed once, compare cheaply, so it scales to millions of documents. Cross-encoder (right): reads both texts together for a sharper score, but needs one model run per query-document pair.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

The meaning score here comes from a deliberately tiny toy model that runs in your browser: a hand-written table maps related words to shared concepts (password, login and credentials all count as "account access"), plus a small slot for each exact word. A real embedding model learns such groupings, and far subtler ones, from data. The behaviour is what matters.

Try this:

  1. Start with paraphrase, no shared words. Word overlap is 0.00 because no content word is shared, yet the meaning score is 0.84. This is the case keyword search misses.
  2. Pick shared words, different meaning ("reset my password" vs "reset my router"). Word overlap rises to 0.33, higher than for the true paraphrase, while the meaning score drops to 0.62. Ranking by overlap would put the wrong answer first.
  3. Pick opposite opinions: "I love this product" vs "I hate this product" scores 0.89. Real embedding models often behave similarly: they capture what a text is about more strongly than whether it agrees or disagrees. Do not use raw similarity to detect contradiction or sentiment.

Normalisation and the cost of dimensions, checked in code:

normalize-and-cost.pyruns with python3 + numpy
# Cosine, dot product and distance agree once vectors are normalised.
# Then: what do dimensions cost at scale?
import numpy as np

rng = np.random.default_rng(0)
docs = rng.normal(size=(1000, 384))       # stand-ins for 1,000 embeddings
docs *= rng.uniform(0.5, 3.0, size=(1000, 1))  # with very different lengths
query = rng.normal(size=384)

def unit(x):
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

cosine = unit(docs) @ unit(query)
dot_raw = docs @ query                     # raw dot: longer vectors win
dot_unit = unit(docs) @ unit(query)        # dot of unit vectors = cosine
dist_unit = np.linalg.norm(unit(docs) - unit(query), axis=1)

top = lambda s: [int(i) for i in np.argsort(-s)[:5]]
print("top 5 by cosine        ", top(cosine))
print("top 5 by raw dot       ", top(dot_raw))
print("top 5 by unit dot      ", top(dot_unit))
print("top 5 by unit distance ", top(-dist_unit))
# for unit vectors: |a - b|^2 = 2 - 2 cos(a, b)
print("identity holds:", np.allclose(dist_unit**2, 2 - 2 * cosine))

print("\nmemory for 10 million vectors")
for dims in (384, 768, 1024, 3072):
    f32 = 10_000_000 * dims * 4 / 1e9
    i8 = 10_000_000 * dims * 1 / 1e9
    print(f"  {dims:>5} dims   float32 {f32:6.1f} GB   int8 {i8:5.1f} GB")

Output

top 5 by cosine         [257, 85, 474, 913, 883]
top 5 by raw dot        [219, 133, 567, 674, 100]
top 5 by unit dot       [257, 85, 474, 913, 883]
top 5 by unit distance  [257, 85, 474, 913, 883]
identity holds: True

memory for 10 million vectors
    384 dims   float32   15.4 GB   int8   3.8 GB
    768 dims   float32   30.7 GB   int8   7.7 GB
   1024 dims   float32   41.0 GB   int8  10.2 GB
   3072 dims   float32  122.9 GB   int8  30.7 GB

The first four lines show that cosine, dot product of unit vectors and distance between unit vectors rank the same five documents, while the raw dot product, fooled by vector length, picks a different five.

Going deeper

Dimensions versus cost

Common embedding sizes range from a few hundred to a few thousand numbers. More dimensions can store finer distinctions, but storage and search time grow linearly (storage precision and re-embedding costs for a real index are in embedding and indexing ):

Memory needed to store 10 million embeddings at different sizes Bars for 10 million vectors: 384 dimensions in float32 need 15.4 GB, 768 need 30.7 GB, 1024 need 41.0 GB, and 3072 need 122.9 GB. Quantising 3072 dimensions to int8 brings it down to 30.7 GB. GB for 10 million vectors 384 dims, float32 15.4 GB 768 dims, float32 30.7 GB 1024 dims, float32 41.0 GB 3072 dims, float32 122.9 GB 3072 dims, int8 30.7 GB Figure 3. Memory grows linearly with dimensions: 10 million x dims x 4 bytes in float32. Quantising to 1 byte per number (int8, green) cuts it by 4x, usually at a small cost in accuracy. Numbers from the code above.

Two techniques soften the trade-off. Quantisation stores each number in fewer bits (int8 or even 1 bit), as covered for vector indexes in Vector databases and ANN indexes . Some models are trained so that the first dimensions carry the most information (an approach called Matryoshka representation learning), which lets you cut a vector to its first few hundred dimensions and keep most of its quality. Only truncate a model trained for it.

Long documents

Embedding models have a maximum input length, and one vector for a 40-page document blurs everything into an average. That is why documents are split into chunks of a few hundred tokens and each chunk is embedded separately ( Document ingestion and chunking ). Also check whether your model expects instructions or prefixes: some are trained with different prefixes for queries and passages, and leaving them out quietly lowers quality.

Multilingual and multimodal

A multilingual model maps text in different languages into one shared space, so a German question can retrieve an English answer. A multimodal model maps images and text into one space, so the text "red running shoes" lands near photos of red running shoes ( Vision-language models in practice ). Support varies a lot between models, and quality usually drops for languages that were rare in training.

Choosing a model

Public benchmarks are the starting point. The Massive Text Embedding Benchmark (MTEB) scores models across many task types (retrieval, clustering, classification, reranking, semantic similarity and more) in many languages. Read it by the task you need, usually retrieval, rather than the overall average, and note model size, dimensions, maximum input length and licence. Then run the top few on your own data: 50 to 200 real queries with known correct documents, scored with recall at k ( Evaluating RAG ). Domain text such as legal clauses, medical notes or code can reorder a leaderboard completely.

Trade-offs

OptionWins whenLoses when
Small model, few dimensions Large corpora, tight latency or memory budgets, on-device Misses fine distinctions; weaker on specialised domains
Large model, many dimensions Hard retrieval tasks where quality matters most Higher embedding cost, storage and search time
Hosted embedding API No infrastructure, strong quality, quick start Per-token cost, data leaves your network, model can be retired
Self-hosted open model Control, privacy, fixed cost at scale, can fine-tune You run GPUs or CPUs and handle upgrades

Common mistakes

  • Mixing embeddings from two models, or two versions of one model, in the same index. Changing models means re-embedding everything.
  • Using a raw dot product on unnormalised vectors and getting results dominated by vector length.
  • Reusing a similarity threshold from a blog post. Score ranges differ per model; calibrate on your own data.
  • Picking the model at the top of the overall leaderboard without testing on your queries and documents.
  • Embedding whole long documents into one vector instead of chunking, or exceeding the model's input limit and silently truncating.

In the wild

Reimers and Gurevych, 2019 (paper)

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (opens in a new tab)

Comparing sentences with a model that reads both at once (a cross-encoder) meant that finding the most similar pair among 10,000 sentences took about 50 million inference computations, roughly 65 hours. Sentence-BERT fine-tuned the network with siamese and triplet structures so that each sentence gets its own embedding, compared with cosine similarity. The paper reports this cut the same search to about 5 seconds while keeping the accuracy, which is the bi-encoder idea behind modern embedding models.

Muennighoff et al. (benchmark)

MTEB: Massive Text Embedding Benchmark and leaderboard (opens in a new tab)

The MTEB paper argues that embeddings were usually evaluated on a narrow set of datasets, so it built a benchmark spanning 8 task types, 58 datasets and 112 languages at launch, and benchmarked 33 models. Its finding was that no single embedding method dominated across all tasks. The public leaderboard linked here keeps that comparison running for new models, which is why you should read it task by task.

Check yourself

  1. All your embeddings are normalised to length 1. Which ranking gives the same order as cosine similarity?

    Show answer

    B. Dot product, and also smallest Euclidean distance For unit vectors the dot product equals cosine, and squared distance equals 2 - 2 cos, so all three rank identically.

  2. Why are bi-encoders used for the first retrieval step over millions of documents?

    Show answer

    B. Document vectors can be computed once in advance, so a query needs one embedding and fast comparisons A cross-encoder must run on every query-document pair. A bi-encoder embeds documents ahead of time, which is what makes search over large collections fast.

  3. You switch to a new embedding model for queries but keep the old document vectors. What happens?

    Show answer

    B. Results become close to random because the two models use different spaces Each model builds its own space. Query and document vectors must come from the same model, so a model change means re-embedding the corpus.

  4. Two models rate the same pair 0.62 and 0.91. What can you conclude?

    Show answer

    B. Nothing about quality: score scales differ between models Absolute scores are not comparable across models. Compare models by ranking quality on your own labelled queries, such as recall at k.

  5. Storage for 1024-dimensional float32 vectors is too large. Which change cuts it 4x while keeping all dimensions?

    Show answer

    B. Quantising to int8 int8 uses 1 byte per number instead of 4, a 4x cut, usually with a small accuracy loss.

Recap

  • Text embedding models map a whole sentence or passage to one vector so that similar meanings get high cosine similarity.
  • Tokens become context-aware vectors, pooling makes one vector, and normalising makes dot product equal cosine.
  • Bi-encoders embed texts separately and scale to millions; cross-encoders read pairs together and are best for reranking.
  • Dimensions trade quality for memory and speed; quantisation and truncatable models soften the cost.
  • Use MTEB by task as a shortlist, then decide with an evaluation on your own queries and documents.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up