Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Representations: Embeddings and Vector Spaces

AI AI-04.01

From symbols to vectors

Why one-hot symbols cannot express similarity, how dense embeddings turn meaning into geometry, and what gets embedded: words, sentences, images, users.

Beginner 10 min Last reviewed

The big idea

Everything in the first three modules of this track worked with symbols. A state was named Ashford, a fact was Parent(ann, bob), a variable was Rain. Symbols are exact, but they are also mute: the name cat says nothing about the name kitten. To a symbolic system they are as different as cat and carburettor.

Modern AI replaces names with positions. Each word, sentence, image or user becomes a list of numbers, a point in a space, placed so that things with similar meaning sit close together. That list is an embeddingA dense vector of numbers that represents an item (word, sentence, image, user) so that similar items have nearby vectors.. Once meaning is geometry, "what is similar to this?" stops being a philosophical question and becomes arithmetic: measure the distance.

Think of a supermarket. Nobody gives the milk a meaningless locker number. Milk sits next to yoghurt and cheese, bread is near the bakery counter, and if you are holding one item, the things you probably also want are within arm's reach. An embedding is a supermarket layout for meaning, computed instead of planned.

How it works

Step 1: One-hot vectors, symbols in disguise

The simplest way to turn a word into numbers is a one-hot vectorA vector with one slot per vocabulary item, all zeros except a single 1 in the slot of the item it represents.. Give each word in the vocabulary a slot; the vector for a word has a 1 in its own slot and 0 everywhere else. With a vocabulary of 50,000 words, every word is a 50,000-long vector containing a single 1.

It looks numeric, but it is still just a name. The dot product of any two different one-hot vectors is 0, so their cosine similarity is 0. cat is exactly as far from kitten as from truck. Nothing learned about one word transfers to any other, and the vectors grow with the vocabulary.

One-hot versus dense representations of the same three words On the left, cat, kitten and truck are one-hot vectors: each has a single 1 in a different slot of a very long vector, and every pair has similarity 0. On the right, the same words are short dense vectors; cat and kitten have similar numbers and similarity 0.87, while kitten and truck have similarity 0.08. The cat to kitten link on the right glows. 0 0 0.87 0.08 cat [1, 0, 0, ... 0] kitten [0, 1, 0, ... 0] truck [0, 0, 1, ... 0] cat [1.0, 0.9, 0.1, 0.0, 0.2] kitten [1.0, 0.9, 0.9, 0.0, 0.1] truck [0.0, 0.0, 0.1, 1.0, 0.9] one-hot: 50,000 slots dense: a few hundred numbers Figure 1. One-hot (left): every pair of words is equally unrelated, similarity 0. Dense (right): the numbers themselves carry meaning, so cat and kitten come out similar (glowing) and kitten and truck do not. Dense values are the toy features from the code below.

Step 2: Dense vectors, where every number carries meaning

A dense vectorA vector whose entries are mostly non-zero real numbers, so information is spread across all dimensions; the opposite of a sparse one-hot vector. uses far fewer dimensions, a few hundred to a few thousand, and every entry is a real number. Imagine, just for intuition, that the dimensions meant "is an animal", "is a pet", "is young", "is a vehicle" and "is large". Then cat and kitten differ only in the "young" slot, and their vectors point in nearly the same direction.

Real embeddings are not hand-designed like that. Their dimensions are learned, and individual numbers rarely mean anything a human would name. What matters is the overall geometry: items used in similar ways end up near each other, and some directions in the space line up with consistent differences, such as young versus adult or singular versus plural.

Words as points in a 2D embedding space, grouped by meaning Thirteen words plotted as points. Animals (cat, kitten, dog, hamster) form a glowing cluster at the top left; vehicles sit at the top right, food at the bottom left and royalty at the bottom right. Distance between points reflects similarity of meaning. cat kitten dog hamster car truck bus apple banana bread king queen prince animals vehicles food royalty Figure 2. A toy 2D picture of an embedding space. Items with similar meaning cluster together (glowing: the animals). Real spaces have hundreds of dimensions; 2D plots like this are projections that keep only part of the structure.

Step 3: Where the numbers come from

Nobody types these vectors in. They are learned from data, guided by one powerful hint known as the distributional hypothesisThe idea that words appearing in similar contexts tend to have similar meanings, the principle behind learned word embeddings.: words that appear in similar contexts have similar meanings. "The ___ purred on the sofa" fits cat and kitten but not truck. A model trained to predict a word from its neighbours (or the neighbours from the word) is pushed to give words with interchangeable contexts similar vectors, because that makes its predictions easier. Classic word-embedding methods are covered in the ML track in Word embeddings ; in modern systems the embeddings are learned inside larger neural networks, as you will see in Embeddings and positional encoding .

Step 4: Not just words

Anything you can feed to a neural network can be embedded, and the same geometry tricks apply:

  • Sentences and documents: one vector for a whole passage, so a question can be matched to the paragraph that answers it. Next lesson: Sentence and document embeddings .
  • Images: photos of beaches cluster together whatever the pixels. Some models put images and text in one shared space, so the caption "a dog on a beach" lands near matching photos ( Vision-language models in practice ).
  • Users and products: a recommender learns a vector per user and per item so that users sit near the items they will like.
  • Code, molecules, songs, support tickets: anything with a notion of similarity.

This is why embeddings are often called the common currency of modern AI. Search, recommendation, clustering, deduplication, classification and retrieval for language models all start the same way: embed, then compare.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Drag the query into the middle of the animal cluster. The top 3 are all animals, whichever measure you pick. Similar meaning, nearby points.
  2. Drag the query to a spot between the animals and the vehicles, say near the top centre. Set k to 5. Neighbours from both groups appear: an ambiguous query sits between meanings.
  3. Switch to cosine and drag the query toward the centre, then slowly outward along the same direction. With cosine only the direction from the origin matters, so the ranking hardly changes; with Euclidean distance it does. Lesson Sentence and document embeddings explains why embedding systems usually normalise vectors so the two agree.

The same comparison in code, with the five made-up features from Figure 1:

one-hot-vs-dense.pyruns with python3 + numpy
# One-hot vs dense vectors: which one knows that a kitten is like a cat?
import numpy as np

words = ["cat", "kitten", "dog", "puppy", "car", "truck"]

# One-hot: one dimension per word, a single 1 in its own slot.
one_hot = np.eye(len(words))

# Dense: a few made-up features per word, for illustration only.
# columns: animal, pet, young, vehicle, size
dense = np.array([
    [1.0, 0.9, 0.1, 0.0, 0.2],  # cat
    [1.0, 0.9, 0.9, 0.0, 0.1],  # kitten
    [1.0, 0.9, 0.1, 0.0, 0.4],  # dog
    [1.0, 0.9, 0.9, 0.0, 0.2],  # puppy
    [0.0, 0.0, 0.1, 1.0, 0.6],  # car
    [0.0, 0.0, 0.1, 1.0, 0.9],  # truck
])

def cos(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

def vec(table, w):
    return table[words.index(w)]

pairs = [("cat", "kitten"), ("cat", "dog"), ("cat", "truck"), ("car", "truck")]
for a, b in pairs:
    oh = cos(vec(one_hot, a), vec(one_hot, b))
    de = cos(vec(dense, a), vec(dense, b))
    print(f"{a:>6} vs {b:<6}  one-hot {oh:.2f}   dense {de:.2f}")

# Directions carry meaning: kitten - cat + dog lands nearest to ...
q = vec(dense, "kitten") - vec(dense, "cat") + vec(dense, "dog")
others = [w for w in words if w not in ("kitten", "cat", "dog")]
scores = {w: cos(q, vec(dense, w)) for w in others}
best = max(scores, key=scores.get)
print("kitten - cat + dog is closest to:", best, f"({scores[best]:.2f})")

Output

   cat vs kitten  one-hot 0.00   dense 0.87
   cat vs dog     one-hot 0.00   dense 0.99
   cat vs truck   one-hot 0.00   dense 0.10
   car vs truck   one-hot 0.00   dense 0.98
kitten - cat + dog is closest to: puppy (1.00)

Two things to notice. One-hot similarity is 0.00 for every pair, related or not. And the dense vectors support arithmetic: kitten - cat + dog lands on puppy, because the difference between kitten and cat is the "young" direction.

Going deeper

Why "nearby" is the whole trick

A dense space gives a model generalisation for free. If a spam filter learns that messages mentioning "prize" are suspicious, and "reward" sits next to "prize", it has partly learned about "reward" too. With one-hot inputs, every word must be learned separately from its own examples.

Directions, not just distances

In learned word spaces, consistent relationships often appear as consistent directions. The best-known example comes from word vectors trained on news text, where the vector for "King" minus "Man" plus "Woman" lands closest to "Queen". Treat this as a tendency, not a law: analogies work for some relations and fail for many others, and they also reproduce whatever biases were in the training text, such as gendered job titles.

How many dimensions?

More dimensions can hold finer distinctions but cost memory and search time: one million items with 1,024 float32 numbers each take 1,000,000 x 1,024 x 4 bytes, about 4 GB. Fewer dimensions are cheaper and can be good enough. The trade-off returns in Sentence and document embeddings and Embedding and indexing .

Trade-offs

OptionWins whenLoses when
Symbols / one-hot Exact identity matters: IDs, product codes, rules and logic; fully interpretable No notion of similarity; vectors grow with the vocabulary; nothing generalises
Hand-designed features Small domains where experts know what matters Expensive to design; miss distinctions nobody thought of
Learned dense embeddings Similarity, generalisation, search by meaning across text, images and users Opaque dimensions; inherit data biases; need a model to compute; can blur exact distinctions

Common mistakes

  • Expecting individual embedding dimensions to have human meanings. Meaning lives in distances and directions, not in slot 37.
  • Comparing vectors produced by two different models. Each model builds its own space; a distance between spaces means nothing.
  • Treating 2D plots of embeddings as the truth. Projections squash hundreds of dimensions into two and can invent or hide clusters.
  • Assuming similar means correct or equal. "Refund allowed" and "refund not allowed" can sit very close together.
  • Forgetting that embeddings learn the associations in their training data, including unwanted ones.

In the wild

Mikolov et al., 2013 (paper)

Efficient Estimation of Word Representations in Vector Space (opens in a new tab)

This paper proposed two simple architectures for learning continuous word vectors from very large text collections, and reported learning high quality vectors from a 1.6 billion word dataset in less than a day. It recalls the word offset technique, where vector("King") - vector("Man") + vector("Woman") is closest to the vector for Queen, and measures how well its vectors capture such syntactic and semantic relationships. Efficient training like this is what made learned word embeddings a standard building block.

Check yourself

  1. What is the cosine similarity between the one-hot vectors for 'cat' and 'kitten'?

    Show answer

    B. 0 Different one-hot vectors have their single 1 in different slots, so their dot product, and therefore their cosine similarity, is 0. One-hot vectors cannot express that two words are related.

  2. Why do learned embeddings put 'cat' and 'kitten' close together?

    Show answer

    B. They appear in similar contexts, and training rewards similar vectors for interchangeable words This is the distributional hypothesis at work: the training objective makes words with similar contexts get similar vectors. Shared spelling plays no role.

  3. A recommender keeps 10 million products with 512 float32 numbers each. Roughly how much memory do the vectors need?

    Show answer

    C. About 20 GB 10,000,000 x 512 x 4 bytes = 20,480,000,000 bytes, about 20 GB. Dimension count is a real cost driver.

  4. You need to match exact part numbers like 'XJ-4471'. Which representation fits best?

    Show answer

    B. Exact symbols or keyword matching Identifiers need exact matching; embeddings may place 'XJ-4471' near 'XJ-4417'. Semantic search systems keep keyword matching for this reason (AI-04.03).

Recap

  • Symbols and one-hot vectors name things but cannot say that two things are similar.
  • Embeddings are dense vectors placed so that distance and direction reflect meaning.
  • They are learned from data, largely from the contexts things appear in.
  • Words, sentences, images, users and products can all be embedded, then compared with the same arithmetic.
  • Embeddings generalise well but are opaque, model-specific and inherit their data's biases.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up