Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Mathematics for ML (only what you need, visually)

Machine Learning ML-01.01

Vectors and vector spaces

See vectors as arrows and as lists of numbers, and learn the dot product and cosine similarity that power embeddings and search.

Beginner 12 min Last reviewed

The big idea

A house listing has a size, a number of rooms and a distance to the station. Write those three numbers in order, [95, 3, 1.2], and you have a vectorAn ordered list of numbers. Geometrically, an arrow from the origin to the point those numbers describe.. That is all a vector is to a computer: an ordered list of numbers. Every input to a model, every row of a dataset and every embedding of a word or an image is a vector.

The trick that makes vectors so useful is that you can also picture them. Two numbers are a point on a page, and an arrow from the origin to that point. Three numbers are an arrow in a room. Five hundred numbers are an arrow in a space you cannot draw, but the same rules still apply: you can measure how long the arrow is, and, most importantly, how much two arrows point the same way. That last measurement is how a search engine decides which documents mean something similar to your question.

Think of vectors as recipes: a cake recipe is "300 g flour, 200 g sugar, 3 eggs". Doubling the recipe doubles every number. Combining two recipes adds them ingredient by ingredient. And two recipes are "similar" when their ingredients come in similar proportions, even if one makes a bigger cake.

How it works

Step 1: Two pictures of the same thing

The vector a = [3, 1] means "3 along the first axis, 1 along the second". As a list it is two numbers; as a picture it is an arrow from (0, 0) to (3, 1). The number of entries is the vector's dimensionThe number of entries in a vector, which is the number of axes of the space it lives in.. A house described by 3 features lives in a 3-dimensional space; a sentence embedding often lives in a space of several hundred or a few thousand dimensions.

Two vectors as arrows Vector a goes from the origin to the point 3, 1. Vector b goes from the origin to the point 1, 2.5. The angle between them is about 50 degrees. -1012345-101234 a = [3, 1] b = [1, 2.5] about 50 deg Figure 1. A vector is a list of numbers and an arrow at the same time. The angle between two arrows says how similar their directions are.

Step 2: Adding and scaling

Two operations come for free:

  • Addition works entry by entry: [3, 1] + [1, 2.5] = [4, 3.5]. As arrows, put the second one's tail at the first one's head; the sum is the arrow to where you end up.
  • Scaling multiplies every entry by the same number: 2 * [3, 1] = [6, 2]. The arrow keeps its direction and gets twice as long. A negative number flips it around.

That is the whole definition of a vector spaceA set of vectors you can add together and scale by numbers, staying inside the set. Every ML feature space is one.: a collection of vectors you can add and scale without leaving it. Every feature space in machine learning is one.

Step 3: Length, measured two ways

A normA function that measures a vector's length. The L2 norm is the straight-line length; the L1 norm adds the absolute values. measures how long a vector is.

  • The L2 norm is the straight-line length, from Pythagoras: . For a = [3, 1]: .
  • The L1 norm adds the absolute values, like walking along city blocks: .

You will meet both again as regularization penalties in Regularization: ridge, lasso, elastic net : an L2 penalty shrinks all weights a little, an L1 penalty pushes some weights to exactly zero.

Step 4: The dot product, a similarity score

The dot productMultiply two vectors entry by entry and add the results. Large and positive when they point the same way, zero when perpendicular, negative when opposite. multiplies matching entries and adds them up:

For our vectors: . Geometrically the dot product equals , where is the angle between them. So it is large when the arrows point the same way, zero when they are perpendicular, and negative when they point in opposite directions.

The dot product is everywhere in ML. A linear model's prediction is a dot product of weights and features. A neural network layer is a stack of dot products. Attention in a transformer scores words against each other with dot products.

Step 5: Cosine similarity, direction only

The dot product mixes two things: direction and length. Often you only care about direction. Two news articles about the same topic should count as similar even if one is ten times longer. Cosine similarityThe dot product of two vectors divided by the product of their lengths: the cosine of the angle between them, from -1 to 1. divides out the lengths:

It runs from 1 (same direction) through 0 (unrelated) to -1 (opposite). For our vectors: , an angle of about 50 degrees. This one formula is how semantic search, recommendation and retrieval augmented generation find "similar" items among millions of embeddings.

vectors.pyruns with python3 + numpy
import numpy as np

a = np.array([3.0, 1.0])
b = np.array([1.0, 2.5])

print("a + b      =", a + b)
print("2 * a      =", 2 * a)
print("L2 norm |a| =", np.linalg.norm(a).round(3))        # sqrt(3^2 + 1^2)
print("L1 norm     =", np.abs(a).sum())                   # |3| + |1|
print("dot a.b     =", a @ b)                             # 3*1 + 1*2.5
cos = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
print("cosine      =", cos.round(3))
print("angle (deg) =", np.degrees(np.arccos(cos)).round(1))

Output

a + b      = [4.  3.5]
2 * a      = [6. 2.]
L2 norm |a| = 3.162
L1 norm     = 4.0
dot a.b     = 5.5
cosine      = 0.646
angle (deg) = 49.8

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Drag b until it points the same way as a. Cosine goes to 1. Now make b twice as long: the dot product doubles, the cosine does not move.
  2. Make the two arrows perpendicular. The dot product drops to 0: the vectors share no direction.
  3. Switch to "projection of b on a". The green arrow is how much of b lies along a. Its length is the dot product divided by the length of a.

Trade-offs

OptionWins whenLoses when
Dot product Vectors are already normalized, or length carries meaning (a stronger signal should score higher) Long vectors dominate just by being long, such as long documents beating short relevant ones
Cosine similarity Only direction matters: comparing embeddings, documents of different lengths Magnitude really matters, such as comparing total spend
Euclidean (L2) distance Actual positions matter, as in k-means or nearest neighbours on scaled features Features are on different scales and you have not standardized them

Common mistakes

  • Comparing vectors from different embedding models. Their axes mean different things, so their similarity is meaningless.
  • Forgetting to scale features before measuring distance. A feature in dollars swamps one in years.
  • Assuming a cosine of 0.8 is "80 percent similar". It is an angle measure, and typical values depend heavily on the model.
  • Mixing up the dot product with element-wise multiplication. a * b in NumPy gives a vector; a @ b gives the dot product, a single number.

In the wild

Vectors are the common currency of modern ML. Search engines and RAG systems turn text into embedding vectors and rank documents by cosine similarity to the query (see Sentence and document embeddings ). Recommender systems represent users and items as vectors and score them with dot products (see Two-tower retrieval and embedding-based retrieval ). Vector databases exist to answer one question fast: which stored vectors point most nearly the same way as this one?

Check yourself

  1. What is the dot product of [2, 0, 1] and [1, 5, 3]?

    Show answer

    A. 5 Multiply matching entries and add: 2 x 1 + 0 x 5 + 1 x 3 = 5. The element-wise product [2, 0, 3] is a different operation.

  2. Two embedding vectors point in exactly the same direction, but one is three times longer. What is their cosine similarity?

    Show answer

    B. 1 Cosine similarity ignores length and looks only at the angle. Same direction means an angle of 0, and cos(0) = 1.

  3. You rank documents by raw dot product with a query, and very long documents always win. What is the usual fix?

    Show answer

    B. Switch to cosine similarity (or normalize the vectors) Length inflates the dot product. Normalizing both vectors, which is what cosine similarity does, compares direction only.

  4. What is the L2 norm of [6, 8]?

    Show answer

    B. 10 sqrt(6^2 + 8^2) = sqrt(36 + 64) = sqrt(100) = 10. The L1 norm would be 14.

Recap

  • A vector is an ordered list of numbers and an arrow in space at the same time.
  • Vectors add entry by entry and scale by multiplying every entry.
  • The L2 norm is straight-line length; the L1 norm adds absolute values.
  • The dot product is large when two vectors point the same way; cosine similarity divides out their lengths.
  • Cosine similarity between embeddings is how search, recommendation and RAG find similar things.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up