Machine Learning ML-01.03
Eigenvectors, eigenvalues and SVD
Find the directions a matrix only stretches, see SVD as rotate-scale-rotate, and use low-rank approximation to compress data and fine-tune models.
The big idea
When a matrix transforms the plane, most arrows get knocked off their original direction. But for many matrices there are a few special directions where the arrow keeps pointing exactly the same way and only gets longer or shorter. Those special directions are eigenvectorsA non-zero vector that a matrix only stretches or shrinks, without changing its direction: A v = lambda v., and the stretch factor along each is its eigenvalueThe factor by which a matrix stretches its eigenvector. Negative means the direction flips..
Why care? Because those directions are the matrix's natural axes. Along them, a complicated transformation becomes simple: stretch by 4 here, by 2 there. A related tool, the singular value decomposition, finds natural axes for any matrix at all, and then lets you throw away the weak ones. That single idea compresses images, powers PCA, drives recommendation systems and lets you fine-tune a huge language model by training only a tiny fraction of its numbers.
Think of kneading dough on a table. Press it with a rolling pin and it spreads mostly in one direction. That direction of greatest stretch is the dough's eigenvector, and how much it spread is the eigenvalue.
How it works
Step 1: Directions that only stretch
The defining equation is short:
"Multiplying by the matrix gives back , just scaled by ." Take
Try : . So is an eigenvector with eigenvalue 4. Try : . Another eigenvector, eigenvalue 2. Any other direction gets rotated as well as stretched.
Step 2: Eigendecomposition
If an n by n matrix has n independent eigenvectors, you can rebuild it from them:
where the columns of are the eigenvectors and is a diagonal matrix of eigenvalues. Read it as three moves: switch into the eigenvector axes, stretch along each axis, switch back. For symmetric matrices, which appear constantly in ML as covariance matrices, the eigenvectors are perpendicular and the eigenvalues are real, so is simply .
Step 3: SVD works for every matrix
Eigendecomposition needs a square matrix and does not always exist. The singular value decompositionWriting any matrix as U Sigma V transposed: a rotation, a scaling along perpendicular axes, and another rotation. (SVD) always works, for any m by n matrix:
- rotates the input space to line up with the matrix's natural input directions.
- is diagonal: it stretches each direction by a singular valueA non-negative number on the diagonal of Sigma in the SVD. Sorted from largest to smallest, they measure how much each direction matters., sorted from largest to smallest.
- rotates the result into the output space.
Rotate, scale, rotate. Every linear transformation, however messy, is just that.
Step 4: Keep the big singular values, drop the rest
Rewrite the SVD as a sum of simple pieces: Each piece is a rank-1 matrixA matrix that is one column pattern times one row pattern. It needs only m + n numbers to describe, instead of m times n.: one column pattern times one row pattern. Because the values are sorted, the first few pieces usually carry most of the matrix. Keep only the top and you get the best possible rank- approximation, a result known as the Eckart-Young theorem.
The storage savings are large. A 100 by 80 matrix holds 8,000 numbers; a rank-3 approximation needs .
import numpy as np
A = np.array([[3.0, 1.0], [1.0, 3.0]])
vals, vecs = np.linalg.eigh(A) # symmetric matrix: real eigenvalues
print("eigenvalues:", vals)
print("eigenvectors (columns):\n", vecs.round(3))
v = vecs[:, 1]
print("A @ v =", (A @ v).round(3), " = 4 * v:", (4 * v).round(3))
rng = np.random.default_rng(0)
M = rng.normal(size=(100, 3)) @ rng.normal(size=(3, 80)) + 0.05 * rng.normal(size=(100, 80))
U, s, Vt = np.linalg.svd(M, full_matrices=False)
print("top 6 singular values:", s[:6].round(1))
for k in (1, 2, 3, 10):
Mk = U[:, :k] @ np.diag(s[:k]) @ Vt[:k]
err = np.linalg.norm(M - Mk) / np.linalg.norm(M)
print(f"rank {k:2d}: relative error {err:.3f}, numbers stored {k * (100 + 80 + 1)} of {100 * 80}")Output
eigenvalues: [2. 4.]
eigenvectors (columns):
[[-0.707 0.707]
[ 0.707 0.707]]
A @ v = [2.828 2.828] = 4 * v: [2.828 2.828]
top 6 singular values: [100.1 87.2 85.2 0.9 0.9 0.9]
rank 1: relative error 0.773, numbers stored 181 of 8000
rank 2: relative error 0.541, numbers stored 362 of 8000
rank 3: relative error 0.027, numbers stored 543 of 8000
rank 10: relative error 0.023, numbers stored 1810 of 8000 That matrix was secretly rank 3 plus a little noise. Look at the singular values: three large ones, then a cliff. Keeping 3 values reproduces it to within the noise, using under 7 percent of the storage.
Try it
Try this:
- Start at k = 1. You see a blurry pattern made of one column shape times one row shape. Step up one at a time and watch details appear in order of importance.
- Find the smallest k where the house and the sun look right to you. Read the storage line: how much did you save?
- Look at the bar chart. Where the bars drop sharply is where extra values stop adding much. Real data almost always has that shape.
Going deeper
PCA is SVD of centered data
Principal component analysis ( Principal Component Analysis ) subtracts the mean from each feature and takes the SVD of the data matrix. The top right-singular vectors are the directions of greatest variance; projecting onto the first two gives the familiar 2D plots of high-dimensional data.
Recommendation by matrix factorization
A user by movie ratings matrix is mostly empty. Factorizing it into a low-rank product (users times k factors, k factors times movies) fills in the gaps: the factors end up capturing tastes like "action" or "romance". See Collaborative filtering and matrix factorization .
LoRA: fine-tuning with low rank
Fine-tuning a large model normally updates huge weight matrices. LoRA ( Parameter-efficient fine-tuning: LoRA and QLoRA ) freezes the original weights and learns the update as a product of two thin matrices, with a small rank such as 8 or 16. A 4096 by 4096 update has about 16.8 million entries; at rank 8 it needs . The bet is that the change needed for a new task is low rank, the same bet image compression makes.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Eigendecomposition | Square, especially symmetric, matrices such as covariance matrices | Non-square or defective matrices, where it does not exist |
| Full SVD | Any matrix; you need exact, numerically stable factors | Very large matrices: cost grows roughly with the cube of the size |
| Truncated or randomized SVD | Large data where only the top k components matter | You need the small components, or exact results |
Common mistakes
- Expecting every matrix to have real eigenvectors. A pure rotation by 90 degrees has none in the real plane, because it turns every direction.
- Using
np.linalg.eigon a symmetric matrix.eighis faster, more accurate and returns sorted real eigenvalues. - Forgetting to center the data before treating its SVD as PCA.
- Picking k by gut feeling. Look at the singular values or the explained variance and choose where the curve flattens.
In the wild
Low-rank structure shows up across production ML. Classic recommender systems factorize huge user-item matrices into a small number of latent factors. Search and NLP pipelines once relied heavily on latent semantic analysis, an SVD of word-document counts. Today, parameter-efficient fine-tuning with LoRA adapters is one of the most common ways teams customize large language models, because the low-rank update is small enough to train and store cheaply.
Check yourself
-
For A = [[2, 0], [0, 5]], what are the eigenvalues?
Show answer
A. 2 and 5 A diagonal matrix stretches the x-axis by 2 and the y-axis by 5, so those axes are the eigenvectors and 2 and 5 are the eigenvalues.
-
What does keeping only the top k singular values give you?
Show answer
B. The best rank-k approximation of the matrix Because singular values are sorted by importance, the top k rank-1 pieces form the best possible rank-k approximation (Eckart-Young).
-
A weight update for a 1024 by 1024 matrix is learned as B A with rank 4. Roughly how many numbers are trained?
Show answer
B. 8,192 B is 1024 by 4 and A is 4 by 1024: 4,096 + 4,096 = 8,192 numbers, under 1 percent of the full matrix.
-
Which matrix has no real eigenvectors?
Show answer
B. A rotation by 90 degrees A 90 degree rotation turns every direction, so no real vector keeps its direction. Its eigenvalues are complex.
Recap
- An eigenvector keeps its direction under the matrix; its eigenvalue is the stretch factor.
- Symmetric matrices have perpendicular eigenvectors and real eigenvalues: their natural axes.
- SVD writes any matrix as rotate, scale, rotate: A = U Sigma V transposed.
- Keeping the top k singular values gives the best rank-k approximation with far less storage.
- PCA, matrix factorization recommenders and LoRA fine-tuning are all low-rank ideas.
Next up
- From notebook to production learning path