Principal Component Analysis

The big idea

A dataset with 64 features is a cloud of points in 64 dimensions. You cannot look at it, many models slow down on it, and a lot of those dimensions are near copies of each other. Most real data is like that: the points do not fill the space, they lie close to a much flatter shape inside it.

Principal Component AnalysisPrincipal Component Analysis: a linear method that finds orthogonal directions of maximum variance in the data and projects the data onto the top few of them. (PCA) finds that flatter shape when it is flat in a straight-line sense. Think of photographing a long, thin baguette. Shot end-on, it is a circle and you learn nothing. Shot from the side, one picture shows almost everything about it. PCA picks the camera angle that shows the most spread, then the next best angle at right angles to the first, and so on.

You keep the first few angles and drop the rest. The explained variance ratioThe fraction of the data's total variance that lies along one principal component. Summed over the kept components, it tells you how much spread the reduced data retains. tells you exactly how much of the spread you kept. PCA is used for compression, for plotting high-dimensional data, for removing noise and for speeding up other models.

How it works

Step 1: Centre (and usually scale) the data

Subtract each feature's mean so the cloud sits around the origin. PCA measures spread around the centre, so without this step the first component just points at the mean. If features use different units (metres and grams, say), also divide by each feature's standard deviation; otherwise the feature with the biggest numbers dominates every component.

Step 2: Find the direction of maximum variance

Pick a unit vector u. Each centred point x lands on the line through u at position u . x, its projectionThe coordinate of a point along a direction: the dot product of the point with a unit vector, which drops the point onto that line at a right angle.. The variance of those positions measures how much spread that direction captures:

where C is the covariance matrixA d by d matrix whose entry (j, k) is the average product of centred features j and k. The diagonal holds each feature's variance; off-diagonal entries show how features move together. of the data. The u that makes this as large as possible is the eigenvector of C with the largest eigenvalue, and the eigenvalue is the variance it captures. That eigenvector is the first principal componentOne of the orthogonal directions found by PCA, ordered by how much variance the data has along it. They are the eigenvectors of the covariance matrix. (PC1).

The rest follow the same rule: PC2 is the direction of most remaining variance at right angles to PC1, and so on. All of them are eigenvectors of C, sorted by eigenvalue. In practice libraries compute them with the SVD of the centred data matrix, which is faster and more stable than forming C.

Step 3: Read the explained variance ratio

Each eigenvalue divided by their sum is that component's explained variance ratio. On the 500 tilted 2D points in the code, the eigenvalues are 3.692 and 0.185. The total is 3.877, so PC1 explains 3.692 / 3.877 = 95.2 percent and PC2 the remaining 4.8 percent. Dropping PC2 loses under 5 percent of the spread.

Step 4: Project to k dimensions, and reconstruct

Stack the top k components as columns of a d by k matrix W. Then:

z is the compressed point with k numbers instead of d. x-hat maps it back to the original space. The squared distance between x and x-hat is exactly the variance you dropped, so PCA is also the best linear compression in the least-squares sense.

Step 5: Choose k

Common rules: keep enough components to reach 90 or 95 percent of the variance; look for the elbow in a scree plotA plot of each principal component's explained variance in order. An elbow where the curve flattens suggests how many components carry signal.; or, best of all, treat k as a hyperparameter and pick the value that gives the best validation score for the model that uses the components. On the digits, 21 components reach 90 percent and 29 reach 95 percent.

Try it

Try this:

1. Drag the angle slowly from 0 to 179 degrees. Where is the captured share largest, and where is it smallest? How far apart are those two angles?
2. Set the axis at right angles to PC1. The share you see is what PC2 would explain. Add it to PC1's share. What do you get, and why?
3. Watch the violet projected dots as you turn the axis. When the variance is highest, are the dots more spread out or more bunched?

Going deeper

Code: PCA by hand, whitening and digits

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# PCA by hand on 2D data: eigenvectors of the covariance matrix
rng = np.random.default_rng(0)
X = rng.multivariate_normal([0, 0], [[3, 1.5], [1.5, 1]], 500)
Xc = X - X.mean(0)
vals, vecs = np.linalg.eigh(np.cov(Xc.T))
order = vals.argsort()[::-1]
vals, vecs = vals[order], vecs[:, order]
print("eigenvalues (variance along each PC):", np.round(vals, 3))
print("explained variance ratio:", np.round(vals / vals.sum(), 3))
print("PC1 direction:", np.round(vecs[:, 0], 3))
p = PCA().fit(X)
print("sklearn agrees:", np.round(p.explained_variance_, 3), np.round(np.abs(p.components_[0]), 3))

# Whitening: decorrelate and scale to unit variance
Z = PCA(whiten=True).fit_transform(X)
print("covariance after whitening:", np.round(np.cov(Z.T), 3).tolist())

# 8x8 digit images: 64 pixel features
D, y = load_digits(return_X_y=True)
pca = PCA().fit(D)
cum = np.cumsum(pca.explained_variance_ratio_)
for k in [2, 10, 20, 40]:
    print(f"{k:>2} components keep {cum[k - 1]:.1%} of the variance")
print(f"components needed for 90%: {np.searchsorted(cum, 0.90) + 1}, for 95%: {np.searchsorted(cum, 0.95) + 1}")

for k in [5, 20, 64]:
    rec = PCA(k).fit(D)
    err = np.mean((D - rec.inverse_transform(rec.transform(D))) ** 2)
    acc = cross_val_score(make_pipeline(StandardScaler(), PCA(k), LogisticRegression(max_iter=2000)), D, y, cv=5).mean()
    print(f"k={k:>2}: reconstruction MSE {err:6.2f}, classifier accuracy {acc:.3f}")

eigenvalues (variance along each PC): [3.692 0.185]
explained variance ratio: [0.952 0.048]
PC1 direction: [-0.882 -0.47 ]
sklearn agrees: [3.692 0.185] [0.882 0.47 ]
covariance after whitening: [[1.0, 0.0], [0.0, 1.0]]
 2 components keep 28.5% of the variance
10 components keep 73.8% of the variance
20 components keep 89.4% of the variance
40 components keep 98.8% of the variance
components needed for 90%: 21, for 95%: 29
k= 5: reconstruction MSE   8.54, classifier accuracy 0.771
k=20: reconstruction MSE   1.98, classifier accuracy 0.899
k=64: reconstruction MSE   0.00, classifier accuracy 0.920

The eigenvalues from the covariance matrix match scikit-learn's explained_variance_ exactly. The sign of a component is arbitrary (here numpy returns (-0.882, -0.47) and scikit-learn (0.882, 0.47)): u and -u describe the same line, so never read meaning into the sign.

Whitening

WhiteningA transform that decorrelates features and scales each to unit variance, so the covariance matrix becomes the identity. In PCA it means dividing each component score by the square root of its eigenvalue. goes one step further: after projecting, divide each component by the square root of its eigenvalue. The result has no correlation between features and unit variance in every direction, which the code confirms with a covariance of exactly [[1, 0], [0, 1]]. Some algorithms (independent component analysis, some older image pipelines) need whitened inputs. The catch: whitening blows up tiny low-variance directions, which are often noise, to the same size as the signal, so combine it with dropping the smallest components.

Compression versus accuracy

On the digits, keeping 5 components gives a reconstruction error (mean squared, pixel values from 0 to 16) of 8.54 and a classifier accuracy of 0.771. With 20 components the error falls to 1.98 and accuracy rises to 0.899, close to the 0.920 from all 64 pixels. Fewer features train faster and can help models that overfit, but PCA picks directions by variance, not by usefulness for your labels. A low-variance direction can be the one that separates the classes, and PCA will throw it away.

Limits

• Linear only. PCA finds flat subspaces. Data on a curved surface, like a rolled-up sheet, needs a non-linear method ( t-SNE and UMAP for visualization, autoencoders for learned compression).
• Sensitive to scale and outliers. One huge outlier can swing PC1 towards itself, because variance squares distances.
• Components are mixtures. Each component blends all original features, which makes it harder to explain than picking a subset of features.
• Fit on training data only. Fit the PCA inside your pipeline on the training fold, then apply it to validation and test data, or you leak information.

Trade-offs

Common mistakes

• Running PCA on unscaled features with different units, so the largest-unit feature becomes PC1.
• Forgetting to centre the data when implementing PCA by hand.
• Fitting PCA on the whole dataset before the train-test split, leaking test information.
• Assuming high-variance directions are the predictive ones; the class signal can hide in a small component.
• Reading meaning into the sign of a component, which is arbitrary.

In the wild

PCA is one of the most used tools in data work. Geneticists project hundreds of thousands of genetic markers onto two components, and the plot often lines up with geography. Face recognition's classic "eigenfaces" were the principal components of face images. In finance, the first component of many stock returns behaves like the overall market. In ML pipelines, PCA compresses embeddings before a nearest-neighbour index, speeds up models on wide tabular data, and is often the first plot you make of a new high-dimensional dataset.

Check yourself

Recap