Autoencoders

The big idea

An autoencoderA neural network trained to reproduce its own input at its output after passing it through a smaller intermediate layer, so it must learn a compact code for the data. is a network whose job sounds pointless: output exactly what you were given. The catch is a narrow layer in the middle. A 64-pixel image has to pass through, say, 2 or 8 numbers on its way to the output. The network can only rebuild the image if those few numbers capture what actually matters about it.

Think of describing a friend's face over the phone to a sketch artist. You cannot read out every pixel; you say "round face, short dark hair, glasses, big smile". The artist rebuilds a face from that short description. If the sketch looks like your friend, your description captured the essentials. The describer is the encoder, the short description is the code, and the artist is the decoder. Train both together on thousands of faces and they invent their own shorthand.

No labels are needed: the input is its own target. That makes autoencoders a tool for compression, for cleaning noisy data, for spotting inputs that do not look like the rest, and for learning features, and they are the starting point for the generative models in the rest of this module.

How it works

Step 1: Encoder, code, decoder

The encoderThe part of an autoencoder that maps an input to its compact code. $f$ maps an input $x$ to a code $z=f(x)$. The decoderThe part of an autoencoder that maps a code back to the input space, producing a reconstruction. $g$ maps the code back, giving a reconstruction $\hat{x}=g(z)$. Both are ordinary networks of the kind you met in Multi-layer networks , usually mirror images of each other: dense layers for tabular data, convolutions for images.

The narrow middle layer is the bottleneckThe narrowest layer of an autoencoder; its size limits how much information the code can carry., and the space of all possible codes is the latent spaceThe space of codes produced by an encoder; nearby points should correspond to similar inputs.. An autoencoder whose code is smaller than its input is called undercomplete. That smallness is the whole point: if the code were as wide as the input, the network could learn to copy pixels straight through and would learn nothing.

Step 2: The reconstruction loss

Training minimises the gap between input and output over the training set. For real-valued data the usual choice is mean squared error:

For pixel values scaled to between 0 and 1 you can also use binary cross-entropy per pixel, with a sigmoid on the output. Worked example with a 4-pixel image: input $(1,0,0.5,0)$, reconstruction $(0.9,0.1,0.4,0)$. The squared errors are $0.01,0.01,0.01,0$, so the per-pixel mean is $0.0075$. Nothing in the loss mentions a label; the data supervises itself.

Step 3: The link to PCA

What would the best linear autoencoder learn? Principal component analysis ( Principal Component Analysis ) already answers "which $k$ directions keep the most of the data". Baldi and Hornik showed in 1989 that a linear network trained with squared error has a unique minimum at exactly the projection onto the subspace of the top principal components; every other critical point is a saddle. So a linear autoencoder is PCA learned by gradient descent (it finds the same subspace, though not necessarily the same axes inside it).

Add nonlinear activations and depth, and the encoder can bend: data that lies on a curved surface can be flattened into a few coordinates, which no straight projection can do. Hinton and Salakhutdinov's 2006 Science paper on deep autoencoders put it this way: a multilayer network "with a small central layer" can learn low-dimensional codes "that work much better than principal components analysis" as a tool to reduce the dimensionality of data, given a good way to initialise the weights.

Step 4: Denoising autoencoders

A different way to stop the network from copying: damage the input and ask for the clean version. A denoising autoencoderAn autoencoder trained to reconstruct the clean input from a deliberately corrupted copy of it, which forces it to learn the structure of the data. takes a corrupted $\tilde{x}$ and is scored against the original $x$:

Vincent, Larochelle, Bengio and Manzagol introduced the idea in 2008. Their corruption chose a fixed fraction of input components at random and forced them to 0, so the network had to "fill in" the blanks. To fill in a missing stroke of a 7, the network must know what 7s look like, which is exactly the structure you want it to learn. The same idea, scaled up, is behind masked autoencoders for images (He et al., 2021), which hide 75 percent of an image's patches and reconstruct the missing pixels as a pretraining task.

Step 5: What the code is good for

• Compression and visualisation: a 2-number code can be plotted, like PCA or t-SNE.
• Features: train the autoencoder on lots of unlabelled data, then feed the codes (or the trained encoder) to a supervised model trained on a few labels.
• Anomaly detection: the network only learns to rebuild inputs like its training data. A transaction or sensor reading that rebuilds badly is unusual. Score each input by its reconstruction errorThe difference between an input and the autoencoder's output for it; large values flag inputs unlike the training data. and flag the ones above a threshold, such as the 99th percentile on normal data ( Anomaly detection covers the evaluation side).

Try it

This is a real autoencoder (64, 64, 2, 64, 64 units) trained offline in numpy on the 1,797 8 by 8 digits that ship with scikit-learn; the decoder's weights are baked into the page, so every image is the network's actual output.

1. Drag the handle into the middle of a cluster of 0s, then a cluster of 6s. The decoder draws a clean digit of that class. The network learned to sort digits by shape without ever being told the labels.
2. Press "Random code in the box" a few times, then drag the handle into an empty corner of the box, such as the top right. Codes far from every training code (the note turns red) often decode to smudges or hybrids. The decoder only learned to draw near codes it was trained on, and nothing in training asked the empty space to mean anything. Hold that thought: it is the problem the next lesson's variational autoencoder fixes.
3. Switch to interpolation, from 1 to 0. The straight line between the two averages crosses other clusters on the way, so the strip passes through other digits rather than morphing smoothly.

Going deeper

Code: PCA, depth, denoising and anomalies

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.neural_network import MLPRegressor
import warnings; warnings.filterwarnings("ignore")      # convergence warnings

X, y = load_digits(return_X_y=True)
X = X / 16.0                                          # 1,797 images, 64 pixels in [0, 1]
rng = np.random.default_rng(0)
idx = rng.permutation(len(X)); tr, te = idx[:1400], idx[1400:]
mse = lambda a, b: np.mean((a - b) ** 2)

def autoencoder(hidden, act="relu", inputs=None):     # an MLP trained to output its own input
    m = MLPRegressor(hidden_layer_sizes=hidden, activation=act, solver="lbfgs", alpha=0,
                     max_iter=1000, random_state=0)
    return m.fit(X[tr] if inputs is None else inputs, X[tr])

# 1. A linear autoencoder with an 8-number bottleneck lands where PCA with 8 components does
pca = PCA(8).fit(X[tr])
lin = autoencoder((8,), act="identity")
print(f"8-number code  PCA {mse(pca.inverse_transform(pca.transform(X[te])), X[te]):.4f}"
      f"   linear AE {mse(lin.predict(X[te]), X[te]):.4f}")
# With only 2 numbers, a nonlinear encoder and decoder (64 -> 64 -> 2 -> 64 -> 64) wins clearly
pca2, ae2 = PCA(2).fit(X[tr]), autoencoder((64, 2, 64))
print(f"2-number code  PCA {mse(pca2.inverse_transform(pca2.transform(X[te])), X[te]):.4f}"
      f"   deep AE   {mse(ae2.predict(X[te]), X[te]):.4f}")
deep = autoencoder((32, 8, 32))                       # 64 -> 32 -> 8 -> 32 -> 64, ReLU

# 2. Denoising: hide 25 percent of the pixels (set them to 0), learn to restore them
def mask(A):
    return A * (rng.random(A.shape) > 0.25)
dae = autoencoder((32, 8, 32), inputs=mask(X[tr]))
noisy = mask(X[te])
print(f"masked input {mse(noisy, X[te]):.4f}   plain AE {mse(deep.predict(noisy), X[te]):.4f}"
      f"   denoising AE {mse(dae.predict(noisy), X[te]):.4f}")

# 3. Anomaly detection: inputs unlike the training data reconstruct badly
odd = rng.permuted(X[te], axis=1)                     # same pixels, scrambled positions
err = lambda A: np.mean((deep.predict(A) - A) ** 2, axis=1)
cut = np.quantile(err(X[tr]), 0.99)                   # threshold: 99th percentile on training
print(f"flagged as anomalies: digits {np.mean(err(X[te]) > cut):.1%}"
      f"   scrambled images {np.mean(err(odd) > cut):.1%}")

8-number code  PCA 0.0245   linear AE 0.0245
2-number code  PCA 0.0534   deep AE   0.0343
masked input 0.0594   plain AE 0.0405   denoising AE 0.0302
flagged as anomalies: digits 1.0%   scrambled images 100.0%

scikit-learn's MLPRegressor becomes an autoencoder when you fit it with the input as the target. Four results, all on 397 held-out digits:

• Linear equals PCA. With an 8-number code, the linear autoencoder's error (0.0245 per pixel) matches PCA's to four decimals, as Baldi and Hornik predict.
• Nonlinearity matters when the code is tiny. With only 2 numbers, PCA's error is 0.0534 and the nonlinear autoencoder's is 0.0343, about a third lower.
• Denoising. With 25 percent of pixels hidden, the masked image is 0.0594 away from the clean one. A plain autoencoder repairs some of that (0.0405) because it pulls inputs toward digit-like shapes anyway; one trained on masked inputs does better (0.0302).
• Anomalies. With a threshold at the 99th percentile of training errors, 1 percent of normal test digits are flagged and every image whose pixels were scrambled is flagged.

In PyTorch the same model is two nn.Sequential stacks, an MSELoss between decoder(encoder(x)) and x, and the training loop from Autograd and your first PyTorch training loop . For images, use convolutions in the encoder and transposed convolutions or upsampling in the decoder.

Other ways to stop copying

A bottleneck is not the only constraint. Sparse autoencoders allow a wide code but penalise how many code units are active at once; contractive autoencoders penalise how much the code changes when the input moves slightly. All of them force the network to keep what is typical about the data and drop the rest.

Trade-offs

Common mistakes

• Making the code as wide as the input with no other constraint, so the network learns to copy and the code means nothing.
• Expecting random points in a plain autoencoder's latent space to decode into realistic samples. It is a compressor, not a generator.
• Training an anomaly detector on data that already contains the anomalies, so it learns to rebuild them too.
• Comparing reconstruction errors across features on different scales; standardise the inputs first.
• Reaching for a deep autoencoder before trying PCA, which is often just as good when the code is not tiny.

Where you will see this

Autoencoders are a standard tool for anomaly detection where labelled failures are rare: unusual transactions, faulty machine sensor readings, defective parts in inspection images, all flagged because they reconstruct badly. Their encoders are used to pretrain features from unlabelled data, most visibly in masked autoencoders for vision transformers. And modern image generators that run "in latent space" use an autoencoder to compress images before the generative model ever sees them, as you will see in the diffusion models lesson at the end of this module.

Check yourself

Recap