Landmark CNNs: LeNet to ResNet

The big idea

Every modern vision model is built from the same parts you have already met: convolutions, nonlinearities, pooling and a classifier head. What changed between 1998 and 2019 was how those parts were arranged, and each landmark network is remembered for one idea that made the next step possible. Learn the six ideas and you can read almost any CNN diagram.

Think of the history of bridges. Stone arches, steel trusses, suspension cables: each design kept what worked and added one trick that let engineers span further. CNNs went the same way, from a few layers reading digits to more than a hundred layers recognizing a thousand object classes, and much of the progress was measured on one benchmark: the ILSVRCThe ImageNet Large Scale Visual Recognition Challenge, a yearly competition (2010 to 2017) on a 1,000-class subset of the ImageNet photo dataset with about 1.2 million training images. image classification challenge.

How it works

Step 1: LeNet (1998): convolution works

LeNet-5, from the 1998 paper "Gradient-based learning applied to document recognition", established the template of Pooling and CNN architecture : 5 by 5 convolutions with shared weights, subsampling (pooling) layers, then fully connected layers. The paper explains why: units in a feature map share the same weights, so they detect the same feature at every location, and a shifted input gives a shifted feature map. A system built around it was reading millions of bank checks per month in the United States. The idea it added: learned, shared convolution kernels trained end to end with backpropagation.

Step 2: AlexNet (2012): scale it up on GPUs

AlexNet applied the same recipe to 1.2 million colour photos in 1,000 classes. It had five convolutional layers and three fully connected layers, 60 million parameters, and won ILSVRC-2012 with a top-5 test error of 15.3 percent against 26.2 percent for the second-best entry. Three ingredients made it trainable: ReLU activations, which the paper reports trained several times faster than tanh ( Activation functions ); a fast GPU implementation of convolution; and dropout in the fully connected layers ( Regularization for deep nets ). The idea it added: depth plus data plus GPUs beats hand-designed features, and that result started the deep learning boom in vision.

Step 3: VGG (2014): small kernels, more depth

VGG asked a simple question: what if every convolution is 3 by 3 and you just go deeper? Its networks have 16 to 19 weight layers, all 3 by 3 convolutions with stride 1 and padding 1, separated by 2 by 2 max pools. Three stacked 3 by 3 layers see a 7 by 7 region but need 27 weights per channel pair instead of 49, and add two extra nonlinearities. VGG placed first in localisation and second in classification at ILSVRC 2014. The idea it added: a uniform, deep stack of small kernels. Its cost was size: VGG-16 has 138 million parameters, most of them in the dense head.

Step 4: Inception / GoogLeNet (2014): go wide, cheaply

Which kernel size is right, 1, 3 or 5? The Inception module uses all of them in parallel and lets training decide, concatenating the results along the channel axis. Run naively, the 5 by 5 branch is very expensive, so each costly branch starts with a 1x1 convolutionA convolution with a 1 by 1 kernel: a learned weighted mix of channels at each pixel, used to reduce or expand the number of channels cheaply. that shrinks the number of channels first. In the code, a 1 by 1 reduction cuts a 5 by 5 convolution from 120 million multiply-adds to 12 million. GoogLeNet, 22 layers deep, set the state of the art at ILSVRC 2014 while using 12 times fewer parameters than AlexNet. The idea it added: multi-scale branches and 1 by 1 bottlenecks to control compute.

Step 5: ResNet (2015): make depth trainable

Simply stacking more layers stopped working: deeper "plain" networks had higher training error than shallower ones, which the ResNet paper calls the degradation problem. It is not overfitting, because the training error itself goes up. ResNet's fix is the residual block from Vanishing and exploding gradients, residual connections : each block learns a correction F(x) and outputs F(x) + x. If a block is not useful, its weights can go to zero and it passes x through unchanged, so extra depth can no longer make things worse, and gradients flow straight through the identity path. ResNet also downsamples with stride-2 convolutions and ends with global average pooling ( Pooling and CNN architecture ). The idea it added: skip connections that make 100+ layer networks trainable.

Step 6: EfficientNet (2019): scale all three dimensions together

Once you have a good network, how should you make it bigger: deeper, wider (more channels), or with higher-resolution inputs? EfficientNet found that balancing all three works better than pushing one, and proposed compound scaling: pick one coefficient phi and scale depth by alpha to the phi, width by beta to the phi and resolution by gamma to the phi.

For its baseline network (found by neural architecture search) the paper reports alpha = 1.2, beta = 1.1, gamma = 1.15, so each step of phi roughly doubles the compute. Width and resolution are squared because compute grows with channels squared and with height times width. The idea it added: principled scaling for a compute budget.

Going deeper

Code: the four tricks in numbers

import numpy as np

def conv_cost(h, w, c_in, c_out, k):
    """Weights and multiply-adds of a k x k convolution with 'same' padding, stride 1."""
    weights = k * k * c_in * c_out
    return weights, weights * h * w

# VGG idea: three stacked 3x3 layers see 7x7 pixels, with fewer weights than one 7x7 layer
C = 256
print(f"one 7x7 layer:       {conv_cost(28, 28, C, C, 7)[0]:>10,} weights")
print(f"three 3x3 layers:    {3 * conv_cost(28, 28, C, C, 3)[0]:>10,} weights  (and 3 ReLUs instead of 1)")

# Inception idea: a 1x1 'bottleneck' shrinks channels before an expensive 5x5 convolution
h = w = 28
direct = conv_cost(h, w, 192, 32, 5)
reduce = conv_cost(h, w, 192, 16, 1)
after = conv_cost(h, w, 16, 32, 5)
print(f"\n5x5 conv, 192 -> 32 channels on 28x28:          {direct[1] / 1e6:6.1f} M multiply-adds")
print(f"1x1 to 16 channels, then 5x5 to 32 channels:    {(reduce[1] + after[1]) / 1e6:6.1f} M multiply-adds")

# ResNet idea: a block outputs F(x) + x, so zero weights give the identity, not zero
rng = np.random.default_rng(0)
x = rng.normal(size=8)
W1, W2 = np.zeros((8, 8)), np.zeros((8, 8))
F = W2 @ np.maximum(0, W1 @ x)
print("\nplain block with zero weights returns x?   ", np.allclose(F, x))
print("residual block with zero weights returns x?", np.allclose(F + x, x))

# EfficientNet idea: scale depth, width and resolution together with one coefficient phi
alpha, beta, gamma = 1.2, 1.1, 1.15        # values the paper found for EfficientNet-B0
print(f"\nalpha * beta^2 * gamma^2 = {alpha * beta**2 * gamma**2:.2f}  (FLOPs grow about 2x per step of phi)")
for phi in [0, 1, 2, 3]:
    d, wd, r = alpha**phi, beta**phi, gamma**phi
    print(f"phi={phi}: depth x{d:.2f}  width x{wd:.2f}  resolution {round(224 * r)}px  FLOPs x{d * wd**2 * r**2:.1f}")

one 7x7 layer:        3,211,264 weights
three 3x3 layers:     1,769,472 weights  (and 3 ReLUs instead of 1)

5x5 conv, 192 -> 32 channels on 28x28:           120.4 M multiply-adds
1x1 to 16 channels, then 5x5 to 32 channels:      12.4 M multiply-adds

plain block with zero weights returns x?    False
residual block with zero weights returns x? True

alpha * beta^2 * gamma^2 = 1.92  (FLOPs grow about 2x per step of phi)
phi=0: depth x1.00  width x1.00  resolution 224px  FLOPs x1.0
phi=1: depth x1.20  width x1.10  resolution 258px  FLOPs x1.9
phi=2: depth x1.44  width x1.21  resolution 296px  FLOPs x3.7
phi=3: depth x1.73  width x1.33  resolution 341px  FLOPs x7.1

The first two blocks show why VGG and Inception were cheaper than they look: stacking small kernels and squeezing channels with 1 by 1 convolutions cut weights and compute by large factors. The residual check shows the key property of F(x) + x: with all-zero weights, a plain block outputs zeros, while a residual block outputs its input exactly. The last block is the compound-scaling arithmetic; the resolutions it prints are illustrative, not the exact sizes of the released models.

What carried forward

Modern CNN backbones are mostly ResNet descendants: residual stages that halve resolution and double channels, 1 by 1 bottlenecks inside each block, batch normalization ( Weight initialization and normalization ) after convolutions, and global average pooling before a small head. Mobile-oriented families split convolutions into cheaper depthwise and pointwise parts, and EfficientNet's baseline is built from such blocks. Since 2020, vision transformers ( Vision transformers ) compete with CNNs, but the lessons of this history, especially residual connections and balanced scaling, are built into them too.

Trade-offs

Common mistakes

• Thinking AlexNet invented convolutional networks. LeNet used the same core ideas over a decade earlier; AlexNet's leap was scale, GPUs, ReLU and dropout.
• Assuming more layers always help. Without residual connections, deeper plain networks get worse even on training data.
• Comparing models by parameter count alone. Compute (multiply-adds) and memory for activations matter as much for speed.
• Reporting ImageNet numbers without saying which ones: top-1 or top-5, validation or test, single model or ensemble.

In the wild

Check yourself

Recap