Pooling and CNN architecture

The big idea

After a convolution you have a feature map that says, pixel by pixel, how strongly a pattern appears. Often you do not need that much precision. "There is a vertical edge somewhere around here" is enough, and it is cheaper to carry forward. PoolingA layer that downsamples a feature map by summarizing each small window, usually with its maximum (max pooling) or its mean (average pooling). does exactly that: it splits the map into small windows and keeps one number per window.

Think of a weather map. A forecaster does not report the temperature of every street; they report the hottest reading in each district. The map gets four times smaller, and if a hot spot moves one street over, the district report does not change. Stack that idea with convolutions and you get a classic convolutional neural network, or CNNConvolutional neural network: a network built mainly from convolution layers, usually with pooling or strided convolutions, followed by layers that make the final prediction..

How it works

Step 1: Max and average pooling

The most common setting is a 2 by 2 window with stride 2, so the windows do not overlap and the height and width are halved. Max poolingPooling that keeps the largest value in each window: the strongest response of the feature in that region. keeps the largest value in each window: "how strongly did the feature fire anywhere in this region?" Average poolingPooling that keeps the mean of each window, smoothing the feature map. keeps the mean: "how much of the feature is in this region on average?"

Pooling has no weights to learn. It is a fixed summary, so it adds no parameters, and it works on each channel separately: 64 feature maps in, 64 smaller feature maps out.

Step 2: Robustness to small shifts

Move an object in an image by one pixel and every value in the feature map moves too. After max pooling, the output often does not change at all: as long as the strong response stays inside the same 2 by 2 window, the window's maximum is the same. In the code, a single peak shifted one pixel diagonally gives an identical pooled map. Shift it two pixels across a window border and the pooled map does change, so the robustness is real but limited to small moves.

Step 3: The receptive field grows with depth

A neuron's receptive fieldThe region of the original input image that can influence one value in a feature map. It grows with every convolution and pooling layer. is the patch of the original image it can see. One 3 by 3 convolution sees 3 by 3 pixels. Stack a second 3 by 3 convolution on top and each of its outputs combines nine first-layer outputs, which together cover 5 by 5 pixels; three layers cover 7 by 7. The VGG paper ( Landmark CNNs: LeNet to ResNet ) used exactly this: three 3 by 3 layers see as much as one 7 by 7 layer with fewer weights (27 versus 49 per channel pair).

Pooling and strides speed this up, because after a 2 by 2 pool each step of the next kernel moves two input pixels instead of one. In the small network of Figure 2, the receptive field goes 3, 4, 8, 10 pixels. That is why deeper layers can respond to large structures such as a whole loop of a digit or a face, while early layers only see edges.

Step 4: The conv-pool-dense stack

The classic CNN repeats a simple block: one or two convolutions with ReLU (padding 1 so the size stays), then a 2 by 2 max pool that halves height and width. Each block typically doubles the number of channels, so the network trades spatial detail for richer features. At the end, the maps are flattened into one vector and passed to one or more dense layers ( Multi-layer networks ) that output class scores, followed by softmax.

The convolution part is the backboneThe feature-extracting part of a network, such as the convolution and pooling layers of a CNN, before the task-specific head.: it turns pixels into features. The dense part is the head: it turns features into a decision. That split is what makes transfer learning ( Transfer learning and fine-tuning ) possible.

Going deeper

Code: pooling, shifts and a shape trace

import numpy as np

def pool(x, size=2, mode="max"):
    """Non-overlapping pooling: split x into size x size windows, keep the max or the mean."""
    h, w = x.shape[0] // size, x.shape[1] // size
    win = x[:h*size, :w*size].reshape(h, size, w, size)
    return win.max(axis=(1, 3)) if mode == "max" else win.mean(axis=(1, 3))

fm = np.array([[1, 3, 2, 0], [5, 6, 1, 2], [0, 2, 9, 4], [1, 1, 3, 7]])  # Figure 1
print("max pool:\n", pool(fm))
print("avg pool:\n", pool(fm, mode="avg"))

# Robustness to small shifts: move a single strong response by one pixel
a = np.zeros((6, 6)); a[2, 2] = 1.0
b = np.zeros((6, 6)); b[3, 3] = 1.0          # same feature, shifted one pixel down-right
print("\nraw maps equal?   ", np.array_equal(a, b))
print("pooled maps equal?", np.array_equal(pool(a), pool(b)))
c = np.zeros((6, 6)); c[2, 4] = 1.0          # shifted two pixels right: crosses a window border
print("shift by 2 equal? ", np.array_equal(pool(a), pool(c)))

# A small CNN for 28x28 digits: trace shapes, parameters and receptive field
layers = [("conv3x3", 3, 1, 8), ("maxpool2", 2, 2, 0), ("conv3x3", 3, 1, 16), ("maxpool2", 2, 2, 0)]
h, c_in, rf, jump, total = 28, 1, 1, 1, 0
print(f"\n{'layer':10} {'output':>12} {'params':>7} {'receptive field':>16}")
print(f"{'input':10} {f'{h}x{h}x{c_in}':>12} {0:>7} {rf:>16}")
for name, k, s, c_out in layers:
    rf += (k - 1) * jump                       # each layer widens the view by (k - 1) input-steps
    jump *= s                                  # stride compounds: neighbours move further apart
    if name.startswith("conv"):
        params = c_out * (k * k * c_in) + c_out    # padding 1 keeps the size
        c_in = c_out
    else:
        params, h = 0, h // s
    total += params
    print(f"{name:10} {f'{h}x{h}x{c_in}':>12} {params:>7} {f'{rf}x{rf}':>16}")
flat = h * h * c_in
fc = flat * 10 + 10
print(f"{'flatten':10} {flat:>12} {0:>7}")
print(f"{'dense 10':10} {10:>12} {fc:>7}")
print(f"total parameters: {total + fc:,}")

# Two stacked 3x3 convs see 5x5; three see 7x7, with fewer weights than one big kernel
C = 64
print(f"\nthree 3x3 layers: {3 * 9 * C * C:,} weights   one 7x7 layer: {49 * C * C:,} weights")

max pool:
 [[6 2]
 [2 9]]
avg pool:
 [[3.75 1.25]
 [1.   5.75]]

raw maps equal?    False
pooled maps equal? True
shift by 2 equal?  False

layer            output  params  receptive field
input           28x28x1       0                1
conv3x3         28x28x8      80              3x3
maxpool2        14x14x8       0              4x4
conv3x3        14x14x16    1168              8x8
maxpool2         7x7x16       0            10x10
flatten             784       0
dense 10             10    7850
total parameters: 9,098

three 3x3 layers: 110,592 weights   one 7x7 layer: 200,704 weights

The table at the end of the output is worth reading line by line. The two convolutions have only 80 and 1,168 parameters, while the single dense layer at the end has 7,850, most of the 9,098 total. That imbalance gets far worse on large images, which is one reason modern CNNs replace flatten-plus-dense with global average pooling.

Global average pooling and strided convolutions

Global average poolingAveraging each whole feature map down to a single number, so a C by H by W tensor becomes a vector of C values regardless of image size. averages each final feature map down to one number, so 512 maps of any size become a vector of 512 values, followed by one small dense layer. It removes most head parameters and lets the same network accept different image sizes. Many modern designs also skip pooling layers in the middle and use convolutions with stride 2 instead, letting the network learn how to downsample rather than always taking a maximum.

Trade-offs

Common mistakes

• Pooling too early or too often, so small objects and fine detail are gone before the network can use them.
• Expecting pooling to make a CNN fully shift-invariant. It only absorbs moves smaller than its window.
• Forgetting that the flatten size depends on the input size, then getting a shape error when the image size changes.
• Assuming a deep layer "sees" only its 3 by 3 kernel. Its receptive field in the original image is far larger.

In the wild

The conv-pool-dense pattern is the design of the early landmark CNNs, from LeNet reading handwritten digits to AlexNet and VGG on ImageNet photos ( Landmark CNNs: LeNet to ResNet ). Later backbones such as ResNet keep the same idea of stages that halve resolution and grow channels, but mostly downsample with stride-2 convolutions and end with global average pooling instead of large dense layers.

Check yourself

Recap