Convolutions

The big idea

A fully connected layer looks at every pixel at once and has to learn separately that an edge in the top-left corner and an edge in the bottom-right corner are the same thing. That is wasteful. A convolutionAn operation that slides a small grid of weights (the kernel) across an input and computes a weighted sum at each position, producing a feature map. does the opposite: it takes one small pattern detector and checks for it everywhere.

Picture reading a page through a small magnifying glass with a stencil on it. You move the glass across the page one step at a time, and at each spot you note how well what you see matches the stencil. The list of notes, laid out in the same grid as the page, is a map of where the pattern appears. In a network the stencil is a 3 by 3 grid of weights, the notes are numbers, and the map is called a feature map.

How it works

Step 1: A kernel and one weighted sum

A kernelA small grid of learnable weights, often 3 by 3, that a convolution slides over its input. Also called a filter. is a small grid of weights, usually 3 by 3. Lay it on a 3 by 3 window of the image, multiply each pixel by the weight on top of it, and add the nine products. That single number is one output value. It is the same weighted sum you met in Forward pass by hand , just over a small patch instead of the whole input.

The kernel in Figure 1 has weights -1 in its left column, 0 in the middle and +1 on the right. So its output is "right column minus left column": large when the window is bright on the right and dark on the left, zero on a flat patch. That is a vertical-edge detector.

Step 2: Slide it everywhere: the feature map

Move the window one pixel to the right and repeat, then again, row by row, until it has visited every position. Each position gives one number, and the grid of numbers is a feature mapThe output grid of a convolution: one value per kernel position, high where the kernel's pattern appears in the input.. A 5 by 5 input with a 3 by 3 kernel has 3 positions across and 3 down, so the map is 3 by 3.

The feature map keeps the layout of the image. The value at row 1, column 2 says how strongly the pattern appears around that spot. So a convolution does not just say "there is an edge"; it says where.

Step 3: Stride and padding control the size

StrideHow many pixels the kernel moves between positions. Stride 2 skips every other position and roughly halves the output size. is how far the kernel moves each step. Stride 1 visits every position; stride 2 skips every other one and roughly halves the height and width.

Without help, the kernel cannot be centred on edge pixels, so each 3 by 3 convolution shrinks the image by 2. PaddingExtra pixels, usually zeros, added around the border of the input so the kernel can be centred on edge pixels and the output keeps its size. adds a border of zeros so it can. With a 3 by 3 kernel, stride 1 and padding 1, the output has the same size as the input, often called "same" padding. No padding is called "valid". For an input of size n, kernel k, padding p and stride s:

For a 28 by 28 digit and a 3 by 3 kernel: no padding gives (28 - 3) / 1 + 1 = 26; padding 1 gives (28 + 2 - 3) / 1 + 1 = 28; stride 2 without padding gives 25 / 2 = 12.5, rounded down to 12, plus 1 = 13. The code below prints all four cases.

Step 4: Parameter sharing

The same nine weights are used at every position. That is parameter sharingUsing the same weights at every position of the input, so a convolution layer has very few parameters and detects a pattern wherever it appears., and it has two big effects:

1. Few parameters. A layer of 64 kernels on an RGB image needs 64 x (3 x 3 x 3) weights plus 64 biases: 1,792 parameters. A dense layer with 64 units on a 224 by 224 RGB image needs 150,528 x 64 + 64 = 9,633,856.
2. Detect anywhere. A kernel that finds an edge in one corner finds it in every other corner too. If the object moves, its response in the feature map moves with it.

Nobody hand-designs the kernels in a real network. They start random and are learned by backpropagation like any other weights. In the 2012 AlexNet paper, the 96 first-layer kernels learned on ImageNet photos turned out to be orientation-selective edge and stripe detectors plus coloured blobs, very much like the hand-made kernels in the lab below.

Try it

Try this:

1. With the vertical-edge kernel, look at the feature map. Which edges of the tall bar light up, and why is one side violet and the other pink? Switch to the horizontal-edge kernel and compare.
2. Pick blur, then sharpen, then outline. Which ones keep flat regions bright, and which turn flat regions to zero? Look at the kernel weights: what do they add up to in each case?
3. Set stride to 2, then padding to 1. Check the feature-map size against the formula, and watch which input pixels the window can now reach.

Going deeper

Code: a convolution by hand

import numpy as np
from scipy.signal import correlate2d

def conv2d(img, k, stride=1, pad=0):
    """Slide kernel k over img; each output cell is the sum of (window * kernel)."""
    x = np.pad(img, pad)                       # zero padding on all four sides
    kh, kw = k.shape
    oh = (x.shape[0] - kh) // stride + 1       # output size: floor((n + 2p - k) / s) + 1
    ow = (x.shape[1] - kw) // stride + 1
    out = np.zeros((oh, ow))
    for i in range(oh):
        for j in range(ow):
            win = x[i*stride:i*stride+kh, j*stride:j*stride+kw]
            out[i, j] = (win * k).sum()
    return out

# The 5x5 image from Figure 1: dark on the left, bright on the right, a step in the edge
img = np.array([[0, 0, 0, 9, 9], [0, 0, 0, 9, 9], [0, 0, 9, 9, 9], [0, 0, 9, 9, 9], [0, 0, 9, 9, 9]])
vert = np.array([[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]])  # right column minus left column
fm = conv2d(img, vert)
print("vertical-edge feature map:\n", fm.astype(int))
print("matches scipy correlate2d:", np.array_equal(fm, correlate2d(img, vert, mode="valid")))

# Output sizes: stride and padding
big = np.zeros((28, 28))
for s, p in [(1, 0), (1, 1), (2, 0), (2, 1)]:
    print(f"28x28, 3x3 kernel, stride {s}, padding {p} -> {conv2d(big, vert, s, p).shape}")

# Other kernels on the same image
blur = np.ones((3, 3)) / 9
sharpen = np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]])
print("\nblur (padding 1):\n", conv2d(img, blur, pad=1).round(1))
print("sharpen (padding 1):\n", conv2d(img, sharpen, pad=1).astype(int))

# Parameter sharing: a conv layer vs a dense layer on a 224x224 RGB image
conv_params = 64 * (3 * 3 * 3) + 64             # 64 kernels of 3x3x3, plus 64 biases
dense_params = (224 * 224 * 3) * 64 + 64        # 64 units, each wired to every input value
print(f"\nconv 3x3, 3 -> 64 channels: {conv_params:,} parameters")
print(f"dense 150,528 -> 64 units:   {dense_params:,} parameters")

vertical-edge feature map:
 [[ 9 27 18]
 [18 27  9]
 [27 27  0]]
matches scipy correlate2d: True
28x28, 3x3 kernel, stride 1, padding 0 -> (26, 26)
28x28, 3x3 kernel, stride 1, padding 1 -> (28, 28)
28x28, 3x3 kernel, stride 2, padding 0 -> (13, 13)
28x28, 3x3 kernel, stride 2, padding 1 -> (14, 14)

blur (padding 1):
 [[0. 0. 2. 4. 4.]
 [0. 1. 4. 7. 6.]
 [0. 2. 5. 8. 6.]
 [0. 3. 6. 9. 6.]
 [0. 2. 4. 6. 4.]]
sharpen (padding 1):
 [[  0   0  -9  27  27]
 [  0   0 -18  18  18]
 [  0  -9  27   9  18]
 [  0  -9  18   9  18]
 [  0  -9  27  18  27]]

conv 3x3, 3 -> 64 channels: 1,792 parameters
dense 150,528 -> 64 units:   9,633,856 parameters

The hand-written loop reproduces Figure 1 and matches SciPy's correlate2d. Blur averages each window, so the sharp 0-to-9 step becomes a gradual ramp. Sharpen has weights that add up to 1, so flat areas keep their value while the edge is exaggerated on both sides (-9 and -18 next to 27). The last two lines are the parameter-sharing arithmetic from Step 4.

Channels in, channels out

Real inputs have channels. For an RGB image, each kernel is 3 x 3 x 3: one 3 by 3 slice per input channel, and the products over all 27 positions are summed (plus a bias) into a single output value. A layer with 64 such kernels produces 64 feature maps, that is 64 output channels. In PyTorch the weight tensor of nn.Conv2d(3, 64, 3) has shape (64, 3, 3, 3): out channels, in channels, kernel height, kernel width. The next layer's kernels then span all 64 channels, which is how later layers combine simple features into complex ones.

Convolution or cross-correlation?

Strictly, the mathematical convolution flips the kernel before sliding it; sliding it unflipped is called cross-correlation. Deep learning libraries compute cross-correlation and call it convolution anyway (PyTorch's documentation says so explicitly). Because the kernels are learned, the difference does not matter: the network simply learns the flipped weights.

Trade-offs

Common mistakes

• Forgetting that a kernel spans all input channels: a 3 by 3 kernel on 64 channels has 576 weights, not 9.
• Getting the output size wrong after stride and padding, then hitting a shape error at the first dense layer. Use the formula.
• Assuming "same" padding works with stride 2. It keeps the size only at stride 1 (PyTorch rejects padding="same" with other strides).
• Thinking someone chooses the kernel values. In a trained network they are learned; presets like Sobel are for intuition.
• Treating convolution as fully position-blind. Sharing weights finds a pattern anywhere, but the feature map still records where it was.

In the wild

Every convolutional network, from digit readers to the vision backbones in phones and cars, is built from this operation, usually with 3 by 3 kernels, stride 1 with same padding inside a stage, and occasional stride-2 layers to shrink the maps. The classic hand-designed kernels in the lab (box blur, sharpen, Sobel edges) are also standard filters in image editors and in libraries such as OpenCV and scikit-image.

Check yourself

Recap