Multi-layer networks

The big idea

One neuron draws one straight line. That was the perceptron's wall: XOR, a circle of one class inside a ring of the other, a spiral, none of them can be split by a single line.

A multi-layer networkA neural network with one or more hidden layers between input and output, where each layer's neurons take the previous layer's outputs as inputs. Also called a multi-layer perceptron (MLP). puts neurons in a row, then another row that reads the first row's outputs, and so on. Think of a factory line. The first station cuts raw sheet metal into simple parts. The next assembles parts into components. The last assembles components into a car. No single station does anything clever, but the chain builds something no single station could.

The rows between input and output are hidden layersA layer of neurons between the input and the output of a network. Its outputs are not seen directly; they are intermediate features the network learns., because nobody tells them what to compute. Training finds out. In an image network, early layers end up detecting edges, middle layers shapes and textures, late layers whole objects.

How it works

Step 1: A layer is many neurons at once

Each neuron in a layer computes its own weighted sum plus bias and applies an activation. Write the weights of all neurons in a layer as the rows of a matrix W, and the whole layer is one matrix multiply:

With 2 inputs and 3 neurons, W is 3 by 2 and b has 3 entries. The next layer does the same to h. The network in Figure 1 has (2 x 3 + 3) + (3 x 3 + 3) + (3 x 1 + 1) = 9 + 12 + 4 = 25 parametersThe numbers a model learns during training; in a neural network, all its weights and biases..

Step 2: Non-linearity makes layers worth stacking

The activation f must not be a straight line. If it were, layer 2 applied to layer 1 would be W2 (W1 x + b1) + b2, which is just another single linear function: a hundred linear layers collapse into one. A bending activation, such as tanh or ReLU ( Activation functions ), is what lets each layer add something new.

Step 3: Hidden neurons build features, later layers combine them

Each hidden neuron draws its own line and outputs "how far on the positive side" a point is. The next layer combines those lines. For XOR, one hidden neuron learns OR, another learns NAND, and the output ANDs them, exactly the hand-built network from the previous lesson. For a circle inside a ring, a few hidden neurons each draw one side of a polygon, and the output fires only inside all of them. With enough sides, the polygon becomes a circle.

Step 4: Universal approximation

The universal approximation theoremThe result that a network with one hidden layer and a non-linear activation can approximate any continuous function on a bounded region as closely as you want, given enough hidden neurons. says one hidden layer is, in principle, enough: with enough neurons it can approximate any continuous function on a bounded region as closely as you like. Intuitively, each hidden neuron contributes one smooth step, and enough steps of the right heights and positions can trace any curve.

In the code, one hidden layer fits a wiggly curve with mean squared error 0.562 with 1 neuron, 0.222 with 2, 0.036 with 4, and essentially 0 with 8. The theorem says nothing about how many neurons you need, whether training will find the weights, or whether the fit generalizes. It is a statement about what is possible, not about what is practical.

Step 5: Width versus depth

WidthThe number of neurons in a layer. is neurons per layer; depthThe number of layers in a network. is the number of layers. For many functions, depth is far more efficient: each layer can fold or reuse what the previous layer built, so a deep network can represent with a few hundred weights what a shallow one needs thousands for. Shapes that repeat, like a spiral that winds around several times, are the textbook case.

In the code, on two interleaved spirals with about 200 parameters each, one hidden layer of 48 neurons reaches 62.0 percent test accuracy, two layers of 12 reach 63.3 percent, and three layers of 9 reach 99.3 percent. Same budget, very different power. Depth has costs too: deeper networks are harder to train, which is what Weight initialization and normalization and Vanishing and exploding gradients, residual connections are about.

Try it

Try this:

1. Choose "circle" with 0 hidden layers and press Train several times. What does the boundary look like, and where does accuracy stop? Now set 1 hidden layer with 1, 2, 3 and then 4 neurons. How many does it take to enclose the inner class?
2. Choose "XOR". Find the smallest network that reaches 100 percent. Does it match the 2-neuron hand-built solution?
3. Choose "spiral". Compare 1 hidden layer of 8 with 2 hidden layers of 8, pressing Train 10 times each. Which wraps around the spiral better?

Going deeper

Code: circles, a wiggly curve, and spirals

import warnings
import numpy as np
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier, MLPRegressor

warnings.filterwarnings("ignore")  # silence convergence warnings for the tiny nets

# 1. A straight-line model cannot do circles; one hidden layer can
X, y = make_circles(400, noise=0.08, factor=0.4, random_state=0)
for hidden in [(), (2,), (3,), (8,)]:
    clf = MLPClassifier(hidden_layer_sizes=hidden if hidden else (1,), activation="tanh" if hidden else "identity",
                        max_iter=3000, random_state=0).fit(X, y)
    print(f"hidden layers {str(hidden):<6} train accuracy {clf.score(X, y):.3f}")

# 2. Universal approximation: one hidden layer, more neurons, closer fit to a wiggly curve
x = np.linspace(-3, 3, 300).reshape(-1, 1)
t = np.sin(3 * x).ravel() + 0.3 * x.ravel() ** 2
for width in [1, 2, 4, 8, 16]:
    m = MLPRegressor(hidden_layer_sizes=(width,), activation="tanh", solver="lbfgs", max_iter=5000, random_state=0).fit(x, t)
    print(f"1 hidden layer, {width:>2} neurons: mean squared error {np.mean((m.predict(x) - t) ** 2):.4f}")

# 3. Width vs depth on two interleaved spirals, with a similar budget of weights
rng = np.random.default_rng(0)
tt = np.sqrt(rng.random(600)) * 3 * np.pi
c = rng.integers(0, 2, 600)
S = np.c_[tt * np.cos(tt + c * np.pi), tt * np.sin(tt + c * np.pi)] / 10 + rng.normal(0, 0.03, (600, 2))
Str, Ste, ctr, cte = train_test_split(S, c, random_state=0)
for shape in [(48,), (12, 12), (9, 9, 9)]:
    clf = MLPClassifier(hidden_layer_sizes=shape, activation="tanh", max_iter=4000, random_state=0).fit(Str, ctr)
    n = sum(w.size for w in clf.coefs_) + sum(b.size for b in clf.intercepts_)
    print(f"shape {str(shape):<10} parameters {n:>3}  test accuracy {clf.score(Ste, cte):.3f}")

hidden layers ()     train accuracy 0.480
hidden layers (2,)   train accuracy 0.545
hidden layers (3,)   train accuracy 0.650
hidden layers (8,)   train accuracy 1.000
1 hidden layer,  1 neurons: mean squared error 0.5624
1 hidden layer,  2 neurons: mean squared error 0.2218
1 hidden layer,  4 neurons: mean squared error 0.0357
1 hidden layer,  8 neurons: mean squared error 0.0000
1 hidden layer, 16 neurons: mean squared error 0.0000
shape (48,)      parameters 193  test accuracy 0.620
shape (12, 12)   parameters 205  test accuracy 0.633
shape (9, 9, 9)  parameters 217  test accuracy 0.993

The first block is the story of this lesson in four lines. With no hidden layer the model is linear and gets 48.0 percent on the circles, no better than guessing. Two tanh neurons get 54.5 percent, three get 65.0, and eight get 100 percent. In theory three neurons can draw a triangle around the inner circle; in practice gradient descent from one random start did not find it, a first hint that capacity and trainability are different things.

How layers see the data

Another way to read a network: each layer warps the space. The first layer stretches and bends the input plane; the next layer bends the result again. The output neuron is still a single neuron drawing a straight line, but in the warped space produced by the last hidden layer. Training searches for a warp in which the classes become linearly separable. The final hidden layer's outputs are the learned features, also called a representation or, in modern systems, an embedding.

Trade-offs

Common mistakes

• Stacking layers with no activation (or a linear one) and expecting more power; it collapses to one linear layer.
• Reading the universal approximation theorem as "one layer is all you ever need".
• Judging a network only by training accuracy; a big enough network can memorize noise.
• Adding width and depth before checking that the data, features and learning rate are sane.

In the wild

The multi-layer perceptron is still a workhorse: the classifier head on top of image and text models, the scoring layers in recommender and ranking systems, and the feed-forward block inside every transformer layer, which is a two-layer MLP applied to each token. The deep networks behind image recognition and language models are long stacks of such layers, combined with special layers (convolutions, attention) that share weights cleverly.

Check yourself

Recap