Forward pass by hand

The big idea

A neural network can sound mysterious until you run one with a pencil. Then it turns out to be a short recipe: multiply, add, bend, repeat. The forward passRunning inputs through a network layer by layer, from input to output, to compute its prediction. is that recipe, and it is all a trained network does when it makes a prediction.

Think of a relay race. The inputs hand their values to the first layer. Each runner in that layer combines what it received, applies its own rule and passes the result on. Nobody looks back; information only moves forward. At the end, the last runner crosses the line with the prediction.

In this lesson you will do the whole race by hand on a network with 2 inputs, 2 hidden neurons and 1 output: 9 parameters in total. Every number below is checked by the code.

How it works

Step 1: Hidden neuron 1

The network has inputs x1 = 1 and x2 = 2. Hidden neuron 1 has weights 0.5 and -0.4 and bias 0.1:

The hidden layer uses ReLU, max(0, z). Since -0.2 is negative, h1 = 0. This neuron is silent for this input. That is normal: different neurons respond to different patterns, and for any given input many ReLUs output zero.

Step 2: Hidden neuron 2

Weights 0.3 and 0.8, bias -0.2:

Positive, so ReLU passes it unchanged: h2 = 1.7.

Step 3: The output neuron

The output neuron treats h1 and h2 as its inputs, with weights 1.2 and -0.7 and bias 0.05:

For a yes/no prediction the output uses a sigmoid: y = 1 / (1 + e^1.14) = 1 / (1 + 3.127) = 0.242. The network gives this input a 24 percent probability of being class 1. Notice that the weight 1.2 did nothing here: it multiplied h1, which was 0.

Step 4: The same thing as matrices

Write the hidden weights as a 2 by 2 matrix W1 (one row per hidden neuron) and the biases as a vector b1. Then the whole hidden layer is one line, and so is the output:

With a batchA group of examples processed together in one forward pass. Stacking inputs as rows of a matrix turns many matrix-vector products into one matrix-matrix product. of inputs stacked as the rows of a matrix X, the hidden layer becomes ReLU(X W1^T + b1): one matrix multiply handles every example at once. That is why networks run on GPUs, which are built to do huge matrix multiplies in parallel. Keeping track of shapesThe sizes of a tensor's dimensions, such as (batch, features). Inner dimensions must match in a matrix multiply. is the main bookkeeping: here X is (4, 2), W1^T is (2, 2), so the hidden activations are (4, 2), and the outputs are (4,).

Try it

Try this:

1. With the defaults, set x2 to 1.0. Does hidden neuron 1 wake up? What is the output now, and why did it go up?
2. Reset, then edit b1 (bias of h1). How large must it be before h1 becomes positive for the default inputs? Watch v1 start to matter.
3. Choose v2 and make it positive. What happens to the output, and what does that tell you about what the output neuron "likes" in h2?

Going deeper

Code: by hand and as a batch

import numpy as np

# A 2-2-1 network: ReLU hidden layer, sigmoid output
x = np.array([1.0, 2.0])
W1 = np.array([[0.5, -0.4],    # weights into hidden neuron 1
               [0.3, 0.8]])    # weights into hidden neuron 2
b1 = np.array([0.1, -0.2])
w2 = np.array([1.2, -0.7])     # weights from the hidden neurons into the output
b2 = 0.05

# Step by step, one neuron at a time
for j in range(2):
    terms = " + ".join(f"{W1[j, k]} * {x[k]}" for k in range(2))
    z = W1[j] @ x + b1[j]
    print(f"hidden {j + 1}: z = {terms} + {b1[j]} = {z:.2f}   ReLU -> {max(0.0, z):.2f}")

h = np.maximum(0, W1 @ x + b1)
z_out = w2 @ h + b2
y = 1 / (1 + np.exp(-z_out))
print(f"output:   z = {w2[0]} * {h[0]:.2f} + {w2[1]} * {h[1]:.2f} + {b2} = {z_out:.3f}   sigmoid -> {y:.4f}")

# The same pass for a batch of 4 inputs at once: one matrix multiply per layer
X = np.array([[1.0, 2.0], [0.0, 0.0], [2.0, -1.0], [-1.0, 1.0]])
H = np.maximum(0, X @ W1.T + b1)          # shape (4, 2)
Y = 1 / (1 + np.exp(-(H @ w2 + b2)))      # shape (4,)
print("\nbatch inputs:\n", X)
print("hidden activations:\n", np.round(H, 2))
print("outputs:", np.round(Y, 4))
print("parameters:", W1.size + b1.size + w2.size + 1)

hidden 1: z = 0.5 * 1.0 + -0.4 * 2.0 + 0.1 = -0.20   ReLU -> 0.00
hidden 2: z = 0.3 * 1.0 + 0.8 * 2.0 + -0.2 = 1.70   ReLU -> 1.70
output:   z = 1.2 * 0.00 + -0.7 * 1.70 + 0.05 = -1.140   sigmoid -> 0.2423

batch inputs:
 [[ 1.  2.]
 [ 0.  0.]
 [ 2. -1.]
 [-1.  1.]]
hidden activations:
 [[0.  1.7]
 [0.1 0. ]
 [1.5 0. ]
 [0.  0.3]]
outputs: [0.2423 0.5424 0.8641 0.4601]
parameters: 9

The first lines reproduce the hand calculation exactly. The batch block runs four inputs through the same network with two matrix multiplies. Check one by hand: for the input (2, -1), h1 = ReLU(1.0 + 0.4 + 0.1) = 1.5 and h2 = ReLU(0.6 - 0.8 - 0.2) = 0, so the output is sigmoid(1.2 x 1.5 + 0.05) = sigmoid(1.85) = 0.864, as printed.

Where the cost goes

Counting operations tells you what a network costs to run. Each weight is used in one multiply and one add per example: our network does 8 multiply-adds per input. A layer with 4096 inputs and 4096 outputs has about 16.8 million weights, and so about 16.8 million multiply-adds per example. Summed over a network, this count (often quoted as FLOPs, about twice the number of multiply-adds) is the main driver of inference speed and cost, a theme that returns in Model serving patterns .

What training will change

The forward pass gives a prediction; a loss ( Loss functions ) compares it to the target. Training then asks, for each of the 9 parameters: if I nudged you a little, how would the loss change? That question is answered by running the computation backwards, which is backpropagation ( Backpropagation ). It reuses every intermediate value computed here, which is why frameworks store them during the forward pass.

Trade-offs

Common mistakes

• Mixing up W x and x W: check which side the batch dimension is on and transpose accordingly.
• Forgetting the bias, or adding it before the multiply instead of after.
• Applying the activation to the output layer twice, once in the model and again in the loss.
• Assuming a zero ReLU output means a bug; for most inputs many units are inactive.

In the wild

Every prediction from a deployed network, from a photo tag to the next word from a language model, is a forward pass: the same multiply, add, bend pattern, repeated through dozens to hundreds of layers with millions or billions of weights. Inference servers spend most of their time doing exactly these matrix multiplies, and techniques such as batching requests together exist to keep that hardware busy.

Check yourself

Recap