Activation functions

The big idea

Every neuron computes a weighted sum z and then passes it through an activation functionThe function a neuron applies to its weighted sum, such as a step, sigmoid or ReLU. It decides the neuron's output.. That one small function decides two things: what shapes the network can build, and how easily gradients can flow back through it during training.

The first job is non-negotiable. Without a bend, any stack of layers is one big linear function, as the code checks. A dimmer switch is a good picture: a straight-through wire passes signals unchanged, while a dimmer that clips at zero (ReLU) or levels off at full brightness (sigmoid) changes the shape of the signal, and stacking such shapes builds curves.

The second job is where activations differ. Training needs the slope f'(z). Where a function is flat, the slope is near zero and the neuron stops learning. That flatness is called saturationWhen an activation function's input is in a flat region, so its derivative is close to zero and almost no gradient flows back through the neuron., and the history of activation functions is largely a history of avoiding it.

How it works

Step 1: Sigmoid

The sigmoidThe S-shaped function 1 / (1 + e^-z), which squashes any number into the range 0 to 1. squashes any number into 0 to 1, which made it the classic choice and still makes it the right output for a yes/no probability. Inside a network it has two problems. Its slope is at most 0.25 (at z = 0) and is only 0.018 at z = 4, so gradients shrink every time they pass through it. And its outputs are always positive, which makes the next layer's weight updates zig-zag.

Step 2: tanh

tanhThe hyperbolic tangent: an S-shaped function from -1 to 1, a rescaled sigmoid centred on zero. is a sigmoid stretched to run from -1 to 1. It is centred on zero and its maximum slope is 1, so it trains better than sigmoid. But it saturates just the same: at z = 4 its slope is 0.001.

Step 3: ReLU

The ReLURectified linear unit: max(0, z). It passes positive inputs unchanged and outputs zero for negative ones. is the simplest possible bend: pass positives through, zero out negatives. Its slope is exactly 1 for every positive input, so it never saturates on that side, and it costs almost nothing to compute. Switching from sigmoid and tanh to ReLU was one of the changes that made deep networks trainable around 2010 to 2012.

Its weakness is the other side. For negative inputs the slope is 0. If a neuron's weighted sum is negative for every example, perhaps after one bad large update, it outputs 0, receives no gradient and never recovers: a dead ReLUA ReLU neuron whose input is negative for every training example, so its output and gradient are always zero and it stops learning..

Step 4: Leaky ReLU, GELU and SiLU

• Leaky ReLU keeps a small slope for negatives, typically 0.01 x z, so no neuron is ever fully dead.
• GELU (Gaussian error linear unit), z times the normal cumulative distribution of z, is a smooth ReLU. Large positive inputs pass, large negative ones go to zero, and near zero it curves gently, even dipping slightly below zero.
• SiLU (also called Swish), z times sigmoid(z), has a very similar shape.

The smooth versions have a slope everywhere and tend to train slightly better in large models. GELU and SiLU (often inside a gated variant) are the standard choices in transformers.

Try it

Try this:

1. With sigmoid, move z from 0 to 4. Watch f'(z) and the "slope^10" number. How small does it get, and what does that mean for a 10-layer sigmoid network?
2. Switch to ReLU and set z to -2. What is the slope? Now pick Leaky ReLU at the same z. What changed?
3. Compare ReLU and GELU around z = -1 to 1. Where do they differ most, and which one has a slope at every point?

Going deeper

Code: values, slopes, collapse and dead units

import numpy as np
from math import erf

sig = lambda z: 1 / (1 + np.exp(-z))
gelu = lambda z: z * 0.5 * (1 + np.vectorize(erf)(z / np.sqrt(2)))
acts = {
    "sigmoid": sig,
    "tanh": np.tanh,
    "relu": lambda z: np.maximum(0, z),
    "leaky": lambda z: np.where(z > 0, z, 0.01 * z),
    "gelu": gelu,
    "silu": lambda z: z * sig(z),
}

# Values and slopes (numerical derivative) at a few inputs
np.set_printoptions(suppress=True)
z = np.array([-4.0, -1.0, 0.5, 2.0, 4.0])
print("z:          ", z)
for name, f in acts.items():
    slope = (f(z + 1e-5) - f(z - 1e-5)) / 2e-5
    print(f"{name:<8} f  ", np.round(f(z), 3) + 0.0, "  f'", np.round(slope, 3) + 0.0)

# Saturation: the gradient through 10 stacked neurons multiplies 10 slopes
print("\nbest-case gradient factor through 10 layers:")
print(f"  sigmoid (max slope 0.25): {0.25 ** 10:.1e}")
print(f"  tanh    (max slope 1.0):  {1.0 ** 10:.1f}, but at z = 2: {(1 - np.tanh(2) ** 2) ** 10:.1e}")
print(f"  relu    (slope 1 if z > 0): {1.0 ** 10:.1f}")

# Without a non-linearity, two layers collapse into one
rng = np.random.default_rng(0)
W1, W2, x = rng.normal(size=(4, 3)), rng.normal(size=(2, 4)), rng.normal(size=3)
print("\ntwo linear layers == one layer W2 @ W1:", np.allclose(W2 @ (W1 @ x), (W2 @ W1) @ x))
print("with tanh in between, still equal?     ", np.allclose(W2 @ np.tanh(W1 @ x), (W2 @ W1) @ x))

# Dead ReLUs: neurons whose input is negative for every example never get a gradient
X = rng.normal(size=(1000, 3))
W, b = rng.normal(size=(3, 50)), np.full(50, -3.0)  # a large negative bias, as after a bad update
dead = np.all(X @ W + b <= 0, axis=0)
print(f"\nReLU units dead on all 1000 inputs: {dead.sum()} of 50")

z:           [-4.  -1.   0.5  2.   4. ]
sigmoid  f   [0.018 0.269 0.622 0.881 0.982]   f' [0.018 0.197 0.235 0.105 0.018]
tanh     f   [-0.999 -0.762  0.462  0.964  0.999]   f' [0.001 0.42  0.786 0.071 0.001]
relu     f   [0.  0.  0.5 2.  4. ]   f' [0. 0. 1. 1. 1.]
leaky    f   [-0.04 -0.01  0.5   2.    4.  ]   f' [0.01 0.01 1.   1.   1.  ]
gelu     f   [ 0.    -0.159  0.346  1.954  4.   ]   f' [-0.001 -0.083  0.867  1.085  1.001]
silu     f   [-0.072 -0.269  0.311  1.762  3.928]   f' [-0.053  0.072  0.74   1.091  1.053]

best-case gradient factor through 10 layers:
  sigmoid (max slope 0.25): 9.5e-07
  tanh    (max slope 1.0):  1.0, but at z = 2: 3.1e-12
  relu    (slope 1 if z > 0): 1.0

two linear layers == one layer W2 @ W1: True
with tanh in between, still equal?      False

ReLU units dead on all 1000 inputs: 7 of 50

The slope table shows saturation directly: sigmoid's slope is 0.018 at z = 4 and tanh's 0.001, while ReLU stays at 1. Even in the best case, a gradient passing through 10 sigmoid layers is multiplied by 0.25^10, about one in a million. With tanh it can survive at z = 0, but at z = 2 it shrinks to 3.1 x 10^-12. This is the vanishing gradient problem, which Vanishing and exploding gradients, residual connections treats in full.

The code also checks the collapse: two linear layers give exactly the same result as one layer with W2 @ W1, and putting tanh between them breaks that equality. Finally, with a bias of -3, 7 of 50 ReLU units output zero on all 1000 inputs. They are dead.

Choosing in practice

Use ReLU (or a smooth relative) in hidden layers of most networks, GELU or SiLU in transformers, and match the output activation to the task: sigmoid for one yes/no probability, softmax for one of many classes ( Multi-class and multi-label classification ), nothing (identity) for regression. Sigmoid and tanh remain inside gates, where a value between 0 and 1 or -1 and 1 is exactly what is needed, as in LSTMs and attention gates.

Trade-offs

Common mistakes

• Using sigmoid in every hidden layer of a deep network and wondering why early layers do not learn.
• Leaving out the activation (or using identity) between layers, collapsing the network into a linear model.
• Applying ReLU to the output of a regression model that must predict negative values.
• Ignoring a large share of dead ReLUs after a too-high learning rate.

In the wild

ReLU and its variants sit in nearly every convolutional network for images. Transformer language models use GELU or SiLU-based gated feed-forward layers in every block. Sigmoids live on in outputs that predict probabilities, such as click-through models, and in the gates of LSTMs. Because every neuron applies its activation, the choice touches billions of computations per prediction, which is one reason the cheap ReLU family has lasted.

Check yourself

Recap