Recurrent neural networks

The big idea

A bag of words ( Bag of words and TF-IDF ) cannot tell "the dog bit the man" from "the man bit the dog": same words, same counts. A plain feed-forward network has a different problem: it takes a fixed number of inputs, but sentences, audio clips and sensor logs come in every length. We want a model that reads a sequence in order, of any length, and remembers what it has seen.

That is what a recurrent neural networkA neural network that processes a sequence one element at a time, updating a hidden state vector from the previous state and the current input with the same weights at every step. (RNN) does. Think of reading a book with a single sticky note. After each sentence you rewrite the note so it captures everything that matters so far, then you move on; you never reread the earlier pages. The note is the RNN's hidden stateThe vector an RNN carries from one time step to the next; a learned summary of everything the network has read so far.. The note has a fixed size, and the habit of rewriting it at every step is exactly what makes the distant past hard to keep, as you will see.

How it works

Step 1: One cell, applied at every step

At each time step $t$ the RNN takes the current input $x_t$ (for text, a word embedding) and the previous hidden state $h_{t-1}$, and produces a new state:

$W$ maps the input into the hidden space, $U$ maps the old state forward, and tanh keeps every entry between -1 and 1. This is the classic "Elman" RNN, from Jeffrey Elman's 1990 paper "Finding structure in time", and it is exactly what PyTorch's nn.RNN computes by default. The state starts as $h_0=0$.

The key design choice: the same $W$, $U$ and $b$ are used at every step. That weight sharingUsing one set of weights at many positions (every time step in an RNN, every image location in a CNN), so the model size does not grow with the input and what is learned in one place applies everywhere. is why an RNN handles any length and why what it learns about "not" at position 3 also applies at position 30. It also keeps the model small. With 300-dimensional word vectors and a hidden size of 128, $W$ has 128 x 300 = 38,400 weights, $U$ has 128 x 128 = 16,384 and $b$ has 128: 54,912 parameters, whether the sentence has 5 words or 5,000.

Step 2: Unroll through time

Drawing the cell once per time step is called unrollingRedrawing a recurrent network as a feed-forward chain with one copy of the cell per time step, all copies sharing the same weights.. The unrolled picture is an ordinary feed-forward network, just a very deep one: a 100-word review becomes a 100-layer network. Everything you know from Forward pass by hand applies; the only twist is that every layer uses the same weights.

What you read off the states depends on the task:

• Many to one: classify a whole sequence (sentiment, spam) from the last state, as in Figure 1.
• Many to many: an output at every step, such as tagging each word with its part of speech, or predicting the next character of text.
• Sequence to sequence: read one sequence and write another of a different length, such as translation. That needs an encoder and a decoder, the topic of Seq2seq and attention .

Step 3: Backpropagation through time

Training an RNN means running ordinary backpropagation ( Backpropagation ) on the unrolled network. This is called backpropagation through timeBackpropagation applied to an RNN unrolled over the sequence; because the weights are shared, the gradient for each weight is the sum of its contributions from every time step. (BPTT), a name popularized by Paul Werbos's 1990 paper "Backpropagation through time: what it does and how to do it". Two details matter:

• Gradients add up over time. Because $U$ appears at every step, its gradient is the sum of one contribution per step. The code below checks that sum against a finite difference.
• Each step back multiplies the gradient by the slope of tanh and by $U^{\top }$:

To learn that the word at step 1 decided the label at step 100, the error must survive 99 of those multiplications.

For long sequences (a novel, hours of audio) the full unrolled graph does not fit in memory. In practice you use truncated BPTTRunning backpropagation through time over windows of a fixed number of steps, carrying the hidden state forward between windows but not the gradient.: process the sequence in chunks of, say, 100 steps, carry the hidden state from one chunk to the next, but stop the gradient at the chunk boundary. It is cheaper, and it means the model is never directly trained to connect events more than one chunk apart.

Step 4: Vanishing and exploding gradients

Here is the catch. Take a one-unit RNN, so $U$ is a single number $w$. Each step back multiplies the gradient by $w$ times a tanh slope, and that slope is at most 1. With $w=0.5$, after 20 steps the gradient has shrunk by at least $0.5^{20}\approx 9.5\times 10^{-7}$: the first words get essentially no learning signal. With $w=1.5$ and inputs that keep tanh near its linear part, the factor can grow toward $1.5^{20}\approx 3,325$ and one update can throw the weights far off.

The same thing happens in many dimensions, with the eigenvalues of $U$ in place of $w$. Pascanu, Mikolov and Bengio (2013) make this precise: for tanh, if the largest eigenvalue of the recurrent matrix has absolute value below 1, the long-term gradient components vanish, and a value above 1 is necessary (but not sufficient) for them to explode. For a sigmoid the threshold is 4 instead of 1, because the sigmoid's slope is at most 1/4. They describe vanishing gradientsGradients that shrink exponentially as they are propagated back through many time steps or layers, so early inputs barely affect learning. as long-term components going "exponentially fast to norm 0, making it impossible for the model to learn correlation between temporally distant events". The problem itself was analysed earlier, by Bengio, Simard and Frasconi in 1994.

This is the RNN version of the deep-network problem in Vanishing and exploding gradients, residual connections , made worse because the same matrix is applied again and again: a deep feed-forward net multiplies different matrices, while an RNN raises one matrix to a power.

Try it

1. Start with w = 0.9 and tanh. The bars shrink steadily to the left: step 0 gets a gradient about a hundred million times smaller than the last step. Raise the sequence length to 60 and watch the left half turn grey.
2. Switch the activation to none (linear) and push w to 1.5. The bars on the left turn red: the gradient explodes. Now switch back to tanh. The state saturates near plus or minus 1, the tanh slopes collapse, and the gradient vanishes instead of exploding.
3. With tanh and w = 1.0, raise the input weight u to 2. Bigger inputs push tanh into its flat region more often, so the gradient vanishes faster even though w is unchanged.

Going deeper

Code: an RNN and BPTT in numpy

import numpy as np
rng = np.random.default_rng(0)

# A vanilla RNN cell: h_t = tanh(W x_t + U h_{t-1} + b); one output y = v . h_T at the end
D, H, T = 3, 8, 20
W, U = rng.normal(0, 0.5, (H, D)), rng.normal(0, 1 / np.sqrt(H), (H, H))
b, v = np.zeros(H), rng.normal(0, 0.5, H)
xs = rng.normal(0, 1, (T, D))

def forward(U):
    hs = [np.zeros(H)]
    for x in xs:                                  # the same weights at every step
        hs.append(np.tanh(W @ x + U @ hs[-1] + b))
    return hs, v @ hs[-1]

hs, y = forward(U)
print(f"{T} steps, hidden size {H}; output y = {y:.4f}")

# Backpropagation through time: walk the unrolled chain backwards, summing U's gradient
dU, dh = np.zeros_like(U), v.copy()               # dy/dh_T = v
norms = []
for t in range(T, 0, -1):
    norms.append(np.linalg.norm(dh))              # size of dy/dh_t
    dz = dh * (1 - hs[t] ** 2)                    # back through tanh
    dU += np.outer(dz, hs[t - 1])                 # this step's share of the shared weight
    dh = U.T @ dz                                 # pass to the previous hidden state
eps, i, j = 1e-6, 2, 5                            # check one entry against finite differences
Up, Um = U.copy(), U.copy(); Up[i, j] += eps; Um[i, j] -= eps
num = (forward(Up)[1] - forward(Um)[1]) / (2 * eps)
print(f"BPTT dy/dU[{i},{j}] = {dU[i, j]:.6f}, finite difference = {num:.6f}")
print("|dy/dh_t| going back from t=20:", " ".join(f"{n:.3f}" for n in norms[::4]))

# Scale the recurrent weights and watch the gradient from 50 steps back
T = 50; xs = rng.normal(0, 1, (T, D))
for scale in [0.5, 1.0, 2.0]:
    Us = scale * U / np.max(np.abs(np.linalg.eigvals(U)))   # largest |eigenvalue| = scale
    hs, _ = forward(Us); dh = v.copy()
    for t in range(T, 0, -1):
        dh = Us.T @ (dh * (1 - hs[t] ** 2))
    print(f"largest |eigenvalue| {scale}: |dy/dh_0| after {T} steps = {np.linalg.norm(dh):.2e}")

20 steps, hidden size 8; output y = -1.0331
BPTT dy/dU[2,5] = 0.079882, finite difference = 0.079882
|dy/dh_t| going back from t=20: 1.400 0.469 0.048 0.003 0.000
largest |eigenvalue| 0.5: |dy/dh_0| after 50 steps = 1.62e-25
largest |eigenvalue| 1.0: |dy/dh_0| after 50 steps = 1.10e-12
largest |eigenvalue| 2.0: |dy/dh_0| after 50 steps = 7.70e-03

The forward pass is a loop that reuses W and U at every step. The backward loop walks the same steps in reverse, adding each step's share to dU, and the result matches a finite-difference estimate to six decimal places, so this really is the gradient. The printed norms show the vanishing problem directly: the gradient with respect to the hidden state falls from 1.4 at the last step to about 0.003 sixteen steps earlier.

The last three lines rescale $U$ so its largest eigenvalue has absolute value 0.5, 1.0 or 2.0 and measure the gradient 50 steps back. At 0.5 it is around $10^{-25}$, at 1.0 still around $10^{-12}$ because tanh's slope is below 1 almost everywhere. At 2.0 it does not explode in this run: the large state saturates tanh, and the small slopes cancel the growth. That is the "necessary but not sufficient" condition from the paper, seen in practice.

What people did about it

• Gradient clipping handles explosions. Pascanu et al. propose rescaling the gradient whenever its norm exceeds a threshold, keeping its direction; in PyTorch this is torch.nn.utils.clip_grad_norm_. It does nothing for vanishing.
• Careful initialization, for example an orthogonal $U$ whose eigenvalues all have absolute value 1, delays the problem but does not remove it once tanh saturates.
• Gated cells fix the cause. LSTMs and GRUs ( LSTM and GRU ) add a path along which the state is updated by addition, controlled by learned gates, so the gradient can travel many steps without being squashed. They replaced the plain RNN in almost every application.
• Attention ( Seq2seq and attention ) and then transformers ( Self-attention, step by step ) went further: every position can look at every other position directly, so the path from word 1 to word 100 is one step long instead of 99.

Sequential is slow

An RNN cannot compute $h_{50}$ before $h_{49}$. Within one sequence the work is strictly serial, so a GPU with thousands of cores waits on one step at a time. Batching many sequences helps, but long sequences still take time proportional to their length. This, as much as vanishing gradients, is why transformers, which process all positions of a sequence in parallel during training, took over.

Trade-offs

Common mistakes

• Thinking the unrolled copies have separate weights. There is one cell; the gradient for its weights is summed over all steps.
• Forgetting to reset the hidden state between unrelated sequences, so one review leaks into the next.
• Training without gradient clipping and blaming the data when the loss suddenly jumps to NaN.
• Expecting a plain RNN to use information from hundreds of steps back. Truncated BPTT and vanishing gradients both limit its reach.
• Padding short sequences with zeros and reading the "last" state from the padding instead of the true last word.

In the wild

Before gated cells, simple recurrent networks were the standard model for sequences in research on language and speech, and the Elman-style cell is still what deep learning libraries ship as their basic recurrent layer (nn.RNN in PyTorch, SimpleRNN in Keras). In practice you will meet its gated successors far more often: LSTMs and GRUs powered speech recognition and machine translation before transformers. The vanishing-gradient analysis from this lesson is the reason those gates exist, and gradient clipping is a standard safeguard when training deep networks of every kind.

Check yourself

Recap