Generative adversarial networks

The big idea

A VAE learns to generate by reconstructing its training data pixel by pixel. A generative adversarial networkA pair of networks trained against each other: a generator that turns random noise into samples, and a discriminator that tries to tell those samples from real data. GAN for short. (GAN) never compares a generated image with a particular real one. Instead it trains a second network to tell real from fake, and the generator learns by trying to fool it.

Think of a forger and an art inspector. The forger paints fakes; the inspector looks at a mix of real paintings and fakes and says which is which. Each time the inspector catches a fake, the forger learns what gave it away. Each time a fake slips through, the inspector learns to look closer. If both keep improving, the fakes end up indistinguishable from real paintings. Neither needs to be told what a "good painting" is; the competition defines it.

How it works

Step 1: Two players

The generatorThe network in a GAN that maps random noise to a sample, such as an image. $G$ maps random noise $z$ (for example 100 or 512 numbers from a normal distribution) to a sample $G(z)$. The discriminatorThe network in a GAN that outputs the probability that its input came from the real data rather than the generator. $D$ is an ordinary binary classifier: $D(x)$ is the probability that $x$ is real. Goodfellow and colleagues introduced the setup in 2014 as a minimax game over one value function:

$D$ maximizes it: the log loss of a classifier with real labelled 1 and fake labelled 0. $G$ minimizes the second term: it wants $D(G(z))$ close to 1.

Step 2: What the game converges to

For a fixed generator with sample distribution $p_g$, the paper shows the best discriminator is

Plug $D^{\ast }$ back in and the generator's objective becomes $-\log 4+2\cdot \mathrm{JSD}(p_{\text{data}}\Vert p_g)$, where JSD is the Jensen-Shannon divergence. It is minimized only when $p_g=p_{\text{data}}$, at which point $D^{\ast }(x)=1/2$ everywhere: the inspector can do no better than a coin flip. That is the ideal. In practice both players are networks updated by small gradient steps, so the game only approximately reaches it, if at all.

Step 3: Training in practice

The paper's algorithm alternates: $k$ gradient steps on the discriminator (they used $k=1$), then one step on the generator. Training the discriminator to completion each time would be too expensive and would overfit.

One change from the textbook game is almost universal. Early in training, fakes are obviously fake and $D(G(z))$ is close to 0, where $\log (1-D(G(z)))$ is flat, so the generator gets almost no gradient. The paper notes this saturation and suggests instead training the generator to maximize $\log D(G(z))$, the non-saturating lossThe generator objective that maximizes log D(G(z)) instead of minimizing log(1 - D(G(z))): same fixed point, much stronger gradients when the discriminator confidently rejects fakes.. It has the same fixed point but much stronger gradients early on.

Step 4: Why GANs are hard to train

Gradient descent on a two-player game is not gradient descent on one loss. Each player's step changes the other's problem, so training can oscillate instead of settling, and the losses say little about sample quality.

The best-known failure is mode collapseA GAN failure where the generator produces only one or a few kinds of output, covering a small part of the data distribution, because those outputs currently fool the discriminator.: the generator finds a few outputs the discriminator currently accepts and maps most of its noise to them, instead of covering all the data. The original paper already warns that $G$ must not be trained too much without updating $D$, to avoid "the Helvetica scenario" in which $G$ collapses too many values of $z$ to the same output. The discriminator then learns to reject that output, the generator jumps to another one, and the pair can chase each other around without ever covering the data.

Many fixes followed: different losses (Wasserstein GANs and gradient penalties), regularizing the discriminator, normalizing layers, and training on minibatch statistics so that a batch of identical samples is easy to spot.

Step 5: StyleGAN highlights

StyleGAN (Karras, Laine and Aila, 2018) is a reference design for high-resolution face generation. Its main changes were to the generator:

• A mapping network. An 8-layer MLP first maps the noise $z$ to an intermediate code $w$ (both 512 numbers). The synthesis network starts from a learned constant rather than from $z$.
• Styles at every resolution. Learned transforms of $w$ set the scale and bias of adaptive instance normalization (AdaIN) after each convolution, so $w$ controls features scale by scale. Copying the coarse styles (4 by 4 to 8 by 8) from one face to another brings pose, face shape, general hair style and eyeglasses; finer styles bring colours and small details.
• Per-pixel noise injected at each layer handles stochastic details such as freckles and hair placement, separately from identity and pose.
• A new dataset. FFHQ, 70,000 high-quality face images at 1024 by 1024, with more variety than earlier face datasets.

Going deeper

Code: a 1-D GAN and a collapse in miniature

import numpy as np

# A 1-D GAN in numpy. Real data: a mix of two bumps at -2 and +2. The generator turns noise z into
# a number; the discriminator outputs the probability that a number is real. Both are 1-32-1 nets.
rng = np.random.default_rng(0)
real = lambda n: np.where(rng.random(n) < 0.5, -2.0, 2.0) + 0.25 * rng.normal(size=n)
def net(seed): r = np.random.default_rng(seed); return [r.normal(0, 1, (1, 32)), np.zeros(32), r.normal(0, 0.3, (32, 1)), np.zeros(1)]
def fwd(p, x): h = np.tanh(x @ p[0] + p[1]); return h, h @ p[2] + p[3]
def back(p, x, h, dout):                               # gradients of the output w.r.t. params and input
    dh = dout @ p[2].T * (1 - h ** 2)
    return [x.T @ dh, dh.sum(0), h.T @ dout, dout.sum(0)], dh @ p[0].T
sig = lambda a: 1 / (1 + np.exp(-a))

# 1. Why the generator does not minimize log(1 - D): when D confidently rejects a fake (D = 0.01),
#    the gradient with respect to D's logit is tiny for that loss and large for the -log D version.
d = 0.01
print(f"D(G(z)) = {d}: gradient of log(1 - D) {-d:.2f}, of -log D {-(1 - d):.2f} (per unit of logit)")

def train(steps=4000, lr=0.02, seed=1):
    G, D = net(seed), net(seed + 1)
    for t in range(steps):
        z = rng.normal(size=(128, 1)); x = real(128)[:, None]
        hg, fake = fwd(G, z)
        for xs, label in [(x, 1.0), (fake, 0.0)]:          # discriminator: log loss, real = 1, fake = 0
            hd, logit = fwd(D, xs)
            g, _ = back(D, xs, hd, (sig(logit) - label) / 128)
            for q, gq in zip(D, g): q -= lr * gq
        hd, logit = fwd(D, fake)                          # generator: non-saturating loss, -log D(G(z))
        _, dx = back(D, fake, hd, (sig(logit) - 1) / 128)
        g, _ = back(G, z, hg, dx)
        for q, gq in zip(G, g): q -= lr * gq
        if t + 1 in (200, 1000, 4000):
            s = fwd(G, rng.normal(size=(4000, 1)))[1].ravel()
            print(f"step {t + 1:4d}: fakes near -2 {np.mean(np.abs(s + 2) < 0.75):4.0%}, near +2 {np.mean(np.abs(s - 2) < 0.75):4.0%}, "
                  f"elsewhere {np.mean((np.abs(s + 2) >= 0.75) & (np.abs(s - 2) >= 0.75)):4.0%}, D(real) {sig(fwd(D, real(4000)[:, None])[1]).mean():.2f}")

    return G, D

G, D = train()
# 2. Mode collapse in miniature: freeze D and let each fake chase it alone (the paper's warning about
#    training G too much without updating D). Every fake slides uphill to wherever D peaks.
x = fwd(G, rng.normal(size=(2000, 1)))[1]
print(f"before: fakes spread (std) {x.std():.2f}, distinct values (to 0.1) {len(np.unique(np.round(x, 1)))}")
for _ in range(3000):
    hd, logit = fwd(D, x)
    x -= 0.05 * back(D, x, hd, sig(logit) - 1)[1]         # gradient step on -log D(x) for each fake
print(f"after:  fakes spread (std) {x.std():.2f}, distinct values (to 0.1) {len(np.unique(np.round(x, 1)))}, "
      f"at {', '.join(f'{v:.1f}' for v in np.unique(np.round(x, 1)))}")

D(G(z)) = 0.01: gradient of log(1 - D) -0.01, of -log D -0.99 (per unit of logit)
step  200: fakes near -2  19%, near +2  19%, elsewhere  62%, D(real) 0.50
step 1000: fakes near -2  33%, near +2  32%, elsewhere  35%, D(real) 0.51
step 4000: fakes near -2  45%, near +2  45%, elsewhere  10%, D(real) 0.52
before: fakes spread (std) 2.04, distinct values (to 0.1) 51
after:  fakes spread (std) 1.88, distinct values (to 0.1) 2, at -1.9, 1.9

Real data is a mix of two bumps at -2 and +2. Generator and discriminator are both tiny 1-32-1 networks with hand-written gradients, updated alternately with the non-saturating loss.

• The saturation problem. When the discriminator rejects a fake with $D=0.01$, the gradient of $\log (1-D)$ with respect to its logit is -0.01, while $-\log D$ gives -0.99, almost 100 times stronger.
• Learning both modes. After 200 steps only 19 percent of fakes sit near each bump and 62 percent are in between. After 4,000 steps it is 45 and 45 percent, with 10 percent elsewhere, and the discriminator outputs 0.52 on real data: close to the coin flip of the ideal.
• Collapse in miniature. Freeze the trained discriminator and let 2,000 fakes each climb $\log D$ on their own, the "generator trained too long against a fixed discriminator" of the paper. Their 51 distinct values (to one decimal) collapse to just 2, at -1.9 and 1.9: each fake runs to wherever the frozen discriminator peaks, and all variety within each mode is lost.

Trade-offs

Common mistakes

• Using the textbook generator loss $\log (1-D(G(z)))$ and getting no learning early on; use the non-saturating version.
• Letting one player win outright: a perfect discriminator gives the generator no useful gradient, and a starved one lets it collapse.
• Judging progress by the losses. They oscillate by design; look at samples and use quality and diversity metrics.
• Checking only a few hand-picked samples, which hides mode collapse. Look at many random samples at once.
• Comparing GANs by visual impressions alone; use the same metric and the same evaluation set for every model.

Where you will see this

GANs have been used for photorealistic faces, image-to-image translation (sketch to photo, day to night), super-resolution and data augmentation. Many current text-to-image systems use diffusion models instead, but adversarial losses live on inside them: the latent diffusion paper's autoencoder, for example, is trained with a perceptual loss plus a patch-based adversarial objective, which keeps reconstructions sharp where pixel losses alone would blur them.

Check yourself

Recap