Image generation

The big idea

A marketing team needs 40 product shots in four seasonal styles by Friday. A text-to-image model takes a sentence such as "a ceramic mug on a wooden table, morning light, shallow depth of field" and returns a new image in a few seconds. Most current models of this kind are diffusion modelsA generative model that learns to reverse a gradual noising process: starting from pure noise, it removes a little noise at each step until an image (or audio, or video) appears..

The trick is surprising. During training, the model sees millions of images with noise added, from a light sprinkle to pure static, and learns one skill: given a noisy image and its caption, predict the noise so it can be removed. To generate, you start from pure random noise and apply that skill again and again. Each step removes a little noise, nudged toward images that fit the caption, until a clean picture is left.

Think of a restorer cleaning a painting so covered in grime that nothing is visible. They clean a little at a time, and with each pass they guess more confidently what is underneath. The caption is a note pinned to the frame saying what the painting shows. Start from different grime and you uncover a different painting that fits the same note.

How it works

Training: learn to remove noise

Take a training image and its caption. Pick a random noise level, mix that much random noise into the image, and ask the network to predict exactly which noise was added. Repeat for millions of image-caption pairs. The network never learns "draw a mug"; it learns "given this noisy picture and the word mug, here is what is probably noise". That one skill is enough to generate.

Generation: many small steps

Generation runs the process backwards. Start with pure noise. At each step the network predicts the noise, which is the same as guessing what the clean image looks like; the samplerThe algorithm that decides how much predicted noise to remove at each step of diffusion generation, and how many steps to take (DDPM, DDIM and others). removes part of it and moves to a slightly less noisy picture. Early steps settle the big layout; late steps add fine detail. Fewer steps are faster but can be rougher; better samplers and distilled models are designed to need fewer.

Latent space: why it is fast enough

Denoising a full-resolution image hundreds of times would be slow. Latent diffusionDiffusion run in the compressed latent space of a pretrained autoencoder instead of on pixels, so each denoising step works on far fewer numbers. first compresses images with an autoencoder and runs the whole loop on the compressed grid. Stable Diffusion v1 uses a downsampling factor of 8: a 512 by 512 RGB image (786,432 numbers) becomes a 64 by 64 grid with 4 channels (16,384 numbers), 48 times fewer. Only the final decode works on pixels.

Text conditioning and guidance

The prompt enters through cross-attention: patches of the noisy latent attend to the prompt's vectors, the same mechanism as in Self-attention, step by step . On its own, that conditioning is often too weak; images drift away from the prompt. Classifier-free guidanceA diffusion sampling trick that runs the denoiser with and without the prompt and moves further in the direction of the difference, trading variety for prompt adherence. fixes this by predicting the noise twice per step, with the prompt and with an empty prompt, and extrapolating:

With $s=1$ you get the plain conditional prediction. With $s=7.5$, the default in the popular diffusers library, every step moves 7.5 times as far in the "more like the prompt" direction. Higher guidance follows the prompt more closely but gives less variety, and the Imagen authors note that very high values damage fidelity, producing highly saturated, unnatural images. Each step costs two network passes, so guidance roughly doubles compute.

Try it

1. With the prompt "a circle" and guidance 1, how many of the eight seeds follow the prompt? This toy's text conditioning is deliberately weak, like a real model's tendency to under-follow prompts.
2. Raise guidance to 2, then to 8. Adherence goes up. What happens to variety: how many seeds give a large circle versus a small one?
3. Set the prompt to "no prompt" and press "New seed" a few times. The shape is chosen by the noise alone. Then pick a prompt again: the same seeds now give circles.

The same sampler in Python, on 2D points instead of pixels, measured over 200 seeds:

# Diffusion with guidance on 2D points instead of pixels. "Training data":
# 40 "cat" points near (2, 0) and 40 "dog" points near (-2, 0). For a fixed
# dataset the ideal denoiser is exact: a softmax-weighted average of the data.
import numpy as np

rng = np.random.default_rng(0)
cats = rng.normal([2, 0], 0.5, (40, 2))
dogs = rng.normal([-2, 0], 0.5, (40, 2))
data = {"cat": cats, "dog": dogs, None: np.vstack([cats, dogs])}

T = 1000
abar = np.cumprod(1 - np.linspace(1e-4, 0.02, T))       # DDPM noise schedule

def guess_x0(x, ab, label):                               # best guess of clean point
    pts = data[label]
    logw = -((x - np.sqrt(ab) * pts) ** 2).sum(1) / (2 * (1 - ab))
    w = np.exp(logw - logw.max())
    return (w[:, None] * pts).sum(0) / w.sum()

def sample(label, g, seed, steps=20, cond=0.35):           # cond < 1: weak text signal
    x = np.random.default_rng(seed).normal(size=2)          # start from pure noise
    ts = np.linspace(T - 1, 0, steps + 1).round().astype(int)[:-1]
    for k, t in enumerate(ts):
        ab = abar[t]
        u = guess_x0(x, ab, None)                           # unconditional guess
        c = u + cond * (guess_x0(x, ab, label) - u)         # conditional guess
        x0 = u + g * (c - u)                                # classifier-free guidance
        e = (x - np.sqrt(ab) * x0) / np.sqrt(1 - ab)        # implied noise
        ab_next = abar[ts[k + 1]] if k + 1 < steps else 1.0
        x = np.sqrt(ab_next) * x0 + np.sqrt(1 - ab_next) * e  # DDIM step
    return x

for g in [0, 1, 2, 4, 8]:
    out = np.array([sample("cat", g, s) for s in range(200)])
    hit = out[:, 0] > 0
    print(f"guidance {g}: {hit.mean():5.0%} look like a cat, "
          f"spread of the cats {out[hit].std(0).mean():.2f}")

guidance 0:   48% look like a cat, spread of the cats 0.44
guidance 1:   69% look like a cat, spread of the cats 0.46
guidance 2:  100% look like a cat, spread of the cats 0.46
guidance 4:  100% look like a cat, spread of the cats 0.39
guidance 8:  100% look like a cat, spread of the cats 0.36

At guidance 0 the prompt is ignored and about half the samples land on each cluster. Guidance 2 is enough to make every sample a "cat", and higher values pull the samples tighter together: the spread falls from 0.46 to 0.36. That is the quality-versus-variety trade-off you tune with the guidance knob in real tools.

Prompting

A good prompt names, roughly in order: the subject ("a ceramic mug with a chipped rim"), the setting ("on a wooden table by a window"), composition ("close-up, centred, shallow depth of field"), lighting ("soft morning light") and style or medium ("product photo", "watercolour", "isometric 3D render"). Concrete visual words beat abstract ones: "golden hour backlight" works better than "beautiful".

• Fix the seed while you iterate. Change one word at a time with the same seed, so you see the effect of the word and not of new noise.
• Generate in batches. Results vary by seed; produce four to eight and pick.
• Negative prompts, where supported, replace the empty prompt in guidance, so the image is pushed away from "blurry, watermark, extra fingers".
• Know the weak spots. Older open models document trouble with legible text, faces and compositions such as "a red cube on top of a blue sphere". Test the model you use on text, counts and exact spatial relations before you rely on it for them.

Control: beyond the prompt

Words are a blunt tool for layout. Three techniques give you more control, all by changing what the denoising loop starts from or listens to:

• Image to image. Add noise to an existing image (a sketch, a photo, last week's banner) part of the way, then denoise with a new prompt. A strength setting from 0 to 1 says how far: low keeps the composition and changes surface details, 1 ignores the input. This is the idea behind SDEdit.
• Inpainting and outpainting. Paint a mask over the region to change ("replace the mug with a teapot") and only that region is regenerated, so it blends with the untouched pixels around it. Outpainting extends the canvas the same way.
• Structural control. ControlNetA method that adds spatial conditions such as edge maps, depth maps or human poses to a pretrained text-to-image diffusion model by training a copy of its encoder while the original stays frozen. feeds the denoiser an extra image such as an edge map, a depth map or a pose skeleton, so the output follows that layout while the prompt sets content and style. Related adapters condition on a reference image to keep a character or product consistent across shots.

Rights, provenance and misuse

Generated images raise questions that the model cannot answer for you: whether the training data was licensed, whether an output closely copies a protected work or a real person's likeness, and how viewers will know an image is synthetic. Check the model's license and the provider's terms for commercial use, keep humans reviewing anything published, and label AI-generated media. Watermarking and content credentials are covered in the video and audio generation lesson at the end of this module.

Trade-offs

Common mistakes

• Changing the prompt and the seed together, so you cannot tell what made the difference.
• Cranking guidance to the maximum to "make it follow the prompt", then getting harsh, samey images.
• Describing layout in words ("the cup on the left, slightly behind the plate") when a sketch with image-to-image or ControlNet would pin it down.
• Regenerating a whole image to fix one region instead of inpainting it.
• Publishing generated images without checking license terms, likeness and trademark issues, or labelling them.

In the wild

Check yourself

Recap