Video and audio generation overview

The big idea

A small studio wants a five-second product clip for each of 200 items, plus a short music bed and a few sound effects. A few years ago that meant a shoot and a composer. Today a model can turn a prompt or a product photo into a short clip with camera motion, and other models write music from a description or a hummed tune. The results can be striking. They are also expensive to make, uneven from one try to the next, and hard to tell apart from real footage.

That last point is why this lesson has two halves. The first is how generation works and what it costs. The second is provenanceThe recorded history of a piece of media: who or what created it, with which tools, and what edits were made since.: how a generated clip can carry a reliable label saying it was made by a model, even after it has been shared and re-encoded.

Think of a banknote. Printing a convincing one is hard work, and printing many is expensive. But the real defence is not the cost; it is the watermark and security features built in when the note is made, which anyone can check later. Generated media needs the same: marks added at creation that survive handling, plus a way to check them.

How it works

From images to video

A video is a stack of images that must agree with each other. Generating frames one at a time and hoping they line up gives flicker and objects that change shape. Current models instead treat the whole clip as one block. A video autoencoder compresses it in space and in time, the compressed block is cut into spacetime patchesSmall blocks of a compressed video that span a few pixels in each direction and a few frames in time; video diffusion transformers use them as tokens., and a diffusion transformerA diffusion model whose denoising network is a transformer over patches instead of a convolutional U-Net. denoises all of them together. OpenAI's 2024 technical report on Sora describes exactly this recipe, and notes that because images are just one-frame videos, one model can be trained on both and can generate clips of different lengths, resolutions and aspect ratios.

The same machinery gives you the editing tools from Image generation , now in time: animate a still image, extend a clip forwards or backwards, or noise an existing video part way and re-denoise it in a new style (the SDEdit idea).

Audio and music

Sound has the same shape as video at a smaller scale: a long signal with structure at several time scales. Speech generation is covered in Speech: ASR and TTS . For music, research models generate from text such as "a calming violin melody backed by a distorted guitar riff", and can be conditioned on a hummed or whistled melody. Sound effects and short music clips come from text-to-audio models, including an open-weights one trained on Creative Commons audio. Cloning a real person's voice is the most sensitive case, which is why consent and labelling matter even more for audio.

What still goes wrong

The Sora report is candid about limits that remain typical: physics such as glass shattering is not modelled correctly, object state does not always change as it should, for example when food is eaten, long clips drift into incoherence, and objects can appear from nowhere. Expect to generate several takes and pick, and plan human review before anything is published.

Why video costs so much

Use the published numbers of one open model, Wan (2025). Its video autoencoder compresses time by 4 and each side by 8, and the transformer takes 2 by 2 patches of that, so the number of tokens per denoising step is:

where T + 1 is the number of frames. A 5-second clip at 16 frames per second has 81 frames, so at 1280 by 720: (1 + 80/4) x 45 x 80 = 21 x 3,600 = 75,600 tokens. One still image at that size is 3,600 tokens. The clip has 21 times the tokens, and since full attention compares every token with every other one, about 21 squared = 441 times the attention work per step, repeated for every denoising step. The Wan authors report that once sequences reach a million tokens, attention can take up to 95 percent of end-to-end training time.

Try it

1. Start at 5 seconds, 16 frames per second, 1280 x 720. Confirm the 75,600 tokens from the worked example. How many times the attention work of one image is that?
2. Double the length to 10 seconds. Tokens roughly double. What happens to the attention bar, and why do providers cap clip length?
3. Go from 1280 x 720 to 1920 x 1080 at 5 seconds. Tokens grow by about 2.25 times; attention by about 5. Which is cheaper: generating at 720p and upscaling, or generating at 1080p?

Provenance in miniature: a signed record bound to the exact bytes of a file.

# A toy content credential: a signed record that binds claims ("made by an
# AI model") to the exact bytes of a file through a hash. Real C2PA manifests
# use X.509 certificates and public-key signatures; HMAC stands in for that here.
import hashlib, hmac, json

SIGNING_KEY = b"generator-private-key"           # stand-in for a private key

def make_credential(media: bytes, claims: dict) -> dict:
    record = {"sha256": hashlib.sha256(media).hexdigest(), **claims}
    body = json.dumps(record, sort_keys=True).encode()
    return {"record": record, "sig": hmac.new(SIGNING_KEY, body, "sha256").hexdigest()}

def verify(media: bytes, cred) -> str:
    if cred is None:
        return "no credential: unknown origin (not proof it is real)"
    body = json.dumps(cred["record"], sort_keys=True).encode()
    if not hmac.compare_digest(cred["sig"], hmac.new(SIGNING_KEY, body, "sha256").hexdigest()):
        return "credential tampered with"
    if cred["record"]["sha256"] != hashlib.sha256(media).hexdigest():
        return "file changed after signing"
    return f"valid: {cred['record']['source']}"

video = bytes(range(256)) * 40                     # stand-in for a generated clip
cred = make_credential(video, {"source": "trainedAlgorithmicMedia", "tool": "video model"})

print("1. as generated:     ", verify(video, cred))
print("2. one byte edited:  ", verify(video[:-1] + b"\x00", cred))
forged = {"record": {**cred["record"], "source": "camera"}, "sig": cred["sig"]}
print("3. claim rewritten:  ", verify(video, forged))
print("4. metadata stripped:", verify(video, None))

1. as generated:      valid: trainedAlgorithmicMedia
2. one byte edited:   file changed after signing
3. claim rewritten:   credential tampered with
4. metadata stripped: no credential: unknown origin (not proof it is real)

The record proves origin only while it travels with the file. Case 4 is the weak spot: strip the metadata and nothing says the clip was generated, and a missing credential proves nothing either way. That is why credentials are paired with watermarks.

Going deeper

Watermarks: marks inside the content

An invisible watermarkA signal hidden in the content itself (pixels, audio samples or word choices) that a detector can find later, used to identify media as AI-generated. is embedded in the pixels or audio samples at generation time. Google's SynthID, for example, is described as imperceptible to people, designed to survive cropping, filters, frame-rate changes and lossy compression for images and video, and to survive added noise, MP3 compression and speed changes for audio; a verification portal launched in 2025. The limits: a mark can be read only by its own detector, a missing watermark proves nothing, and a determined attacker may degrade one. Watermarks raise the cost of passing off generated media; they do not make it impossible.

Content credentials: signed history

The C2PACoalition for Content Provenance and Authenticity: an open standard for attaching signed provenance records (content credentials) to media files. standard attaches a signed manifest, called a content credential, to a file. It holds assertions such as the creating tool, edits made and whether AI was used (through a digital source type field), bound to the file by cryptographic hashes and signed with X.509 certificates. Any change to the file or the record breaks the match. Metadata can be stripped, so the standard's "durable" credentials combine this hard binding with a soft binding, a watermark or fingerprint that lets a checker find the manifest again in a cloud store. The C2PA explainer is careful about one more limit: provenance tells you where content came from, not whether it is true.

Rules are arriving

The EU AI Act's transparency article requires providers of systems that generate synthetic audio, images, video or text to mark outputs in a machine-readable, detectable way, and requires deployers who publish deepfakes to disclose that they are artificial. It has applied since 2 August 2026. The Commission's FAQ notes one grace period: systems already on the market before that date have until 2 December 2026 to meet the marking duty. Other jurisdictions and platforms have their own labelling rules; see the AI regulation and governance lesson in the safety module.

Trade-offs

Common mistakes

• Estimating video cost from image cost per frame. Tokens grow with length and resolution, and attention work grows with their square.
• Treating a missing watermark or credential as proof that media is real.
• Expecting correct physics, stable text and consistent objects in long clips without reviewing every take.
• Cloning a voice or likeness without documented consent, or publishing generated media without a label.
• Stripping metadata in your own processing pipeline (resizing, transcoding) and silently losing the credentials you were given.

In the wild

Check yourself

Recap