Multimodal models

The big idea

A transformer does not know what a word is. It sees a sequence of vectors, one per token, and learns patterns over them. So the trick to making a language model "see" is not to teach it a new skill from scratch, but to turn an image into a sequence of vectors that look like tokens, put them in the sequence next to the text, and let attention do the rest.

That is what most multimodal modelsModels that take in, and sometimes generate, more than one kind of data, such as text, images, audio or video, in a single model. do. A vision encoder cuts the image into small patches and turns each into a vector. A small projection layer maps those vectors into the language model's embedding space. To the transformer, the image is now just a few hundred extra tokens in the prompt, and it can attend from "What colour is the car?" straight to the patches showing the car.

Think of a translator who only reads English. To let them work on a Japanese document, you do not teach them Japanese; you hire an interpreter who renders each paragraph into English and slips it into the file. The interpreter is the vision encoder plus projector, and the translator works exactly as before.

How it works

Step 1: patches

A vision transformerA transformer applied to images by cutting the image into fixed-size patches, flattening each patch into a vector, and treating the patches as a token sequence. (ViT) cuts an image into a grid of square patches, often 14 or 16 pixels wide. A 224 by 224 image with 14-pixel patches becomes a 16 by 16 grid, 256 patches. Each patch is flattened into a vector of its pixel values and passed through a linear layer, plus a position embedding so the model knows where the patch sits in the grid.

Step 2: the vision encoder

The patch vectors go through a stack of transformer blocks, exactly like text, with full (not causal) attention, so every patch can see every other. Out come visual featuresVectors produced by a vision encoder that describe the content of image patches in a form a downstream model can use.: one vector per patch, now describing content ("fur", "edge of a wheel") rather than raw pixels. Many multimodal models use an encoder pretrained on image-caption pairs so its features already line up with language.

Step 3: projection into the token space

The encoder's vectors have a different size and meaning than the language model's token embeddings. A projectorA small network, often a single linear layer or a two-layer MLP, that maps a vision or audio encoder's output vectors into the language model's embedding space so they can be used as tokens., often a single linear layer or a small two-layer network, maps each one into the language model's embedding space. The results are image tokensVectors standing for parts of an image, produced by a vision encoder and projector, placed in a language model's input sequence alongside text tokens.: they do not correspond to any word in the vocabulary, but they live in the same space and the transformer treats them like any other token.

Step 4: one sequence

The prompt becomes text tokens, then a few hundred image tokens, then more text. The language model runs as usual. Attention lets "car" in the question pull information from the patches that show the car. Training typically happens in stages: first train only the projector on image-caption pairs so the image tokens "make sense" to the frozen language model, then fine-tune more of the model on visual question-answer and instruction data.

Try it

Try this:

1. At the defaults, a 336 by 336 image with 14-pixel patches is a 24 by 24 grid: 576 image tokens, the same context as roughly 430 words of text.
2. Raise both sides to 1,344 pixels. The grid becomes 96 by 96, 9,216 tokens. Doubling the side quadruples the tokens, and attention cost grows faster still ( Context windows and the KV cache ).
3. Turn on the 2 x 2 merge. The 1,344 image drops to 2,304 tokens. Merging or pooling neighbouring patches is a common way to keep high-resolution images affordable, at some cost in fine detail.

The pipeline in NumPy, with toy sizes:

# Turn an image into "image tokens" and splice them into a text sequence.
import numpy as np

rng = np.random.default_rng(0)
img = rng.random((32, 32, 3))                 # a toy 32 x 32 RGB image
p = 8                                         # patch size in pixels

# 1. Cut into p x p patches and flatten: 16 patches of 8*8*3 values
patches = img.reshape(32 // p, p, 32 // p, p, 3).transpose(0, 2, 1, 3, 4)
patches = patches.reshape(-1, p * p * 3)
print("patches:", patches.shape)

# 2. Vision encoder: one feature vector per patch (toy: one layer)
W_enc = rng.normal(size=(p * p * 3, 48)) / 14
feats = np.tanh(patches @ W_enc)              # (16, 48) visual features

# 3. A projector maps visual features into the LLM's embedding size (64)
W_proj = rng.normal(size=(48, 64)) / 7
img_tokens = feats @ W_proj                   # (16, 64) "image tokens"

# 4. Text tokens come from the LLM's own embedding table
vocab = rng.normal(size=(1000, 64))
text_ids = [17, 402, 9, 311]                  # e.g. "What is in"
question = [88, 5]                            # e.g. "this image ?"
seq = np.vstack([vocab[text_ids], img_tokens, vocab[question]])
print("sequence fed to the transformer:", seq.shape)

# Real numbers: patches per image for common sizes and patch widths
for side, ps in [(224, 14), (336, 14), (224, 16), (1344, 14)]:
    n = (side // ps) ** 2
    print(f"{side} x {side} px, {ps} px patches: {n:,} image tokens")

patches: (16, 192)
sequence fed to the transformer: (22, 64)
224 x 224 px, 14 px patches: 256 image tokens
336 x 336 px, 14 px patches: 576 image tokens
224 x 224 px, 16 px patches: 196 image tokens
1344 x 1344 px, 14 px patches: 9,216 image tokens

The 32 by 32 image becomes 16 patches of 192 values, then 16 image tokens of the same size as the language model's word embeddings, sandwiched between 4 and 2 text tokens into a 22-token sequence.

Going deeper

High-resolution images

Encoders are trained at a fixed resolution, but users send screenshots and documents full of small text. Common approaches: cut a large image into tiles, encode each tile at the native resolution plus a downscaled overview, and concatenate the tokens; or train the encoder to accept variable sizes directly. Either way, detailed images cost many tokens, which is why APIs often let you choose a low or high detail setting and price images by size.

Audio

Audio enters the same way. The waveform is converted into a spectrogramA picture of sound: energy at each frequency over time, usually on a perceptual (mel) frequency scale. The standard input to speech models., an audio encoder turns short frames of it into vectors, and a projector maps them into the language model's space. Longer audio means more tokens, at a rate set by the encoder's frame rate and any pooling. Speech-in, text-out models handle transcription, translation and spoken questions this way ( Speech: ASR and TTS ).

Any-to-any models

Going the other way, generating images or audio, needs an output path. Two common designs:

• Discrete media tokens: a separate tokenizer turns images or audio into codes from a fixed codebook. The language model learns to predict those codes like words, and a decoder turns them back into pixels or sound.
• Hand-off to a generator: the language model produces a description or a set of conditioning vectors, and a dedicated image or audio generator, often a diffusion model ( Image generation ), produces the output.

Models trained to read and write several modalities in one sequence are called any-to-any or natively multimodal. The engineering principle stays the same as the input side: everything becomes a token sequence.

Trade-offs

Common mistakes

• Thinking the model "looks" at the image the way you do. It reads a few hundred patch vectors; small text or tiny objects may not survive the patching and pooling.
• Sending a huge screenshot when a crop of the relevant region would be cheaper and more accurate.
• Forgetting that images consume context tokens and cost money like text does.
• Trusting counts and precise positions (how many people, which pixel) without checking. Spatial precision is a known weak spot.
• Treating an image caption from the model as ground truth for downstream automation without evaluating it ( Vision-language models in practice ).

In the wild

Check yourself

Recap