Vision-language models in practice

The big idea

Your finance team gets 3,000 receipt photos a month and types each one into a spreadsheet. A vision-language modelA language model that also accepts images as input and answers in text about what they show, including any text inside them. (VLM) can read a photo, find the merchant, date, line items and total, and return them as JSON in a few seconds. The same model can describe a product photo, read a chart, answer questions about a 40-page PDF or tell an agent which button to click in a screenshot.

What it cannot do is be trusted blindly. It reads the image after it has been resized and cut into patches, so small print can blur into guesses. It counts roughly, places things approximately, and when a digit is unreadable it may produce a plausible one instead of saying so.

Think of a fast, well-read new hire looking at documents over your shoulder from across the desk. They understand what a receipt is and what a chart means far better than any template matcher. But if the print is tiny they squint and guess, and you would not let them file expense reports without someone checking that the numbers add up.

How it works

What vision-language models are good at

• Scene understanding: what is in a photo, what is happening, whether a product image matches its listing.
• Reading text (OCROptical character recognition: turning text that appears in an image into machine-readable characters.): signs, labels, receipts, handwriting of reasonable quality. Unlike a classic OCR engine, the model also understands what it reads, so "find the invoice number" works without a template.
• Documents: forms, invoices, contracts, slides. Layout carries meaning (a number under "Total" is the total) and the model uses it.
• Charts and diagrams: the trend, the largest bar, what the axes mean. Exact values read off a plot are estimates.
• Screenshots: error dialogs, dashboards and app screens, which is what lets computer-use agents work ( Coding agents and computer-use agents ).

What the model actually receives

Before the model sees your image, the API usually shrinks it. Each provider has a maximum resolution and a token budget per image, and larger images are downscaled to fit, keeping their aspect ratio. The model then reads the image as a grid of patches, one visual tokenA token that stands for one patch of an image in a vision-language model's input. Images cost context and money through these tokens, like text does. per patch, so images cost context and money the same way text does.

Worked example with one provider's documented rule (28 by 28 pixel patches, at most 1,568 tokens on its standard tier). A 1920 by 1080 screenshot would need 69 by 39 = 2,691 patches, too many, so it is shrunk to 1456 by 819: 52 by 30 = 1,560 tokens. Every detail is now 76 percent of its original size. A 12-megapixel phone photo shrinks much more, to about 31 percent of each side, and text that was 28 pixels tall becomes about 9.

Prompting with images

• Image first, question after. Providers recommend putting documents and PDFs before the question, the same advice as for long text; do the same with images.
• Be specific. "Extract merchant, date (YYYY-MM-DD), each line item with quantity and unit price, tax and total" beats "What does this receipt say?"
• Ask for structure. A JSON schema ( Structured outputs ) gives you fields your code can check.
• Give an escape hatch. Tell the model to write null and list the field under unreadable when it cannot read something. Without that, a blurry digit becomes a confident guess.
• Transcribe, then reason. For charts and tables, ask for the values it reads first, then the conclusion. You can check the transcription, and errors stop compounding silently.
• Label multiple images. "Image 1 is the invoice, image 2 is the delivery note. List differences." Otherwise the model may mix them up.
• Crop rather than shout. If the answer lives in one corner, send that corner at full resolution. Anthropic reports a consistent gain on image evals when the model has a crop tool it can use to zoom in on a region itself.

Where they fail

• Small text and dense tables, especially after downscaling or heavy JPEG compression.
• Counting and precise positions. Approximate counts of many small objects; bounding boxes and coordinates that are close but not exact.
• Fine spatial relations. Whether two thin lines cross or two circles touch can be surprisingly hard (see the BlindTest study below).
• Low-quality input: rotated, blurry or tiny images (under about 200 pixels) raise the error rate.
• Confident filling-in. When text is illegible, the model can produce something plausible rather than admitting it cannot read it. This is the image version of hallucination, covered in the Hallucinations lesson of the safety module.
• Detecting AI-generated images. Do not use a general VLM as a fake detector.

Try it

1. Start with the receipt photo, whole image, standard tier. How tall are the digits after resizing? Now crop to half of each side. The token count barely changes, but what happens to the digits?
2. Switch to the laptop screenshot. Compare the standard and high-resolution tiers. How many more tokens does high resolution cost, and what does it buy you?
3. Try the chart. Sending it whole on the standard tier shrinks the axis labels; cropping to half makes it cheaper and sharp. When would cropping lose information you need?

The resizing rule in Python, following the reference implementation in Anthropic's docs. The last three lines are the receipt from the lab:

# What does a vision model actually receive? Resize an image to fit a model's
# limits (long edge and visual-token cap), then count 28 x 28 px patches.
# Follows the reference rule in Anthropic's vision docs (2026).
from math import ceil

PATCH = 28
TIERS = {"standard": (1568, 1568), "high-res": (2576, 4784)}  # (max edge, max tokens)

def tokens(w, h):
    return ceil(w / PATCH) * ceil(h / PATCH)

def fits(w, h, edge, cap):
    return (ceil(w / PATCH) * PATCH <= edge and ceil(h / PATCH) * PATCH <= edge
            and tokens(w, h) <= cap)

def fit(W, H, edge, cap):
    """Largest aspect-preserving size that fits both limits."""
    if fits(W, H, edge, cap):
        return W, H
    if H > W:                                  # search along the long edge
        h, w = fit(H, W, edge, cap)
        return w, h
    aspect = W / H
    lo, hi = 1, W                              # lo always fits, hi never does
    while lo + 1 < hi:
        mid = (lo + hi) // 2
        if fits(mid, max(round(mid / aspect), 1), edge, cap):
            lo = mid
        else:
            hi = mid
    return lo, max(round(lo / aspect), 1)

for W, H in [(1000, 1000), (1920, 1080), (1075, 1520), (3840, 2160)]:
    row = [f"{W}x{H}:"]
    for name, (edge, cap) in TIERS.items():
        w, h = fit(W, H, edge, cap)
        row.append(f"{name} {w}x{h} = {tokens(w, h)} tokens")
    print("  ".join(row))

# A phone photo of a receipt: 3024 x 4032, printed digits about 28 px tall.
for crop in [1, 0.5, 0.25]:
    W, H = int(3024 * crop), int(4032 * crop)
    w, h = fit(W, H, *TIERS["standard"])
    print(f"send {W}x{H}: model sees {w}x{h}, {tokens(w, h)} tokens, "
          f"digits {28 * w / W:.1f} px tall")

1000x1000:  standard 1000x1000 = 1296 tokens  high-res 1000x1000 = 1296 tokens
1920x1080:  standard 1456x819 = 1560 tokens  high-res 1920x1080 = 2691 tokens
1075x1520:  standard 924x1307 = 1551 tokens  high-res 1075x1520 = 2145 tokens
3840x2160:  standard 1456x819 = 1560 tokens  high-res 2576x1449 = 4784 tokens
send 3024x4032: model sees 952x1270, 1564 tokens, digits 8.8 px tall
send 1512x2016: model sees 952x1270, 1564 tokens, digits 17.6 px tall
send 756x1008: model sees 756x1008, 972 tokens, digits 28.0 px tall

Cropping to the region you need is the cheapest quality fix there is: the quarter crop costs fewer tokens than the full photo and keeps the digits at full size.

Going deeper

Check what you can, review the rest

Extraction quality is not a property of the model alone; it is the model plus your checks. Receipts and invoices are full of redundancy: line items should sum to a subtotal, tax should match a rate, dates must exist. A few lines of code catch many misreads before anyone sees them.

# Never trust an extraction you can check. These dicts stand in for JSON a
# vision model returned for three receipt photos (made-up values).
from datetime import date

def checks(r):
    problems = []
    items = round(sum(qty * price for _, qty, price in r["items"]), 2)
    if abs(items + r["tax"] - r["total"]) > 0.01:
        problems.append(f"items {items} + tax {r['tax']} != total {r['total']}")
    try:
        date.fromisoformat(r["date"])
    except ValueError:
        problems.append(f"bad date {r['date']!r}")
    if any(v is None for v in (r["merchant"], r["total"])):
        problems.append("missing required field")
    if r.get("unreadable"):
        problems.append(f"model flagged unreadable: {r['unreadable']}")
    return problems

receipts = [
    {"merchant": "Corner Cafe", "date": "2026-03-14", "tax": 1.20, "total": 16.20,
     "items": [("latte", 2, 4.50), ("bagel", 2, 3.00)]},
    {"merchant": "Hardware Co", "date": "2026-03-15", "tax": 2.40, "total": 32.40,
     "items": [("hinge", 4, 6.50), ("screws", 1, 3.80)]},   # 6.50 misread?
    {"merchant": "Book Nook", "date": "2026-02-30", "tax": 0.00, "total": 12.00,
     "items": [("paperback", 1, 12.00)], "unreadable": "store address"},
]
for r in receipts:
    p = checks(r)
    print(f"{r['merchant']:12} {'auto-accept' if not p else 'human review: ' + '; '.join(p)}")

Corner Cafe  auto-accept
Hardware Co  human review: items 29.8 + tax 2.4 != total 32.4
Book Nook    human review: bad date '2026-02-30'; model flagged unreadable: store address

The first receipt is accepted. The second fails the sum check, probably a misread unit price, and goes to a person. The third has a date that does not exist (30 February) and a field the model honestly flagged as unreadable. Every human correction is a labelled example: add it to your eval set ( Building an eval set ) and measure field-level accuracy per document type.

Documents and PDFs

A PDF with a real text layer gives you exact characters for free; a VLM adds the layout and the figures. That is why some APIs send each page as both its extracted text and an image. Scanned PDFs have no text layer, so everything rides on the image. For long documents, the page count drives cost: each page is an image plus its text.

Classic OCR or a VLM?

A dedicated OCR engine returns characters with positions and confidence scores, is cheap, and is very good on clean scans. It knows nothing about meaning. A VLM understands layout and context but costs more per page and can invent text. Many production systems combine them: OCR for exact characters, a language model to map them to fields, and the VLM for messy photos, charts and handwriting.

Trade-offs

Common mistakes

• Sending a 12-megapixel photo and assuming the model sees every pixel. It sees a downscaled copy; resize or crop yourself so you know what it gets.
• Asking a vague question ("what is this?") when you need specific fields in a specific format.
• Not giving the model a way to say "unreadable", then being surprised by confident wrong digits.
• Trusting counts, coordinates or values read off a chart without verification.
• Measuring accuracy on a few clean sample images instead of a labelled set of the messy photos users actually send.

In the wild

Check yourself

Recap