Design a visual search system

The problem

A shopper sees a lamp in a friend's living room, takes a photo and wants to buy it, or something like it. They cannot describe it in words ("brass, curved neck, mushroom shade" is not how people search). A visual search system takes the photo, finds the lamp in it, and returns the most similar products from a catalog of a billion images, in well under a second.

The photo is messy: a sofa, a plant and a window are in the frame, the light is yellow, the lamp is at an angle. The catalog is clean studio shots on white backgrounds, and it changes every day as sellers upload new products. And "similar" is not one thing: the same product, the same style, or the same category at a lower price are all reasonable answers.

Think of a shop assistant shown a phone photo. They do not look at the whole picture; they point at the lamp, form an impression of it ("mid-century, brass, small"), and walk to the aisle where such lamps are rather than checking every item in the store. Visual search does the same three steps: detect and crop the object, embed it, and look it up in an index that only visits the right aisle.

Requirements

Functional

• Given a photo (camera, screenshot or an image already on the platform), return the top 20 to 50 visually similar items, optionally filtered by category, price or availability.
• Detect the objects in the photo and let the user pick one; search for each separately.
• Newly uploaded items must be searchable within hours.
• Log queries, crops, results shown, clicks, saves and purchases.

Non-functional

• Latency: under about 500 ms end to end, of which detection and embedding take maybe 100 to 200 ms on a GPU and the index lookup tens of milliseconds.
• Scale: a billion or more images, thousands of queries per second at peak.
• Quality: relevance measured offline with labelled sets and online with engagement; no near-duplicate spam filling the results.
• Cost: index memory is the main bill, so vectors must be compressed.

Out of scope

• Text search and hybrid ranking ( Design a search ranking system ).
• Training the general-purpose image backbone from scratch.
• Moderation of uploaded images ( Design a spam / content moderation classifier ).

Estimates

Assumed numbers for a large shopping or image platform, not any company's figures:

# Back-of-envelope sizing for a visual search system. Every input is an assumption; change them and re-run.
catalog_images = 1_000_000_000    # product and user images in the index
objects_per_image = 1.5           # detected objects indexed separately (whole image plus crops)
dim = 256                         # embedding size
new_per_day = 5_000_000           # images uploaded per day
queries_per_day = 50_000_000
peak_factor = 3
pq_bytes = 64                     # product-quantized code per vector

vectors = catalog_images * objects_per_image
print(f"vectors in the index        {vectors / 1e9:.1f} billion")
print(f"raw float32 embeddings      {vectors * dim * 4 / 1e12:.2f} TB")
print(f"float16                     {vectors * dim * 2 / 1e12:.2f} TB")
print(f"PQ codes ({pq_bytes} bytes)          {vectors * pq_bytes / 1e9:,.0f} GB, fits in RAM across a few machines")
print(f"binary codes ({dim} bits)      {vectors * dim / 8 / 1e9:,.0f} GB")
peak_qps = queries_per_day / 86_400 * peak_factor
print(f"queries at peak             {peak_qps:,.0f} per second")
print(f"brute force at peak         {peak_qps * vectors * dim * 2 / 1e15:,.1f} PFLOP/s of dot products: why we need ANN")
print(f"new vectors per day         {new_per_day * objects_per_image / 1e6:.1f} million to embed and insert")

vectors in the index        1.5 billion
raw float32 embeddings      1.54 TB
float16                     0.77 TB
PQ codes (64 bytes)          96 GB, fits in RAM across a few machines
binary codes (256 bits)      48 GB
queries at peak             1,736 per second
brute force at peak         1.3 PFLOP/s of dot products: why we need ANN
new vectors per day         7.5 million to embed and insert

• Index size: 1 billion images with 1.5 indexed objects each is 1.5 billion vectors. At 256 float32 dimensions that is 1.54 TB, 0.77 TB in float16, and 96 GB as 64-byte product-quantized codes.
• Brute force is out of the question: 1,736 queries per second at peak against 1.5 billion vectors would need about 1.3 PFLOP/s of dot products. An approximate index scans a tiny fraction.
• Freshness: 5 million uploads a day become 7.5 million new vectors to embed and insert daily.

Data and labels

Two kinds of data, for two jobs:

• Training the embedding needs pairs that should be close: the same product photographed in different settings, a user's photo and the catalog item they then bought, two images saved to the same board or bought together. Hard negatives (same category, different product) teach the fine distinctions.
• Evaluating relevance needs judgments: for a query image and a result, is it the same product, similar, or irrelevant? Human raters are the gold standard but costly; a cheaper proxy is "relevant if the two images share a category or product label".

Pinterest's paper "Visual Discovery at Pinterest" (Zhai et al., WWW 2017) used exactly that proxy for offline comparison: 2,000 query images searched against an indexed evaluation set, a result counted as relevant when the two images shared a class label, scored with precision at 1, 5 and 10, in addition to human evaluation. The proxy is fast and repeatable for comparing models; human relevance labelsGraded judgments by trained raters of whether each result is relevant to a query, used as ground truth for offline evaluation of search and recommendation. catch what it misses (two "lamps" can look nothing alike).

Features: the image embedding

An image embeddingA vector produced by a neural network such that images that look alike, or show the same thing, have nearby vectors. maps an image to a vector where nearby means similar. Options, roughly in historical order:

• A layer of a classification network. Take a pretrained CNN and use an intermediate layer as the embedding. Pinterest compared layers and found intermediate features (pool5, fc6, fc7) retrieved better than the most semantic ones (fc8, fc1000); among raw features ResNet152 pool5 with L2 distance did best.
• Fine-tuned with metric learning: train with a contrastive or triplet loss on same-product pairs, so the space reflects "same product" rather than "same ImageNet class" ( Transfer learning and fine-tuning ).
• Image-text models trained on image-caption pairs, which also allow searching images with text and mixing both.

Compress for scale. Pinterest binarized its features and compared them with Hamming distance; when binarized, VGG16 fc6 performed best among the features it tested and was 16 times smaller than raw ResNet pool5 features. Today product quantization (PQ) is the common choice: split the vector into chunks and store each chunk as the id of its nearest centroid, so 256 floats become 64 bytes.

Model choice and training

The detection front-end

Embedding the whole photo mixes the lamp with the sofa and the window. An object detectorA model that finds objects in an image and returns a bounding box, a class and a confidence score for each one. ( Object detection ) finds candidate objects; the system crops each box, embeds the crop, and searches with it. The detector also tells the system the category, which can restrict the search to the right part of the index.

Detection costs latency, and the choice is a speed-quality trade. Pinterest's paper reports a Faster R-CNN detector at 272 ms against 59 ms for its SSD variant on the same evaluation set; SSD had higher overall precision (0.502 vs 0.426) but lower recall (0.371 vs 0.470) and visibly worse box quality. Pinterest's Flashlight launched first with a manual crop box the user dragged, and added automatic detection later, which simplified the interface to clickable dots on detected objects.

For the catalog side, Pinterest Lens applied object detection to the billions of images on the platform so that a query object could be matched against similar objects inside catalog images, not only against whole images. Indexing objects multiplies the index size (the 1.5 objects per image above) but finds the lamp in a photo of a whole room.

Re-ranking

The nearest neighbours by embedding are candidates. A light re-ranker then combines visual similarity with metadata (category agreement with the detector, text annotations, price, availability, seller quality) and removes near-duplicates. Pinterest's original system did exactly this in two steps: each index shard's leaf ranker retrieved nearest neighbours, then re-ranked the top candidates using additional metadata such as annotations, and a root ranker merged the shards' results.

High-level design

Deep dives

Deep dive 1: Crop first

The sample below simulates 2,000 products in 40 categories (products in one category look alike), 25 catalog images each, as 64-dimensional embeddings. A user's photo mixes the product with a busy background; a detector's crop keeps a little of the background.

# Visual search on synthetic embeddings: cropping the query, an IVF index, and index freshness.
import numpy as np

rng = np.random.default_rng(8)
P, per, d = 2_000, 25, 64                               # products, catalog images per product, embedding size
cat_c = rng.normal(0, 1, (40, d))                       # 40 categories: products in one look alike
centre = cat_c[rng.integers(0, 40, P)] + 0.7 * rng.normal(0, 1, (P, d))
unit = lambda v: v / np.linalg.norm(v, axis=-1, keepdims=True)
items = unit(centre.repeat(per, 0) + 0.8 * rng.normal(0, 1, (P * per, d)))
label = np.arange(P).repeat(per)

def p_at_5(q, truth):                                   # share of the top 5 that show the same product
    top = np.argsort(-(q @ items.T), axis=1)[:, :5]
    return (label[top] == truth[:, None]).mean()

truth = rng.integers(0, P, 500)
scene = rng.normal(0, 1, (500, d))                      # a user's photo: the product plus a busy background
whole = unit(centre[truth] + 2.0 * scene + 0.8 * rng.normal(0, 1, (500, d)))
crop = unit(centre[truth] + 0.6 * scene + 0.8 * rng.normal(0, 1, (500, d)))   # detector box around the object
print(f"precision@5: whole photo {p_at_5(whole, truth):.2f}, detected crop {p_at_5(crop, truth):.2f}")

# IVF: k-means cells, search only the nprobe closest cells.
nlist = 100
cent = items[rng.choice(len(items), nlist, replace=False)]
for _ in range(10):
    cell = np.argmax(items @ cent.T, 1)
    cent = unit(np.array([items[cell == c].mean(0) for c in range(nlist)]))
cell = np.argmax(items @ cent.T, 1)
exact = np.argsort(-(crop @ items.T), axis=1)[:, :10]
for nprobe in (1, 4, 16):
    hits, scanned = 0, 0
    for i, q in enumerate(crop):
        cand = np.flatnonzero(np.isin(cell, np.argsort(-(cent @ q))[:nprobe]))
        found = cand[np.argsort(-(items[cand] @ q))[:10]]
        hits += len(np.intersect1d(found, exact[i])); scanned += len(cand)
    print(f"IVF nprobe {nprobe:2d}: recall@10 {hits / exact.size:.3f}, vectors scanned {scanned / len(crop):,.0f} of {len(items):,}")

# Freshness: 200 new products launch today; users search for them with photos.
new_c = rng.normal(0, 1, (200, d))
new_items = unit(new_c.repeat(5, 0) + 0.6 * rng.normal(0, 1, (1000, d)))
qn = unit(new_c + 0.3 * rng.normal(0, 1, (200, d)) + 0.6 * rng.normal(0, 1, (200, d)))
all_items = np.vstack([items, new_items])
top_after = np.argmax(qn @ all_items.T, 1)
right = (top_after >= len(items)) & ((top_after - len(items)) // 5 == np.arange(200))
print(f"queries for 200 new products: right product at rank 1 before indexing 0%, after {right.mean():.0%}; "
      f"old-index best match similarity {(qn @ items.T).max(1).mean():.2f} vs {(qn @ new_items.T).max(1).mean():.2f}")

precision@5: whole photo 0.37, detected crop 0.93
IVF nprobe  1: recall@10 0.815, vectors scanned 638 of 50,000
IVF nprobe  4: recall@10 0.999, vectors scanned 2,048 of 50,000
IVF nprobe 16: recall@10 1.000, vectors scanned 7,988 of 50,000
queries for 200 new products: right product at rank 1 before indexing 0%, after 100%; old-index best match similarity 0.48 vs 0.75

• Cropping is the biggest single win. With the whole photo, 37 percent of the top 5 results show the right product; with the detected crop, 93 percent. The background dominated the whole-photo embedding.

Deep dive 2: The ANN index

Exact nearest neighbours over a billion vectors is too slow, so the index is approximate ( Vector databases and ANN indexes ). Two families dominate:

• IVF (inverted file): cluster the vectors with k-means into nlist cells; at query time, compare the query with the cell centroids and scan only the nprobe closest cells. Usually combined with PQ codes (IVF-PQ) for memory.
• Graph indexes such as HNSWHierarchical Navigable Small World: an ANN index that links each vector to near neighbours in a layered graph and searches by greedy walks from coarse to fine layers.: very fast and accurate, but memory-hungry and harder to update in place at very large scale.

The sample's second part builds an IVF index with 100 cells over the 50,000 catalog vectors. Searching the closest cell only (nprobe 1) scans 638 vectors and finds 81.5 percent of the true top 10; 4 cells scan 2,048 and find 99.9 percent; 16 cells scan 7,988 and find all of them. Recall rises fast with nprobe, and the cost rises linearly: tune nprobe against latency on real queries. The quantity to report is recall@kThe share of the true k nearest neighbours (from exact search) that an approximate index returns. against exact search, measured on a sample of real queries, separately from relevance.

Other approximations serve the same goal: the PinSage paper notes that approximate nearest neighbours in its embedding space can be found efficiently with locality sensitive hashing.

Deep dive 3: Freshness and model updates

A product uploaded this morning that is not in the index does not exist for visual search. In the sample, 200 new products launch after the index was built: until they are indexed, no query for them can find the right product (the best match in the old index has similarity 0.48, a different product); once they are added, the right product is at rank 1 for 100 percent of those queries.

Two kinds of update:

• New items, same model: embed and insert incrementally. IVF makes this easy: assign the new vector to its nearest centroid and append it to that cell's list. Pinterest's incremental fingerprinting service split images into daily epochs by upload date: each day a new epoch with that day's uploads was added and only the missing features computed, since old images do not change.
• New model: every vector changes, and old and new embeddings are not comparable. Re-embed the whole catalog in a batch job, build a new index next to the old one, switch traffic once it is complete, and keep the old one for rollback. PinSage's MapReduce pipeline computed embeddings for 3 billion items in less than 24 hours, which is what makes regular full refreshes possible.

Cell centroids also drift as the catalog changes; rebuild the IVF clustering periodically, or cells become unbalanced and slow.

Try it

1. With 16 cells and 1 cell searched, the starting query finds 80 percent of its true 10 neighbours while scanning only 59 of 800 vectors. Search 2 cells: 90 percent from 107 vectors; 3 cells: 100 percent from 170. Look at which red rings turn green: they sit just across a cell boundary.
2. Switch to 32 cells with 1 searched: recall drops to 50 percent, scanning only 24 vectors; 2 cells bring it back to 100 percent from 65. More, smaller cells make each probe cheaper but boundaries more frequent.
3. Press "Query a new product" with 16 cells and 1 searched: recall is 90 percent. Now set new uploads to "uploaded, index not updated": 9 of the true 10 neighbours are new uploads that sit in no cell, so recall falls to 10 percent, and searching all 8 cells the slider allows does not help. Choose "added to the index": recall is back to 90 percent.

Going deeper

Evaluation

• Offline relevance: precision at k on a labelled query set, with labels from category or product matches for speed and human raters for the final decision. Measure separately for whole-image queries, crops, and each major category.
• Index quality: recall at k of the ANN index against exact search, and latency at P50 and P99.
• Online: clicks, saves and purchases per search, and the share of searches with any engagement. Pinterest reports that its early Flashlight visual search, powered mostly by embedding retrieval with minimal re-ranking because crops had no metadata, reached two thirds of the engagement rate of its long-developed text search.

Monitoring

Track index lag (newest indexed upload vs newest upload), the share of catalog items missing embeddings, detector confidence distributions per category, ANN recall on a fixed probe set after every rebuild, P99 latency per shard, and the share of results that are near-duplicates.

Bottlenecks and failure modes

Trade-offs we made

Extensions

• Multimodal queries: "like this, but in red" by combining image and text embeddings.
• Shop the look: detect every object in a scene and search for each.
• Personalized re-ranking with user history, as in Design a feed ranking system .
• Graph embeddings that mix visual features with engagement, as PinSage does ( Graph-based recommendations ).

In the wild

Where you will see this: camera search in shopping apps, "similar items" under product photos, reverse image search engines, and duplicate and near-duplicate detection in photo libraries all use the same detect, embed, ANN and re-rank pipeline.

Check yourself

Recap