Object detection

The big idea

A classifier answers "is there a cat in this photo?". A self-driving car needs more: every pedestrian, cyclist and car, and exactly where each one is. Object detectionThe task of finding every object of interest in an image and returning, for each, a class label, a bounding box and a confidence score. outputs a list of objects, each with a class, a rectangle around it and a confidence score. The number of objects is not known in advance: zero, one or fifty.

Think of a stocktaker with a clipboard walking past a shelf. For every item they see, they write down what it is and which shelf slot it occupies. If two colleagues both count the same tin of soup, the duplicate has to be crossed out at the end. A detector works the same way: it proposes many candidate boxes, scores each, and then removes the duplicates. Three questions shape this lesson: how do you compare two boxes, how does a network propose boxes, and how do you score the whole list?

How it works

Step 1: Bounding boxes

A bounding boxThe smallest axis-aligned rectangle that encloses an object, stored as four numbers. is an axis-aligned rectangle stored as four numbers. Two formats are common: corners (x0, y0, x1, y1), or centre and size (cx, cy, w, h). COCO annotations use the top-left corner plus width and height; the original YOLO predicts the centre relative to a grid cell and the width and height relative to the whole image. Mixing formats up is the most common bug in detection code, so convert explicitly at every boundary.

Each detection is then a tuple: class, box and a confidence score. A detector usually outputs hundreds or thousands of raw candidates per image; most are later thrown away by a score threshold and by non-max suppression (Step 4).

Step 2: Intersection over union

To compare a predicted box with a ground-truth box you need one number for "how well do they overlap". IoUIntersection over union: the area where two boxes overlap divided by the area they cover together. 1 means identical boxes, 0 means no overlap., intersection over union, divides the overlap area by the combined area:

Worked example: a ground-truth box 3 wide and 4 tall (area 12) and a prediction of the same size shifted right by 1. The overlap is 2 by 4 = 8. The union is 12 + 12 - 8 = 16. IoU = 8 / 16 = 0.5. Being off by a third of the width already halves the score, which is why IoU is a strict measure. It is also scale-free: a 10-pixel error matters a lot on a small box and little on a big one.

IoU does three jobs in a detector. During training it decides which predictions count as matching an object. In Step 4 it decides which boxes are duplicates. At evaluation it decides whether a detection is a hit: a true positiveA detection whose IoU with an unmatched ground-truth box of the same class reaches the evaluation threshold, such as 0.5. needs IoU above a threshold, traditionally 0.5.

Step 3: Proposing boxes: anchors or points

A detector runs a CNN backbone ( Landmark CNNs: LeNet to ResNet ) to get a feature map, then a small head that, at every feature-map location, predicts class scores and a box. The question is what the box is predicted relative to.

Anchor-based. At every location, the head starts from a fixed set of anchor boxesPreset reference boxes of several sizes and aspect ratios placed at every feature-map location; the detector predicts offsets from them instead of raw coordinates. and predicts, for each anchor, an objectness or class score and four offsets that shift and stretch it. Faster R-CNN introduced the idea with 3 scales (box areas of 128 squared, 256 squared and 512 squared pixels) and 3 aspect ratios (1:1, 1:2 and 2:1), so k = 9 anchors at each sliding position. During training, an anchor whose IoU with a ground-truth box is above 0.7 is labelled positive and one below 0.3 for every object is labelled negative. Offsets are usually parameterized relative to the anchor, for example the centre shift divided by anchor width and the log of the width ratio, so one small number fixes a box of any size.

Anchor-free. Anchors add hyperparameters (how many, which sizes, which ratios) that must suit the dataset. FCOS drops them: every feature-map location inside an object predicts the four distances from itself to the box's left, top, right and bottom sides, written l, t, r and b, plus a class score. Its authors describe it as anchor box free as well as proposal free, avoiding the anchor IoU computations during training and all the hyperparameters related to anchor boxes. A related design predicts each object as its centre point plus a width and a height.

One stage or two. Faster R-CNN is a two-stage detectorA detector that first proposes candidate regions (with a region proposal network) and then classifies and refines each region in a second stage.: a region proposal network suggests regions that might hold an object, then a second network classifies and refines each one. One-stage detectors such as YOLO predict final classes and boxes directly from the feature map in a single pass, which is simpler and faster.

Step 4: Non-maximum suppression

A good detector fires several times on the same object: neighbouring cells and anchors all see it. Non-maximum suppressionA post-processing step that keeps the highest-scoring box, removes every remaining box that overlaps it by more than an IoU threshold, and repeats with the next survivor. (NMS) cleans this up greedily:

1. Sort the boxes by score.
2. Keep the highest-scoring box.
3. Delete every remaining box whose IoU with it is above a threshold, often 0.5.
4. Repeat with the highest-scoring survivor until none are left.

It is normally run per class, so a person box never suppresses an overlapping bicycle box.

The threshold is a trade-off. Too low and two genuinely overlapping objects, such as people in a crowd, get merged into one. Too high and duplicates survive and count as false positives.

Step 5: YOLO and the one-stage family

YOLO (You Only Look Once, 2015) frames detection as a single regression problem from pixels to boxes and class probabilities. It divides the image into an S by S grid; each cell predicts B boxes, each with x, y, w, h and a confidence, plus C class probabilities. For PASCAL VOC the paper uses S = 7, B = 2 and C = 20, so the whole output is one 7 by 7 by 30 tensor, produced in one forward pass. The base model ran at 45 frames per second and Fast YOLO at 155. The paper notes that NMS was less critical for it than for earlier detectors but still added 2 to 3 points of mAP, and that it struggled with small objects in groups, such as flocks of birds, because each cell predicts only a few boxes.

The family kept evolving. YOLOv2 removed the fully connected layers and switched to anchor boxes, choosing the anchor shapes by running k-means on the training boxes with 1 - IoU as the distance ( k-means clustering ). Later versions moved back to anchor-free heads: YOLOX switched the YOLO detector to an anchor-free design in 2021, and Ultralytics describes YOLOv8 (released January 2023) as using an anchor-free split head. The constant is the one-stage idea: a single network pass, then NMS, fast enough for video.

Step 6: Scoring a detector with mAP

You cannot score a detector with accuracy, because there is no fixed number of answers. Instead, sort all detections of one class across the test set by confidence and walk down the list. Each detection is a true positive if it matches a not-yet-matched ground-truth box with IoU above the threshold, otherwise a false positive (a duplicate of an already matched object also counts as false). After each step compute precision and recall ( Classification metrics ). Average precisionAverage precision: the area under a class's precision-recall curve, computed from detections ranked by confidence. (AP) is the area under the resulting precision-recall curve. mAPMean average precision: AP averaged over all classes, and in COCO also over IoU thresholds from 0.50 to 0.95. averages AP over classes.

The IoU threshold matters. PASCAL VOC traditionally uses a single IoU of 0.50. COCO's primary metric averages AP over 10 IoU thresholds, 0.50, 0.55 and so on up to 0.95, and over all 80 categories; the COCO site notes that this rewards detectors with better localization and that it makes no distinction between AP and mAP. So "AP 50" and "COCO AP" for the same model can differ a lot, and you must always say which one you report.

Try it

Try this:

1. Choose "shifted by a third of the width". Check the panel: 8 / 16 = 0.5, exactly at the threshold. Now raise the threshold to 0.75, the stricter COCO-style setting. The same box becomes a false positive.
2. Choose "twice as tall". The prediction contains the whole object, yet IoU is only 0.5. Why does IoU punish boxes that are too big as hard as boxes that are misplaced?
3. Choose "small box inside a big one", then drag the violet corner handle to grow the prediction. How large must it get before it counts as a hit at 0.5?

Going deeper

Code: IoU, NMS and AP from scratch

import numpy as np

def iou(a, b):                       # boxes as [x0, y0, x1, y1]
    w = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = w * h
    union = (a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - inter
    return inter / union

def nms(boxes, scores, thr=0.5):     # greedy non-maximum suppression
    order, keep = list(np.argsort(scores)[::-1]), []
    while order:
        best = order.pop(0); keep.append(best)
        order = [i for i in order if iou(boxes[best], boxes[i]) < thr]
    return keep

# One image, two cats. The detector fires five times.
truth = [[10, 10, 50, 50], [60, 20, 90, 60]]
boxes = [[12, 8, 52, 48], [8, 12, 48, 52], [62, 22, 92, 58], [15, 15, 55, 55], [58, 18, 86, 64]]
scores = np.array([0.92, 0.90, 0.81, 0.40, 0.66])
print("IoU of each box with cat 1:", [round(iou(b, truth[0]), 2) for b in boxes])
keep = nms(boxes, scores)
print("NMS keeps boxes", [int(i) for i in keep], "with scores", scores[keep])

def average_precision(dets, truths, thr):
    """dets: (image, score, box); truths: {image: [boxes]}. All-point interpolated AP."""
    dets = sorted(dets, key=lambda d: -d[1]); used = set(); tp = []
    for img, _, box in dets:
        ious = [iou(box, t) for t in truths[img]]
        j = int(np.argmax(ious)) if ious else -1
        ok = j >= 0 and ious[j] >= thr and (img, j) not in used
        if ok: used.add((img, j))
        tp.append(ok)
    tp = np.array(tp); n_true = sum(len(v) for v in truths.values())
    recall = np.cumsum(tp) / n_true
    precision = np.cumsum(tp) / np.arange(1, len(tp) + 1)
    p_interp = np.maximum.accumulate(precision[::-1])[::-1]   # best precision at >= this recall
    return np.sum(np.diff(np.concatenate([[0], recall])) * p_interp)

# Three images: image 1 has one dog, image 2 has a bird the detector never finds.
truths = {0: truth, 1: [[20, 30, 70, 80]], 2: [[40, 40, 60, 60]]}
others = [(1, 0.88, [25, 30, 72, 84]), (1, 0.70, [100, 100, 120, 120]), (2, 0.30, [0, 0, 15, 15])]
raw = [(0, s, b) for s, b in zip(scores, boxes)] + others
after = [(0, scores[i], boxes[i]) for i in keep] + others
print(f"AP at IoU 0.50 without NMS: {average_precision(raw, truths, 0.5):.3f}")
for t in (0.5, 0.75, 0.9):
    print(f"AP at IoU {t:.2f} with NMS:    {average_precision(after, truths, t):.3f}")
coco = np.mean([average_precision(after, truths, t) for t in np.arange(0.5, 0.96, 0.05)])
print(f"COCO-style AP, mean over IoU 0.50:0.05:0.95: {coco:.3f}")

IoU of each box with cat 1: [0.82, 0.82, 0.0, 0.62, 0.0]
NMS keeps boxes [0, 2] with scores [0.92 0.81]
AP at IoU 0.50 without NMS: 0.625
AP at IoU 0.50 with NMS:    0.750
AP at IoU 0.75 with NMS:    0.750
AP at IoU 0.90 with NMS:    0.000
COCO-style AP, mean over IoU 0.50:0.05:0.95: 0.500

The first lines reproduce Figure 2: boxes 0.90 and 0.40 overlap the 0.92 box with IoU 0.68 and 0.62 and are suppressed, and 0.66 loses to 0.81. Then the code scores detections on three images with four objects. Without NMS, the duplicate 0.90 box is ranked second and counts as a false positive, so AP at IoU 0.50 is 0.625; with NMS it rises to 0.750. The 0.25 still missing is the bird no detection found: recall never passes 3 of 4. The kept boxes overlap their objects with IoU around 0.8, so AP stays 0.750 at threshold 0.75 but drops to 0 at 0.90, and the COCO-style average over 0.50 to 0.95 is 0.500.

Training a detector

The loss has two parts. A classification loss (cross-entropy or a variant) on each candidate's class, including a "background" outcome for candidates that match nothing; and a box regression loss on the offsets, applied only to candidates matched to an object. Matching is the subtle part: anchor-based detectors match by IoU thresholds, as in Faster R-CNN's 0.7 and 0.3, and anchor-free ones by whether the location falls inside the object. Because most candidates are background, the classes are heavily imbalanced ( Classification metrics ), and detectors either sample a balanced subset or reweight the loss.

Without NMS: detection as set prediction

NMS is a hand-written step outside the network, and its threshold is a guess. DETR (2020) treats detection as direct set prediction: a transformer outputs a fixed number of predictions in parallel, and a loss based on one-to-one bipartite matching with the ground truth forces unique predictions. Its authors write that this removes the need for hand-designed components such as NMS and anchor generation. The trade-off is a heavier, transformer-based model ( Vision transformers covers transformers for vision).

Trade-offs

Common mistakes

• Mixing box formats: treating (x, y, w, h) as corners, or normalized coordinates as pixels. IoU then looks random.
• Running NMS across classes, so a confident "person" box deletes an overlapping "bicycle" box.
• Quoting "mAP" without the IoU threshold. AP at 0.50 and COCO AP over 0.50 to 0.95 are different numbers.
• Forgetting to transform boxes when you flip, crop or resize images for augmentation ( Data augmentation ).
• Setting the confidence threshold before computing mAP. AP needs the full ranked list, including low-scoring detections.

In the wild

Detection is everywhere a system needs to count or locate things: pedestrian and vehicle detection in driver assistance, people counting in shops and stations, product recognition at self-checkouts, face detection in camera apps, document layout analysis that finds tables and figures, and camera traps that spot animals in wildlife surveys. One-stage YOLO-style models are a common choice where speed matters, as on live video; two-stage and transformer detectors are common where accuracy matters more than latency.

Check yourself

Recap