Image segmentation

The big idea

A detector draws a rectangle around a pedestrian. But a rectangle around a person also contains road, sky and parts of a parked car. A self-driving car planning a path, a radiologist measuring a tumour or a photo app blurring the background needs the exact outline. Image segmentationAssigning a label to every pixel of an image, so the output is a mask with the same height and width as the input. gives it: the output is a label for every single pixel.

Think of a colouring book. Detection is drawing a box around each animal on the page. Segmentation is colouring inside the lines: every pixel of the giraffe gets giraffe colour, every pixel of grass gets grass colour. And there are two ways to colour: all giraffes the same yellow, or each giraffe its own shade so you can count them. That choice is exactly the difference between the main kinds of segmentation.

How it works

Step 1: Three kinds of segmentation

• Semantic segmentationSegmentation that assigns a class label to every pixel without distinguishing separate objects of the same class. assigns a class to every pixel: sky, road, car, person. Two cars side by side become one region of "car" pixels; the output cannot tell you there are two.
• Instance segmentationSegmentation that detects each object and gives it its own mask and id, so objects of the same class are kept apart. finds each object and gives it its own mask: car 1, car 2, person 1. It usually only covers countable objects and leaves the background unlabelled. Mask R-CNN is the classic design: it extends Faster R-CNN ( Object detection ) with a branch that predicts an object mask in parallel with the existing branch for box recognition.
• Panoptic segmentationSegmentation that gives every pixel both a class and, for countable objects, an instance id, unifying semantic and instance segmentation., named in a 2018 paper, unifies the two: every pixel gets a class, and pixels of countable objects also get an instance id. The paper calls amorphous regions such as sky and road "stuff" and countable objects "things", and proposes a single metric, panoptic quality (PQ), for both.

Step 2: Dense prediction with an encoder-decoder

A classification CNN ( Pooling and CNN architecture ) shrinks the image step by step with pooling and stride until it is a single vector. That throws away exactly what segmentation needs: where things are. The fix is to turn the network into a function from an image to an image. A fully convolutional networkA convolutional network with no fully connected layers, so it accepts inputs of any size and outputs a spatial map of predictions. (FCN) replaces the dense layers with convolutions; the 2014 FCN paper describes networks that take input of arbitrary size and produce correspondingly-sized output, and combines a deep, coarse layer with a shallow, fine one to get detailed masks.

The general shape is an encoder-decoderA network that first compresses the input into a small, deep representation (encoder) and then expands it back to full resolution (decoder).. The encoder is a normal CNN backbone: resolution goes down, channels go up, and features become more semantic ("this region is a car"). The decoder upsamples back to full resolution, and a final 1 by 1 convolution turns each pixel's feature vector into class scores. Training is ordinary cross-entropy ( Multi-class and multi-label classification ), applied at every pixel and averaged.

Step 3: U-Net and skip connections

The decoder has a problem: after four rounds of downsampling, the fine detail of boundaries is gone, and upsampling cannot invent it. U-NetAn encoder-decoder segmentation network whose decoder concatenates the matching encoder feature maps through skip connections, giving a U-shaped diagram. (2015) solves this by copying each encoder feature map across to the decoder stage of the same resolution.

The U-Net paper describes each encoder step as two 3 by 3 convolutions with ReLU followed by 2 by 2 max pooling with stride 2, doubling the number of feature channels at each downsampling. Each decoder step upsamples, applies a 2 by 2 "up-convolution" that halves the channels, concatenates the corresponding encoder feature map, and applies two more 3 by 3 convolutions. A final 1 by 1 convolution maps each 64-channel feature vector to the classes; the whole network has 23 convolutional layers. The original used unpadded convolutions, so encoder maps had to be cropped before concatenation; most modern implementations pad instead ( Convolutions ) so the sizes match.

Why it matters: U-Net was built for biomedical images, where labelled data is scarce. The paper reports training end to end from very few images, relying on heavy data augmentation with elastic deformations ( Data augmentation ), and segmenting a 512 by 512 image in under a second on a GPU. The U shape became the default for any image-to-image task; the original denoising diffusion model (DDPM) uses a U-Net as its denoising network, for example.

Step 4: Segment Anything: promptable segmentation

Classic segmentation models know a fixed list of classes. Segment Anything (2023) changed the question from "label every pixel with one of these classes" to "give me the mask of whatever I point at". The promptable segmentationA model that returns a segmentation mask for any object indicated by a prompt such as a click, a box or a rough mask, without being limited to a fixed class list. model, SAM, has three parts:

1. A heavy image encoder that turns the image into an embedding, run once per image.
2. A prompt encoder for points, boxes or masks (the paper also shows early results with free-form text).
3. A lightweight mask decoder that combines the two into masks.

Because the expensive embedding is reused, each new click costs only the decoder: the paper reports about 50 ms per prompt in a web browser, fast enough for interactive editing. A single click can be ambiguous (the shirt or the whole person?), so SAM predicts several candidate masks per prompt. It was trained on SA-1B, over 1 billion masks on 11 million images, built with a "data engine" in which the model helped annotators and then generated masks itself. The authors report zero-shot transfer to new image distributions and tasks that is often competitive with fully supervised models. SAM masks have no class names; pair it with a detector or a classifier when you need labels.

Going deeper

Code: masks, instances and metrics

import numpy as np
from scipy import ndimage

# A 7 x 10 scene. Classes: 0 sky, 1 road, 2 car, 3 person. Two cars touch each other.
sem = np.zeros((7, 10), int)
sem[3:] = 1                     # road
sem[3:5, 1:4] = 2               # car A
sem[3:5, 4:7] = 2               # car B, right next to A
sem[2:5, 8] = 3                 # person
print("semantic mask (one class id per pixel):\n", sem)

# Instances from the semantic mask alone: connected components of the car class
cars, n = ndimage.label(sem == 2)
print("connected car blobs found:", n, "(there are really 2 cars)")
inst = np.zeros_like(sem)
inst[3:5, 1:4], inst[3:5, 4:7], inst[2:5, 8] = 1, 2, 3   # true instance ids
print("instance ids:\n", inst)

def scores(pred, truth, k=4):
    acc = (pred == truth).mean()
    ious = []
    for c in range(k):
        inter = ((pred == c) & (truth == c)).sum()
        union = ((pred == c) | (truth == c)).sum()
        ious.append(inter / union)
    return acc, np.array(ious)

lazy = np.where(sem == 3, 1, sem)              # a model that never predicts "person"
for name, pred in [("perfect", sem), ("never person", lazy)]:
    acc, ious = scores(pred, sem)
    print(f"{name:13} pixel accuracy {acc:.3f}  IoU per class {np.round(ious, 2)}  mIoU {ious.mean():.3f}")

# Dice for one binary mask: 2|A n B| / (|A| + |B|)
a = sem == 2                                   # true car pixels (12)
b = np.zeros_like(a); b[3:5, 1:6] = True       # predicted car mask misses the last column
dice = 2 * (a & b).sum() / (a.sum() + b.sum())
iou = (a & b).sum() / (a | b).sum()
print(f"car mask: Dice {dice:.3f}  IoU {iou:.3f}")

semantic mask (one class id per pixel):
 [[0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 3 0]
 [1 2 2 2 2 2 2 1 3 1]
 [1 2 2 2 2 2 2 1 3 1]
 [1 1 1 1 1 1 1 1 1 1]
 [1 1 1 1 1 1 1 1 1 1]]
connected car blobs found: 1 (there are really 2 cars)
instance ids:
 [[0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 3 0]
 [0 1 1 1 2 2 2 0 3 0]
 [0 1 1 1 2 2 2 0 3 0]
 [0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 0]]
perfect       pixel accuracy 1.000  IoU per class [1. 1. 1. 1.]  mIoU 1.000
never person  pixel accuracy 0.957  IoU per class [1.  0.9 1.  0. ]  mIoU 0.724
car mask: Dice 0.909  IoU 0.833

The semantic mask is Figure 1 as numbers. Running connected-component labelling on the car pixels finds one blob, not two: touching objects of one class cannot be separated after the fact, which is why instance segmentation needs its own model. The metric lines show a classic trap. A model that never predicts "person" still gets 0.957 pixel accuracy, because people are 3 of 70 pixels; its mean IoU is 0.724, with person IoU 0.

Metrics

• Pixel accuracy is the share of correct pixels. Easy to read, but dominated by large classes such as road and sky.
• IoU per class, then averaged over classes, is mIoUMean intersection over union: IoU between predicted and true masks computed per class, then averaged over classes; the standard semantic segmentation metric., the standard semantic segmentation score. Each class counts equally regardless of its size, so a missed small class hurts.
• Dice, 2|A n B| / (|A| + |B|), is common in medical imaging and as a training loss for small structures. For the same masks it is always at least as large as IoU: the code's car mask scores Dice 0.909 and IoU 0.833.
• Instance segmentation uses mask AP, the detection AP of Object detection with mask IoU in place of box IoU; panoptic segmentation uses PQ.

Class imbalance and boundaries

Most pixels are easy background, and most errors sit on thin boundaries. Weighted cross-entropy, Dice loss or a mix of both counter the imbalance ( Imbalanced data ). Masks are also expensive to label by hand, which is exactly why interactive tools like SAM matter for building datasets.

Trade-offs

Common mistakes

• Reporting pixel accuracy on imbalanced scenes. Report per-class IoU and mIoU so small classes are visible.
• Resizing masks with bilinear interpolation, which invents fractional class ids. Use nearest-neighbour for label masks.
• Augmenting the image but not its mask, or flipping one and not the other.
• Expecting semantic segmentation to count objects. Touching objects of one class merge; use instance or panoptic models.
• Downsampling too aggressively with no skip connections, then getting blobby masks with soft, misplaced boundaries.

In the wild

Segmentation runs wherever exact shape matters: drivable-area and lane masks in driver assistance, organ and tumour outlines in medical imaging (the original home of U-Net), portrait mode and background replacement in video calls, crop and field boundaries in satellite imagery, and defect outlines in factory inspection. Promptable models now sit inside image editors and labelling tools, where a click selects an object and an annotator only corrects the edges.

Check yourself

Recap