Machine Learning ML-04.07
Multi-class vs multi-label classification
Multi-class gives each example exactly one class; multi-label can give it several at once. Definitions, examples, softmax vs sigmoid, loss and metrics.
The big idea
Most real classification problems have more than two answers. A support ticket goes to one of eight teams; a handwritten digit is one of ten; a photo of a plant is one of hundreds of species. These are multi-classClassification where each example belongs to exactly one of three or more classes. problems: pick exactly one.
Other problems allow several answers at once. A news article can be about politics and economics; a photo can contain a dog, a cat and a beach. These are multi-labelClassification where each example can have any number of labels at the same time, including none. problems: say yes or no to each label independently.
The difference is like a single-choice exam question versus a "tick all that apply" question. It decides the output layer, the loss, and how you measure success.
| multi-class | multi-label | |
|---|---|---|
| Labels per example | exactly one of three or more classes | any number, including none |
| Example | a handwritten digit is one of 0 to 9 | a photo is tagged "dog" and "beach" |
| Output layer | one softmax over all classes | one sigmoid per label |
| Probabilities | sum to 1 across classes | each label between 0 and 1 on its own |
| Loss | categorical cross-entropy | binary cross-entropy per label |
| Prediction | the class with the highest probability | every label above its threshold |
How it works
Step 1: One-vs-rest, the simple adapter
Any binary classifier can handle many classes with one-vs-restTraining one binary classifier per class (this class against all others) and predicting the class whose classifier is most confident. (OvR): train one model per class, "cat vs not cat", "dog vs not dog", and so on, then predict the class whose model gives the highest score. It is easy and works with anything, but each model is trained separately, so their scores are not on a shared scale and the probabilities do not add up to 1.
Step 2: Softmax, classes that compete
The cleaner approach computes one score per class, called a logitThe raw, unnormalised score a model produces for a class before it is turned into a probability., then turns all scores into probabilities together with the softmaxThe function exp(z_k) / sum_j exp(z_j). It turns a vector of scores into probabilities that are positive and sum to 1.:
Exponentiating makes every score positive and exaggerates the gaps; dividing by the sum makes the probabilities add to 1. Worked example with logits 2.0, 1.0 and 0.1: e^2.0 = 7.39, e^1.0 = 2.72, e^0.1 = 1.11, total 11.21. So the probabilities are 7.39 / 11.21 = 0.659, 0.242 and 0.099. Raising one class's score necessarily lowers the others: the classes compete for a fixed budget of probability.
Training uses categorical cross-entropyThe loss -log p of the probability the model gave to the true class, averaged over examples. The multi-class version of binary cross-entropy.: the loss is -log of the probability assigned to the true class. Logistic regression with softmax is called multinomial logistic regression, and a softmax layer is the standard last layer of neural network classifiers ( The perceptron and the artificial neuron ). With two classes, softmax reduces exactly to the sigmoid.
Step 3: Multi-label, a sigmoid per label
For multi-label problems, the labels should not compete: an image being "outdoors" says nothing about whether it also contains a dog. So give each label its own sigmoid and its own binary cross-entropy, and threshold each one separately. The probabilities can sum to anything from 0 to the number of labels.
Step 4: Averaging metrics across classes
Precision, recall and F1 are defined for one class at a time. With several classes you compute them per class and then average, and the way you average changes the story:
- Macro averagingComputing a metric for each class separately and taking the plain average, so every class counts equally regardless of size.: average the per-class scores. Every class counts the same, so a rare class you never predict drags the score down hard.
- Micro averagingPooling all true positives, false positives and false negatives across classes before computing the metric, so frequent classes dominate.: pool all decisions first, then compute. Big classes dominate. For single-label multi-class problems, micro F1 equals accuracy.
- Weighted averaging: per-class scores weighted by class size. Sits in between.
Use macro averaging when every class matters equally, especially rare ones; use micro or weighted when overall throughput matters. Always look at the per-class numbers too.
Try it
Try this:
- Raise the dog score until it passes cat. On the left, cat's probability falls as dog's rises; on the right, cat's sigmoid does not move at all.
- Set all three scores to 3. Softmax says one third each: it cannot express "all three are present". The sigmoids say about 0.95 each: all three tags on.
- Lower the temperature to 0.25, then raise it to 4. Low temperature sharpens softmax toward the top class; high temperature flattens it toward uniform. The same knob controls randomness when language models sample text ( Decoding and sampling ).
Going deeper
Code: softmax, one-vs-rest, averaging and multi-label
import numpy as np
from sklearn.datasets import load_iris, make_classification, make_multilabel_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import f1_score
# Softmax by hand
z = np.array([2.0, 1.0, 0.1])
p = np.exp(z) / np.exp(z).sum()
print("logits", z, "-> softmax", p.round(3), "sum", p.sum().round(3))
# Iris: 3 flower species, softmax (multinomial) vs one-vs-rest
X, y = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
soft = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
ovr = OneVsRestClassifier(LogisticRegression(max_iter=1000)).fit(Xtr, ytr)
print(f"\niris softmax acc {soft.score(Xte, yte):.3f} one-vs-rest acc {ovr.score(Xte, yte):.3f}")
print("one flower, softmax probs:", soft.predict_proba(Xte[:1]).round(3)[0])
# Macro vs micro on imbalanced classes: 80 / 15 / 5 percent
X, y = make_classification(n_samples=6000, n_features=10, n_informative=5, n_classes=3,
weights=[0.8, 0.15, 0.05], class_sep=0.7, random_state=1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
pred = LogisticRegression(max_iter=1000).fit(Xtr, ytr).predict(Xte)
print("\nper-class F1:", f1_score(yte, pred, average=None).round(3))
print(f"micro F1 {f1_score(yte, pred, average='micro'):.3f} "
f"macro F1 {f1_score(yte, pred, average='macro'):.3f} "
f"weighted F1 {f1_score(yte, pred, average='weighted'):.3f}")
# Multi-label: each example can have several of 4 tags; one sigmoid per tag
X, Y = make_multilabel_classification(n_samples=2000, n_features=20, n_classes=4, random_state=0)
Xtr, Xte, Ytr, Yte = train_test_split(X, Y, test_size=0.5, random_state=0)
ml = OneVsRestClassifier(LogisticRegression(max_iter=2000)).fit(Xtr, Ytr)
P = ml.predict_proba(Xte[:1])[0]
print("\nmulti-label example: true tags", Yte[0], "probs", P.round(2), "sum", P.sum().round(2))
print(f"tags per example on average: {Y.sum(axis=1).mean():.2f}")
print(f"micro F1 {f1_score(Yte, ml.predict(Xte), average='micro'):.3f} "
f"exact match (all 4 tags right) {(ml.predict(Xte) == Yte).all(axis=1).mean():.3f}")Output
logits [2. 1. 0.1] -> softmax [0.659 0.242 0.099] sum 1.0
iris softmax acc 0.983 one-vs-rest acc 0.933
one flower, softmax probs: [0.002 0.268 0.73 ]
per-class F1: [0.892 0.317 0. ]
micro F1 0.807 macro F1 0.403 weighted F1 0.759
multi-label example: true tags [0 0 1 0] probs [0.16 0.29 1. 0.25] sum 1.71
tags per example on average: 1.86
micro F1 0.784 exact match (all 4 tags right) 0.434 On the iris flowers, softmax logistic regression scores 0.983 and one-vs-rest 0.933 on the same split. In the multi-label part, one example's tag probabilities are 0.16, 0.29, 1.00 and 0.25: they sum to 1.71, which is fine because each is a separate yes or no. Exact-match accuracy (all four tags right) is only 0.434 while micro F1 is 0.784: for multi-label problems, insist on per-label metrics rather than all-or-nothing accuracy.
Many classes and hierarchies
With thousands of classes (products, species, intents), the softmax over all of them becomes expensive and many classes have few examples. Common answers are hierarchical classification (first department, then product type), retrieval-style models that embed inputs and classes in the same space and pick the nearest class, and top-k accuracy as the metric ("is the right answer among the five suggestions?").
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Softmax (multinomial) | Exactly one correct class; you want probabilities that sum to 1 | Examples can belong to several classes or none |
| One-vs-rest | Your favourite model is binary-only; classes are added over time | You need comparable, calibrated scores across classes |
| Sigmoid per label | Tagging: any number of labels per example | Labels are mutually exclusive; it can output contradictory sets |
| Macro averaging | Rare classes matter as much as common ones | Tiny classes make the score noisy |
| Micro averaging | Overall volume of correct decisions matters | It hides failure on rare classes |
Common mistakes
- Using softmax for a tagging problem, which forces labels to compete and caps the total at 1.
- Using independent sigmoids for exclusive classes, then getting two or zero classes predicted.
- Reporting only micro F1 or accuracy when a rare class matters.
- Using a single threshold of 0.5 for every label in multi-label problems; tune each label's threshold on validation data.
- Stratifying multi-class splits but forgetting that rare classes may have only a handful of test examples.
In the wild
Image classifiers trained on large benchmarks end in a softmax over hundreds or thousands of classes and are scored on top-1 and top-5 accuracy. Content platforms tag posts with many topics at once using one sigmoid per tag. Support systems route tickets with a multi-class model and fall back to a human when the top softmax probability is low. Medical image models often predict several findings per scan, a multi-label problem evaluated per finding.
Check yourself
-
Logits are 1.0, 1.0 and 1.0. What does softmax output?
Show answer
B. 0.33, 0.33, 0.33 Equal scores give equal shares of a probability budget that sums to 1. Independent sigmoids would give 0.73 each instead.
-
A photo-tagging model must output any of 50 tags. Which output layer?
Show answer
B. 50 independent sigmoids with binary cross-entropy Tags are not mutually exclusive. Each needs its own yes or no probability, so a sigmoid per tag.
-
Micro F1 is 0.95 but macro F1 is 0.55 on a 10-class problem. What is the most likely explanation?
Show answer
B. The model does well on frequent classes and poorly on some rare ones Micro averaging is dominated by frequent classes; macro gives every class equal weight. A big gap points to rare classes being neglected.
-
For single-label multi-class classification, micro-averaged F1 equals what?
Show answer
B. Accuracy Every wrong prediction is one false positive for the predicted class and one false negative for the true class, so pooled precision, pooled recall and accuracy coincide.
Recap
- Multi-class means exactly one class per example; multi-label means any number of labels.
- One-vs-rest adapts binary models; softmax turns all class scores into probabilities that sum to 1.
- Multi-label problems use an independent sigmoid and binary cross-entropy per label, each with its own threshold.
- Macro averaging counts every class equally; micro averaging is dominated by frequent classes.
- Always inspect per-class metrics, especially for rare classes.
Next up
- From notebook to production learning path