Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Classification

Machine Learning ML-04.07

Multi-class vs multi-label classification

Multi-class gives each example exactly one class; multi-label can give it several at once. Definitions, examples, softmax vs sigmoid, loss and metrics.

Intermediate 10 min Last reviewed

The big idea

Most real classification problems have more than two answers. A support ticket goes to one of eight teams; a handwritten digit is one of ten; a photo of a plant is one of hundreds of species. These are multi-classClassification where each example belongs to exactly one of three or more classes. problems: pick exactly one.

Other problems allow several answers at once. A news article can be about politics and economics; a photo can contain a dog, a cat and a beach. These are multi-labelClassification where each example can have any number of labels at the same time, including none. problems: say yes or no to each label independently.

The difference is like a single-choice exam question versus a "tick all that apply" question. It decides the output layer, the loss, and how you measure success.

multi-classmulti-label
Labels per exampleexactly one of three or more classesany number, including none
Examplea handwritten digit is one of 0 to 9a photo is tagged "dog" and "beach"
Output layerone softmax over all classesone sigmoid per label
Probabilitiessum to 1 across classeseach label between 0 and 1 on its own
Losscategorical cross-entropybinary cross-entropy per label
Predictionthe class with the highest probabilityevery label above its threshold

How it works

Step 1: One-vs-rest, the simple adapter

Any binary classifier can handle many classes with one-vs-restTraining one binary classifier per class (this class against all others) and predicting the class whose classifier is most confident. (OvR): train one model per class, "cat vs not cat", "dog vs not dog", and so on, then predict the class whose model gives the highest score. It is easy and works with anything, but each model is trained separately, so their scores are not on a shared scale and the probabilities do not add up to 1.

Step 2: Softmax, classes that compete

The cleaner approach computes one score per class, called a logitThe raw, unnormalised score a model produces for a class before it is turned into a probability., then turns all scores into probabilities together with the softmaxThe function exp(z_k) / sum_j exp(z_j). It turns a vector of scores into probabilities that are positive and sum to 1.:

Exponentiating makes every score positive and exaggerates the gaps; dividing by the sum makes the probabilities add to 1. Worked example with logits 2.0, 1.0 and 0.1: e^2.0 = 7.39, e^1.0 = 2.72, e^0.1 = 1.11, total 11.21. So the probabilities are 7.39 / 11.21 = 0.659, 0.242 and 0.099. Raising one class's score necessarily lowers the others: the classes compete for a fixed budget of probability.

Training uses categorical cross-entropyThe loss -log p of the probability the model gave to the true class, averaged over examples. The multi-class version of binary cross-entropy.: the loss is -log of the probability assigned to the true class. Logistic regression with softmax is called multinomial logistic regression, and a softmax layer is the standard last layer of neural network classifiers ( The perceptron and the artificial neuron ). With two classes, softmax reduces exactly to the sigmoid.

Step 3: Multi-label, a sigmoid per label

For multi-label problems, the labels should not compete: an image being "outdoors" says nothing about whether it also contains a dog. So give each label its own sigmoid and its own binary cross-entropy, and threshold each one separately. The probabilities can sum to anything from 0 to the number of labels.

Three output heads for three kinds of problem An input feeds three scores. Top row: softmax turns the scores into probabilities 0.66, 0.24 and 0.10 that sum to 1, and the highest is chosen. Bottom row: the same scores go through separate sigmoids, giving 0.88, 0.73 and 0.52, each compared with its own threshold, so several labels can be on at once. input features scores 2.0, 1.0, 0.1 softmax 0.66, 0.24, 0.10 sum 1: pick cat sigmoid each 0.88, 0.73, 0.52 each above 0.5: all three multi-class: pick one multi-label: tag many Figure 1. The same three scores (2.0, 1.0, 0.1), two heads. Softmax makes the classes share one unit of probability and picks one. Independent sigmoids judge each label alone, so 0.88, 0.73 and 0.52 can all pass a 0.5 threshold.

Step 4: Averaging metrics across classes

Precision, recall and F1 are defined for one class at a time. With several classes you compute them per class and then average, and the way you average changes the story:

  • Macro averagingComputing a metric for each class separately and taking the plain average, so every class counts equally regardless of size.: average the per-class scores. Every class counts the same, so a rare class you never predict drags the score down hard.
  • Micro averagingPooling all true positives, false positives and false negatives across classes before computing the metric, so frequent classes dominate.: pool all decisions first, then compute. Big classes dominate. For single-label multi-class problems, micro F1 equals accuracy.
  • Weighted averaging: per-class scores weighted by class size. Sits in between.
Per-class F1 and three averages Bars for a three-class model on data that is 80, 15 and 5 percent of each class. F1 is 0.892 for class A, 0.317 for class B and 0.000 for class C. Micro F1 is 0.807, weighted F1 is 0.759, and macro F1 is only 0.403. class A (80%) 0.892 class B (15%) 0.317 class C (5%) 0.000 micro F1 0.807 weighted F1 0.759 macro F1 0.403 Figure 2. Numbers from the code below. The model never correctly predicts the rare class C. Micro F1 (0.807) barely notices; macro F1 (0.403) is dragged down by it. Report both.

Use macro averaging when every class matters equally, especially rare ones; use micro or weighted when overall throughput matters. Always look at the per-class numbers too.

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Raise the dog score until it passes cat. On the left, cat's probability falls as dog's rises; on the right, cat's sigmoid does not move at all.
  2. Set all three scores to 3. Softmax says one third each: it cannot express "all three are present". The sigmoids say about 0.95 each: all three tags on.
  3. Lower the temperature to 0.25, then raise it to 4. Low temperature sharpens softmax toward the top class; high temperature flattens it toward uniform. The same knob controls randomness when language models sample text ( Decoding and sampling ).

Going deeper

Code: softmax, one-vs-rest, averaging and multi-label

multiclass.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.datasets import load_iris, make_classification, make_multilabel_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier
from sklearn.metrics import f1_score

# Softmax by hand
z = np.array([2.0, 1.0, 0.1])
p = np.exp(z) / np.exp(z).sum()
print("logits", z, "-> softmax", p.round(3), "sum", p.sum().round(3))

# Iris: 3 flower species, softmax (multinomial) vs one-vs-rest
X, y = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
soft = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
ovr = OneVsRestClassifier(LogisticRegression(max_iter=1000)).fit(Xtr, ytr)
print(f"\niris softmax acc {soft.score(Xte, yte):.3f}   one-vs-rest acc {ovr.score(Xte, yte):.3f}")
print("one flower, softmax probs:", soft.predict_proba(Xte[:1]).round(3)[0])

# Macro vs micro on imbalanced classes: 80 / 15 / 5 percent
X, y = make_classification(n_samples=6000, n_features=10, n_informative=5, n_classes=3,
                           weights=[0.8, 0.15, 0.05], class_sep=0.7, random_state=1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)
pred = LogisticRegression(max_iter=1000).fit(Xtr, ytr).predict(Xte)
print("\nper-class F1:", f1_score(yte, pred, average=None).round(3))
print(f"micro F1 {f1_score(yte, pred, average='micro'):.3f}   "
      f"macro F1 {f1_score(yte, pred, average='macro'):.3f}   "
      f"weighted F1 {f1_score(yte, pred, average='weighted'):.3f}")

# Multi-label: each example can have several of 4 tags; one sigmoid per tag
X, Y = make_multilabel_classification(n_samples=2000, n_features=20, n_classes=4, random_state=0)
Xtr, Xte, Ytr, Yte = train_test_split(X, Y, test_size=0.5, random_state=0)
ml = OneVsRestClassifier(LogisticRegression(max_iter=2000)).fit(Xtr, Ytr)
P = ml.predict_proba(Xte[:1])[0]
print("\nmulti-label example: true tags", Yte[0], "probs", P.round(2), "sum", P.sum().round(2))
print(f"tags per example on average: {Y.sum(axis=1).mean():.2f}")
print(f"micro F1 {f1_score(Yte, ml.predict(Xte), average='micro'):.3f}   "
      f"exact match (all 4 tags right) {(ml.predict(Xte) == Yte).all(axis=1).mean():.3f}")

Output

logits [2.  1.  0.1] -> softmax [0.659 0.242 0.099] sum 1.0

iris softmax acc 0.983   one-vs-rest acc 0.933
one flower, softmax probs: [0.002 0.268 0.73 ]

per-class F1: [0.892 0.317 0.   ]
micro F1 0.807   macro F1 0.403   weighted F1 0.759

multi-label example: true tags [0 0 1 0] probs [0.16 0.29 1.   0.25] sum 1.71
tags per example on average: 1.86
micro F1 0.784   exact match (all 4 tags right) 0.434

On the iris flowers, softmax logistic regression scores 0.983 and one-vs-rest 0.933 on the same split. In the multi-label part, one example's tag probabilities are 0.16, 0.29, 1.00 and 0.25: they sum to 1.71, which is fine because each is a separate yes or no. Exact-match accuracy (all four tags right) is only 0.434 while micro F1 is 0.784: for multi-label problems, insist on per-label metrics rather than all-or-nothing accuracy.

Many classes and hierarchies

With thousands of classes (products, species, intents), the softmax over all of them becomes expensive and many classes have few examples. Common answers are hierarchical classification (first department, then product type), retrieval-style models that embed inputs and classes in the same space and pick the nearest class, and top-k accuracy as the metric ("is the right answer among the five suggestions?").

Trade-offs

OptionWins whenLoses when
Softmax (multinomial) Exactly one correct class; you want probabilities that sum to 1 Examples can belong to several classes or none
One-vs-rest Your favourite model is binary-only; classes are added over time You need comparable, calibrated scores across classes
Sigmoid per label Tagging: any number of labels per example Labels are mutually exclusive; it can output contradictory sets
Macro averaging Rare classes matter as much as common ones Tiny classes make the score noisy
Micro averaging Overall volume of correct decisions matters It hides failure on rare classes

Common mistakes

  • Using softmax for a tagging problem, which forces labels to compete and caps the total at 1.
  • Using independent sigmoids for exclusive classes, then getting two or zero classes predicted.
  • Reporting only micro F1 or accuracy when a rare class matters.
  • Using a single threshold of 0.5 for every label in multi-label problems; tune each label's threshold on validation data.
  • Stratifying multi-class splits but forgetting that rare classes may have only a handful of test examples.

In the wild

Image classifiers trained on large benchmarks end in a softmax over hundreds or thousands of classes and are scored on top-1 and top-5 accuracy. Content platforms tag posts with many topics at once using one sigmoid per tag. Support systems route tickets with a multi-class model and fall back to a human when the top softmax probability is low. Medical image models often predict several findings per scan, a multi-label problem evaluated per finding.

Check yourself

  1. Logits are 1.0, 1.0 and 1.0. What does softmax output?

    Show answer

    B. 0.33, 0.33, 0.33 Equal scores give equal shares of a probability budget that sums to 1. Independent sigmoids would give 0.73 each instead.

  2. A photo-tagging model must output any of 50 tags. Which output layer?

    Show answer

    B. 50 independent sigmoids with binary cross-entropy Tags are not mutually exclusive. Each needs its own yes or no probability, so a sigmoid per tag.

  3. Micro F1 is 0.95 but macro F1 is 0.55 on a 10-class problem. What is the most likely explanation?

    Show answer

    B. The model does well on frequent classes and poorly on some rare ones Micro averaging is dominated by frequent classes; macro gives every class equal weight. A big gap points to rare classes being neglected.

  4. For single-label multi-class classification, micro-averaged F1 equals what?

    Show answer

    B. Accuracy Every wrong prediction is one false positive for the predicted class and one false negative for the true class, so pooled precision, pooled recall and accuracy coincide.

Recap

  • Multi-class means exactly one class per example; multi-label means any number of labels.
  • One-vs-rest adapts binary models; softmax turns all class scores into probabilities that sum to 1.
  • Multi-label problems use an independent sigmoid and binary cross-entropy per label, each with its own threshold.
  • Macro averaging counts every class equally; micro averaging is dominated by frequent classes.
  • Always inspect per-class metrics, especially for rare classes.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up