Machine Learning ML-01.10
Information theory for ML
Measure surprise with entropy, see why cross-entropy is the classification loss, and use KL divergence to compare two distributions.
The big idea
When a classifier predicts "cat 0.9, dog 0.1" and the picture really is a cat, it did well. If it predicted "cat 0.2, dog 0.8", it did badly. But how badly, as a single number we can minimize? Information theory answers that with one idea: surprise. An outcome you thought was likely is not surprising; an outcome you thought was nearly impossible is very surprising. The standard classification loss, cross-entropy, simply measures how surprised the model was by the true answers.
Think of the weather forecast. A forecaster who says "sunny" in the desert is rarely wrong and rarely surprised. A forecaster who says "1 percent chance of rain" and then it pours has been badly surprised, and you should trust them less. Information theory turns that intuition into bits you can count.
How it works
Step 1: Surprise is measured in bits
The information contentThe surprise of an outcome with probability p, measured as minus log base 2 of p, in bits. (or surprise) of an outcome with probability is
A fair coin landing heads () carries 1 bit. A 1-in-8 outcome carries 3 bits. A certain outcome () carries 0 bits: no surprise at all. Using the natural log instead gives units called nats; ML libraries usually use natural logs, which just rescales everything.
Step 2: Entropy, the average surprise
EntropyThe average surprise of a distribution: minus the sum of p times log p over all outcomes. High when outcomes are evenly spread, zero when one outcome is certain. is the expected surprise of a distribution:
For a coin with chance of heads: a fair coin has 1 bit of entropy, the maximum. A 90/10 coin has 0.47 bits. A two-headed coin has 0. Entropy measures uncertainty: how many yes-or-no questions, on average, you need to pin down the outcome.
Step 3: Cross-entropy, the classification loss
Now there are two distributions: the truth and the model's prediction . Cross-entropyThe average surprise when outcomes come from p but you predicted q: minus the sum of p times log q. The standard loss for classification. is the average surprise when the world follows but you expected :
For a labeled example, the truth is one-hot: probability 1 on the correct class, 0 elsewhere. So cross-entropy collapses to : the surprise at the right answer.
- The model gave the right class 0.8: loss bits.
- The model gave it 0.2: loss bits.
- The model gave it 0.001: loss about 10 bits. Confident wrong answers are punished hard.
That steep penalty is exactly what you want from a training signal, and it is why cross-entropy (also called log loss) is the standard loss for classifiers and for language models predicting the next token ( Loss functions ).
Step 4: KL divergence, the extra surprise
KL divergenceKullback-Leibler divergence: cross-entropy minus entropy. The extra surprise from using q when the truth is p. Zero only when q equals p, and not symmetric. measures how much worse is than the best possible guess, which is itself:
It is never negative, and zero only when . It is not symmetric: in general, so it is not a true distance. Since does not depend on the model, minimizing cross-entropy and minimizing KL divergence are the same thing during training.
import numpy as np
def entropy(p):
p = np.asarray(p, dtype=float)
p = p[p > 0]
return float(-(p * np.log2(p)).sum()) + 0.0 # + 0.0 turns -0.0 into 0.0
def cross_entropy(p, q):
return -(np.asarray(p) * np.log2(np.asarray(q))).sum()
print("fair coin H =", round(entropy([0.5, 0.5]), 3), "bits")
print("90/10 coin H =", round(entropy([0.9, 0.1]), 3), "bits")
print("certain coin H =", round(entropy([1.0, 0.0]), 3), "bits")
truth = [0, 1, 0] # the label is class 1 (one-hot)
good = [0.1, 0.8, 0.1] # confident and right
bad = [0.6, 0.2, 0.2] # confident and wrong
print("cross-entropy, good model:", round(cross_entropy(truth, good), 3))
print("cross-entropy, bad model: ", round(cross_entropy(truth, bad), 3))
p, q = np.array([0.9, 0.1]), np.array([0.5, 0.5])
print("KL(p || q) =", round(cross_entropy(p, q) - entropy(p), 3), "bits")Output
fair coin H = 1.0 bits
90/10 coin H = 0.469 bits
certain coin H = 0.0 bits
cross-entropy, good model: 0.322
cross-entropy, bad model: 2.322
KL(p || q) = 0.531 bits Try it
Try this:
- Slide from 0.5 to 0.99 and watch entropy fall: a predictable coin carries less information per flip.
- Set and move the model's belief . Cross-entropy is lowest exactly when , and KL divergence hits zero there.
- Set and . The cross-entropy shoots up: being confidently wrong is the costliest mistake.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Cross-entropy (log loss) | Training classifiers and language models; rewards calibrated probabilities | Labels are very noisy, since confident mistakes on mislabeled data dominate; label smoothing helps |
| Accuracy as a metric | Reporting results in plain terms | As a training loss: it is flat almost everywhere, so it gives no gradient |
| KL divergence | Comparing two models' distributions: distillation, VAEs, keeping a fine-tuned model close to its base in RLHF | You need a symmetric distance; consider Jensen-Shannon divergence instead |
Common mistakes
- Feeding probabilities of exactly 0 into a log, which gives infinity. Libraries clip or work from logits for this reason.
- Applying softmax and then a loss that applies softmax again. In PyTorch,
CrossEntropyLossexpects raw logits. - Assuming KL divergence is symmetric.
- Comparing losses computed with log base 2 and natural log as if they were the same units.
In the wild
Every large language model is pretrained by minimizing cross-entropy on the next token, and its perplexity is just the exponential of that cross-entropy. Knowledge distillation trains a small model to match a large model's output distribution by minimizing KL divergence ( Distillation and synthetic data ). RLHF adds a KL penalty that keeps a fine-tuned model from drifting too far from its original behaviour ( RLHF ). Decision trees choose splits that most reduce entropy ( Decision trees ).
Check yourself
-
What is the entropy of a fair 4-sided die, in bits?
Show answer
B. 2 Four equally likely outcomes: -4 x 0.25 x log2(0.25) = 2 bits. You need two yes-or-no questions to identify the result.
-
The true class is 'dog'. The model assigns dog a probability of 0.5. What is the cross-entropy loss in bits?
Show answer
B. 1 With a one-hot label, cross-entropy is -log2 of the probability of the true class: -log2(0.5) = 1 bit.
-
When is the KL divergence between p and q exactly zero?
Show answer
B. When q = p KL divergence is the extra surprise from using q instead of p. It is zero only when the two distributions are identical.
-
Why is cross-entropy a better training loss than accuracy for a classifier?
Show answer
B. It changes smoothly with the predicted probabilities, giving a useful gradient, and it punishes confident mistakes Accuracy only changes when a prediction flips, so its gradient is zero almost everywhere. Cross-entropy responds to every change in confidence.
Recap
- Surprise is minus log p: rare outcomes carry more information, certain ones carry none.
- Entropy is average surprise, highest when outcomes are evenly spread.
- Cross-entropy measures surprise at the truth under the model's prediction, and is the standard classification loss.
- KL divergence is cross-entropy minus entropy: the extra cost of being wrong, zero only when q equals p.
- LLM pretraining, distillation and RLHF are all built on these quantities.
Next up
- From notebook to production learning path