Machine Learning ML-01.06
Distributions you will meet
Recognise the Bernoulli, binomial, Poisson, uniform, normal and exponential distributions, know where each appears, and compute mean and variance.
The big idea
A distributionA description of how likely each possible value of a random variable is, as a PMF for discrete values or a PDF for continuous ones. is the shape of uncertainty: which values are likely, which are rare. The good news is that a handful of shapes cover a remarkable amount of the real world. Clicks or no clicks, the number of support tickets per hour, the time between server failures, measurement noise: each of these follows one of six standard distributions so reliably that recognising them is a superpower. It tells you which loss function to use, which model assumptions are reasonable, and when your data looks suspicious.
Think of them as the common chord shapes of music. Real songs are endlessly varied, but most are built from a small set of shapes, and once you recognise them you hear structure everywhere.
How it works
Two numbers summarize any distribution. The meanThe average value you would get over many draws, also called the expectation: E[X]. (or expectation, ) is where it balances. The varianceThe average squared distance from the mean, measuring spread. Its square root is the standard deviation. measures spread: the average squared distance from the mean, . Its square root, the standard deviation, is in the same units as the data.
Discrete distributions: counting things
- Bernoulli(p): one yes-or-no trial with success probability . Did this user click? Mean , variance . Binary classifiers model each label this way, which is why logistic regression outputs a probability .
- Binomial(n, p): the number of successes in independent Bernoulli trials. How many of 10 shown ads get clicked? Mean , variance .
- Poisson(rate): the number of events in a fixed interval when they happen independently at a steady average rate. Tickets per hour, typos per page, requests per second. Mean and variance are both equal to the rate.
Continuous distributions: measuring things
- Uniform(a, b): every value between and equally likely. Random initial guesses, random sampling. Mean , variance .
- Normal(mean, sd): the bell curve. It appears whenever many small independent effects add up, which is why heights, measurement errors and averages are roughly normal. About 68 percent of values fall within one standard deviation of the mean and about 95 percent within two.
- Exponential(rate): waiting time until the next event in a Poisson process. Time until the next request, the next failure. Mean . It is memorylessThe property that the remaining wait does not depend on how long you have already waited.: having waited 5 minutes tells you nothing about how much longer you will wait.
import numpy as np
rng = np.random.default_rng(7)
n = 100_000
samples = {
"bernoulli p=0.3": rng.binomial(1, 0.3, n),
"binomial n=10 p=0.3": rng.binomial(10, 0.3, n),
"poisson rate=3": rng.poisson(3, n),
"uniform 0..1": rng.uniform(0, 1, n),
"normal 0, 1": rng.normal(0, 1, n),
"exponential rate=2": rng.exponential(1 / 2, n),
}
for name, x in samples.items():
print(f"{name:20s} mean {x.mean():6.3f} variance {x.var():6.3f}")Output
bernoulli p=0.3 mean 0.300 variance 0.210
binomial n=10 p=0.3 mean 3.001 variance 2.078
poisson rate=3 mean 2.997 variance 2.986
uniform 0..1 mean 0.500 variance 0.083
normal 0, 1 mean -0.003 variance 0.999
exponential rate=2 mean 0.501 variance 0.252 The sampled means and variances land within a whisker of the formulas, a preview of the law of large numbers in Statistics: sampling, CLT and confidence intervals .
Try it
Try this:
- Pick the binomial with and slide from 1 to 40. A lopsided pair of bars turns into a bell. That is the central limit theorem waving at you.
- Pick the Poisson and slide the rate. Notice the mean and variance stay equal. If your real count data has a variance far bigger than its mean, it is not Poisson, a sign of bursts or mixed groups.
- Pick the exponential and raise the rate. Faster events mean shorter waits: the mean is 1 divided by the rate.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Assume a standard distribution | The data plausibly matches it: simple formulas, small samples go far | The data has heavy tails, several groups or bursts, and the assumption hides that |
| Work with the empirical distribution | Plenty of data and no good parametric fit; bootstrapping, histograms | Small samples, or you need to extrapolate beyond what you have seen |
| Model the target with a matching loss | Counts with a Poisson loss, binary labels with log loss | You force squared error onto everything by habit |
Common mistakes
- Assuming everything is normal. Incomes, response times and file sizes are skewed with long tails; their mean can be far from the typical value.
- Using a normal model for counts near zero, which predicts impossible negative counts.
- Treating events as independent when they come in bursts, which makes Poisson badly underestimate variance.
- Confusing standard deviation (spread of the data) with standard error (uncertainty of an average), covered next.
In the wild
The choice of loss function in ML often encodes a distribution: squared error corresponds to normal noise, log loss to a Bernoulli label, and Poisson regression to counts. Capacity planning for web services models request arrivals as Poisson and waiting times as exponential. A/B testing platforms model conversions as binomial to decide whether a new variant really performs better.
Check yourself
-
You count how many orders arrive per minute at a steady average of 4. Which distribution fits best?
Show answer
B. Poisson with rate 4 Independent events counted in a fixed interval at a steady rate are the textbook Poisson case.
-
A binomial has n = 20 and p = 0.25. What are its mean and variance?
Show answer
A. 5 and 3.75 Mean np = 5; variance np(1 - p) = 20 x 0.25 x 0.75 = 3.75.
-
Server failures arrive at an average rate of 2 per day. What is the average waiting time to the next failure?
Show answer
B. Half a day Waiting times in a Poisson process are exponential with mean 1 / rate = 0.5 days, and memorylessness means the time already waited does not matter.
-
About what share of a normal distribution lies within two standard deviations of the mean?
Show answer
C. 95 percent Roughly 68 percent within one standard deviation and 95 percent within two.
Recap
- Mean says where a distribution balances; variance and standard deviation say how spread out it is.
- Bernoulli is one yes or no; binomial counts yeses in n trials; Poisson counts events per interval.
- Uniform is flat, normal is the bell that appears when many small effects add up, exponential models waiting times.
- Matching the distribution to your data guides the choice of loss and catches suspicious data.
Next up
- From notebook to production learning path