Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Mathematics for ML (only what you need, visually)

Machine Learning ML-01.05

Probability fundamentals

Learn events, random variables, conditional probability and Bayes' theorem, and see why a positive medical test can still mean you are probably fine.

Beginner 15 min Last reviewed

The big idea

Machine learning is reasoning under uncertainty. A spam filter is never sure; it is 97 percent confident. A weather model does not say "rain", it says "70 percent chance of rain". Probability is the language for saying exactly how sure you are, and for updating that certainty when new evidence arrives.

Here is a puzzle that trips up most people, including many doctors. A disease affects 1 percent of people. A test catches 90 percent of sick people and correctly clears 95 percent of healthy ones. You test positive. How likely is it that you are sick? Most people guess around 90 percent. The real answer is about 15 percent. By the end of this lesson that number will feel obvious, and the reason it surprises people is the same reason models trained on imbalanced data behave strangely.

How it works

Step 1: Events, outcomes and random variables

A probabilityA number between 0 and 1 measuring how likely an event is; the probabilities of all possible outcomes add up to 1. is a number from 0 (never) to 1 (certain). An event is something that may or may not happen: "the die shows a 6", "this email is spam". A random variableA quantity whose value depends on a random outcome, such as the number shown by a die or tomorrow's temperature. is a number that depends on chance: the value a die shows, the time until the next customer arrives.

Random variables come in two kinds:

  • Discrete ones take separate values (a die: 1 to 6). They have a PMFProbability mass function: the probability of each separate value of a discrete random variable. giving each value's probability, for a fair die .
  • Continuous ones can take any value in a range (a height). Any exact value has probability 0, so instead they have a PDFProbability density function: a curve whose area over an interval is the probability of landing in that interval., a curve whose area over an interval is the probability of landing there.

Both have a CDFCumulative distribution function: the probability that the variable is less than or equal to a value., , which climbs from 0 to 1.

Step 2: Joint, marginal and conditional

With two random things, say whether a person is sick and whether their test is positive:

  • The joint probability is how likely both are.
  • The marginal probability ignores the other one: add the joint probabilities over every value of .
  • The conditional probability is how likely is once you know happened:

Conditioning means zooming into the world where happened and asking what fraction of it also has . Two events are independentKnowing one event happened does not change the probability of the other: P(A given B) = P(A). when knowing one tells you nothing about the other: .

Step 3: Bayes' theorem

Rearranging the definition gives Bayes' theoremP(A given B) = P(B given A) times P(A) divided by P(B). It flips a conditional probability and updates a prior belief with evidence.:

  • is the priorYour belief about something before seeing the new evidence, often the base rate in the population.: how common the disease is.
  • is the test's sensitivity.
  • is the chance of any positive: true positives plus false positives, .

So . About 15 percent.

The intuition is clearer with 1,000 people. 10 are sick and 9 of them test positive. 990 are healthy, and 5 percent of them, about 49.5, also test positive. Of the roughly 58.5 positives, only 9 are actually sick.

Bayes as areas A square of all people. A narrow strip on the left is sick people, mostly shaded green for testing positive. Across the wide healthy region, a thin red band at the top is healthy people who test positive. The red band is about five times the green area. sick (drawn wider) healthy test + healthy but test + (false positives) healthy and test - (most people) Figure 1. Everyone who tests positive is in the glowing green strip or the red band. The red band of false positives is much bigger, because healthy people vastly outnumber sick ones.

Ignoring the prior is called base rate neglectJudging a probability from evidence while ignoring how common the thing was to begin with.. In ML it shows up whenever a classifier is trained on balanced data and then deployed where positives are rare: precision collapses, for exactly this reason ( Imbalanced data ).

bayes.pyruns with python3 + numpy
prior = 0.01          # 1 in 100 people has the disease
sens = 0.90           # P(positive | sick)
spec = 0.95           # P(negative | healthy)

p_pos = sens * prior + (1 - spec) * (1 - prior)   # total probability of a positive
posterior = sens * prior / p_pos                  # Bayes' theorem
print(f"P(positive)          = {p_pos:.4f}")
print(f"P(sick | positive)   = {posterior:.3f}")

# Same thing with 1,000 people
sick, healthy = 1000 * prior, 1000 * (1 - prior)
tp, fp = sick * sens, healthy * (1 - spec)
print(f"{tp:.1f} true positives, {fp:.1f} false positives -> {tp / (tp + fp):.3f}")

Output

P(positive)          = 0.0585
P(sick | positive)   = 0.154
9.0 true positives, 49.5 false positives -> 0.154

Try it

This interactive needs JavaScript. The diagrams and text above cover the same idea.

Try this:

  1. Leave the defaults (1 percent, 90 percent, 95 percent) and read the answer: about 15 percent.
  2. Raise specificity to 99.9 percent. Watch the red band shrink and the answer jump. For rare conditions, false positives matter more than sensitivity.
  3. Raise "how common" to 30 percent, as if testing only people with symptoms. The same test is now much more trustworthy, because the prior changed.

Trade-offs

OptionWins whenLoses when
Report a single best guess Downstream code needs a yes or no decision Callers cannot tell a confident answer from a coin flip
Report a probability Decisions depend on costs, such as reviewing anything above 30 percent fraud risk The probabilities are poorly calibrated and get read too literally (see ML-04.06)
Update with Bayes as evidence arrives Evidence comes in pieces and priors are known, such as spam filters and diagnostics The prior is unknown or contested, and it dominates the answer

Common mistakes

  • Confusing with . The chance a sick person tests positive is not the chance a positive tester is sick.
  • Ignoring the base rate when the event is rare.
  • Assuming events are independent when they are not, such as treating two reviews from the same person as independent evidence.
  • Reading a PDF value as a probability. Densities can exceed 1; only areas under the curve are probabilities.

In the wild

Naive Bayes spam filters ( Naive Bayes ) apply Bayes' theorem word by word to decide whether an email is spam. Fraud and anomaly systems deal with tiny base rates, which is why they are judged on precision and recall instead of accuracy. Medical screening programmes are designed around exactly the calculation in this lesson: a cheap first test followed by a more specific second test for the positives.

Check yourself

  1. P(rain) = 0.3 and P(cloudy and rain) = 0.27. What is P(cloudy given rain)?

    Show answer

    B. 0.9 P(cloudy given rain) = P(cloudy and rain) / P(rain) = 0.27 / 0.3 = 0.9.

  2. A disease affects 1 in 1,000 people. A test has 99 percent sensitivity and 99 percent specificity. Roughly what share of positives are actually sick?

    Show answer

    C. About 9 percent Out of 100,000 people: 100 sick, 99 test positive. 99,900 healthy, 1 percent of them is 999 false positives. 99 / (99 + 999) is about 9 percent.

  3. Which statement about a PDF is true?

    Show answer

    C. The area under it over an interval is a probability For continuous variables, a single point has probability 0. Probabilities are areas under the density curve, and the density itself can exceed 1.

  4. Two events are independent when:

    Show answer

    B. P(A given B) equals P(A) Independence means learning B happened tells you nothing new about A. Events that cannot both happen are mutually exclusive, which is a different idea.

Recap

  • Probabilities run from 0 to 1; random variables are numbers that depend on chance.
  • Discrete variables have a PMF, continuous ones a PDF whose areas are probabilities, and both have a CDF.
  • Conditional probability zooms into the world where B happened: P(A given B) = P(A and B) / P(B).
  • Bayes' theorem combines a prior with evidence; ignoring the prior is base rate neglect.
  • Rare events make even accurate tests produce mostly false positives.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up