AI AI-03.01
Probability for AI (recap)
Why AI reasons with probabilities, joint distributions and inference by enumeration, and why a full joint table explodes as variables grow.
The big idea
The logic of the last module answers "true", "false" or "unknown". Real agents need more. A self-driving car's camera sees something that is probably a plastic bag. A support bot reads a message that is likely about billing. A doctor's test comes back positive, which raises the chance of disease without settling it. Treating these as true or false throws information away; treating them as unknown is useless. ProbabilityA number between 0 and 1 expressing how strongly an agent believes a statement, updated as evidence arrives. is the language for degrees of belief, and it comes with exact rules for updating them.
Think of a weather forecaster. "70 percent chance of rain" is not a hedge; it is a precise statement that, over many such days, it rains on about 7 in 10. And when the forecaster sees dark clouds, the number moves in a principled way. AI systems reason the same way, and the main challenge, which this module is about, is doing it with many variables at once without the bookkeeping exploding.
How it works
Step 1: Random variables and the joint distribution
A random variable is something whose value is uncertain: (yes or no), (on or off), (wet grass or dry). A complete assignment of values to every variable is one possible world, exactly like a model in Propositional logic . The joint distributionThe probability of every complete combination of values of all the variables. Any question about those variables can be answered from it. assigns a probability to every possible world, and the probabilities add up to 1.
With three yes/no variables there are worlds, so the joint distribution is an 8-row table. Everything you might ask about rain, sprinklers and wet grass can be answered from those 8 numbers.
Step 2: Conditional probability and Bayes' rule
The probability of given that you know is
which means: among the worlds where holds, what fraction also have ? Rearranging gives Bayes' rule, which turns "how likely is the evidence if the cause is present" into "how likely is the cause given the evidence":
Step 3: Inference by enumeration
Given the joint table, any question is answered the same way: add up the probabilities of the worlds that match the question and the evidence, and divide by the total for the evidence. That is inference by enumerationAnswering a probability question by summing the joint probabilities of all worlds consistent with the query and the evidence, then normalising..
Try it
Here the joint table for rain, sprinkler and wet grass is built from three small tables (how likely rain is, how the sprinkler depends on rain, how wet grass depends on both), then queried by enumeration:
from itertools import product
# Three yes/no variables: Rain, Sprinkler, Wet grass.
P_RAIN = 0.2
P_SPRINKLER = {True: 0.01, False: 0.4} # given Rain
P_WET = {(True, True): 0.99, (True, False): 0.9, # given (Sprinkler,
(False, True): 0.8, (False, False): 0.0} # Rain)
def p(rain, spr, wet):
"""Probability of one complete world, by the chain rule."""
pr = P_RAIN if rain else 1 - P_RAIN
ps = P_SPRINKLER[rain] if spr else 1 - P_SPRINKLER[rain]
pw = P_WET[(spr, rain)] if wet else 1 - P_WET[(spr, rain)]
return pr * ps * pw
joint = {w: p(*w) for w in product([True, False], repeat=3)}
print("rain sprinkler wet P")
for (r, s, w), pr in joint.items():
print(f"{r!s:6} {s!s:9} {w!s:6} {pr:.4f}")
print(f"sum = {sum(joint.values()):.4f}")
def ask(query, evidence):
"""P(query | evidence) by enumeration: add up matching worlds."""
def total(cond):
return sum(pr for w, pr in joint.items()
if all(w[i] == v for i, v in cond.items()))
return total({**evidence, **query}) / total(evidence)
RAIN, SPR, WET = 0, 1, 2
print(f"P(rain) = {P_RAIN:.3f}")
print(f"P(rain | wet) = {ask({RAIN: True}, {WET: True}):.3f}")
both = ask({RAIN: True}, {WET: True, SPR: True})
print(f"P(rain | wet, sprinkler) = {both:.3f}")
for n in (3, 10, 30, 100): # rows in a full joint table
print(f"{n:3} yes/no variables -> {2 ** n:.3g} rows")Output
rain sprinkler wet P
True True True 0.0020
True True False 0.0000
True False True 0.1584
True False False 0.0396
False True True 0.2880
False True False 0.0320
False False True 0.0000
False False False 0.4800
sum = 1.0000
P(rain) = 0.200
P(rain | wet) = 0.358
P(rain | wet, sprinkler) = 0.007
3 yes/no variables -> 8 rows
10 yes/no variables -> 1.02e+03 rows
30 yes/no variables -> 1.07e+09 rows
100 yes/no variables -> 1.27e+30 rows Read the three answers in order:
- Before looking at the grass, the chance of rain is 0.2.
- Seeing wet grass raises it to 0.358: rain is one of the two ways grass gets wet.
- Learning that the sprinkler was on drops it to 0.007, far below where it started. The sprinkler already explains the wet grass, and in this world it is rarely switched on when it rains. Evidence for one cause makes a competing cause less likely: this is called explaining away, and it is the kind of reasoning that is natural with probabilities and awkward with rules.
The last lines show the problem this module solves.
Going deeper
Why exact inference explodes
The joint table grows as for yes/no variables, and it is not just a storage problem. You would also need data to estimate every number, and enumeration must sum over all the worlds consistent with the evidence. With 100 variables, both are hopeless.
The way out is that most variables do not directly affect most others. Whether the grass is wet does not depend on the day's stock prices. IndependenceA and B are independent if learning one tells you nothing about the other: P(A | B) = P(A). and, more usefully, conditional independence, "once you know whether it rained and whether the sprinkler ran, the clouds tell you nothing more about the grass", let the joint distribution be built from many small tables. The code above already did this: 1 + 2 + 4 = 7 numbers defined all 8 worlds. For a hundred variables, each depending on a few others, the saving is astronomical. That is a Bayesian network, the subject of Bayesian networks .
Where the numbers come from
The probabilities in a model come from data (counting how often the sprinkler is on when it rains), from experts, or from both. Learning them from data is the bridge to machine learning, where models are fitted by maximum likelihood and its relatives. And modern language models are themselves probability models: they assign a probability to every possible next token, which Decoding and sampling uses to generate text.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Logic (true / false / unknown) | Rules that are definite, such as legal or mathematical constraints | Noisy evidence and degrees of belief |
| Full joint distribution | A handful of variables; exact answers by enumeration | Grows as 2^n: impossible to store, estimate or sum at scale |
| Structured models (Bayesian networks) | Many variables with few direct influences | Need the structure; some queries remain hard |
Common mistakes
- Confusing with . The chance of wet grass given rain (0.8 or more) is not the chance of rain given wet grass (0.358).
- Ignoring the prior. A rare cause stays fairly unlikely even after supporting evidence; the classic positive-test example in Probability fundamentals shows why.
- Treating independence as the default. Assuming variables are independent when they are not, such as two symptoms of the same disease, double-counts evidence.
- Forgetting to normalise. Conditional probabilities must divide by the probability of the evidence.
In the wild
Where you will see this: spam filters, medical decision support, fault diagnosis in machines and networks, robot localisation from noisy sensors, and fraud scoring all reason with probabilities rather than hard rules. Language models output a probability distribution over the next token at every step, and calibration, whether a model's stated confidence matches how often it is right, is a key concern when deploying any AI system.
Check yourself
-
What does a joint distribution over Rain, Sprinkler and Wet give you?
Show answer
B. A probability for every combination of the three values, from which any question can be answered Every query about those variables can be answered by summing the right rows of the joint table.
-
Wet grass raised P(rain) from 0.2 to 0.358. Learning the sprinkler was on then dropped it to 0.007. What is this called?
Show answer
B. Explaining away The sprinkler already explains the wet grass, so the competing explanation, rain, becomes less likely.
-
How many rows does a full joint table need for 20 yes/no variables?
Show answer
C. About a million 2^20 is 1,048,576, about a million. Each extra variable doubles it.
-
What makes reasoning with many variables tractable?
Show answer
B. Conditional independence, which lets the joint be built from small tables When each variable directly depends on only a few others, a handful of small tables defines the whole joint distribution.
Recap
- Probability expresses degrees of belief and gives exact rules for updating them with evidence.
- The joint distribution assigns a probability to every possible world; inference by enumeration sums the matching worlds.
- Bayes' rule turns P(evidence | cause) into P(cause | evidence); explaining away falls out naturally.
- A full joint table doubles with every variable, so it cannot scale.
- Conditional independence lets small tables define the joint, which is what Bayesian networks exploit.