Skip to content
learn
  1. Learn
  2. Artificial Intelligence
  3. Reasoning Under Uncertainty

AI AI-03.01

Probability for AI (recap)

Why AI reasons with probabilities, joint distributions and inference by enumeration, and why a full joint table explodes as variables grow.

Beginner 8 min Last reviewed

The big idea

The logic of the last module answers "true", "false" or "unknown". Real agents need more. A self-driving car's camera sees something that is probably a plastic bag. A support bot reads a message that is likely about billing. A doctor's test comes back positive, which raises the chance of disease without settling it. Treating these as true or false throws information away; treating them as unknown is useless. ProbabilityA number between 0 and 1 expressing how strongly an agent believes a statement, updated as evidence arrives. is the language for degrees of belief, and it comes with exact rules for updating them.

Think of a weather forecaster. "70 percent chance of rain" is not a hedge; it is a precise statement that, over many such days, it rains on about 7 in 10. And when the forecaster sees dark clouds, the number moves in a principled way. AI systems reason the same way, and the main challenge, which this module is about, is doing it with many variables at once without the bookkeeping exploding.

How it works

Step 1: Random variables and the joint distribution

A random variable is something whose value is uncertain: (yes or no), (on or off), (wet grass or dry). A complete assignment of values to every variable is one possible world, exactly like a model in Propositional logic . The joint distributionThe probability of every complete combination of values of all the variables. Any question about those variables can be answered from it. assigns a probability to every possible world, and the probabilities add up to 1.

With three yes/no variables there are worlds, so the joint distribution is an 8-row table. Everything you might ask about rain, sprinklers and wet grass can be answered from those 8 numbers.

Step 2: Conditional probability and Bayes' rule

The probability of given that you know is

which means: among the worlds where holds, what fraction also have ? Rearranging gives Bayes' rule, which turns "how likely is the evidence if the cause is present" into "how likely is the cause given the evidence":

Step 3: Inference by enumeration

Given the joint table, any question is answered the same way: add up the probabilities of the worlds that match the question and the evidence, and divide by the total for the evidence. That is inference by enumerationAnswering a probability question by summing the joint probabilities of all worlds consistent with the query and the evidence, then normalising..

Try it

Here the joint table for rain, sprinkler and wet grass is built from three small tables (how likely rain is, how the sprinkler depends on rain, how wet grass depends on both), then queried by enumeration:

joint.pyruns with python3 (standard library)
from itertools import product

# Three yes/no variables: Rain, Sprinkler, Wet grass.
P_RAIN = 0.2
P_SPRINKLER = {True: 0.01, False: 0.4}       # given Rain
P_WET = {(True, True): 0.99, (True, False): 0.9,  # given (Sprinkler,
         (False, True): 0.8, (False, False): 0.0}  #        Rain)

def p(rain, spr, wet):
    """Probability of one complete world, by the chain rule."""
    pr = P_RAIN if rain else 1 - P_RAIN
    ps = P_SPRINKLER[rain] if spr else 1 - P_SPRINKLER[rain]
    pw = P_WET[(spr, rain)] if wet else 1 - P_WET[(spr, rain)]
    return pr * ps * pw

joint = {w: p(*w) for w in product([True, False], repeat=3)}
print("rain   sprinkler wet    P")
for (r, s, w), pr in joint.items():
    print(f"{r!s:6} {s!s:9} {w!s:6} {pr:.4f}")
print(f"sum = {sum(joint.values()):.4f}")

def ask(query, evidence):
    """P(query | evidence) by enumeration: add up matching worlds."""
    def total(cond):
        return sum(pr for w, pr in joint.items()
                   if all(w[i] == v for i, v in cond.items()))
    return total({**evidence, **query}) / total(evidence)

RAIN, SPR, WET = 0, 1, 2
print(f"P(rain)                  = {P_RAIN:.3f}")
print(f"P(rain | wet)            = {ask({RAIN: True}, {WET: True}):.3f}")
both = ask({RAIN: True}, {WET: True, SPR: True})
print(f"P(rain | wet, sprinkler) = {both:.3f}")

for n in (3, 10, 30, 100):          # rows in a full joint table
    print(f"{n:3} yes/no variables -> {2 ** n:.3g} rows")

Output

rain   sprinkler wet    P
True   True      True   0.0020
True   True      False  0.0000
True   False     True   0.1584
True   False     False  0.0396
False  True      True   0.2880
False  True      False  0.0320
False  False     True   0.0000
False  False     False  0.4800
sum = 1.0000
P(rain)                  = 0.200
P(rain | wet)            = 0.358
P(rain | wet, sprinkler) = 0.007
  3 yes/no variables -> 8 rows
 10 yes/no variables -> 1.02e+03 rows
 30 yes/no variables -> 1.07e+09 rows
100 yes/no variables -> 1.27e+30 rows

Read the three answers in order:

  • Before looking at the grass, the chance of rain is 0.2.
  • Seeing wet grass raises it to 0.358: rain is one of the two ways grass gets wet.
  • Learning that the sprinkler was on drops it to 0.007, far below where it started. The sprinkler already explains the wet grass, and in this world it is rarely switched on when it rains. Evidence for one cause makes a competing cause less likely: this is called explaining away, and it is the kind of reasoning that is natural with probabilities and awkward with rules.

The last lines show the problem this module solves.

Size of a full joint table as the number of yes/no variables grows A log-scale bar chart of the rows needed for a full joint distribution over yes/no variables: 8 rows for 3 variables, about a thousand for 10, about a billion for 30, and about ten to the thirtieth for 100, which glows red. rows in the joint tablefor comparison 3 variables 8 fits on an index card 10 variables about 10^3 a long spreadsheet 30 variables about 10^9 about 8 GB of 8-byte numbers 100 variables 10^30 beyond all storage on Earth Figure 1. Each extra yes/no variable doubles the joint table. A medical or sensor model with 100 variables would need about 10^30 numbers, impossible to store, estimate or sum over.

Going deeper

Why exact inference explodes

The joint table grows as for yes/no variables, and it is not just a storage problem. You would also need data to estimate every number, and enumeration must sum over all the worlds consistent with the evidence. With 100 variables, both are hopeless.

The way out is that most variables do not directly affect most others. Whether the grass is wet does not depend on the day's stock prices. IndependenceA and B are independent if learning one tells you nothing about the other: P(A | B) = P(A). and, more usefully, conditional independence, "once you know whether it rained and whether the sprinkler ran, the clouds tell you nothing more about the grass", let the joint distribution be built from many small tables. The code above already did this: 1 + 2 + 4 = 7 numbers defined all 8 worlds. For a hundred variables, each depending on a few others, the saving is astronomical. That is a Bayesian network, the subject of Bayesian networks .

Where the numbers come from

The probabilities in a model come from data (counting how often the sprinkler is on when it rains), from experts, or from both. Learning them from data is the bridge to machine learning, where models are fitted by maximum likelihood and its relatives. And modern language models are themselves probability models: they assign a probability to every possible next token, which Decoding and sampling uses to generate text.

Trade-offs

OptionWins whenLoses when
Logic (true / false / unknown) Rules that are definite, such as legal or mathematical constraints Noisy evidence and degrees of belief
Full joint distribution A handful of variables; exact answers by enumeration Grows as 2^n: impossible to store, estimate or sum at scale
Structured models (Bayesian networks) Many variables with few direct influences Need the structure; some queries remain hard

Common mistakes

  • Confusing with . The chance of wet grass given rain (0.8 or more) is not the chance of rain given wet grass (0.358).
  • Ignoring the prior. A rare cause stays fairly unlikely even after supporting evidence; the classic positive-test example in Probability fundamentals shows why.
  • Treating independence as the default. Assuming variables are independent when they are not, such as two symptoms of the same disease, double-counts evidence.
  • Forgetting to normalise. Conditional probabilities must divide by the probability of the evidence.

In the wild

Where you will see this: spam filters, medical decision support, fault diagnosis in machines and networks, robot localisation from noisy sensors, and fraud scoring all reason with probabilities rather than hard rules. Language models output a probability distribution over the next token at every step, and calibration, whether a model's stated confidence matches how often it is right, is a key concern when deploying any AI system.

Check yourself

  1. What does a joint distribution over Rain, Sprinkler and Wet give you?

    Show answer

    B. A probability for every combination of the three values, from which any question can be answered Every query about those variables can be answered by summing the right rows of the joint table.

  2. Wet grass raised P(rain) from 0.2 to 0.358. Learning the sprinkler was on then dropped it to 0.007. What is this called?

    Show answer

    B. Explaining away The sprinkler already explains the wet grass, so the competing explanation, rain, becomes less likely.

  3. How many rows does a full joint table need for 20 yes/no variables?

    Show answer

    C. About a million 2^20 is 1,048,576, about a million. Each extra variable doubles it.

  4. What makes reasoning with many variables tractable?

    Show answer

    B. Conditional independence, which lets the joint be built from small tables When each variable directly depends on only a few others, a handful of small tables defines the whole joint distribution.

Recap

  • Probability expresses degrees of belief and gives exact rules for updating them with evidence.
  • The joint distribution assigns a probability to every possible world; inference by enumeration sums the matching worlds.
  • Bayes' rule turns P(evidence | cause) into P(cause | evidence); explaining away falls out naturally.
  • A full joint table doubles with every variable, so it cannot scale.
  • Conditional independence lets small tables define the joint, which is what Bayesian networks exploit.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy