Machine Learning ML-04.03
Naive Bayes
Bayes' theorem plus one bold simplification: a spam filter that trains in one pass, needs little data, and uses Laplace smoothing to avoid zeros.
The big idea
A spam filter sees "free money" in an email. Spam often contains "free"; it often contains "money". Each word is a small piece of evidence, and Bayes' theorem tells you how to combine evidence into a probability. The difficulty is that words come in combinations, and you will never have enough data to estimate the probability of every combination.
Naive BayesA classifier that applies Bayes' theorem while assuming every feature is independent of the others once the class is known. makes one bold simplification: it pretends that, once you know whether an email is spam, each word appears independently of the others. Like a detective who scores each clue separately and multiplies the odds together, ignoring that some clues tend to arrive in pairs. The assumption is clearly false ("free" and "prize" travel together), yet the classifier works surprisingly well, trains in a single pass over the data, and needs very few examples.
How it works
Step 1: Bayes' theorem for classes
For a message with words w_1, ..., w_n, we want the probability of each class:
The first term is the priorThe probability of a class before looking at the features, usually its share of the training data., the share of spam in training. The second is the likelihood of seeing exactly these words in spam, and it is the problem: there are far too many word combinations to count.
Step 2: The naive assumption
Assume the words are conditionally independentTwo features are conditionally independent given the class if, once you know the class, learning one feature tells you nothing more about the other. given the class. Then the joint likelihood breaks into a product of single-word likelihoods that are easy to count:
Step 3: Train by counting
Training is nothing more than counting. For each class, count how often each word appears and divide by the total words in that class. Take a toy training set of six messages: three spam ("win free money now", "free prize claim now", "cheap money offer") and three ham ("meeting at noon", "lunch at noon tomorrow", "project meeting notes"). The spam messages contain 11 words in total, 2 of them "money", so P(money | spam) = 2/11 = 0.18. The ham messages never contain "money", so P(money | ham) = 0.
Step 4: Laplace smoothing
That zero is a disaster. A single word never seen in ham makes the whole product for ham zero, no matter how much other evidence points to ham. A ham email that happens to mention money would be impossible to classify as ham.
Laplace smoothingAdding a small count (usually 1) to every word in every class before computing probabilities, so no word ever gets probability zero. adds a pseudo-count alpha (usually 1) to every word in every class:
where |V| is the vocabulary size, 15 words here. Now P(money | spam) = (2 + 1) / (11 + 15) = 0.115 and P(money | ham) = (0 + 1) / (10 + 15) = 0.040. Unseen words get a small but nonzero probability.
Step 5: Classify a new message
Score "free money meeting". The priors are equal (3 spam, 3 ham), so they cancel. With smoothing:
- spam: P(free | spam) x P(money | spam) x P(meeting | spam) = 0.115 x 0.115 x 0.038 = 0.00051
- ham: 0.040 x 0.040 x 0.120 = 0.00019
Normalise: P(spam) = 0.00051 / (0.00051 + 0.00019) = 0.73. Two spammy words beat one hammy word.
In practice you work with logarithms. Multiplying hundreds of small probabilities underflows to zero in floating point, while adding their logs is stable. In log space naive Bayes is a linear model: each word adds a fixed weight to the log-odds, just like logistic regression. The difference is how the weights are found: naive Bayes counts, logistic regression optimises.
Going deeper
Code: the toy filter and a learning race
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
# Six tiny training messages
docs = ["win free money now", "free prize claim now", "cheap money offer",
"meeting at noon", "lunch at noon tomorrow", "project meeting notes"]
labels = np.array([1, 1, 1, 0, 0, 0]) # 1 = spam
vec = CountVectorizer()
X = vec.fit_transform(docs)
vocab = vec.get_feature_names_out()
print("vocabulary size:", len(vocab))
for alpha in [1.0, 1e-10]:
nb = MultinomialNB(alpha=alpha, force_alpha=True).fit(X, labels)
P = np.exp(nb.feature_log_prob_) # P(word | class), rows: ham, spam
i = list(vocab).index("money")
j = list(vocab).index("meeting")
print(f"\nalpha = {alpha:g}")
print(f" P(money | spam) = {P[1, i]:.3f} P(money | ham) = {P[0, i]:.3f}")
print(f" P(meeting | spam) = {P[1, j]:.3f} P(meeting | ham) = {P[0, j]:.3f}")
msg = vec.transform(["free money meeting"])
print(" 'free money meeting' -> P(spam) =", nb.predict_proba(msg)[0, 1].round(3))
# A bigger synthetic corpus: who needs less data?
rng = np.random.default_rng(0)
spam_w = "free win prize money cheap offer click now deal bonus".split()
ham_w = "meeting noon project notes lunch report team call review plan".split()
both_w = "the a you your today please thanks hello time new".split()
def message(spam):
own, other = (spam_w, ham_w) if spam else (ham_w, spam_w)
words = list(rng.choice(own, 3)) + list(rng.choice(other, 1)) + list(rng.choice(both_w, 4))
return " ".join(words)
y = (rng.random(4000) < 0.4).astype(int)
text = [message(s) for s in y]
Xall = CountVectorizer().fit_transform(text)
Xte, yte = Xall[3000:], y[3000:]
print("\ntrain size naive Bayes logistic regression")
for n in [10, 30, 100, 1000]:
nb = MultinomialNB().fit(Xall[:n], y[:n])
lr = LogisticRegression(max_iter=1000).fit(Xall[:n], y[:n])
print(f"{n:>10} {nb.score(Xte, yte):.3f} {lr.score(Xte, yte):.3f}")Output
vocabulary size: 15
alpha = 1
P(money | spam) = 0.115 P(money | ham) = 0.040
P(meeting | spam) = 0.038 P(meeting | ham) = 0.120
'free money meeting' -> P(spam) = 0.727
alpha = 1e-10
P(money | spam) = 0.182 P(money | ham) = 0.000
P(meeting | spam) = 0.000 P(meeting | ham) = 0.200
'free money meeting' -> P(spam) = 1.0
train size naive Bayes logistic regression
10 0.692 0.659
30 0.818 0.796
100 0.948 0.998
1000 1.000 1.000 With smoothing (alpha = 1), the probabilities match the hand calculation and "free money meeting" scores 0.727. With smoothing effectively off (alpha close to 0), the probabilities contain hard zeros and the score jumps to 1.0: the answer is decided by which class has fewer never-seen words, not by the evidence.
The second part trains both models on a synthetic corpus. With 10 or 30 messages, naive Bayes is ahead (0.692 against 0.659, then 0.818 against 0.796). By 100 messages logistic regression has overtaken it, 0.998 against 0.948. This is the typical pattern: naive Bayes learns fast from little data, and a discriminative model catches up and passes it as data grows, because it is not limited by the independence assumption.
Variants for different features
- Multinomial NB: word counts. The standard for text.
- Bernoulli NB: word present or absent. Works for short texts.
- Gaussian NB: continuous features, each modelled as a normal distribution per class.
Why it works despite being wrong
The independence assumption double-counts correlated evidence ("free" and "prize" both fire), which pushes probabilities toward 0 and 1. So naive Bayes probabilities are badly calibrated ( Probability calibration ). But classification only needs the right class to win, and the ranking of classes is often still correct even when the probabilities are exaggerated.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Naive Bayes | Text with many features, few labels, or a need to train in seconds and update online | You need trustworthy probabilities, or features interact strongly |
| Logistic regression | Enough data; calibrated-ish probabilities; correlated features | Very little labelled data |
| Fine-tuned language model | Meaning matters more than keywords; large budgets | Tight latency, tiny budgets, or a need for full transparency |
Common mistakes
- Forgetting smoothing, so one unseen word zeroes out a class.
- Multiplying raw probabilities instead of summing logs, and getting underflow to 0.
- Using naive Bayes probabilities as real risks; they are usually far too confident.
- Fitting the vectorizer's vocabulary on all data before splitting, leaking test words into training.
- Using Gaussian NB on counts or Multinomial NB on negative continuous values.
In the wild
Naive Bayes was one of the first widely used statistical spam filters, and Bayesian filtering in mail software followed the same recipe: count words per class, smooth, combine log-likelihoods. It remains a common first baseline for text classification such as support ticket routing, document categorisation and sentiment, because it trains in one pass, updates incrementally as new labelled messages arrive, and handles vocabularies of hundreds of thousands of words cheaply.
Check yourself
-
What exactly does the 'naive' assumption say?
Show answer
B. Features are independent of each other once the class is known Conditional independence given the class lets the joint likelihood factor into a product of single-feature likelihoods.
-
The word 'invoice' never appeared in spam during training. Without smoothing, what happens to an email containing it?
Show answer
B. Its spam probability becomes exactly 0, whatever the other words say P(invoice | spam) = 0 makes the whole product for spam zero. Laplace smoothing gives unseen words a small nonzero probability instead.
-
With 20 labelled tickets, naive Bayes beats logistic regression. With 50,000 it loses. Why?
Show answer
B. NB's strong assumption helps with little data, but limits it once data can support a more flexible model Strong assumptions reduce variance when data is scarce. With plenty of data, the bias from the false independence assumption dominates.
-
Why do implementations sum log-probabilities instead of multiplying probabilities?
Show answer
B. Multiplying hundreds of small numbers underflows to 0 in floating point log(a x b) = log a + log b gives the same ranking without underflow. It also shows naive Bayes is linear in log space.
Recap
- Naive Bayes applies Bayes' theorem and assumes features are independent given the class.
- Training is counting: priors from class shares, likelihoods from per-class feature frequencies.
- Laplace smoothing adds a pseudo-count so unseen words never force a probability to zero.
- In log space it is a linear model; it learns fast from little data but its probabilities are overconfident.
- It remains a strong, cheap baseline for text classification such as spam filtering.
Next up
- From notebook to production learning path