Skip to content
learn
  1. Learn
  2. Machine Learning
  3. Classification

Machine Learning ML-04.03

Naive Bayes

Bayes' theorem plus one bold simplification: a spam filter that trains in one pass, needs little data, and uses Laplace smoothing to avoid zeros.

Intermediate 10 min Last reviewed

The big idea

A spam filter sees "free money" in an email. Spam often contains "free"; it often contains "money". Each word is a small piece of evidence, and Bayes' theorem tells you how to combine evidence into a probability. The difficulty is that words come in combinations, and you will never have enough data to estimate the probability of every combination.

Naive BayesA classifier that applies Bayes' theorem while assuming every feature is independent of the others once the class is known. makes one bold simplification: it pretends that, once you know whether an email is spam, each word appears independently of the others. Like a detective who scores each clue separately and multiplies the odds together, ignoring that some clues tend to arrive in pairs. The assumption is clearly false ("free" and "prize" travel together), yet the classifier works surprisingly well, trains in a single pass over the data, and needs very few examples.

How it works

Step 1: Bayes' theorem for classes

For a message with words w_1, ..., w_n, we want the probability of each class:

The first term is the priorThe probability of a class before looking at the features, usually its share of the training data., the share of spam in training. The second is the likelihood of seeing exactly these words in spam, and it is the problem: there are far too many word combinations to count.

Step 2: The naive assumption

Assume the words are conditionally independentTwo features are conditionally independent given the class if, once you know the class, learning one feature tells you nothing more about the other. given the class. Then the joint likelihood breaks into a product of single-word likelihoods that are easy to count:

The naive Bayes assumption as a graph A glowing class node labelled spam or ham at the top, with arrows down to three word nodes: free, money and meeting. Between the word nodes, dashed red lines with the label no links assumed show that the model ignores any connection between words once the class is known. class spam or ham free money meeting dashed: links between words, assumed away Figure 1. Naive Bayes assumes the class alone explains each word. The dashed links between words are what it ignores, which is why each word's evidence can simply be multiplied in.

Step 3: Train by counting

Training is nothing more than counting. For each class, count how often each word appears and divide by the total words in that class. Take a toy training set of six messages: three spam ("win free money now", "free prize claim now", "cheap money offer") and three ham ("meeting at noon", "lunch at noon tomorrow", "project meeting notes"). The spam messages contain 11 words in total, 2 of them "money", so P(money | spam) = 2/11 = 0.18. The ham messages never contain "money", so P(money | ham) = 0.

Step 4: Laplace smoothing

That zero is a disaster. A single word never seen in ham makes the whole product for ham zero, no matter how much other evidence points to ham. A ham email that happens to mention money would be impossible to classify as ham.

Laplace smoothingAdding a small count (usually 1) to every word in every class before computing probabilities, so no word ever gets probability zero. adds a pseudo-count alpha (usually 1) to every word in every class:

where |V| is the vocabulary size, 15 words here. Now P(money | spam) = (2 + 1) / (11 + 15) = 0.115 and P(money | ham) = (0 + 1) / (10 + 15) = 0.040. Unseen words get a small but nonzero probability.

Step 5: Classify a new message

Score "free money meeting". The priors are equal (3 spam, 3 ham), so they cancel. With smoothing:

  • spam: P(free | spam) x P(money | spam) x P(meeting | spam) = 0.115 x 0.115 x 0.038 = 0.00051
  • ham: 0.040 x 0.040 x 0.120 = 0.00019

Normalise: P(spam) = 0.00051 / (0.00051 + 0.00019) = 0.73. Two spammy words beat one hammy word.

Each word adds its own evidence Horizontal bars for the message free money meeting. free and money each add 1.06 to the log-odds of spam, shown in red. meeting subtracts 1.14, shown in green. The total log-odds is +0.98, which corresponds to a spam probability of 0.73. log-oddseffect free +1.06 toward spam money +1.06 toward spam meeting -1.14 toward ham total log-odds +0.98 P(spam) = 0.73 Figure 2. In log space, naive Bayes is a sum: each word adds log(P(word | spam) / P(word | ham)) to the score. The total, +0.98, maps to a spam probability of 0.73.

In practice you work with logarithms. Multiplying hundreds of small probabilities underflows to zero in floating point, while adding their logs is stable. In log space naive Bayes is a linear model: each word adds a fixed weight to the log-odds, just like logistic regression. The difference is how the weights are found: naive Bayes counts, logistic regression optimises.

Going deeper

Code: the toy filter and a learning race

naive-bayes.pyruns with python3 + numpy + scikit-learn
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression

# Six tiny training messages
docs = ["win free money now", "free prize claim now", "cheap money offer",
        "meeting at noon", "lunch at noon tomorrow", "project meeting notes"]
labels = np.array([1, 1, 1, 0, 0, 0])            # 1 = spam
vec = CountVectorizer()
X = vec.fit_transform(docs)
vocab = vec.get_feature_names_out()
print("vocabulary size:", len(vocab))

for alpha in [1.0, 1e-10]:
    nb = MultinomialNB(alpha=alpha, force_alpha=True).fit(X, labels)
    P = np.exp(nb.feature_log_prob_)             # P(word | class), rows: ham, spam
    i = list(vocab).index("money")
    j = list(vocab).index("meeting")
    print(f"\nalpha = {alpha:g}")
    print(f"  P(money | spam) = {P[1, i]:.3f}   P(money | ham) = {P[0, i]:.3f}")
    print(f"  P(meeting | spam) = {P[1, j]:.3f}   P(meeting | ham) = {P[0, j]:.3f}")
    msg = vec.transform(["free money meeting"])
    print("  'free money meeting' -> P(spam) =", nb.predict_proba(msg)[0, 1].round(3))

# A bigger synthetic corpus: who needs less data?
rng = np.random.default_rng(0)
spam_w = "free win prize money cheap offer click now deal bonus".split()
ham_w = "meeting noon project notes lunch report team call review plan".split()
both_w = "the a you your today please thanks hello time new".split()
def message(spam):
    own, other = (spam_w, ham_w) if spam else (ham_w, spam_w)
    words = list(rng.choice(own, 3)) + list(rng.choice(other, 1)) + list(rng.choice(both_w, 4))
    return " ".join(words)
y = (rng.random(4000) < 0.4).astype(int)
text = [message(s) for s in y]
Xall = CountVectorizer().fit_transform(text)
Xte, yte = Xall[3000:], y[3000:]
print("\ntrain size   naive Bayes   logistic regression")
for n in [10, 30, 100, 1000]:
    nb = MultinomialNB().fit(Xall[:n], y[:n])
    lr = LogisticRegression(max_iter=1000).fit(Xall[:n], y[:n])
    print(f"{n:>10}   {nb.score(Xte, yte):.3f}         {lr.score(Xte, yte):.3f}")

Output

vocabulary size: 15

alpha = 1
  P(money | spam) = 0.115   P(money | ham) = 0.040
  P(meeting | spam) = 0.038   P(meeting | ham) = 0.120
  'free money meeting' -> P(spam) = 0.727

alpha = 1e-10
  P(money | spam) = 0.182   P(money | ham) = 0.000
  P(meeting | spam) = 0.000   P(meeting | ham) = 0.200
  'free money meeting' -> P(spam) = 1.0

train size   naive Bayes   logistic regression
        10   0.692         0.659
        30   0.818         0.796
       100   0.948         0.998
      1000   1.000         1.000

With smoothing (alpha = 1), the probabilities match the hand calculation and "free money meeting" scores 0.727. With smoothing effectively off (alpha close to 0), the probabilities contain hard zeros and the score jumps to 1.0: the answer is decided by which class has fewer never-seen words, not by the evidence.

The second part trains both models on a synthetic corpus. With 10 or 30 messages, naive Bayes is ahead (0.692 against 0.659, then 0.818 against 0.796). By 100 messages logistic regression has overtaken it, 0.998 against 0.948. This is the typical pattern: naive Bayes learns fast from little data, and a discriminative model catches up and passes it as data grows, because it is not limited by the independence assumption.

Variants for different features

  • Multinomial NB: word counts. The standard for text.
  • Bernoulli NB: word present or absent. Works for short texts.
  • Gaussian NB: continuous features, each modelled as a normal distribution per class.

Why it works despite being wrong

The independence assumption double-counts correlated evidence ("free" and "prize" both fire), which pushes probabilities toward 0 and 1. So naive Bayes probabilities are badly calibrated ( Probability calibration ). But classification only needs the right class to win, and the ranking of classes is often still correct even when the probabilities are exaggerated.

Trade-offs

OptionWins whenLoses when
Naive Bayes Text with many features, few labels, or a need to train in seconds and update online You need trustworthy probabilities, or features interact strongly
Logistic regression Enough data; calibrated-ish probabilities; correlated features Very little labelled data
Fine-tuned language model Meaning matters more than keywords; large budgets Tight latency, tiny budgets, or a need for full transparency

Common mistakes

  • Forgetting smoothing, so one unseen word zeroes out a class.
  • Multiplying raw probabilities instead of summing logs, and getting underflow to 0.
  • Using naive Bayes probabilities as real risks; they are usually far too confident.
  • Fitting the vectorizer's vocabulary on all data before splitting, leaking test words into training.
  • Using Gaussian NB on counts or Multinomial NB on negative continuous values.

In the wild

Naive Bayes was one of the first widely used statistical spam filters, and Bayesian filtering in mail software followed the same recipe: count words per class, smooth, combine log-likelihoods. It remains a common first baseline for text classification such as support ticket routing, document categorisation and sentiment, because it trains in one pass, updates incrementally as new labelled messages arrive, and handles vocabularies of hundreds of thousands of words cheaply.

Check yourself

  1. What exactly does the 'naive' assumption say?

    Show answer

    B. Features are independent of each other once the class is known Conditional independence given the class lets the joint likelihood factor into a product of single-feature likelihoods.

  2. The word 'invoice' never appeared in spam during training. Without smoothing, what happens to an email containing it?

    Show answer

    B. Its spam probability becomes exactly 0, whatever the other words say P(invoice | spam) = 0 makes the whole product for spam zero. Laplace smoothing gives unseen words a small nonzero probability instead.

  3. With 20 labelled tickets, naive Bayes beats logistic regression. With 50,000 it loses. Why?

    Show answer

    B. NB's strong assumption helps with little data, but limits it once data can support a more flexible model Strong assumptions reduce variance when data is scarce. With plenty of data, the bias from the false independence assumption dominates.

  4. Why do implementations sum log-probabilities instead of multiplying probabilities?

    Show answer

    B. Multiplying hundreds of small numbers underflows to 0 in floating point log(a x b) = log a + log b gives the same ranking without underflow. It also shows naive Bayes is linear in log space.

Recap

  • Naive Bayes applies Bayes' theorem and assumes features are independent given the class.
  • Training is counting: priors from class shares, likelihoods from per-class feature frequencies.
  • Laplace smoothing adds a pseudo-count so unseen words never force a probability to zero.
  • In log space it is a linear model; it learns fast from little data but its probabilities are overconfident.
  • It remains a strong, cheap baseline for text classification such as spam filtering.

Get new lessons by email

Get new posts on backend and AI internals. No spam, unsubscribe anytime.

Your email is only used to send new posts. Privacy

Next up