Bag of words and TF-IDF

The big idea

You have 50,000 support tickets and want to route each one to billing, shipping or tech support. The simplest useful representation of a ticket ignores grammar and word order completely and just records which words appear and how often. That is the bag of wordsA document representation that counts how often each vocabulary word appears, ignoring word order.: tip the words of a document into a bag, shake it, and count what is inside.

It sounds too crude to work, yet a ticket containing "refund", "charged" and "invoice" is about billing whatever the order of those words. The catch is that raw counts are dominated by words like "the" and "please", which appear everywhere and say nothing. TF-IDF fixes that by boosting words that are frequent in this document but rare in the collection, the way a good editor highlights the words that make a document different from all the others.

How it works

Step 1: The document-term matrix

Build a vocabulary of every term in the training documents ( Text preprocessing and tokenization basics ). Each document becomes a row vector with one entry per term: the number of times that term occurs. Stack the rows and you have a document-term matrixA matrix with one row per document and one column per vocabulary term; each entry counts, or weights, that term in that document..

Take three documents: "the cat sat on the mat", "the dog sat on the log" and "the cat chased the dog". The vocabulary has 8 terms (cat, chased, dog, log, mat, on, sat, the), so each document is a vector of length 8. The first one is [1, 0, 0, 0, 1, 1, 1, 2]: "the" twice, five other words once, three words zero times.

Real vocabularies have tens or hundreds of thousands of terms and each document uses only a few hundred, so almost every entry is zero. Libraries store the matrix as a sparse matrixA matrix format that stores only the non-zero entries and their positions, saving memory when almost all entries are zero. that keeps only the non-zero entries.

Step 2: Term frequency and document frequency

The count of a term in one document is its term frequencyHow often a term occurs in one document; the TF in TF-IDF. (tf). On its own it rewards common words: "the" has the highest count in all three example documents. What we want to know is how much a word tells us about this document, compared with the others. The document frequencyThe number of documents in the collection that contain a term at least once. (df) of a term is the number of documents that contain it. "the" has df = 3 out of 3; "mat" has df = 1.

Step 3: Inverse document frequency

IDFInverse document frequency: a weight that is high for terms that occur in few documents and low for terms that occur in most of them, typically log(N / df). turns df into a weight that is large for rare terms and small for common ones. The classic form is

where N is the number of documents. With N = 3: "the" gets ln(3 / 3) = 0, so it vanishes; "cat", "sat" and "on" (df = 2) get ln 1.5 = 0.41; "mat" (df = 1) gets ln 3 = 1.10. In document 1, "mat" now has the highest weight: it is the word that sets this document apart.

Why a logarithm? Without it, a term in 1 document would weigh 1,000 times more than a term in all of them, and a single rare typo would swamp everything. The log compresses that range.

Step 4: Variants you will meet

TF-IDF is a family, not one formula. scikit-learn's TfidfVectorizer by default uses a smoothed idf, ln((1 + N) / (1 + df)) + 1, which acts as if one extra document contained every term, so no idf is ever 0 and terms in every document keep a weight of 1. It then scales each document vector to length 1 (L2 normalization), so long and short documents become comparable. Other common options are sublinear tf (1 + ln tf), so the tenth "refund" counts less than the first, and n-grams: with ngram_range=(1, 2) pairs of adjacent words such as "not good" become features too, which brings back a little word order.

Try it

Try this:

1. Switch to raw counts: "the" is the biggest number in every row. Switch back to TF-IDF: its idf is 0 and the gold outline moves to "mat", "log" and "chased".
2. Choose scikit-learn default. "the" is back with weight above 0 because the smoothed idf never reaches 0. In document 1 it even has the largest weight, because it occurs twice. What does that tell you about removing stop words or using sublinear tf?
3. Edit document 3 to "the mat" and watch "mat" lose weight in document 1: its df went from 1 to 2.

Going deeper

Code: TF-IDF by hand and a tiny classifier

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression

docs = ["the cat sat on the mat", "the dog sat on the log", "the cat chased the dog"]
cv = CountVectorizer(); X = cv.fit_transform(docs)          # sparse document-term matrix
print("terms:", list(cv.get_feature_names_out()))
print("counts:\n", X.toarray())

# TF-IDF by hand (scikit-learn's default) and with the library
n, df = len(docs), (X.toarray() > 0).sum(0)
idf = np.log((1 + n) / (1 + df)) + 1                        # smooth idf
W = X.toarray() * idf; W /= np.linalg.norm(W, axis=1, keepdims=True)
tv = TfidfVectorizer(); W_sk = tv.fit_transform(docs).toarray()
print("idf:", np.round(idf, 3), " matches library:", np.allclose(W, W_sk))
print("doc 1 weights:", {t: float(v) for t, v in zip(cv.get_feature_names_out(), np.round(W[0], 2))})

# A linear classifier on TF-IDF: 12 tiny reviews, 1 = positive
reviews = ["great food and friendly staff", "loved the pasta, great service", "friendly and fast, will return",
           "the best pizza in town", "great value, lovely staff", "fast service and tasty food",
           "cold food and rude staff", "slow service, never again", "the pizza was not good",
           "rude waiter and cold pasta", "not worth the price", "terrible, slow and expensive"]
y = np.array([1] * 6 + [0] * 6)
vec = TfidfVectorizer(); Xr = vec.fit_transform(reviews)
clf = LogisticRegression(C=10).fit(Xr, y)
w, names = clf.coef_[0], vec.get_feature_names_out()
print(f"\n{len(names)} features; with bigrams there would be",
      len(TfidfVectorizer(ngram_range=(1, 2)).fit(reviews).get_feature_names_out()))
print("most positive words:", list(names[np.argsort(w)[-4:][::-1]]))
print("most negative words:", list(names[np.argsort(w)[:4]]))
test = ["friendly staff and great pasta", "the food was not good", "not bad at all, great"]
for t, p in zip(test, clf.predict_proba(vec.transform(test))[:, 1]):
    print(f"  P(positive) {p:.2f}  {t}")

terms: ['cat', 'chased', 'dog', 'log', 'mat', 'on', 'sat', 'the']
counts:
 [[1 0 0 0 1 1 1 2]
 [0 0 1 1 0 1 1 2]
 [1 1 1 0 0 0 0 2]]
idf: [1.288 1.693 1.288 1.693 1.693 1.288 1.288 1.   ]  matches library: True
doc 1 weights: {'cat': 0.37, 'chased': 0.0, 'dog': 0.0, 'log': 0.0, 'mat': 0.49, 'on': 0.37, 'sat': 0.37, 'the': 0.58}

32 features; with bigrams there would be 75
most positive words: ['great', 'fast', 'friendly', 'town']
most negative words: ['cold', 'rude', 'slow', 'not']
  P(positive) 0.84  friendly staff and great pasta
  P(positive) 0.19  the food was not good
  P(positive) 0.51  not bad at all, great

The hand computation matches scikit-learn's TfidfVectorizer exactly, including the smoothed idf of 1.0 for "the" and the unit-length rows. The second half trains logistic regression ( Logistic regression ) on TF-IDF features of 12 short restaurant reviews. Its largest weights are readable: "great", "fast" and "friendly" push towards positive, "cold", "rude", "slow" and "not" towards negative. It scores "the food was not good" at 0.19 because "not" is a strong negative word. But it gives "not bad at all, great" only 0.51: it sees "not" and "great" and cannot know that "not bad" is praise. That is the bag-of-words blind spot, and with 12 examples bigrams are too sparse to fix it.

Why linear models work so well here

A TF-IDF vector has thousands of dimensions, and in such high-dimensional sparse spaces classes are often close to linearly separable. Logistic regression or a linear SVM ( Support Vector Machines and kernels ) trains in seconds on hundreds of thousands of documents, needs no GPU, and every weight maps to a word you can read and audit. That makes TF-IDF plus a linear model the baseline to beat before reaching for neural networks ( Learning curves and baselines ). The same vectors power search: rank documents by the cosine similarity between their TF-IDF vector and the query's. BM25, the scoring function behind classic search engines, is a refined TF-IDF.

Limits

Bag of words ignores order ("dog bites man" equals "man bites dog") and treats every word as unrelated to every other: "great" and "excellent" are as different as "great" and "terrible". Word embeddings ( Word embeddings ) attack the second problem, and sequence models attack the first.

Trade-offs

Common mistakes

• Fitting the vectorizer on all data, test set included. The vocabulary and idf leak test information; fit on training data only ( Data leakage ).
• Building the vocabulary again at prediction time, so column 57 means a different word in production than in training. Save and reuse the fitted vectorizer.
• Converting the sparse matrix to a dense array on a large corpus and running out of memory.
• Comparing documents of very different lengths without normalizing the vectors.
• Expecting bag of words to understand negation or word order.

In the wild

TF-IDF and its relative BM25 have long powered keyword search engines, and they still sit inside modern retrieval systems as the "sparse" half of hybrid search next to dense embeddings ( Hybrid search and reranking ). Spam filters, support-ticket routers, document deduplication and keyword extraction tools often run on TF-IDF vectors with a linear model, because they are fast, cheap and easy to explain.

Check yourself

Recap