Text preprocessing and tokenization basics

The big idea

A model cannot read. "The food was great" has to become numbers before any classifier, clustering method or neural network can touch it. The first step is deciding what the units are and how much of the original text's mess to keep. TokenizationSplitting text into units called tokens, such as words, pieces of words or characters, which are then mapped to integer ids. splits text into units, and preprocessing cleans the text before or after that split.

Think of sorting a pile of handwritten shopping lists to count what people buy. You would treat "Milk", "milk" and "MILK!" as the same item, you might ignore "and" and "some", and you would count "apples" and "apple" together. Each of those choices makes counting easier, and each can throw away something that mattered. That trade-off runs through this whole lesson.

How it works

Step 1: Normalize

NormalizationMaking different spellings of the same text identical before tokenizing, for example by lowercasing, unifying Unicode forms and removing punctuation. makes text that should match actually match. Typical steps:

• Lowercase, so "Apple" and "apple" are one token. But this merges Apple the company with the fruit, and "US" with "us".
• Unicode normalization. The same visible character can be stored in several ways: an accented e can be one code point or an e plus a combining accent, and the "fi" ligature is a single character. Python's unicodedata.normalize("NFKC", text) folds many of these into one form.
• Punctuation, numbers, URLs: remove, keep, or replace with a placeholder. For sentiment, "!!" may be signal; for topic labelling, it is noise.

Step 2: Tokenize into words

The simplest tokenizer splits on whitespace and punctuation. Real text is harder than it looks: "don't", "e-mail", "New York", "3.5" and emoji all need decisions. scikit-learn's default tokenizer keeps tokens of 2 or more letters or digits, so the word "a" and the "I" in "I loved it" silently disappear.

Each distinct token goes into a vocabularyThe fixed set of tokens a model knows, each mapped to an integer id. and gets an integer id. A word-level vocabulary has a hard problem: any word not seen in training, a typo, a new product name, becomes an unknown token, usually written [UNK], and its meaning is lost.

Step 3: Stop words

Stop wordsVery frequent words such as the, and or of that are often removed before counting because they carry little topic information. are frequent words like "the", "is" and "of". Removing them shrinks the data and stops them from dominating word counts in topic-style tasks. But scikit-learn's own documentation warns that its built-in English list has several known issues and that popular stop lists can include words that are informative for some tasks. The code below shows the trap: that list contains "not" and "five", so "NOT open, five stars" loses both its negation and its rating. Never remove stop words for sentiment without checking the list.

Step 4: Stemming or lemmatization

Both collapse word forms so that "connect", "connected" and "connections" count as one feature.

• StemmingCutting words down to a common stem with suffix-stripping rules, without a dictionary; the stem need not be a real word. applies suffix-stripping rules. The classic Porter stemmer (1980) conflates CONNECT, CONNECTED, CONNECTING, CONNECTION and CONNECTIONS into one stem. It is fast and needs no dictionary, but stems need not be words ("studies" becomes "studi"), and Porter himself notes that conflating words like RELATE and RELATIVITY may be unreasonable for some collections.
• LemmatizationReducing a word to its dictionary form (lemma) using a vocabulary and the word's part of speech, for example mice to mouse and better to good. returns the dictionary form, using a lexicon and the word's part of speech: "mice" to "mouse", "ran" to "run", "better" (as an adjective) to "good". It is more accurate and slower, and it needs language resources.

Going deeper

Code: the pipeline on one sentence

import re, unicodedata
from sklearn.feature_extraction.text import CountVectorizer, ENGLISH_STOP_WORDS

# 1. Normalization: Unicode NFKC, lowercase, strip accents (each step is a choice)
raw = "The Café is NOT open ... five stars?! Visit 2x２ times"
nfkc = unicodedata.normalize("NFKC", raw)              # ligature fi -> f i, full-width 2 -> 2
no_acc = "".join(ch for ch in unicodedata.normalize("NFD", nfkc.lower()) if not unicodedata.combining(ch))
print("raw:       ", ascii(raw))
print("normalized:", ascii(no_acc))

# 2. Word tokenization: scikit-learn's default keeps tokens of 2+ word characters
tokens = CountVectorizer().build_analyzer()(no_acc)
print("tokens:    ", tokens)

# 3. Stop words: the built-in English list also removes "not"
print(f"stop list size {len(ENGLISH_STOP_WORDS)}, 'not' in list: {'not' in ENGLISH_STOP_WORDS}")
print("no stops:  ", [t for t in tokens if t not in ENGLISH_STOP_WORDS])

# 4. Stemming (rules, no dictionary) vs lemmatization (dictionary + part of speech)
def stem(w):                                           # a toy suffix stripper in Porter's spirit
    for suf, rep in [("sses", "ss"), ("ies", "i"), ("ions", ""), ("ion", ""), ("ing", ""), ("ed", ""), ("s", "")]:
        if w.endswith(suf) and len(w) - len(suf) >= 3:
            return w[: len(w) - len(suf)] + rep
    return w
LEMMA = {"connected": "connect", "connecting": "connect", "connections": "connection",   # a real lemmatizer
         "studies": "study", "studying": "study", "ran": "run", "running": "run",  # looks these up in
         "better": "good", "mice": "mouse"}                                         # a dictionary
for w in ["connected", "connecting", "connections", "studies", "studying", "running", "ran", "better", "mice"]:
    print(f"  {w:12} stem {stem(w):10} lemma {LEMMA.get(w, w)}")

# 5. Word vs subword tokens: a word vocabulary has no entry for a new word
corpus = "the model is believable . the results are unbelievable . we believe the model".split()
vocab = sorted(set(corpus))
sentence = "unbelievably believable results".split()
print("word ids:  ", [w if w in vocab else "[UNK]" for w in sentence])

PIECES = {"un", "believ", "able", "ably", "result", "s", "the", "model"} | set("abcdefghijklmnopqrstuvwxyz")
def subword(word):                                     # greedy longest match, as WordPiece does
    out, i = [], 0
    while i < len(word):
        j = max(j for j in range(i + 1, len(word) + 1) if word[i:j] in PIECES)
        out.append(word[i:j]); i = j
    return out
print("subwords:  ", [subword(w) for w in sentence])

raw:        'The Cafe\u0301 is NOT open ... \ufb01ve stars?! Visit 2x\uff12 times'
normalized: 'the cafe is not open ... five stars?! visit 2x2 times'
tokens:     ['the', 'cafe', 'is', 'not', 'open', 'five', 'stars', 'visit', '2x2', 'times']
stop list size 318, 'not' in list: True
no stops:   ['cafe', 'open', 'stars', 'visit', '2x2', 'times']
  connected    stem connect    lemma connect
  connecting   stem connect    lemma connect
  connections  stem connect    lemma connection
  studies      stem studi      lemma study
  studying     stem study      lemma study
  running      stem runn       lemma run
  ran          stem ran        lemma run
  better       stem better     lemma good
  mice         stem mice       lemma mouse
word ids:   ['[UNK]', 'believable', 'results']
subwords:   [['un', 'believ', 'ably'], ['believ', 'able'], ['result', 's']]

NFKC turns the ligature into "fi" and the full-width digit into a plain 2, and stripping the combining accent turns "Cafe" plus accent into "cafe". The tokenizer drops nothing important here, but the stop list removes "not" and "five". The toy stemmer maps the "connect" family to one stem but produces non-words ("studi", "runn"); the lemma lookup gives real words and handles irregular forms ("ran", "mice") that no suffix rule can.

Word, character or subword tokens

The last two lines show the problem with word tokens and its modern fix. The word "unbelievably" never appeared in the tiny corpus, so a word vocabulary maps it to [UNK]. A subword tokenizerTokenization that splits rare words into smaller frequent pieces, so any word can be represented without an unknown token. splits it into known pieces, "un" + "believ" + "ably", so the model still sees familiar parts.

• Word tokens: short sequences and readable features, but a huge vocabulary and no way to handle new words.
• Character tokens: tiny vocabulary and nothing is ever unknown, but sequences get long and each token means little.
• Subword tokens sit in between: frequent words stay whole and rare words split into pieces. Byte pair encoding was adapted for this in neural machine translation in 2015, to encode rare and unknown words as sequences of subword units. BERT uses WordPiece with a 30,000-token vocabulary and greedy longest-match splitting, the same idea as the code's subword function.

How BPE learns its merges is covered step by step in Tokenization .

Classic pipeline or modern model?

For linear models on counts ( Bag of words and TF-IDF ), aggressive normalization, stop words and stemming often help: they shrink the vocabulary and pool evidence. For pretrained neural models, do the opposite. Feed raw text to the model's own tokenizer: it was trained on unnormalized text, and its subword vocabulary already handles case, rare words and punctuation. Lowercasing or stemming before a pretrained model only removes information it knows how to use.

Trade-offs

Common mistakes

• Removing stop words for sentiment and losing "not", "no" and "never".
• Preprocessing training text one way and live text another, for example lowercasing only in the training notebook. Put preprocessing in the same pipeline object that serves predictions.
• Fitting the vocabulary on all data, including the test set ( Data leakage ).
• Stemming or lowercasing text before a pretrained transformer's tokenizer.
• Ignoring Unicode: the same word stored with different code points becomes two tokens.

In the wild

Search engines have long used stemming so that a query for "running shoes" also finds "run" and "runner", and libraries such as NLTK, spaCy and scikit-learn ship tokenizers, stop lists, stemmers and lemmatizers. Spam filters and ticket routers built on word counts use the classic pipeline. Every large language model starts with a learned subword tokenizer, which is why the same sentence can cost a different number of tokens in different models.

Check yourself

Recap