Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Tokenization and Normalization

The chapter opener showed a bookshop returning nothing for “car repair” because “automobile repair” is stored under a different token. That failure begins at the very first stage of the retrieval pipeline: how raw text is split into tokens and normalized before any matching happens. This section walks through that stage, starting with the whitespace-and-punctuation splitting from 1 - Classical Text Retrieval and ending with a rule-based language detector that decides which downstream pipeline to invoke.

The naive tokenizer and where it breaks

Chapter 1 split text at whitespace and punctuation boundaries. The simplest rendering of that approach is a one-line tokenizer:

def word_tokenize(text: str) -> list[str]:
    text = re.sub(r'[^\w\-]+', ' ', text)
    return [t for t in text.split(' ') if t]

It substitutes any non-word character with a space, then splits on whitespace. It works on well-behaved English prose. Give it a sentence that exercises real-world text and it fails in several distinct ways at once:

I buy my parents’ 10% of U.K. startup for $1.4 billion. Dr. Watson’s cat called Mrs. Hersley and it was w.r.o.n.g., more to come ...

Running the naive tokenizer produces:

['I', 'buy', 'my', 'parents', '10', 'of', 'U', 'K', 'startup',
 'for', '1', '4', 'billion', 'Dr', 'Watson', 's', 'cat', 'called',
 'Mrs', 'Hersley', 'and', 'it', 'was', 'w', 'r', 'o', 'n', 'g',
 'more', 'to', 'come']

Four systematic problems appear:

For a retrieval-only pipeline over English, this behaviour is not fatal: users rarely search for “U.K.” with the period, and possessive stripping helps recall. But the distinctions are gone for good, and any task that needs sentence structure or numeric precision needs a better tokenizer up front.

Modern tokenization with nltk and spaCy

Both nltk and spaCy ship word tokenizers built from rules combined with abbreviation and exception lists learned from text corpora. Running each on the demonstration sentence:

import nltk
import spacy

sentence = ("I buy my parents' 10% of U.K. startup for $1.4 billion. "
            "Dr. Watson's cat called Mrs. Hersley and it was w.r.o.n.g., more to come ...")

nltk.word_tokenize(sentence)
# ['I', 'buy', 'my', 'parents', "'", '10', '%', 'of', 'U.K.', 'startup', 'for',
#  '$', '1.4', 'billion', '.', 'Dr.', 'Watson', "'s", 'cat', 'called', 'Mrs.',
#  'Hersley', 'and', 'it', 'was', 'w.r.o.n.g.', ',', 'more', 'to', 'come', '...']

[t.text for t in spacy.load('en_core_web_sm')(sentence)]
# ['I', 'buy', 'my', 'parents', "'", '10', '%', 'of', 'U.K.', 'startup', 'for',
#  '$', '1.4', 'billion', '.', 'Dr.', 'Watson', "'s", 'cat', 'called', 'Mrs.',
#  'Hersley', 'and', 'it', 'was', 'w.r.o.n.g', '.', ',', 'more', 'to', 'come', '...']

The two outputs agree on every token except one: nltk treats w.r.o.n.g. as a single abbreviation, spaCy splits the trailing period as sentence-terminating punctuation. On real text the two libraries usually agree; artificial or novel abbreviations are where they diverge.

Compared with the naive regex, the trained tokenizers fix several of the earlier failures but not all of them, and the two possessive shapes still behave differently:

For retrieval, we still want a leaner token list than the tokenizer alone provides. The standard cleanup after tokenization is:

  1. Drop single-letter tokens and pure punctuation tokens.

  2. Drop numbers and currency tokens unless the collection is numeric (a scientific-paper index would keep them).

  3. Drop possessive tokens ("'s") once the possessive information has been used.

The chapter’s demo notebook composes these three cleanup steps on top of a regex tokenizer and compares the intermediate outputs side by side.

Case, Unicode, and accents

A tokenizer separates words. Normalization decides which surface forms of the same word collapse to the same token. Consider three variants of one city name:

Zurich, Zürich, ZURICH

They refer to the same city. To match a query for Zurich against a document that spells the city Zürich, the index has to normalize both spellings to a shared form before storing them. Three transforms handle almost all cases in European text:

After all three, the three variants above become zurich. Users can type any spelling and match any document.

In practice, modern text-processing stacks apply Unicode normalization by default because it has no downside, case-fold when the retrieval scenario is case-insensitive (most web search), and fold accents only when the user population cannot easily type them.

Sentence segmentation

Some tasks need whole sentences as input, not tokens. Part-of-speech taggers work sentence by sentence because word ambiguity resolves in context. RAG systems (see the retrieval-augmented generation chapter) build larger chunks by grouping consecutive sentences until they hit a size limit. Both need a reliable answer to the question: where does one sentence end and the next begin?

Splitting on . is the naive approach and fails on the same abbreviations we saw earlier. In “Dr. Watson’s cat called Mrs. Hersley.”, the periods after Dr and Mrs are not sentence boundaries; only the trailing period is. Multi-line quoted dialogue, ellipses, and decimal numbers add more edge cases.

Two libraries handle this well:

from nltk.tokenize import sent_tokenize
sent_tokenize("Dr. Watson's cat called Mrs. Hersley. She was Egyptian.")
# ["Dr. Watson's cat called Mrs. Hersley.", "She was Egyptian."]

Sentence segmentation is one of the few pieces of a classical pipeline that stayed useful into the LLM era. Every RAG system built on top of embeddings still needs to decide where its chunks begin and end, and sentence boundaries are the most reliable low-cost signal for that decision.

Word boundaries are not always spaces

Not every writing system uses spaces to separate words, and not every “word” in a modern document is meant to be a single token. Both cases break the “split on whitespace” assumption at the root.

Scriptio continua languages, including Chinese, Japanese, Thai, and classical Greek, do not use spaces at all. A single sentence is a continuous string of characters:

莎拉波娃现在居住在美国东南部的佛罗里达。

莎拉波娃    现在    居住   在    美国    东南部       的      佛罗里达
Sharapova  now     lives  in   US     southeast    in      Florida

Code identifiers pack multiple words into a single token by convention: QueryParser, assertEquals, word_tokenize. A code-search index that stores only whole identifiers cannot answer a query for parser.

Spoken-language transcripts come from a continuous phoneme stream. Speakers do not pause between words; the pauses that exist mark breath, hesitation, or emphasis, and rarely align with word boundaries. Even the pause at a sentence end is unreliable in fluent speech. The segmenter has to infer word boundaries from the phoneme sequence itself, aided by a pronunciation dictionary and a language model, and non-native pronunciation or unclear articulation shifts where those boundaries land.

There are two ways to break these continuous streams into useful tokens:

This workaround for unsegmentable text turned out to be a foundational idea. Byte Pair Encoding (BPE) and WordPiece take the same principle (match parts of words when whole-word matching fails) and learn the token vocabulary from a corpus instead of fixing it to trigrams. Both power today’s large language models, and the semantic-search chapter covers them in detail.

Language detection

Both indexing documents and understanding queries depend on knowing what language the input is in. A German document benefits from Snowball’s German rules and the German stop-word list; running it through the English pipeline strips accents but leaves compounds and inflections untouched. A query like “Bücher von Goethe” from the chapter opener needs to be recognized as German before the retrieval system can restrict results to German-language books or apply German-specific normalization.

For long documents, a small number of hand-crafted rules is enough:

For document-side indexing this is usually enough: a paragraph or a page provides many opportunities for the rules to fire, and the winning language emerges quickly. Once the language is known, the pipeline branches: a German document goes to the German Snowball stemmer, an English document to the English WordNet lemmatizer, and so on.

Short queries break the rules, though. “Mein computer” mixes German with an English loanword; “pain” is a French word for bread and an English word for suffering. Neither has enough evidence for the rules to fire reliably. That regime needs a statistical classifier over character n-grams, developed with Naive Bayes in Intent Routing and Classification.

Where this sits in modern retrieval

Tokenization and normalization are the least glamorous part of the retrieval stack and still the largest source of quality issues in production. Modern lexical retrievers like Lucene, Solr, and Elasticsearch expose the whole pipeline as a chain of configurable analyzers: a tokenizer, a lowercase filter, a Unicode-normalization filter, a stop-word filter, a stemmer, sometimes a compound splitter. Dense retrieval sidesteps most of these choices by learning them implicitly in the embedding model, but hybrid stacks (BM25 combined with dense retrieval) still need the lexical side, and the lexical side still starts with these steps. Getting them wrong at ingestion time is expensive to fix later because the index has to be rebuilt.

The next section takes tokens as given and looks at the second half of the recall gap: collapsing “car”, “cars”, “carrying”, and “carried” so a document mentioning any of them matches a query for any other.

Part I: Foundations
3 - Advanced Text Processing
Part I: Foundations
Stemming and Lemmatization