Why does one word not always equal one token?

Why does one word not always equal one token?

A word is a linguistic unit, while a token is a unit chosen for a model’s representation. Their boundaries sometimes coincide, but there is no rule requiring one token per word. Tokenizers are designed around a vocabulary and segmentation strategy, not around a universal dictionary of words.

Common patterns influence the vocabulary

Frequently occurring pieces may be stored as single tokens. An unfamiliar name or longer word may need several smaller pieces. In an illustrative vocabulary, “sunlight” could be one token, while “sunlightproof” could require “sunlight” and “proof.” A real tokenizer might split both examples differently.

Capitalization, punctuation, and neighboring spaces can also affect the selected units. The letters in “Lake,” “lake,” and “ lake” need not produce identical tokens. This makes rough estimates useful only as estimates.

Languages do not share one word-to-token ratio

Writing systems vary, and some languages do not mark word boundaries with spaces. Tokenizer training data and vocabulary allocation also vary. A short phrase in one language may therefore require more tokens than a similarly sized phrase in another.

Code adds further variation through indentation, symbols, unusual identifiers, and repeated punctuation. Counting only space-separated words misses much of that structure.

Measure the text you will actually send

When a prompt must fit a context limit, count the formatted request with the intended tokenizer. Include relevant conversation history and special formatting where the application exposes them. If you shorten a document, preserve important meaning and verify the final token count instead of assuming a fixed reduction for every word removed.