How does a tokenizer divide text into smaller units?

How does a tokenizer divide text into smaller units?

A tokenizer divides text according to its configured processing rules and vocabulary. It does not simply read a sentence as a human would and mark each dictionary word. The same input can be divided differently by tokenizers trained for different models.

Preparing candidate pieces

Some tokenizers first normalize text, such as by applying a Unicode normalization rule. Others preserve distinctions that a different tokenizer might remove. An initial splitting stage can then separate text around spaces or punctuation, depending on the tokenizer design.

This is why developers should not assume every tokenizer lowercases text or discards extra spaces. Such changes can matter for code, names, and exact quotations.

Applying the learned vocabulary

A subword tokenizer uses its vocabulary and segmentation algorithm to produce final units. Byte-pair encoding, for example, builds larger units by learning useful combinations of smaller pieces. WordPiece and Unigram use different selection approaches. Hugging Face explains these distinctions in its tokenization algorithms guide.

Consider “moonwalking!” A toy vocabulary might produce “moon,” “walk,” “ing,” and “!”. Another could include “moonwalking” as one item. Neither split tells us whether the sentence is sensible; tokenization is about representation.

Returning usable model input

The tokenizer converts the selected pieces into IDs and may add special markers. Some interfaces also return offsets that connect token positions to spans in the original text. These offsets are useful for highlighting search results or extracting named entities. To inspect an actual split, run the exact text through the tokenizer paired with the target model.