What is tokenization, and why is it necessary for LLMs?

What is tokenization, and why is it necessary for LLMs?

Tokenization converts text into units that a language model can represent and process. In common text LLMs, it provides the bridge between flexible human writing and a fixed vocabulary of model inputs. It also gives generated outputs a way to return to readable text.

Balancing vocabulary and sequence length

A whole-word vocabulary would need to handle spelling variations, names, technical terms, and newly invented words. A character-only representation keeps the vocabulary smaller but usually produces longer sequences. Subword tokenization offers a compromise by reusing pieces across many words.

For example, a toy tokenizer could represent “repainting” using “re,” “paint,” and “ing.” Those pieces could also appear in other words. A real tokenizer may choose different boundaries because its vocabulary and rules were learned or configured differently.

Preparing input without interpreting it

Tokenization identifies text units and maps them to IDs. It does not by itself decide whether an argument is correct or understand the author’s intention. Contextual interpretation happens through the model’s learned numerical computations.

Some pipelines normalize text, apply initial splitting rules, segment those pieces, and add special markers. The exact sequence is tokenizer-specific, as described in the Hugging Face tokenization pipeline.

Why the model and tokenizer must match

A model’s embedding table expects the vocabulary used during its training. Replacing the tokenizer can alter the meaning of every input index, even when the original text stays unchanged. For applications, the practical rule is to count, encode, and decode text with a compatible tokenizer rather than treating tokenization as an interchangeable text-cleaning step.