What is a token in artificial intelligence?

What is a token in artificial intelligence?

A token is a unit that a model uses to represent input or output. For text LLMs, it may correspond to a complete word, part of a word, punctuation, whitespace-related text, or a byte sequence. It is a computational unit, so it does not have to match what a reader considers a word.

A token is a vocabulary item

Imagine a toy tokenizer whose vocabulary contains “rain,” “coat,” and “!”. It could represent “raincoat!” using those three units. A different tokenizer might include “raincoat” as one vocabulary item and produce only two. These are illustrative splits, not claims about any particular model.

Tokens can also serve structural purposes. Some tokenizers reserve special items to mark boundaries, padding, or the end of generated text. A special token’s role comes from how the model and application were designed to use it.

Tokens become model inputs

A tokenizer associates each vocabulary item with an integer ID. An embedding layer then maps the ID to a vector that the neural network can process. The token, its ID, and its embedding are related, but they are different objects.

Why token counts matter

Context limits and generation limits are often expressed in tokens. Tokenization therefore affects how much material fits in a request and how much output a model can produce. Languages, code, emoji, and unusual names can have different tokenization patterns. Use the intended model’s tokenizer for a reliable count instead of assuming that every word costs one token.