What is a token ID?

What is a token ID?

A token ID is an integer that identifies an entry in a tokenizer’s vocabulary. It allows software to represent a sequence of text units as a sequence of numbers. The number is an index, not a definition or a measure of how important the token is.

Think of an ID as a catalog number

In a toy vocabulary, “river” might have ID 18 and “window” might have ID 19. Their neighboring numbers would not imply that rivers and windows have similar meanings. Those numbers are only illustrative: real values must be obtained from the specific tokenizer being used.

Likewise, the same visible text can have different IDs in different vocabularies. Passing one model’s token IDs to another model without a compatible tokenizer can change the interpreted input completely.

From identifier to embedding

A model typically uses the ID to select a row from a learned embedding matrix. That row contains a vector of numerical values used by the neural network. Training can change those vector values while leaving the token’s ID unchanged.

This distinction explains why adding 1 to a token ID does not move to a slightly different meaning. Meaning-related behavior emerges from learned representations and the model’s subsequent computations.

IDs also appear in output

When a model selects an output token, its ID can be passed to the matching decoder to recover text. Some IDs correspond to special markers rather than ordinary language. Hugging Face’s tokenizer reference documents how its tokenizer interfaces convert text and token IDs.