Why do token IDs not contain the meanings of words?

Why do token IDs not contain the meanings of words?

Token IDs identify vocabulary entries; they do not encode a dictionary definition or an ordered scale of meaning. Their job is to make text units easy to look up. Meaning-related behavior comes from learned vectors and the computations that use them.

A number can label without describing

Imagine a library assigning book 52 to astronomy and book 53 to gardening. The neighboring labels do not make the subjects similar. In the same way, a tokenizer could assign neighboring IDs to unrelated pieces of text.

A consistent relabeling of vocabulary entries, together with the corresponding embedding and output mappings, would preserve what the system represents. The particular integer is less important than using that integer consistently throughout the pipeline.

Embeddings supply learned numerical structure

After tokenization, an ID selects a vector from the model’s embedding table. Training changes those vectors so that they help the model perform its learning task. Two tokens can have related learned representations even if their ID numbers are far apart.

Conversely, two adjacent IDs can select very different vectors. Subtracting their IDs therefore provides no useful semantic distance. Comparing appropriately trained vectors can be meaningful, but only in a compatible representation space and with a suitable similarity measure.

Context adds another layer

Even a learned token embedding is not a complete, context-sensitive definition. The same token can appear in jokes, technical explanations, and contradictory statements. Later network layers incorporate the available context to produce representations for those particular uses. Keeping identifiers, initial vectors, and contextual representations separate makes the text-processing pipeline much easier to reason about.