What is the difference between a token ID and an embedding?

What is the difference between a token ID and an embedding?

A token ID is an integer label for a vocabulary entry. An embedding is a vector of numerical values used to represent an item in the model’s computations. The ID selects a representation; the embedding supplies the representation’s values.

Follow a toy lookup

Suppose a tokenizer assigns “cloud” the illustrative ID 27. A small example embedding table might map row 27 to [0.4, -0.2, 0.9]. The ID has one integer value, while this example embedding has three numerical components. Real models often use much larger vectors.

The values are not obtained by performing arithmetic on 27. They are parameters stored in the table and commonly learned during training. If training updates the row, the embedding can change while the token remains ID 27.

Identity and relationships are separate

IDs make lookup reliable. They do not place vocabulary items in meaningful numerical order. Embeddings can support useful relationships because their values are optimized as part of a learning process.

Even then, the embedding is not necessarily a complete dictionary definition. A token can have different roles in different sentences, and a model’s later layers combine its initial vector with the available context.

Keep the three stages distinct

The text unit is first identified by a token ID, then represented by an initial embedding, and finally transformed into contextual hidden representations. Confusing these stages leads to mistakes such as treating nearby ID numbers as similar words or assuming that one fixed lookup vector captures every possible meaning. For debugging, inspect each stage according to its own role.