Why are text inputs converted into embeddings before entering a neural network?

Why are text inputs converted into embeddings before entering a neural network?

Text inputs are converted into embeddings so an LLM can work with useful numerical representations instead of treating vocabulary IDs as meaningful measurements. More precisely, the embedding lookup is typically a layer inside the neural network; it prepares representations for the later processing layers.

Token numbers are only labels

Suppose a toy tokenizer assigns “cat” ID 8 and “mountain” ID 9. Feeding those integers as ordinary magnitudes would suggest an ordering or distance that has no semantic justification. The ID difference of one does not mean the concepts are almost the same.

An embedding table instead assigns each token a vector of trainable values. The network uses the vector’s learned structure, while the integer remains a reliable index for selecting it.

Dense representations support learning

A one-hot vector is another way to identify a token: one position is active and every other position is zero. An embedding lookup can be understood as efficiently selecting the same learned row that a one-hot input would select through a matrix operation.

The resulting dense vector can be combined with other vectors through the model’s layers. Training adjusts its values in ways that help the full system perform its objective. PyTorch documents this lookup behavior in its Embedding reference.

Context comes from later computation

The initial embedding for a token is generally fixed for a given model state. Later layers incorporate surrounding information and position to create contextual representations. This is how a model can process the same vocabulary item differently across sentences without requiring a new token ID for every possible use.