What is an embedding?

What is an embedding?

An embedding is a numerical representation that places an item in a vector space. In machine learning, embeddings are commonly learned so that the representation helps a model perform a task. The item might be a token, a sentence, a photograph, or a product in a catalog.

A vector shaped by its objective

Imagine a system trained to retrieve answers to support questions. It could learn to place “reset my password” near relevant account-recovery instructions. A system trained to recommend products might instead learn relationships based on customer behavior. Both use embeddings, but they organize information around different objectives.

An embedding’s individual coordinates usually do not have obvious labels. It is misleading to assume one coordinate always means “happiness” and another always means “size.” Useful information can be distributed across many values.

Token embeddings and document embeddings differ

Inside an LLM, a token ID commonly selects a vector from an embedding table. PyTorch’s Embedding documentation describes this lookup operation. The vector is then processed by further network layers.

A sentence or document embedding is typically produced by an encoder that combines information across text. It is often used for retrieval or clustering. Taking any internal LLM vector does not automatically produce a high-quality search embedding.

Similarity needs a compatible space

Embeddings can be compared using measures such as cosine similarity when the representation and task support that comparison. Vectors from unrelated models are not automatically comparable, even if they have the same number of components. Choose a compatible encoder and evaluate the relationships it actually captures.