How are embeddings learned during model training?

How are embeddings learned during model training?

Embeddings are commonly learned by adjusting numerical values so that a model becomes better at a training objective. The values may begin with an initialization that contains no useful task-specific structure, or they may start from an already trained model. Repeated updates shape representations that support the task.

A prediction creates a learning signal

Consider a toy language model trained on “The gardener watered the plants.” It uses the preceding text to predict a later token and compares its prediction with the observed target. A loss function expresses how poorly the prediction fits that target.

Backpropagation computes how changes in trainable parameters would affect the loss. An optimizer then updates parameters, including embedding values when those values are trainable. Across many examples, this can produce representations that help the model use language patterns more effectively.

Different objectives teach different relationships

A retrieval model may be trained to give a query and a relevant passage compatible representations while distinguishing irrelevant passages. A recommendation model may instead learn from user-item interactions. Both produce embeddings, but the meaning of closeness follows the objective and data.

Training does not require a person to assign a definition to every coordinate. Useful structure can emerge from optimizing performance across examples.

Learning is different from using an embedding

At inference time, generating a vector normally uses fixed model parameters. A user submitting a new sentence does not automatically retrain the encoder. Some systems support later fine-tuning or online learning, but that is an explicit additional process. When embeddings change after a model update, stored search vectors may need to be regenerated for consistent comparison.