How does a loss function measure the quality of a model’s prediction?

How does a loss function measure the quality of a model’s prediction?

A loss function assigns a numerical penalty to a prediction according to a chosen objective. Training uses that number to compare possible parameter settings. Lower loss generally means the prediction better matches the training target under that particular rule.

Different tasks need different losses

A model predicting delivery time might use squared error. If it predicts 40 minutes and the observed time is 30, the squared error is 100. Larger deviations receive increasingly large penalties.

A next-token language model usually uses cross-entropy. For a single target token, the loss depends on the negative logarithm of the probability assigned to that token.

For example, assigning the correct token probability 0.8 gives a smaller penalty than assigning it 0.1. This encourages the model to place more probability on observed continuations.

A training signal across many examples

Training commonly averages or otherwise aggregates losses across positions and examples. The resulting objective provides a direction for gradient-based updates.

One sentence can have several reasonable continuations, while a dataset records only the continuation that occurred. The objective measures fit to those observations rather than a universal judgment about all possible responses.

Loss and user value are different measures

A lower language-model loss can accompany better prediction, but it does not independently prove factual accuracy, helpfulness, or fairness.

Engineers therefore combine loss monitoring with evaluations of the behavior they actually need. PyTorch's optimization tutorial illustrates how loss, backpropagation, and optimizer steps form a training loop.