What is a forward pass in neural-network training?

What is a forward pass in neural-network training?

A forward pass is the computation that carries an input through a neural network to produce an output. It uses the current parameter values. During training, that output is used to calculate a loss; the forward pass alone does not update the learned weights.

From input to prediction

Consider a model that classifies a short message as positive or negative. The text becomes token IDs, which are mapped to vectors. Layers then transform those vectors into representations suitable for the task.

A final layer produces scores for the possible categories. Those scores can be converted into probabilities, such as 0.8 for positive and 0.2 for negative. The exact operations depend on the model architecture.

Why intermediate values matter

The temporary values produced inside layers are called activations. They change with the input, even when the parameters remain fixed.

During gradient-based training, a framework typically retains information from the forward computation so it can calculate derivatives later. Memory-saving methods may instead recompute some intermediate values during the backward pass.

For a language model, training can compute next-token predictions for many positions in a sequence within a single forward computation, while preserving the causal restriction on future context.

How the training cycle continues

The loss compares the output with training targets. The backward pass computes gradients, and the optimizer applies parameter updates.

Inference also uses forward computations, but normally skips gradient calculation and training updates. This distinction explains how the same model can support both learning from examples and answering requests.