How does the learning rate affect gradient descent and model training?

How does the learning rate affect gradient descent and model training?

The learning rate controls the scale of parameter updates during optimization. In basic gradient descent, the optimizer multiplies the gradient by the learning rate before subtracting it from the current parameter value.

It is a hyperparameter chosen or scheduled by the training process, rather than an ordinary model weight learned directly from prediction examples.

Steps that are too large or too small

Imagine minimizing the simple loss (weight − 3)². Starting with weight 0, the gradient is −6. With learning rate 0.1, the next weight is 0.6, moving toward the minimum at 3.

With learning rate 1, the next weight is 6. Another update returns it to 0, so this example oscillates instead of settling. An extremely small learning rate would move in the right direction very slowly.

Why schedules are useful

Training can use larger updates during one phase and smaller updates during another. A warmup period gradually increases the rate at the beginning, while a decay schedule reduces it later.

Different optimizers interpret their settings differently, so a value that works well in one training setup may fail in another. Batch size, architecture, precision, and data also influence stability.

Choose by measured behavior

Watch training loss, validation performance, and signs of unstable updates. Compare learning-rate choices under similar conditions instead of judging only the earliest loss decrease.

Stanford's optimization notes discuss learning-rate behavior and update methods. No single rate guarantees fast convergence or good generalization across every model.