Learning Rate

A hyperparameter that controls how much the model changes its weights in response to the estimated error each time the model weights are updated.

Learning Rate Scheduler

A Scheduler is an algorithm that changes the learning rate during training.

Types of Learning Rate Schedulers

Step Decay

The LR is constant for a few epochs, then drops by a factor (e.g., 0.1) suddenly.

• Mechanism: “Every 30 epochs, cut LR in half.”
• Best For: Simple computer vision tasks (ResNets) trained with SGD.

Cosine Annealing

The LR decreases following a cosine curve (smooth wave). It starts high and drops rapidly to near zero by the end of training.

• Mechanism: $L{R}_t={\eta }_{min}+\frac{1}{2}({\eta }_{max}-{\eta }_{min})(1+\cos (\frac{t}{T}\pi ))$
• Best For: Modern Deep Learning. It is very popular because it doesn’t require tuning “step” milestones like Step Decay does.

ReduceLROnPlateau

(The “Patience” Scheduler) This is reactive rather than pre-planned. It watches validation loss. If the loss stops going down for a set number of epochs (patience), it slashes the LR.

• Mechanism: “If val_loss doesn’t improve for 10 epochs, multiply LR by 0.1.”
• Best For: When you don’t know how long training will take.

Warmup + Linear Decay

The LR starts at 0 and grows linearly to a peak (warmup), then decays linearly to 0.

• Best For: Transformers (BERT, GPT). Transformers are very unstable at the start of training; “warming up” the LR prevents the gradients from exploding in the first few batches.