Optimizer

An algorithm used to adjust the model’s parameters to minimize the Loss Function.

Types:

1. Gradient Descent

The most basic & common approach.

2. Momentum-Based Optimizers

To fix the oscillation of SGD, we add “momentum.” If the gradient points in the same direction repeatedly, the optimizer accelerates. If it changes direction, it slows down (dampening oscillation).

SGD with Momentum

It adds a fraction of the previous update vector to the current update.

• Formula:
  $v_t=\gamma v_{t-1}+\eta \nabla J(\theta )$

3. Adaptive Learning Rate Optimizers

Standard SGD uses the same Learning Rate for every single parameter. However, sparse features (rare words in NLP, rare chemical bonds in GNNs) need larger updates, while frequent features need smaller updates.

RMSprop (Root Mean Square Propagation)

Adapts the Learning Rate for each parameter by dividing the gradient by a running average of its recent magnitude.

• Effect: It penalizes weights with steep gradients (slowing them down) and boosts weights with small gradients.

Adam (Adaptive Moment Estimation)

The current industry standard. It combines the best of Momentum and RMSprop.

1. Momentum: Keeps a moving average of past gradients (first moment).
2. RMSprop: Keeps a moving average of past squared gradients (second moment).

• Pros: Works well with almost no tuning. Fast convergence.
• Cons: Can sometimes generalize worse than SGD for specific computer vision tasks.

AdamW (Adam with Weight Decay)

A variation of Adam that decouples Weight Decay (L2 regularization) from the gradient update.

• Use Case: This is the standard for training Transformers (e.g., GPT, BERT) and modern deep learning models, as it fixes a mathematical flaw in how the original Adam implementation handled regularization.

4. Second-Order Optimizers

Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS)

A popular quasi-Newton method particularly effective for optimizing large-scale problems

• How it works: It uses the Hessian matrix (second derivative) to estimate the curvature of the loss landscape, not just the slope.
• Pros: Extremely precise. It converges in far fewer steps than Adam/SGD.
• Cons: Extremely memory hungry and computationally expensive per step.
• Use Case: Small datasets, physics simulations, or finding the precise ground state of a molecule (geometry optimization). It is rarely used in large Deep Learning but is gold for scientific computing.

Step Size Calculation

$\text{Step Size}=\underset{\text{Scheduler}}{\underbrace{{\eta }_t}}\times \underset{\text{Optimizer}}{\underbrace{\frac{1}{\sqrt{v_t}+ϵ}}}$

• ${\eta }_t$: The Global Learning Rate at time t. This is what the LR Scheduler reduces (e.g., from 1e−3 to 1e−4).
• $1/\sqrt{(}{v}_t)$: The Adaptive Term. This is what Adaptive Learning Rate Optimizers calculate per parameter based on past gradient variance.

Practical Rule of Thumb: Start with Adam