A regularization technique that gradually reduces the magnitude of a model’s weights during training. It works by subtracting a small fraction of the current weights from themselves at each update step, effectively nudging parameters toward zero.
The primary goal of weight decay is to improve a model’s ability to generalize to new, unseen data by controlling its complexity. By penalizing large weights, it discourages the model from “memorizing” noise or specific details in the training set that do not exist in the broader population.
Smaller weights typically result in smoother mathematical functions that are less sensitive to small changes in input, leading to more robust predictions.
Also helps avoid exploding gradients