Loss Function

In Machine Learning, a loss function is a mathematical function that quantifies the difference between a model’s predicted output and the actual true values.

The primary goal during model training is to minimize this loss value by changing model parameters (see Gradient Descent and Backpropogation).

Types:

1. Regression Loss Functions

Used when predicting continuous numbers

Mean Squared Error (MSE) / L2 Loss

Default loss function for most regression problems. It squares the difference between the prediction and the target.
$MSE=\frac{1}{n}{\sum }_{i=1}^n(y_i-{\hat{y}}_i{)}^2$

• Pros:
  • The squaring penalizes large errors much more heavily than small ones.
  • It is convex and differentiable everywhere, making it ideal for gradient descent.
• Cons:
  • Extremely sensitive to outliers. A single bad prediction can skew the entire loss calculation.

Mean Absolute Error (MAE) / L1 Loss

Calculates the absolute difference between prediction and target. $MAE=\frac{1}{n}{\sum }_{i=1}^n|y_i-{\hat{y}}_i|$
- Pros:
- Robust to outliers. If dataset has noisy data or anomalies, MAE usually yields a better general fit than MSE.
- Cons:
- It is not differentiable at 0 (the V-shape bottom), which can complicate optimization near the solution.

Huber Loss

Hybrid function. It acts like MSE when the error is small (quadratic) and like MAE when the error is large (linear).

$L_{\delta }(y,\hat{y})=\{\begin{array}{ll}\frac{1}{2}(y-\hat{y}{)}^2& \text{for }|y-\hat{y}|\le \delta \\ \delta \cdot (|y-\hat{y}|-\frac{1}{2}\delta )& \text{otherwise}\end{array}$

• Use Case:
  • Robust regression where you want differentiability but need to ignore extreme outliers.

2. Classification Loss Functions

Used when predicting probabilities of classes (e.g., Cat vs. Dog, Spam vs. Not Spam).

Binary Cross-Entropy (Log Loss)

The standard for binary classification (0 or 1). It measures the distance between the actual class and the predicted probability.

$L=-\frac{1}{n}{\sum }_{i=1}^n[y_{i}log({\hat{y}}_i)+(1-y_i)log(1-{\hat{y}}_i)]$

• Mechanism:
  • If the actual label is 1 (y=1), the second term disappears, and we just maximize log(y^).
  • If the model predicts a probability of 0.1 for a true class of 1, the loss explodes, heavily penalizing confident wrong answers.

Categorical Cross-Entropy

The extension of Log Loss for multi-class problems (e.g., MNIST digits 0–9). It is typically used in conjunction with a Softmax activation function in the final layer.

$L=-{\sum }_{i=1}^C{y}_{i}log({\hat{y}}_i)$

Hinge Loss

Primarily used in Support Vector Machines (SVMs). It cares about the margin of the classification, not just whether it was right or wrong.

$L=max(0,1-y\cdot {\hat{y}}_i))$

• Use Case:
  • When you want to maximize the margin between decision boundaries.

3. Specialized Loss Functions

Used in special cases.