Normalization Layers

Neural Network layers that force the data to stay within a standard range (mean 0, variance 1) as it flows through the network. This prevents “co-variate shift” (where the data distribution changes wildly between layers).

Types

Batch Normalization

• Normalizes across the batch dimension.
• Standard for Images/Convolutional Neural Networks (CNN).
• Great for CNNs with large, consistent batches but struggles with small or variable batches.

Layer Normalization

• Normalizes across the feature dimension.
• Standard for NLP/Transformers and often used in GNNs where batch sizes vary.

Math is as follows:

x is the input.

1. Calculate the Mean ($\mu$) and Variance (${\sigma }^2$) across the features:
   $\mu =\frac{1}{H}{\sum }_{i=1}^H{x}_i$
   ${\sigma }^2=\frac{1}{H}{\sum }_{i=1}^H(x_i-\mu {)}^2$
2. Normalize:
   $\hat{x}=\frac{x-\mu }{\sqrt{{\sigma }^2+ϵ}}$
   $ϵ$ is a small constant to prevent divide by zero.
3. Scale and shift:
   $y=\gamma \odot \hat{x}+\beta$
   $\gamma$ and $\beta$ are learnable parameters.

Instance Normalization

• Normalizes each sample individually. Used in Style Transfer (GANs).