A mathematical formula to determine if a neuron should be activated based on the weighted sum of its inputs. They introduce non-linearity which allows the model to learn complex patterns. Without them, a Neural Network is just linear regression.
Types
1. ReLU (Rectified Linear Unit):
- The industry standard.
- Fast and efficient
2. Leaky ReLU:
- Fixes the “dying ReLU” problem where neurons get stuck at 0 by allowing a small gradient for negative values.
3. Sigmoid:
- Older function.
- Rarely used in hidden layers now because they cause vanishing gradients.
4. GELU (Gaussian Error Linear Unit):
- The modern standard for Transformers (BERT, GPT).
- Smoother version of ReLU.
