An optimization algorithm that is used to find the optimal set of parameters that minimize the Loss Function
Complementary process with Backpropogation, which computes the Gradient for Gradient Descent.
How it works:
Uses the Gradient and updates the parameters.
Types of Gradient Descent
1. Batch Gradient Descent:
- Calculates the Gradient of the loss function using the entire dataset
- Accurate but slow and computationally expensive, especially with large datasets.
2. Stochastic Gradient Descent (SGD):
- Updates model parameters using just one data point at a time.
- Quicker but may sometimes be erratic and can lead to more fluctuations in the path to the minimum.
3. Mini-Batch Gradient Descent:
- Splits the dataset into small batches and updates the parameters after each batch.
- Balances the accuracy of batch gradient descent with the speed of SGD, making it a popular choice in practice.