An unsupervised Neural Network that learns to compress data to lower dimensions and then reconstruct the data to match the original input as closely as possible. The lower dimensional space is called the Latent Space.
Types:
- Vanilla Autoencoder:
- Symmetrically structured, feed-forward neural network with a single lower dimensional hidden layer.
- Denoising Autoencoder:
- Injects random noise into the input data and forces the network to remove that noise during reconstruction. Used for image denoising, audio filtering, and extracting robust, generalizable data features.
- Sparse Autoencoder:
- Adds a sparsity penalty to the network’s loss function. This forces only a small fraction of the hidden nodes to “activate” at any given time, even if the layer has a high overall capacity. Often used for Mechanistic Interpretability.
- Contractive Autoencoder:
- Adds a penalty based on the Frobenius Norm of the encoder’s Jacobian matrix. This forces the model to learn representations that are smooth and stable against small changes/variations in the input.
- Convolutional Autoencoder:
- Replaces fully connected, dense layers with convolutional layers, making them highly efficient for grid-like data. Used for image compression and reconstructing corrupted visual data.
- Variational Autoencoder:
- Rather than mapping data to a single fixed point in the latent space, VAEs map inputs to a probability distribution. Allows you to sample from the probability distribution to generate entirely new, synthetic data