Rupan

Neural Network

3 min read

A machine learning model that consists of layers of interconnected nodes. Essentially global function appropriators (see Universal Approximation Theorem).

Types:

  • Feedforward Neural Network (FNN) / Multilayer Perceptron (MLP)
  • Convolutional Neural Network (CNN)
  • Recurrent Neural Network (RNN)
  • Graph Neural Network (GNN)

How it works:

Neural networks consist of layers, which contain nodes. Data flows through the layers and is augmented by each nodes weights and biases.

The model is trained with a Training Loop.

Layers:

Types:

1. Linear Layers

The most basic building blocks. Every input is connected to every output.

  • Input: Vector (Batch, Size)
  • Use Case: Tabular data, final classification head, simple regressions

Standard Fully Connected Layer

Identity (Pass-through) - Useful for debugging or residual connections

2. Convolutional Layers

These scan the input with a kernel (filter) to detect patterns (edges, shapes).

  • Input: (Batch, Channels, Dimension...)
  • Use Case: Images (2D), Audio (1D), Video/Medical Scans (3D).

1D Convolution (Audio / Time Series / Text)

  • Input: (Batch, Channels, Length)

2D Convolution (Images)

  • Input: (Batch, Channels, Height, Width)

3D Convolution (Video / MRI)

  • Input: (Batch, Channels, Depth, Height, Width)

3. Pooling Layers

These reduce the spatial dimensions of the data to reduce computation and force the model to learn “invariant” features (e.g., recognizing a cat regardless of where it is in the picture).

Max Pooling (Selects the brightest pixel/strongest feature)

  • Best for extracting dominant features (edges).

Average Pooling (Averages the area)

  • Best for smoothing.

Adaptive Pooling

  • “I don’t care what the input size is, give me a 1x1 output.”

4. Recurrent Layers

These process sequences step-by-step, maintaining a “hidden state” (memory) of what they saw previously.

  • Input: (Batch, Sequence_Length, Features)
  • Use Case: NLP, Time Series, Molecular Strings (SMILES).

RNN (Basic Recurrent Neural Network)

  • Simple, but suffers from vanishing gradients.

LSTM (Long Short-Term Memory)

  • The gold standard for RNNs. Has “gates” to forget or keep memory.

GRU (Gated Recurrent Unit)

  • A simplified, faster LSTM.

5. Transformer Layers

The modern standard replacing RNNs. They process the entire sequence at once using “Self-Attention.”

Multi-Head Attention

  • The core mechanism: “How much does Word A relate to Word B?”

Transformer Encoder Layer

  • A bundle of Attention + FeedForward + Norm

6. Regularization Layers

These don’t learn parameters; they modify data during training to prevent overfitting. (e.g., Dropout)

Dropout

  • Randomly zeros out elements with probability p.
  • Forces the network not to rely on any single neuron.

Normalization Layers

7. Utility Layers

Helpers to reshape or transform data.

Flatten

  • Smashes dimensions. (Batch, 32, 10, 10) -> (Batch, 3200)
  • Used to connect Conv layers to Linear layers.

Embedding

  • Lookup table. Converts integer IDs (e.g., Word #55) into Vectors.
  • Essential for NLP and Graphs.

Backlinks