A type of neural network architecture designed to process sequential data. Unlike previous models (like RNNs or LSTMs) that process data word-by-word, Transformers process entire sequences at once, making them faster and better at capturing long-distance relationships in text.
Key Innovations
Self-Attention (The Context Engine)
In the sentence “The animal didn’t cross the street because it was too tired,” a Transformer uses Self-Attention to determine that “it” refers to “animal.”
- It creates a weighted map for every word in a sequence relative to every other word.
- This allows the model to focus on the most relevant parts of the input, regardless of how far apart they are.
Positional Encoding
Since Transformers process all words simultaneously (parallelization), they lose the natural sense of word order. Positional Encoding adds a unique mathematical signal to each word’s embedding so the model knows its specific position in the sequence (1st, 2nd, etc.).
Multi-Head Attention
Instead of looking at the sequence through one lens, the model uses multiple heads to attend to different types of information simultaneously. One head might focus on grammar, another on lexical meaning, and another on historical references within the text.
Architecture
Encoder
Processes the input sequence and creates a high-dimensional representation.
Decoder
Takes the encoder’s output and previously generated tokens to predict the next token in the sequence.
NOTE:
Modern LLMs are decoder-only. They are Autoregressive Models, meaning they process text from left to right. They are “blind” to any words that come after the one they are currently predicting.