From Probabilistic Sequence Models to Self-Attention, Bidirectional Encoding, and Autoregressive Text Generation
Before modern transformers, AI shifted from static classification to generating synthetic sequential data.
Hidden Markov Models captured short-term state transitions. Great for speech, but failed at long context.
Restricted Boltzmann Machines learned compressed representations by reconstructing inputs via energy states.
Generated images pixel-by-pixel sequentially. Ensured local consistency, but suffered from extremely slow generation.
Breakthrough audio model generating raw waveforms. Produced human-like speech, but computationally expensive.
The Core Challenge: Early probabilistic models captured local patterns, but lacked diversity, long-range coherence, and creativity. This spurred the development of continuous latent space models.
Classic Autoencoder: Encodes input into a single fixed vector. Good for compression, but cannot generate new data.
VAE: Encodes input into a probabilistic distribution (Mean $\mu$ and Variance $\sigma^2$), allowing sampling of brand new points!
Direct sampling is non-differentiable. VAEs sample noise $\epsilon \sim \mathcal{N}(0, I)$ and scale it:
This allows backpropagation to flow smoothly through the encoder!
VAEs balance reconstruction quality with a continuous latent space:
Introduced by Ian Goodfellow (2014), GANs framed generation as a zero-sum game between two competing networks.
Takes random noise vector $z$ and creates synthetic samples $G(z)$. Objective: Fool the discriminator into thinking the fake sample is real!
Evaluates real images $x$ and generated images $G(z)$. Objective: Correctly classify real as 1 and fake as 0!
🌟 Strengths: Produces hyper-realistic, high-frequency image details without blurriness.
⚠️ Challenges: Training instability & Mode Collapse (generator repeats identical samples).
Seq2Seq models (RNN/LSTM) compressed an entire 100-word sentence into a single fixed vector.
Like forcing someone to memorize an entire textbook page and summarize it in one single word!
Instead of one fixed summary vector, Attention allows the model to look back at all input tokens at every decoding step!
Acts like a highlighter—dynamically zooming in on relevant source words when generating each output word!
Attention converts retrieval into a continuous, differentiable database lookup.
Represents the target token's question: "What information am I looking for right now?"
Represents the input tokens' labels: "What content or topic do I contain?"
Represents the actual information content to be passed forward if matched.
For large vector dimensions $d_k$, dot products grow large in magnitude, pushing softmax into regions with extremely small gradients (vanishing gradient problem).
Scaling by $\sqrt{d_k}$ keeps variance at 1 and keeps gradients stable!
Instead of looping over words one-by-one, $Q, K, V$ are packed into matrices, allowing GPU tensor math to compute attention for all words simultaneously!
"Attention Is All You Need" replaced recurrent loops completely with self-attention layers.
A single attention head might focus on subject-verb relationships. Multi-Head Attention runs multiple operations in parallel.
Tracks grammatical links (e.g. connecting verb "flew" to subject "bat").
Tracks contextual meaning (e.g. connecting "bat" to "cave" to infer animal).
Tracks adjacent token relationships and phrase boundaries.
Since self-attention operates on sets (permutation-invariant), we must inject word order explicitly into token embeddings!
Uses fixed sine & cosine functions of different frequencies:
Extrapolates to longer sequences, but fixed.
Assigns a dedicated trainable vector to every index (0 to 512).
Used in BERT & GPT-2. Simple, but cannot extend beyond max length.
Rotates token vectors in complex space based on position.
Used in LLaMA, Mistral, GPT-4. Generalizes to long contexts!
Released by Google (2018), BERT is an Encoder-Only architecture designed for deep language understanding.
Traditional left-to-right models miss context following a word. BERT reads the entire sentence simultaneously!
"The bat flew out of the cave."
BERT uses both "flew" (left) and "cave" (right) to know bat means an animal, not sports gear.
BERT is pretrained on unlabelled text (Wikipedia & BookCorpus) via two self-supervised tasks:
15% of input tokens are randomly replaced with a [MASK] token.
Forces BERT to use bidirectional context to fill in the blanks!
Model is fed pairs of sentences and predicts if Sentence B logically follows Sentence A.
Introduced by OpenAI (2018), GPT flips the script: an Decoder-Only architecture designed for text creation.
GPT predicts the next token in a sequence based on past tokens, one step at a time.
To prevent the model from "cheating" by peeking at future words during training, an upper-triangular mask sets future attention weights to $-\infty$.
Token at index $i$ can ONLY attend to tokens at indices $\le i$!
Trained on billions of tokens (web text, books, code) to minimize next-token prediction error.
Learns grammar, world facts, reasoning patterns, and coding syntax automatically!
Raw LLMs complete text, but can output harmful or unhelpful responses. Reinforcement Learning from Human Feedback aligns the model:
import numpy as np
def softmax(x):
e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
return e_x / np.sum(e_x, axis=-1, keepdims=True)
# Toy Embeddings (4 words, dim=4)
np.random.seed(42)
X = np.random.randn(4, 4)
# Linear Projections Q, K, V
W_q = np.random.randn(4, 4)
W_k = np.random.randn(4, 4)
W_v = np.random.randn(4, 4)
Q = np.dot(X, W_q)
K = np.dot(X, W_k)
V = np.dot(X, W_v)
# Scaled Dot-Product Attention
d_k = Q.shape[-1]
scores = np.dot(Q, K.T) / np.sqrt(d_k)
weights = softmax(scores)
output = np.dot(weights, V)
print("Attention Weights Matrix:\n", np.round(weights, 3))
print("\nContextualized Vector:\n", np.round(output[0], 3))
1. Takes input token embeddings X.
2. Projects into Query, Key, and Value matrices (Q, K, V).
3. Computes dot-product $Q \cdot K^T / \sqrt{d_k}$.
4. Applies softmax to produce probability weights.
5. Multiplies weights by V to get contextual embeddings!
| Architecture | Type | Attention Mask | Primary Strengths | Famous Models |
|---|---|---|---|---|
| BERT | Encoder-Only | None (Bidirectional) | Deep language comprehension, classification, QA, NER | BERT, RoBERTa, DeBERTa |
| GPT | Decoder-Only | Causal (Left-to-Right) | Fluent text generation, dialogue, coding, zero-shot tasks | GPT-4, LLaMA, Claude, Mistral |
| T5 / BART | Encoder-Decoder | Encoder: Bidirectional Decoder: Causal |
Translation, summarization, text-to-text transformation | T5, BART, Whisper |
| VAEs / GANs | Latent / Adversarial | N/A | Image synthesis, audio generation, style transfer | StyleGAN, VAE, Stable Diffusion |
VAEs and GANs proved neural networks could generate authentic new data distributions.
Removed recurrent bottlenecks and fixed information loss in long sequences.
Decoder-only scaling combined with RLHF unlocked modern conversational foundation models.