From Causal & Masked Objectives to LoRA PEFT Adaptation, SFT, and Human-Preference Alignment via RLHF
A pretraining task dictates both how model weights are updated during training AND how the model generates during inference.
Given prompt "The cat sat on the", the model calculates cross-entropy loss against ground-truth target "mat".
Loss error signals propagate backward via Adam optimizer to update weights.
Given user prompt "Translate to French: I love cats", the model uses the SAME autoregressive next-token prediction loop.
Outputs "J'aime" $\to$ "les" $\to$ "chats" sequentially.
Measures disagreement between model's predicted probability distribution $P$ and true target token $y$:
Correct predictions lower loss; wrong guesses raise loss!
Calculates parameter gradients $\nabla_W \mathcal{L}$ with adaptive momentum and learning rates:
Nudges billions of weights step-by-step toward mastering grammar and reasoning.
Predicts next token using left-only past context. Upper-triangular causal mask prevents looking ahead.
GPT-4, LLaMA, DeepSeekMasks 15% random tokens; uses bidirectional (left + right) context to reconstruct hidden tokens.
BERT, RoBERTa, DeBERTaAligns paired embeddings (e.g. image + caption) while pushing mismatched pairs apart in vector space.
CLIP, ImageBindWeb scraping, deduplication, toxicity filtering, BPE tokenization.
Thousands of GPUs initialized with random Gaussian weight matrices.
Self-supervised prediction, loss computation & backpropagation.
Generalist base model ready for downstream task adaptation.
A pretrained base model is a raw text completer. Prompt it with "How to bake a cake?" and it might just auto-complete with more internet questions instead of giving instructions!
Fine-tuning applies Transfer Learning—reusing general pretraining knowledge to specialize the model as an interactive assistant that follows user instructions precisely!
Updates ALL parameters ($100\%$ of base model weights $W_0$).
Freezes base weights $W_0$ and updates only a tiny fraction ($< 1\%$) of parameters.
Freeze base transformer layers; train only a newly added task-specific classification head on top.
Unfreeze and fine-tune only specific selected layers (e.g. top 2 transformer layers or bias terms).
Insert tiny bottleneck linear layers after self-attention; train only adapter parameters.
Instead of updating weight matrix $W_0 \in \mathbb{R}^{d \times k}$ directly, LoRA decomposes update matrix $\Delta W$ into two low-rank matrices $A$ and $B$:
Where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ with rank $r \ll \min(d, k)$ (e.g. $r = 8$ or $16$).
Reduces trainable parameters from $d \times k$ down to $r \times (d + k)$ (over 99% reduction!).
During deployment, low-rank matrices $B \cdot A$ can be multiplied and added directly into base weights $W_0$.
No extra latency added during live generation!
Allows fine-tuning a massive 70B parameter LLM on a single consumer GPU (24GB VRAM) with zero performance degradation compared to 16-bit fine-tuning!
| Method | Trainable Parameters | Memory Footprint | Inference Latency Overhead | Performance vs Full FT |
|---|---|---|---|---|
| Full Fine-Tuning | 100% | Very High (4x Base Model) | None | 100% (Baseline) |
| Feature-Based | < 0.1% | Very Low | Minimal | Lower (Rigid) |
| Adapters | ~ 1% | Low | Small Overhead (+2-5%) | Comparable |
| LoRA | 0.01% - 0.1% | Extremely Low | Zero Overhead (Merged) | Matches Full FT |
| QLoRA | 0.01% - 0.1% | Lowest (4-bit Base) | Zero Overhead | Matches Full FT |
SFT trains models on curated prompt-response pairs so they behave as instruction-following assistants.
"When a measure becomes a target, it ceases to be a good measure."
If long answers were rewarded during instruction tuning, models learn to write verbose, bloated text. Solved via RLHF alignment!
RLHF aligns model outputs with human values (Helpful, Honest, Harmless):
Fine-tune base model on high-quality demonstration prompts & human answers.
Collect human rankings of multiple generated responses; train a neural Reward Model $R(x, y)$.
Optimize policy model using Proximal Policy Optimization (PPO) to maximize reward score.
Without guardrails, PPO RL optimization will "game" the reward model, outputting gibberish that scores artificially high.
The Role of $\beta D_{KL}$: Acts as an elastic leash, penalizing the RL model if its output probability distribution drifts too far from the reference SFT model!
import numpy as np
# Base Frozen Weights (dim_in=1000, dim_out=1000)
d, k = 1000, 1000
np.random.seed(42)
W0 = np.random.randn(d, k)
# LoRA Low-Rank Matrices (rank r = 4)
r = 4
A = np.random.randn(r, k)
B = np.zeros((d, r)) # Init B to zeros so delta starts at 0
# Compute Parameters Saved
full_params = d * k
lora_params = r * (d + k)
# Forward Pass Simulation
X = np.random.randn(1, d)
base_out = np.dot(X, W0)
lora_delta = np.dot(np.dot(X, B), A)
final_out = base_out + lora_delta
print("Full Model Parameters:", full_params)
print("LoRA Trainable Parameters:", lora_params)
print("Parameter Reduction:", round((1 - lora_params/full_params)*100, 2), "%")
print("Initial LoRA Delta Magnitude:", round(np.linalg.norm(lora_delta), 6))
1. Base weight matrix W0 ($1000 \times 1000$) has 1,000,000 parameters.
2. Low-rank matrices A and B ($r=4$) have only 8,000 parameters.
3. 99.2% Parameter Reduction with identical output representation capacity!
| Phase | Objective Task | Dataset Scale | Primary Outcome |
|---|---|---|---|
| Pretraining | Next-token prediction (CLM) / MLM | Trillions of tokens (Unlabeled) | Broad linguistic & world knowledge base |
| Supervised Fine-Tuning | Instruction-Response mapping (SFT) | 10k - 100k curated pairs | Interactive instruction following |
| PEFT (LoRA) | Low-rank update $B \cdot A$ | Domain specific | 99%+ compute/RAM savings |
| RLHF (PPO) | Reward Model scoring + KL penalty | Human rank preferences | Helpful, safe, human-aligned assistant |
Autoregressive Causal LM forms the foundational knowledge engine for modern LLMs.
LoRA & QLoRA make adapting giant 70B+ models practical on accessible hardware.
Combines Reward Models, PPO, and KL penalties to build safe, aligned AI assistants.