What Are Foundation Models & How Do Models Learn?

Architecture Scale, Emergent Abilities, Efficiency Engineering, and Pretraining Paradigms (Supervised, Unsupervised & Self-Supervised)

Scale & Emergence Quantization & Distillation Self-Supervised Learning

The Paradigm Shift: Narrow AI vs. Foundation Models

Traditional Narrow AI

Designed from scratch for single, specific tasks (e.g. spam filter, Sentiment Classifier, ResNet-50 image tagger).

Inflexible: Requires a new model for every new task.
Fragmented data & heavy manual labelling.
Cannot generalize to unseen domains.

General-Purpose Foundation Models

Trained ONCE on massive, multimodal datasets (text, code, vision, audio) and adapted to thousands of downstream tasks.

One core base model handles code, translation & QA.
Minimal fine-tuning or zero-shot prompt engineering.
Transfer learning at unprecedented scale.

The 3 Pillars of Foundation Models

1. Massive Scale

Trained on billions to trillions of parameters and petabytes of data.

Captures deep non-linear relationships across language, code, and physical concepts.

2. Emergent Abilities

Qualitatively new skills appear unexpectedly as parameter scale increases!

Reasoning, multi-step math, code execution & puzzle solving emerge naturally without explicit programming.

3. Multimodality

Unifies text, vision, audio, and sensor inputs in a shared embedding space.

Models like Gemini & GPT-4o reason seamlessly across images, spoken voice, and text.

Building a Foundation Model: Infrastructure & Cost

Constructing a frontier foundation model is like building a skyscraper—requiring massive capital & compute.

1. Data Scale

Trillions of tokens of high-quality text, code, web documents, books, and synthetic datasets.

Token Filtering & De-duplication

2. Compute Clusters

Tens of thousands of GPUs/TPUs (e.g. H100 / H200 clusters) running continuously for months.

High-Speed InfiniBand Fabrics

3. Financial Investment

State-of-the-art training runs cost between $50 Million to $500+ Million per run!

Power & Cooling Costs

Efficiency Engineering: Leaner, Faster Models

Running giant 70B+ parameter models on cloud servers is expensive. We compress them using 3 key techniques:

Quantization

Reduces precision of weights from FP32/FP16 down to INT8 or INT4.

16-bit $\to$ 4-bit (75% Memory Reduction)

Enables running 8B models directly on local laptops and edge devices!

Pruning

Identifies and zero-out redundant or non-critical weight connections.

Sparse Weights Matrix

Accelerates matrix multiplication inference speeds.

Knowledge Distillation

Train a compact Student model to mimic the probability distributions of a massive Teacher model.

Teacher (405B) $\to$ Student (8B)

The Current Foundation Model Landscape

Modality Domain	Open-Source / Open-Weights	Proprietary / API Access	Key Breakthrough Focus
Language & Reasoning	Meta Llama 3.3, DeepSeek R1 & V3, Mistral NeMo	OpenAI o3 / GPT-4.5, Anthropic Claude 3.7, Google Gemini 1.5	Chain-of-Thought reasoning, long-context windows (2M tokens)
Vision & Image Gen	FLUX.1, Stable Diffusion XL / 3.5	OpenAI DALL·E 3, Midjourney v6	Photorealistic text-to-image, typography rendering
Audio & Voice	Whisper (Speech-to-Text), SeamlessM4T	Google Chirp 2, ElevenLabs Voice Engine	Real-time speech translation, low-latency voice agents

Core Limitations of Foundation Models

1. Hallucinations

Models predict plausible-sounding but false facts because they predict text statistically rather than querying a verified database.

Fix: Retrieval-Augmented Generation (RAG).

2. Pretraining Biases

Unfiltered web scraping causes models to inherit and amplify human stereotypes, toxicity, and societal prejudices.

Fix: RLHF alignment & safety guardrails.

3. Knowledge Cutoff

Pretrained weights are static snapshots of data up to their training end date. They are blind to real-time events.

Fix: Live web search tools & API function calling.

How Do Models Learn? The 3 Paradigms

AI models aren't born smart—they acquire knowledge through foundational pretraining. Think of hiring three robot chefs:

Supervised Learning

Chef 1: Follows labeled recipes with step-by-step instructions and immediate corrections.

Inputs + Human Labels

Unsupervised Learning

Chef 2: Studies thousands of unlabeled cookbooks to discover common cooking patterns.

Clustering Raw Data

Self-Supervised Learning

Chef 3: Hides ingredients from itself and plays games predicting missing steps!

Data Creates Its Own Labels

Supervised Learning: Learning from Labeled Data

How Supervised Training Works

The model receives input data $X$ alongside ground-truth target labels $Y$.

Model outputs prediction $\hat{Y} = f(X; W)$.
Calculates loss $\mathcal{L}(\hat{Y}, Y)$.
Updates weights via backpropagation: $W \leftarrow W - \alpha \nabla_W \mathcal{L}$.

The Labeling Bottleneck

High Expense: Human annotators must manually label millions of images or text pairs.
Labeler Bias: Annotator mistakes pollute training data.
Hard to Scale: Impossible to label petabytes of internet-scale data manually!

Unsupervised Learning: Clustering & Pattern Discovery

Finding Hidden Structure

Operates on unlabeled data $X$. Discovers clusters, latent features, and dimensionality reduction without human guidance.

K-Means Clustering: Groups similar items by distance metric.
PCA: Reduces features while preserving variance.

The Catch: Surface-Level Features

Because no semantic objective is enforced, unsupervised models often cluster by shallow visual cues (e.g. background color rather than animal anatomy).

Lacks deep semantic understanding needed for complex reasoning tasks.

Self-Supervised Learning (SSL): The Engine of GenAI

SSL turns unlabeled data into a supervised task by hiding parts of the input and predicting them!

Why SSL Transformed AI

Zero Cost Labels: Uses raw web text, audio, and video directly without human labeling!
Infinite Scalability: Scales endlessly as data volume and compute increase.
Rich Representations: Forces model to master grammar, context, semantics, and world logic.

The SSL Paradigm

Input: "The quick [MASK] fox jumps over the lazy dog."
Supervised Task generated automatically:
Feature: Visible tokens $\to$ Target: "brown"

The 3 Core Self-Supervised Tasks

1. Masked Language Modeling

Hide random tokens in a sentence; model uses bidirectional context to guess them.

Used in BERT & RoBERTa

2. Causal Autoregressive LM

Predict the next word in a sequence given preceding visible context.

Used in GPT-4, LLaMA & DeepSeek

3. Contrastive Learning

Pull matching image-text pairs close in vector space; push non-matching pairs apart.

Used in CLIP & Multimodal Vision

Live Code: Simulating INT8 Model Quantization

import numpy as np

# Simulate FP32 Neural Network Weights
np.random.seed(42)
weights_fp32 = np.random.uniform(-2.5, 2.5, size=(4, 4))

# Compute Scale Factor for INT8 (-128 to 127)
max_abs = np.max(np.abs(weights_fp32))
scale = max_abs / 127.0

# Quantize FP32 -> INT8
weights_int8 = np.round(weights_fp32 / scale).astype(np.int8)

# Dequantize back to FP32 for inference
reconstructed_fp32 = weights_int8 * scale
quantization_error = np.mean(np.abs(weights_fp32 - reconstructed_fp32))

print("Original FP32 Weights (bytes:", weights_fp32.nbytes, "):\n", np.round(weights_fp32, 2))
print("\nQuantized INT8 Weights (bytes:", weights_int8.nbytes, "):\n", weights_int8)
print("\nMemory Saved:", round((1 - weights_int8.nbytes/weights_fp32.nbytes)*100, 1), "%")
print("Mean Error:", round(quantization_error, 4))

Quantization Math

1. High-precision FP32 weights consume 4 bytes (32 bits) per parameter.

2. Maps continuous range $[-X_{max}, X_{max}]$ into 8-bit integers $[-128, 127]$.

3. Saves 75% RAM while preserving over 99% model accuracy!

Pretraining Paradigms Comparison

Learning Paradigm	Analogy	How Model Learns	Key Advantages	Limitations
Supervised	Teacher with labeled flashcards	Compares prediction to explicit target labels	High accuracy on narrow tasks	Expensive human labeling, hard to scale
Unsupervised	Sorting photos by visual likeness	Groups unlabeled data by distance metrics	Scales easily, finds hidden clusters	Learns shallow, surface-level features
Self-Supervised	Solving fill-in-the-blank puzzles	Hides parts of data & predicts missing pieces	Zero label cost, deep semantic representations	Complex self-task setup

Session 8 Summary

The Foundation Model Blueprint

1. Train Once, Adapt Many

Foundation models replace narrow tools with versatile base intelligence.

2. Self-Supervised Engine

SSL unlocks petabyte scale by turning raw data into self-labeled learning tasks.

3. Leaner Deployment

Quantization and distillation enable efficient edge and cloud execution.

Return to SRM PDP Talk Hub

1 / 15