Training Foundation Models

Foundation models are large neural networks trained on broad datasets and adapted to many downstream tasks. Examples include large language models, multimodal transformers, vision foundation models, audio-language systems, and general-purpose embedding models.

Training these systems requires coordinated advances in:

optimization
distributed systems
data engineering
numerical stability
infrastructure reliability
hardware utilization

Foundation model training differs from ordinary deep learning mainly in scale. The underlying mathematical principles remain similar, but the operational constraints become much more severe.

A small model may train on one GPU for hours. A foundation model may require thousands of accelerators running continuously for weeks or months.

What Defines a Foundation Model

A foundation model typically has several properties:

Property	Description
Large parameter count	Millions to trillions of parameters
Broad pretraining data	Diverse internet-scale datasets
General-purpose representations	Useful across many tasks
Transferability	Fine-tuned or prompted for downstream use
Emergent capabilities	Behaviors not explicitly supervised

Most modern foundation models are transformer-based because transformers scale efficiently with data and compute.

Common foundation model categories include:

Type	Example tasks
Language models	Text generation, reasoning, coding
Vision models	Classification, segmentation, detection
Multimodal models	Vision-language understanding
Audio models	Speech recognition, synthesis
Embedding models	Retrieval and semantic search

Scaling Laws

One of the central discoveries in modern deep learning is that model performance often follows predictable scaling behavior.

Performance depends on:

parameter count
training data size
compute budget

Empirical scaling laws often resemble power-law relationships:

L(N) = A N^{-\alpha} + C,

where:

Symbol	Meaning
$L(N)$	Loss
$N$	Scale variable
$A, C$	Constants
$\alpha$	Scaling exponent

Increasing model size, data, or compute generally improves performance, though with diminishing returns.

Scaling laws influenced modern foundation model design because they suggested that larger models trained on more data would continue improving predictably.

This shifted research from hand-designed architectures toward large-scale optimization and infrastructure.

Compute-Optimal Training

Training budgets are finite. A key question becomes how to allocate compute between:

larger models
more training tokens
longer training duration

Suppose:

Variable	Meaning
$P$	Parameter count
$T$	Training tokens
$C$	Total compute

Approximate transformer training cost scales as:

C \propto P T.

If the model is too large for the available data, parameters are undertrained. If the dataset is too large for the model, capacity may be insufficient.

Modern training recipes attempt to balance model size and data volume to maximize performance for a fixed compute budget.

This idea is often called compute-optimal training.

Token-Based Training

Large language models are usually trained in terms of tokens rather than epochs.

A token is a subword unit produced by tokenization.

Example:

"foundation models are powerful"

might tokenize into:

["foundation", " models", " are", " powerful"]

Training progress is often measured as:

\text{tokens processed}.

For example:

Model	Approximate training tokens
Small language model	Billions
Mid-scale LLM	Hundreds of billions
Frontier-scale LLM	Trillions

Unlike classical datasets, internet-scale corpora may not have clean epoch boundaries. Data pipelines therefore stream tokens continuously.

Data Pipelines

Foundation models require enormous datasets.

Data engineering becomes a major component of the system.

Typical stages include:

Stage	Purpose
Crawling	Collect raw data
Deduplication	Remove repeated content
Filtering	Remove low-quality data
Language identification	Separate languages
Safety filtering	Remove harmful content
Tokenization	Convert text to token IDs
Sharding	Split data across workers

Data quality strongly affects model quality.

A smaller high-quality dataset may outperform a much larger noisy dataset.

Streaming Datasets

Large datasets are rarely stored as one monolithic file.

Instead, they are sharded into many files:

shard_00000.bin
shard_00001.bin
shard_00002.bin
...

Workers stream shards in parallel.

Advantages include:

Benefit	Reason
Parallel reading	Multiple workers load simultaneously
Fault tolerance	Corruption affects only one shard
Distributed access	Nodes read different shards
Incremental updates	New shards can be added

Streaming avoids loading the entire dataset into memory.

Transformer Training

Most foundation models are transformers.

A simplified decoder-only transformer computes:

x \rightarrow \text{Embedding} \rightarrow \text{Transformer Blocks} \rightarrow \text{Output Projection}.

Each transformer block contains:

self-attention
feedforward networks
residual connections
normalization layers

Training is autoregressive.

Given tokens:

[t_1, t_2, \ldots, t_n],

the model predicts:

t_{i+1}

from earlier tokens.

The objective is usually next-token prediction:

L = - \sum_{i=1}^{n} \log p_\theta(t_i \mid t_{<i}).

Mixed Precision Training

Foundation model training almost always uses mixed precision.

Instead of float32 everywhere, systems use:

Format	Common use
fp16	Earlier mixed precision systems
bf16	Modern large-scale training
fp32	Master weights or sensitive operations

Mixed precision reduces:

memory usage
communication volume
training time

Bfloat16 became especially important because it preserves the exponent range of float32, improving numerical stability.

A typical configuration:

Tensor type	Precision
Activations	bf16
Gradients	bf16
Matrix multiplications	bf16
Optimizer state	fp32
Master weights	fp32

Parallelism Strategies

Foundation models are too large for simple data parallelism alone.

Modern systems combine multiple parallelism dimensions.

Parallelism	Purpose
Data parallelism	Scale training throughput
Tensor parallelism	Split large matrix operations
Pipeline parallelism	Split sequential layers
Sharded optimizers	Reduce replicated optimizer state
Expert parallelism	Route sparse experts across devices

Large training systems often organize GPUs into groups.

Example:

Parallelism dimension	Size
Data parallel	16
Tensor parallel	8
Pipeline parallel	4

Total GPUs:

16 \times 8 \times 4 = 512.

Each GPU participates in several communication groups simultaneously.

Memory Optimization

Memory becomes a dominant constraint.

Main memory consumers include:

Component	Scaling behavior
Parameters	Proportional to model size
Optimizer state	Often 2 to 8 times parameter size
Gradients	Similar to parameter size
Activations	Depend on batch and sequence length

Techniques used to reduce memory include:

Technique	Purpose
Activation checkpointing	Trade compute for memory
Gradient accumulation	Simulate large batches
ZeRO/FSDP	Shard optimizer state and parameters
Quantization	Lower precision storage
Offloading	Move state to CPU or NVMe

Without these methods, large models may not fit even across many GPUs.

Throughput Optimization

Training cost is dominated by accelerator time.

Suppose a training run uses:

2,000 GPUs
$2 per GPU-hour
30 days

Cost:

2000 \times 24 \times 30 \times 2 = 2{,}880{,}000.

Even small inefficiencies become expensive.

Important throughput metrics include:

Metric	Meaning
Tokens per second	Language-model throughput
FLOPs utilization	Fraction of peak compute used
GPU utilization	Accelerator activity
Communication overhead	Time spent synchronizing
Data pipeline latency	Waiting for input data

High-performance training systems carefully overlap:

communication
data loading
computation
checkpointing

Learning Rate Schedules

Foundation models often use carefully tuned learning rate schedules.

A common pattern:

warmup
plateau or cosine decay
gradual reduction

Warmup stabilizes early optimization.

Example cosine schedule:

\eta_t = \eta_{\min} + \frac{1}{2} (\eta_{\max} - \eta_{\min}) \left( 1 + \cos\left(\frac{\pi t}{T}\right) \right).

genui{“math_block_widget_always_prefetch_v2”:{“content”:"\eta_t=\eta_{\min}+\frac{1}{2}(\eta_{\max}-\eta_{\min})\left(1+\cos\left(\frac{\pi t}{T}\right)\right)"}}

Warmup is especially important for large batch training because early gradients can be unstable.

Gradient Stability

Large models are sensitive to numerical instability.

Common problems include:

Problem	Symptom
Exploding gradients	Loss divergence
Vanishing gradients	Slow learning
Overflow	NaNs
Underflow	Zero gradients
Activation spikes	Instability

Stabilization techniques include:

Technique	Purpose
Gradient clipping	Limit update magnitude
Normalization layers	Stabilize activations
Residual connections	Improve gradient flow
Careful initialization	Prevent early divergence
Adaptive optimizers	Stabilize updates

Gradient clipping often uses:

g \leftarrow g \cdot \min\left( 1, \frac{\tau}{\|g\|} \right),

where $\tau$ is the clipping threshold.

Evaluation During Training

Foundation model evaluation is expensive.

Evaluations may include:

Evaluation type	Example
Validation perplexity	Language modeling
Benchmark suites	Reasoning and QA
Human preference evaluation	Alignment
Safety testing	Harmful outputs
Retrieval quality	Embedding models

Frequent evaluation slows training, but infrequent evaluation risks wasting compute on bad runs.

Many systems therefore run lightweight validation frequently and expensive benchmark suites less often.

Alignment and Post-Training

Pretraining produces a general-purpose model, but not necessarily a helpful or safe assistant.

Modern systems often add:

Stage	Purpose
Supervised fine-tuning	Teach instruction following
Preference optimization	Align outputs with preferences
RLHF	Reinforcement learning from human feedback
Constitutional methods	Rule-guided alignment
Safety tuning	Reduce harmful behavior

The final model is therefore the result of several training stages, not just one pretraining run.

Infrastructure Reliability

Foundation model training depends heavily on infrastructure engineering.

Key requirements include:

Requirement	Reason
Fault tolerance	Failures are inevitable
Distributed checkpointing	Large model state
Monitoring systems	Detect hangs and instability
Cluster scheduling	Coordinate resources
High-bandwidth networking	Synchronization efficiency
Storage throughput	Massive datasets and checkpoints

At large scale, infrastructure limitations often dominate algorithmic limitations.

Environmental and Economic Cost

Foundation model training consumes substantial energy and compute resources.

Costs include:

accelerator manufacturing
electricity
cooling
datacenter infrastructure
engineering labor

Efficiency improvements therefore matter economically and environmentally.

Important efficiency directions include:

Direction	Goal
Better optimizers	Fewer training steps
Sparse models	Lower compute
Quantization	Lower memory and energy
Smaller high-quality datasets	Better data efficiency
Efficient architectures	Higher throughput

Emergent Behavior

As models scale, new capabilities sometimes appear unexpectedly.

Examples may include:

in-context learning
chain-of-thought reasoning
tool use
multilingual transfer
coding ability

These behaviors are called emergent because they were not explicitly programmed.

However, emergence is often gradual rather than sudden when measured carefully.

Understanding why scaling produces these capabilities remains an active research area.

The Central Constraint

Foundation model training is fundamentally constrained by:

\text{compute} \times \text{data} \times \text{memory} \times \text{communication}.

Every design decision affects one or more of these factors.

For example:

Decision	Tradeoff
Larger model	Better capacity, higher memory
Longer context	Better reasoning, more compute
More GPUs	More throughput, more communication
Larger batches	Better hardware utilization, harder optimization

Training systems therefore balance mathematical efficiency with systems efficiency.

From Research to Infrastructure

Early deep learning research focused primarily on architecture design. Foundation model training shifted much of the challenge toward systems engineering.

Modern training requires expertise in:

optimization theory
distributed systems
networking
compiler systems
numerical methods
storage infrastructure
data engineering

As models scale, the boundary between machine learning research and large-scale systems engineering becomes increasingly blurred.

A modern foundation model is simultaneously:

a statistical learning system
a distributed computation graph
a large-scale numerical optimization problem
a data processing pipeline
a fault-tolerant infrastructure system