Large Language Models

In 2017, the transformer architecture introduced in Attention Is All You Need achieved state-of-the-art machine translation. By 2020, GPT-3 demonstrated that scaling transformers to 175 billion parameters unlocked remarkable capabilities—from code generation to creative writing—that smaller models simply could not achieve. The field had discovered a profound truth: scale fundamentally changes what models can do.

This wasn't merely incremental improvement. Researchers observed qualitative changes in behavior as models grew—capabilities that emerged suddenly rather than gradually, abilities that defied prediction from smaller-scale experiments. Understanding transformer scaling has become essential for anyone working at the frontier of machine learning, influencing everything from research priorities to infrastructure investments worth billions of dollars.

The modern understanding of transformer scaling is grounded in scaling laws—empirical relationships that predict model performance as a function of compute, data, and parameters. These laws, discovered through systematic experimentation, provide the theoretical foundation for billion-dollar training decisions.

The Kaplan Scaling Laws (OpenAI, 2020)

The seminal work by Kaplan et al. established that language model loss follows predictable power-law relationships:

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}$$

Where $L$ is the loss, $N$ is the number of parameters, and $N_c$ and $\alpha_N$ are fitted constants. Similar relationships exist for compute $C$ and dataset size $D$.

Key findings from the Kaplan laws:

1. Smooth power-law scaling: Loss decreases as a power-law with increased compute, parameters, or data
2. Compute-efficient scaling: Optimal model size scales sublinearly with compute budget
3. Parameter efficiency: For fixed compute, larger models trained on less data often outperform smaller models trained longer
4. Data requirements: Optimal data size scales roughly linearly with model size

The Chinchilla Scaling Laws (DeepMind, 2022)

Hoffmann et al. revised our understanding with the Chinchilla paper, demonstrating that many large models were significantly undertrained. Their analysis suggested a more balanced approach to scaling:

Chinchilla optimal scaling:

• For compute-optimal training, parameters and data should scale roughly equally
• The optimal model size is approximately $N^* \propto C^{0.5}$ (not $C^{0.73}$)
• The optimal number of training tokens is approximately $D^* \propto C^{0.5}$

This had profound implications:

Chinchilla (70B parameters) outperformed Gopher (280B parameters) while using the same compute, simply by training longer on more data.

The Unified Scaling Law

Integrating the dimensions of scale, performance can be modeled as:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

Where:

• $E$ is the irreducible entropy of natural language (estimated ~1.69 nats for English)
• $A/N^\alpha$ captures parameter scaling
• $B/D^\beta$ captures data scaling

This formulation reveals a crucial insight: performance is bottlenecked by the smaller of the two terms. An undertrained large model or an overtrained small model both waste compute. Optimal allocation balances both contributions.

Perhaps the most fascinating aspect of scaling is the emergence of capabilities that cannot be predicted from smaller models. These emergent abilities appear suddenly once a model reaches a critical scale threshold, transforming from near-random performance to rapid improvement.

Defining Emergence

An ability is considered emergent if:

1. It is absent or random in smaller models
2. It appears abruptly at scale rather than improving gradually
3. It cannot be predicted by extrapolating from smaller models

This is distinct from smooth scaling, where doubling parameters yields predictable, incremental gains. Emergence represents a phase transition—a qualitative change in what the model can do.

Theories of Emergence

Several hypotheses attempt to explain why emergence occurs:

1. Task Decomposition Threshold Complex tasks require multiple sub-capabilities. Performance remains low until all required sub-capabilities reach sufficient quality, then jumps rapidly. For example, arithmetic requires digit recognition, positional understanding, and carry propagation—all must work for the overall task to succeed.

2. Circuit Formation Neural networks form computational circuits that perform specific functions. Emergent abilities may correspond to circuits that only form (or become reliably activated) above certain parameter counts.

3. Grokking Phenomenon Models sometimes exhibit 'grokking'—sudden generalization long after memorization. This may happen at scale when sufficient capacity enables the model to discover generalizable algorithms rather than memorizing patterns.

4. Measurement Artifacts Some researchers argue emergence is partially an artifact of discrete metrics. When measuring accuracy on binary tasks, gradual improvements in underlying probability appear as sudden jumps. Continuous metrics may show smoother scaling.

Implications for Model Development

Emergent capabilities shape how frontier AI labs approach development:

• Scale-first mentality: The possibility of valuable emergent capabilities incentivizes training ever-larger models
• Capability elicitation: Extensive testing is required to discover what abilities a model has acquired
• Safety evaluation: New safety concerns may emerge unpredictably at scale
• Benchmark design: Traditional benchmarks may not capture emergent abilities; new evaluation paradigms are needed
• Competitive pressure: The unpredictability creates FOMO-driven scaling races between labs

Understanding emergence is crucial because it fundamentally changes the return profile of scale investments: you're not just getting proportionally better performance, you might unlock entirely new capabilities.

Scaling transformers is not merely about adding more parameters—it requires architectural innovations that maintain training stability, computational efficiency, and model capability as dimensions increase. The standard transformer has evolved substantially to enable billion-parameter scale.

Core Modifications for Stability

Pre-Layer Normalization (Pre-LN)

The original transformer used post-layer normalization, applying LayerNorm after the attention/FFN sublayers. This creates gradient scaling issues at extreme depths. Pre-LN applies normalization before each sublayer:

# Post-LN (original, unstable at depth)
x = x + Attention(LayerNorm(x))  # ❌

# Pre-LN (modern, stable)
x = x + Attention(LayerNorm(x))  # ✓ Norm before, not after
x = x + FFN(LayerNorm(x))

Pre-LN enables training models with 100+ layers where Post-LN would diverge.

RMSNorm

Root Mean Square Layer Normalization simplifies LayerNorm by removing mean centering:

$$\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{n}\sum x_i^2 + \epsilon}} \cdot \gamma$$

This reduces computation (~30% faster) while maintaining performance, important at scale.

DeepNorm

For extremely deep transformers (1000+ layers), DeepNorm modifies initialization and residual connections:

$$x_{l+1} = \text{LN}(\alpha x_l + \text{Sublayer}(x_l))$$

Where $\alpha > 1$ amplifies residual connections, and sublayer weights are scaled down accordingly.

Model Dimension Scaling

When scaling model size, practitioners must decide how to allocate additional parameters across dimensions:

Common configurations:

Training models with hundreds of billions of parameters requires sophisticated distributed systems. A single GPU cannot hold the model weights, let alone the gradients and optimizer states. Modern training infrastructure employs multiple parallelism strategies working in concert.

Memory Requirements

For a model with $N$ parameters using mixed precision training:

A 70B parameter model requires ~1.6 TB just for the training state—no single GPU has this capacity (as of 2024, the largest GPU has 80GB). This necessitates distributed training.

Parallelism Strategies

1. Data Parallelism (DP)

The simplest form: replicate the entire model on each device, split batches across devices, and average gradients.

• ✓ Scales linearly with devices for batch size
• ✗ Requires each device to hold full model
• ✗ Gradient synchronization becomes bottleneck at scale

2. Tensor Parallelism (TP)

Split individual operations across devices. For matrix multiplication $Y = XW$, split $W$ across devices:

         GPU 0          GPU 1
    [W_0, W_1]    [W_2, W_3]
    Y_0 = X @ W_0  Y_1 = X @ W_1  ...
    AllGather([Y_0, Y_1, Y_2, Y_3]) → Y

• ✓ Reduces per-device memory
• ✗ Requires high-bandwidth interconnect (NVLink)
• ✗ Typically limited to 8 devices (single node)

3. Pipeline Parallelism (PP)

Split model layers across devices, process micro-batches in pipeline fashion:

Time →
GPU 0 (Layers 0-7):   [B1] [B2] [B3] [B4]     [B1 bwd] [B2 bwd]...
GPU 1 (Layers 8-15):       [B1] [B2] [B3] [B4] [B1 bwd]...
GPU 2 (Layers 16-23):           [B1] [B2] [B3] [B4]...

• ✓ Works across nodes with commodity interconnect
• ✓ Reduces per-device memory
• ✗ Pipeline bubbles reduce efficiency (~50-80% utilization)

4. Sequence Parallelism (SP)

Split along sequence dimension, particularly for attention computation where memory scales as O(n²).

5. ZeRO (Zero Redundancy Optimizer)

Shards optimizer states, gradients, and optionally weights across data-parallel ranks:

Understanding the economics of large-scale training is essential for practitioners. The costs involved shape research priorities, influence architectural decisions, and determine who can participate in frontier AI development.

Compute Cost Estimation

Training compute is measured in FLOPs (floating-point operations). For a transformer:

$$C \approx 6ND$$

Where:

• $C$ = total training FLOPs
• $N$ = model parameters
• $D$ = training dataset size (tokens)

The factor of 6 comes from 2× (forward + backward) × 3 (matrix operations per layer).

Example calculations:

Hardware Utilization

Theoretical FLOPS ≠ achieved FLOPS. Model FLOPs Utilization (MFU) measures actual efficiency:

$$\text{MFU} = \frac{\text{Achieved FLOPS}}{\text{Peak Hardware FLOPS}}$$

A rule of thumb: Plan for ~40% MFU for large-scale training. A 50% MFU is excellent; 55%+ is exceptional.

Training Cost Components

Total Training Cost Breakdown (70B model, 2T tokens):

├── GPU Compute (H100 cloud): $7.5M
│   └── 30M GPU hours × $2.50/hr (negotiated rate)
│
├── Infrastructure: $1.5M
│   ├── Networking (InfiniBand/RoCE fabric)
│   ├── Storage (100+ TB high-speed)
│   └── Cooling and power overhead
│
├── Engineering: $1M
│   ├── Training infrastructure (6 months × 5 engineers)
│   ├── Data pipeline development
│   └── Monitoring and recovery systems
│
├── Data: $500K
│   ├── Acquisition and licensing
│   ├── Cleaning and preprocessing
│   └── Quality filtering
│
└── Experiments: $2M
    ├── Architecture ablations at smaller scales
    ├── Hyperparameter search
    └── Failed training runs

────────────────────────────────────────
TOTAL: ~$12.5M for production 70B model

The Compute Divide

The economics of scale create a stark divide in AI capability:

This concentration of capability raises important questions about:

• Democratization of AI research
• Replication and verification of results
• Safety testing by independent parties
• Regulatory oversight and accountability

Open-weight models (LLaMA, Mistral) partially address this by allowing smaller players to access frontier capabilities through fine-tuning rather than pre-training.

Given a fixed compute budget, how should resources be allocated? This is the central practical question of scaling. The answer depends on your goals, constraints, and downstream use cases.

The Scaling Decision Framework

Step 1: Define Your Objective

Different objectives favor different scales:

Step 2: Account for Inference

For deployment, inference cost often dominates:

$$\text{Total Cost} = C_{\text{training}} + N_{\text{inferences}} \times C_{\text{inference}}$$

For high-volume applications, the Chinchilla-optimal model may be suboptimal overall. A smaller, overtrained model might:

• Cost more to train (suboptimal compute allocation)
• Cost much less per inference
• Yield lower total cost at high query volume

Beyond Loss: What Scaling Optimizes

Scaling laws optimize for loss on the training distribution. But practitioners care about:

• Task-specific accuracy: May not correlate perfectly with loss
• Reasoning capability: May require architectural changes, not just scale
• Factual grounding: Can improve or plateau unpredictably with scale
• Instruction following: Depends heavily on fine-tuning, not just pre-training scale
• Safety properties: May require specific interventions beyond scale

The uncomfortable truth: Scale is necessary but not sufficient. A 100B model trained on poor data will underperform a 10B model trained on excellent data. Scale amplifies the effect of other factors—good data becomes great, bad data becomes terrible.

The Future of Scaling

Current trends suggest:

1. Diminishing returns approaching: The gains from pure scale may be tapering
2. Data becomes the bottleneck: High-quality data is increasingly scarce
3. Architectural innovation matters more: Post-transformer architectures may unlock new scaling regimes
4. Inference optimization critical: As models deploy widely, efficiency innovations become paramount
5. Specialization over monolithic scale: Multiple specialized models may outperform single general models

The era of 'just scale it' may be transitioning to an era of 'scale it intelligently'.

Transformer scaling has transformed machine learning from an incremental science to one characterized by dramatic capability leaps. Understanding scale is now essential knowledge for anyone working with modern language models.

What's next:

With an understanding of how transformers scale, we turn to what they learn during pre-training. The next page explores pre-training objectives—the loss functions and training tasks that give large language models their remarkable capabilities. We'll see how simple objectives like next-token prediction create models that understand language, reason about the world, and follow complex instructions.