training cost
**Training Cost** refers to the **total computational resources, time, energy, and financial expense required to train a machine learning model** — for large language models this has grown from thousands of dollars (GPT-2 in 2019) to tens of millions of dollars (GPT-4 in 2023) to projected hundreds of millions (frontier models in 2025+), driven by scaling laws that show model quality improves predictably with more compute, creating a compute arms race that makes training cost the defining constraint of modern AI development.
**What Is Training Cost?**
- **Definition**: The total expense of computing all the gradient updates needed to train a model to convergence — encompassing GPU/TPU rental or ownership, electricity, networking infrastructure, cooling, engineering salaries, data acquisition, and failed experiments.
- **Why It Matters**: Training cost determines who can build frontier AI models. When training costs reach $100M+, only a handful of organizations (OpenAI, Google, Meta, Anthropic, xAI) can compete. This has profound implications for AI concentration, accessibility, and safety.
- **The Scaling Reality**: Every 10× increase in training compute has historically delivered meaningful capability improvements, incentivizing ever-larger training runs.
**Training Cost of Notable Models**
| Model | Year | Parameters | Training Compute | Estimated Cost | Hardware |
|-------|------|-----------|-----------------|---------------|----------|
| **GPT-2** | 2019 | 1.5B | ~1 PF-day | ~$50K | TPU v3 |
| **GPT-3** | 2020 | 175B | ~3,640 PF-days | ~$4.6M | V100 cluster |
| **PaLM** | 2022 | 540B | ~25,000 PF-days | ~$8-12M | TPU v4 |
| **LLaMA-2 70B** | 2023 | 70B | ~6,000 PF-days | ~$2-4M | A100 cluster |
| **GPT-4** | 2023 | ~1.8T (rumored) | ~100,000+ PF-days | ~$60-100M | A100 cluster |
| **Llama 3 405B** | 2024 | 405B | ~40,000 PF-days | ~$50-80M | H100 cluster |
| **Frontier models** | 2025+ | 1T+ | 500,000+ PF-days | ~$200-500M | H100/B200 clusters |
**Components of Training Cost**
| Component | Share of Total | Description |
|-----------|---------------|------------|
| **GPU/TPU Compute** | 60-80% | Accelerator rental or amortized purchase cost |
| **Electricity** | 5-15% | Power for compute + cooling (training Llama-3: ~30 GWh) |
| **Networking** | 5-10% | InfiniBand/NVLink for distributed training communication |
| **Engineering** | 5-15% | ML researchers, systems engineers ($200-500K/year each) |
| **Data** | 2-5% | Acquisition, cleaning, filtering, human annotation |
| **Failed Experiments** | 20-50% of total budget | Hyperparameter searches, diverged runs, restarts |
**Cost Optimization Strategies**
| Strategy | Savings | Trade-off |
|----------|---------|-----------|
| **Mixed Precision (FP16/BF16)** | ~2× throughput | Negligible quality loss with loss scaling |
| **Gradient Checkpointing** | ~60% memory reduction | 20-30% slower (recomputation) |
| **Data Parallelism** | Near-linear scaling to 1000s of GPUs | Communication overhead at extreme scale |
| **MoE Architecture** | 3-5× less compute per token for same quality | Higher total memory, routing complexity |
| **Efficient Architectures (FlashAttention)** | 2-3× attention speedup | Minor implementation effort |
| **Spot/Preemptible Instances** | 60-70% cost reduction | Requires checkpointing, interruption handling |
| **Distillation** | Train small model from large model outputs | Requires teacher model (already trained) |
**Training Cost is the defining constraint of modern AI development** — scaling from thousands to hundreds of millions of dollars as models grow in size and capability, determining which organizations can build frontier AI systems, driving the development of cost-reduction techniques from mixed precision to MoE architectures, and raising fundamental questions about the concentration, sustainability, and accessibility of advanced AI research.