Training a Large Language Model (LLM) is one of the most resource-intensive tasks in machine learning. Models like GPT-4 cost over $100 million to train. This guide breaks down the complete process into understandable steps.
The Three Phases of LLM Training
- Pre-training - Learn language from massive text data
- Supervised Fine-tuning (SFT) - Learn to follow instructions
- RLHF - Align with human preferences
Phase 1: Pre-training
Pre-training teaches the model to understand and generate language by predicting the next token in a sequence. This is where 99% of compute is spent.
1.1 Data Collection
LLMs are trained on trillions of tokens from diverse sources:
- Common Crawl - Web pages (filtered for quality)
- Wikipedia - High-quality encyclopedic knowledge
- Books - Fiction and non-fiction (Books3, etc.)
- Code - GitHub, StackOverflow
- Scientific papers - arXiv, PubMed
- Curated datasets - RedPajama, The Pile, FineWeb
Example: Llama 3 Training Data
Meta's Llama 3 was trained on 15 trillion tokens—7x more than Llama 2. Data quality is now more important than quantity.
1.2 Data Preprocessing
- Deduplication - Remove duplicate documents (MinHash, SimHash)
- Quality filtering - Remove low-quality, toxic, or copyrighted content
- Language filtering - Keep target languages (often English-focused)
- Tokenization - Convert text to tokens using BPE or SentencePiece
1.3 Tokenization
Text is converted into tokens—subword units the model can process:
Common tokenizers:
- BPE (Byte Pair Encoding) - Used by GPT models
- SentencePiece - Used by Llama, T5
- Tiktoken - OpenAI's fast BPE implementation
1.4 Model Architecture
Most LLMs use the Transformer decoder architecture:
| Model | Parameters | Layers | Hidden Size |
|---|---|---|---|
| GPT-3 | 175B | 96 | 12,288 |
| Llama 3 70B | 70B | 80 | 8,192 |
| GPT-4 (est.) | 1.8T (MoE) | 120 | ~16,000 |
1.5 Training Infrastructure
Pre-training requires massive GPU clusters:
Hardware Requirements (70B Model)
- GPUs: 2,000+ NVIDIA H100s
- Memory: 80GB HBM3 per GPU
- Interconnect: NVLink + InfiniBand
- Training time: 2-4 months
- Cost: $20-50 million (compute only)
1.6 Training Techniques
- Mixed Precision (BF16/FP16) - Reduce memory, speed up training
- Data Parallelism - Distribute batches across GPUs
- Tensor Parallelism - Split layers across GPUs
- Pipeline Parallelism - Split model stages
- ZeRO Optimization - Shard optimizer states (DeepSpeed)
- Gradient Checkpointing - Trade compute for memory
Phase 2: Supervised Fine-tuning (SFT)
Pre-trained models are "completion machines"—they continue text but don't follow instructions. SFT teaches them to be helpful assistants.
SFT Process
- Collect instruction-response pairs (10K-100K examples)
- Format as conversations: User → Assistant
- Fine-tune on this data with lower learning rate
- Typically 1-3 epochs, few hours to days
Example SFT data format:
SFT Data Sources
- Human-written - Expensive but highest quality
- ShareGPT - User conversations with ChatGPT
- Synthetic - Generated by stronger models (GPT-4)
- Open datasets - Alpaca, Dolly, OpenAssistant
Phase 3: RLHF (Reinforcement Learning from Human Feedback)
RLHF aligns the model with human preferences—making it helpful, harmless, and honest.
3.1 Train a Reward Model
- Generate multiple responses to the same prompt
- Human annotators rank responses from best to worst
- Train a model to predict these rankings
3.2 PPO Training
Use Proximal Policy Optimization (PPO) to fine-tune the LLM:
- LLM generates a response
- Reward model scores the response
- Update LLM to maximize reward while staying close to SFT model
RLHF Alternatives
- DPO (Direct Preference Optimization) - No reward model needed
- RLAIF - Use AI instead of human feedback
- Constitutional AI - Self-critique and revision
Cost Breakdown
| Phase | Compute Cost | Data Cost | Time |
|---|---|---|---|
| Pre-training (70B) | $20-50M | $1-5M | 2-4 months |
| SFT | $10-100K | $50-500K | Days |
| RLHF | $100K-1M | $500K-2M | Weeks |
Training Your Own LLM
For most use cases, fine-tuning an existing model is more practical:
Practical Options
- LoRA/QLoRA - Fine-tune with minimal resources (single GPU)
- RAG - Add knowledge without fine-tuning
- Base models - Llama 3, Mistral, Qwen (free to use)
- APIs - OpenAI fine-tuning API for custom models
Key Takeaways
- LLM training has 3 phases: Pre-training → SFT → RLHF
- Pre-training costs millions; fine-tuning costs thousands
- Data quality matters more than quantity
- Most applications should fine-tune existing models, not train from scratch
- LoRA/QLoRA makes fine-tuning accessible on consumer hardware
Explore AI Terminology
Browse 5,000+ AI definitions in our comprehensive glossary.
Browse AI Glossary →