pretraining
**Pre-training** is the **initial training phase where models learn general patterns from large unlabeled datasets** — creating foundation models that capture broad language or vision understanding, which can then be fine-tuned for specific downstream tasks with much less data and compute.
**What Is Pre-Training?**
- **Definition**: Training on large, general datasets before specialization.
- **Objective**: Learn universal representations (language patterns, visual features).
- **Scale**: Billions of tokens/images, weeks-months of compute.
- **Output**: Foundation model or base model.
**Why Pre-Training Works**
- **Transfer Learning**: General knowledge transfers to specific tasks.
- **Data Efficiency**: Fine-tuning needs much less task-specific data.
- **Emergence**: Capabilities arise from scale that can't be directly trained.
- **Cost Amortization**: One expensive pre-train, many cheap fine-tunes.
- **Better Representations**: Self-supervised learning captures structure.
**Pre-Training Objectives**
**Language Models**:
```
Objective | Description
----------------------|----------------------------------
Causal LM (GPT) | Predict next token: P(x_t | x_{
Pre-Training — Building Foundation Models
self-supervised learning on trillions of tokens to learn general representations
Pre-Training Pipeline
Web Corpus
CommonCrawl
Wikipedia
Books / Code
arXiv / Stack
10–15 T tokens
(deduplicated, filtered)
Self-Supervised Obj.
Next-Token Prediction
P(x_t | x_1...x_{t-1})
cross-entropy loss
causal LM (decoder-only)
or MLM (encoder: BERT)
Distributed Training
1000–16000 GPUs
FSDP + TP + PP
bf16 mixed precision
weeks to months
$10M–$100M+ compute
H100/B200 clusters
Foundation Model
general-purpose LLM
emergent capabilities
in-context learning
few-shot reasoning
→ fine-tune for any task
GPT-4, Llama 3, Claude
Scaling Laws (Chinchilla)
compute budget (FLOPs)
loss
7B
70B
405B
L(N,D) ∝ (N/N₀)^α + (D/D₀)^β — loss is predictable from scale
Pre-Training by the Numbers
Llama 3 405B:
15.6T tokens · 16K H100s · 54 days
3.8×10²⁵ FLOPs · context 8K→128K
GPT-4 (estimated):
~13T tokens · ~25K A100s · ~90 days
MoE ~1.8T params (220B active)
Chinchilla rule:
optimal: tokens ≈ 20× params
Pre-training is the most expensive and irreversible step — it determines what the model can ever learn to do.
```
**Code Example**:
```python
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
Trainer,
TrainingArguments,
DataCollatorForLanguageModeling,
)
# Load base architecture
model = AutoModelForCausalLM.from_config(config)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
# Pre-training data
def tokenize(examples):
return tokenizer(examples["text"], truncation=True, max_length=2048)
tokenized_dataset = dataset.map(tokenize, batched=True)
# Training arguments for pre-training
training_args = TrainingArguments(
output_dir="./pretrained-model",
per_device_train_batch_size=8,
gradient_accumulation_steps=16,
learning_rate=3e-4,
warmup_steps=2000,
max_steps=500000,
bf16=True,
save_steps=5000,
)
# Data collator for causal LM
data_collator = DataCollatorForLanguageModeling(
tokenizer=tokenizer, mlm=False
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_dataset,
data_collator=data_collator,
)
trainer.train()
```
**Pre-Training vs. Fine-Tuning**
```
Aspect | Pre-Training | Fine-Tuning
----------------|-------------------|------------------
Data | Billions tokens | Thousands-millions
Compute | $1M+ | $10-$10K
Time | Weeks-months | Hours-days
Objective | General LM | Task-specific
Who does it | AI labs | Everyone
Learning rate | Higher (1e-4) | Lower (1e-5)
```
Pre-training is **the foundation of modern AI** — by investing massive resources once to create powerful general-purpose models, the community enables efficient specialization through fine-tuning, democratizing access to capabilities that would be impossible to train from scratch.