iteration
**Training Terminology: Epochs, Batches, Iterations**
**Definitions**
**Batch**
A subset of training examples processed together:
```python
batch_size = 32 # Process 32 examples at once
```
**Iteration (Step)**
One forward + backward pass on a single batch:
```
1 iteration = process 1 batch = 1 gradient update
```
**Epoch**
One complete pass through the entire training dataset:
```
1 epoch = dataset_size / batch_size iterations
```
**Example Calculation**
```
Dataset: 10,000 examples
Batch size: 32
Iterations per epoch: 10,000 / 32 ≈ 312
Training for 3 epochs = 3 × 312 = 936 total iterations
```
**Effective Batch Size**
**Gradient Accumulation**
Process more examples before updating weights:
```python
accumulation_steps = 4
effective_batch_size = batch_size × accumulation_steps
# 32 × 4 = 128 effective batch size
```
Why use it:
- Fit larger effective batches on limited GPU memory
- More stable gradients
**Distributed Training**
With multiple GPUs:
```
global_batch_size = batch_size × num_gpus × accumulation_steps
```
**LLM Training Scale**
**Pretraining**
| Model | Tokens | Epochs | Notes |
|-------|--------|--------|-------|
| GPT-3 | 300B | <1 | Never repeats data |
| Llama 2 | 2T | ~1 | Some repetition |
| Llama 3 | 15T | ~4 on some data | Selective repetition |
**Fine-Tuning**
| Method | Typical Epochs |
|--------|----------------|
| SFT | 1-3 |
| LoRA | 1-5 |
| Full fine-tuning | 1-3 |
More epochs risk overfitting on small datasets.
**Training Code Example**
```python
num_epochs = 3
batch_size = 32
accumulation_steps = 4
for epoch in range(num_epochs):
for i, batch in enumerate(dataloader):
# Forward pass
loss = model(batch)
loss = loss / accumulation_steps
loss.backward()
# Update only every N steps
if (i + 1) % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad()
print(f"Completed epoch {epoch + 1}")
```
**Monitoring Progress**
```
Step 1000: loss=2.34, lr=0.0001
Step 2000: loss=1.87, lr=0.0001
Epoch 1/3 complete
...
```