epoch
**Epoch, Batch, and Iteration** are **the fundamental time-keeping units of neural network training** — defining how training data is organized, processed, and used to update model parameters. Understanding their relationship is essential for configuring training runs, interpreting loss curves, setting learning rate schedules, and comparing results across different research papers and implementations.
**Core Definitions**
**Epoch** — one complete pass through the entire training dataset.
- Every training sample has been seen exactly once
- After each epoch, the dataset is typically shuffled before the next pass
- Most vision models train for tens to hundreds of epochs; ResNet-50 on ImageNet trains for 90 epochs
- LLM pre-training often completes well under 1 epoch (the dataset is larger than the compute budget can exhaust)
**Mini-batch (Batch)** — a subset of training samples processed together in a single forward-backward pass.
- All samples in the batch are processed in parallel on the GPU
- The loss is averaged over all samples in the batch before backpropagation
- Typical sizes: 32, 64, 128, 256 for vision; 2M-16M tokens for LLM training
- Smaller batches: more gradient noise, potentially better generalization, less parallelism
- Larger batches: less noise, more stable training, better hardware utilization
**Iteration (Step)** — one weight update from one mini-batch.
- One iteration = one forward pass + one backward pass + one optimizer step
- This is the fundamental unit of training time: most training logs report metrics per step
- Learning rate schedulers count steps, not epochs
**The Mathematical Relationship**
$$\text{Iterations per epoch} = \left\lceil \frac{N_{\text{train}}}{B} \right\rceil$$
$$\text{Total iterations} = \text{Epochs} \times \text{Iterations per epoch}$$
Example: ImageNet (1.28M images), batch size 256, 90 epochs:
- Iterations per epoch: $1{,}280{,}000 / 256 = 5{,}000$
- Total iterations: $90 \times 5{,}000 = 450{,}000$
**Training Loop Structure**
```python
for epoch in range(num_epochs): # outer loop: dataset passes
dataloader.shuffle() # randomize order each epoch
for batch_x, batch_y in dataloader: # inner loop: mini-batches
optimizer.zero_grad() # clear previous gradients
predictions = model(batch_x) # forward pass
loss = criterion(predictions, batch_y) # compute loss
loss.backward() # backpropagate gradients
optimizer.step() # update weights
iteration += 1 # count step
validate(model) # evaluate after each epoch
```
This triple structure — dataset → epoch → batch → iteration — is the heartbeat of all neural network training.
**LLM Pre-training: Token-Based Counting**
Large language models redefine these concepts around tokens rather than samples:
- **Token batch**: Global batch size measured in tokens, not samples. LLaMA 3 used 4M tokens/batch; GPT-3 used 3.2M tokens/batch
- **Training tokens**: Total tokens processed = global batch size × total steps. LLaMA 3.1 was trained on 15 trillion tokens.
- **Epoch**: LLM training rarely completes even 1 epoch — the Chinchilla paper shows that for compute-optimal training, models should be trained on 20× more tokens than parameters, which for a 70B model means 1.4T tokens — most datasets aren't that large, so epochs are rare
**Learning Rate Scheduling and Steps**
Learning rate schedules operate on steps, not epochs:
| Schedule Type | Step Behavior | Used In |
|--------------|---------------|--------|
| **Linear warmup** | LR increases from 0 to $\eta_{max}$ over first $T_{warmup}$ steps | LLMs, transformers |
| **Cosine decay** | LR follows cosine from $\eta_{max}$ to $\eta_{min}$ over $T$ steps | GPT, LLaMA, most modern LLMs |
| **Step decay** | Multiply by 0.1 at milestone steps/epochs | ResNet ImageNet training |
| **Constant** | Fixed LR throughout | Simple baselines, evaluation |
Standard LLM training: 1-2% warmup steps, then cosine decay for remainder.
**Shuffling and Data Order**
Shuffle training data before each epoch:
- Prevents the model from learning spurious order-dependent patterns
- Ensures different batches each epoch, improving sample diversity
- For LLM training: documents are shuffled and concatenated (then split into fixed-length sequences), so epoch boundaries are approximate
**Gradient Accumulation and Virtual Batch Size**
When GPU memory limits batch size, gradient accumulation enables larger **virtual** (effective) batches:
$$B_{\text{effective}} = B_{\text{micro}} \times N_{\text{accum}} \times N_{\text{GPUs}}$$
One **iteration** in terms of weight updates corresponds to $N_{\text{accum}}$ forward-backward micro-steps. Training logs typically count optimizer steps (weight updates), not micro-steps.
**Practical Guidance**
- **How many epochs for my task?**
- Image classification (from scratch): 90-300 epochs
- Fine-tuning a pre-trained vision model: 10-30 epochs
- SFT fine-tuning an LLM: 1-3 epochs over instruction data
- LLM pre-training: <1 epoch (token-budget limited)
- **How should I pick batch size?**
- Use the largest batch that fits in memory
- Scale learning rate proportionally: $\eta \propto \sqrt{B}$ (square root rule) or $\eta \propto B$ (linear scaling for SGD)
- For LLMs: target 1M-16M tokens/batch for stable training
- **Should I care about epochs or steps?**
- For fixed datasets: epochs make sense (you know when data is exhausted)
- For streaming/large-scale training: steps are the natural unit (you set a compute budget)
- Learning rate schedules always use steps
- Early stopping monitors validation metrics after each epoch
Epoch, batch, and iteration are the vocabulary of training — every training script, research paper, and debugging conversation uses these terms, and their precise relationship determines how learning rate, regularization, and compute budget interact.