Training Terminology: Epochs, Batches, Iterations

Definitions

Batch A subset of training examples processed together:

batch_size = 32  # Process 32 examples at once

Iteration (Step) One forward + backward pass on a single batch:

1 iteration = process 1 batch = 1 gradient update

Epoch One complete pass through the entire training dataset:

1 epoch = dataset_size / batch_size iterations

Example Calculation

Dataset: 10,000 examples
Batch size: 32
Iterations per epoch: 10,000 / 32 ≈ 312

Training for 3 epochs = 3 × 312 = 936 total iterations

Effective Batch Size

Gradient Accumulation Process more examples before updating weights:

accumulation_steps = 4
effective_batch_size = batch_size × accumulation_steps
# 32 × 4 = 128 effective batch size

Why use it:

• Fit larger effective batches on limited GPU memory
• More stable gradients

Distributed Training With multiple GPUs:

global_batch_size = batch_size × num_gpus × accumulation_steps

LLM Training Scale

Pretraining

Fine-Tuning

More epochs risk overfitting on small datasets.

Training Code Example

num_epochs = 3
batch_size = 32
accumulation_steps = 4

for epoch in range(num_epochs):
    for i, batch in enumerate(dataloader):
        # Forward pass
        loss = model(batch)
        loss = loss / accumulation_steps
        loss.backward()

        # Update only every N steps
        if (i + 1) % accumulation_steps == 0:
            optimizer.step()
            optimizer.zero_grad()

    print(f"Completed epoch {epoch + 1}")

Monitoring Progress

Step 1000: loss=2.34, lr=0.0001
Step 2000: loss=1.87, lr=0.0001
Epoch 1/3 complete
...