batch learning
**Batch learning** (also called **offline learning**) is the traditional machine learning paradigm where the model is trained on a **fixed, complete dataset** gathered before training begins. The model sees all training data (potentially in multiple epochs) and does not update after deployment.
**How Batch Learning Works**
- **Collect**: Gather all training data before training begins.
- **Train**: Process the entire dataset (typically multiple passes/epochs), optimizing parameters on the complete dataset.
- **Evaluate**: Test on held-out validation and test sets.
- **Deploy**: Deploy the fixed, trained model for inference.
- **Refresh** (optional): Periodically retrain from scratch on updated data.
**Advantages**
- **Optimization Quality**: Multiple passes over the complete dataset allow thorough optimization. Better convergence guarantees than online learning.
- **Reproducibility**: Fixed dataset and deterministic shuffling make results reproducible.
- **Well-Understood Theory**: Standard ML theory (VC dimension, PAC learning, bias-variance tradeoff) is built on batch learning assumptions.
- **Easy Evaluation**: Clear train/validation/test splits enable robust performance estimation.
- **Simpler Implementation**: No need to handle streaming data, concept drift, or incremental updates.
**Disadvantages**
- **Staleness**: The model's knowledge is frozen at training time. It doesn't learn from new data until retrained.
- **Retraining Cost**: Full retraining on growing datasets becomes increasingly expensive.
- **Data Storage**: Must store the entire training dataset.
- **Latency**: There's a delay between new data becoming available and the model incorporating it.
**Batch Learning for LLMs**
- **Pre-Training**: LLM pre-training is fundamentally batch learning — models are trained on a fixed corpus (Common Crawl, Wikipedia, books, code).
- **Knowledge Cutoff**: The "knowledge cutoff date" of LLMs is a direct consequence of batch learning — the model only knows what was in its training data.
- **Periodic Retraining**: Major model releases (GPT-3 → GPT-4 → GPT-4o) represent retraining cycles with updated data.
**When to Use Batch Learning**
- Data distribution is relatively stable.
- Complete datasets are available before training.
- High accuracy and well-calibrated predictions are critical.
- Retraining frequency (weekly, monthly) matches data staleness tolerance.
Batch learning remains the **dominant paradigm** for most ML applications, including LLM pre-training, because it provides the most stable and well-understood training dynamics.