mini-batch online learning
**Mini-batch online learning** is a hybrid approach that combines aspects of batch and online learning by **updating the model with small batches of streaming data** rather than one example at a time or waiting for the complete dataset. It provides a practical middle ground for real-world systems.
**How It Works**
- **Accumulate**: Collect a small batch of new examples (e.g., 32–256 examples).
- **Compute Gradients**: Calculate the gradient of the loss across the mini-batch.
- **Update Model**: Apply the gradient update to model parameters.
- **Continue**: Move to the next mini-batch as data arrives.
**Why Mini-Batches Instead of Single Examples?**
- **Gradient Stability**: Single-example gradients are very noisy — they point in unpredictable directions. Mini-batch gradients average over multiple examples, providing a much more reliable update direction.
- **Hardware Efficiency**: GPUs are designed for parallel computation. Processing one example at a time wastes GPU capacity. Mini-batches fill the GPU's parallel compute units.
- **Learning Rate Sensitivity**: Single-example updates require very small learning rates to avoid instability. Mini-batches allow larger, more effective learning rates.
**Mini-Batch vs. Other Approaches**
| Approach | Batch Size | Update Frequency | Gradient Quality |
|----------|-----------|------------------|------------------|
| **Full Batch** | Entire dataset | Once per epoch | Best (exact gradient) |
| **Mini-Batch** | 32–256 | After each batch | Good (approximate gradient) |
| **Online (SGD)** | 1 | After each example | Noisy (stochastic) |
| **Mini-Batch Online** | 32–256 (streaming) | As data arrives | Good + adaptive |
**Applications**
- **Real-Time Model Adaptation**: Update recommendation models as new user interactions arrive in small batches.
- **Streaming Analytics**: Process log streams or sensor data in micro-batches.
- **Continual Fine-Tuning**: Periodically micro-fine-tune LLMs on recent data batches.
- **Federated Learning**: Clients compute updates on local mini-batches and share aggregated gradients.
**Practical Considerations**
- **Batch Size Selection**: Larger batches are more stable but introduce more latency before each update. Typical range: 32–256.
- **Learning Rate Scheduling**: Online mini-batch updates often benefit from warm-up and decay schedules.
- **Validation**: Periodically evaluate on a held-out set to detect degradation.
Mini-batch online learning is how most **production ML systems** actually operate — it balances the theoretical purity of online learning with the practical stability of batch training.