convergence

Convergence occurs when training loss stops meaningfully improving, indicating the model has learned available patterns. **Signs of convergence**: Loss plateaus, validation metrics stable, gradient norms decrease, weight changes diminish. **Types**: **Loss convergence**: Training loss stops decreasing. **Validation convergence**: Validation metrics plateau (may diverge from train = overfitting). **Weight convergence**: Parameters stabilize. **Factors affecting convergence**: Learning rate (too high = no convergence, too low = slow), model capacity, data quality, optimization algorithm. **Convergence vs optimality**: Converged model not necessarily optimal. May be local minimum or saddle point. **Non-convergence issues**: Loss oscillating, NaN, increasing - indicate training problems. **Practical convergence**: Rarely reach true minimum. Stop when good enough or overfitting. **For LLMs**: Often train until compute budget exhausted rather than waiting for convergence. Scaling laws predict loss at given compute. **Monitoring**: Watch loss curves, compare train/val, check learning rate wasnt too aggressive. **Early stopping**: If validation stops improving, stop before full convergence to prevent overfitting.