overfitting

**Overfitting and Underfitting** — the two fundamental failure modes in machine learning, related to the bias-variance tradeoff. **Underfitting (High Bias)** - Model is too simple to capture the data pattern - High training error AND high validation error - Fix: Increase model capacity, train longer, reduce regularization **Overfitting (High Variance)** - Model memorizes training data including noise - Low training error BUT high validation error - Fix: More data, regularization (dropout, weight decay), data augmentation, early stopping **Diagnosis** - Plot training vs. validation loss curves - If both high: underfitting - If training low but validation high: overfitting - If both low and converging: good fit **Bias-Variance Tradeoff** - Bias: Error from overly simple assumptions - Variance: Error from sensitivity to training data fluctuations - Total error = Bias$^2$ + Variance + Irreducible noise - Goal: Minimize total error, not just one component **Modern deep learning** often defies the classical tradeoff — very large models can generalize well with proper regularization (double descent phenomenon).