Bias-Variance Tradeoff

The bias-variance tradeoff is the principle that a model's expected prediction error on new data can be split into error from wrong assumptions (bias), error from sensitivity to the particular training sample (variance) and unavoidable noise, and that making a model more flexible typically lowers bias while raising variance. It explains why a model that fits its training data perfectly can still predict poorly, and why the most flexible model is not automatically the best one.

For squared error, the decomposition is exact. The expected error at a point equals bias squared plus variance plus irreducible noise. Bias is the gap between the average prediction, taken over many hypothetical training sets, and the true value; variance is how much predictions scatter around their own average from one training set to another; irreducible noise is randomness in the target that no model could predict. The framing was popularized for neural networks by Stuart Geman, Elie Bienenstock and Rene Doursat in "Neural Networks and the Bias/Variance Dilemma" (1992) and is a core chapter of Hastie, Tibshirani and Friedman's "The Elements of Statistical Learning." The decomposition is exact for squared loss; for classification error, analogous but messier decompositions exist.

A polynomial-fitting example makes the two failure modes concrete. Suppose 20 noisy points are drawn from a smooth curve. A straight line (degree 1) misses the curve's shape however it is trained: high bias, low variance, so it underfits, and training and validation error are both high. A degree-3 polynomial follows the curve and varies modestly between samples. A degree-15 polynomial threads almost every training point, wiggling wildly between them: low bias, very high variance, so training error is tiny while validation error is large, which is overfitting. Plotting validation error against complexity gives the familiar U-shaped curve, whose minimum marks the practical sweet spot.

Many standard techniques are best understood as moving along this tradeoff. Regularization such as ridge or lasso penalties and dropout deliberately accepts some bias to shrink variance. Leo Breiman's bagging (1996) and random forests average many high-variance trees, cutting variance without raising bias much, while boosting builds a strong learner from weak high-bias ones by reducing bias step by step. More training data reduces variance but leaves bias untouched, which is why adding data rescues overfit models but not underfit ones. Cross-validation estimates where on the curve a model sits, and a learning curve comparing training and validation error diagnoses whether to collect data (a large gap indicates variance) or enlarge the model (both errors high indicates bias).

Modern deep learning complicates the classical U-shaped picture. Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal (2019) described "double descent": as capacity rises past the interpolation threshold, where a model fits the training data exactly, test error peaks and then falls again, and Preetum Nakkiran and colleagues (2019) documented it across deep networks. Heavily overparameterized networks can generalize well despite fitting training data perfectly, partly because training procedures favor simple solutions among the many that fit. This does not repeal the decomposition; it shows that parameter count alone is a poor proxy for effective complexity, and that the tradeoff is not a strict law forcing a loss of one error term for every gain in the other.

Common pitfalls follow from treating the tradeoff as a rule rather than a lens. Tuning on the test set understates variance, a single train-test split can give a misleadingly lucky estimate, and neglecting irreducible noise leads to chasing accuracy that does not exist. Another misconception is that variance means "randomness in the model"; it is sensitivity to which training sample was drawn, which is why a model with a fixed seed can still have high variance.

Read the bias-variance tradeoff as a diagnostic for why models fail on unseen data: bias means the model cannot represent the pattern, variance means it chases the noise in one sample, and good practice is finding the flexibility, data and regularization that balance them, remembering that overparameterized networks can break the simple U-shaped story.