Random Forest

A random forest fixes a single decision tree's biggest weakness — that it's unstable, changing noticeably if the training data shifts even slightly — by training hundreds of trees instead of one and letting them vote. Leo Breiman formalized the method in his 2001 paper "Random Forests," building on the bagging technique he'd introduced in 1996, and the trick that makes the ensemble actually work is forcing every tree to disagree with the others: each tree sees a different bootstrap sample of the training rows, and at every split, each tree is only allowed to consider a random subset of the features — scikit-learn's default is the square root of the total feature count — rather than all of them. Individually weaker and more random than a single well-tuned tree, the resulting trees make different mistakes, and averaging those mistakes away is the entire point.

Bagging and feature randomness attack two different sources of error, and a forest needs both. Bootstrap aggregating alone — training many trees on different random samples of the same full feature set — reduces variance, but if every tree still gets to pick from the same strongest features, the trees end up correlated with each other anyway, which limits how much averaging can help; restricting each split to a random subset of features forces different trees down genuinely different paths, decorrelating them further and squeezing more benefit out of the averaging step. Breiman's own benchmark results against boosting algorithms of the era showed this combination matching or beating the state of the art on classification and regression tasks while being far easier to tune.

Out-of-bag error gives a random forest a built-in validation set, for free. Because each tree is trained on a bootstrap sample — roughly 63% of the rows, sampled with replacement — the remaining 37% left out of that particular tree's training set can be used to evaluate that tree without touching a separate held-out set at all; averaging each row's out-of-bag predictions across every tree that didn't see it during training produces an honest performance estimate without the data-splitting overhead described in cross-validation. This was one of the original selling points of Breiman's method and remains a distinguishing feature of random forests compared to most other algorithms, which require a separate validation procedure to get the same kind of estimate.

A random forest's feature importance scores come essentially for free, and they are a different, more useful byproduct than most single-model outputs. By tracking how much each feature's splits reduce impurity on average across all the trees in the forest — or, more robustly, how much a model's accuracy drops when a feature's values are randomly shuffled (permutation importance) — a random forest ranks which inputs actually mattered to its predictions without any extra modeling work, which is why it's a standard first tool for exploratory feature analysis on a new tabular dataset even when the final production model ends up being something else.

The trade a random forest makes is the same one dropout makes inside a single neural network: deliberately injecting randomness — in which rows each tree sees, in which features each split can consider — costs each individual tree some accuracy, but buys enough diversity across hundreds of trees that the vote beats any one of them. It is ensembling made nearly automatic, which is exactly why it became, and largely remains, the default first thing to try on a new tabular dataset before reaching for anything more complicated.