bagging (bootstrap aggregating)
**Bagging (Bootstrap Aggregating)** is an ensemble learning method that improves model accuracy and stability by training multiple instances of the same base learner on different bootstrap samples (random samples with replacement) of the training data, then aggregating their predictions through voting (classification) or averaging (regression). Introduced by Leo Breiman in 1996, bagging reduces variance without increasing bias, making it particularly effective for high-variance, low-bias base learners.
**Why Bagging Matters in AI/ML:**
Bagging provides **reliable variance reduction** that stabilizes predictions from unstable models (decision trees, neural networks, k-NN with low k), consistently improving generalization performance while providing natural out-of-bag estimation for validation.
• **Bootstrap sampling** — Each base learner trains on a bootstrap sample of size N drawn with replacement from the original N training examples; each sample contains ~63.2% unique examples (by the birthday paradox), with ~36.8% left out as "out-of-bag" (OOB) examples
• **Variance reduction** — For N models with prediction variance σ² and pairwise correlation ρ, bagging reduces variance to (ρ·σ² + (1-ρ)·σ²/N); the benefit is greatest when ρ is small (diverse models) and diminishes for highly correlated predictors
• **Out-of-bag estimation** — Each training example is excluded from ~36.8% of bootstrap samples; using these models to predict on their OOB examples provides a nearly unbiased estimate of generalization error without needing a separate validation set
• **Parallel training** — All base learners train independently on their bootstrap samples, enabling embarrassingly parallel training across multiple GPUs, machines, or nodes with no communication overhead during training
• **Random Forest extension** — Random Forest extends bagging by additionally sampling a random subset of features at each split (√p for classification, p/3 for regression), further decorrelating trees to maximize ensemble benefit beyond standard bagging
| Property | Value | Notes |
|----------|-------|-------|
| Base Learners | 10-1000 (typically 100-500) | Diminishing returns beyond ~200 |
| Bootstrap Fraction | ~63.2% unique per sample | 1 - (1 - 1/N)^N ≈ 1 - 1/e |
| OOB Sample Fraction | ~36.8% per model | Free validation estimate |
| Aggregation | Majority vote / average | Soft voting (probabilities) preferred |
| Variance Reduction | Up to 1/N (uncorrelated) | Typically 40-80% reduction |
| Bias Change | None (same base learner) | Bagging does not reduce bias |
| Training Parallelism | Fully parallel | No inter-model dependencies |
**Bagging is a foundational ensemble technique that reliably improves prediction stability and accuracy by training diverse models on bootstrap samples and averaging their outputs, providing variance reduction with parallel training efficiency and free out-of-bag error estimation that makes it indispensable for building robust, production-quality machine learning systems.**