boosting
**Boosting** is an **ensemble technique where models are trained sequentially, with each new model specifically targeting the errors made by the previous models** — unlike bagging (which trains independent models in parallel to reduce variance), boosting builds an additive chain where Model 2 focuses on the examples Model 1 got wrong, Model 3 focuses on what Models 1+2 still get wrong, and so on, progressively reducing both bias and variance to produce the most powerful supervised learning algorithms available for structured/tabular data (XGBoost, LightGBM, CatBoost).
**What Is Boosting?**
- **Definition**: A family of ensemble algorithms that convert many "weak learners" (models slightly better than random) into a single "strong learner" by training them sequentially — each weak learner focuses on the mistakes of the previous ones, and the final prediction is a weighted combination of all learners.
- **The Intuition**: Imagine a student (Model 1) takes a test and gets 30% of questions wrong. A tutor (Model 2) then specifically drills those 30% of hard questions. A second tutor (Model 3) drills the remaining errors. After 100 tutoring sessions, the student masters the entire test.
- **Key Difference from Bagging**: Bagging trains independent models to reduce variance. Boosting trains dependent models (each one depends on previous errors) to reduce bias and variance.
**How Gradient Boosting Works**
| Step | Process | What the Model Learns |
|------|---------|----------------------|
| 1. Train Tree 1 | Fit to the target y | Rough overall pattern |
| 2. Compute residuals | $r_1 = y - hat{y}_1$ (what Tree 1 got wrong) | Errors of Tree 1 |
| 3. Train Tree 2 | Fit to residuals $r_1$ | How to fix Tree 1's errors |
| 4. Update prediction | $hat{y} = hat{y}_1 + eta cdot hat{y}_2$ (η = learning rate) | Combined prediction |
| 5. Compute new residuals | $r_2 = y - (hat{y}_1 + eta cdot hat{y}_2)$ | Remaining errors |
| 6. Repeat N times | Each tree fixes the remaining residual | Progressively better fit |
**Boosting Algorithms Timeline**
| Algorithm | Year | Key Innovation | Status |
|-----------|------|---------------|--------|
| **AdaBoost** | 1997 | Reweight misclassified examples | Historic, still used for simple tasks |
| **Gradient Boosting (GBM)** | 1999 | Fit residuals using gradient descent in function space | Foundation of modern boosting |
| **XGBoost** | 2014 | Regularization + parallelized splits + missing value handling | Dominated Kaggle 2014-2020 |
| **LightGBM** | 2017 | Histogram binning + leaf-wise growth + GOSS | Fastest, most memory-efficient |
| **CatBoost** | 2017 | Native categorical encoding + ordered boosting | Best for categorical-heavy data |
**Critical Hyperparameters**
| Parameter | Effect | Too Low | Too High |
|-----------|--------|---------|----------|
| **n_estimators** (# trees) | Number of sequential models | Underfitting | Overfitting (mitigated by early stopping) |
| **learning_rate** (η) | Shrinkage per tree | Needs many more trees | Overfits quickly |
| **max_depth** | Individual tree complexity | Weak learners (good for boosting) | Each tree overfits |
| **subsample** | Fraction of data per tree | More regularization | Less regularization |
**Rule of thumb**: Use a low learning rate (0.01-0.1) with many trees (500-5000) and early stopping.
**Boosting is the most powerful supervised learning paradigm for structured data** — sequentially building an additive ensemble where each model corrects the errors of its predecessors, powering the XGBoost/LightGBM/CatBoost family that dominates tabular data competitions and production systems, with the critical requirement of proper learning rate and early stopping tuning to prevent the overfitting that sequential error-correction can cause.