Bagging Bootstrap Aggregation Variance Reduction

# Bagging: Bootstrap Aggregation & Variance Reduction

## Introduction & Motivation

Bagging trains base learners on bootstrap samples (random sampling with replacement) and averages predictions. Reduces variance without bias increase. Foundation for Random Forest, Extra Trees. Parallelizable; complements boosting.

Motivation: Single learner overfits. Multiple independent learners via resampling reduce variance. Averaging stable predictions; each base learner sees different data distribution.

Applications: Ensemble methods, outlier detection, robustness to dataset variations.

---

## Core Concepts & Theory

### Bootstrap Sample

Sample n points from n points with replacement. ~63.2% unique; 36.8% duplicates.

### Variance Reduction

Var[Ensemble] = (1 - ρ)σ²/m + ρσ² where ρ is correlation, m is learners.

---

## Summary & Key Takeaways

Bagging reduces variance via bootstrap sampling and averaging, foundation for ensemble methods.

Principles:
1. Bootstrap creates diverse datasets.
2. Independent learners reduce variance.
3. Averaging stabilizes predictions.
4. Works well with unstable learners (trees).
5. Parallel training efficient.

---

---

## Appendix: Practical Labs

### Lab 1: Bagging Classifier

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

base = DecisionTreeClassifier(max_depth=5)
bagging = BaggingClassifier(base_estimator=base, n_estimators=10, random_state=42)
bagging.fit(X_train, y_train)

y_pred = bagging.predict(X_test)
acc = accuracy_score(y_test, y_pred)

print(f"Accuracy: {acc:.4f}")
assert acc > 0.7, "Should exceed baseline"
print("✓ Bagging working")

if __name__ == "__main__":
 print("Lab 1: Bagging - PASSED")

### Lab 2: Bootstrap Samples

import numpy as np
from sklearn.utils import resample

X = np.arange(10).reshape(-1, 1)
y = np.arange(10)

n_samples = 100
bootstrap_samples = []

for _ in range(5):
 X_boot, y_boot = resample(X, y, n_samples=len(X), random_state=42)
 bootstrap_samples.append(X_boot.flatten())

print(f"Bootstrap samples created: {len(bootstrap_samples)}")
assert len(bootstrap_samples) == 5, "Should have 5 bootstrap samples"
print("✓ Bootstrap sampling working")

if __name__ == "__main__":
 print("Lab 2: Bootstrap - PASSED")

### Lab 3: OOB Error

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

bagging = BaggingClassifier(DecisionTreeClassifier(), n_estimators=10, oob_score=True, random_state=42)
bagging.fit(X_train, y_train)

oob_score = bagging.oob_score_

print(f"OOB score: {oob_score:.4f}")
assert 0.5 < oob_score < 1.0, "OOB should be reasonable"
print("✓ OOB error working")

if __name__ == "__main__":
 print("Lab 3: OOB - PASSED")

### Lab 4: Ensemble Size

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

for n in [5, 10, 20]:
 bagging = BaggingClassifier(DecisionTreeClassifier(), n_estimators=n, random_state=42)
 bagging.fit(X_train, y_train)
 acc = accuracy_score(y_test, bagging.predict(X_test))
 print(f"n_estimators={n}: {acc:.4f}")

print("✓ Ensemble size working")

if __name__ == "__main__":
 print("Lab 4: Ensemble Size - PASSED")