Ensemble Methods Bagging Boosting Stacking
# Ensemble Methods: Bagging, Boosting & Stacking
## Introduction & Motivation
Ensemble methods: combine multiple models. Bagging: parallel training; reduce variance. Boosting: sequential training; reduce bias. Stacking: meta-learner on base models. Applications: classification, regression, improved robustness.
Motivation: Single model prone to overfitting. Ensemble averaging reduces error through diversity.
Applications: Kaggle competitions, production systems.
---
## Core Concepts & Theory
### Bagging
Bootstrap aggregation; parallel models.
### Boosting
Sequential; weight examples; reduce bias.
### Stacking
Meta-learner combines base learners.
---
## Mathematical Formulation
Bagging prediction:
$$\hat{y} = \frac{1}{M} \sum_{m=1}^M f_m(x)$$
Boosting (AdaBoost):
$$F(x) = ext{sign}\left(\sum_{m=1}^M \alpha_m f_m(x)
ight)$$
Stacking:
$$\hat{y} = g(f_1(x), f_2(x), \ldots, f_M(x))$$
where g = meta-learner.
---
## Advanced Theory & Extensions
### Random Forests
Bagging decision trees; feature subsampling.
### Gradient Boosting
Fit residuals iteratively; XGBoost, LightGBM.
### Voting Ensembles
Majority vote; weighted voting.
---
## Computational Considerations
Bagging: O(M · model_time) parallel; M = num models.
Boosting: O(M · model_time) sequential.
Stacking: Training + meta-training.
---
## Practical Implementation Strategies
### Diversity
Ensure model diversity; different architectures/data.
### Early Stopping
Monitor ensemble performance; stop adding models.
### Hyperparameter Tuning
Tune base models independently; meta-learner.
---
## Benchmark Datasets & Evaluation
MNIST: Bagging improves generalization.
ImageNet: Ensemble of CNNs; state-of-the-art.
Tabular Data: XGBoost standard; excellent performance.
---
## Key Challenges & Limitations
### Computational Cost
M models; M× training time.
### Diversity-Accuracy Tradeoff
Too similar → no benefit; too different → poor individual performance.
### Correlation
Correlated errors → less benefit from ensemble.
---
## Hyperparameter Tuning
Bagging sample fraction: 0.5-1.0; typically 1.0 (with replacement).
Number of models M: 10-100; diminishing returns.
Boosting learning rate: 0.01-0.1; step size.
---
## Real-World Applications & Case Studies
Random Forests: Classification standard; robust.
Gradient Boosting: Tabular data; XGBoost competitive.
Neural Ensemble: Multiple CNNs averaged; accuracy boost.
---
## Integration with Other Methods
Ensemble + Regularization → complementary.
Ensemble + Data Aug → improved robustness.
---
## Summary & Key Takeaways
Ensemble methods via bagging, boosting, and stacking combine multiple models to reduce variance and bias through diversity and aggregation.
Principles:
1. Bagging: parallel; reduce variance.
2. Boosting: sequential; reduce bias.
3. Stacking: meta-learner.
4. Diversity: key for benefit.
5. Aggregation: averaging or voting.
---
---
## Appendix: Practical Labs
### Lab 1: Bagging Classifier
import numpy as np
class BaggingEnsemble:
def __init__(self, base_model_fn, num_models=10):
self.base_model_fn = base_model_fn
self.num_models = num_models
self.models = []
def fit(self, X, y):
"""Train bagging ensemble"""
n_samples = len(X)
for _ in range(self.num_models):
# Bootstrap sample
indices = np.random.choice(n_samples, size=n_samples, replace=True)
X_boot = X[indices]
y_boot = y[indices]
# Train model
model = self.base_model_fn()
model.fit(X_boot, y_boot)
self.models.append(model)
def predict(self, X):
"""Average predictions"""
predictions = np.array([model.predict(X) for model in self.models])
return predictions.mean(axis=0)
# Test (simplified)
np.random.seed(42)
class DummyModel:
def fit(self, X, y): pass
def predict(self, X): return np.random.randn(len(X))
ensemble = BaggingEnsemble(DummyModel, num_models=10)
X = np.random.randn(100, 10)
y = np.random.randint(0, 2, 100)
ensemble.fit(X, y)
preds = ensemble.predict(X[:10])
assert preds.shape == (10,), "Prediction shape"
print("✓ Bagging ensemble working")
if __name__ == "__main__":
print("Lab 1: BaggingEnsemble - PASSED")### Lab 2: Boosting (AdaBoost-like)
import numpy as np
def boost_classifier(X, y, num_rounds=10):
"""Simplified boosting"""
n_samples = len(X)
# Initialize weights
weights = np.ones(n_samples) / n_samples
models = []
alphas = []
for _ in range(num_rounds):
# Train model on weighted data
indices = np.random.choice(n_samples, size=n_samples, replace=True, p=weights)
# Prediction (simplified: random)
y_pred = np.random.randint(0, 2, n_samples)
# Error rate
error = (y_pred != y).sum() / n_samples
if error >= 0.5 or error == 0:
break
# Model weight
alpha = 0.5 * np.log((1 - error) / (error + 1e-8))
alphas.append(alpha)
models.append(y_pred)
# Update weights
weights = weights * np.exp(-alpha * y * (2 * y_pred - 1))
weights = weights / weights.sum()
return models, alphas
# Test
np.random.seed(42)
X = np.random.randn(100, 10)
y = np.random.randint(0, 2, 100) * 2 - 1 # {-1, 1}
models, alphas = boost_classifier(X, y, num_rounds=10)
assert len(models) > 0, "Models trained"
print("✓ Boosting working")
if __name__ == "__main__":
print("Lab 2: Boosting - PASSED")### Lab 3: Stacking
import numpy as np
def stacking_ensemble(X_train, y_train, X_test, base_models, meta_model):
"""Train and predict with stacking"""
# Train base models and generate meta-features
meta_features_train = []
for model in base_models:
model.fit(X_train, y_train)
pred = model.predict(X_train)
meta_features_train.append(pred)
X_meta_train = np.column_stack(meta_features_train)
# Train meta-model
meta_model.fit(X_meta_train, y_train)
# Test prediction
meta_features_test = []
for model in base_models:
pred = model.predict(X_test)
meta_features_test.append(pred)
X_meta_test = np.column_stack(meta_features_test)
# Meta prediction
y_pred = meta_model.predict(X_meta_test)
return y_pred
# Test (simplified)
np.random.seed(42)
class SimpleModel:
def fit(self, X, y): pass
def predict(self, X): return np.random.randn(len(X))
X_train = np.random.randn(100, 10)
y_train = np.random.randint(0, 2, 100)
X_test = np.random.randn(20, 10)
base_models = [SimpleModel() for _ in range(3)]
meta_model = SimpleModel()
preds = stacking_ensemble(X_train, y_train, X_test, base_models, meta_model)
assert preds.shape == (20,), "Prediction shape"
print("✓ Stacking working")
if __name__ == "__main__":
print("Lab 3: Stacking - PASSED")### Lab 4: Ensemble Diversity
import numpy as np
def measure_ensemble_diversity(predictions):
"""Measure pairwise prediction diversity"""
num_models = len(predictions)
pairwise_diffs = []
for i in range(num_models):
for j in range(i + 1, num_models):
# Disagreement rate
diff = (predictions[i] != predictions[j]).mean()
pairwise_diffs.append(diff)
return np.mean(pairwise_diffs) if pairwise_diffs else 0
# Test
np.random.seed(42)
predictions = [
np.random.randint(0, 2, 100),
np.random.randint(0, 2, 100),
np.random.randint(0, 2, 100)
]
diversity = measure_ensemble_diversity(predictions)
assert 0 <= diversity <= 1, "Diversity in [0,1]"
print("✓ Ensemble diversity measurement working")
if __name__ == "__main__":
print("Lab 4: EnsembleDiversity - PASSED")