Ensemble Methods Boosting Bagging Voting
# Ensemble Methods: Boosting, Bagging & Voting
## Introduction & Motivation
Ensemble Methods: combine multiple models. Reduce variance and bias. Boosting, bagging, stacking, voting. Applications: competitions (Kaggle), production systems, robustness.
Motivation: Diversity reduces error; stronger predictions.
Applications: Robustness, accuracy improvement, competitions.
---
## Core Concepts & Theory
### Bagging
Bootstrap aggregation; parallel training.
### Boosting
Sequential error correction; weighted samples.
### Voting
Aggregate predictions; hard or soft.
---
## Mathematical Formulation
Ensemble prediction (soft voting):
$$\hat{y} = \frac{1}{M} \sum_m f_m(x)$$
AdaBoost weight update:
$$w_t^{i} \propto w_{t-1}^{i} \cdot \exp(-\alpha_t y_i f_t(x_i))$$
Stacking meta-learner:
$$y = h(f_1(x), f_2(x), \ldots, f_M(x))$$
---
## Advanced Theory & Extensions
### Gradient Boosting
Gradient-based sequential improvement.
### Stacking
Learn meta-model on predictions.
### Blending
Holdout set for meta-learning.
---
## Computational Considerations
Bagging: O(M·model_time) parallel.
Boosting: O(M·model_time) sequential.
Stacking: O(M·CV_folds).
---
## Practical Implementation Strategies
### Diversity
Use different models; different subsets.
### Weighted Voting
Assign weights by performance.
### Cross-validation
Robust ensemble evaluation.
---
## Benchmark Datasets & Evaluation
Machine Learning Competitions: Kaggle.
Classification Benchmarks: UCI datasets.
MNIST/CIFAR: Standard ensembles.
---
## Key Challenges & Limitations
### Correlation
Correlated models don't diversify.
### Computational Cost
Multiple models; training overhead.
### Overfitting
Meta-learner can overfit.
---
## Hyperparameter Tuning
Number of models M: 5-100; accuracy-cost.
Model diversity: Different architectures.
Voting weight: Accuracy or per-model.
---
## Real-World Applications & Case Studies
Kaggle Competitions: Top solutions use ensembles.
Production ML: Robustness via diverse models.
Medicine: Multiple diagnostic models.
---
## Integration with Other Methods
Ensemble + Regularization → robustness.
Ensemble + Different features → diversity.
---
## Summary & Key Takeaways
Ensemble Methods via boosting, bagging, and voting enable robust predictions through model diversity and aggregation strategies.
Principles:
1. Diversity: decorrelated models.
2. Bagging: parallel aggregation.
3. Boosting: sequential correction.
4. Voting: soft or hard aggregation.
5. Stacking: meta-learner.
---
---
## Appendix: Practical Labs
### Lab 1: Soft Voting
import numpy as np
def soft_voting_ensemble(predictions_list, weights=None):
"""Soft voting: average predictions"""
num_models = len(predictions_list)
if weights is None:
weights = np.ones(num_models) / num_models
# Average probabilities
ensemble_pred = np.zeros_like(predictions_list[0])
for m, pred in enumerate(predictions_list):
ensemble_pred += weights[m] * pred
return ensemble_pred
# Test
np.random.seed(42)
pred1 = np.random.rand(10, 5)
pred2 = np.random.rand(10, 5)
pred3 = np.random.rand(10, 5)
ensemble = soft_voting_ensemble([pred1, pred2, pred3])
assert ensemble.shape == pred1.shape, "Ensemble shape"
print("✓ Soft voting working")
if __name__ == "__main__":
print("Lab 1: SoftVoting - PASSED")### Lab 2: Hard Voting
import numpy as np
def hard_voting_ensemble(predictions_list):
"""Hard voting: majority vote"""
num_models = len(predictions_list)
num_samples = len(predictions_list[0])
num_classes = predictions_list[0].shape[1]
votes = np.zeros((num_samples, num_classes))
for pred in predictions_list:
predicted_classes = np.argmax(pred, axis=1)
votes[np.arange(num_samples), predicted_classes] += 1
# Majority class
ensemble_pred = np.argmax(votes, axis=1)
return ensemble_pred
# Test
np.random.seed(42)
pred1 = np.random.rand(10, 5)
pred2 = np.random.rand(10, 5)
pred3 = np.random.rand(10, 5)
ensemble = hard_voting_ensemble([pred1, pred2, pred3])
assert ensemble.shape == (10,), "Ensemble shape"
assert np.all((ensemble >= 0) & (ensemble < 5)), "Valid classes"
print("✓ Hard voting working")
if __name__ == "__main__":
print("Lab 2: HardVoting - PASSED")### Lab 3: Boosting Weights
import numpy as np
def adaboost_weight_update(errors, alpha):
"""Update sample weights in AdaBoost"""
# Weight update: increase for misclassified
weights_new = np.exp(alpha * errors)
# Normalize
weights_new = weights_new / weights_new.sum()
return weights_new
# Test
np.random.seed(42)
errors = np.random.rand(100) # 0=correct, 1=wrong
alpha = 0.5
weights = adaboost_weight_update(errors, alpha)
assert weights.shape == errors.shape, "Weight shape"
assert np.isclose(weights.sum(), 1.0), "Normalized"
print("✓ AdaBoost update working")
if __name__ == "__main__":
print("Lab 3: AdaBoostUpdate - PASSED")### Lab 4: Ensemble Diversity Metric
import numpy as np
def ensemble_disagreement(predictions_list):
"""Measure disagreement (diversity) in ensemble"""
num_models = len(predictions_list)
num_samples = len(predictions_list[0])
# Pairwise disagreement
total_disagreement = 0
for i in range(num_samples):
preds_i = [pred[i] for pred in predictions_list]
pred_classes = [np.argmax(p) for p in preds_i]
# Count disagreement
for m1 in range(num_models):
for m2 in range(m1+1, num_models):
if pred_classes[m1] != pred_classes[m2]:
total_disagreement += 1
# Average
max_disagreement = num_samples * (num_models * (num_models - 1) / 2)
diversity = total_disagreement / max_disagreement if max_disagreement > 0 else 0
return diversity
# Test
np.random.seed(42)
pred1 = np.random.rand(10, 5)
pred2 = np.random.rand(10, 5)
pred3 = np.random.rand(10, 5)
diversity = ensemble_disagreement([pred1, pred2, pred3])
assert 0 <= diversity <= 1, "Diversity in [0,1]"
print("✓ Ensemble diversity working")
if __name__ == "__main__":
print("Lab 4: EnsembleDiversity - PASSED")