Ensemble Methods Model Combination Collective Intelligence
# Ensemble Methods: Model Combination & Collective Intelligence
## Introduction & Motivation
Ensemble: combine multiple models for better prediction. Bagging: independent, average predictions. Boosting: sequential, focus on hard examples. Stacking: meta-learner aggregates base models. Applications: reduced variance, robustness, state-of-the-art performance.
Motivation: Individual models make errors; ensembles leverage diversity. Uncorrelated errors cancel via averaging.
Applications: Kaggle competitions, production ML, uncertainty estimation.
---
## Core Concepts & Theory
### Diversity
Key property: models make different errors. Correlation ↓ → ensemble gain ↑.
### Averaging vs Weighted Voting
Simple average: equal weights. Weighted: learned weights (cross-validation).
### Stacking
Meta-features: train base models; use outputs as features for meta-learner.
---
## Mathematical Formulation
Ensemble prediction (regression):
$$\hat{y} = \frac{1}{M} \sum_{m=1}^M f_m(x)$$
Weighted ensemble:
$$\hat{y} = \sum_{m=1}^M w_m f_m(x), \quad \sum w_m = 1$$
Bias-Variance Decomposition (ensemble error reduction):
$$ ext{Var}( ext{ensemble}) = \frac{1}{M} \overline{ ext{Var}} - \frac{1}{M^2} \sum_{m
eq n} ext{Cov}(f_m, f_n)$$
---
## Advanced Theory & Extensions
### Gradient Boosting
Iterative residual fitting; sequential error correction.
### Snapshot Ensembles
Cyclic learning rate; capture multiple local optima.
### Mixture of Experts
Gating network selects expert per input; adaptive ensemble.
---
## Computational Considerations
Prediction: O(M × forward_time) for M models.
Training Bagging: O(M × training_time) parallelizable.
Boosting: O(M × training_time) sequential.
Stacking: O(M × training_time) + meta-learner training.
---
## Practical Implementation Strategies
### Model Diversity
Train on different subsets, architectures, hyperparameters.
### Regularization
Individual models slightly underfitted; ensemble fits well.
### Early Stopping
Bag models trained for different epochs; natural diversity.
---
## Benchmark Datasets & Evaluation
ImageNet: Ensembles improve 1-3% accuracy over single models.
COCO (Detection): Top systems use 5-10 model ensembles.
NLP Tasks: Ensemble standard for competitions.
---
## Key Challenges & Limitations
### Computational Cost
Inference M× slower; deployment challenging.
### Correlation Between Models
Similar architectures → correlated errors; limited gain.
### Hyperparameter Tuning
Ensemble hyperparameters (weights, diversity) add complexity.
---
## Hyperparameter Tuning
Ensemble size M: 3-10; diminishing returns after 5.
Model diversity: Different architectures, training data, initialization.
Weighting strategy: Uniform vs learned via cross-validation.
---
## Real-World Applications & Case Studies
Kaggle: Top solutions typically 5-20 model ensembles.
XGBoost: Gradient boosting standard for tabular data.
Computer Vision: Ensemble of ResNets, EfficientNets, ViTs.
---
## Integration with Other Methods
Ensemble + Distillation → compress ensemble into single student.
Ensemble + Uncertainty → ensemble variance as uncertainty.
---
## Summary & Key Takeaways
Ensemble methods combine multiple models leveraging prediction diversity, achieving improved accuracy and robustness through averaging, boosting, or stacking.
Principles:
1. Diversity key: uncorrelated errors enable averaging benefit.
2. Bagging: independent, variance reduction via averaging.
3. Boosting: sequential, focus on hard examples.
4. Stacking: meta-learner aggregates base model outputs.
5. Ensemble gain ∝ (individual accuracy, diversity).
---
---
## Appendix: Practical Labs
### Lab 1: Ensemble Averaging
import torch
import torch.nn as nn
import numpy as np
def ensemble_average(predictions):
"""Average ensemble predictions"""
# predictions: [M, N] where M=models, N=samples
ensemble_pred = predictions.mean(dim=0)
return ensemble_pred
# Test
M, N = 10, 100 # 10 models, 100 samples
predictions = torch.rand(M, N)
ensemble = ensemble_average(predictions)
print(f"Ensemble shape: {ensemble.shape}")
assert ensemble.shape == (N,), "Should output N predictions"
assert (ensemble >= predictions.min()).all() and (ensemble <= predictions.max()).all(), "Should be in range"
print("✓ Ensemble averaging working")
if __name__ == "__main__":
print("Lab 1: Averaging - PASSED")### Lab 2: Weighted Voting
import torch
import torch.nn as nn
import numpy as np
def weighted_ensemble(predictions, weights):
"""Weighted ensemble with learned weights"""
# predictions: [M, N]
# weights: [M]
weights = weights / weights.sum() # Normalize
weighted_pred = (predictions * weights.view(-1, 1)).sum(dim=0)
return weighted_pred
# Test
M, N = 10, 100
predictions = torch.rand(M, N)
weights = torch.ones(M)
ensemble = weighted_ensemble(predictions, weights)
print(f"Weighted ensemble shape: {ensemble.shape}")
assert ensemble.shape == (N,), "Should output N predictions"
print("✓ Weighted ensemble working")
if __name__ == "__main__":
print("Lab 2: Weighted - PASSED")### Lab 3: Stacking
import torch
import torch.nn as nn
class StackingEnsemble(nn.Module):
def __init__(self, base_models, meta_model):
super().__init__()
self.base_models = nn.ModuleList(base_models)
self.meta_model = meta_model
def forward(self, x):
# Base model predictions
base_outputs = []
for model in self.base_models:
out = model(x)
base_outputs.append(out)
# Meta-features: concatenate base outputs
meta_features = torch.cat(base_outputs, dim=1)
# Meta-learner
ensemble_pred = self.meta_model(meta_features)
return ensemble_pred
# Test
# Create simple base models and meta-learner
base_models = [nn.Sequential(nn.Linear(10, 20), nn.ReLU(), nn.Linear(20, 5)) for _ in range(3)]
meta_model = nn.Linear(15, 5) # 3 models × 5 outputs = 15 features
stack = StackingEnsemble(base_models, meta_model)
x = torch.randn(8, 10)
output = stack(x)
print(f"Stacking output shape: {output.shape}")
assert output.shape == (8, 5), "Should output class logits"
print("✓ Stacking ensemble working")
if __name__ == "__main__":
print("Lab 3: Stacking - PASSED")### Lab 4: Diversity Measurement
import torch
import numpy as np
def compute_ensemble_diversity(predictions):
"""Measure prediction diversity (pairwise disagreement)"""
# predictions: [M, N, C] (models, samples, classes)
M = predictions.shape[0]
disagreement = 0
for i in range(M):
for j in range(i + 1, M):
pred_i = predictions[i].argmax(dim=1)
pred_j = predictions[j].argmax(dim=1)
disagreement += (pred_i != pred_j).float().mean()
num_pairs = M * (M - 1) / 2
diversity = disagreement / num_pairs
return diversity.item()
# Test
M, N, C = 5, 100, 10
predictions = torch.randn(M, N, C)
diversity = compute_ensemble_diversity(predictions)
print(f"Ensemble diversity: {diversity:.4f}")
assert 0 <= diversity <= 1, "Diversity should be in [0,1]"
print("✓ Diversity measurement working")
if __name__ == "__main__":
print("Lab 4: Diversity - PASSED")