Ensemble Methods for Robust Predictions
# Ensemble Methods for Robust Predictions
## Introduction & Motivation
Ensemble methods combine multiple models to achieve superior predictive performance, robustness, and uncertainty quantification. Critical for high-stakes engineering decisions where individual model failures must be minimized through diverse model combinations.
Motivation: Use ensembles for robust and calibrated predictions.
Applications: Risk prediction, robust forecasting, uncertainty estimation, decision support.
---
## Core Concepts & Theory
### Base Learners
Diverse model components.
### Aggregation
Combining predictions.
### Bootstrap
Resampling for diversity.
### Voting/Stacking
Ensemble architectures.
---
## Mathematical Formulation
Voting:
$$\hat{y} = \frac{1}{M} \sum_{m=1}^M f_m(x)$$
Boosting:
$$f(x) = \sum_{m=1}^M \alpha_m h_m(x)$$
Bagging Variance Reduction:
$$ ext{Var}[\hat{f}] = \frac{1}{M^2}\sum ext{Var}[f_m]$$
---
## Advanced Theory & Extensions
### Stacking
Meta-learner combination.
### Adaboost
Adaptive boosting weights.
### Random Forests
Bootstrap aggregating.
---
## Computational Considerations
Bagging: O(M·N·log N) for M models, N samples.
Boosting: Sequential O(M·N).
Stacking: O(M²·N).
---
## Practical Implementation Strategies
### Base Learner Selection
Diverse architectures.
### Weighting Schemes
Model-specific confidence.
### Cross-validation
Avoiding overfitting.
---
## Benchmark Datasets & Evaluation
UCI Repository: Classification tasks.
Kaggle: Competition benchmarks.
Domain Datasets: Engineering applications.
---
## Key Challenges & Limitations
### Computational Cost
Multiple model training.
### Model Correlation
Redundant predictions.
### Interpretability
Understanding ensemble decisions.
---
## Hyperparameter Tuning
Number of models: 10-100.
Base learner type: Diverse.
Aggregation weights: Learned or uniform.
---
## Real-World Applications & Case Studies
Medical Diagnosis: Multi-model consensus.
Process Control: Robust predictions.
Financial Forecasting: Ensemble forecasts.
---
## Integration with Other Methods
Ensembles + deep learning; + uncertainty quantification; + optimization.
---
## Summary & Key Takeaways
Ensembles combine models for robustness.
Principles:
1. Diversity: Uncorrelated models.
2. Aggregation: Combining predictions.
3. Training: Parallel or sequential.
4. Weighting: Confidence-based combination.
5. Evaluation: Robustness assessment.
---
## Appendix: Practical Labs
### Lab 1: Voting Ensemble
import numpy as np
class VotingEnsemble:
def __init__(self, n_models=5):
self.models = [np.random.randn(10, 1) for _ in range(n_models)]
def train_models(self, X, y):
"""Train base learners"""
for i, model in enumerate(self.models):
# Simplified linear regression
self.models[i] = np.linalg.lstsq(X, y, rcond=None)[0]
def predict(self, X):
"""Ensemble prediction via voting"""
predictions = np.array([X @ m for m in self.models])
return np.mean(predictions, axis=0)
ensemble = VotingEnsemble(n_models=5)
X_train = np.random.randn(50, 10)
y_train = X_train[:, 0] + np.random.randn(50) * 0.1
ensemble.train_models(X_train, y_train)
X_test = np.random.randn(5, 10)
predictions = ensemble.predict(X_test)
print(f"✓ Ensemble predictions: {predictions[:3]}")### Lab 2: Boosting
import numpy as np
class AdaBoost:
def __init__(self, n_models=10):
self.n_models = n_models
self.models = []
self.alphas = []
def train(self, X, y):
"""AdaBoost training"""
weights = np.ones(len(X)) / len(X)
for _ in range(self.n_models):
# Weighted model training
model = np.linalg.lstsq(X * np.sqrt(weights[:, None]), y * np.sqrt(weights), rcond=None)[0]
predictions = X @ model
errors = (predictions != y).astype(int)
error_rate = np.sum(weights * errors)
if error_rate > 0.5:
break
# Update weights
alpha = 0.5 * np.log((1 - error_rate) / (error_rate + 1e-10))
weights *= np.exp(-alpha * y * predictions)
weights /= np.sum(weights)
self.models.append(model)
self.alphas.append(alpha)
print(f"✓ Boosting implemented")### Lab 3: Bagging with Bootstrapping
import numpy as np
class BaggingEnsemble:
def __init__(self, n_estimators=10):
self.n_estimators = n_estimators
self.estimators = []
def fit(self, X, y):
"""Bootstrap aggregating"""
n_samples = len(X)
for _ in range(self.n_estimators):
# Bootstrap sample
indices = np.random.choice(n_samples, n_samples, replace=True)
X_boot = X[indices]
y_boot = y[indices]
# Train model
coef = np.linalg.lstsq(X_boot, y_boot, rcond=None)[0]
self.estimators.append(coef)
def predict(self, X):
"""Average predictions"""
predictions = np.array([X @ est for est in self.estimators])
return np.mean(predictions, axis=0)
bagging = BaggingEnsemble(n_estimators=10)
X = np.random.randn(100, 10)
y = X[:, 0] + np.random.randn(100) * 0.5
bagging.fit(X, y)
pred = bagging.predict(X[:5])
print(f"✓ Bagging convergence")### Lab 4: Stacking Ensemble
import numpy as np
class StackingEnsemble:
def __init__(self, n_base=5):
self.base_models = [np.random.randn(10, 1) for _ in range(n_base)]
self.meta_model = np.random.randn(n_base, 1)
def train(self, X, y):
"""Train base and meta models"""
# Train base models
for i in range(len(self.base_models)):
self.base_models[i] = np.linalg.lstsq(X, y, rcond=None)[0]
# Generate meta-features
meta_features = np.array([X @ m for m in self.base_models]).T
# Train meta-model
self.meta_model = np.linalg.lstsq(meta_features, y, rcond=None)[0]
def predict(self, X):
"""Stacked prediction"""
meta_features = np.array([X @ m for m in self.base_models]).T
return meta_features @ self.meta_model
stacking = StackingEnsemble(n_base=5)
X = np.random.randn(100, 10)
y = X[:, 0] + X[:, 1] + np.random.randn(100) * 0.5
stacking.train(X, y)
pred = stacking.predict(X[:5])
print(f"✓ Stacking complete")---