Ensemble Methods Voting Stacking Meta-Learners
# Ensemble Methods: Voting & Stacking Meta-Learners
## Introduction & Motivation
Voting averages predictions from multiple classifiers; Stacking trains meta-learner on base predictions. Combines diverse models; reduces variance and bias. Improves over individual models; Kaggle competition standard.
Motivation: Diversity in ensemble reduces overfitting. Different algorithms capture complementary patterns. Meta-learner learns optimal combination.
Applications: Competition machine learning, robust prediction systems, hybrid models.
---
## Core Concepts & Theory
### Voting
Soft: average probabilities.
Hard: majority vote.
### Stacking
Level 0: base learners (diverse models).
Level 1: meta-learner trained on base predictions.
---
## Mathematical Formulation
Soft voting:
$$\hat{p}_c = \frac{1}{m} \sum_{i=1}^{m} p_i^c$$
Stacking:
$$\hat{y} = f_{ ext{meta}}([f_1(\mathbf{x}), f_2(\mathbf{x}), ..., f_m(\mathbf{x})])$$
---
## Computational Considerations
Training: Sum of base learners + meta-learner.
Inference: All base learners run; then meta-learner.
---
## Practical Implementation Strategies
### Base Learner Diversity
Mix algorithms: tree, linear, SVM, neural net.
### Cross-Validation for Stacking
Avoid data leakage; use CV predictions for meta-learner training.
---
## Summary & Key Takeaways
Voting and stacking combine diverse base learners via averaging or meta-learning, achieving superior generalization.
Principles:
1. Diversity improves ensemble.
2. Soft voting preferred for probability calibration.
3. Stacking learns optimal combination.
4. Cross-validation critical to avoid overfitting.
5. Computationally expensive but effective.
---
---
## Appendix: Practical Labs
### Lab 1: Voting Classifier
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = make_classification(n_samples=150, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
lr = LogisticRegression(max_iter=200)
dt = DecisionTreeClassifier(max_depth=5)
svm = SVC(kernel='rbf', probability=True)
voting = VotingClassifier(estimators=[('lr', lr), ('dt', dt), ('svm', svm)], voting='soft')
voting.fit(X_train, y_train)
y_pred = voting.predict(X_test)
acc = accuracy_score(y_test, y_pred)
print(f"Voting accuracy: {acc:.4f}")
assert acc > 0.6, "Should exceed baseline"
print("✓ Voting classifier working")
if __name__ == "__main__":
print("Lab 1: Voting - PASSED")### Lab 2: Stacking Classifier
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = make_classification(n_samples=150, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
base_learners = [
('dt', DecisionTreeClassifier(max_depth=5)),
('svm', SVC(kernel='rbf', probability=True))
]
meta_learner = LogisticRegression(max_iter=200)
stacking = StackingClassifier(estimators=base_learners, final_estimator=meta_learner)
stacking.fit(X_train, y_train)
y_pred = stacking.predict(X_test)
acc = accuracy_score(y_test, y_pred)
print(f"Stacking accuracy: {acc:.4f}")
assert acc > 0.6, "Should exceed baseline"
print("✓ Stacking classifier working")
if __name__ == "__main__":
print("Lab 2: Stacking - PASSED")### Lab 3: Base Learner Diversity
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = make_classification(n_samples=150, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
models = [
LogisticRegression(max_iter=200),
DecisionTreeClassifier(max_depth=5),
SVC(kernel='rbf'),
RandomForestClassifier(n_estimators=50)
]
for model in models:
model.fit(X_train, y_train)
acc = accuracy_score(y_test, model.predict(X_test))
print(f"{model.__class__.__name__}: {acc:.4f}")
print("✓ Base learner diversity working")
if __name__ == "__main__":
print("Lab 3: Diversity - PASSED")### Lab 4: Soft vs Hard Voting
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = make_classification(n_samples=150, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
lr = LogisticRegression(max_iter=200)
dt = DecisionTreeClassifier(max_depth=5)
svm = SVC(kernel='rbf', probability=True)
soft_voting = VotingClassifier(estimators=[('lr', lr), ('dt', dt), ('svm', svm)], voting='soft')
hard_voting = VotingClassifier(estimators=[('lr', lr), ('dt', dt), ('svm', svm)], voting='hard')
soft_voting.fit(X_train, y_train)
hard_voting.fit(X_train, y_train)
soft_acc = accuracy_score(y_test, soft_voting.predict(X_test))
hard_acc = accuracy_score(y_test, hard_voting.predict(X_test))
print(f"Soft voting: {soft_acc:.4f}, Hard voting: {hard_acc:.4f}")
assert soft_acc > 0.5 and hard_acc > 0.5, "Both should exceed baseline"
print("✓ Soft vs hard voting working")
if __name__ == "__main__":
print("Lab 4: Soft vs Hard - PASSED")