Stacking (Stacked Generalization) is an ensemble technique where a "meta-learner" model is trained to optimally combine the predictions of multiple diverse "base learners" — instead of simple averaging or voting, stacking learns WHEN to trust each model (Model A is best for young customers, Model B is best for high-income customers) by using the base models' predictions as input features to a second-level model, typically achieving the highest performance of any ensemble method and serving as the winning strategy in many Kaggle competitions.

What Is Stacking?

• Definition: A two-level ensemble method where Level 1 consists of diverse base models that generate predictions, and Level 2 is a meta-learner (usually a simple linear model) that takes those predictions as features and learns the optimal way to combine them.
• Why It's Better Than Simple Averaging: Averaging weights all models equally. Stacking learns that "trust the Random Forest more for these types of inputs and the Neural Network more for those types" — capturing conditional expertise that uniform weighting cannot.
• The Key Insight: Different models have different strengths. A linear model might be best for extrapolation, a tree model for capturing interactions, and a neural network for non-linear patterns. Stacking automatically allocates trust based on each model's demonstrated accuracy on different regions of the data.

How Stacking Works

Preventing Data Leakage in Stacking

The critical mistake: training base models on the same data used to generate meta-features → the meta-learner overfits to training set predictions.

Solution: K-Fold Out-of-Fold Predictions

Each training example gets a prediction from a model that never saw it — preventing leakage.

Common Stacking Architectures

Python Implementation

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC

stacker = StackingClassifier(
    estimators=[
        ('rf', RandomForestClassifier(n_estimators=100)),
        ('svm', SVC(probability=True)),
    ],
    final_estimator=LogisticRegression(),
    cv=5  # Out-of-fold predictions (prevents leakage)
)
stacker.fit(X_train, y_train)

Stacking is the most powerful ensemble technique for combining diverse models — learning the optimal conditional weighting of base model predictions through a meta-learner that captures when each model is most trustworthy, consistently achieving top performance in competitions and production systems where maximizing accuracy justifies the additional complexity of a multi-model pipeline.