Stacking (stacked generalization) is the ensemble learning technique that trains a meta-model to optimally combine predictions from multiple diverse base models, learning through cross-validation which base learners to trust for different types of inputs — consistently outperforming simple averaging or voting by discovering complementary strengths across algorithms, making it the dominant ensemble strategy in machine learning competitions and a robust approach for production systems where no single model excels across all data patterns.

What Is Stacking?

• Architecture: Layer 0 (base models: RF, XGBoost, SVM, Neural Net) → Layer 1 (meta-model: logistic regression or linear model) → Final prediction.
• Key Insight: Different models make different mistakes — a meta-learner can identify which model to trust for which inputs.
• Cross-Validation Requirement: Base model predictions used for meta-training must come from out-of-fold predictions to prevent data leakage and overfitting.
• Meta-Features: The meta-model's input features are the predictions (or probabilities) from each base model.

Why Stacking Matters

• Superior Performance: Typically beats any individual base model and outperforms simple averaging by 1-5% on benchmarks.
• Diversity Exploitation: A random forest might excel on categorical features while a neural network handles continuous interactions — stacking learns to route decisions appropriately.
• Competition Dominance: Nearly every top Kaggle submission uses stacking or its variants.
• Robustness: Less sensitive to individual model failures since the meta-learner can down-weight unreliable base models.
• Flexible Architecture: Any combination of models can serve as base learners — mixing paradigms (tree-based, linear, neural) maximizes diversity.

How Stacking Works

Step 1 — Generate Out-of-Fold Predictions:

• Split training data into K folds.
• For each base model, train on K-1 folds and predict on the held-out fold.
• Concatenate held-out predictions to create meta-features for the full training set.

Step 2 — Train Meta-Model:

• Use the out-of-fold predictions as features and original labels as targets.
• Fit a simple meta-model (logistic regression is standard) to learn optimal combination.

Step 3 — Final Prediction:

• Train all base models on full training data.
• Generate predictions on test data from each base model.
• Feed base predictions through the trained meta-model for final output.

Stacking Variants

Best Practices

• Meta-Model Simplicity: Use logistic regression or ridge — complex meta-models overfit to the small number of meta-features.
• Base Model Diversity: Maximize architectural diversity (trees, linear, neural, nearest-neighbor) — correlated base models add no value.
• Sufficient Folds: Use 5-10 fold CV to generate reliable out-of-fold predictions.
• Probability Outputs: Feed predicted probabilities (not classes) to the meta-model for maximum information transfer.

Stacking is the principled way to let models vote on the answer — going beyond democratic averaging to intelligent weighting where a meta-learner discovers exactly when to trust each expert, consistently producing the most robust predictions achievable from a given set of base models.