stacking
**Stacking** (stacked generalization) is the **ensemble learning technique that trains a meta-model to optimally combine predictions from multiple diverse base models, learning through cross-validation which base learners to trust for different types of inputs** — consistently outperforming simple averaging or voting by discovering complementary strengths across algorithms, making it the dominant ensemble strategy in machine learning competitions and a robust approach for production systems where no single model excels across all data patterns.
**What Is Stacking?**
- **Architecture**: Layer 0 (base models: RF, XGBoost, SVM, Neural Net) → Layer 1 (meta-model: logistic regression or linear model) → Final prediction.
- **Key Insight**: Different models make different mistakes — a meta-learner can identify which model to trust for which inputs.
- **Cross-Validation Requirement**: Base model predictions used for meta-training must come from out-of-fold predictions to prevent data leakage and overfitting.
- **Meta-Features**: The meta-model's input features are the predictions (or probabilities) from each base model.
**Why Stacking Matters**
- **Superior Performance**: Typically beats any individual base model and outperforms simple averaging by 1-5% on benchmarks.
- **Diversity Exploitation**: A random forest might excel on categorical features while a neural network handles continuous interactions — stacking learns to route decisions appropriately.
- **Competition Dominance**: Nearly every top Kaggle submission uses stacking or its variants.
- **Robustness**: Less sensitive to individual model failures since the meta-learner can down-weight unreliable base models.
- **Flexible Architecture**: Any combination of models can serve as base learners — mixing paradigms (tree-based, linear, neural) maximizes diversity.
**How Stacking Works**
**Step 1 — Generate Out-of-Fold Predictions**:
- Split training data into K folds.
- For each base model, train on K-1 folds and predict on the held-out fold.
- Concatenate held-out predictions to create meta-features for the full training set.
**Step 2 — Train Meta-Model**:
- Use the out-of-fold predictions as features and original labels as targets.
- Fit a simple meta-model (logistic regression is standard) to learn optimal combination.
**Step 3 — Final Prediction**:
- Train all base models on full training data.
- Generate predictions on test data from each base model.
- Feed base predictions through the trained meta-model for final output.
**Stacking Variants**
| Variant | Description | Use Case |
|---------|-------------|----------|
| **Standard Stacking** | Single-layer meta-model on base predictions | Default approach |
| **Multi-Level Stacking** | Multiple meta-model layers (stack of stacks) | Competitions (diminishing returns) |
| **Blending** | Uses hold-out set instead of cross-validation | Faster, simpler, slightly less optimal |
| **Feature-Weighted Stacking** | Meta-model also receives original features | When base models miss important signals |
| **Stacking with Diversity** | Deliberately train weaker but diverse base models | Maximum complementarity |
**Best Practices**
- **Meta-Model Simplicity**: Use logistic regression or ridge — complex meta-models overfit to the small number of meta-features.
- **Base Model Diversity**: Maximize architectural diversity (trees, linear, neural, nearest-neighbor) — correlated base models add no value.
- **Sufficient Folds**: Use 5-10 fold CV to generate reliable out-of-fold predictions.
- **Probability Outputs**: Feed predicted probabilities (not classes) to the meta-model for maximum information transfer.
Stacking is **the principled way to let models vote on the answer** — going beyond democratic averaging to intelligent weighting where a meta-learner discovers exactly when to trust each expert, consistently producing the most robust predictions achievable from a given set of base models.