target encoding
**Target Encoding (Mean Encoding)** is a **supervised categorical encoding technique that replaces each category with the mean of the target variable for that category** — transforming "New York" into 0.82 (82% of New York customers bought) and "Paris" into 0.23 (23% of Paris customers bought), providing a single numeric column that captures predictive signal for high-cardinality features (1000+ categories) where one-hot encoding would create an impractically wide matrix, but requiring careful regularization to prevent severe overfitting.
**What Is Target Encoding?**
- **Definition**: For each category, calculate the mean of the target variable across all training examples with that category, then replace the category string with that mean value — converting a categorical feature into a continuous numeric feature that directly encodes predictive information.
- **Why It's Powerful**: It converts high-cardinality categories (10,000 zip codes, 50,000 product IDs) into a single informative numeric column — impossible with one-hot encoding (10,000 new columns) and uninformative with label encoding (arbitrary integers).
- **Why It's Dangerous**: Without regularization, target encoding is the most common source of data leakage in machine learning pipelines — a category with 1 observation and a positive target gets encoded as 1.0, giving the model perfect (but fake) signal.
**How Target Encoding Works**
| City | Target (Bought?) | Count | Mean Target | Encoded Value |
|------|-----------------|-------|-------------|---------------|
| New York | 1,1,1,0,1 | 5 | 0.80 | 0.80 |
| Paris | 0,0,1,0,0 | 5 | 0.20 | 0.20 |
| Tokyo | 1,1,0,1,1 | 5 | 0.80 | 0.80 |
| Berlin | 1 | 1 | 1.00 | ⚠️ Overfitting! |
**The Overfitting Problem and Solutions**
| Problem | Example | Solution |
|---------|---------|---------|
| **Small category** | Berlin has 1 sample → mean = 1.0 (perfect but fake) | Smoothing with global mean |
| **Target leakage** | Encoding uses the same data the model trains on | Compute encoding inside cross-validation folds |
| **Rare categories** | A new city in test data has no encoding | Fall back to global mean |
**Smoothing Formula**
$ ext{Encoded} = frac{n imes ext{category\_mean} + m imes ext{global\_mean}}{n + m}$
Where n = category count, m = smoothing parameter. For Berlin (n=1, global_mean=0.5, m=10): $(1 × 1.0 + 10 × 0.5) / (1 + 10) = 0.545$ — pulled toward the global mean instead of the unreliable 1.0.
**Cross-Validation Encoding (K-Fold)**
| Fold | Training Data | Encoding Source | Test Data |
|------|-------------|----------------|-----------|
| Fold 1 as test | Compute means from Folds 2-5 | Means from Folds 2-5 | Apply to Fold 1 |
| Fold 2 as test | Compute means from Folds 1,3-5 | Means from Folds 1,3-5 | Apply to Fold 2 |
This ensures no data point's own target value contributes to its encoding — preventing leakage.
**When to Use Target Encoding**
| Scenario | Use Target Encoding? | Reason |
|----------|---------------------|--------|
| High-cardinality (1000+ categories) | Yes ✓ | One-hot creates too many columns |
| Low-cardinality (3-10 categories) | Usually no | One-hot works fine and is simpler |
| Tree-based models (XGBoost, LightGBM) | Yes ✓ | Trees benefit from continuous signal |
| Linear models | Carefully | Must ensure proper smoothing |
**Target Encoding is the most powerful encoding for high-cardinality categorical features** — converting thousands of categories into a single informative numeric column that captures predictive signal, with the critical requirement that smoothing and cross-validation encoding must be used to prevent the data leakage that makes naive target encoding notoriously prone to overfitting.