encoding
**One-Hot Encoding** is the **standard technique for converting categorical variables into a binary matrix representation that machine learning models can process** — where each unique category becomes its own column with values 0 or 1 (Red → [1,0,0], Blue → [0,1,0], Green → [0,0,1]), avoiding the false ordinal assumption that Label Encoding introduces (Red=0, Blue=1, Green=2 implies Blue is "between" Red and Green), making it the default encoding for linear models and neural networks.
**What Is One-Hot Encoding?**
- **Definition**: A transformation that converts a single categorical column with K unique values into K binary columns — each row has exactly one "1" (hot) and K-1 "0"s (cold), creating a sparse binary representation.
- **Why Not Just Numbers?**: If you encode Red=0, Blue=1, Green=2 (Label Encoding), a linear model learns weights where Blue is literally "between" Red and Green mathematically. This is nonsensical for nominal categories. One-hot encoding gives each category its own independent coefficient.
**Example**
| Original | Red | Green | Blue |
|----------|-----|-------|------|
| Red | 1 | 0 | 0 |
| Blue | 0 | 0 | 1 |
| Green | 0 | 1 | 0 |
| Red | 1 | 0 | 0 |
**When to Use One-Hot Encoding**
| Model Type | Use One-Hot? | Reason |
|-----------|-------------|--------|
| **Linear Regression / Logistic** | Yes (required) | Cannot handle nominal categories as integers |
| **Neural Networks** | Yes (standard) | Independent dimensions for each category |
| **SVM** | Yes | Distance-based, needs proper encoding |
| **KNN** | Yes | Distance calculation needs binary dimensions |
| **Decision Trees / Random Forest** | Optional | Trees split on individual features, can use label encoding |
| **XGBoost / LightGBM** | Optional | LightGBM has native categorical support |
**The High-Cardinality Problem**
| Feature | Unique Values | One-Hot Columns | Problem |
|---------|--------------|----------------|---------|
| Color | 3 | 3 | Fine |
| Country | 195 | 195 | Manageable |
| Zip Code | 41,000+ | 41,000+ | Too many columns — model becomes slow, sparse, overfitting |
| User ID | 1,000,000+ | 1,000,000+ | Completely impractical |
**Solutions for high cardinality**:
- **Target Encoding**: Replace category with mean of target variable.
- **Frequency Encoding**: Replace category with its count.
- **Embeddings**: Learn dense vector representations (standard in deep learning).
- **Hash Encoding**: Map categories to a fixed number of buckets.
**The Dummy Variable Trap**
- **Problem**: With K one-hot columns, the last column is perfectly predictable from the first K-1 (if all are 0, the last must be 1). This creates multicollinearity in linear models.
- **Solution**: Drop one column (`drop_first=True` in pandas). Use K-1 columns instead of K.
```python
import pandas as pd
pd.get_dummies(df["color"], drop_first=True)
```
**One-Hot Encoding is the default categorical encoding for most machine learning models** — providing each category with an independent dimension that prevents false ordinal assumptions, with the key trade-off being dimensionality explosion for high-cardinality features that requires alternative encoding strategies like target encoding or embeddings.