normalization
**Normalization and Standardization** are **feature scaling techniques that transform numeric features to comparable ranges** — essential preprocessing for distance-based algorithms (KNN, SVM) and gradient-based methods (neural networks, logistic regression) because unscaled features with different magnitudes (Age 0-100 vs Salary 0-200,000) cause the larger-magnitude features to dominate distance calculations and gradient updates, leading to biased models and slow convergence.
**Why Scale Features?**
- **The Problem**: If you measure distances between data points using Age (0-100) and Salary (0-200,000), Salary dominates the distance calculation because its values are 2,000× larger — a difference of $10,000 in salary overwhelms a difference of 10 years in age, even though both might be equally important.
- **Which Algorithms Need Scaling**: Distance-based (KNN, SVM, K-Means), gradient-based (Neural Networks, Logistic Regression, Linear Regression with regularization). Tree-based models (Random Forest, XGBoost) do NOT need scaling because they split on individual features independently.
**Standardization (Z-Score Normalization)**
- **Formula**: $X_{new} = frac{X - mu}{sigma}$
- **Result**: Mean = 0, Standard Deviation = 1
- **Range**: Unbounded (typically -3 to +3, but outliers can be ±10+)
- **Best For**: Most ML algorithms — robust to outliers because outliers don't affect the mean/std as severely as they affect min/max
| Feature | Original | Standardized |
|---------|----------|-------------|
| Age = 25 | 25 | -1.2 |
| Age = 50 | 50 | 0.0 |
| Age = 75 | 75 | +1.2 |
| Salary = $30K | 30,000 | -1.0 |
| Salary = $60K | 60,000 | 0.0 |
| Salary = $90K | 90,000 | +1.0 |
**Normalization (Min-Max Scaling)**
- **Formula**: $X_{new} = frac{X - X_{min}}{X_{max} - X_{min}}$
- **Result**: All values mapped to [0, 1]
- **Best For**: Neural networks (bounded activations), image data (pixels 0-255 → 0-1), algorithms requiring bounded input
| Feature | Original | Normalized |
|---------|----------|-----------|
| Age = 25 | 25 | 0.25 |
| Age = 50 | 50 | 0.50 |
| Age = 75 | 75 | 0.75 |
**Comparison**
| Property | Standardization (Z-Score) | Normalization (Min-Max) |
|----------|--------------------------|------------------------|
| **Output range** | Unbounded (~-3 to +3) | Fixed [0, 1] |
| **Outlier sensitivity** | Moderate (outliers shift mean/std slightly) | High (one outlier compresses all other values) |
| **Best for** | General ML, regression, SVM | Neural networks, image data |
| **Preserves zero** | Yes (sparse data friendly) | No |
| **Rule of thumb** | "When in doubt, standardize" | When bounded input is required |
**Critical Rule: Fit on Train, Transform Both**
```python
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # Learn mean/std from train
X_test_scaled = scaler.transform(X_test) # Apply train's mean/std to test
```
Never call `fit_transform` on test data — that would leak test statistics into the scaler, causing data leakage.
**Normalization and Standardization are the essential preprocessing steps for fair feature comparison** — ensuring that all features contribute proportionally to model learning regardless of their original scale, with standardization as the safe default for most algorithms and min-max normalization for neural networks and bounded-input requirements.