Normalization and Standardization are feature scaling techniques that transform numeric features to comparable ranges — essential preprocessing for distance-based algorithms (KNN, SVM) and gradient-based methods (neural networks, logistic regression) because unscaled features with different magnitudes (Age 0-100 vs Salary 0-200,000) cause the larger-magnitude features to dominate distance calculations and gradient updates, leading to biased models and slow convergence.

Why Scale Features?

• The Problem: If you measure distances between data points using Age (0-100) and Salary (0-200,000), Salary dominates the distance calculation because its values are 2,000× larger — a difference of $10,000 in salary overwhelms a difference of 10 years in age, even though both might be equally important.
• Which Algorithms Need Scaling: Distance-based (KNN, SVM, K-Means), gradient-based (Neural Networks, Logistic Regression, Linear Regression with regularization). Tree-based models (Random Forest, XGBoost) do NOT need scaling because they split on individual features independently.

Standardization (Z-Score Normalization)

• Formula: $X_{new} = frac{X - mu}{sigma}$
• Result: Mean = 0, Standard Deviation = 1
• Range: Unbounded (typically -3 to +3, but outliers can be ±10+)
• Best For: Most ML algorithms — robust to outliers because outliers don't affect the mean/std as severely as they affect min/max

Normalization (Min-Max Scaling)

• Formula: $X_{new} = frac{X - X_{min}}{X_{max} - X_{min}}$
• Result: All values mapped to [0, 1]
• Best For: Neural networks (bounded activations), image data (pixels 0-255 → 0-1), algorithms requiring bounded input

Comparison

Critical Rule: Fit on Train, Transform Both

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # Learn mean/std from train
X_test_scaled = scaler.transform(X_test)          # Apply train's mean/std to test

Never call fit_transform on test data — that would leak test statistics into the scaler, causing data leakage.

Normalization and Standardization are the essential preprocessing steps for fair feature comparison — ensuring that all features contribute proportionally to model learning regardless of their original scale, with standardization as the safe default for most algorithms and min-max normalization for neural networks and bounded-input requirements.