Data Normalization Preprocessing Standardization Scaling Encoding
# Data Normalization & Preprocessing: Standardization, Scaling & Encoding
## Introduction & Motivation
Preprocessing: prepare raw data for training. Standardization: zero mean, unit variance; center data. Min-max scaling: [0,1] or [-1,1] range; preserves structure. Normalization: per-sample; ensures comparable magnitude. One-hot encoding: categorical → binary vectors. Applications: essential for all supervised learning; impacts convergence, performance.
Motivation: Raw features often: different scales, different ranges. Preprocessing ensures fair contribution from all features.
Applications: Computer vision, NLP, tabular data.
---
## Core Concepts & Theory
### Standardization (Z-score)
Subtract mean, divide by std: (X - μ) / σ.
### Min-Max Scaling
Scale to [0,1]: (X - min) / (max - min).
### L2 Normalization
Per-sample unit norm; divide by L2 norm.
---
## Mathematical Formulation
Standardization:
$$X' = \frac{X - \mu}{\sigma}$$
where μ = mean, σ = standard deviation.
Min-Max Scaling:
$$X' = \frac{X - X_{\min}}{X_{\max} - X_{\min}}$$
L2 Normalization:
$$x'_i = \frac{x_i}{\|x\|_2} = \frac{x_i}{\sqrt{\sum_j x_j^2}}$$
One-Hot Encoding:
$$e_c = [0, \ldots, 1, \ldots, 0]^T \quad ext{(1 at position c)}$$
---
## Advanced Theory & Extensions
### Robust Scaling
Use median, IQR; robust to outliers.
### Power Transformations
Log, Box-Cox; handle skewed distributions.
### Quantile Normalization
Map to uniform quantiles; distribution-free.
---
## Computational Considerations
Standardization: O(N) one-pass; O(1) per sample.
Min-Max: O(N) for min/max; O(1) per sample.
L2 Norm: O(D) per sample; D = dimensionality.
---
## Practical Implementation Strategies
### Fit on Training Data
Compute μ, σ on train; apply to test. Avoid data leakage.
### Handle Missing Values
Mean/median imputation, or remove rows.
### Categorical Encoding
One-hot for tree-based; embeddings for deep learning.
---
## Benchmark Datasets & Evaluation
ImageNet: Normalize by channel mean/std; [0, 255] → [-1, 1].
Tabular Data: Standardization typical; reduces feature bias.
NLP: Tokenize, embed; no explicit normalization often.
---
## Key Challenges & Limitations
### Data Leakage
Fit on full data → biased estimates. Must fit on train.
### Outliers
Z-score sensitive; robust scaling better.
### Categorical Encoding
One-hot sparse; high dimensionality for many categories.
---
## Hyperparameter Tuning
Normalization method: Standardization default.
Outlier handling: Remove or robust scaling.
Encoding: One-hot for sparse, embeddings for dense.
---
## Real-World Applications & Case Studies
ImageNet: Channel standardization; [-1, 1] range.
Kaggle Competitions: Standardization for tabular; best practice.
Neural Networks: Batch norm reduces normalization need.
---
## Integration with Other Methods
Preprocessing + Batch Norm → redundant; either sufficient.
Preprocessing + Outlier Detection → robust pipelines.
---
## Summary & Key Takeaways
Preprocessing via standardization, scaling, and encoding prepares data for training, ensuring fair feature contribution and stable convergence.
Principles:
1. Standardization: zero mean, unit variance; default.
2. Fit on train, apply to test; prevent leakage.
3. Scaling: [0,1] or [-1,1]; preserves relative structure.
4. Categorical: one-hot for algorithms; embeddings for deep.
5. Remove outliers or use robust scaling.
---
---
## Appendix: Practical Labs
### Lab 1: Standardization
import torch
import numpy as np
class StandardScaler:
def __init__(self):
self.mean = None
self.std = None
def fit(self, X):
"""Fit on training data"""
self.mean = X.mean(axis=0)
self.std = X.std(axis=0) + 1e-8
def transform(self, X):
"""Apply to data"""
return (X - self.mean) / self.std
def fit_transform(self, X):
"""Fit and transform"""
self.fit(X)
return self.transform(X)
# Test
np.random.seed(42)
X_train = np.random.randn(100, 5)
X_test = np.random.randn(20, 5)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
assert np.allclose(X_train_scaled.mean(axis=0), 0, atol=1e-6), "Mean should be 0"
assert np.allclose(X_train_scaled.std(axis=0), 1, atol=1e-6), "Std should be 1"
print("✓ Standardization working")
if __name__ == "__main__":
print("Lab 1: Standardization - PASSED")### Lab 2: Min-Max Scaling
import numpy as np
class MinMaxScaler:
def __init__(self, feature_range=(0, 1)):
self.feature_range = feature_range
self.min = None
self.max = None
def fit(self, X):
"""Fit on training data"""
self.min = X.min(axis=0)
self.max = X.max(axis=0)
def transform(self, X):
"""Apply scaling"""
X_scaled = (X - self.min) / (self.max - self.min + 1e-8)
a, b = self.feature_range
return X_scaled * (b - a) + a
def fit_transform(self, X):
self.fit(X)
return self.transform(X)
# Test
np.random.seed(42)
X_train = np.random.randn(100, 5) * 10 + 50
scaler = MinMaxScaler(feature_range=(0, 1))
X_scaled = scaler.fit_transform(X_train)
assert X_scaled.min() >= 0, "Min should be >= 0"
assert X_scaled.max() <= 1, "Max should be <= 1"
assert np.isfinite(X_scaled).all(), "All finite"
print("✓ Min-max scaling working")
if __name__ == "__main__":
print("Lab 2: MinMax - PASSED")### Lab 3: L2 Normalization
import torch
import numpy as np
def l2_normalize(X, axis=1):
"""L2 normalize per sample"""
norm = np.linalg.norm(X, axis=axis, keepdims=True)
return X / (norm + 1e-8)
# Test
np.random.seed(42)
X = np.random.randn(32, 100)
X_norm = l2_normalize(X, axis=1)
# Check norms
norms = np.linalg.norm(X_norm, axis=1)
assert np.allclose(norms, 1.0), "Norms should be 1"
assert np.isfinite(X_norm).all(), "All finite"
print("✓ L2 normalization working")
if __name__ == "__main__":
print("Lab 3: L2 - PASSED")### Lab 4: One-Hot Encoding
import numpy as np
def one_hot_encode(X, n_classes=None):
"""One-hot encode categorical labels"""
if n_classes is None:
n_classes = X.max() + 1
encoded = np.zeros((len(X), n_classes))
for i, x in enumerate(X):
encoded[i, int(x)] = 1
return encoded
# Test
np.random.seed(42)
X = np.array([0, 1, 2, 1, 0])
encoded = one_hot_encode(X, n_classes=3)
assert encoded.shape == (5, 3), "Should be [5, 3]"
assert (encoded.sum(axis=1) == 1).all(), "Each row sums to 1"
assert (encoded == 0) | (encoded == 1) | (encoded.all()), "Binary values"
print("✓ One-hot encoding working")
if __name__ == "__main__":
print("Lab 4: OneHot - PASSED")