Imbalanced Classification Sampling Strategies Smote
# Imbalanced Classification: Sampling Strategies & SMOTE
## Introduction & Motivation
Imbalanced data: minority class rare; standard metrics misleading. Oversampling: duplicate minority; simple but overfits. Undersampling: remove majority; lose information. SMOTE: synthetic minority; generate realistic samples. Class weights: scale loss inversely to frequency. Applications: fraud detection, medical diagnosis, rare diseases.
Motivation: Accuracy high despite poor minority performance. Imbalance strategies focus on minority, improve robustness.
Applications: Fraud, anomaly detection, medical ML.
---
## Core Concepts & Theory
### Oversampling
Duplicate minority class samples; increase class balance.
### Undersampling
Remove majority class samples; balance via reduction.
### SMOTE (Synthetic Minority Over-sampling)
Interpolate between minority samples; generate synthetic examples.
---
## Mathematical Formulation
SMOTE (single point):
$$x_{ ext{new}} = x_i + r \cdot (x_k - x_i)$$
where x_i = minority sample, x_k = random neighbor, r ∈ [0, 1].
Class weights (loss):
$$L_{ ext{weighted}} = \sum_i w_{y_i} L(y_i, \hat{y}_i)$$
$$w_c = \frac{N}{n_c \cdot C} \quad ext{(inverse frequency)}$$
Focal loss (reweighting):
$$L_{ ext{focal}} = -\alpha (1 - p_t)^\gamma \log(p_t)$$
---
## Advanced Theory & Extensions
### Borderline SMOTE
Oversample near decision boundary; more targeted.
### ADASYN
Adaptive oversampling; more samples for harder cases.
### Cost-Sensitive Learning
Different misclassification costs; custom weighting.
---
## Computational Considerations
Oversampling: O(1) duplicate; negligible.
Undersampling: O(N) filtering; fast.
SMOTE: O(k·m) for m minority samples, k neighbors; reasonable.
---
## Practical Implementation Strategies
### Sampling Strategy
Oversample minority + undersample majority; balance ~1:10.
### SMOTE Neighbors
k=5 typical; more for smoother interpolation.
### Train-Test Split
Apply SMOTE on train only; prevent leakage.
---
## Benchmark Datasets & Evaluation
Credit Card Fraud: Typical 1:1000 imbalance; SMOTE + class weights.
Medical Diagnosis: Rare disease; 1:100 imbalance common.
Kaggle: Domain-specific; oversampling/undersampling standard.
---
## Key Challenges & Limitations
### Overfitting Risk
Oversampling exact copies; model memorizes.
### Information Loss
Undersampling discards data; may remove informative samples.
### SMOTE Artifacts
Interpolation may create unrealistic samples at class boundaries.
---
## Hyperparameter Tuning
Sampling ratio: 1:1 to 1:10; empirical on holdout.
SMOTE neighbors k: 3-10; smaller for dense regions.
Class weight: Inverse frequency; or data-driven.
---
## Real-World Applications & Case Studies
Fraud Detection: Combine oversampling + focal loss; both critical.
Medical Imaging: Oversampling minority disease class.
Anomaly Detection: SMOTE for known anomalies; PU learning for unknowns.
---
## Integration with Other Methods
Sampling + Threshold Tuning → adjust decision boundary.
Sampling + Ensemble → diverse minority representation.
---
## Summary & Key Takeaways
Imbalanced classification via oversampling, SMOTE, and class weighting addresses class imbalance, enabling minority-class-aware learning.
Principles:
1. Oversampling: simple; risk of overfitting.
2. SMOTE: synthetic; more robust than duplication.
3. Undersampling: fast; loses information.
4. Class weights: implicit; coupled to loss.
5. Combine strategies: oversampling + weights often best.
---
---
## Appendix: Practical Labs
### Lab 1: Oversampling and Undersampling
import numpy as np
from collections import Counter
def oversample_minority(X, y):
"""Oversample minority class to balance"""
classes, counts = np.unique(y, return_counts=True)
minority_class = classes[np.argmin(counts)]
max_count = counts.max()
minority_idx = np.where(y == minority_class)[0]
n_to_generate = max_count - len(minority_idx)
# Sample with replacement
new_idx = np.random.choice(minority_idx, size=n_to_generate, replace=True)
X_new = np.vstack([X, X[new_idx]])
y_new = np.hstack([y, y[new_idx]])
return X_new, y_new
# Test
np.random.seed(42)
X = np.random.randn(100, 10)
y = np.hstack([np.ones(95), np.zeros(5)]) # 95:5 imbalance
X_balanced, y_balanced = oversample_minority(X, y)
assert len(y_balanced) > len(y), "Should increase data"
assert Counter(y_balanced)[0] == Counter(y_balanced)[1], "Should be balanced"
print("✓ Oversampling working")
if __name__ == "__main__":
print("Lab 1: Oversampling - PASSED")### Lab 2: SMOTE Implementation
import numpy as np
def smote(X, y, k=5, ratio=1.0):
"""SMOTE: generate synthetic minority samples"""
minority_class = np.argmin(np.bincount(y))
minority_idx = np.where(y == minority_class)[0]
X_minority = X[minority_idx]
n_minority = len(minority_idx)
n_to_generate = int(n_minority * ratio)
X_synthetic = []
for _ in range(n_to_generate):
# Random minority sample
idx = np.random.choice(n_minority)
x_i = X_minority[idx]
# Find k nearest neighbors
dists = np.linalg.norm(X_minority - x_i, axis=1)
neighbors_idx = np.argsort(dists)[1:k+1]
# Random neighbor
x_k = X_minority[np.random.choice(neighbors_idx)]
# Interpolate
r = np.random.rand()
x_new = x_i + r * (x_k - x_i)
X_synthetic.append(x_new)
X_synthetic = np.array(X_synthetic)
X_new = np.vstack([X, X_synthetic])
y_new = np.hstack([y, np.full(len(X_synthetic), minority_class)])
return X_new, y_new
# Test
np.random.seed(42)
X = np.random.randn(100, 10)
y = np.hstack([np.ones(95), np.zeros(5)])
X_smote, y_smote = smote(X, y, k=3, ratio=1.0)
assert len(y_smote) > len(y), "Should generate samples"
assert (y_smote == 0).sum() > (y == 0).sum(), "Minority increased"
print("✓ SMOTE working")
if __name__ == "__main__":
print("Lab 2: SMOTE - PASSED")### Lab 3: Class Weights
import torch
import numpy as np
def compute_class_weights(y):
"""Compute weights inversely proportional to class frequency"""
classes, counts = np.unique(y, return_counts=True)
weights = len(y) / (len(classes) * counts)
weight_dict = {c: w for c, w in zip(classes, weights)}
return weight_dict
# Test
np.random.seed(42)
y = np.hstack([np.ones(95), np.zeros(5)])
weights = compute_class_weights(y)
assert weights[0] > weights[1], "Minority should have higher weight"
assert abs(weights[0] + weights[1] - 2.0) < 1e-6, "Weights should sum to 2"
print("✓ Class weights working")
if __name__ == "__main__":
print("Lab 3: Weights - PASSED")### Lab 4: Imbalance Metrics (Precision, Recall, F1)
import numpy as np
def compute_metrics(y_true, y_pred):
"""Compute precision, recall, F1 for binary classification"""
tp = ((y_pred == 1) & (y_true == 1)).sum()
fp = ((y_pred == 1) & (y_true == 0)).sum()
fn = ((y_pred == 0) & (y_true == 1)).sum()
precision = tp / (tp + fp + 1e-8)
recall = tp / (tp + fn + 1e-8)
f1 = 2 * (precision * recall) / (precision + recall + 1e-8)
return precision, recall, f1
# Test
np.random.seed(42)
y_true = np.hstack([np.ones(95), np.zeros(5)])
y_pred = np.hstack([np.ones(90), np.zeros(10)]) # Miss some positives
precision, recall, f1 = compute_metrics(y_true, y_pred)
assert 0 <= precision <= 1, "Precision in [0,1]"
assert 0 <= recall <= 1, "Recall in [0,1]"
assert 0 <= f1 <= 1, "F1 in [0,1]"
assert np.isfinite([precision, recall, f1]).all(), "All finite"
print("✓ Imbalance metrics working")
if __name__ == "__main__":
print("Lab 4: Metrics - PASSED")