imbalanced classification smote class weights threshold adjustment
# Imbalanced Classification: SMOTE, Class Weights & Threshold Adjustment
## Introduction & Motivation
Imbalanced datasets (rare positive class) bias models toward majority class. SMOTE (Synthetic Minority Oversampling) generates synthetic minority samples via k-NN interpolation. Class weights penalize misclassification of rare class. Threshold adjustment trades off precision-recall. Critical for fraud detection, disease diagnosis, anomaly detection.
Motivation: Standard accuracy misleading when class skew extreme (1% positive). Must optimize for recall/precision/F1.
Applications: Credit card fraud, medical diagnosis (rare disease), anomaly detection, click-through rate prediction.
---
## Core Concepts & Theory
### Imbalanced Learning
Naive classifier: always predict majority. No information gain; 99% accuracy but useless.
### SMOTE
Oversample minority: find k-NN in minority class; interpolate new samples.
### Class Weights
Penalize minority misclassification more: weight_pos > weight_neg.
### Cost-Sensitive Learning
Asymmetric loss: different cost for FP vs. FN.
---
## Mathematical Formulation
SMOTE sampling:
$$x_{ ext{syn}} = x_i + r \cdot (x_{nn_j} - x_i), \quad r \in [0,1]$$
Weighted cross-entropy:
$$\mathcal{L} = -\sum_i [y_i \log \hat{p}_i + (1-y_i) \log(1-\hat{p}_i)] \cdot w_{y_i}$$
Class weights:
$$w_0 = \frac{n}{2n_0}, \quad w_1 = \frac{n}{2n_1}$$
---
## Advanced Theory & Extensions
### SMOTE Variants
Borderline-SMOTE (focus on boundary samples); SVM-SMOTE (use SVM support vectors).
### Cost-Sensitive Thresholding
Adjust threshold P(y=1|x) > τ based on cost ratio.
### Ensemble Methods
RandomUnderSampler + RandomOverSampler in pipeline.
---
## Computational Considerations
SMOTE: O(k × m) where k = k-NN, m = minority samples.
Class weighting: O(1) in loss computation.
Threshold search: O(n) to compute metrics across thresholds.
---
## Practical Implementation Strategies
### Stratified CV
Maintain class ratio in train/validation splits.
### Pipeline Design
SMOTE on train only; prevent data leakage to test.
### Metric Selection
Recall: cost of missed positives; Precision: cost of false alarms; F1: balance.
---
## Benchmark Datasets & Evaluation
Fraud Detection: Kaggle Credit Card Fraud (284K samples, 0.17% positive).
Medical: UCI Breast Cancer (569 samples, 37% positive).
Metrics: Precision, Recall, F1, ROC-AUC, PR-AUC, cost-sensitive accuracy.
---
## Key Challenges & Limitations
### Overgeneralization
SMOTE may create unrealistic samples in high dimensions.
### Computational Cost
Large oversampling multiplies training time.
### Class Distribution Shift
Train distribution != test distribution (requires calibration).
---
## Hyperparameter Tuning
SMOTE k: 5-10; balance locality-diversity.
Class weight ratio: 2-100; heavier weights for rarer classes.
Threshold: 0.5-0.95; optimize for business metric.
---
## Real-World Applications & Case Studies
Stripe: Fraud detection via imbalanced learning + real-time thresholding.
Healthcare: Rare disease diagnosis via SMOTE + cost-sensitive trees.
Anomaly Detection: Unsupervised + imbalanced; few known anomalies.
---
## Integration with Other Methods
Imbalanced + Ensemble → balanced random forests, easy ensemble.
Imbalanced + Deep Learning → focal loss (down-weight easy examples).
---
## Summary & Key Takeaways
Imbalanced classification requires SMOTE oversampling, class weighting, and threshold adjustment to optimize for minority class performance.
Principles:
1. SMOTE generates synthetic minority samples via k-NN.
2. Class weights penalize minority misclassification.
3. Stratified CV maintains class ratio.
4. Threshold optimization balances precision-recall.
5. Evaluate with F1, recall, AUC; not accuracy.
---
---
## Appendix: Practical Labs
### Lab 1: SMOTE Oversampling
import numpy as np
from collections import Counter
def simple_smote(X, y, k=5, oversampling_factor=2):
"""Basic SMOTE implementation"""
X_minority = X[y == 1]
n_new = len(X_minority) * (oversampling_factor - 1)
X_syn = []
for _ in range(int(n_new)):
idx = np.random.randint(len(X_minority))
x_i = X_minority[idx]
# k-NN among minority
distances = np.linalg.norm(X_minority - x_i, axis=1)
knn_idx = np.argsort(distances)[1:k+1]
nn_idx = np.random.choice(knn_idx)
x_nn = X_minority[nn_idx]
# Interpolate
r = np.random.rand()
x_syn = x_i + r * (x_nn - x_i)
X_syn.append(x_syn)
X_syn = np.array(X_syn)
X_new = np.vstack([X, X_syn])
y_new = np.concatenate([y, np.ones(len(X_syn))])
return X_new, y_new
# Data
np.random.seed(42)
X = np.random.randn(100, 5)
y = np.concatenate([np.zeros(95), np.ones(5)])
X_bal, y_bal = simple_smote(X, y, k=3, oversampling_factor=3)
print(f"Original: {Counter(y)}, Balanced: {Counter(y_bal)}")
assert len(y_bal) > len(y), "Should oversample"
assert (y_bal == 1).sum() > (y == 1).sum(), "Should add minority"
print("✓ SMOTE oversampling working")
if __name__ == "__main__":
print("Lab 1: SMOTE - PASSED")### Lab 2: Class Weights
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
# Imbalanced data
np.random.seed(42)
X = np.random.randn(200, 5)
y = np.concatenate([np.zeros(190), np.ones(10)])
# Compute class weights
n_pos = (y == 1).sum()
n_neg = (y == 0).sum()
weight_pos = n_neg / (n_pos + 1e-8)
weight_neg = 1.0
print(f"Class weights: pos={weight_pos:.2f}, neg={weight_neg:.2f}")
# Train without weights
model_unweighted = LogisticRegression(max_iter=200, random_state=42)
model_unweighted.fit(X, y)
# Train with weights (sklearn balanced option)
model_weighted = LogisticRegression(max_iter=200, class_weight='balanced', random_state=42)
model_weighted.fit(X, y)
f1_unweighted = f1_score(y, model_unweighted.predict(X))
f1_weighted = f1_score(y, model_weighted.predict(X))
print(f"F1 unweighted: {f1_unweighted:.4f}, weighted: {f1_weighted:.4f}")
assert f1_weighted >= 0, "F1 should be non-negative"
print("✓ Class weighting working")
if __name__ == "__main__":
print("Lab 2: Class Weights - PASSED")### Lab 3: Threshold Optimization
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_recall_curve, f1_score
# Data
np.random.seed(42)
X = np.random.randn(200, 5)
y = np.concatenate([np.zeros(180), np.ones(20)])
model = LogisticRegression(max_iter=200, class_weight='balanced', random_state=42)
model.fit(X, y)
probs = model.predict_proba(X)[:, 1]
# Find optimal threshold
thresholds = np.linspace(0, 1, 50)
f1_scores = []
for threshold in thresholds:
y_pred = (probs >= threshold).astype(int)
if (y_pred == 1).sum() > 0:
f1 = f1_score(y, y_pred)
else:
f1 = 0
f1_scores.append(f1)
optimal_threshold = thresholds[np.argmax(f1_scores)]
print(f"Optimal threshold: {optimal_threshold:.3f}, max F1: {max(f1_scores):.4f}")
assert 0 <= optimal_threshold <= 1, "Threshold should be in [0,1]"
assert max(f1_scores) >= 0, "F1 should be non-negative"
print("✓ Threshold optimization working")
if __name__ == "__main__":
print("Lab 3: Threshold - PASSED")### Lab 4: Precision-Recall Trade-off
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_recall_curve
# Data
np.random.seed(42)
X = np.random.randn(300, 5)
y = np.concatenate([np.zeros(270), np.ones(30)])
model = LogisticRegression(max_iter=200, class_weight='balanced', random_state=42)
model.fit(X, y)
probs = model.predict_proba(X)[:, 1]
precision, recall, thresholds = precision_recall_curve(y, probs)
print(f"Precision at recall 0.5: {precision[np.argmin(np.abs(recall - 0.5))]:.3f}")
print(f"Number of thresholds: {len(thresholds)}")
assert len(precision) > 0, "Should have precision values"
assert len(recall) == len(precision), "Precision-recall should match"
assert all(0 <= p <= 1 for p in precision), "Precision in [0,1]"
print("✓ Precision-recall curve working")
if __name__ == "__main__":
print("Lab 4: PR Curve - PASSED")