imbalanced learning class weights resampling strategies
# Imbalanced Learning: Class Weights & Resampling Strategies
## Introduction & Motivation
Imbalanced Learning: skewed class distributions. Rare positive class; common negative. Resampling, cost-sensitive learning. Applications: fraud detection, medical diagnosis, anomaly detection.
Motivation: Improve performance on minority class.
Applications: Fraud, disease, rare events.
---
## Core Concepts & Theory
### Class Imbalance
Unequal class frequencies; data imbalance.
### Resampling
Oversampling minority; undersampling majority.
### Cost-Sensitive Learning
Asymmetric loss weights.
---
## Mathematical Formulation
Class weights:
$$w_c = \frac{N}{C \cdot N_c}$$
where C=number of classes, N_c=class c count.
Weighted cross-entropy:
$$L = -\sum_c w_c y_c \log(\hat{y}_c)$$
SMOTE resampling:
$$x_{ ext{new}} = x + \lambda (x_{nn} - x)$$
---
## Advanced Theory & Extensions
### SMOTE
Synthetic Minority Oversampling.
### Threshold Adjustment
Shift classification boundary.
### Ensemble Methods
Balanced bagging; cascade classifiers.
---
## Computational Considerations
Resampling: O(N) for oversampling.
SMOTE: O(N_minority·k_neighbors).
Cost computation: O(batch_size).
---
## Practical Implementation Strategies
### Class Weights
Automatic weight calculation.
### Focal Loss
Down-weight easy samples.
### Threshold Tuning
Adjust decision boundary.
---
## Benchmark Datasets & Evaluation
Credit Card Fraud: Highly imbalanced.
Rare Diseases: Medical imbalance.
Anomaly Datasets: Imbalanced by nature.
---
## Key Challenges & Limitations
### Too Much Oversampling
Overfitting to minority.
### Information Loss
Undersampling loses data.
### Metric Selection
Accuracy misleading; use AUC, F1.
---
## Hyperparameter Tuning
Class weight ratio: 1-100 depending on imbalance.
Oversampling ratio: Match or reduce imbalance.
Threshold: 0.3-0.7 depending on cost.
---
## Real-World Applications & Case Studies
Fraud Detection: Rare positive.
Disease Diagnosis: Uncommon conditions.
Anomaly Detection: Rare anomalies.
---
## Integration with Other Methods
Imbalanced + Ensemble → robust minority.
Imbalanced + Active Learning → efficient labeling.
---
## Summary & Key Takeaways
Imbalanced Learning via class weighting and resampling enables minority class improvement through cost-sensitive training and synthetic sample generation.
Principles:
1. Class weights: asymmetric importance.
2. Resampling: balance distributions.
3. SMOTE: synthetic generation.
4. Metrics: AUC, F1, precision-recall.
5. Threshold: adjust boundary.
---
---
## Appendix: Practical Labs
### Lab 1: Class Weight Calculation
import numpy as np
def calculate_class_weights(labels, num_classes=2):
"""Calculate class weights for imbalance"""
weights = np.zeros(num_classes)
total_samples = len(labels)
for c in range(num_classes):
class_count = (labels == c).sum()
if class_count > 0:
weights[c] = total_samples / (num_classes * class_count)
return weights
# Test
np.random.seed(42)
labels = np.concatenate([np.zeros(900), np.ones(100)]).astype(int)
weights = calculate_class_weights(labels, num_classes=2)
assert len(weights) == 2, "Weight count"
assert weights[1] > weights[0], "Minority class higher weight"
print("✓ Class weights working")
if __name__ == "__main__":
print("Lab 1: ClassWeights - PASSED")### Lab 2: SMOTE Oversampling
import numpy as np
def smote_oversample(X_minority, k=5):
"""SMOTE: Synthetic Minority Oversampling"""
synthetic_samples = []
for i in range(len(X_minority)):
# Find k-nearest neighbors
distances = np.linalg.norm(X_minority - X_minority[i], axis=1)
knn_indices = np.argsort(distances)[1:k+1]
# Generate synthetic samples
for _ in range(k):
neighbor_idx = np.random.choice(knn_indices)
neighbor = X_minority[neighbor_idx]
# Linear interpolation
alpha = np.random.rand()
synthetic = X_minority[i] + alpha * (neighbor - X_minority[i])
synthetic_samples.append(synthetic)
return np.array(synthetic_samples)
# Test
np.random.seed(42)
X_minority = np.random.randn(20, 5)
synthetic = smote_oversample(X_minority, k=5)
assert synthetic.shape[0] >= len(X_minority), "At least original size"
print("✓ SMOTE working")
if __name__ == "__main__":
print("Lab 2: SMOTE - PASSED")### Lab 3: Weighted Loss
import numpy as np
def weighted_cross_entropy(logits, targets, class_weights):
"""Weighted cross-entropy loss"""
exp_logits = np.exp(logits - np.max(logits, axis=1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=1, keepdims=True)
batch_size = len(logits)
loss_per_sample = -np.log(probs[np.arange(batch_size), targets] + 1e-8)
# Apply class weights
weights = class_weights[targets]
weighted_loss = (loss_per_sample * weights).mean()
return weighted_loss
# Test
np.random.seed(42)
logits = np.random.randn(32, 2)
targets = np.random.randint(0, 2, 32)
class_weights = np.array([1.0, 10.0])
loss = weighted_cross_entropy(logits, targets, class_weights)
assert np.isfinite(loss), "Loss finite"
print("✓ Weighted loss working")
if __name__ == "__main__":
print("Lab 3: WeightedLoss - PASSED")### Lab 4: Threshold Optimization
import numpy as np
def optimize_threshold(predictions, targets, metric='f1'):
"""Find optimal classification threshold"""
thresholds = np.linspace(0, 1, 100)
best_threshold = 0.5
best_score = 0
for threshold in thresholds:
predicted = (predictions >= threshold).astype(int)
# Compute metric
tp = ((predicted == 1) & (targets == 1)).sum()
fp = ((predicted == 1) & (targets == 0)).sum()
fn = ((predicted == 0) & (targets == 1)).sum()
if metric == 'f1':
precision = tp / (tp + fp + 1e-8)
recall = tp / (tp + fn + 1e-8)
score = 2 * precision * recall / (precision + recall + 1e-8)
elif metric == 'gmean':
spec = 1 - (fp / ((fp + ((targets == 0).sum())) + 1e-8))
sens = tp / (tp + fn + 1e-8)
score = np.sqrt(spec * sens)
else:
score = tp / (tp + fn + 1e-8) # Recall
if score > best_score:
best_score = score
best_threshold = threshold
return best_threshold
# Test
np.random.seed(42)
preds = np.random.rand(100)
targets = np.random.randint(0, 2, 100)
threshold = optimize_threshold(preds, targets, metric='f1')
assert 0 <= threshold <= 1, "Valid threshold"
print("✓ Threshold optimization working")
if __name__ == "__main__":
print("Lab 4: ThresholdOptimization - PASSED")