loss functions cross-entropy focal loss class imbalance
# Loss Functions: Cross-Entropy, Focal Loss & Class Imbalance
## Introduction & Motivation
Loss function: optimization objective. Cross-entropy: standard classification. Focal loss: address class imbalance via down-weighting easy examples. Smooth L1: robust regression. Applications: all supervised learning; critical for performance.
Motivation: Loss directly drives optimization; choice impacts convergence, final performance, robustness.
Applications: Classification, regression, object detection, imbalanced datasets.
---
## Core Concepts & Theory
### Cross-Entropy
Standard log loss; symmetric treatment of classes.
### Focal Loss
Modulation term: (1-p_t)^γ; focus on hard examples.
### Class Weights
Scale loss by inverse class frequency; address imbalance.
---
## Mathematical Formulation
Cross-entropy (classification):
$$L = -\sum_c y_c \log(\hat{y}_c)$$
Focal Loss:
$$L = -\alpha (1 - p_t)^\gamma \log(p_t)$$
where p_t = predicted probability for true class, γ = focusing parameter.
Smooth L1 (Huber loss):
$$L = \begin{cases} 0.5 x^2 & |x| < 1 \\ |x| - 0.5 & |x| \geq 1 \end{cases}$$
---
## Advanced Theory & Extensions
### Lovasz-Softmax
Direct IoU optimization for segmentation.
### Label Smoothing
Soften targets; regularization effect.
### Curriculum Loss
Weight examples by difficulty; gradual curriculum.
---
## Computational Considerations
Cross-entropy: O(N·C) for N samples, C classes.
Focal loss: O(N·C) with modulation.
Class weights: O(1) per sample; overhead negligible.
---
## Practical Implementation Strategies
### Weighting Scheme
Inverse frequency, effective number, learnable.
### Focal Parameter γ
Typically 2; balance between hard/easy.
### Label Smoothing α
0.1-0.2; regularization strength.
---
## Benchmark Datasets & Evaluation
CIFAR-10: Cross-entropy standard baseline.
COCO (Detection): Focal loss standard; 3-4% AP improvement.
Imbalanced Data: Class weights essential.
---
## Key Challenges & Limitations
### Hyperparameter Sensitivity
Focal γ, weights, smoothing interact.
### Computational Stability
Numerical issues with extreme probabilities.
### Task Dependency
Optimal loss function problem-specific.
---
## Hyperparameter Tuning
Focal γ: 1.5-2.5; balance easy/hard.
Class weights: Inverse frequency or effective number.
Label smoothing: 0.0-0.2; regularization.
---
## Real-World Applications & Case Studies
Object Detection: Focal loss reduces background class dominance.
Medical Imaging: Class weights for rare diseases.
NLP: Cross-entropy standard; label smoothing helps.
---
## Integration with Other Methods
Loss + Regularization → combined objective.
Loss + Class Weights → tailored optimization.
---
## Summary & Key Takeaways
Loss functions via cross-entropy and focal loss guide optimization, with class weighting addressing imbalance through reweighting and hard example focus.
Principles:
1. Cross-entropy: symmetric log loss.
2. Focal loss: down-weight easy examples.
3. Class weights: inverse frequency balancing.
4. Label smoothing: soft targets as regularization.
5. Task-dependent: choice impacts convergence and performance.
---
---
## Appendix: Practical Labs
### Lab 1: Cross-Entropy Loss
import torch
import torch.nn.functional as F
import numpy as np
def cross_entropy_loss(logits, targets):
"""Compute cross-entropy loss manually"""
# logits: [N, C]
# targets: [N] (class indices)
log_probs = F.log_softmax(logits, dim=1)
loss = -log_probs.gather(1, targets.unsqueeze(1)).mean()
return loss
# Test
logits = torch.randn(32, 10)
targets = torch.randint(0, 10, (32,))
loss = cross_entropy_loss(logits, targets)
assert loss > 0, "Loss should be positive"
assert np.isfinite(loss.item()), "Loss should be finite"
print(f"Cross-entropy loss: {loss:.4f}")
print("✓ Cross-entropy working")
if __name__ == "__main__":
print("Lab 1: CE - PASSED")### Lab 2: Focal Loss
import torch
import torch.nn.functional as F
import numpy as np
def focal_loss(logits, targets, gamma=2.0, alpha=0.25):
"""Compute focal loss for class imbalance"""
log_probs = F.log_softmax(logits, dim=1)
probs = torch.exp(log_probs)
# Get probability of true class
p_t = probs.gather(1, targets.unsqueeze(1)).squeeze(1)
# Focal loss: -alpha * (1 - p_t)^gamma * log(p_t)
focal = -alpha * (1 - p_t) ** gamma * torch.log(p_t + 1e-8)
return focal.mean()
# Test
logits = torch.randn(32, 10)
targets = torch.randint(0, 10, (32,))
loss = focal_loss(logits, targets, gamma=2.0)
assert loss > 0, "Loss should be positive"
print(f"Focal loss: {loss:.4f}")
print("✓ Focal loss working")
if __name__ == "__main__":
print("Lab 2: Focal - PASSED")### Lab 3: Class Weighting
import torch
import numpy as np
def compute_class_weights(targets, num_classes):
"""Compute weights as inverse class frequency"""
class_counts = torch.bincount(targets, minlength=num_classes).float()
weights = 1.0 / (class_counts + 1e-8)
weights = weights / weights.sum() * num_classes
return weights
# Test
targets = torch.tensor([0, 0, 0, 1, 1, 2, 2, 2, 2]) # Imbalanced
num_classes = 3
weights = compute_class_weights(targets, num_classes)
print(f"Class weights: {weights}")
assert weights.shape == (num_classes,), "Should have weights per class"
assert (weights > 0).all(), "Weights should be positive"
print("✓ Class weighting working")
if __name__ == "__main__":
print("Lab 3: Weighting - PASSED")### Lab 4: Loss Comparison
import torch
import torch.nn.functional as F
import numpy as np
def compare_losses(logits, targets):
"""Compare different loss functions"""
ce = F.cross_entropy(logits, targets)
focal = focal_loss(logits, targets, gamma=2.0)
return {'ce': ce.item(), 'focal': focal.item()}
def focal_loss(logits, targets, gamma=2.0, alpha=0.25):
log_probs = F.log_softmax(logits, dim=1)
probs = torch.exp(log_probs)
p_t = probs.gather(1, targets.unsqueeze(1)).squeeze(1)
focal = -alpha * (1 - p_t) ** gamma * torch.log(p_t + 1e-8)
return focal.mean()
# Test
logits = torch.randn(32, 10)
targets = torch.randint(0, 10, (32,))
losses = compare_losses(logits, targets)
print(f"Loss comparison: CE={losses['ce']:.4f}, Focal={losses['focal']:.4f}")
assert losses['ce'] > 0 and losses['focal'] > 0, "Losses should be positive"
print("✓ Loss comparison working")
if __name__ == "__main__":
print("Lab 4: Comparison - PASSED")