loss functions cross-entropy focal loss contrastive
# Loss Functions: Cross-Entropy, Focal Loss & Contrastive
## Introduction & Motivation
Loss functions: model training objectives. Cross-entropy: classification standard. Focal loss: address class imbalance. Contrastive loss: learn embeddings; distance-based. Applications: classification, imbalanced data, metric learning, similarity learning.
Motivation: Different tasks require different loss functions. Appropriate loss design improves convergence and performance.
Applications: Classification, metric learning, ranking.
---
## Core Concepts & Theory
### Cross-Entropy Loss
Categorical classification; probabilistic interpretation.
### Focal Loss
Downweight easy examples; focus on hard negatives.
### Contrastive Loss
Minimize distance similar pairs; maximize dissimilar.
---
## Mathematical Formulation
Cross-entropy:
$$L = -\sum_c y_c \log(p_c)$$
where y = one-hot, p = predicted probability.
Focal loss:
$$L = -\alpha_t (1 - p_t)^\gamma \log(p_t)$$
where p_t = probability of ground truth, γ = focusing parameter.
Contrastive loss (Siamese):
$$L = (1-Y) \frac{1}{2}D^2 + Y \frac{1}{2}\{\max(0, m-D)\}^2$$
where Y = label, D = distance, m = margin.
---
## Advanced Theory & Extensions
### Triplet Loss
Anchor-positive-negative; margin separation.
### NT-Xent (InfoNCE)
Normalized temperature-scaled cross-entropy.
### ArcFace Loss
Angular margin; face recognition.
---
## Computational Considerations
Cross-entropy: O(C) per sample where C = classes.
Focal: Same as cross-entropy; scaling factor.
Contrastive: O(N²) pairwise distances; all-pairs.
---
## Practical Implementation Strategies
### Label Smoothing
Soften targets; improve generalization.
### Class Weighting
Balance imbalanced classes; weighted average.
### Hard Negative Mining
Focus on challenging examples; curriculum.
---
## Benchmark Datasets & Evaluation
CIFAR-10: Cross-entropy baseline; simple.
ImageNet: Focal loss for imbalanced subsets.
VoxCeleb: Contrastive loss for speaker verification.
---
## Key Challenges & Limitations
### Class Imbalance
Standard losses suboptimal; weighting needed.
### Convergence Behavior
Different losses vary stability; hyperparameter tuning.
### Computational Cost
Contrastive O(N²); approximations needed for scale.
---
## Hyperparameter Tuning
Focal loss α: 0.25-0.75; class frequency dependent.
Focal loss γ: 1.5-2.5; focusing strength.
Contrastive margin: 0.5-1.0; task dependent.
---
## Real-World Applications & Case Studies
Image Classification: Cross-entropy standard; focal for imbalance.
Face Recognition: ArcFace loss; large-scale deployment.
Metric Learning: Contrastive/triplet; embedding space.
---
## Integration with Other Methods
Loss + Regularization → complementary objectives.
Loss + Data Augmentation → improved robustness.
---
## Summary & Key Takeaways
Loss functions via cross-entropy, focal, and contrastive methods provide appropriate training objectives for classification and metric learning tasks.
Principles:
1. Cross-entropy: probabilistic classification.
2. Focal: handle class imbalance.
3. Contrastive: learn embeddings.
4. Weighting: balance classes.
5. Tuning: hyperparameter dependent.
---
---
## Appendix: Practical Labs
### Lab 1: Cross-Entropy Loss
import numpy as np
def cross_entropy_loss(logits, labels):
"""Categorical cross-entropy"""
# Softmax probabilities
exp_logits = np.exp(logits - np.max(logits, axis=-1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=-1, keepdims=True)
# Cross-entropy
batch_size = logits.shape[0]
ce_loss = -np.log(probs[np.arange(batch_size), labels] + 1e-8)
return ce_loss.mean()
# Test
np.random.seed(42)
logits = np.random.randn(32, 10)
labels = np.random.randint(0, 10, 32)
loss = cross_entropy_loss(logits, labels)
assert np.isfinite(loss), "Loss finite"
assert loss > 0, "Loss positive"
print("✓ Cross-entropy loss working")
if __name__ == "__main__":
print("Lab 1: CrossEntropy - PASSED")### Lab 2: Focal Loss
import numpy as np
def focal_loss(logits, labels, alpha=0.25, gamma=2.0):
"""Focal loss for imbalanced classification"""
# Softmax probabilities
exp_logits = np.exp(logits - np.max(logits, axis=-1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=-1, keepdims=True)
# Focal loss
batch_size = logits.shape[0]
p_t = probs[np.arange(batch_size), labels]
focal = -alpha * (1 - p_t) ** gamma * np.log(p_t + 1e-8)
return focal.mean()
# Test
np.random.seed(42)
logits = np.random.randn(32, 10)
labels = np.random.randint(0, 10, 32)
loss = focal_loss(logits, labels, alpha=0.25, gamma=2.0)
assert np.isfinite(loss), "Loss finite"
print("✓ Focal loss working")
if __name__ == "__main__":
print("Lab 2: FocalLoss - PASSED")### Lab 3: Contrastive Loss
import numpy as np
def contrastive_loss(embeddings, labels, margin=1.0):
"""Siamese contrastive loss"""
# Pairwise distances
distances = np.linalg.norm(embeddings[:, np.newaxis] - embeddings[np.newaxis, :], axis=-1)
# Labels: 1 if same, 0 if different
same_class = (labels[:, np.newaxis] == labels[np.newaxis, :]).astype(float)
# Loss
loss_pos = same_class * distances ** 2
loss_neg = (1 - same_class) * np.maximum(0, margin - distances) ** 2
return (loss_pos + loss_neg).mean() / 2
# Test
np.random.seed(42)
embeddings = np.random.randn(32, 64)
labels = np.random.randint(0, 10, 32)
loss = contrastive_loss(embeddings, labels, margin=1.0)
assert np.isfinite(loss), "Loss finite"
print("✓ Contrastive loss working")
if __name__ == "__main__":
print("Lab 3: ContrastiveLoss - PASSED")### Lab 4: Loss Comparison
import numpy as np
def compare_losses(logits, labels):
"""Compare different loss functions"""
# Cross-entropy
exp_logits = np.exp(logits - np.max(logits, axis=-1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=-1, keepdims=True)
batch_size = logits.shape[0]
p_t = probs[np.arange(batch_size), labels]
ce_loss = -np.log(p_t + 1e-8).mean()
# Focal loss
focal = -0.25 * (1 - p_t) ** 2.0 * np.log(p_t + 1e-8)
focal_loss = focal.mean()
# MSE loss (alternative)
one_hot = np.eye(logits.shape[1])[labels]
mse_loss = ((probs - one_hot) ** 2).mean()
return {
"cross_entropy": ce_loss,
"focal": focal_loss,
"mse": mse_loss
}
# Test
np.random.seed(42)
logits = np.random.randn(32, 10)
labels = np.random.randint(0, 10, 32)
losses = compare_losses(logits, labels)
assert all(np.isfinite(v) for v in losses.values()), "All losses finite"
print("✓ Loss comparison working")
if __name__ == "__main__":
print("Lab 4: LossComparison - PASSED")