Vision Transformers ViT

# Loss Functions for Different Tasks

## Introduction & Motivation

Loss Functions: measure prediction error. Cross-entropy, MSE, Triplet loss. Applications: objective definition, training guidance.

Motivation: Define task-specific optimization targets.

Applications: Classification, regression, metric learning.

---

## Core Concepts & Theory

### Classification Losses

Cross-entropy, focal loss.

### Regression Losses

MSE, MAE, Huber.

### Ranking Losses

Contrastive, triplet.

### Specialized Losses

Dice, Focal, focal-tversky.

---

## Mathematical Formulation

Cross-Entropy: L = -\sum_i y_i \log(\hat{y}_i)

Focal Loss: L = -\sum_i (1-p_t)^\gamma \log(p_t)

Triplet Loss: L = \max(d(a, p) - d(a, n) + m, 0)

Dice Loss: L = 1 - \frac{2|X \cap Y|}{|X| + |Y|}

---

## Advanced Theory & Extensions

### Weighted Loss

Class balancing.

### Label Smoothing

Confidence calibration.

### Multi-Margin Losses

Multiple objectives.

---

## Computational Considerations

Cross-entropy: O(batch·classes).

Triplet: O(batch²).

Focal: O(batch·classes).

---

## Practical Implementation Strategies

### Loss Weighting

Class imbalance handling.

### Negative Mining

Hard example selection.

### Loss Annealing

Dynamic weight adjustment.

---

## Benchmark Datasets & Evaluation

ImageNet: Standard benchmarks.

Imbalanced datasets: Loss effectiveness.

Medical imaging: Dice/Focal comparison.

---

## Key Challenges & Limitations

### Numerical Stability

Log computations.

### Class Imbalance

Rare class weighting.

### Hyperparameter Tuning

Loss configuration.

---

## Hyperparameter Tuning

Focal gamma: 0-3.

Triplet margin: 0.5-2.0.

Label smoothing: 0.1-0.3.

---

## Real-World Applications & Case Studies

Image Classification: Cross-entropy.

Medical Segmentation: Dice/Focal.

Face Recognition: Triplet/ArcFace.

---

## Integration with Other Methods

Losses + metrics for evaluation; + weighting for imbalance handling.

---

## Summary & Key Takeaways

Loss Functions guide optimization toward task objectives.

Principles:
1. Cross-entropy: Probability matching.
2. Focal loss: Hard example focus.
3. Triplet loss: Metric learning.
4. Dice loss: Overlap maximization.
5. Weighting: Imbalance handling.

---

## Appendix: Practical Labs

### Lab 1: Cross-Entropy Loss

import numpy as np

def cross_entropy_loss(logits, labels):
 batch_size = logits.shape[0]
 probs = np.exp(logits) / np.sum(np.exp(logits), axis=1, keepdims=True)
 loss = -np.mean(np.log(probs[np.arange(batch_size), labels] + 1e-8))
 return loss

np.random.seed(42)
logits = np.random.randn(32, 10)
labels = np.random.randint(0, 10, 32)
loss = cross_entropy_loss(logits, labels)
assert np.isfinite(loss), "Loss is finite"
print("✓ Cross-entropy loss working")

### Lab 2: Focal Loss

import numpy as np

def focal_loss(logits, labels, gamma=2.0, alpha=0.25):
 probs = np.exp(logits) / np.sum(np.exp(logits), axis=1, keepdims=True)
 batch_size = logits.shape[0]
 p_t = probs[np.arange(batch_size), labels]
 loss = -alpha * (1 - p_t) ** gamma * np.log(p_t + 1e-8)
 return np.mean(loss)

np.random.seed(42)
logits = np.random.randn(32, 10)
labels = np.random.randint(0, 10, 32)
loss = focal_loss(logits, labels)
assert np.isfinite(loss), "Focal loss finite"
print("✓ Focal loss working")

### Lab 3: Triplet Loss

import numpy as np

def triplet_loss(anchor, positive, negative, margin=1.0):
 d_ap = np.linalg.norm(anchor - positive, axis=1)
 d_an = np.linalg.norm(anchor - negative, axis=1)
 loss = np.maximum(d_ap - d_an + margin, 0)
 return np.mean(loss)

np.random.seed(42)
anchor = np.random.randn(32, 128)
positive = anchor + np.random.randn(32, 128) * 0.1
negative = np.random.randn(32, 128)
loss = triplet_loss(anchor, positive, negative)
assert loss >= 0, "Triplet loss non-negative"
print("✓ Triplet loss working")

### Lab 4: Dice Loss

import numpy as np

def dice_loss(predictions, targets, smooth=1.0):
 intersection = np.sum(predictions * targets)
 union = np.sum(predictions) + np.sum(targets)
 dice = (2 * intersection + smooth) / (union + smooth)
 return 1 - dice

np.random.seed(42)
preds = np.random.rand(32, 128, 128)
targets = np.random.rand(32, 128, 128).round()
loss = dice_loss(preds, targets)
assert 0 <= loss <= 1, "Dice loss in valid range"
print("✓ Dice loss working")

---