loss function basics

**Loss Function** — the mathematical function that measures how wrong the model's predictions are, providing the signal that guides training through gradient descent. **Classification Losses** - **Cross-Entropy Loss**: $L = -\sum y_i \log(\hat{y}_i)$ — standard for classification. Penalizes confident wrong predictions heavily - **Binary Cross-Entropy (BCE)**: For two-class problems or multi-label classification - **Focal Loss**: Down-weights easy examples, focuses on hard ones. Developed for object detection with class imbalance **Regression Losses** - **MSE (Mean Squared Error)**: $L = \frac{1}{n}\sum(y - \hat{y})^2$ — penalizes large errors quadratically - **MAE (Mean Absolute Error)**: $L = \frac{1}{n}\sum|y - \hat{y}|$ — more robust to outliers - **Huber Loss**: MSE for small errors, MAE for large errors (best of both) **Other Important Losses** - **Contrastive Loss**: Pull similar pairs together, push dissimilar apart (CLIP, SimCLR) - **Triplet Loss**: Anchor closer to positive than negative by margin - **KL Divergence**: Measure difference between two probability distributions (used in VAE, knowledge distillation) - **CTC Loss**: For sequence-to-sequence without alignment (speech recognition) **Choosing the right loss function** is one of the most impactful design decisions — it directly defines what the model optimizes for.