supervised contrastive learning

**Supervised Contrastive Learning (SupCon)** is an **extension of contrastive learning that leverages label information** — treating all samples of the same class as positives and samples of different classes as negatives, producing better-structured representations than standard cross-entropy training. **How Does SupCon Work?** - **Positive Set**: All augmented views of all samples with the same label (not just augmented views of the same instance). - **Loss**: $mathcal{L} = -sum_{i} frac{1}{|P(i)|} sum_{p in P(i)} log frac{exp(z_i cdot z_p / au)}{sum_{a eq i} exp(z_i cdot z_a / au)}$ - **Contrast**: Pull same-class representations together, push different-class representations apart. - **Training**: Two-stage — SupCon on the encoder, then cross-entropy on a linear classifier. **Why It Matters** - **Better Representations**: Produces more structured, class-aware feature spaces than cross-entropy alone. - **Robustness**: More robust to natural corruptions, label noise, and hyperparameter sensitivity. - **Transfer**: Better linear probe performance than cross-entropy-trained features. **Supervised Contrastive Learning** is **SimCLR with labels** — using class supervision to define positive pairs more accurately and learn cleaner decision boundaries.