supervised contrastive learning class-level positive pairs

# Supervised Contrastive Learning: Class-Level Positive Pairs

## Introduction & Motivation

Supervised contrastive learning extends contrastive methods to labeled data. Positive pairs: same class (multiple samples). Learns discriminative representations; outperforms standard cross-entropy baseline. Stronger inductive bias; class separability explicit in loss. Applications: classification, metric learning, open-set recognition.

Motivation: Leverage labels to define positive pairs explicitly. Cross-entropy only constrains decision boundary; contrastive enforces inter-class separation throughout embedding space.

Applications: Metric learning, image retrieval, face recognition, open-set classification.

---

## Core Concepts & Theory

### Positive Set

All samples of same class treated as positives; enable large positive sets. Batch must have multiple samples per class for effectiveness.

### Inter-Class Separation

Negative pairs span different classes; distance in embedding space reflects class distance.

### Temperature & Concentration

Lower temperature concentrates mass on hard negatives; affects convergence.

---

## Mathematical Formulation

Supervised contrastive loss:
$$\ell_i = -\frac{1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp( ext{sim}(z_i, z_p) / au)}{\sum_{a=1}^{N} \exp( ext{sim}(z_i, z_a) / au)}$$

Relation to cross-entropy:
Supervised contrastive generalizes cross-entropy; at τ→0, becomes nearest-class-mean classifier.

---

## Advanced Theory & Extensions

### Hard Negative Mining

Focus on hard negatives (closest different-class); online hard negative queue.

### Multi-Prototype

Class represented by multiple prototypes; handles class imbalance.

### Focal Contrastive

Weight negatives by margin; hard negatives weighted higher.

---

## Computational Considerations

Batch composition: Require multiple samples per class; affects batch construction.

Positive set size: Larger → more stable, but requires batch strategy.

Computational cost: O(batch_size × |P(i)|) per sample.

---

## Practical Implementation Strategies

### Batch Construction

Ensure multiple samples per class (e.g., 4 samples × 8 classes = batch 32).

### Class-Aware Sampling

Sample classes uniformly; within-class random sampling.

### Mixup with Contrastive

Combine mixup augmentation with supervised contrastive.

---

## Benchmark Datasets & Evaluation

CIFAR-10/100: Standard classification; supervised contrastive +2-3% vs cross-entropy.

ImageNet: Large-scale; supervised contrastive strong baseline.

Metrics: Top-1/5 accuracy, embedding quality (clustering, purity).

---

## Key Challenges & Limitations

### Batch Dependency

Performance sensitive to batch composition; require class balance.

### Scalability

Large number of classes challenges; positives may be sparse.

### Hyperparameter Interaction

Temperature, learning rate, batch size tightly coupled.

---

## Hyperparameter Tuning

Temperature τ: 0.05-0.5; class-specific tuning possible.

Positive weight: 1.0 (equal to negatives); alternative: weight by |P(i)|.

Learning rate: 0.5-2.0 (higher than cross-entropy).

---

## Real-World Applications & Case Studies

CIFAR Classification: Supervised contrastive achieves 96.1% on CIFAR-10 (vs. 95.5% cross-entropy).

Face Recognition: Inter-person separation via contrastive loss.

Medical Diagnosis: Disease class separation in embedding space.

---

## Integration with Other Methods

Supervised contrastive + Self-supervised → pre-train self-supervised, fine-tune supervised contrastive.

Supervised contrastive + Metric learning → siamese networks with contrastive loss.

---

## Summary & Key Takeaways

Supervised contrastive learning leverages class labels to define positive pairs, achieving superior generalization via explicit inter-class separation in embedding space.

Principles:
1. Positive pairs: same-class samples; multiple per batch.
2. NT-Xent loss adapted for class-level positives.
3. Batch construction critical; require class-balanced sampling.
4. Lower temperature encourages hard negative mining.
5. Strong transfer learning baseline for downstream tasks.

---

---

## Appendix: Practical Labs

### Lab 1: Class-Stratified Batch

import torch
import numpy as np

def stratified_batch_sampler(labels, batch_size=32, samples_per_class=4):
 """Sample batch with equal samples per class"""
 unique_classes = torch.unique(labels)
 n_classes = len(unique_classes)
 
 if batch_size != n_classes * samples_per_class:
 raise ValueError(f"Batch size {batch_size} != n_classes {n_classes} * samples_per_class {samples_per_class}")
 
 batch_indices = []
 for c in unique_classes:
 class_indices = torch.where(labels == c)[0]
 sampled = np.random.choice(class_indices.cpu().numpy(), size=samples_per_class, replace=False)
 batch_indices.extend(sampled)
 
 return torch.tensor(batch_indices)

# Test
labels = torch.tensor([0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3])
batch_idx = stratified_batch_sampler(labels, batch_size=12, samples_per_class=3)

print(f"Batch indices: {batch_idx}")
assert len(batch_idx) == 12, "Should have batch_size samples"
print("✓ Stratified batch sampler working")

if __name__ == "__main__":
 print("Lab 1: Stratified Batch - PASSED")

### Lab 2: Positive Set Construction

import torch
import numpy as np

def construct_positive_set(labels, exclude_idx):
 """Construct positive set for sample at exclude_idx"""
 target_label = labels[exclude_idx]
 positive_indices = torch.where(labels == target_label)[0]
 
 # Remove self
 positive_indices = positive_indices[positive_indices != exclude_idx]
 
 return positive_indices

# Test
labels = torch.tensor([0, 0, 1, 1, 2, 2, 0])
pos_set = construct_positive_set(labels, exclude_idx=0)

print(f"Positive set for index 0 (label {labels[0]}): {pos_set}")
assert len(pos_set) >= 1, "Should have at least one positive"
assert labels[pos_set[0]] == labels[0], "All positives should match class"
print("✓ Positive set construction working")

if __name__ == "__main__":
 print("Lab 2: Positive Set - PASSED")

### Lab 3: Supervised Contrastive Loss

import torch
import torch.nn.functional as F
import numpy as np

def supervised_contrastive_loss(z, labels, tau=0.1):
 """Supervised contrastive loss with class-level positives"""
 batch_size = z.size(0)
 z_norm = F.normalize(z, dim=1)
 sim = torch.mm(z_norm, z_norm.T) / tau
 
 # Create positive mask (same class, excluding self)
 mask = (labels.unsqueeze(1) == labels.unsqueeze(0)).float()
 mask = mask - torch.eye(batch_size)
 
 # Numerator: positives
 exp_sim = torch.exp(sim)
 pos_sum = (mask * exp_sim).sum(dim=1, keepdim=True)
 
 # Denominator: all except self
 total_sum = exp_sim.sum(dim=1, keepdim=True) - torch.diag_embed(torch.diag(exp_sim))
 
 loss = -torch.log(pos_sum / total_sum + 1e-8)
 
 # Average over samples with positives
 valid = (mask.sum(dim=1) > 0)
 return loss[valid].mean() if valid.sum() > 0 else loss.mean()

# Test
z = torch.randn(16, 128)
labels = torch.tensor([0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 0, 1, 2, 3])

loss = supervised_contrastive_loss(z, labels, tau=0.1)

print(f"Supervised contrastive loss: {loss:.4f}")
assert 0 < loss < 10, "Loss should be in reasonable range"
assert np.isfinite(loss.item()), "Loss should be finite"
print("✓ Supervised contrastive loss working")

if __name__ == "__main__":
 print("Lab 3: Supervised Contrastive - PASSED")

### Lab 4: Class Separation Metric

import torch
import torch.nn.functional as F
import numpy as np

def compute_class_separation(z, labels):
 """Measure inter-class vs intra-class distance"""
 unique_labels = torch.unique(labels)
 z_norm = F.normalize(z, dim=1)
 
 intra_distances = []
 inter_distances = []
 
 for c1 in unique_labels:
 mask_c1 = labels == c1
 z_c1 = z_norm[mask_c1]
 
 # Intra-class: pairwise within class
 if len(z_c1) > 1:
 pairwise = torch.mm(z_c1, z_c1.T)
 intra_distances.append((1 - pairwise[~torch.eye(len(z_c1), dtype=torch.bool)]).mean().item())
 
 # Inter-class: distance to other classes
 for c2 in unique_labels:
 if c1 < c2:
 z_c2 = z_norm[labels == c2]
 pairwise = torch.mm(z_c1, z_c2.T)
 inter_distances.append((1 - pairwise).mean().item())
 
 intra_mean = np.mean(intra_distances) if intra_distances else 0
 inter_mean = np.mean(inter_distances) if inter_distances else 1
 
 return intra_mean, inter_mean

# Test
z = torch.randn(20, 64)
labels = torch.tensor([0]*5 + [1]*5 + [2]*5 + [3]*5)

intra, inter = compute_class_separation(z, labels)

print(f"Intra-class distance: {intra:.4f}, Inter-class: {inter:.4f}")
assert 0 <= intra <= 2, "Intra-class distance should be valid"
assert 0 <= inter <= 2, "Inter-class distance should be valid"
assert inter > intra, "Inter-class should be larger than intra-class"
print("✓ Class separation metric working")

if __name__ == "__main__":
 print("Lab 4: Class Separation - PASSED")