simsiam
**SimSiam** (Simple Siamese Networks) is a **self-supervised representation learning method that learns useful visual representations without requiring negative sample pairs, large batches, or momentum encoders — achieving competitive performance with contrastive methods through a remarkably minimal architecture consisting of two weight-sharing encoders and a stop-gradient operation that prevents representational collapse** — published by Kaiming He et al. (Facebook AI Research, 2021) as a theoretical and empirical demonstration that the seemingly essential components of contrastive self-supervised learning were not actually necessary.
**What Is SimSiam?**
- **Siamese Network**: Two identical encoder networks (sharing weights) process two differently augmented views of the same image — each producing a feature vector.
- **Predictor MLP**: One branch passes its representation through an additional small MLP (the predictor) before computing the similarity loss.
- **Stop-Gradient**: The other branch's representation is treated as a constant — gradients are not propagated through it during the backward pass.
- **Loss Function**: Negative cosine similarity between the predicted representation of one branch and the stopped-gradient representation of the other — minimized to encourage the two views to agree.
- **No Negatives, No Momentum**: Unlike SimCLR (needs large batches of negatives) and BYOL/MoCo (needs momentum encoder), SimSiam needs neither.
**Why Stop-Gradient Prevents Collapse**
The critical question: why doesn't SimSiam collapse to a trivial solution (all representations identical)?
- **Intuitive Answer**: Stop-gradient creates an implicit expectation-maximization (EM) algorithm — one branch optimizes the predictor to match a "target" (the other branch), while the target is periodically updated by weight sharing. Neither branch fully controls the target.
- **Theoretical Analysis**: With stop-gradient, SimSiam alternates between optimizing the predictor (E-step: find best prediction of fixed representations) and updating the encoder (M-step: improve representations to make them more predictable).
- **Symmetrical Update**: The loss is computed in both directions (both branches act as predictor and target), stabilizing the asymmetric update.
**Comparison with Alternatives**
| Method | Negatives Needed | Momentum Encoder | Large Batch | Stop-Gradient |
|--------|-----------------|-----------------|------------|---------------|
| **SimCLR** | Yes (2×4096) | No | Yes | No |
| **MoCo v2** | Yes (queue 65536) | Yes | No | No |
| **BYOL** | No | Yes | No | No |
| **SimSiam** | No | No | No | Yes |
| **Barlow Twins** | No | No | No | No (cross-corr) |
**Performance and Impact**
- **ImageNet Linear Evaluation**: SimSiam achieves ~71% top-1 with ResNet-50 — competitive with SimCLR and BYOL despite the greater simplicity.
- **Transfer Learning**: Features transfer well to detection and segmentation on COCO and Pascal VOC.
- **Theoretical Impact**: SimSiam drove the field to understand *why* self-supervised methods work — the stop-gradient analysis revealed implicit EM optimization hiding in SSL methods.
SimSiam is **the minimalist proof that self-supervised learning needs less than we thought** — its discovery that useful representations emerge from simple similarity maximization with stop-gradient reshaped the theory of SSL and inspired a generation of even simpler, more scalable methods.