SimSiam (Simple Siamese Networks) is a self-supervised representation learning method that learns useful visual representations without requiring negative sample pairs, large batches, or momentum encoders — achieving competitive performance with contrastive methods through a remarkably minimal architecture consisting of two weight-sharing encoders and a stop-gradient operation that prevents representational collapse — published by Kaiming He et al. (Facebook AI Research, 2021) as a theoretical and empirical demonstration that the seemingly essential components of contrastive self-supervised learning were not actually necessary.

What Is SimSiam?

• Siamese Network: Two identical encoder networks (sharing weights) process two differently augmented views of the same image — each producing a feature vector.
• Predictor MLP: One branch passes its representation through an additional small MLP (the predictor) before computing the similarity loss.
• Stop-Gradient: The other branch's representation is treated as a constant — gradients are not propagated through it during the backward pass.
• Loss Function: Negative cosine similarity between the predicted representation of one branch and the stopped-gradient representation of the other — minimized to encourage the two views to agree.
• No Negatives, No Momentum: Unlike SimCLR (needs large batches of negatives) and BYOL/MoCo (needs momentum encoder), SimSiam needs neither.

Why Stop-Gradient Prevents Collapse

The critical question: why doesn't SimSiam collapse to a trivial solution (all representations identical)?

• Intuitive Answer: Stop-gradient creates an implicit expectation-maximization (EM) algorithm — one branch optimizes the predictor to match a "target" (the other branch), while the target is periodically updated by weight sharing. Neither branch fully controls the target.
• Theoretical Analysis: With stop-gradient, SimSiam alternates between optimizing the predictor (E-step: find best prediction of fixed representations) and updating the encoder (M-step: improve representations to make them more predictable).
• Symmetrical Update: The loss is computed in both directions (both branches act as predictor and target), stabilizing the asymmetric update.

Comparison with Alternatives

Performance and Impact

• ImageNet Linear Evaluation: SimSiam achieves ~71% top-1 with ResNet-50 — competitive with SimCLR and BYOL despite the greater simplicity.
• Transfer Learning: Features transfer well to detection and segmentation on COCO and Pascal VOC.
• Theoretical Impact: SimSiam drove the field to understand why self-supervised methods work — the stop-gradient analysis revealed implicit EM optimization hiding in SSL methods.

SimSiam is the minimalist proof that self-supervised learning needs less than we thought — its discovery that useful representations emerge from simple similarity maximization with stop-gradient reshaped the theory of SSL and inspired a generation of even simpler, more scalable methods.