moco (momentum contrast)

MoCo (Momentum Contrast) enables contrastive learning with large negative sample pools using a momentum encoder. **Problem solved**: Contrastive learning needs many negatives, but large batches are expensive. MoCo decouples batch size from negative count. **Mechanism**: Query encoder (updated by gradient), key encoder (momentum-updated copy), queue of recent keys as negatives. Contrastive loss with query, positive key, and queue negatives. **Momentum update**: θ_k = m·θ_k + (1-m)·θ_q, typical m=0.999. Slow update keeps key encoder consistent, prevents representation drift. **Queue**: FIFO queue stores recent mini-batch keys as negatives. Queue size (e.g., 65536) >> batch size. **InfoNCE loss**: Match query to positive key against all queue negatives. **MoCo v2**: Improved augmentations (from SimCLR), MLP projection head, cosine learning rate. **MoCo v3**: Applied to Vision Transformers, replaced queue with batch negatives. **Advantages**: Memory efficient, works with normal batch sizes, consistent representations through momentum encoder. **Impact**: Demonstrated self-supervised can match supervised ImageNet pre-training. Influential architecture for contrastive learning.