MoCo (Momentum Contrast) enables contrastive learning with large negative sample pools using a momentum encoder. Problem solved: Contrastive learning needs many negatives, but large batches are expensive. MoCo decouples batch size from negative count. Mechanism: Query encoder (updated by gradient), key encoder (momentum-updated copy), queue of recent keys as negatives. Contrastive loss with query, positive key, and queue negatives. Momentum update: θ_k = m·θ_k + (1-m)·θ_q, typical m=0.999. Slow update keeps key encoder consistent, prevents representation drift. Queue: FIFO queue stores recent mini-batch keys as negatives. Queue size (e.g., 65536) >> batch size. InfoNCE loss: Match query to positive key against all queue negatives. MoCo v2: Improved augmentations (from SimCLR), MLP projection head, cosine learning rate. MoCo v3: Applied to Vision Transformers, replaced queue with batch negatives. Advantages: Memory efficient, works with normal batch sizes, consistent representations through momentum encoder. Impact: Demonstrated self-supervised can match supervised ImageNet pre-training. Influential architecture for contrastive learning.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.