memory bank

**Memory Bank** is a **data structure used in contrastive self-supervised learning to store a large collection of negative sample representations** — enabling effective contrastive learning with small batch sizes by decoupling the number of negatives from the batch size. **What Is a Memory Bank?** - **Structure**: A dictionary/queue storing feature vectors from previous forward passes. - **Size**: Typically 4K-65K entries (much larger than a single batch). - **Update**: Features are computed with the current encoder and stored. Older entries are replaced (FIFO or random). - **Used By**: MoCo (momentum-updated queue), InstDisc (full memory bank). **Why It Matters** - **GPU Efficiency**: Small batches fit on any GPU, but the memory bank provides thousands of negatives for the contrastive loss. - **Staleness Trade-off**: Stored features were computed by an older version of the encoder -> stale representations. - **MoCo Solution**: Uses a slowly-updated momentum encoder to reduce staleness. **Memory Bank** is **the archive of past representations** — a clever trick that provides a large, diverse pool of negatives without requiring massive batch sizes.