infonce loss

**InfoNCE Loss** is a **contrastive learning objective that estimates mutual information between representations** — by training a model to identify the correct "positive" sample from a set of "negative" distractors, forming the core loss function behind CPC, MoCo, and SimCLR. **What Is InfoNCE?** - **Formula**: $mathcal{L} = -log frac{exp(sim(z_i, z_j^+)/ au)}{sum_{k=0}^{K} exp(sim(z_i, z_k)/ au)}$ - **Positive Pair** ($z_i, z_j^+$): Two augmented views of the same sample. - **Negatives** ($z_k$): All other samples in the batch (or memory bank). - **Temperature** ($ au$): Controls the sharpness of the distribution. **Why It Matters** - **Foundation**: The mathematical engine behind modern contrastive self-supervised learning. - **Mutual Information**: Lower bound on the mutual information $I(X; Z)$ between input and representation. - **Scalability**: Performance improves with more negatives (larger batch size or memory bank). **InfoNCE** is **the core loss function of contrastive learning** — teaching representations by distinguishing the real match from thousands of imposters.