InfoNCE Loss is a contrastive learning objective that estimates mutual information between representations — by training a model to identify the correct "positive" sample from a set of "negative" distractors, forming the core loss function behind CPC, MoCo, and SimCLR.

What Is InfoNCE?

• Formula: $mathcal{L} = -log frac{exp(sim(z_i, z_j^+)/ au)}{sum_{k=0}^{K} exp(sim(z_i, z_k)/ au)}$
• Positive Pair ($z_i, z_j^+$): Two augmented views of the same sample.
• Negatives ($z_k$): All other samples in the batch (or memory bank).
• Temperature ($ au$): Controls the sharpness of the distribution.

Why It Matters

• Foundation: The mathematical engine behind modern contrastive self-supervised learning.
• Mutual Information: Lower bound on the mutual information $I(X; Z)$ between input and representation.
• Scalability: Performance improves with more negatives (larger batch size or memory bank).

InfoNCE is the core loss function of contrastive learning — teaching representations by distinguishing the real match from thousands of imposters.