barlow twins

**Barlow Twins** is a **self-supervised learning method that learns representations by enforcing the cross-correlation matrix of embeddings to approach the identity matrix** — making the representation invariant to augmentations while avoiding redundancy between dimensions. **How Does Barlow Twins Work?** - **Input**: Two augmented views of each image, encoded into embeddings $Z_A$ and $Z_B$. - **Loss**: Cross-correlation matrix $C_{ij} = frac{sum_b z_{b,i}^A z_{b,j}^B}{sqrt{sum_b (z_{b,i}^A)^2}sqrt{sum_b (z_{b,j}^B)^2}}$. - **Objective**: Push diagonal elements toward 1 (invariance) and off-diagonal toward 0 (reduce redundancy). - **Inspiration**: Neuroscientist Horace Barlow's redundancy-reduction hypothesis. **Why It Matters** - **Simple**: No momentum encoder, no memory bank, no asymmetric architectures. - **No Negatives**: Like BYOL, avoids the need for explicit negative samples. - **Conceptual Elegance**: Directly optimizes information-theoretic properties of the representation. **Barlow Twins** is **making features independent and informative** — using a redundancy-reduction principle from neuroscience to learn powerful, non-degenerate representations.