stop-gradient in self-supervised

**Stop-gradient in self-supervised learning** is the **operation that blocks gradient backpropagation through selected branches so target networks remain stable and collapse is avoided** - by freezing one side of the objective during each update, methods such as BYOL and DINO-style variants maintain directional learning signals. **What Is Stop-Gradient?** - **Definition**: Computational graph operation that treats tensor as constant during backpropagation. - **Typical Placement**: Applied on teacher outputs or target branch embeddings. - **Optimization Role**: Prevents mutual shortcut updates that can drive trivial solutions. - **Framework Support**: Implemented as detach operation in major deep learning libraries. **Why Stop-Gradient Matters** - **Collapse Resistance**: Blocks degenerate co-adaptation between student and teacher branches. - **Stable Targets**: Keeps supervision signal anchored while student learns. - **Convergence Quality**: Reduces oscillation and objective instability. - **Method Simplicity**: Achieves major stability gains with minimal implementation cost. - **Broad Utility**: Useful in self-distillation, contrastive variants, and hybrid objectives. **How It Is Used** **Teacher Branch Freeze**: - Teacher outputs are detached before loss computation. - Student receives gradient, teacher does not. **Symmetric Objectives**: - In two-view losses, stop-gradient may alternate across branches. - Maintains balanced learning dynamics. **Token-Level Settings**: - Patch targets can also be detached to stabilize dense objectives. - Helpful in masked token distillation methods. **Engineering Checks** - **Graph Verification**: Confirm no gradient flows into detached branch. - **Entropy Monitoring**: Detect collapse despite stop-gradient if other hyperparameters are mis-set. - **Loss Weighting**: Keep branch losses balanced to prevent dominance. Stop-gradient in self-supervised learning is **a critical stabilization primitive that keeps target signals fixed enough for meaningful representation learning** - it is one of the smallest code-level changes with one of the largest effects on self-supervised training reliability.