sharpening in self-supervised

**Sharpening in self-supervised learning** is the **temperature-based target transformation that makes teacher probability distributions more confident and less uniform** - by lowering temperature before softmax, training receives clearer discrimination signals across semantic dimensions. **What Is Sharpening?** - **Definition**: Applying low-temperature softmax to teacher logits to reduce entropy of target distributions. - **Core Effect**: Higher probability mass on a few dimensions and lower mass on irrelevant dimensions. - **Primary Role**: Improve supervisory signal strength in self-distillation losses. - **Common Pairing**: Typically used with centering to avoid trivial dominant channels. **Why Sharpening Matters** - **Signal Clarity**: Student receives less ambiguous targets and learns faster semantic structure. - **Collapse Prevention**: Uniform targets are discouraged, reducing non-informative solutions. - **Feature Separation**: Encourages sharper clusters in embedding space. - **Downstream Benefit**: Improves linear evaluation and retrieval ranking consistency. - **Stability Balance**: Proper temperature prevents both noisy and overconfident extremes. **How Sharpening Works** **Step 1**: - Compute centered teacher logits and divide by temperature value T below 1. - Lower T yields sharper target distribution, higher T yields softer distribution. **Step 2**: - Apply softmax to obtain teacher probabilities and train student to match targets. - Tune temperature schedule across epochs to balance stability and discrimination. **Practical Guidance** - **Temperature Range**: Values around 0.04 to 0.2 are common depending on architecture and objective. - **Schedule Design**: Warm temperature early can help stability, sharper temperature later can improve separation. - **Diagnostics**: Track target entropy and feature collapse indicators during training. Sharpening in self-supervised learning is **the decisiveness control that turns flat targets into informative supervision** - with centering and momentum updates, it becomes a core ingredient for high-quality representation learning.