centering in self-supervised
**Centering in self-supervised learning** is the **target normalization strategy that subtracts a running mean from teacher logits so one class channel does not dominate training** - this keeps target distributions balanced and prevents trivial fixed-output solutions in non-label supervision pipelines.
**What Is Centering?**
- **Definition**: A moving-average correction applied to teacher outputs before softmax target generation.
- **Core Mechanism**: Subtract running center vector from teacher logits to remove persistent bias.
- **Primary Goal**: Prevent output collapse where every sample maps to nearly identical teacher probabilities.
- **Typical Use**: DINO-like student-teacher setups with multi-view consistency objectives.
**Why Centering Matters**
- **Collapse Resistance**: Reduces risk of constant class preference across all images.
- **Target Diversity**: Preserves spread across output dimensions for richer supervision.
- **Training Stability**: Smooths batch-to-batch drift in teacher target statistics.
- **Representation Quality**: Improves feature separability in downstream linear probing.
- **Recipe Compatibility**: Works with sharpening, momentum encoders, and multi-crop views.
**How Centering Works**
**Step 1**:
- Compute teacher logits for current batch and update an exponential moving average center vector.
- Keep the center update slow enough to avoid noisy oscillation.
**Step 2**:
- Subtract center vector from teacher logits before temperature scaling and softmax.
- Feed normalized soft targets to student loss for cross-view alignment.
**Practical Guidance**
- **Momentum Choice**: High center momentum improves stability in large-batch runs.
- **Monitoring**: Track per-dimension target entropy to detect imbalance early.
- **Numerics**: Compute center updates in float32 even when model trains in mixed precision.
Centering in self-supervised learning is **a small normalization step that prevents target bias from derailing representation learning** - it is one of the highest leverage stabilizers in modern non-label student-teacher training.