self-distillation
**Self-distillation** is the **training strategy where a model learns from softened targets produced by another instance of itself or an exponential moving average teacher** - in vision transformers, it improves calibration, representation smoothness, and low-label transfer.
**What Is Self-Distillation?**
- **Definition**: Student model matches probabilistic outputs or features from a teacher derived from the same architecture family.
- **Teacher Sources**: EMA teacher, previous epoch checkpoint, or stronger pretrained variant.
- **Target Type**: Logits, intermediate features, or token-level distributions.
- **Objective Blend**: Distillation loss is combined with supervised or self-supervised base loss.
**Why Self-Distillation Matters**
- **Generalization Gain**: Soft targets carry richer uncertainty information than hard labels.
- **Calibration Improvement**: Reduces extreme confidence spikes in predictions.
- **Stability**: Teacher signal regularizes optimization, especially in deep models.
- **Label Efficiency**: Helps student perform better with fewer labeled examples.
- **Compression Path**: Supports transfer from large teacher to smaller deployment models.
**Distillation Configurations**
**Logit Distillation**:
- Student matches teacher class probability distribution.
- Temperature controls softness of targets.
**Feature Distillation**:
- Align intermediate representations between teacher and student.
- Useful for dense tasks and architecture transfer.
**Token Distillation**:
- Match patch-level outputs for richer spatial guidance.
- Effective in ViT pipelines.
**Implementation Guidance**
- **Temperature Tuning**: Too sharp targets reduce benefit, too soft can dilute signal.
- **Loss Weighting**: Balance distillation and base losses across training stages.
- **Teacher Stability**: EMA teachers often provide smoother supervision.
Self-distillation is **a high-impact regularization and transfer mechanism that lets models learn from structured soft supervision instead of only hard targets** - it is a core ingredient in modern ViT training recipes.