self-distillation

**Self-Distillation** is a **knowledge distillation technique where the teacher and student share the same architecture** — the model distills knowledge into itself, either by using a deeper version as teacher, using earlier training checkpoints, or distilling from the full model into auxiliary classifiers at intermediate layers. **How Does Self-Distillation Work?** - **Same Architecture**: Teacher and student have identical structure (unlike traditional KD where teacher is larger). - **Variants**: - **Born-Again Networks**: Train student = teacher architecture on teacher's soft labels. - **DINO**: EMA teacher provides targets for the student (self-distillation with momentum). - **Intermediate Classifiers**: Auxiliary classifiers at hidden layers distill from the final classifier. - **Surprise**: Self-distilled models often outperform the original teacher! **Why It Matters** - **Free Performance**: Improves accuracy without increasing model size or changing architecture. - **Label Smoothing Effect**: Soft targets provide richer training signal than hard labels. - **Foundation Models**: DINO and DINOv2 are fundamentally self-distillation frameworks. **Self-Distillation** is **the student becoming the teacher** — a model improving itself by learning from its own refined outputs.