online distillation

**Online Distillation** is a **knowledge distillation approach where teacher and student networks are trained simultaneously** — rather than the traditional offline approach where the teacher is pre-trained and fixed. Both networks learn from each other during training. **How Does Online Distillation Work?** - **Mutual Learning** (DML): Two networks are trained in parallel. Each one uses the other's soft predictions as additional supervision. - **Co-Distillation**: Multiple models exchange knowledge during training rounds. - **ONE (One-for-all)**: A single multi-branch network where branches distill knowledge to each other. - **No Pre-Training**: Unlike offline KD, no separate teacher training phase is needed. **Why It Matters** - **Efficiency**: Eliminates the expensive pre-training phase for the teacher model. - **Mutual Benefit**: Both networks improve from the knowledge exchange — even models of the same size benefit. - **Ensemble Effect**: The aggregated knowledge from multiple online students often exceeds any single model. **Online Distillation** is **collaborative learning between networks** — where models teach each other simultaneously, improving together without a pre-trained teacher.