multi-teacher distillation

**Multi-Teacher Distillation** is a **knowledge distillation approach where a single student learns from multiple teacher models simultaneously** — combining knowledge from diverse teachers that may have different architectures, training data, or areas of expertise. **How Does Multi-Teacher Work?** - **Aggregation**: Teacher predictions are combined by averaging, weighted averaging, or learned attention. - **Specialization**: Different teachers may specialize in different classes or domains. - **Loss**: $mathcal{L} = mathcal{L}_{CE} + sum_t alpha_t cdot mathcal{L}_{KD}(student, teacher_t)$ - **Ensemble-Like**: The student effectively distills the knowledge of an ensemble into a single model. **Why It Matters** - **Diversity**: Multiple teachers provide diverse perspectives, reducing bias and improving generalization. - **Ensemble Compression**: Compresses an ensemble of large models into one small model for deployment. - **Multi-Domain**: Teachers trained on different domains contribute complementary knowledge. **Multi-Teacher Distillation** is **learning from a panel of experts** — absorbing diverse knowledge from multiple specialists into a single efficient model.