mutual learning

**Mutual Learning** is a **collaborative training strategy where two or more networks train simultaneously and teach each other** — each network uses the other's soft predictions as an additional supervisory signal, improving both models beyond what either could achieve alone. **How Does Mutual Learning Work?** - **Setup**: Two (or more) networks with the same or different architectures, trained on the same data. - **Loss**: Each network optimizes: $mathcal{L} = mathcal{L}_{CE} + alpha cdot D_{KL}(p_1 || p_2)$ (and vice versa). - **No Pre-Training**: Unlike traditional KD, no pre-trained teacher is needed. - **Paper**: Zhang et al., "Deep Mutual Learning" (2018). **Why It Matters** - **Mutual Improvement**: Even two identical networks improve each other through mutual learning (surprising result). - **Ensemble Effect**: Each network benefits from the regularizing effect of the other's predictions. - **Efficiency**: Achieves distillation benefits without the cost of pre-training a large teacher model. **Mutual Learning** is **peer tutoring for neural networks** — two models learning together and teaching each other, achieving better results than studying alone.