mean teacher

**Mean Teacher** is a **semi-supervised learning method that maintains an exponential moving average (EMA) of model weights as the "teacher"** — the student model is trained on labeled data, while a consistency loss encourages the student to match the teacher's predictions on unlabeled data. **How Does Mean Teacher Work?** - **Student**: Trained with gradient descent on labeled loss + consistency loss. - **Teacher**: $ heta_{teacher} = alpha cdot heta_{teacher} + (1-alpha) cdot heta_{student}$ (EMA of student weights, $alpha approx 0.999$). - **Consistency**: $mathcal{L}_{cons} = ||f_{teacher}(x + ext{noise}) - f_{student}(x + ext{noise'})||^2$. - **Paper**: Tarvainen & Valpola (2017). **Why It Matters** - **Stable Targets**: The EMA teacher produces more stable and accurate targets than the student alone. - **Foundation**: Inspired BYOL (self-supervised), EMA-based methods in detection, and modern SSL frameworks. - **Effective**: Significant improvement over Temporal Ensembling with negligible additional compute. **Mean Teacher** is **the smoothed mentor** — an exponentially averaged version of the model that provides stable, high-quality targets for semi-supervised learning.