temporal ensembling

**Temporal Ensembling** is a **semi-supervised learning method that maintains an exponential moving average of each sample's prediction over training epochs** — using these accumulated predictions as soft targets for a consistency loss on unlabeled data. **How Does Temporal Ensembling Work?** - **Accumulate**: After each epoch, update the EMA prediction for each sample: $ ilde{z}_i = alpha ilde{z}_i + (1-alpha) z_i$. - **Target**: Use the corrected EMA $hat{z}_i = ilde{z}_i / (1 - alpha^t)$ as the target. - **Consistency Loss**: $mathcal{L} = ||z_i - hat{z}_i||^2$ (current prediction should match accumulated predictions). - **Paper**: Laine & Aila (2017). **Why It Matters** - **Ensemble Effect**: The EMA predictions implicitly ensemble the model across training epochs. - **Simple**: No additional model or parameters (just a prediction buffer). - **Limitation**: Targets update only once per epoch (slow). Mean Teacher addresses this. **Temporal Ensembling** is **memory of past predictions** — using the accumulated history of a sample's predictions as a stable learning target.