Temporal Ensembling is a semi-supervised learning method that maintains an exponential moving average of each sample's prediction over training epochs — using these accumulated predictions as soft targets for a consistency loss on unlabeled data.

How Does Temporal Ensembling Work?

• Accumulate: After each epoch, update the EMA prediction for each sample: $ ilde{z}_i = alpha ilde{z}_i + (1-alpha) z_i$.
• Target: Use the corrected EMA $hat{z}_i = ilde{z}_i / (1 - alpha^t)$ as the target.
• Consistency Loss: $mathcal{L} = ||z_i - hat{z}_i||^2$ (current prediction should match accumulated predictions).
• Paper: Laine & Aila (2017).

Why It Matters

• Ensemble Effect: The EMA predictions implicitly ensemble the model across training epochs.
• Simple: No additional model or parameters (just a prediction buffer).
• Limitation: Targets update only once per epoch (slow). Mean Teacher addresses this.

Temporal Ensembling is memory of past predictions — using the accumulated history of a sample's predictions as a stable learning target.