Temperature in Distillation is the softmax scaling parameter $ au$ used to control the smoothness of the teacher's output distribution — higher temperature produces softer probabilities that reveal more dark knowledge, while lower temperature produces sharper, more confident distributions.

How Does Temperature Work?

• Softmax: $p_i = frac{exp(z_i / au)}{sum_j exp(z_j / au)}$
• $ au = 1$: Standard softmax. One class dominates.
• $ au = 5-20$: Softer distribution. Non-dominant classes become visible.
• $ au ightarrow infty$: Uniform distribution (maximum entropy).
• Training: Both teacher and student use the same $ au$ during distillation.

Why It Matters

• Information Extraction: Higher $ au$ extracts more dark knowledge from the teacher's logits.
• Typical Values: $ au = 3-10$ works well in practice. Too high dilutes the signal.
• Scaling: The distillation loss is multiplied by $ au^2$ to maintain gradient magnitude across temperatures.

Temperature is the zoom lens on dark knowledge — adjusting how much inter-class similarity information is exposed from the teacher's output distribution.