temperature in distillation

**Temperature in Distillation** is the **softmax scaling parameter $ au$ used to control the smoothness of the teacher's output distribution** — higher temperature produces softer probabilities that reveal more dark knowledge, while lower temperature produces sharper, more confident distributions. **How Does Temperature Work?** - **Softmax**: $p_i = frac{exp(z_i / au)}{sum_j exp(z_j / au)}$ - **$ au = 1$**: Standard softmax. One class dominates. - **$ au = 5-20$**: Softer distribution. Non-dominant classes become visible. - **$ au ightarrow infty$**: Uniform distribution (maximum entropy). - **Training**: Both teacher and student use the same $ au$ during distillation. **Why It Matters** - **Information Extraction**: Higher $ au$ extracts more dark knowledge from the teacher's logits. - **Typical Values**: $ au = 3-10$ works well in practice. Too high dilutes the signal. - **Scaling**: The distillation loss is multiplied by $ au^2$ to maintain gradient magnitude across temperatures. **Temperature** is **the zoom lens on dark knowledge** — adjusting how much inter-class similarity information is exposed from the teacher's output distribution.