dark knowledge

**Dark Knowledge** is the **rich information contained in a teacher model's soft output distribution** — the relative probabilities assigned to incorrect classes reveal the model's learned similarity structure, which is far more informative than the hard one-hot label. **What Is Dark Knowledge?** - **Example**: For an image of a cat, the teacher might output: cat=0.85, dog=0.10, fox=0.03, car=0.001. - **Information**: The high probability for "dog" tells the student that cats and dogs look similar. "Car" being near-zero teaches they are unrelated. - **Hard Labels**: Only say "cat." No information about similarity to other classes. - **Temperature**: Higher temperature ($ au$) softens the distribution, revealing more dark knowledge. **Why It Matters** - **Richer Supervision**: Dark knowledge provides orders of magnitude more information per training sample than hard labels. - **Generalization**: Students trained on soft targets generalize better because they learn inter-class relationships. - **Foundation**: The entire knowledge distillation framework is built on the insight that dark knowledge exists and is transferable. **Dark Knowledge** is **the hidden curriculum in a teacher's predictions** — the subtle class-similarity information that hard labels completely discard.