selective knowledge distillation

**Selective Knowledge Distillation** is a **distillation approach that carefully chooses which knowledge to transfer from teacher to student** — rather than blindly mimicking all teacher outputs, selectively transferring only the most informative or relevant knowledge for the student's capacity. **How Does Selective KD Work?** - **Sample Selection**: Focus on hard or informative samples where the teacher's guidance is most valuable. - **Channel Selection**: Transfer only the most important feature channels, not all intermediate representations. - **Class Selection**: For many-class problems, distill from the top-k most relevant classes only. - **Confidence-Based**: Weight the distillation loss by teacher's confidence — focus on samples where teacher is most certain. **Why It Matters** - **Efficiency**: Not all teacher knowledge is equally useful for the student. Selective transfer avoids noise. - **Capacity Match**: A small student may not have capacity to absorb everything — selective KD prioritizes. - **Performance**: Often outperforms full distillation by reducing the "noise" of irrelevant teacher signals. **Selective Knowledge Distillation** is **curated mentoring** — choosing the most important lessons to teach rather than overwhelming the student with everything.