teaching assistant

**Teaching Assistant (TA)** in knowledge distillation is a **technique that introduces an intermediate-sized model between a very large teacher and a very small student** — bridging the capacity gap that causes direct distillation to fail when the teacher is too powerful relative to the student. **How Does TA Work?** - **Problem**: When the capacity gap between teacher and student is too large, the student cannot effectively learn from the teacher's complex output distribution. - **Solution**: Train an intermediate "teaching assistant" model from the teacher first, then use the TA to train the final student. - **Chain**: Teacher -> TA -> Student. Each step has a manageable capacity gap. - **Paper**: Mirzadeh et al., "Improved Knowledge Distillation via Teacher Assistant" (2020). **Why It Matters** - **Bridging the Gap**: A ResNet-110 teacher may not distill well to a ResNet-8 student directly. A ResNet-32 TA bridges the gap. - **Multi-Step**: Multiple TAs can be chained for very large capacity gaps. - **Practical**: Important when the deployment target has extremely limited resources. **Teaching Assistant** is **the bridge between master and novice** — an intermediate model that translates expert knowledge into a form that a small student can actually absorb.