teacher-student framework

**Teacher-Student Framework** is the **general paradigm where a pre-trained "teacher" model guides the training of a "student" model** — the teacher provides soft targets, intermediate features, or other supervision signals that help the student learn better than it could from data alone. **What Is the Teacher-Student Framework?** - **Teacher**: Large, accurate, pre-trained model (or an ensemble). Fixed during distillation. - **Student**: Smaller, efficient model to be deployed. Trained to mimic the teacher. - **Supervision**: Teacher's soft outputs (KD), features (FitNets), attention maps, or relational structure. - **Applications**: Model compression, SSL (DINO), semi-supervised learning, domain adaptation. **Why It Matters** - **Universal Pattern**: The teacher-student paradigm appears across model compression, self-supervised learning, and semi-supervised learning. - **Flexibility**: The teacher can be a larger model, an ensemble, or even the same model at a different training stage (self-distillation). - **Deployment**: Enables deploying compact, fast models that retain the accuracy of much larger ones. **Teacher-Student Framework** is **the master-apprentice relationship of deep learning** — the universal pattern of knowledge transfer from a capable model to a practical one.