few-shot distillation

**Few-Shot Distillation** is a **knowledge distillation approach that works with only a small number of labeled examples** — combining the teacher's dark knowledge with data augmentation and meta-learning techniques to effectively train a student model from very limited data. **How Does Few-Shot Distillation Work?** - **Setup**: Very few labeled examples (1-10 per class) available for distillation. - **Teacher**: Provides soft labels for the limited data + any augmented versions. - **Augmentation**: Heavy data augmentation (CutMix, MixUp, RandAugment) to amplify the small dataset. - **Meta-Learning**: Some approaches use meta-learning to optimize the distillation procedure itself. **Why It Matters** - **Low-Resource**: Many real-world applications have very limited labeled data for the target domain. - **Domain Shift**: When the teacher was trained on domain A but the student needs to operate on domain B with few examples. - **Rapid Deployment**: Enables quick model deployment in new domains without extensive data collection. **Few-Shot Distillation** is **learning from a teacher with almost no examples** — maximizing knowledge transfer efficiency when data is extremely scarce.