zero-shot distillation

**Zero-Shot Distillation** is a **variant of data-free distillation where the student is trained without any real data or data generation process** — relying entirely on the teacher's learned parameters and the structure of the output space to transfer knowledge. **How Does Zero-Shot Distillation Work?** - **Crafted Inputs**: Generate pseudo-data by optimizing random noise to maximize specific class activations in the teacher. - **Model Inversion**: Use gradient-based optimization to "invert" the teacher — finding inputs that produce representative outputs. - **Dirichlet Sampling**: Sample from the simplex of class probabilities to create diverse soft label targets. - **Difference from Data-Free**: Zero-shot is even more restrictive — no generator network training, just direct optimization. **Why It Matters** - **Extreme Constraint**: When not even a generator can be trained (no compute budget for data generation). - **Model IP**: Enables knowledge transfer from a black-box teacher API with minimal queries. - **Research**: Explores the fundamental limits of how much knowledge can be extracted from a model without data. **Zero-Shot Distillation** is **knowledge transfer at the extreme** — distilling a model's knowledge with literally zero training examples from any source.