data-free distillation

**Data-Free Distillation** is a **knowledge distillation technique that works without access to the original training data** — using the teacher model itself to generate synthetic training data, or leveraging statistics stored in the teacher's batch normalization layers to guide data synthesis. **How Does Data-Free Distillation Work?** - **Generator**: Train a generator network to produce images that maximize the teacher's output diversity. - **BN Statistics**: Use the running mean and variance stored in BatchNorm layers as targets for synthetic data statistics. - **Adversarial**: Generate data that is hard for the student but easy for the teacher -> maximally informative. - **No Real Data**: The entire distillation happens with synthetic data only. **Why It Matters** - **Privacy**: Original training data may be confidential, proprietary, or deleted after teacher training. - **Practical**: Many deployed models have no associated training data pipeline available for re-training. - **Regulation**: GDPR and similar regulations may prohibit retaining training data. **Data-Free Distillation** is **extracting knowledge without the textbook** — training a student using only the teacher model itself, when the original training data is unavailable.