progressive distillation
**Progressive Distillation** is a knowledge distillation technique specifically designed for accelerating diffusion model sampling by iteratively training student models that perform the same denoising in half the steps of their teacher. Each distillation round halves the required sampling steps, and after K rounds, the original N-step process is compressed to N/2^K steps, enabling efficient few-step generation while preserving sample quality.
**Why Progressive Distillation Matters in AI/ML:**
Progressive distillation provides a **systematic, principled approach to accelerating diffusion models** by 100-1000×, compressing thousands of sampling steps into 4-8 steps with minimal quality degradation through iterative halving of the denoising schedule.
• **Step halving** — Each distillation round trains a student to match the teacher's two-step output in a single step: student(x_t, t→t-2Δ) ≈ teacher(teacher(x_t, t→t-Δ), t-Δ→t-2Δ); the student learns to "skip" every other step while producing equivalent results
• **Iterative compression** — Starting from a 1024-step teacher: Round 1 produces a 512-step student, Round 2 produces a 256-step student, ..., Round 8 produces a 4-step student; each round uses the previous student as the new teacher
• **v-prediction parameterization** — Progressive distillation works best with v-prediction (v = α_t·ε - σ_t·x) rather than ε-prediction, as v-prediction provides more stable training targets during distillation, especially for large step sizes
• **Quality preservation** — Each halving step introduces minimal quality loss (~0.5-1.0 FID increase per round); after 8 rounds (1024→4 steps), total quality degradation is typically 3-8 FID points, a favorable tradeoff for 256× speed improvement
• **Classifier-free guidance distillation** — Extended to distill classifier-free guided models by incorporating the guidance computation into the student, further reducing inference cost by eliminating the need for dual (conditional + unconditional) forward passes
| Distillation Round | Steps | Speedup | Typical FID Impact |
|-------------------|-------|---------|-------------------|
| Teacher (base) | 1024 | 1× | Baseline |
| Round 1 | 512 | 2× | +0.1-0.3 |
| Round 2 | 256 | 4× | +0.2-0.5 |
| Round 4 | 64 | 16× | +0.5-1.5 |
| Round 6 | 16 | 64× | +1.5-3.0 |
| Round 8 | 4 | 256× | +3.0-8.0 |
**Progressive distillation is the most systematic technique for accelerating diffusion model inference, iteratively halving the sampling steps through teacher-student knowledge transfer until few-step generation is achieved with controlled quality tradeoffs, enabling practical deployment of diffusion models in latency-sensitive applications.**