lifelong distillation

**Lifelong Distillation** is a **continual learning technique that uses knowledge distillation to prevent catastrophic forgetting** — when learning a new task, the model distills its own knowledge of previous tasks from its earlier version, maintaining old competencies while acquiring new ones. **How Does Lifelong Distillation Work?** - **Process**: Before learning task $t+1$, save a copy of the model trained on tasks $1..t$. - **Loss**: $mathcal{L} = mathcal{L}_{new\_task} + alpha cdot mathcal{L}_{KD}(model_{new}, model_{old})$ - **Effect**: The KD term penalizes the model for changing its outputs on old-task inputs. - **Related**: Learning without Forgetting (LwF) uses this exact approach. **Why It Matters** - **No Data Storage**: Unlike experience replay, lifelong distillation doesn't need to store old training data. - **Privacy**: Old data may be unavailable or confidential — distillation preserves knowledge without data. - **Scalability**: Works for arbitrarily many sequential tasks. **Lifelong Distillation** is **self-teaching through time** — a model using its past self as a teacher to remember what it learned before.