born-again networks

**Born-Again Networks (BAN)** is a **self-distillation technique where a model is re-trained using its own soft predictions as targets** — the student has the identical architecture as the teacher, yet consistently outperforms the original teacher model. **How Do Born-Again Networks Work?** - **Step 1**: Train a teacher model normally with hard labels. - **Step 2**: Train a student (same architecture) using the teacher's soft output distribution as the target. - **Step 3**: Optionally repeat — use the student as the new teacher and train another generation. - **Result**: Each generation improves, even with identical architecture. **Why It Matters** - **Free Improvement**: Same model, same data, better accuracy. The soft labels provide a richer training signal. - **Dark Knowledge**: The teacher's soft outputs encode class-similarity information not present in hard labels. - **Sequence**: Multiple generations of born-again training yield diminishing but consistent improvements. **Born-Again Networks** are **reincarnation for neural nets** — proving that being trained on your own refined knowledge makes you smarter than your previous self.