dino (self-distillation with no labels)

**DINO** (Self-DIstillation with NO labels) is a **self-supervised learning approach for Vision Transformers** — demonstrating that self-supervised ViT features explicitly contain scene layout and object segmentation information, which usually requires supervised learning. **What Is DINO?** - **Definition**: A self-supervised method using knowledge distillation without labels. - **Architecture**: Student and Teacher networks with the same architecture but different parameters. - **Update Rule**: Teacher is updated as an exponential moving average (EMA) of the student. - **Key Insight**: Self-supervision on ViTs automatically leads to class-specific features. **Why DINO Matters** - **Emergent Segmentation**: Attention maps automatically segment objects without supervision. - **k-NN Performance**: Features work incredibly well with simple k-nearest neighbor classifiers. - **No Labels Needed**: Unlocks learning from massive uncurated image datasets. - **Teacher-Student Stability**: Solves collapse issues common in self-supervised learning without negative pairs. **How It Works** - **Multi-Crop Strategy**: Feeds global and local crops to student, only global to teacher. - **Cross-Entropy Loss**: Minimizes distance between student and teacher probability distributions. - **Centering & Sharpening**: Prevents mode collapse (outputting same class for everything). **DINO** is **a landmark in unsupervised vision** — proving that supervision is not necessary for models to "understand" object boundaries and semantic categories.