token labeling

**Token Labeling** is a **training strategy for Vision Transformers that assigns individual labels to each patch token** — rather than only supervising the CLS token, providing dense supervision that encourages every token to learn meaningful representations. **How Does Token Labeling Work?** - **Teacher Model**: A pre-trained CNN or ViT generates soft predictions for each spatial region. - **Token Labels**: Each patch token receives a soft label from the teacher's corresponding spatial prediction. - **Dual Loss**: $mathcal{L} = mathcal{L}_{CLS} + alpha cdot mathcal{L}_{tokens}$ (supervise both CLS and individual tokens). - **Paper**: Jiang et al. (2021, "All Tokens Matter"). **Why It Matters** - **Dense Supervision**: Every token learns to be discriminative, not just the CLS token. - **Better Features**: Token-level supervision produces better intermediate features for downstream tasks. - **Free Improvement**: ~1% accuracy gain on ImageNet with no architectural changes. **Token Labeling** is **teaching every patch to recognize** — providing individual supervision to each token for richer, more discriminative ViT features.