token labeling in vit
**Token Labeling** is the **dense supervision technique that attaches labels to every patch token so Vision Transformers learn fine-grained correspondences rather than just relying on the class token** — it trains the student network to mimic the per-patch predictions of a stronger teacher, boosting accuracy on ImageNet and segmentation tasks.
**What Is Token Labeling?**
- **Definition**: A training strategy where each patch token receives a soft label generated by a high-capacity teacher, and the loss aggregates over all patches rather than a single CLS token.
- **Key Feature 1**: Soft labels come from a teacher network (e.g., EfficientNet) that provides probability distributions for each spatial region.
- **Key Feature 2**: The student ViT uses a small head per token that predicts the teacher label, enforcing fine spatial alignment.
- **Key Feature 3**: Aggregated loss blends token-level supervision with the original classification loss to preserve global semantics.
- **Key Feature 4**: Token labeling pairs naturally with distillation and strong augmentations like mixup.
**Why Token Labeling Matters**
- **Higher Accuracy**: Provides 1-2 point gains on ImageNet by teaching the model to pay attention to every region.
- **Spatial Awareness**: Encourages tokens to represent actionable features like edges, textures, or object parts.
- **Teacher Guidance**: The student inherits the localization knowledge of the teacher without needing bounding boxes.
- **Multi-Task Ready**: Token-level outputs can double as segmentation or localization maps for downstream heads.
- **Compatibility**: Works without architectural changes, simply by adding a per-token projection and loss term.
**Labeling Patterns**
**Dense Soft Labels**:
- Teacher outputs a distribution over classes for every token.
- Student learns to match these distributions, capturing uncertainty.
**Binary Maps**:
- For some tasks, teachers provide foreground/background probabilities per patch.
- Student learns to attend to regions relevant to the high-level class.
**Hybrid Loss**:
- Combine token loss with CLS loss in a weighted sum, ensuring the model still respects global prediction quality.
**How It Works / Technical Details**
**Step 1**: During each forward pass, copy the teacher predictions for the corresponding image and pass them through a softmax to produce token-level targets.
**Step 2**: Student token representations pass through a shared projection head to predict the same distribution. The token loss (e.g., KL divergence) is averaged across tokens and added to the standard cross-entropy on the class token.
**Comparison / Alternatives**
| Aspect | Token Labeling | Standard ViT | Segmentation Distillation |
|--------|----------------|--------------|---------------------------|
| Supervision | Dense | Global | Dense but task-specific |
| Teacher Usage | Token-wise | Optional | Pixel-wise |
| Downstream Map | Ready | Requires probing | Task-specific |
| Complexity | Slight extra head | Baseline | Similar or higher |
**Tools & Platforms**
- **timm**: Supports token labeling by enabling `token_label` configs and teacher checkpoints.
- **Hugging Face**: Allows writing custom loss that combines token and CLS components.
- **Weights & Biases**: Visualizes token attention and label fidelity to ensure the student matches the teacher.
- **Distillation Libraries**: (e.g., TinyML Distiller) provide utilities to store teacher logits per patch.
Token labeling is **the dense teacher supervision that turns every patch into a lesson, raising ViT fidelity without touching the architecture** — it ensures every token has a role beyond a silent placeholder.