ibot pre-training
**iBOT pre-training** is the **self-supervised vision transformer method that combines masked patch prediction with online token-level self-distillation** - it aligns global and local representations across views, producing strong semantic features without manual labels.
**What Is iBOT?**
- **Definition**: Image BERT style training that uses teacher-student framework with masked tokens and patch-level targets.
- **Dual Objective**: Global view alignment plus masked patch token prediction.
- **Online Distillation**: Teacher network updates by momentum from student weights.
- **Token Supervision**: Encourages meaningful patch embeddings, not only image-level embeddings.
**Why iBOT Matters**
- **Dense Feature Quality**: Patch-level targets improve segmentation and localization transfer.
- **Label-Free Learning**: Learns high-level semantics from unlabeled data.
- **Strong Benchmarks**: Delivers competitive results on linear probe and fine-tuning tasks.
- **Representation Diversity**: Combines global invariance with local detail modeling.
- **Modern Influence**: Informs many later token-centric self-supervised methods.
**Training Mechanics**
**View Augmentation**:
- Generate multiple crops and perturbations of each image.
- Feed views to student and teacher branches.
**Teacher-Student Targets**:
- Teacher produces soft targets for global and token-level outputs.
- Student matches targets with masked and unmasked inputs.
**Momentum Update**:
- Teacher parameters follow exponential moving average of student.
- Stabilizes targets during training.
**Implementation Notes**
- **Temperature Settings**: Critical for stable soft target distributions.
- **Mask Ratio**: Influences balance between local reconstruction and global alignment.
- **Batch Diversity**: Large and diverse batches improve representation quality.
iBOT pre-training is **a powerful blend of masked modeling and self-distillation that yields highly transferable ViT representations without labels** - it is especially effective when dense token quality is a priority.