ibot

**iBOT** is a **self-supervised vision transformer pre-training framework** — performing masked image modeling (MIM) with an online tokenizer to learn high-level semantic abstractions without human annotations. **What Is iBOT?** - **Definition**: Image BERT Pre-training with Online Tokenizer. - **Core Mechanism**: Distills knowledge from an online teacher network to a student network. - **Innovation**: Avoids the need for a pre-trained tokenizer (unlike BEiT) by learning one jointly. - **Result**: Learns both local (patch-level) and global (image-level) features simultaneously. **Why iBOT Matters** - **Semantic Richness**: Captures better semantic meaning than pure contrastive methods (like DINO). - **Efficiency**: Eliminates the multi-stage training pipeline required by BEiT. - **Robustness**: Performs exceptionally well on partial or corrupted images. - **Flexibility**: Works on various vision transformer architectures (ViT, Swin). **Key Components** - **Masked Image Modeling (MIM)**: Reconstructs masked patches (like BERT in NLP). - **Self-Distillation**: Teacher network guides the student's learning. - **Online Tokenizer**: Dynamically generates discrete tokens for image patches during training. **iBOT** is **a pivotal advance in self-supervised vision** — bridging the gap between masked modeling and contrastive learning for superior visual understanding.