masked image modeling
**Masked image modeling (MIM)** is the **self-supervised training paradigm where a model reconstructs hidden image patches from visible context** - this forces ViT encoders to learn semantic and structural representations instead of memorizing local texture shortcuts.
**What Is Masked Image Modeling?**
- **Definition**: Randomly mask a subset of patches and train model to predict pixel or token targets for masked regions.
- **Mask Ratio**: Often high, such as 40 to 75 percent, to create meaningful reconstruction challenge.
- **Target Choices**: Raw pixels, quantized tokens, or latent features.
- **Backbone Fit**: ViT token structure makes masking straightforward and efficient.
**Why MIM Matters**
- **Unlabeled Learning**: Extracts supervision from raw image structure.
- **Context Reasoning**: Encourages understanding of global layout and object relationships.
- **Transfer Performance**: Pretrained encoders perform strongly on many downstream tasks.
- **Data Scalability**: Benefits from large unlabeled corpora.
- **Architectural Flexibility**: Supports lightweight or heavy decoders depending on objective.
**MIM Variants**
**Pixel Reconstruction**:
- Predict normalized pixel values for masked patches.
- Simple but can emphasize low-level detail.
**Token Reconstruction**:
- Predict discrete visual tokens from tokenizer.
- Often yields stronger semantic abstraction.
**Feature Reconstruction**:
- Match teacher or latent feature targets.
- Balances detail and semantic fidelity.
**Training Flow**
**Step 1**:
- Sample mask pattern, remove masked patches from encoder input, and process visible tokens.
**Step 2**:
- Decoder predicts masked targets and optimization minimizes reconstruction loss over masked positions.
Masked image modeling is **a versatile and scalable self-supervised framework that teaches ViTs to infer missing visual context from surrounding evidence** - it is now a core building block for modern vision pretraining.