beit pre-training
**BEiT pre-training** is the **masked image modeling framework that predicts discrete visual tokens from masked patches, analogous to masked language modeling in NLP** - by reconstructing semantic token targets instead of raw pixels, BEiT encourages higher-level representation learning.
**What Is BEiT?**
- **Definition**: Bidirectional Encoder representation from Image Transformers using masked token prediction.
- **Target Source**: Discrete tokens generated by an external image tokenizer.
- **Objective**: Predict masked token IDs from visible context.
- **Architecture**: ViT encoder with prediction head over visual vocabulary.
**Why BEiT Matters**
- **Semantic Focus**: Token targets can emphasize object-level structure beyond low-level pixels.
- **NLP Analogy**: Brings proven masked-token paradigm into vision domain.
- **Transfer Quality**: Produces strong initialization for classification and dense tasks.
- **Research Influence**: Inspired many tokenized and hybrid MIM methods.
- **Flexible Extension**: Works with richer tokenizers and multi-task pretraining.
**BEiT Pipeline**
**Tokenizer Stage**:
- Pretrain or load visual tokenizer that maps image patches to discrete IDs.
- Build vocabulary for masked prediction.
**Masked Encoding Stage**:
- Mask patches in input and process visible tokens through ViT encoder.
- Predict token IDs for masked locations.
**Optimization Stage**:
- Minimize cross-entropy over masked token positions.
- Fine-tune encoder for downstream supervised tasks.
**Practical Considerations**
- **Tokenizer Quality**: Strong tokenizer improves target signal quality.
- **Vocabulary Size**: Too small loses detail, too large can hurt stability.
- **Compute Cost**: Extra tokenizer pipeline increases pretraining complexity.
BEiT pre-training is **a semantic masked-token approach that pushes ViT encoders toward richer abstraction during self-supervised learning** - it remains a key method in the evolution of modern vision pretraining.