beit (bert pre-training of image transformers)
**BEiT (BERT Pre-Training of Image Transformers)** is a self-supervised pre-training method for Vision Transformers that adapts BERT's masked language modeling objective to images by masking random image patches and training the model to predict discrete visual tokens generated by a pre-trained discrete VAE (dVAE) tokenizer. This approach pre-trains ViT on unlabeled images by treating image patches as "visual words" in a visual vocabulary.
**Why BEiT Matters in AI/ML:**
BEiT established the **masked image modeling (MIM) paradigm** for self-supervised visual pre-training, demonstrating that BERT-style masked prediction works for images when combined with discrete visual tokenization, achieving superior transfer performance over contrastive learning methods.
• **Discrete visual tokenizer** — A pre-trained discrete VAE (dVAE from DALL-E) maps each 16×16 image patch to a discrete token from a vocabulary of 8192 visual words; these discrete tokens serve as prediction targets analogous to word tokens in BERT
• **Masked patch prediction** — During pre-training, ~40% of image patches are randomly masked, and the ViT encoder must predict the discrete visual token IDs of the masked patches from the visible context; the loss is cross-entropy over the 8192-token vocabulary
• **Two-stage approach** — Stage 1: train the dVAE tokenizer on images (DALL-E's tokenizer); Stage 2: pre-train the ViT using the frozen tokenizer's outputs as prediction targets for masked patches; the tokenizer provides the "visual vocabulary" that makes masked prediction meaningful
• **Blockwise masking** — BEiT uses blockwise masking (masking contiguous blocks of patches rather than random individual patches) to create more challenging prediction tasks that require understanding spatial relationships
• **Transfer learning** — After pre-training, the ViT encoder is fine-tuned on downstream tasks (classification, detection, segmentation) with the pre-trained weights providing a strong initialization; BEiT pre-training improves ImageNet accuracy by 1-3% and downstream task performance by 2-5%
| Component | BEiT | MAE | BERT (NLP) |
|-----------|------|-----|-----------|
| Masking | ~40% patches | ~75% patches | ~15% tokens |
| Target | Discrete visual tokens | Raw pixel values | Token IDs |
| Tokenizer | Pre-trained dVAE | None needed | WordPiece |
| Encoder | Full ViT (all patches) | ViT (visible only) | Full BERT |
| Decoder | Linear classification head | Lightweight decoder | Linear head |
| Pre-train Data | ImageNet-1K/22K | ImageNet-1K | BookCorpus + Wiki |
| ImageNet Fine-tune | 83.2% (ViT-B) | 83.6% (ViT-B) | N/A |
**BEiT pioneered masked image modeling for Vision Transformers, adapting BERT's masked prediction paradigm to visual data through discrete tokenization, establishing the MIM pre-training approach that outperforms contrastive methods and inspired the subsequent wave of masked autoencoder research including MAE, SimMIM, and iBOT.**