BEiT (BERT Pre-Training of Image Transformers) is a self-supervised pre-training method for Vision Transformers that adapts BERT's masked language modeling objective to images by masking random image patches and training the model to predict discrete visual tokens generated by a pre-trained discrete VAE (dVAE) tokenizer. This approach pre-trains ViT on unlabeled images by treating image patches as "visual words" in a visual vocabulary.

Why BEiT Matters in AI/ML: BEiT established the masked image modeling (MIM) paradigm for self-supervised visual pre-training, demonstrating that BERT-style masked prediction works for images when combined with discrete visual tokenization, achieving superior transfer performance over contrastive learning methods.

• Discrete visual tokenizer — A pre-trained discrete VAE (dVAE from DALL-E) maps each 16×16 image patch to a discrete token from a vocabulary of 8192 visual words; these discrete tokens serve as prediction targets analogous to word tokens in BERT • Masked patch prediction — During pre-training, ~40% of image patches are randomly masked, and the ViT encoder must predict the discrete visual token IDs of the masked patches from the visible context; the loss is cross-entropy over the 8192-token vocabulary • Two-stage approach — Stage 1: train the dVAE tokenizer on images (DALL-E's tokenizer); Stage 2: pre-train the ViT using the frozen tokenizer's outputs as prediction targets for masked patches; the tokenizer provides the "visual vocabulary" that makes masked prediction meaningful • Blockwise masking — BEiT uses blockwise masking (masking contiguous blocks of patches rather than random individual patches) to create more challenging prediction tasks that require understanding spatial relationships • Transfer learning — After pre-training, the ViT encoder is fine-tuned on downstream tasks (classification, detection, segmentation) with the pre-trained weights providing a strong initialization; BEiT pre-training improves ImageNet accuracy by 1-3% and downstream task performance by 2-5%

BEiT pioneered masked image modeling for Vision Transformers, adapting BERT's masked prediction paradigm to visual data through discrete tokenization, establishing the MIM pre-training approach that outperforms contrastive methods and inspired the subsequent wave of masked autoencoder research including MAE, SimMIM, and iBOT.