mae pre-training

**MAE pre-training (Masked Autoencoders)** is the **efficient MIM approach that encodes only visible patches and reconstructs masked patches with a lightweight decoder** - by avoiding full-token encoding during pretraining, MAE reduces compute cost while learning high-quality transferable representations. **What Is MAE?** - **Definition**: Masked autoencoding framework with asymmetric encoder-decoder design for vision transformers. - **Asymmetry**: Heavy encoder sees visible tokens only; small decoder reconstructs masked content. - **High Masking**: Typical mask ratio near 75 percent improves efficiency and representation quality. - **Transfer Strategy**: Decoder is discarded after pretraining; encoder is fine-tuned downstream. **Why MAE Matters** - **Efficiency**: Encoding only visible patches lowers pretraining FLOPs significantly. - **Strong Transfer**: MAE encoders perform well on classification, detection, and segmentation. - **Scalable Objective**: Works across model sizes and large unlabeled datasets. - **Optimization Stability**: Reconstruction objective provides dense training signal. - **Practical Adoption**: Widely used baseline for self-supervised ViT pipelines. **MAE Pipeline** **Masking Stage**: - Randomly hide large fraction of patch tokens. - Keep positional metadata for reconstruction alignment. **Encoder Stage**: - Process only visible tokens through ViT encoder. - Produce compact latent representation. **Decoder Stage**: - Insert mask tokens, decode full sequence, and reconstruct masked patch targets. - Compute loss only on masked patches. **Deployment Notes** - **Fine-Tuning**: Use pretrained encoder with task head and smaller learning rate. - **Mask Ratio Tuning**: Too low reduces challenge, too high can reduce stability. - **Normalization Targets**: Pixel normalization improves reconstruction behavior. MAE pre-training is **an efficient and high-impact self-supervised recipe that turns sparse visible context into strong general-purpose vision features** - it remains one of the most reliable starting points for ViT pretraining.