imagenet-21k pre-training

**ImageNet-21k pre-training** is the **supervised large-scale initialization strategy where ViT models learn from over twenty thousand classes before fine-tuning on target datasets** - it provides broad semantic coverage and strong transfer foundations for many downstream vision tasks. **What Is ImageNet-21k Pre-Training?** - **Definition**: Supervised training on the ImageNet-21k taxonomy with millions of labeled images. - **Label Structure**: Fine-grained hierarchy encourages rich semantic discrimination. - **Common Pipeline**: Pretrain on 21k classes, then fine-tune on ImageNet-1k or domain-specific sets. - **Historical Role**: Important milestone in early strong ViT transfer results. **Why ImageNet-21k Matters** - **Transfer Gains**: Provides notable boosts over training from scratch on smaller datasets. - **Label Quality**: Curated labels are cleaner than many web-scale corpora. - **Reproducibility**: Standard benchmark dataset enables fair model comparison. - **Compute Efficiency**: Smaller than web-scale sets while still yielding strong features. - **Practical Accessibility**: Easier to manage than ultra-large private corpora. **Training Considerations** **Class Imbalance Handling**: - Long tail classes need balanced sampling or reweighting. - Prevents dominant class bias. **Resolution and Augmentation**: - Typical pretraining at moderate resolution with strong augmentation. - Fine-tune later at higher resolution. **Fine-Tuning Protocol**: - Lower learning rates and positional embedding interpolation for resolution changes. - Evaluate across multiple downstream tasks. **Comparison Context** - **Versus ImageNet-1k**: Usually stronger transfer and better robustness. - **Versus Web-Scale**: Less noisy but smaller, often lower asymptotic ceiling. - **Versus Self-Supervised**: Supervised labels help class alignment, self-supervised helps domain breadth. ImageNet-21k pre-training is **a high-value supervised initialization path that balances dataset quality, scale, and reproducibility for ViT development** - it remains a strong baseline in many production and research workflows.