label smoothing in vit

**Label smoothing in ViT** is the **regularization method that replaces hard one-hot targets with softened distributions to reduce overconfidence and improve calibration** - instead of forcing probability one for a single class, it reserves small mass for other classes and encourages less extreme logits. **What Is Label Smoothing?** - **Definition**: Modify target vector so true class gets 1 - epsilon and remaining classes share epsilon. - **Regularization Mechanism**: Penalizes overly sharp probability outputs. - **Typical Values**: Epsilon around 0.05 to 0.2 depending on dataset and augmentation strength. - **Loss Integration**: Applied directly in cross entropy computation. **Why Label Smoothing Matters** - **Generalization**: Reduces overfitting by discouraging memorization of hard labels. - **Calibration**: Produces more realistic confidence scores at inference time. - **Stability**: Limits extreme logits that can destabilize mixed precision optimization. - **Noise Tolerance**: Slightly reduces impact of mislabeled samples. - **Recipe Synergy**: Works well with mixup, CutMix, and strong augmentation policies. **Smoothing Configurations** **Fixed Epsilon**: - Constant smoothing value throughout training. - Simple and commonly effective. **Scheduled Epsilon**: - Start higher then reduce near end for sharper decision boundaries. - Useful in long training runs. **Class-Aware Smoothing**: - Different epsilon values by class frequency. - Can improve rare class handling. **How It Works** **Step 1**: Build softened label distribution for each sample by allocating most probability to target class and small residual across others. **Step 2**: Compute cross entropy against softened targets, producing gradients that discourage extreme certainty. **Tools & Platforms** - **PyTorch cross entropy**: Supports label smoothing parameter directly. - **timm recipes**: Includes tuned defaults for ViT families. - **Calibration metrics**: ECE and reliability diagrams validate impact. Label smoothing is **a simple but effective calibration tool that helps ViTs generalize better by reducing pathological confidence spikes** - it keeps classifier behavior more realistic under real world variation.