asam

**ASAM** (Adaptive Sharpness-Aware Minimization) is an **improvement over SAM that uses adaptive perturbation sizes for each parameter** — normalizing the perturbation by the parameter magnitude, making SAM scale-invariant and more effective across different network architectures. **How Does ASAM Differ from SAM?** - **SAM**: Uses a uniform perturbation ball -> biased toward large-magnitude parameters. - **ASAM**: $hat{epsilon}_i = ho cdot |w_i| cdot g_i / ||w odot g||$ (perturbation proportional to parameter magnitude). - **Scale Invariance**: Adaptive normalization ensures that the sharpness measure is invariant to parameter rescaling. - **Paper**: Kwon et al. (2021). **Why It Matters** - **Better Generalization**: ASAM consistently outperforms SAM by 0.1-0.5% on ImageNet across architectures. - **Robustness**: Less sensitive to the perturbation radius $ ho$ hyperparameter. - **Theory**: Addresses the known theoretical limitation of SAM's non-adaptive perturbation. **ASAM** is **SAM with proportional fairness** — ensuring that each parameter gets a perturbation sized appropriately for its scale, improving both theory and practice.