sharpness-aware minimization

**SAM** (Sharpness-Aware Minimization) is an **optimization technique that simultaneously minimizes both the loss value and the loss sharpness** — by seeking parameters that lie in flat regions of the loss landscape where the worst-case loss within a perturbation neighborhood is minimized. **How Does SAM Work?** - **Objective**: $min_ heta max_{||epsilon|| leq ho} mathcal{L}( heta + epsilon)$ (minimize worst-case loss in a ball of radius $ ho$). - **Inner Step**: Compute the adversarial perturbation: $hat{epsilon} = ho cdot abla_ heta mathcal{L} / || abla_ heta mathcal{L}||$. - **Outer Step**: Compute gradient at $ heta + hat{epsilon}$ and use it to update $ heta$. - **Cost**: 2x forward-backward passes per step (double the compute of standard SGD). **Why It Matters** - **Generalization**: Consistently improves test accuracy across architectures and datasets (0.5-2% on ImageNet). - **Simple**: Drop-in replacement for standard optimizers with minimal code changes. - **Theory**: Directly optimizes for flat minima, connecting the flatness-generalization hypothesis to a practical algorithm. **SAM** is **the optimizer that seeks wide valleys** — trading compute for generalization by explicitly avoiding the sharp, brittle minima that lead to overfitting.