SAM (Sharpness-Aware Minimization) is an optimization technique that simultaneously minimizes both the loss value and the loss sharpness — by seeking parameters that lie in flat regions of the loss landscape where the worst-case loss within a perturbation neighborhood is minimized.

How Does SAM Work?

• Objective: $min_ heta max_{||epsilon|| leq ho} mathcal{L}( heta + epsilon)$ (minimize worst-case loss in a ball of radius $ ho$).
• Inner Step: Compute the adversarial perturbation: $hat{epsilon} = ho cdot \nabla_ heta mathcal{L} / ||\nabla_ heta mathcal{L}||$.
• Outer Step: Compute gradient at $ heta + hat{epsilon}$ and use it to update $ heta$.
• Cost: 2x forward-backward passes per step (double the compute of standard SGD).

Why It Matters

• Generalization: Consistently improves test accuracy across architectures and datasets (0.5-2% on ImageNet).
• Simple: Drop-in replacement for standard optimizers with minimal code changes.
• Theory: Directly optimizes for flat minima, connecting the flatness-generalization hypothesis to a practical algorithm.

SAM is the optimizer that seeks wide valleys — trading compute for generalization by explicitly avoiding the sharp, brittle minima that lead to overfitting.