flat minima

**Flat Minima** are **regions of the loss landscape where the loss remains low over a wide neighborhood of parameter values** — characterized by small eigenvalues of the Hessian matrix, and empirically associated with better generalization performance. **What Are Flat Minima?** - **Definition**: A minimum where the loss changes slowly when parameters are perturbed -> wide valley. - **Hessian**: Small eigenvalues of $H = abla^2 mathcal{L}$ indicate flatness. - **Measures**: Sharpness (max eigenvalue), trace of Hessian, volume of the low-loss region. - **Connection**: Flat minima have high posterior volume in Bayesian interpretation -> preferred by Occam's razor. **Why It Matters** - **Generalization**: Flat minima generalize better because the solution is robust to parameter perturbations (which approximate the distribution shift between train and test). - **SAM**: Sharpness-Aware Minimization explicitly seeks flat minima by optimizing worst-case loss in a perturbation ball. - **Batch Size**: Large batch SGD tends to find sharp minima; small batch SGD finds flatter ones. **Flat Minima** are **the wide valleys of good generalization** — solutions where the model's performance is robust to small changes in its parameters.