sharp minima

**Sharp Minima** are **regions of the loss landscape where the loss increases rapidly when parameters are perturbed** — characterized by large eigenvalues of the Hessian matrix, and empirically associated with poorer generalization to unseen data. **What Are Sharp Minima?** - **Definition**: A minimum where even small perturbations cause significant loss increase -> narrow valley. - **Hessian**: Large eigenvalues indicate high curvature (sharpness). - **Large Batch Training**: Gradient descent with very large batch sizes tends to converge to sharp minima. - **Overfitting**: Sharp minima are often associated with overfitting because they represent "brittle" solutions. **Why It Matters** - **Generalization Gap**: The train-test performance gap is often larger for models converging to sharp minima. - **Batch Size Effect**: This explains why large-batch training often degrades test accuracy. - **Mitigation**: Learning rate warmup, SAM, and noise injection help steer optimization toward flatter minima. **Sharp Minima** are **the narrow canyons of the loss landscape** — precise solutions that work perfectly on training data but crumble under the slightest perturbation.