loss landscape smoothness

**Loss Landscape Smoothness** refers to the **geometric properties of the loss function surface in parameter space** — smooth landscapes (low curvature, wide minima) correlate with better generalization, while rough landscapes (sharp minima, high curvature) correlate with poor generalization. **Smoothness Metrics** - **Hessian Eigenvalues**: The eigenvalues of the loss Hessian measure local curvature — smaller eigenvalues = smoother. - **Sharpness**: The maximum loss change within a neighborhood of the minimum — sharp minima generalize poorly. - **Filter Normalization**: Visualize the loss landscape by plotting loss along random directions, normalized by filter norms. - **PAC-Bayes**: Sharpness-aware generalization bounds relate the width of minima to generalization error. **Why It Matters** - **Generalization**: Models converging to flat minima generalize better — SAM (Sharpness-Aware Minimization) explicitly seeks flat minima. - **Batch Size**: Large batch sizes tend to find sharp minima — small batches explore more and find flatter minima. - **Architecture**: Skip connections (ResNets) create smoother loss landscapes — one reason they train more easily. **Loss Landscape Smoothness** is **the geometry of good solutions** — flatter, smoother loss landscapes produce models that generalize better.