loss landscape analysis

**Loss Landscape Analysis** is the **study of the geometry of a neural network's loss function in parameter space** — visualizing and characterizing the shape of the high-dimensional loss surface to understand optimization, generalization, and the relationship between flat/sharp minima. **What Is Loss Landscape Analysis?** - **Visualization**: Project the high-dimensional loss surface onto 1D or 2D slices for visualization. - **Methods**: Random direction projection, filter-normalized plots (Li et al., 2018), PCA of training trajectories. - **Features**: Minima, saddle points, barriers between minima, flatness/sharpness. **Why It Matters** - **Flat vs. Sharp Minima**: Flat minima (wide valleys) often correlate with better generalization. - **Optimization**: The landscape shape determines whether optimizers converge successfully. - **Architecture Dependence**: Skip connections (ResNet) create smoother landscapes than plain networks. **Loss Landscape Analysis** is **cartography for optimization** — mapping the terrain that gradient descent must navigate to find good solutions.