random matrix theory

**Random Matrix Theory (RMT)** applied to deep learning is the **mathematical study of the eigenvalue distributions of weight matrices and Hessian matrices** — providing insights into network training dynamics, generalization, and the structure of the loss landscape. **What Does RMT Tell Us About DNNs?** - **Weight Matrices**: Well-trained networks develop heavy-tailed eigenvalue distributions (not the Marchenko-Pastur distribution of random matrices). - **Hessian Spectrum**: The eigenvalue distribution of the Hessian reveals the curvature of the loss landscape — many near-zero eigenvalues + a few large ones. - **Generalization**: The heavy-tail exponent $alpha$ of weight matrix eigenvalues correlates with generalization quality. **Why It Matters** - **Diagnostics**: Analyzing weight eigenspectra can predict model quality without validation data. - **Double Descent**: RMT provides theoretical explanations for the double descent phenomenon. - **Pruning**: Eigenvalue analysis identifies which weight matrices are over-parameterized (pruneable). **Random Matrix Theory** is **spectral analysis for neural networks** — reading the eigenvalue fingerprints of weight matrices to understand what the network has learned.