spectral normalization

**Spectral Normalization** is a **weight normalization technique that constrains each weight matrix's spectral norm (largest singular value) to a target value** — controlling the Lipschitz constant of each layer to stabilize training and improve adversarial robustness. **How Spectral Normalization Works** - **Spectral Norm**: $sigma(W) = max_{|v|=1} |Wv|$ — the largest singular value of the weight matrix. - **Normalization**: $hat{W} = W / sigma(W)$ — divide by the spectral norm so each layer has Lipschitz constant ≤ 1. - **Power Iteration**: Estimate $sigma(W)$ efficiently using one step of power iteration per training step. - **Application**: Applied to every weight matrix (linear, conv) in the network. **Why It Matters** - **GAN Stability**: Originally introduced for stabilizing GAN discriminator training (Miyato et al., 2018). - **Robustness**: Constraining spectral norms improves adversarial robustness by limiting sensitivity. - **Lightweight**: Power iteration adds negligible computational cost — one extra matrix-vector product per layer. **Spectral Normalization** is **capping the sensitivity of each layer** — normalizing weight matrices to control how much each layer amplifies perturbations.