gradient penalty

**Gradient Penalty** is a **regularization technique used primarily in GAN training (WGAN-GP)** — penalizing the norm of the discriminator's gradient with respect to its input, enforcing the Lipschitz constraint required by the Wasserstein distance formulation. **How Does Gradient Penalty Work?** - **WGAN-GP**: $mathcal{L}_{GP} = lambda cdot mathbb{E}_{hat{x}}[(|| abla_{hat{x}} D(hat{x})||_2 - 1)^2]$ - **Interpolation**: $hat{x} = alpha x_{real} + (1-alpha) x_{fake}$ with $alpha sim U(0,1)$. - **Target**: The gradient norm should be 1 everywhere along interpolation paths. - **Paper**: Gulrajani et al., "Improved Training of Wasserstein GANs" (2017). **Why It Matters** - **GAN Stability**: Replaced weight clipping in WGAN, dramatically improving training stability and sample quality. - **Lipschitz Constraint**: Provides a soft, differentiable enforcement of the 1-Lipschitz constraint. - **Widely Adopted**: Standard in most modern GAN architectures (StyleGAN, BigGAN, etc.). **Gradient Penalty** is **the smoothness enforcer for GANs** — ensuring the discriminator function changes gradually, preventing the adversarial training from becoming unstable.