Gradient Penalty is a regularization technique used primarily in GAN training (WGAN-GP) — penalizing the norm of the discriminator's gradient with respect to its input, enforcing the Lipschitz constraint required by the Wasserstein distance formulation.

How Does Gradient Penalty Work?

• WGAN-GP: $mathcal{L}_{GP} = lambda cdot mathbb{E}_{hat{x}}[(||\nabla_{hat{x}} D(hat{x})||_2 - 1)^2]$
• Interpolation: $hat{x} = alpha x_{real} + (1-alpha) x_{fake}$ with $alpha sim U(0,1)$.
• Target: The gradient norm should be 1 everywhere along interpolation paths.
• Paper: Gulrajani et al., "Improved Training of Wasserstein GANs" (2017).

Why It Matters

• GAN Stability: Replaced weight clipping in WGAN, dramatically improving training stability and sample quality.
• Lipschitz Constraint: Provides a soft, differentiable enforcement of the 1-Lipschitz constraint.
• Widely Adopted: Standard in most modern GAN architectures (StyleGAN, BigGAN, etc.).

Gradient Penalty is the smoothness enforcer for GANs — ensuring the discriminator function changes gradually, preventing the adversarial training from becoming unstable.