Generative Adversarial Networks Adversarial Generative Modeling
# Generative Adversarial Networks & Adversarial Generative Modeling
## Introduction & Motivation
Generative Adversarial Networks, introduced by Goodfellow and colleagues in 2014, reframed generative modeling as a two-player game rather than a direct likelihood-maximization problem. Instead of writing down an explicit probability density and fitting it to data by maximum likelihood, a GAN pits two neural networks against each other: a generator that maps random noise to synthetic samples, and a discriminator that tries to distinguish those synthetic samples from real training data. The generator is trained to fool the discriminator, and the discriminator is trained to catch the generator, and under idealized conditions this adversarial pressure drives the generator's output distribution toward the true data distribution without ever needing to evaluate a tractable likelihood.
This likelihood-free property was, at the time, a major departure from prior generative approaches such as variational autoencoders and autoregressive models, both of which require an explicit or approximate likelihood term in the training objective. GANs instead only require that one can sample from the generator and pass the result through a discriminator, which turns out to be enough to produce remarkably sharp, high-fidelity samples, particularly in image generation, where early GANs produced noticeably crisper outputs than contemporaneous VAEs.
The practical importance of GANs extends well past their original image-synthesis demonstrations. Architectures descended from the original formulation underpin super-resolution systems, image-to-image translation (style transfer, colorization, domain adaptation), data augmentation pipelines for scarce-data domains such as medical imaging, deepfake and voice-synthesis technology (with attendant ethical and security considerations), and synthetic training data generation for downstream discriminative models. Even as diffusion models have surpassed GANs on many image-fidelity benchmarks in recent years, GANs remain relevant for applications demanding fast, single-pass sample generation (a diffusion model typically requires dozens to thousands of denoising steps, while a trained GAN generator needs only one forward pass), and the adversarial training paradigm itself has been repurposed well beyond generative modeling, informing domain adaptation, adversarial robustness research, and certain self-supervised representation-learning schemes.
Studying GANs is also valuable because they are a canonical example of a min-max game formulated as a neural network training objective, and the instabilities that arise during GAN training (mode collapse, oscillation, vanishing gradients for the generator) are illustrative of broader difficulties that appear whenever two learned components are trained against each other, a pattern that recurs in actor-critic reinforcement learning, self-play multi-agent systems, and certain robust-optimization formulations.
## Core Concepts & Theory
The generator network, typically denoted G, takes as input a vector z sampled from a simple prior distribution (commonly a standard multivariate Gaussian or uniform distribution) defined over a lower-dimensional latent space, and maps it through a series of upsampling and convolutional (or transposed-convolutional) layers to produce a synthetic sample G(z) living in the same space as the real data, such as an image. The discriminator network, denoted D, takes either a real training sample or a generated sample and outputs a scalar in the range zero to one, interpreted as the discriminator's estimated probability that the input is real rather than generated.
Training alternates, or in some formulations proceeds simultaneously, between two update steps. The discriminator is updated to increase D of real data points toward one and decrease D of generated points toward zero, effectively training it as a binary classifier distinguishing real from fake. The generator is updated to increase D of its own generated points, i.e., to make the discriminator more likely to be fooled. This is the min-max game: the discriminator maximizes its ability to tell real from fake, while the generator minimizes the discriminator's success, and at the (idealized) Nash equilibrium of this game, the generator distribution exactly matches the true data distribution, and the discriminator can do no better than random guessing (outputting one half everywhere).
A key theoretical result from the original GAN paper is that, for a fixed generator, the optimal discriminator has a closed form in terms of the true data density and the generator's implied density, and substituting this optimal discriminator back into the generator's objective reveals that the generator is implicitly minimizing the Jensen-Shannon divergence between the real data distribution and the generated distribution. This connects GAN training, despite its game-theoretic framing, to a familiar statistical distance between distributions, though this connection also explains some of the practical training difficulties: the Jensen-Shannon divergence behaves poorly (its gradient vanishes or becomes uninformative) when the real and generated distributions have little or no overlapping support, which is common early in training when the generator produces poor samples easily distinguished by the discriminator.
Mode collapse is the most notorious pathology in GAN training: rather than covering the full diversity of the true data distribution, the generator collapses onto producing only a small number of distinct outputs (or even a single output) that reliably fool the current discriminator. This occurs because the generator's objective only requires fooling the discriminator, not covering the data distribution, so a generator can achieve a low loss by finding a small number of highly convincing modes rather than modeling the full distribution's diversity. Mode collapse is closely tied to the non-convex, non-cooperative nature of the underlying game, in which the generator and discriminator can enter oscillatory dynamics rather than converging to the desired equilibrium.
## Mathematical Formulation
The original (non-saturating in its standard practical form, but here shown in its saturating minimax form) GAN objective is a two-player value function optimized jointly by the discriminator (maximizing) and generator (minimizing):
$$ \min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{data}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] $$
where the first expectation encourages the discriminator to assign high probability to real samples drawn from the true data distribution, and the second expectation encourages the discriminator to assign low probability to generated samples, while the generator seeks to make this second term large (i.e., make D of G of z close to one) so as to minimize the overall objective from its own perspective.
In practice, this saturating formulation produces weak gradients for the generator early in training, when the discriminator easily identifies generated samples as fake (D of G of z near zero), because the gradient of the log of one minus D of G of z vanishes as D of G of z approaches zero. The standard fix, proposed in the original paper itself, is the non-saturating generator loss, in which the generator instead directly maximizes the log-probability the discriminator assigns to its samples being real:
$$ \max_G \; \mathbb{E}_{z \sim p_z(z)}[\log D(G(z))] $$
This reformulated objective has the same fixed point as the original minimax game (the generator still wants to fool the discriminator) but produces much stronger gradients when the generator is performing poorly, and it is the loss function used in essentially all practical GAN implementations rather than the pure minimax form.
The Wasserstein GAN (WGAN) reformulation replaces the Jensen-Shannon-divergence-based objective with the Earth Mover's (Wasserstein-1) distance between the real and generated distributions, motivated by the observation that the Wasserstein distance remains meaningful and provides useful gradients even when two distributions have disjoint support, unlike Jensen-Shannon divergence. Via the Kantorovich-Rubinstein duality, the Wasserstein distance can be estimated as a supremum over one-Lipschitz functions:
$$ W(p_{data}, p_g) = \sup_{\|f\|_L \leq 1} \; \mathbb{E}_{x \sim p_{data}}[f(x)] - \mathbb{E}_{x \sim p_g}[f(x)] $$
where the discriminator (renamed the "critic" in WGAN terminology, since it no longer outputs a probability) is trained to approximate this one-Lipschitz function f, subject to a Lipschitz constraint enforced either by weight clipping (the original WGAN approach) or, more effectively, by a gradient penalty term (WGAN-GP) that penalizes deviations of the critic's gradient norm from one along interpolated points between real and generated samples.
## Advanced Theory & Extensions
Conditional GANs extend the basic framework by supplying both the generator and discriminator with additional conditioning information, such as a class label, a text description, or a source image, allowing controllable generation rather than unconditional sampling from the learned data distribution. The generator becomes a function of both the noise vector and the condition, and the discriminator is trained to judge not just realism but also consistency with the supplied condition, penalizing generated samples that look realistic but do not match the conditioning input. This conditioning mechanism underlies practical systems such as image-to-image translation frameworks (pix2pix, which uses paired training data, and CycleGAN, which relaxes this to unpaired data via a cycle-consistency loss that requires translating an image to a target domain and back to recover the original), text-to-image synthesis GANs, and super-resolution GANs that condition on a low-resolution input.
StyleGAN and its successors introduced an alternative generator architecture in which the latent code is first mapped through a non-linear mapping network into an intermediate latent space, and this intermediate representation is then injected into each layer of a synthesis network via adaptive instance normalization, controlling the "style" of the generated image at different spatial resolutions (coarse styles like pose and face shape at low resolution, fine styles like color and micro-texture at high resolution). This architecture dramatically improved both sample quality and the semantic disentanglement of the latent space, meaning that meaningful, interpretable directions in latent space (such as a direction corresponding to age or the presence of eyeglasses) become easier to discover and manipulate.
Self-attention GANs (SAGAN) and later BigGAN incorporated self-attention layers into the generator and discriminator to better model long-range spatial dependencies that convolutions alone struggle to capture (since a single convolutional kernel has a limited receptive field), and BigGAN in particular demonstrated that scaling up model capacity and batch size substantially, combined with techniques such as truncating the latent noise distribution at sampling time to trade off diversity for fidelity, could push GAN image quality to levels competitive with early diffusion models on large, diverse datasets such as ImageNet.
Adversarial training as a general technique extends beyond pure generative modeling. Domain-adversarial neural networks use a discriminator to encourage a feature extractor to produce representations that are invariant to a nuisance variable such as the source domain in domain adaptation, by training the feature extractor to fool a domain-classifying discriminator while a task-specific head remains accurate. Similarly, certain self-supervised and representation-learning schemes borrow the adversarial framing to encourage learned representations to satisfy statistical independence or invariance properties that are difficult to enforce through a direct loss term alone.
## Computational Considerations
GAN training is notoriously sensitive to hyperparameters and prone to instability, more so than typical supervised deep learning, because it involves a coupled, non-cooperative optimization between two networks rather than a single well-behaved loss surface being minimized. Common failure modes include mode collapse (discussed above), oscillatory non-convergence (the generator and discriminator cycle through states without settling near an equilibrium), and vanishing gradients when the discriminator becomes too strong relative to the generator, leaving the generator with no useful signal to improve.
Practical stabilization techniques include using the non-saturating generator loss rather than the pure minimax form, applying spectral normalization to the discriminator's weight matrices to bound its Lipschitz constant and prevent it from becoming arbitrarily confident, using a learning rate and update schedule that keeps the generator and discriminator roughly balanced (sometimes via a "two time-scale update rule," using different learning rates for each network), applying label smoothing on the discriminator's real-sample targets, and adding small amounts of instance noise to both real and generated samples early in training to increase the overlap between their supports and thereby provide more informative gradients.
Compute costs for GAN training are driven primarily by the cost of forward and backward passes through both networks at every training step, and by the number of discriminator updates performed per generator update, a ratio that is itself a tunable hyperparameter (WGAN-style training commonly performs multiple discriminator updates per generator update to keep the critic well-trained relative to the generator). High-resolution GANs such as StyleGAN also require progressive growing or carefully designed multi-scale architectures to remain trainable at high resolution, since directly training a generator and discriminator at, for instance, 1024 by 1024 resolution from the start is highly unstable.
## Practical Implementation Strategies
A robust baseline GAN implementation should start with a well-tested architecture family (DCGAN-style convolutional architectures for images, or a WGAN-GP formulation for improved training stability) rather than attempting to design a novel architecture from scratch, since small architectural choices (the placement of batch normalization, the choice of upsampling method, the use of spectral normalization) have an outsized effect on training stability relative to typical supervised networks.
Monitoring GAN training is qualitatively different from monitoring supervised training, because the generator and discriminator losses do not necessarily decrease monotonically, and a "good" loss trajectory for one network often looks like an oscillating or slowly changing trajectory rather than a smooth descent. Practitioners therefore rely heavily on qualitative sample inspection (viewing generated samples at regular intervals) and quantitative sample-quality metrics such as the Inception Score and the Frechet Inception Distance (FID), which compare statistics of features extracted from real and generated images using a pretrained classifier network, with lower FID indicating generated samples whose feature distribution more closely matches that of real data.
Data preprocessing matters more for GANs than for many supervised tasks: normalizing pixel values to match the range and distribution expected by the generator's final activation function (commonly tanh, implying pixel normalization to the range negative one to one), ensuring balanced representation of classes or modes in conditional settings, and using appropriate data augmentation (differentiable augmentation applied consistently to both real and generated samples, a technique shown to substantially improve GAN performance in limited-data regimes) all materially affect training outcomes.
## Benchmark Datasets & Evaluation
CIFAR-10 and downsampled ImageNet variants remain common benchmarks for measuring unconditional and class-conditional GAN sample quality via FID and Inception Score, providing a standardized, relatively fast-to-train setting for comparing architectural and training innovations. CelebA and its high-resolution variant CelebA-HQ are standard for face generation research, given the widespread interest in controllable face synthesis and editing. LSUN provides large collections of scene images (bedrooms, churches, and so on) commonly used to evaluate GANs at higher resolutions than CIFAR-10 permits. Full-resolution ImageNet, at 128 by 128 resolution and above, is the standard large-scale, high-diversity benchmark used by BigGAN and subsequent large-scale generative models to demonstrate both fidelity and diversity across a thousand object categories.
Evaluation of generative models is inherently more difficult than evaluation of discriminative models, since there is no single ground-truth label to compare against. The Frechet Inception Distance compares the mean and covariance of Inception-network features extracted from real and generated image sets, assuming these features are approximately Gaussian-distributed, and has become the de facto standard metric despite known limitations (sensitivity to sample size, reliance on a specific pretrained feature extractor, and known cases where FID does not correlate perfectly with human perceptual judgments of quality). Precision and recall metrics for generative models attempt to separately quantify sample fidelity (precision, whether generated samples fall within the support of the real data distribution) and sample diversity or mode coverage (recall, whether the generator covers the full support of the real data distribution), addressing a blind spot in aggregate metrics like FID that can be fooled by generators exhibiting partial mode collapse alongside high-fidelity samples.
## Key Challenges & Limitations
Mode collapse and training instability remain the central practical challenges of GAN-based generative modeling, and despite substantial research into stabilization techniques (gradient penalties, spectral normalization, alternative divergences), no single fix reliably eliminates these issues across all architectures and datasets, meaning GAN training still typically requires more hyperparameter tuning and manual monitoring than training a comparably sized diffusion model or autoregressive model.
Evaluation remains an open challenge: no single automatic metric perfectly captures the multiple desiderata of realism, diversity, and semantic correctness, and human evaluation, while more reliable for perceptual quality judgments, is expensive, slow, and itself subject to inconsistency across raters and evaluation protocols.
GANs lack an explicit, tractable likelihood, which is a strength for sample quality but a weakness for certain use cases: unlike autoregressive models or normalizing flows, a trained GAN generator cannot directly provide a likelihood estimate for a given data point, complicating tasks such as anomaly detection via likelihood thresholding or principled model comparison via held-out log-likelihood, a common practice in other generative modeling families.
The misuse potential of high-fidelity conditional GANs, particularly for face and voice synthesis, has become a significant societal concern, driving research into GAN-generated content detection, provenance watermarking of generated media, and policy discussions around the responsible deployment of synthesis technology, considerations that are now a standard part of responsible-deployment discussions for any high-fidelity conditional generative system, GAN-based or otherwise.
## Hyperparameter Tuning
The ratio of discriminator to generator updates per training step is a critical and somewhat model-specific hyperparameter, with WGAN-style formulations commonly benefiting from performing several discriminator (critic) updates for every generator update, to keep the critic close to its optimum relative to the current generator, while other formulations perform a strict one-to-one alternation.
Learning rates for the generator and discriminator are frequently set independently rather than shared, and the Adam optimizer with reduced first-moment decay (a lower beta-one value than the common default of 0.9, often set to around 0.5 or even zero) is a widely used default for GAN training, motivated empirically by improved stability relative to the standard Adam defaults tuned for supervised learning.
The strength of the gradient penalty coefficient in WGAN-GP, the choice of spectral normalization versus gradient penalty (or their combination) for enforcing the Lipschitz constraint on the discriminator, the latent noise dimensionality, and the batch size (larger batch sizes have been empirically linked to improved GAN sample quality, as demonstrated prominently by BigGAN) are all hyperparameters that require dataset- and architecture-specific tuning, typically guided by tracking FID on a held-out validation set at regular checkpoints throughout training rather than by the raw generator or discriminator loss values, which, as discussed, are not reliable indicators of sample quality during GAN training.
## Real-World Applications & Case Studies
Image super-resolution systems such as SRGAN and ESRGAN use adversarial training combined with perceptual (feature-space) losses to upsample low-resolution images while producing sharp, plausible high-frequency detail that pure pixel-wise regression losses (which tend to produce blurry outputs due to averaging over plausible high-frequency completions) cannot achieve.
Data augmentation via GAN-generated synthetic samples has proven valuable in domains where labeled data is scarce or expensive to collect, such as medical imaging, where GANs have been used to generate synthetic pathological examples to augment training sets for downstream diagnostic classifiers, though care is required to validate that synthetic samples do not introduce artifacts that mislead the downstream model.
Image-to-image translation systems, including pix2pix for paired translation tasks (such as converting building facade labels into photorealistic facade images) and CycleGAN for unpaired translation (such as converting photographs into the visual style of a particular painter, or converting images between summer and winter appearances), have found application in creative tools, style transfer products, and simulation-to-real domain adaptation pipelines used to bridge the gap between synthetic training environments and real-world deployment for robotics and autonomous systems.
## Integration with Other Methods
GANs and variational autoencoders have been combined in hybrid architectures (VAE-GAN) that use an adversarial loss in place of or alongside a VAE's typical pixel-wise reconstruction loss, aiming to combine the VAE's stable, encoder-based training and structured latent space with the GAN's sharper sample quality.
GAN-based adversarial losses are frequently used as an auxiliary loss term alongside other objectives rather than as the sole training signal, for instance combining an adversarial loss with a pixel-wise or perceptual reconstruction loss in image translation and super-resolution systems, since a pure adversarial loss alone provides weak supervision about the specific target output and benefits from being anchored by a more direct reconstruction signal.
More recently, GAN-style discriminators have been explored as a training signal within diffusion model distillation pipelines, where a discriminator helps a fast, few-step "student" generator match the sample distribution of a slow, many-step diffusion "teacher," illustrating how adversarial training continues to play a role even in architectures whose primary generative mechanism (iterative denoising) is not itself adversarial.
## Future Research Directions
Improving the theoretical understanding of GAN training dynamics, particularly convergence guarantees for the non-convex, non-cooperative game played by the generator and discriminator, remains an active area, since existing convergence results typically rely on simplifying assumptions (such as the discriminator being optimal at every step) that do not hold in practice.
Combining the fast, single-pass sampling of GANs with the training stability and strong likelihood-based theoretical grounding of diffusion models is a recurring research theme, including work on adversarial diffusion distillation that trains few-step or single-step generators using a combination of distillation and adversarial objectives to approach diffusion-level quality without the associated sampling cost.
Research into more reliable, standardized generative model evaluation, including sample-based metrics that better correlate with human perceptual judgment and metrics that separately and robustly quantify fidelity and diversity, continues to be a priority as generative models are deployed in increasingly consequential real-world applications where sample quality failures carry real costs.
## Summary & Key Takeaways
Generative Adversarial Networks formulate generative modeling as a min-max game between a generator that produces synthetic samples and a discriminator that distinguishes real from generated data, with the generator implicitly minimizing a statistical divergence (Jensen-Shannon in the original formulation, Wasserstein distance in WGAN) between the true and generated data distributions as training converges toward an ideal equilibrium.
Training instability, particularly mode collapse and oscillatory non-convergence, is the central practical challenge of GAN-based modeling, addressed through techniques such as the non-saturating generator loss, spectral normalization, gradient penalties, and careful balancing of generator and discriminator update schedules.
Architectural innovations including conditional GANs, StyleGAN's style-based generator, and self-attention mechanisms have substantially improved sample quality, controllability, and the semantic structure of the learned latent space, while adversarial training as a general technique has found application well beyond pure generative modeling, in domain adaptation, representation learning, and diffusion model distillation.
Despite being partly superseded by diffusion models on many image-fidelity benchmarks, GANs remain practically relevant wherever fast, single-pass generation is required, and the adversarial training paradigm they popularized continues to influence generative modeling research broadly.
Keywords: generative adversarial network, GAN, generator discriminator, adversarial training, mode collapse, Jensen-Shannon divergence, Wasserstein GAN, WGAN-GP, gradient penalty, spectral normalization, conditional GAN, StyleGAN, CycleGAN, pix2pix, BigGAN, Frechet Inception Distance, Inception Score, image-to-image translation, super-resolution GAN, adversarial diffusion distillation
---
## Appendix: Practical Labs
### Lab 1: Toy 1D GAN Training Loop with Minimax and Non-Saturating Losses
import numpy as np
np.random.seed(0)
class LinearModel:
"""A minimal single-layer model mapping inputs to a scalar via a
linear transform followed by a sigmoid (for the discriminator) or
identity (for the generator's raw output before reshaping)."""
def __init__(self, in_dim, out_dim):
self.W = np.random.randn(in_dim, out_dim) * 0.1
self.b = np.zeros(out_dim)
def forward(self, x):
return x @ self.W + self.b
def backward(self, x, grad_out, lr):
grad_W = x.T @ grad_out / x.shape[0]
grad_b = grad_out.mean(axis=0)
self.W -= lr * grad_W
self.b -= lr * grad_b
def sigmoid(x):
return 1.0 / (1.0 + np.exp(-np.clip(x, -30, 30)))
def true_data_sampler(n):
# Real data: samples from a Gaussian centered at 4.0 with std 0.5
return np.random.randn(n, 1) * 0.5 + 4.0
def noise_sampler(n):
return np.random.randn(n, 1) * 1.0
def train_toy_gan(n_steps=2000, batch_size=64, lr=0.05, use_nonsaturating=True):
G = LinearModel(1, 1)
D = LinearModel(1, 1)
for step in range(n_steps):
# --- Discriminator update ---
real = true_data_sampler(batch_size)
z = noise_sampler(batch_size)
fake = G.forward(z)
d_real_logits = D.forward(real)
d_fake_logits = D.forward(fake)
d_real = sigmoid(d_real_logits)
d_fake = sigmoid(d_fake_logits)
# Binary cross-entropy gradient w.r.t. logits: (pred - target)
grad_d_real = (d_real - 1.0)
grad_d_fake = (d_fake - 0.0)
grad_d_logits = np.concatenate([grad_d_real, grad_d_fake], axis=0)
d_input = np.concatenate([real, fake], axis=0)
D.backward(d_input, grad_d_logits, lr)
# --- Generator update ---
z = noise_sampler(batch_size)
fake = G.forward(z)
d_fake_logits = D.forward(fake)
d_fake = sigmoid(d_fake_logits)
if use_nonsaturating:
# Non-saturating loss: maximize log(D(G(z))) => grad w.r.t. logits is (d_fake - 1)
grad_wrt_fake_logits = (d_fake - 1.0)
else:
# Saturating minimax loss: minimize log(1 - D(G(z))) => grad is -(1 - d_fake) flipped
grad_wrt_fake_logits = -(1.0 - d_fake)
grad_wrt_fake = grad_wrt_fake_logits @ D.W.T
G.backward(z, grad_wrt_fake, lr)
return G, D
def test_toy_gan_training():
G, D = train_toy_gan(n_steps=3000, use_nonsaturating=True)
z = noise_sampler(2000)
fake_samples = G.forward(z)
gen_mean = fake_samples.mean()
gen_std = fake_samples.std()
print(f"Generator output mean: {gen_mean:.3f} (target ~4.0)")
print(f"Generator output std: {gen_std:.3f} (target ~0.5)")
# The generator should have moved its mean substantially toward the
# real data mean of 4.0, starting from a mean near 0.
assert gen_mean > 1.5, "Generator mean did not shift toward real data mean"
assert gen_std > 0.05, "Generator collapsed to near-zero variance (mode collapse)"
print("Toy GAN training test passed.")
if __name__ == "__main__":
test_toy_gan_training()### Lab 2: Non-Saturating vs. Saturating Generator Gradients Near Discriminator Saturation
import numpy as np
def sigmoid(x):
return 1.0 / (1.0 + np.exp(-np.clip(x, -30, 30)))
def saturating_generator_gradient(d_fake):
"""Gradient magnitude of the saturating loss log(1 - D(G(z))) w.r.t. D(G(z)),
which the generator wants to minimize (push toward log(1 - d) being very negative,
i.e. d close to 1 is good for the generator, but gradients vanish as d -> 0)."""
d_fake = np.clip(d_fake, 1e-7, 1 - 1e-7)
# d/dd [log(1 - d)] = -1 / (1 - d); this is the raw gradient signal magnitude
return np.abs(-1.0 / (1.0 - d_fake))
def nonsaturating_generator_gradient(d_fake):
"""Gradient magnitude of the non-saturating loss -log(D(G(z))) w.r.t. D(G(z))."""
d_fake = np.clip(d_fake, 1e-7, 1 - 1e-7)
# d/dd [-log(d)] = -1/d
return np.abs(-1.0 / d_fake)
def compare_gradient_strength(d_fake_values):
results = []
for d in d_fake_values:
sat_grad = saturating_generator_gradient(np.array([d]))[0]
nonsat_grad = nonsaturating_generator_gradient(np.array([d]))[0]
results.append((d, sat_grad, nonsat_grad))
return results
def test_nonsaturating_provides_stronger_early_gradients():
# Early in training the discriminator easily rejects the generator,
# so D(G(z)) is close to 0 (the discriminator confidently says "fake").
early_training_d_fake = 0.02
late_training_d_fake = 0.5
sat_early = saturating_generator_gradient(np.array([early_training_d_fake]))[0]
nonsat_early = nonsaturating_generator_gradient(np.array([early_training_d_fake]))[0]
print(f"Early training (D(G(z))={early_training_d_fake}):")
print(f" Saturating loss gradient magnitude: {sat_early:.3f}")
print(f" Non-saturating loss gradient magnitude: {nonsat_early:.3f}")
# The non-saturating loss should provide a much stronger gradient signal
# exactly when the generator needs it most (early, when it is easily detected).
assert nonsat_early > sat_early, (
"Non-saturating gradient should exceed saturating gradient when D(G(z)) is small"
)
assert nonsat_early > 40.0, "Non-saturating gradient should be strongly amplified near d=0"
results = compare_gradient_strength([0.02, 0.1, 0.3, 0.5, 0.7, 0.9])
print("
d_fake | saturating_grad | nonsaturating_grad")
for d, sat, nonsat in results:
print(f"{d:.2f} | {sat:14.3f} | {nonsat:.3f}")
print("Non-saturating gradient advantage test passed.")
if __name__ == "__main__":
test_nonsaturating_provides_stronger_early_gradients()### Lab 3: Wasserstein Critic with a Gradient Penalty (WGAN-GP) Regularization Term
import numpy as np
np.random.seed(1)
class Critic:
"""A simple two-layer critic (no sigmoid output, unlike a standard
GAN discriminator, since a Wasserstein critic outputs an unbounded score)."""
def __init__(self, in_dim, hidden_dim=8):
self.W1 = np.random.randn(in_dim, hidden_dim) * 0.3
self.b1 = np.zeros(hidden_dim)
self.W2 = np.random.randn(hidden_dim, 1) * 0.3
self.b2 = np.zeros(1)
def forward(self, x):
h = np.tanh(x @ self.W1 + self.b1)
out = h @ self.W2 + self.b2
return out.squeeze(-1), h
def numerical_gradient_norm(self, x, eps=1e-4):
"""Estimate the gradient norm of the critic output w.r.t. its input x,
via central finite differences, for use in a gradient penalty term."""
grads = np.zeros_like(x)
for i in range(x.shape[1]):
x_plus = x.copy()
x_minus = x.copy()
x_plus[:, i] += eps
x_minus[:, i] -= eps
out_plus, _ = self.forward(x_plus)
out_minus, _ = self.forward(x_minus)
grads[:, i] = (out_plus - out_minus) / (2 * eps)
return np.linalg.norm(grads, axis=1)
def wasserstein_critic_loss(critic, real, fake, gp_lambda=10.0):
"""Computes the WGAN-GP critic loss: maximize E[critic(real)] - E[critic(fake)],
minus a gradient penalty that discourages the critic's gradient norm from
deviating from 1 along random interpolations between real and fake samples."""
real_scores, _ = critic.forward(real)
fake_scores, _ = critic.forward(fake)
wasserstein_estimate = real_scores.mean() - fake_scores.mean()
# Interpolate between real and fake samples with a random mixing coefficient per sample
batch_size = real.shape[0]
epsilon = np.random.uniform(0, 1, size=(batch_size, 1))
interpolated = epsilon * real + (1 - epsilon) * fake
grad_norms = critic.numerical_gradient_norm(interpolated)
gradient_penalty = gp_lambda * np.mean((grad_norms - 1.0) ** 2)
# The critic wants to MAXIMIZE the Wasserstein estimate while minimizing
# the penalty, so the loss to minimize is the negative estimate plus the penalty.
critic_loss = -wasserstein_estimate + gradient_penalty
return critic_loss, wasserstein_estimate, gradient_penalty
def test_gradient_penalty_behavior():
critic = Critic(in_dim=2)
real = np.random.randn(32, 2) * 0.5 + np.array([3.0, 3.0])
fake = np.random.randn(32, 2) * 0.5 + np.array([-1.0, -1.0])
loss, w_estimate, gp = wasserstein_critic_loss(critic, real, fake, gp_lambda=10.0)
print(f"Wasserstein estimate: {w_estimate:.3f}")
print(f"Gradient penalty term: {gp:.3f}")
print(f"Total critic loss: {loss:.3f}")
assert gp >= 0.0, "Gradient penalty must be non-negative (it's a squared deviation)"
assert np.isfinite(loss), "Critic loss should be a finite number"
# If real and fake are far apart, an untrained critic's gradient norm along
# the interpolation path should generally be measurably different from exactly 1,
# confirming the penalty term is actually sensing something.
assert gp > 1e-6, "Gradient penalty should be non-trivial for an untrained critic"
print("WGAN-GP critic loss test passed.")
if __name__ == "__main__":
test_gradient_penalty_behavior()### Lab 4: Detecting Mode Collapse via Sample Diversity Metrics
import numpy as np
np.random.seed(2)
def generate_multimodal_target(n_samples, n_modes=4, spread=0.3):
"""Ground truth data has n_modes distinct clusters arranged on a circle."""
mode_indices = np.random.randint(0, n_modes, size=n_samples)
angles = 2 * np.pi * mode_indices / n_modes
centers = np.stack([np.cos(angles), np.sin(angles)], axis=1) * 3.0
noise = np.random.randn(n_samples, 2) * spread
return centers + noise, mode_indices
def assign_to_nearest_mode(samples, n_modes=4):
angles = 2 * np.pi * np.arange(n_modes) / n_modes
mode_centers = np.stack([np.cos(angles), np.sin(angles)], axis=1) * 3.0
dists = np.linalg.norm(samples[:, None, :] - mode_centers[None, :, :], axis=2)
return np.argmin(dists, axis=1)
def mode_coverage_score(generated_samples, n_modes=4, capture_radius=1.0):
"""Fraction of modes that have at least one generated sample within capture_radius,
and the fraction of generated samples that fall within any mode's capture radius."""
angles = 2 * np.pi * np.arange(n_modes) / n_modes
mode_centers = np.stack([np.cos(angles), np.sin(angles)], axis=1) * 3.0
dists = np.linalg.norm(
generated_samples[:, None, :] - mode_centers[None, :, :], axis=2
)
nearest_mode = np.argmin(dists, axis=1)
nearest_dist = np.min(dists, axis=1)
captured_mask = nearest_dist < capture_radius
modes_covered = len(set(nearest_mode[captured_mask].tolist()))
coverage_fraction = modes_covered / n_modes
high_quality_fraction = captured_mask.mean()
return coverage_fraction, high_quality_fraction
def simulate_mode_collapsed_generator(n_samples):
"""Simulates a generator that has collapsed onto only one of the four modes."""
single_mode_angle = 0.0
center = np.array([np.cos(single_mode_angle), np.sin(single_mode_angle)]) * 3.0
return center + np.random.randn(n_samples, 2) * 0.3
def simulate_healthy_generator(n_samples, n_modes=4):
samples, _ = generate_multimodal_target(n_samples, n_modes=n_modes, spread=0.3)
return samples
def test_mode_collapse_detection():
n_samples = 500
n_modes = 4
collapsed_samples = simulate_mode_collapsed_generator(n_samples)
healthy_samples = simulate_healthy_generator(n_samples, n_modes=n_modes)
collapsed_coverage, collapsed_quality = mode_coverage_score(
collapsed_samples, n_modes=n_modes
)
healthy_coverage, healthy_quality = mode_coverage_score(
healthy_samples, n_modes=n_modes
)
print(f"Collapsed generator: mode coverage={collapsed_coverage:.2f}, "
f"high-quality fraction={collapsed_quality:.2f}")
print(f"Healthy generator: mode coverage={healthy_coverage:.2f}, "
f"high-quality fraction={healthy_quality:.2f}")
assert collapsed_coverage <= 0.25 + 1e-9, "Collapsed generator should cover ~1 of 4 modes"
assert healthy_coverage >= 0.75, "Healthy generator should cover most modes"
assert healthy_coverage > collapsed_coverage, (
"Healthy generator must show strictly higher mode coverage than the collapsed one"
)
print("Mode collapse detection test passed.")
if __name__ == "__main__":
test_mode_collapse_detection()