generative adversarial networks gans adversarial learning
# Generative Adversarial Networks: GANs & Adversarial Learning
## Introduction & Motivation
Generative Adversarial Networks: two networks compete. Generator creates samples; discriminator classifies real vs. fake. Adversarial loss: zero-sum game. Applications: image generation, style transfer, data synthesis, super-resolution.
Motivation: Learn generative models without explicit density. Competitive training drives realism.
Applications: Image generation, data augmentation, synthetic media.
---
## Core Concepts & Theory
### Generator Network
Maps noise to data distribution.
### Discriminator Network
Classifies real vs. generated samples.
### Adversarial Loss
Zero-sum game between G and D.
---
## Mathematical Formulation
GAN objective:
$$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{ ext{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$$
Generator loss (non-saturating):
$$L_G = -\mathbb{E}_{z \sim p_z}[\log D(G(z))]$$
Discriminator loss:
$$L_D = -\mathbb{E}_{x \sim p_{ ext{data}}}[\log D(x)] - \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$$
---
## Advanced Theory & Extensions
### Conditional GANs
Class-conditioned generation.
### Wasserstein GANs
Wasserstein distance; stable training.
### Spectral Normalization
Discriminator Lipschitz constraint.
---
## Computational Considerations
Generator: O(latent_dim → image_res²·channels).
Discriminator: O(image_res²·channels → binary).
Training: alternating updates; computational cost.
---
## Practical Implementation Strategies
### Batch Normalization
Stabilize training; discriminator normalization strategy.
### Learning Rate Scheduling
Different rates for G and D.
### Spectral Norm
Enforce Lipschitz continuity.
---
## Benchmark Datasets & Evaluation
MNIST: Digit generation baseline.
CelebA: Face generation benchmark.
ImageNet: Large-scale synthesis.
---
## Key Challenges & Limitations
### Mode Collapse
Generator produces limited diversity.
### Training Instability
Discriminator overpowers generator.
### Evaluation Difficulty
No likelihood; IS/FID metrics.
---
## Hyperparameter Tuning
Learning rate G: 0.0001-0.0004; slower than D.
Learning rate D: 0.0004-0.0008; higher than G.
Batch size: 64-256; stability.
---
## Real-World Applications & Case Studies
Image Synthesis: Photorealistic generation.
Style Transfer: CycleGAN image-to-image.
Data Augmentation: Synthetic training data.
---
## Integration with Other Methods
GAN + Classifier → semi-supervised learning.
GAN + Encoder → image-to-image translation.
---
## Summary & Key Takeaways
Generative Adversarial Networks via adversarial learning enable image generation through competing generator and discriminator networks.
Principles:
1. Adversarial objective: G vs. D.
2. Generator: noise to data.
3. Discriminator: real/fake classification.
4. Wasserstein: training stability.
5. Applications: synthesis, augmentation.
---
---
## Appendix: Practical Labs
### Lab 1: Generator and Discriminator Loss
import numpy as np
def compute_gan_losses(real_logits, fake_logits):
"""Compute GAN losses"""
# Discriminator loss
real_loss = -np.log(1 / (1 + np.exp(-real_logits)) + 1e-8).mean()
fake_loss = -np.log(1 - 1 / (1 + np.exp(-fake_logits)) + 1e-8).mean()
d_loss = real_loss + fake_loss
# Generator loss (non-saturating)
g_loss = -np.log(1 / (1 + np.exp(-fake_logits)) + 1e-8).mean()
return d_loss, g_loss
# Test
np.random.seed(42)
real_logits = np.random.randn(32)
fake_logits = np.random.randn(32) - 1
d_loss, g_loss = compute_gan_losses(real_logits, fake_logits)
assert np.isfinite(d_loss), "D loss finite"
assert np.isfinite(g_loss), "G loss finite"
print("✓ GAN losses working")
if __name__ == "__main__":
print("Lab 1: GANLosses - PASSED")### Lab 2: Wasserstein Distance
import numpy as np
def wasserstein_distance(real_samples, fake_samples):
"""Compute Wasserstein distance (1D approximation)"""
real_sorted = np.sort(real_samples)
fake_sorted = np.sort(fake_samples)
# Pad to same size
max_len = max(len(real_sorted), len(fake_sorted))
real_pad = np.pad(real_sorted, (0, max_len - len(real_sorted)))
fake_pad = np.pad(fake_sorted, (0, max_len - len(fake_sorted)))
# Wasserstein distance
w_dist = np.mean(np.abs(real_pad - fake_pad))
return w_dist
# Test
np.random.seed(42)
real = np.random.randn(100)
fake = np.random.randn(100) + 0.5
w_dist = wasserstein_distance(real, fake)
assert w_dist >= 0, "Distance non-negative"
print("✓ Wasserstein distance working")
if __name__ == "__main__":
print("Lab 2: WassersteinDistance - PASSED")### Lab 3: Mode Coverage Analysis
import numpy as np
def analyze_mode_collapse(fake_samples, num_modes=10, threshold=0.1):
"""Detect mode collapse"""
# Cluster samples into modes
covered_modes = 0
for mode_idx in range(num_modes):
mode_center = mode_idx * (1.0 / num_modes)
# Check if generator covers this mode
in_mode = np.abs(fake_samples - mode_center) < threshold
if in_mode.sum() > 0:
covered_modes += 1
coverage = covered_modes / num_modes
return coverage
# Test
np.random.seed(42)
fake_samples = np.random.randn(1000) * 0.5 # Limited diversity
coverage = analyze_mode_collapse(fake_samples)
assert 0 <= coverage <= 1, "Coverage in [0,1]"
print("✓ Mode coverage analysis working")
if __name__ == "__main__":
print("Lab 3: ModeCollapse - PASSED")### Lab 4: Inception Score Approximation
import numpy as np
def approximate_inception_score(fake_logits, num_classes=10):
"""Approximate Inception Score from logits"""
# Softmax
exp_logits = np.exp(fake_logits - np.max(fake_logits, axis=1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=1, keepdims=True)
# Marginal distribution
p_y = probs.mean(axis=0)
# KL divergence
kl_divs = []
for prob in probs:
kl = np.sum(prob * (np.log(prob + 1e-8) - np.log(p_y + 1e-8)))
kl_divs.append(kl)
# Inception Score
is_score = np.exp(np.mean(kl_divs))
return is_score
# Test
np.random.seed(42)
fake_logits = np.random.randn(100, 10)
is_score = approximate_inception_score(fake_logits)
assert is_score >= 1, "IS >= 1"
print("✓ Inception score working")
if __name__ == "__main__":
print("Lab 4: InceptionScore - PASSED")