Generative Adversarial Networks GAN Architecture Adversarial Training

# Generative Adversarial Networks: GAN Architecture & Adversarial Training

## Introduction & Motivation

GANs train two neural networks—generator and discriminator—in adversarial competition. Generator creates synthetic samples; discriminator distinguishes fake from real. Reaches Nash equilibrium; generator produces high-quality, realistic data. Foundation for image synthesis, super-resolution, style transfer.

Motivation: Traditional generative models (VAE, autoregressive) have limitations. Adversarial training enables implicit density modeling without explicit likelihood. Generator learns data distribution directly.

Applications: Image generation, data augmentation, style transfer, super-resolution, deepfakes.

---

## Core Concepts & Theory

### Generator & Discriminator

Generator G: z → x (latent noise to fake samples).
Discriminator D: x → [0,1] (real/fake classification).

### Adversarial Loss

Binary cross-entropy: min_G max_D E[log D(x)] + E[log(1 - D(G(z)))].

### Nash Equilibrium

Training converges when D correctly classifies (50/50) and G replicates data distribution.

---

## Mathematical Formulation

Minimax objective:
$$\min_G \max_D \mathbb{E}_{x \sim p_{ ext{data}}}[\log D(\mathbf{x})] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(\mathbf{z})))]$$

Wasserstein GAN (improved stability):
$$\min_G \max_D \mathbb{E}_{x}[D(\mathbf{x})] - \mathbb{E}_{z}[D(G(\mathbf{z}))]$$

---

## Advanced Theory & Extensions

### Mode Collapse

Generator produces limited sample diversity; discriminator overfits to specific modes. Addressed via spectral normalization, feature matching, minibatch discrimination.

### Conditional GAN (cGAN)

Condition both G and D on class label y: G(z, y), D(x, y). Enables class-conditional generation.

---

## Computational Considerations

Generator: Deconvolution layers (transpose convolution) or upsampling + convolution.

Discriminator: Standard CNN with batch normalization, spectral normalization for stability.

Training: ~O(iterations × (G forward + D forward + backward)) per step.

---

## Practical Implementation Strategies

### Training Stability

1. Use batch normalization in generator, spectral normalization in discriminator.
2. Label smoothing (real=0.9, fake=0.1) instead of (1, 0).
3. Train discriminator k steps per generator step (k=1-5).

### Latent Space Interpolation

Sample z1, z2; interpolate z(t) = (1-t)z1 + tz2; visualize G(z(t)) smoothness.

---

## Benchmark Datasets & Evaluation

CIFAR-10: 32×32 color images; Inception Score, FID.

CelebA: Face images; visual quality assessment.

Inception Score (IS): Entropy of predicted class distribution; higher = better diversity + confidence.

Fréchet Inception Distance (FID): Wasserstein distance between real/fake feature distributions; lower = better.

---

## Key Challenges & Limitations

### Mode Collapse

Generator ignores latent diversity; produces repeated samples.

### Training Instability

Discriminator loss → 0 too quickly; generator receives vanishing gradients.

### Evaluation Difficulty

No direct likelihood; rely on proxy metrics (IS, FID, human evaluation).

---

## Hyperparameter Tuning

Learning rates: G_lr ≈ 0.0002, D_lr ≈ 0.0002 (lower than supervised).

Batch size: 64-256 (larger stabilizes).

k (D steps per G step): 1-5.

---

## Real-World Applications & Case Studies

StyleGAN: Learned hierarchical image generation; high-resolution face synthesis.

CycleGAN: Unpaired image-to-image translation (photos ↔ paintings).

Super-Resolution GAN (SRGAN): Upscale low-res → high-res images.

---

## Integration with Other Methods

GAN + Reinforcement Learning → inverse reward learning.

GAN + Anomaly Detection → learn generator on normal data; OOD samples have high reconstruction error.

---

## Summary & Key Takeaways

GANs use adversarial competition between generator and discriminator to learn implicit data distributions, enabling high-quality synthetic data generation.

Principles:
1. Generator creates samples from noise; discriminator classifies real/fake.
2. Minimax game converges to Nash equilibrium.
3. Training is unstable; use batch norm, spectral norm, label smoothing.
4. Mode collapse limits diversity; addressed via architectural/training improvements.
5. Evaluate via IS, FID; no direct likelihood available.

---

---

## Appendix: Practical Labs

### Lab 1: Basic GAN on MNIST

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np

# Generator
class Generator(nn.Module):
 def __init__(self, latent_dim=100):
 super().__init__()
 self.model = nn.Sequential(
 nn.Linear(latent_dim, 128),
 nn.ReLU(),
 nn.Linear(128, 256),
 nn.ReLU(),
 nn.Linear(256, 784),
 nn.Tanh()
 )
 def forward(self, z):
 return self.model(z)

# Discriminator
class Discriminator(nn.Module):
 def __init__(self):
 super().__init__()
 self.model = nn.Sequential(
 nn.Linear(784, 256),
 nn.ReLU(),
 nn.Linear(256, 128),
 nn.ReLU(),
 nn.Linear(128, 1),
 nn.Sigmoid()
 )
 def forward(self, x):
 return self.model(x)

# Setup
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
G = Generator().to(device)
D = Discriminator().to(device)
g_opt = optim.Adam(G.parameters(), lr=0.0002, betas=(0.5, 0.999))
d_opt = optim.Adam(D.parameters(), lr=0.0002, betas=(0.5, 0.999))
criterion = nn.BCELoss()

# Data
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
mnist = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
loader = DataLoader(mnist, batch_size=64, shuffle=True)

# Train 1 epoch
latent_dim = 100
g_loss_log, d_loss_log = [], []
for real_x, _ in loader:
 real_x = real_x.view(real_x.size(0), -1).to(device)
 batch_size = real_x.size(0)
 
 # D update
 d_opt.zero_grad()
 real_y = torch.ones(batch_size, 1).to(device)
 fake_y = torch.zeros(batch_size, 1).to(device)
 d_real = D(real_x)
 d_loss_real = criterion(d_real, real_y)
 
 z = torch.randn(batch_size, latent_dim).to(device)
 fake_x = G(z)
 d_fake = D(fake_x.detach())
 d_loss_fake = criterion(d_fake, fake_y)
 
 d_loss = d_loss_real + d_loss_fake
 d_loss.backward()
 d_opt.step()
 d_loss_log.append(d_loss.item())
 
 # G update
 g_opt.zero_grad()
 z = torch.randn(batch_size, latent_dim).to(device)
 fake_x = G(z)
 d_fake = D(fake_x)
 g_loss = criterion(d_fake, real_y)
 g_loss.backward()
 g_opt.step()
 g_loss_log.append(g_loss.item())

avg_g_loss = np.mean(g_loss_log)
avg_d_loss = np.mean(d_loss_log)
print(f"Generator loss: {avg_g_loss:.4f}, Discriminator loss: {avg_d_loss:.4f}")
assert avg_g_loss > 0 and avg_d_loss > 0, "Losses should be positive"
assert avg_g_loss < 10 and avg_d_loss < 10, "Losses should be reasonable"
print("✓ Basic GAN training working")

if __name__ == "__main__":
 print("Lab 1: Basic GAN - PASSED")

### Lab 2: Latent Space Interpolation

import torch
import torch.nn as nn
import numpy as np

# Reuse Generator from Lab 1
class Generator(nn.Module):
 def __init__(self, latent_dim=100):
 super().__init__()
 self.model = nn.Sequential(
 nn.Linear(latent_dim, 128),
 nn.ReLU(),
 nn.Linear(128, 256),
 nn.ReLU(),
 nn.Linear(256, 784),
 nn.Tanh()
 )
 def forward(self, z):
 return self.model(z)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
G = Generator().to(device)
latent_dim = 100

# Sample two latent vectors
z1 = torch.randn(1, latent_dim).to(device)
z2 = torch.randn(1, latent_dim).to(device)

# Interpolate
n_steps = 5
interpolated = []
for t in np.linspace(0, 1, n_steps):
 z_t = (1 - t) * z1 + t * z2
 x_t = G(z_t).detach().cpu().numpy()
 interpolated.append(x_t)

print(f"Interpolation steps: {len(interpolated)}")
assert len(interpolated) == n_steps, "Should have n_steps samples"
assert all(x.shape == (1, 784) for x in interpolated), "All samples should have shape (1, 784)"
print("✓ Latent space interpolation working")

if __name__ == "__main__":
 print("Lab 2: Interpolation - PASSED")

### Lab 3: Discriminator Loss Monitoring

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np

class Generator(nn.Module):
 def __init__(self, latent_dim=100):
 super().__init__()
 self.model = nn.Sequential(
 nn.Linear(latent_dim, 128),
 nn.ReLU(),
 nn.Linear(128, 256),
 nn.ReLU(),
 nn.Linear(256, 784),
 nn.Tanh()
 )
 def forward(self, z):
 return self.model(z)

class Discriminator(nn.Module):
 def __init__(self):
 super().__init__()
 self.model = nn.Sequential(
 nn.Linear(784, 256),
 nn.ReLU(),
 nn.Linear(256, 128),
 nn.ReLU(),
 nn.Linear(128, 1),
 nn.Sigmoid()
 )
 def forward(self, x):
 return self.model(x)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
G = Generator().to(device)
D = Discriminator().to(device)
d_opt = optim.Adam(D.parameters(), lr=0.0002)
criterion = nn.BCELoss()

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
mnist = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
loader = DataLoader(mnist, batch_size=64, shuffle=True)

latent_dim = 100
d_losses = []
for real_x, _ in loader:
 real_x = real_x.view(real_x.size(0), -1).to(device)
 batch_size = real_x.size(0)
 
 d_opt.zero_grad()
 real_y = torch.ones(batch_size, 1).to(device)
 fake_y = torch.zeros(batch_size, 1).to(device)
 
 d_real = D(real_x)
 d_loss_real = criterion(d_real, real_y)
 
 z = torch.randn(batch_size, latent_dim).to(device)
 fake_x = G(z).detach()
 d_fake = D(fake_x)
 d_loss_fake = criterion(d_fake, fake_y)
 
 d_loss = d_loss_real + d_loss_fake
 d_loss.backward()
 d_opt.step()
 d_losses.append(d_loss.item())

avg_d_loss = np.mean(d_losses)
print(f"Average discriminator loss: {avg_d_loss:.4f}")
assert len(d_losses) > 0, "Should have loss values"
assert 0 < avg_d_loss < 10, "D loss should be in reasonable range"
print("✓ Discriminator loss monitoring working")

if __name__ == "__main__":
 print("Lab 3: D Loss Monitoring - PASSED")

### Lab 4: Mode Collapse Detection

import torch
import torch.nn as nn
import numpy as np
from scipy.spatial.distance import pdist

class Generator(nn.Module):
 def __init__(self, latent_dim=100):
 super().__init__()
 self.model = nn.Sequential(
 nn.Linear(latent_dim, 128),
 nn.ReLU(),
 nn.Linear(128, 256),
 nn.ReLU(),
 nn.Linear(256, 784),
 nn.Tanh()
 )
 def forward(self, z):
 return self.model(z)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
G = Generator().to(device)
latent_dim = 100

# Generate samples
n_samples = 100
z = torch.randn(n_samples, latent_dim).to(device)
fake_x = G(z).detach().cpu().numpy()

# Compute pairwise distances
distances = pdist(fake_x, metric='euclidean')
mean_distance = np.mean(distances)
std_distance = np.std(distances)
diversity_ratio = std_distance / (mean_distance + 1e-8)

print(f"Mean pairwise distance: {mean_distance:.4f}")
print(f"Diversity ratio: {diversity_ratio:.4f}")
assert mean_distance > 0, "Distances should be positive"
assert diversity_ratio > 0, "Diversity ratio should be positive"
assert len(fake_x) == n_samples, "Should have n_samples"
print("✓ Mode collapse detection working")

if __name__ == "__main__":
 print("Lab 4: Mode Collapse - PASSED")