Variational Autoencoder VAE Generative Models

# Variational Autoencoder: VAE & Generative Models

## Introduction & Motivation

Variational autoencoders: probabilistic generative models. Encoder: map to latent distribution. Decoder: reconstruct from samples. ELBO: evidence lower bound; training objective. Applications: image generation, data augmentation, disentangled representations.

Motivation: GANs difficult to train; VAEs provide stable alternative. Learn interpretable latent spaces.

Applications: Image generation, representation learning.

---

## Core Concepts & Theory

### Latent Distribution

Gaussian prior; learned posterior.

### Reparameterization Trick

Differentiable sampling; gradient flow.

### ELBO Loss

Reconstruction + KL divergence.

---

## Mathematical Formulation

Encoder (posterior):
$$q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$$

Reparameterization:
$$z = \mu + \sigma \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

ELBO objective:
$$\mathcal{L} = \mathbb{E}_{q_\phi(z|x)}[\log p_ heta(x|z)] - ext{KL}(q_\phi(z|x) \| p(z))$$

---

## Advanced Theory & Extensions

### Beta-VAE

Balance reconstruction and KL; disentanglement.

### Hierarchical VAE

Hierarchical latent variables; multi-scale.

### Conditional VAE

Conditioned on class label; CVAE.

---

## Computational Considerations

VAE training: O(batch_size · latent_dim).

Inference: O(encoder + decoder).

Memory: Store latent distributions.

---

## Practical Implementation Strategies

### KL Annealing

Gradually increase KL weight; avoid posterior collapse.

### Reconstruction Loss

L2 for continuous; BCE for binary.

### Latent Regularization

Additional constraints; disentanglement.

---

## Benchmark Datasets & Evaluation

MNIST: VAE standard; generation quality.

CelebA: Face generation; high-resolution.

FashionMNIST: Fashion images.

---

## Key Challenges & Limitations

### Posterior Collapse

KL term zeros out; latent unused.

### Blurry Generation

Averaging over modes; reconstruction loss.

### Limited Expressiveness

Simple Gaussian posterior; limited distributions.

---

## Hyperparameter Tuning

Latent dimension: 10-100 typical.

KL weight β: 0.1-1.0; annealing useful.

Architecture depth: 3-4 layers typical.

---

## Real-World Applications & Case Studies

Image Generation: MNIST, CelebA.

Data Augmentation: Generate training data.

Disentanglement: Interpretable latent factors.

---

## Integration with Other Methods

VAE + GAN → hybrid generative.

VAE + Classification → semi-supervised.

---

## Summary & Key Takeaways

Variational autoencoders via reparameterization and ELBO provide stable generative modeling with interpretable latent representations.

Principles:
1. Encoder-decoder: probabilistic mapping.
2. Reparameterization: gradient-friendly sampling.
3. ELBO: reconstruction + regularization.
4. KL divergence: latent distribution.
5. Disentanglement: interpretable factors.

---

---

## Appendix: Practical Labs

### Lab 1: Reparameterization Trick

import numpy as np

def reparameterize(mu, log_var):
 """Reparameterization trick: differentiable sampling"""
 # Sample epsilon
 epsilon = np.random.randn(*mu.shape)
 
 # Compute standard deviation from log variance
 std = np.exp(0.5 * log_var)
 
 # Reparameterize
 z = mu + std * epsilon
 
 return z

# Test
np.random.seed(42)
mu = np.random.randn(32, 20)
log_var = np.random.randn(32, 20)

z = reparameterize(mu, log_var)

assert z.shape == mu.shape, "Sample shape"
print("✓ Reparameterization trick working")

if __name__ == "__main__":
 print("Lab 1: Reparameterization - PASSED")

### Lab 2: ELBO Loss

import numpy as np

def vae_loss(x_true, x_recon, mu, log_var, beta=1.0):
 """VAE ELBO loss"""
 # Reconstruction loss (L2)
 recon_loss = np.mean((x_true - x_recon) ** 2)
 
 # KL divergence: KL(N(mu, var) || N(0, I))
 kl_loss = -0.5 * np.mean(1 + log_var - mu**2 - np.exp(log_var))
 
 # ELBO
 elbo = recon_loss + beta * kl_loss
 
 return elbo, recon_loss, kl_loss

# Test
np.random.seed(42)
x_true = np.random.randn(32, 784)
x_recon = np.random.randn(32, 784)
mu = np.random.randn(32, 20)
log_var = np.random.randn(32, 20)

elbo, recon, kl = vae_loss(x_true, x_recon, mu, log_var, beta=1.0)

assert np.isfinite(elbo), "ELBO finite"
print("✓ ELBO loss working")

if __name__ == "__main__":
 print("Lab 2: ELBOLoss - PASSED")

### Lab 3: Posterior Collapse Detection

import numpy as np

def detect_posterior_collapse(log_var, threshold=1e-2):
 """Detect posterior collapse via KL divergence"""
 # KL divergence for each sample
 kl_per_sample = -0.5 * np.mean(1 + log_var, axis=1)
 
 # Proportion of dims with low KL
 collapsed = kl_per_sample < threshold
 
 collapse_ratio = collapsed.mean()
 
 return collapse_ratio

# Test
np.random.seed(42)
log_var = np.random.randn(100, 20) - 5 # Low variance → collapse

collapse_ratio = detect_posterior_collapse(log_var)

assert 0 <= collapse_ratio <= 1, "Ratio in [0,1]"
print("✓ Posterior collapse detection working")

if __name__ == "__main__":
 print("Lab 3: PosteriorCollapse - PASSED")

### Lab 4: Generation Quality Metrics

import numpy as np

def compute_reconstruction_quality(x_true, x_recon):
 """Compute reconstruction metrics"""
 # MSE
 mse = np.mean((x_true - x_recon) ** 2)
 
 # PSNR (Peak Signal-to-Noise Ratio)
 max_val = 1.0
 psnr = 20 * np.log10(max_val) - 10 * np.log10(mse + 1e-8)
 
 # SSIM (simplified)
 mu_x = x_true.mean()
 mu_y = x_recon.mean()
 sigma_x = x_true.var()
 sigma_y = x_recon.var()
 sigma_xy = np.cov(x_true.flatten(), x_recon.flatten())[0, 1]
 
 c1, c2 = 0.01, 0.03
 ssim = ((2 * mu_x * mu_y + c1) * (2 * sigma_xy + c2)) / \
 ((mu_x**2 + mu_y**2 + c1) * (sigma_x + sigma_y + c2))
 
 return {"mse": mse, "psnr": psnr, "ssim": ssim}

# Test
np.random.seed(42)
x_true = np.random.rand(100, 784)
x_recon = x_true + 0.1 * np.random.randn(100, 784)

metrics = compute_reconstruction_quality(x_true, x_recon)

assert all(np.isfinite(v) for v in metrics.values()), "Metrics finite"
print("✓ Generation quality metrics working")

if __name__ == "__main__":
 print("Lab 4: ReconstructionQuality - PASSED")