Variational Autoencoders Vaes

# Variational Autoencoders (VAEs)

## Introduction & Motivation

VAEs: probabilistic latent variable models. Encoder-decoder with KL divergence. Applications: generative modeling, latent space manipulation.

Motivation: Learn interpretable latent representations.

Applications: Image generation, anomaly detection.

---

## Core Concepts & Theory

### Encoder Network

Map data to latent distribution.

### Reparameterization Trick

Sample latent variables differentiably.

### KL Divergence

Regularize latent distribution.

### Decoder Network

Reconstruct from latent samples.

---

## Mathematical Formulation

ELBO:
$$\mathcal{L} = \mathbb{E}[log p(x|z)] - D_{KL}(q(z|x) || p(z))$$

KL Divergence (Gaussian):
$$D_{KL} = -\frac{1}{2} \sum (1 + log \sigma^2 - \mu^2 - \sigma^2)$$

Reparameterization:
$$z = \mu + \sigma \odot \epsilon, \quad \epsilon \sim N(0, I)$$

---

## Advanced Theory & Extensions

### Beta-VAE

Balance reconstruction and regularization.

### Hierarchical VAE

Multi-level latent hierarchy.

### VQ-VAE

Vector quantized latent space.

---

## Computational Considerations

Encoder: O(x·h·z).

Reparameterization: O(z).

Decoder: O(z·h·x).

---

## Practical Implementation Strategies

### KL Annealing

Gradually increase KL weight.

### Gradient Checkpointing

Reduce memory usage.

### Batch Normalization

Stabilize training.

---

## Benchmark Datasets & Evaluation

MNIST: Simple benchmark.

CelebA: Complex image generation.

ImageNet: Large-scale evaluation.

---

## Key Challenges & Limitations

### Posterior Collapse

KL divergence goes to zero.

### Blurry Reconstructions

Trade-off between reconstruction and regularization.

### Training Instability

Balancing multiple objectives.

---

## Hyperparameter Tuning

Beta (KL weight): 0.1-1.0.

Learning rate: 1e-4 to 1e-3.

Latent dimension: 8-512.

---

## Real-World Applications & Case Studies

Image Generation: Novel sample creation.

Anomaly Detection: Reconstruction-based outliers.

Data Interpolation: Latent space traversal.

---

## Integration with Other Methods

VAE + GAN hybrid models; + flow-based models for posterior.

---

## Summary & Key Takeaways

VAEs learn probabilistic latent representations via encoder-decoder architecture.

Principles:
1. Encoder: Posterior inference network.
2. Decoder: Generative likelihood network.
3. KL divergence: Prior regularization.
4. ELBO: Variational lower bound.
5. Reparameterization: Gradient-friendly sampling.

---

## Appendix: Practical Labs

### Lab 1: Reparameterization Trick

import numpy as np

def reparameterize(mu, logvar):
 """Sample from latent distribution using reparameterization trick"""
 sigma = np.exp(0.5 * logvar)
 epsilon = np.random.randn(*mu.shape)
 z = mu + sigma * epsilon
 return z

np.random.seed(42)
mu = np.random.randn(16, 32)
logvar = np.log(np.ones((16, 32)) * 0.1)
z = reparameterize(mu, logvar)
assert z.shape == mu.shape, "Correct latent shape"
print("✓ Reparameterization trick working")

### Lab 2: KL Divergence Loss

import numpy as np

def kl_divergence_loss(mu, logvar):
 """Compute KL divergence between q(z|x) and N(0,I)"""
 kl = -0.5 * np.sum(1 + logvar - mu**2 - np.exp(logvar))
 return kl

np.random.seed(42)
mu = np.random.randn(16, 32)
logvar = np.log(np.ones((16, 32)) * 0.1)
kl = kl_divergence_loss(mu, logvar)
assert kl >= 0, "Non-negative KL divergence"
print(f"✓ KL divergence loss working: {kl:.4f}")

### Lab 3: ELBO Computation

import numpy as np

def elbo_loss(x, x_recon, mu, logvar):
 """Compute Evidence Lower Bound"""
 # Reconstruction loss (mean squared error)
 recon_loss = np.mean((x - x_recon) ** 2)
 
 # KL divergence
 kl = -0.5 * np.sum(1 + logvar - mu**2 - np.exp(logvar))
 
 # ELBO
 elbo = recon_loss + kl
 return elbo, recon_loss, kl

np.random.seed(42)
x = np.random.randn(16, 784)
x_recon = np.random.randn(16, 784)
mu = np.random.randn(16, 32)
logvar = np.log(np.ones((16, 32)) * 0.1)
elbo, recon, kl = elbo_loss(x, x_recon, mu, logvar)
assert elbo > 0, "Positive ELBO"
print(f"✓ ELBO working: ELBO={elbo:.4f}, Recon={recon:.4f}, KL={kl:.4f}")

### Lab 4: Latent Space Interpolation

import numpy as np

def interpolate_latent(z1, z2, steps=5):
 """Interpolate between two latent points"""
 interpolations = []
 for t in np.linspace(0, 1, steps):
 z_interp = (1 - t) * z1 + t * z2
 interpolations.append(z_interp)
 return np.array(interpolations)

np.random.seed(42)
z1 = np.random.randn(32)
z2 = np.random.randn(32)
interp = interpolate_latent(z1, z2, steps=5)
assert interp.shape[0] == 5, "Correct interpolation steps"
assert np.allclose(interp[0], z1), "Start point preserved"
assert np.allclose(interp[-1], z2), "End point preserved"
print("✓ Latent space interpolation working")

---