variational autoencoders vae probabilistic generative model

# Variational Autoencoders: VAE Probabilistic Generative Model

## Introduction & Motivation

VAEs learn probabilistic latent representations via encoder-decoder architecture. Encoder maps data to latent distribution (mean, variance); decoder samples and reconstructs. ELBO (evidence lower bound) objective balances reconstruction and regularization. Foundation for interpretable generative models, disentangled representations.

Motivation: Unlike standard autoencoders, VAEs enforce structured latent space via KL regularization. Enables principled sampling, interpolation, semi-supervised learning.

Applications: Image generation, semi-supervised learning, disentangled representation learning, anomaly detection.

---

## Core Concepts & Theory

### Encoder & Decoder

Encoder q_φ(z|x): approximates posterior p(z|x).
Decoder p_θ(x|z): reconstructs x from latent z.

### Reparameterization Trick

Sample z = μ + ε ⊙ σ where ε ~ N(0,I). Enables backprop through stochastic sampling.

### ELBO

log p(x) ≥ E_q[log p(x|z)] - KL(q(z|x) || p(z)).
First term: reconstruction; second: latent regularization.

---

## Mathematical Formulation

ELBO objective:
$$\mathcal{L} = \mathbb{E}_{q_\phi(\mathbf{z}|\mathbf{x})}[\log p_ heta(\mathbf{x}|\mathbf{z})] - ext{KL}(q_\phi(\mathbf{z}|\mathbf{x}) \| p(\mathbf{z}))$$

Reparameterization:
$$\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, \mathbf{I})$$

KL divergence (Gaussian):
$$ ext{KL} = -\frac{1}{2} \sum_j (1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2)$$

---

## Advanced Theory & Extensions

### β-VAE

Add weight β to KL term: β·KL(q||p). Increases regularization; improves disentanglement.

### Hierarchical VAE

Stack VAEs; encode latents hierarchically. Captures multi-scale structure.

---

## Computational Considerations

Encoder: MLP or CNN to (μ, log σ²).

Decoder: Transpose network or upsampling + CNN.

Training: ~O(batch_size × latent_dim) per step.

---

## Practical Implementation Strategies

### Beta Scheduling

Start β=0, gradually increase to 1 over epochs. Prevents posterior collapse.

### Latent Dimensionality

Balance reconstruction vs. regularization; typical 8-128 for images.

### Loss Weight Balance

λ_recon, λ_kl tuned via validation.

---

## Benchmark Datasets & Evaluation

MNIST: 28×28 grayscale; visual quality + disentanglement.

CelebA: 64×64 face images; generation quality.

Metrics: ELBO, reconstruction MSE, KL divergence, mutual information (for disentanglement).

---

## Key Challenges & Limitations

### Posterior Collapse

KL → 0; encoder ignored. Solutions: β-scheduling, free bits.

### Blurry Reconstructions

Gaussian reconstruction loss encourages averaging. Use VQ-VAE or hierarchical VAE.

---

## Hyperparameter Tuning

Learning rate: 0.001-0.01.

β (KL weight): 0.01-1.0; higher = more regularization.

Latent dim: 8-128.

---

## Real-World Applications & Case Studies

β-VAE: Disentangled representations; transfer learning.

Temporal VAE: Video generation via factored latents.

Semi-supervised VAE: Label + unlabeled data via M1 or M2 framework.

---

## Integration with Other Methods

VAE + RL → probabilistic world models.

VAE + Adversarial → adversarially learned inference (ALI).

---

## Summary & Key Takeaways

VAEs combine autoencoders with probabilistic latent models via reparameterization and ELBO, enabling structured generative modeling.

Principles:
1. Encoder maps to latent distribution; decoder reconstructs.
2. Reparameterization trick enables gradient-based sampling.
3. ELBO balances reconstruction and KL regularization.
4. Posterior collapse and blurry outputs are main challenges.
5. β-VAE improves disentanglement via increased regularization.

---

---

## Appendix: Practical Labs

### Lab 1: Basic VAE on MNIST

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np

class VAE(nn.Module):
 def __init__(self, latent_dim=20):
 super().__init__()
 self.latent_dim = latent_dim
 
 self.encoder = nn.Sequential(
 nn.Linear(784, 512),
 nn.ReLU(),
 nn.Linear(512, 256),
 nn.ReLU()
 )
 self.fc_mu = nn.Linear(256, latent_dim)
 self.fc_log_sigma = nn.Linear(256, latent_dim)
 
 self.decoder = nn.Sequential(
 nn.Linear(latent_dim, 256),
 nn.ReLU(),
 nn.Linear(256, 512),
 nn.ReLU(),
 nn.Linear(512, 784),
 nn.Sigmoid()
 )
 
 def encode(self, x):
 h = self.encoder(x)
 mu = self.fc_mu(h)
 log_sigma = self.fc_log_sigma(h)
 return mu, log_sigma
 
 def reparameterize(self, mu, log_sigma):
 std = torch.exp(0.5 * log_sigma)
 eps = torch.randn_like(std)
 z = mu + eps * std
 return z
 
 def decode(self, z):
 return self.decoder(z)
 
 def forward(self, x):
 mu, log_sigma = self.encode(x.view(-1, 784))
 z = self.reparameterize(mu, log_sigma)
 recon = self.decode(z)
 return recon, mu, log_sigma, z

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = VAE(latent_dim=20).to(device)
optimizer = optim.Adam(model.parameters(), lr=0.001)

transform = transforms.Compose([transforms.ToTensor()])
mnist = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
loader = DataLoader(mnist, batch_size=128, shuffle=True)

def vae_loss(recon, x, mu, log_sigma):
 bce = nn.functional.binary_cross_entropy(recon, x.view(-1, 784), reduction='sum')
 kl = -0.5 * torch.sum(1 + log_sigma - mu.pow(2) - log_sigma.exp())
 return bce + kl

loss_log = []
for x, _ in loader:
 x = x.to(device)
 optimizer.zero_grad()
 recon, mu, log_sigma, z = model(x)
 loss = vae_loss(recon, x, mu, log_sigma)
 loss.backward()
 optimizer.step()
 loss_log.append(loss.item())

avg_loss = np.mean(loss_log)
print(f"Average ELBO loss: {avg_loss:.2f}")
assert avg_loss > 0, "Loss should be positive"
assert len(loss_log) > 0, "Should have loss values"
print("✓ Basic VAE training working")

if __name__ == "__main__":
 print("Lab 1: Basic VAE - PASSED")

### Lab 2: Latent Space Traversal

import torch
import torch.nn as nn
import numpy as np

class VAE(nn.Module):
 def __init__(self, latent_dim=20):
 super().__init__()
 self.latent_dim = latent_dim
 self.encoder = nn.Sequential(nn.Linear(784, 512), nn.ReLU(), nn.Linear(512, 256), nn.ReLU())
 self.fc_mu = nn.Linear(256, latent_dim)
 self.fc_log_sigma = nn.Linear(256, latent_dim)
 self.decoder = nn.Sequential(nn.Linear(latent_dim, 256), nn.ReLU(), nn.Linear(256, 512), nn.ReLU(), nn.Linear(512, 784), nn.Sigmoid())
 
 def decode(self, z):
 return self.decoder(z)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = VAE(latent_dim=20).to(device)

# Sample latent and traverse one dimension
z = torch.randn(1, 20).to(device)
traversals = []
for val in np.linspace(-3, 3, 7):
 z_copy = z.clone()
 z_copy[0, 0] = val
 recon = model.decode(z_copy).detach().cpu().numpy()
 traversals.append(recon)

print(f"Traversal samples: {len(traversals)}")
assert len(traversals) == 7, "Should have 7 samples"
assert all(x.shape == (1, 784) for x in traversals), "All should have shape (1, 784)"
print("✓ Latent traversal working")

if __name__ == "__main__":
 print("Lab 2: Traversal - PASSED")

### Lab 3: KL Divergence Monitoring

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np

class VAE(nn.Module):
 def __init__(self, latent_dim=20):
 super().__init__()
 self.latent_dim = latent_dim
 self.encoder = nn.Sequential(nn.Linear(784, 512), nn.ReLU(), nn.Linear(512, 256), nn.ReLU())
 self.fc_mu = nn.Linear(256, latent_dim)
 self.fc_log_sigma = nn.Linear(256, latent_dim)
 self.decoder = nn.Sequential(nn.Linear(latent_dim, 256), nn.ReLU(), nn.Linear(256, 512), nn.ReLU(), nn.Linear(512, 784), nn.Sigmoid())
 
 def encode(self, x):
 h = self.encoder(x)
 return self.fc_mu(h), self.fc_log_sigma(h)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = VAE().to(device)

transform = transforms.Compose([transforms.ToTensor()])
mnist = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
loader = DataLoader(mnist, batch_size=128, shuffle=True)

kl_values = []
for x, _ in loader:
 x = x.to(device)
 mu, log_sigma = model.encode(x.view(-1, 784))
 kl = -0.5 * torch.mean(1 + log_sigma - mu.pow(2) - log_sigma.exp())
 kl_values.append(kl.item())

avg_kl = np.mean(kl_values)
print(f"Average KL divergence: {avg_kl:.4f}")
assert avg_kl > 0, "KL should be positive"
assert avg_kl < 100, "KL should be reasonable"
print("✓ KL monitoring working")

if __name__ == "__main__":
 print("Lab 3: KL Monitoring - PASSED")

### Lab 4: Reconstruction Quality

import torch
import torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np

class VAE(nn.Module):
 def __init__(self, latent_dim=20):
 super().__init__()
 self.latent_dim = latent_dim
 self.encoder = nn.Sequential(nn.Linear(784, 512), nn.ReLU(), nn.Linear(512, 256), nn.ReLU())
 self.fc_mu = nn.Linear(256, latent_dim)
 self.fc_log_sigma = nn.Linear(256, latent_dim)
 self.decoder = nn.Sequential(nn.Linear(latent_dim, 256), nn.ReLU(), nn.Linear(256, 512), nn.ReLU(), nn.Linear(512, 784), nn.Sigmoid())
 
 def encode(self, x):
 h = self.encoder(x)
 mu = self.fc_mu(h)
 log_sigma = self.fc_log_sigma(h)
 return mu, log_sigma
 
 def reparameterize(self, mu, log_sigma):
 std = torch.exp(0.5 * log_sigma)
 eps = torch.randn_like(std)
 return mu + eps * std
 
 def forward(self, x):
 mu, log_sigma = self.encode(x)
 z = self.reparameterize(mu, log_sigma)
 recon = self.decoder(z)
 return recon

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = VAE().to(device)
model.eval()

transform = transforms.Compose([transforms.ToTensor()])
mnist = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
loader = DataLoader(mnist, batch_size=100, shuffle=True)

mse_values = []
with torch.no_grad():
 for x, _ in loader:
 x = x.to(device)
 x_flat = x.view(-1, 784)
 recon = model(x_flat)
 mse = nn.functional.mse_loss(recon, x_flat, reduction='mean')
 mse_values.append(mse.item())
 if len(mse_values) >= 5:
 break

avg_mse = np.mean(mse_values)
print(f"Average reconstruction MSE: {avg_mse:.4f}")
assert avg_mse > 0, "MSE should be positive"
assert avg_mse < 1, "MSE should be reasonable"
print("✓ Reconstruction quality working")

if __name__ == "__main__":
 print("Lab 4: Reconstruction - PASSED")