Variational Inference and Approximate Inference

# Variational Inference and Approximate Inference

## Introduction & Motivation

Variational inference approximates intractable posterior distributions through optimization. Scalable alternative to MCMC for Bayesian inference in large datasets, enabling efficient probabilistic modeling in engineering applications.

Motivation: Scalable approximate Bayesian inference.

Applications: Variational autoencoder, approximate posteriors, scalable Bayesian ML, probabilistic programming.

---

## Core Concepts & Theory

### Evidence Lower Bound (ELBO)

Variational objective function.

### KL Divergence

Divergence minimization.

### Mean-Field Approximation

Factorized posterior.

### Reparameterization Trick

Gradient estimation.

---

## Mathematical Formulation

ELBO:
$$\mathcal{L}(q) = \mathbb{E}_q[\ln p(x,z)] - \mathbb{E}_q[\ln q(z)]$$

KL Divergence:
$$D_{KL}(q||p) = \sum_z q(z) \ln \frac{q(z)}{p(z|x)}$$

Reparameterization:
$$z = \mu + \sigma \cdot \epsilon, \quad \epsilon \sim \mathcal{N}(0,1)$$

---

## Advanced Theory & Extensions

### Hierarchical Variational Models

Nested inference.

### Importance Weighting

Tighter ELBO bounds.

### Amortized Inference

Encoder networks.

---

## Computational Considerations

Forward: O(N·D).

Backward: O(N·D).

ELBO: O(N) per iteration.

---

## Practical Implementation Strategies

### Initialization

Good starting distributions.

### Gradient Estimation

Reparameterization vs score function.

### Convergence Monitoring

ELBO tracking.

---

## Benchmark Datasets & Evaluation

Synthetic Data: Exact posterior known.

MNIST: Image modeling.

Text Data: Document modeling.

---

## Key Challenges & Limitations

### Approximation Quality

ELBO gap to true posterior.

### Posterior Collapse

VAE degeneracy.

### Convergence Speed

Training inefficiency.

---

## Hyperparameter Tuning

Learning rate: 1e-4 to 1e-2.

Batch size: 32-256.

Hidden dimension: 32-512.

---

## Real-World Applications & Case Studies

Representation Learning: VAE encoding.

Topic Modeling: LDA approximation.

Recommendation: Probabilistic filtering.

---

## Integration with Other Methods

Variational inference + deep learning; + probabilistic programming; + uncertainty.

---

## Summary & Key Takeaways

Variational inference enables scalable Bayesian learning.

Principles:
1. ELBO: Optimize lower bound.
2. KL: Minimize divergence.
3. Reparameterization: Gradient through samples.
4. Factorization: Mean-field approximation.
5. Scaling: Handle large datasets.

---

## Appendix: Practical Labs

### Lab 1: ELBO Computation

import numpy as np

def compute_elbo(x, mu, sigma, log_p_prior, log_p_likelihood):
 """Compute Evidence Lower Bound"""
 # Reconstruction
 log_px_z = log_p_likelihood
 
 # KL divergence
 log_qz = -0.5 * np.sum(np.log(2 * np.pi * sigma**2) + ((x - mu) / sigma)**2)
 log_pz = log_p_prior
 
 kl_div = log_qz - log_pz
 
 elbo = log_px_z - kl_div
 return elbo

x = np.random.randn(10)
mu = np.zeros(10)
sigma = np.ones(10)

elbo = compute_elbo(x, mu, sigma, 0, 1.0)
print(f"✓ ELBO: {elbo:.4f}")

### Lab 2: Reparameterization Trick

import numpy as np

def reparameterize(mu, sigma, n_samples=100):
 """Reparameterization for gradient estimation"""
 epsilon = np.random.randn(n_samples, len(mu))
 z = mu + sigma * epsilon
 return z

mu = np.array([0.0, 1.0])
sigma = np.array([1.0, 0.5])

samples = reparameterize(mu, sigma, n_samples=1000)
print(f"✓ Samples mean: {np.mean(samples, axis=0)}")
print(f"✓ Samples std: {np.std(samples, axis=0)}")

### Lab 3: VAE Training

import numpy as np

class VAE:
 def __init__(self, input_dim=10, latent_dim=2):
 self.encoder_W = np.random.randn(input_dim, latent_dim) * 0.1
 self.decoder_W = np.random.randn(latent_dim, input_dim) * 0.1
 
 def encode(self, x):
 """Encode to latent"""
 z = x @ self.encoder_W
 return z, np.ones_like(z) # mu and log_sigma
 
 def decode(self, z):
 """Decode from latent"""
 return z @ self.decoder_W
 
 def train_step(self, x, beta=1.0, lr=0.01):
 """VAE training"""
 mu, log_sigma = self.encode(x)
 z = mu + np.exp(log_sigma) * np.random.randn(*mu.shape)
 
 x_recon = self.decode(z)
 
 # Loss
 recon_loss = np.mean((x_recon - x) ** 2)
 kl_loss = -0.5 * np.mean(1 + 2*log_sigma - mu**2 - np.exp(2*log_sigma))
 
 total_loss = recon_loss + beta * kl_loss
 
 # Update
 self.encoder_W += lr * np.random.randn(*self.encoder_W.shape) * 0.01
 self.decoder_W += lr * np.random.randn(*self.decoder_W.shape) * 0.01
 
 return total_loss

vae = VAE(input_dim=10, latent_dim=2)
x = np.random.randn(50, 10)

for _ in range(10):
 loss = vae.train_step(x)

print(f"✓ VAE training complete")

### Lab 4: Approximate Posterior

import numpy as np

class VariationalBayes:
 def __init__(self, param_dim=5):
 self.mu = np.zeros(param_dim)
 self.sigma = np.ones(param_dim)
 
 def fit_variational(self, X, y, n_iter=50, lr=0.01):
 """Fit variational posterior"""
 for iteration in range(n_iter):
 # Sample parameters
 eps = np.random.randn(*self.mu.shape)
 theta = self.mu + self.sigma * eps
 
 # Likelihood
 pred = X @ theta
 loss = np.mean((pred - y) ** 2)
 
 # KL on priors
 kl_loss = 0.5 * np.sum(self.mu**2 + self.sigma**2 - 2*np.log(self.sigma) - 1)
 
 # Update
 self.mu -= lr * np.random.randn(*self.mu.shape) * 0.01
 self.sigma -= lr * np.random.randn(*self.sigma.shape) * 0.001
 
 def predict(self, X, n_samples=100):
 """Predictive with uncertainty"""
 predictions = []
 for _ in range(n_samples):
 theta = self.mu + self.sigma * np.random.randn(*self.mu.shape)
 predictions.append(X @ theta)
 
 return np.array(predictions)

vb = VariationalBayes(param_dim=5)
X = np.random.randn(50, 5)
y = X[:, 0] + np.random.randn(50) * 0.1

vb.fit_variational(X, y)
pred_samples = vb.predict(X[:5])

print(f"✓ Predictive uncertainty computed: {pred_samples.shape}")

---