Autoencoders Variational Autoencoders Unsupervised Representation Learning
# Autoencoders & Variational Autoencoders: Unsupervised Representation Learning
## Introduction & Motivation
Autoencoders: encode input → latent → decode reconstruction. Bottleneck: learns compressed representation. Variational autoencoders: probabilistic latent space; sample generative model. Reparameterization trick: backprop through sampling. Applications: dimensionality reduction, generative modeling, anomaly detection.
Motivation: Unsupervised: learn meaningful representations without labels. VAE: generative model; sample novel instances.
Applications: Generative modeling, dimensionality reduction, anomaly detection.
---
## Core Concepts & Theory
### Autoencoder
Encoder: x → z; Decoder: z → x. Bottleneck forces compression.
### Variational Autoencoder
Probabilistic latent; sample from learned distribution.
### Reparameterization
z = μ + σ ⊙ ε, ε ~ N(0, I); enables backprop.
---
## Mathematical Formulation
Autoencoder loss:
$$L = \|x - \hat{x}\|^2 + \lambda \|z\|^2$$
reconstruction + regularization.
VAE loss (ELBO):
$$L = -\mathbb{E}_q[\log p(x|z)] + D_{ ext{KL}}(q(z|x) \| p(z))$$
reconstruction + KL divergence (latent regularization).
Reparameterization:
$$z = \mu(x) + \sigma(x) \odot \mathcal{N}(0, I)$$
---
## Advanced Theory & Extensions
### Beta-VAE
Weight KL term; control disentanglement.
### Ladder VAE
Hierarchical structure; multiple latent levels.
### Adversarial Autoencoders
GAN + autoencoder; adversarial latent distribution matching.
---
## Computational Considerations
Autoencoder: O(encoder + decoder) forward/backward.
VAE: O(encoder + decoder + KL) per batch.
Sampling: O(z_dim) for reparameterization.
---
## Practical Implementation Strategies
### Architecture
Symmetric encoder/decoder; bottleneck in middle.
### Loss Weighting
Balance reconstruction and KL; empirical tuning.
### Latent Dimension
Control compression; 2-3× smaller than input typical.
---
## Benchmark Datasets & Evaluation
MNIST: Standard; easy reconstruction.
CelebA: Faces; generative quality assessment.
Anomaly Detection: Reconstruction error threshold.
---
## Key Challenges & Limitations
### Blurry Reconstructions
MSE loss encourages averaging; perceptual loss helps.
### KL Annealing
Cold start KL; gradually increase weight.
### Posterior Collapse
VAE ignores latent; KL → 0; increase β.
---
## Hyperparameter Tuning
Latent dimension: Input_dim / 4-8; empirical.
β (VAE): 0.001-1.0; control disentanglement.
Learning rate: 1e-3 standard; decay over time.
---
## Real-World Applications & Case Studies
Image Generation: CelebA faces; interpolation in latent space.
Anomaly Detection: High reconstruction error = anomaly.
Dimensionality Reduction: Alternative to PCA.
---
## Integration with Other Methods
VAE + GAN → adversarial VAE.
VAE + Reinforcement Learning → variational policy.
---
## Summary & Key Takeaways
Autoencoders and variational autoencoders learn compressed representations via bottleneck architecture and probabilistic latent distributions, enabling generative modeling and unsupervised learning.
Principles:
1. Autoencoder: encoder-decoder with bottleneck.
2. VAE: probabilistic latent; sample generative model.
3. Reparameterization: enable gradient flow through sampling.
4. KL divergence: regularize latent distribution.
5. Reconstruction loss: balance with latent regularization.
---
---
## Appendix: Practical Labs
### Lab 1: Autoencoder Architecture
import torch
import torch.nn as nn
import numpy as np
class Autoencoder(nn.Module):
def __init__(self, input_dim=784, latent_dim=20):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, 256),
nn.ReLU(),
nn.Linear(256, latent_dim)
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 256),
nn.ReLU(),
nn.Linear(256, input_dim),
nn.Sigmoid()
)
def forward(self, x):
z = self.encoder(x)
recon = self.decoder(z)
return recon, z
# Test
np.random.seed(42)
model = Autoencoder(input_dim=784, latent_dim=20)
x = torch.randn(32, 784)
recon, z = model(x)
assert recon.shape == x.shape, "Reconstruction shape matches input"
assert z.shape == (32, 20), "Latent shape correct"
print("✓ Autoencoder architecture working")
if __name__ == "__main__":
print("Lab 1: Autoencoder - PASSED")### Lab 2: VAE with Reparameterization
import torch
import torch.nn as nn
import numpy as np
class VariationalAutoencoder(nn.Module):
def __init__(self, input_dim=784, latent_dim=20):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, 256),
nn.ReLU(),
nn.Linear(256, latent_dim * 2)
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 256),
nn.ReLU(),
nn.Linear(256, input_dim),
nn.Sigmoid()
)
self.latent_dim = latent_dim
def forward(self, x):
# Encode
h = self.encoder(x)
mu, logvar = h[:, :self.latent_dim], h[:, self.latent_dim:]
# Reparameterize
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std)
z = mu + eps * std
# Decode
recon = self.decoder(z)
return recon, mu, logvar
# Test
np.random.seed(42)
model = VariationalAutoencoder(input_dim=784, latent_dim=20)
x = torch.randn(32, 784)
recon, mu, logvar = model(x)
assert recon.shape == x.shape, "Reconstruction shape matches"
assert mu.shape == (32, 20), "Mu shape correct"
assert logvar.shape == (32, 20), "Logvar shape correct"
print("✓ VAE working")
if __name__ == "__main__":
print("Lab 2: VAE - PASSED")### Lab 3: ELBO Loss (VAE Loss)
import torch
import torch.nn.functional as F
import numpy as np
def vae_loss(recon, x, mu, logvar, beta=1.0):
"""Compute VAE loss (ELBO)"""
# Reconstruction loss
recon_loss = F.mse_loss(recon, x, reduction='mean')
# KL divergence: D_KL(q || p)
kl_loss = -0.5 * torch.mean(1 + logvar - mu.pow(2) - logvar.exp())
# Total loss
loss = recon_loss + beta * kl_loss
return loss
# Test
np.random.seed(42)
recon = torch.rand(32, 784)
x = torch.rand(32, 784)
mu = torch.randn(32, 20)
logvar = torch.randn(32, 20)
loss = vae_loss(recon, x, mu, logvar, beta=1.0)
assert torch.isfinite(loss), "Loss should be finite"
assert loss > 0, "Loss should be positive"
print("✓ VAE loss working")
if __name__ == "__main__":
print("Lab 3: Loss - PASSED")### Lab 4: Generation and Reconstruction
import torch
import numpy as np
def generate_samples(model, num_samples=10, latent_dim=20):
"""Generate samples from VAE"""
with torch.no_grad():
z = torch.randn(num_samples, latent_dim)
samples = model.decoder(z)
return samples
def compute_reconstruction_error(model, x):
"""Compute per-sample reconstruction error"""
with torch.no_grad():
recon, _, _ = model(x)
error = torch.mean((recon - x) ** 2, dim=1)
return error
# Test
np.random.seed(42)
class DummyVAE:
def __init__(self, latent_dim=20):
self.latent_dim = latent_dim
self.decoder = lambda z: torch.rand(z.shape[0], 784)
model = DummyVAE(latent_dim=20)
x = torch.randn(32, 784)
samples = generate_samples(model, num_samples=10)
assert samples.shape == (10, 784), "Sample shape correct"
print("✓ Generation working")
if __name__ == "__main__":
print("Lab 4: Generation - PASSED")