Autoencoders Deep Unsupervised Learning Representation Learning
# Autoencoders: Deep Unsupervised Learning & Representation Learning
## Introduction & Motivation
Autoencoders learn data representations via reconstruction—encode high-dimensional data into low-dimensional latent space, decode back to original space. Unsupervised dimensionality reduction with non-linear mappings (via deep layers). Foundation for variational autoencoders, generative models, anomaly detection.
Motivation: Linear methods (PCA) insufficient for complex data. Autoencoders capture non-linear structure via deep neural networks. No labels required; reconstruction loss guides learning.
Applications: Dimensionality reduction, denoising, image generation, anomaly detection, feature learning.
---
## Core Concepts & Theory
### Architecture
Encoder: \mathbf{z} = f_{ ext{enc}}(\mathbf{x}) compresses to latent code.
Decoder: \hat{\mathbf{x}} = f_{ ext{dec}}(\mathbf{z}) reconstructs from latent.
Loss: \mathcal{L} = \|\mathbf{x} - \hat{\mathbf{x}}\|^2 (MSE reconstruction).
---
## Mathematical Formulation
Bottleneck:
$$\mathbf{z} = f_{ ext{enc}}(\mathbf{x}) \in \mathbb{R}^k, \quad k \ll d$$
Reconstruction:
$$\hat{\mathbf{x}} = f_{ ext{dec}}(\mathbf{z})$$
Objective:
$$\min_{ heta} \mathbb{E}[\|\mathbf{x} - f_{ ext{dec}}(f_{ ext{enc}}(\mathbf{x}))\|^2]$$
---
## Advanced Theory & Extensions
### Sparse Autoencoders
L1 penalty on latent code; encourages sparsity (few active units).
### Denoising Autoencoders
Input corrupted; reconstruct clean—learns robust features.
### Variational Autoencoders (VAE)
Probabilistic framework; latent distribution N(0,I); generative model.
---
## Computational Considerations
Training: O(n imes d imes ext{iterations}) via SGD.
Inference (encoding): O(d imes k imes L) for L layers.
---
## Practical Implementation Strategies
### Latent Dimension
k = d/2 to d/10 typical; cross-validation guides choice.
### Architecture Symmetry
Encoder and decoder mirror structure for balance.
### Regularization
Dropout, L2 weight decay prevent overfitting.
---
## Benchmark Datasets & Evaluation
MNIST: Digits; reconstruction quality visual.
CelebA: Faces; disentangled representation evaluation.
---
## Key Challenges & Limitations
### Reconstruction vs. Representation
Balancing perfect reconstruction vs. meaningful latent code.
### Training Instability
Deep networks require careful learning rate, batch normalization.
---
## Hyperparameter Tuning
Latent dim \in {10, 32, 64\}, learning_rate \in {0.001, 0.01\}, batch_size \in {32, 64, 128\}.
---
## Real-World Applications & Case Studies
Image Denoising: Trained on clean images; denoise via reconstruction.
Anomaly Detection: Reconstruction error indicates anomalies.
---
## Integration with Other Methods
Autoencoder + Classification Head → Semi-supervised learning.
---
## Future Research Directions
Disentangled representations; hierarchical autoencoders; adversarial robustness.
---
## Summary & Key Takeaways
Autoencoders learn non-linear representations via reconstruction, discovering compressed latent codes suitable for downstream tasks.
Principles:
1. Encoder compresses data to latent bottleneck.
2. Decoder reconstructs from latent code.
3. Reconstruction loss drives unsupervised learning.
4. Bottleneck width controls compression.
5. Extensions (VAE, denoising) enable diverse applications.
---
---
## Appendix: Practical Labs
### Lab 1: Basic Autoencoder
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
X, _ = load_digits(return_X_y=True)
X = StandardScaler().fit_transform(X).astype('float32')
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
class Autoencoder(nn.Module):
def __init__(self):
super().__init__()
self.encoder = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 16))
self.decoder = nn.Sequential(nn.Linear(16, 32), nn.ReLU(), nn.Linear(32, 64))
def forward(self, x):
z = self.encoder(x)
return self.decoder(z)
model = Autoencoder()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.MSELoss()
X_train_t = torch.FloatTensor(X_train)
for epoch in range(20):
opt.zero_grad()
recon = model(X_train_t)
loss = loss_fn(recon, X_train_t)
loss.backward()
opt.step()
print(f"Final loss: {loss.item():.4f}")
assert loss.item() < 5.0, "Loss should decrease"
print("✓ Autoencoder working")
if __name__ == "__main__":
print("Lab 1: Autoencoder - PASSED")### Lab 2: Latent Representations
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True)
X = StandardScaler().fit_transform(X).astype('float32')
class Autoencoder(nn.Module):
def __init__(self):
super().__init__()
self.encoder = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 8))
self.decoder = nn.Sequential(nn.Linear(8, 32), nn.ReLU(), nn.Linear(32, 64))
def encode(self, x):
return self.encoder(x)
def forward(self, x):
return self.decoder(self.encode(x))
model = Autoencoder()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
X_t = torch.FloatTensor(X)
for epoch in range(20):
opt.zero_grad()
loss = nn.MSELoss()(model(X_t), X_t)
loss.backward()
opt.step()
# Extract latent codes
z = model.encode(X_t).detach().numpy()
print(f"Latent code shape: {z.shape}")
assert z.shape == (1797, 8), "Should have 8-D latent"
print("✓ Latent representation working")
if __name__ == "__main__":
print("Lab 2: Latent Representation - PASSED")### Lab 3: Reconstruction Quality
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
X, _ = load_digits(return_X_y=True)
X = StandardScaler().fit_transform(X).astype('float32')
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
class Autoencoder(nn.Module):
def __init__(self):
super().__init__()
self.encoder = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 16))
self.decoder = nn.Sequential(nn.Linear(16, 32), nn.ReLU(), nn.Linear(32, 64))
def forward(self, x):
return self.decoder(self.encoder(x))
model = Autoencoder()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
X_train_t = torch.FloatTensor(X_train)
X_test_t = torch.FloatTensor(X_test)
for epoch in range(20):
opt.zero_grad()
loss = nn.MSELoss()(model(X_train_t), X_train_t)
loss.backward()
opt.step()
with torch.no_grad():
test_loss = nn.MSELoss()(model(X_test_t), X_test_t).item()
print(f"Test reconstruction MSE: {test_loss:.4f}")
assert test_loss > 0 and test_loss < 5, "Should have reasonable reconstruction"
print("✓ Reconstruction quality working")
if __name__ == "__main__":
print("Lab 3: Reconstruction Quality - PASSED")### Lab 4: Latent Dimension Effect
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
X, _ = load_digits(return_X_y=True)
X = StandardScaler().fit_transform(X).astype('float32')
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
X_train_t = torch.FloatTensor(X_train)
X_test_t = torch.FloatTensor(X_test)
latent_dims = [4, 8, 16, 32]
for z_dim in latent_dims:
class Autoencoder(nn.Module):
def __init__(self, z_dim):
super().__init__()
self.encoder = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, z_dim))
self.decoder = nn.Sequential(nn.Linear(z_dim, 32), nn.ReLU(), nn.Linear(32, 64))
def forward(self, x):
return self.decoder(self.encoder(x))
model = Autoencoder(z_dim)
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(20):
opt.zero_grad()
loss = nn.MSELoss()(model(X_train_t), X_train_t)
loss.backward()
opt.step()
with torch.no_grad():
test_loss = nn.MSELoss()(model(X_test_t), X_test_t).item()
print(f"Latent dim {z_dim}: Test MSE {test_loss:.4f}")
print("✓ Latent dimension effect working")
if __name__ == "__main__":
print("Lab 4: Latent Dimension - PASSED")