diffusion models denoising score matching and generative modeling
# Diffusion Models: Denoising, Score Matching, and Generative Modeling
## 1. Introduction & Motivation
Diffusion models have emerged as a powerful class of generative models that iteratively denoise random noise to produce samples from a learned distribution. Unlike generative adversarial networks (GANs) requiring adversarial training or variational autoencoders (VAEs) requiring explicit likelihood bounds, diffusion models offer stable training through a principled objective: reversing a noise corruption process.
The core insight is to learn the reverse of a forward noise corruption process. The forward process gradually adds Gaussian noise to data, while the reverse process learns to remove that noise. By training a neural network to predict noise or score functions, diffusion models can generate high-quality samples that rival or exceed GANs and VAEs in sample quality and diversity.
Diffusion models have achieved state-of-the-art results in image generation (DDPM, Latent Diffusion), text generation, and conditional generation tasks. The framework's mathematical elegance, training stability, and interpretability have made it a dominant generative modeling paradigm.
This article provides comprehensive coverage of diffusion model theory, variants, training objectives, conditional generation, and practical implementation.
## 2. Core Concepts & Theory
### 2.1 Forward Diffusion Process
The forward process gradually corrupts data x_0 by adding Gaussian noise over T time steps:
$$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t I)$$
where beta_t in (0, 1) are predefined noise schedule parameters. This defines a Markov chain:
$$q(x_{0:T}) = q(x_0) \prod_{t=1}^{T} q(x_t | x_{t-1})$$
The schedule {beta_t} typically increases over time, with beta_1 < beta_2 < ... < beta_T .
### 2.2 Closed Form: Reparameterization
A key property is computing x_t directly from x_0 without iterating:
$$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$$
where alpha_t = 1 - beta_t , alpha_bar_t = prod(alpha_s) , and epsilon ~ N(0, I) .
This reparameterization enables efficient training by sampling arbitrary time steps.
### 2.3 Reverse Process
The reverse process p_theta learns to denoise:
$$p_ heta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_ heta(x_t, t), \sigma_t^2 I)$$
By the properties of conditional Gaussians, the reverse process is also Gaussian:
$$q(x_{t-1} | x_t, x_0) = \mathcal{N}(x_{t-1}; ilde{\mu}_t(x_t, x_0), ilde{\beta}_t I)$$
where:
$$ ilde{\mu}_t = \frac{\sqrt{1 - \beta_t} \cdot x_0 + \sqrt{\beta_t} x_t}{1 - \bar{\alpha}_t}$$
$$ ilde{\beta}_t = \frac{(1 - \bar{\alpha}_{t-1}) \beta_t}{1 - \bar{\alpha}_t}$$
### 2.4 Training Objective: Evidence Lower Bound
The negative log-likelihood is bounded by:
$$-\log p_ heta(x_0) \leq \mathbb{E}_q\left[D_{KL}(q(x_T | x_0) \| p(x_T)) + \sum_{t=2}^{T} D_{KL}(q(x_{t-1}|x_t, x_0) \| p_ heta(x_{t-1}|x_t)) - \log p_ heta(x_0|x_1) ight]$$
For Gaussian distributions, the KL divergence between two Gaussians is:
$$D_{KL}(\mathcal{N}(\mu_1, \Sigma_1) \| \mathcal{N}(\mu_2, \Sigma_2)) = \frac{1}{2}\left( ext{tr}(\Sigma_2^{-1}\Sigma_1) + (\mu_2 - \mu_1)^T \Sigma_2^{-1} (\mu_2 - \mu_1) - k + \log \frac{|\Sigma_2|}{|\Sigma_1|} ight)$$
### 2.5 Noise Prediction Objective
Rather than predicting the mean directly, it's common to predict the noise epsilon added at step t:
$$p_ heta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_ heta^{\epsilon}(x_t, t), \sigma_t^2 I)$$
where:
$$\mu_ heta^{\epsilon}(x_t, t) = \frac{x_t - \sqrt{\beta_t} \epsilon_ heta(x_t, t)}{\sqrt{1 - \beta_t}}$$
The training objective becomes:
$$\mathcal{L}_t = \mathbb{E}_{x_0, \epsilon} \left\|\epsilon - \epsilon_ heta(x_t, t) ight\|_2^2$$
## 3. Mathematical Formulation
### 3.1 Variance Schedule Design
Common schedules balance early and late denoising:
Linear schedule:
$$\beta_t = \beta_{\min} + \frac{t}{T}(\beta_{\max} - \beta_{\min})$$
Typically: beta_min = 0.0001, beta_max = 0.02.
Cosine schedule:
$$\bar{\alpha}_t = \frac{f(t)}{f(0)}, \quad f(t) = \cos\left(\frac{t/T + s}{1+s}\pi/2
ight)^2$$
where s is a small offset (e.g., 0.008). Cosine schedules often provide superior performance.
### 3.2 Continuous-Time Diffusion
Generalizing to continuous time, the process can be modeled as a stochastic differential equation (SDE):
$$dx_t = f(x_t, t) dt + g(t) dw_t$$
where f is drift, g is diffusion coefficient, and w_t is Brownian motion.
The score function nabla_x log p_t(x_t) satisfies:
$$p(x_0) = \int p(x_0 | x_t) p(x_t) dx_t$$
### 3.3 Score Matching
Rather than predicting noise, some formulations predict the score (gradient of log probability):
$$s_ heta(x_t, t) \approx abla_{x_t} \log p_t(x_t)$$
The score is related to noise prediction by:
$$s_ heta(x_t, t) = -\frac{\epsilon_ heta(x_t, t)}{\sqrt{1 - \bar{\alpha}_t}}$$
Score-based generative models learn this gradient directly, which can be more stable during training.
### 3.4 DDIM: Accelerated Sampling
Denoising Diffusion Implicit Models (DDIM) provide a non-Markovian reverse process:
$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \left(\frac{x_t - \sqrt{1-\bar{\alpha}_t} \epsilon_ heta(x_t, t)}{\sqrt{\bar{\alpha}_t}} ight) + \sqrt{1-\bar{\alpha}_{t-1} - \eta^2 \sigma_t^2} \epsilon_ heta(x_t, t) + \eta \sigma_t \epsilon$$
where eta is a parameter controlling stochasticity. Setting eta = 0 produces deterministic trajectories, enabling fewer sampling steps with acceptable quality loss.
## 4. Advanced Theory & Extensions
### 4.1 Latent Diffusion Models
Rather than diffusing high-dimensional pixel space, compress to latent space first:
$$z = E(x), \quad x = D(z)$$
where E and D are encoder and decoder. Diffuse in latent space:
$$p_ heta(z_0) = \int p_ heta(z_0 | z_T) p(z_T) dz_T$$
This reduces computational cost quadratically (e.g., 64x for 8x compression) while maintaining quality.
### 4.2 Classifier-Free Guidance
For conditional diffusion, guide generation toward class c without a separate classifier:
$$\epsilon_ heta^{ ext{guided}}(x_t, t, c) = \epsilon_ heta(x_t, t, \emptyset) + w \cdot (\epsilon_ heta(x_t, t, c) - \epsilon_ heta(x_t, t, \emptyset))$$
where w is a guidance scale. Higher w increases class adherence at cost of sample diversity.
### 4.3 Textual Inversion and Embeddings
Optimize learnable embeddings theta* to represent concepts:
$$\min_ heta \mathcal{L}(D(E( ext{concept})), D(E( ext{prompt}[ heta])))$$
Enables binding specific concepts to text token embeddings, allowing fine-grained control over generation.
### 4.4 LoRA Fine-Tuning for Diffusion
Low-rank adaptation for efficient diffusion fine-tuning:
$$\delta W = A B^T, \quad A \in \mathbb{R}^{d imes r}, B \in \mathbb{R}^{d imes r}$$
Update only low-rank matrices instead of full weight matrices, reducing parameters by 100x while maintaining quality.
## 5. Computational Considerations
### 5.1 Memory and Time Complexity
Training memory: Model weights O(model size), Activations O(T * B * d^2) for T timesteps, batch B, and dimension d, Gradient storage O(T * B * d^2). Typical GPU memory: 40-80GB for base diffusion models.
Sampling time: DDPM O(T) forward passes (1000 steps typical, ~20-50s), DDIM O(T/k) with k acceleration factor (50 steps, ~1-2s), Latent diffusion 10x speedup from lower resolution.
### 5.2 Gradient Checkpointing
Trade memory for computation by recomputing activations:
- Save checkpoints every sqrt(T) layers
- Recompute forward pass during backward
- Memory: O(sqrt(T)) instead of O(T)
- Computation: ~20% overhead
### 5.3 Mixed Precision Training
Use lower precision (16-bit) for forward/backward while maintaining 32-bit for loss computation:
- Memory reduction: ~50%
- Speed improvement: ~2-3x on modern GPUs
- Minimal quality loss with appropriate scaling
### 5.4 Inference Optimization
Quantization: 8-bit weight quantization reduces model size by 4x.
Caching: Precompute noise schedules and √ terms to avoid redundant computation.
Batched sampling: Process multiple samples simultaneously, amortizing overhead.
## 6. Practical Implementation Strategies
### 6.1 Noise Schedule Selection
Critical for performance. Cosine schedules generally outperform linear:
- Cosine: Better at both extremes (early and late denoising)
- Linear: Simpler, competitive but slightly lower quality
- Sigmoid: Steeper at extremes, more aggressive early denoising
Select based on dataset:
- High-frequency details: Faster schedule
- Low-frequency structure: Slower schedule
### 6.2 Model Architecture
UNet with residual blocks: Standard backbone
- Skip connections prevent gradient vanishing
- Multi-scale processing captures hierarchical structure
- Attention layers (often at low resolution) improve quality
Group normalization: Preferred over batch norm
- Batch size independent
- More stable with varying batch sizes
- Particularly beneficial for 32GB+ models
Sinusoidal time embeddings: Encode timestep t:
$$ ext{PE}(t, 2i) = \sin(t / 10000^{2i/d})$$
$$ ext{PE}(t, 2i+1) = \cos(t / 10000^{2i/d})$$
### 6.3 Conditional Generation
Class-conditional: Embed class label and inject into model:
$$\epsilon_ heta(x_t, t, ext{emb}(c))$$
Text-conditional: Encode text using CLIP or other encoder:
$$\epsilon_ heta(x_t, t, E_{ ext{text}}( ext{prompt}))$$
Attention-based conditioning: Cross-attention layers fuse conditioning:
$$ ext{Attn}(Q, K_{ ext{cond}}, V_{ ext{cond}})$$
### 6.4 Training Strategies
Weighting schemes: Weight loss by timestep to balance difficulty:
$$\mathcal{L} = \sum_{t=1}^{T} w_t \mathbb{E}[\|\epsilon - \epsilon_ heta(x_t, t)\|^2]$$
Common weights:
- Uniform: w_t = 1 (baseline)
- Minimum SNR:
$$ w_t = 1 / ext{SNR}(t) $$
(often best)
- Exponential: w_t = e^{-alpha t} EMA (Exponential Moving Average): Maintain shadow weights for more stable sampling:
$$ heta_{ ext{ema}} \leftarrow au heta_{ ext{ema}} + (1 - au) heta$$
With tau = 0.9999 typical.
## 7. Benchmark Datasets & Evaluation
### 7.1 Image Generation
CIFAR-10:
- Resolution: 32×32 pixels
- Classes: 10 (airplane, automobile, etc.)
- DDPM: FID 3.17 (state-of-the-art for diffusion)
- GAN baseline: FID 2.0-3.0
- Evaluation metric: Fréchet Inception Distance (FID)
CelebA-HQ:
- Resolution: 256×256
- Faces: 30K high-quality images
- Latent Diffusion: FID 5.08
- Baseline GAN: FID 3-5
- Evaluation: FID, IS (Inception Score)
ImageNet 256×256:
- 1000 classes, 1.3M images
- DDPM: FID 3.04
- Diffusion models very competitive or exceed GANs
- Metrics: FID, IS, precision/recall
### 7.2 Conditional Generation Benchmarks
Text-to-Image (COCO Captions):
- 123K train, 40K test captions
- Metrics: CLIP Score, FID-30K, Inception Score
- DDPM with guidance: CLIP ~0.28, FID ~15-20
- Modern (Stable Diffusion): CLIP ~0.30, FID ~10-12
SuperResolution (DIV2K):
- 800 training, 100 validation high-res images
- Metrics: PSNR, SSIM, LPIPS
- Diffusion-based SR: PSNR 30-32dB (vs. 28-30dB GANs)
### 7.3 Evaluation Metrics
Fréchet Inception Distance (FID):
$$ ext{FID} = \|\mu_r - \mu_g\|^2 + ext{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2})$$
Lower is better; measures distance between distributions of real and generated images.
Inception Score (IS):
$$IS = \exp(\mathbb{E}_x D_{KL}(p(y|x) \| p(y)))$$
Measures class confidence and diversity; standard metric though limited.
CLIP Score: Similarity between generated images and text prompts:
$$ ext{CLIP} = ext{cosine}( ext{CLIP}_{ ext{image}}, ext{CLIP}_{ ext{text}})$$
## 8. Key Challenges & Limitations
### 8.1 Slow Sampling
DDPM requires 1000 denoising steps (~50-100s), impractical for interactive applications:
$$ ext{Sample time} = O(T \cdot ext{forward pass time})$$
Solutions:
- DDIM reduces steps to 50 with minimal quality loss
- Latent diffusion speeds up 10x through compression
- Distillation techniques reduce steps further
### 8.2 Training Instability
Despite stability claims, large models can be unstable:
- Loss spikes at certain timesteps
- EMA weight divergence
- Training collapse with poor hyperparameters
Mitigation: Use EMA, careful learning rate schedules, monitor validation metrics frequently.
### 8.3 Sample Quality vs Diversity
There's a trade-off controlled by guidance scale in conditional generation:
- Low w: High diversity, lower adherence to condition
- High w: Exact condition adherence, lower diversity
- Optimal: w = 7-15 typically
### 8.4 Hallucination and Semantic Errors
Despite high FID scores, models can generate semantically incorrect images:
- Text-to-image: Failing to render specific objects from prompts
- Super-resolution: Introducing artifacts not in low-res input
- Conditional generation: Ignoring important aspects of condition
## 9. Hyperparameter Tuning & Optimization
### 9.1 Architecture Hyperparameters
Channel multipliers: [1, 2, 4, 4] typical for 4 resolution levels
- Controls capacity at each resolution
- Higher multipliers increase model size and quality
- Diminishing returns beyond 4 levels
Attention heads: 8-16 typical
- More heads don't necessarily improve quality
- Higher computational cost
- Standard: 8 heads at all resolutions
Number of residual blocks: 2-3 per resolution level
- Deeper networks capture more complex patterns
- Computational cost linear in depth
- 2 blocks: ~base performance, 3 blocks: ~5% improvement
### 9.2 Training Hyperparameters
Learning rate: eta = 1e-4 to 1e-3 typical
- Adam optimizer with beta_1 = 0.95 , beta_2 = 0.999 - Learning rate warmup: Linear increase over first 10K steps
- Cosine annealing decay over rest of training
Batch size: B = 128-512 typical
- Larger batches more stable, better FID
- Limited by GPU memory
- Gradient accumulation simulates larger batches
EMA decay: tau = 0.9999 standard
- Higher values (0.99995-0.99999) for better quality
- Slower weight updates
- Essential for FID improvement
### 9.3 Noise Schedule Hyperparameters
Cosine schedule offset: s = 0.008 (standard)
- Controls strength of early denoising
- Smaller s: More aggressive early denoising
- Larger s: More linear schedule
Max beta : Typically 0.02 for cosine schedule
- Controls noise at final step
- beta_max = 0.02 standard
- Higher values add more noise (more aggressive)
## 10. Real-World Applications & Case Studies
### 10.1 High-Resolution Image Generation
Problem: Generate 512×512 high-quality images from text descriptions
Architecture:
- Stable Diffusion (latent diffusion)
- Base: 4x downsampled latent space
- UNet: 4 resolution levels, 4 residual blocks each
- Attention: 8 heads at 32×32 and lower resolution
Results:
- CLIP score: 0.305 (strong text alignment)
- User preference: 80% vs. Midjourney (comparable quality)
- Speed: 5-8 seconds per image on A100 GPU
Key techniques:
- Latent diffusion reduces 512×512 → 128×128 computation
- Classifier-free guidance (w=7.5) balances quality and adherence
- CLIP embeddings as text encoding
### 10.2 Super-Resolution
Problem: Upscale 64×64 low-resolution images to 256×256
Architecture:
- Conditional latent diffusion
- Input: Low-res image + bicubic interpolation
- Conditioning: Image encoder (first 3 conv blocks of ResNet)
- 5-step DDIM sampling
Results:
- PSNR: 31.2 dB (vs. 28.5 dB bicubic baseline)
- SSIM: 0.908 (vs. 0.85 bicubic)
- LPIPS: 0.12 (perceptual quality, vs. 0.25 baseline)
Practical considerations:
- Training: 500K images, 4 A100s, 72 hours
- Inference: 0.5-1 second per image (5 DDIM steps)
### 10.3 Inpainting and Editing
Problem: Fill missing regions while preserving context
Architecture:
- Masked input: Known regions concatenated with original
- Progressive inpainting: Denoise from noise, only modifying masked region
- Cross-attention: Condition on text describing desired inpainting
Results:
- User study: 4.2/5.0 consistency with known regions
- 3.8/5.0 naturalness of generated content
- Inference: 2-3 seconds
Key insight: Masking mechanism prevents overwriting known regions during denoising.
### 10.4 Text-to-3D Generation
Problem: Generate 3D shapes from text descriptions
Architecture:
- Score distillation sampling (SDS) loss
- Teacher: Pre-trained text-to-image diffusion model
- Student: Neural radiance field (NeRF) or mesh decoder
- Optimization: Gradient flow from teacher diffusion model
Results:
- Visual quality: 4.0/5.0 user ratings
- Semantic alignment: 82% align with text description
- Generation time: 10-30 minutes (computationally expensive)
Challenges:
- Slow optimization loop (many teacher forward passes)
- Limited geometric accuracy
- Ambiguous text descriptions produce inconsistent shapes
## 11. Integration with Other Methods
### 11.1 Combining with Pre-trained Encoders
Use CLIP or other vision-language models for conditioning:
$$\epsilon_ heta(x_t, t, ext{CLIP}( ext{text}))$$
Benefits:
- Better text-image alignment
- Leverages large-scale pretraining
- Enables zero-shot generation for unseen concepts
### 11.2 Diffusion for Adaptation and Editing
Iterative refinement loop:
1. Generate initial sample with diffusion
2. Evaluate with auxiliary model (e.g., classifier)
3. Adjust conditioning and regenerate
Used for adversarial robustness, style transfer, and domain adaptation.
### 11.3 Sequential Diffusion Models for Language
Apply diffusion framework to text:
$$q(x_t^{[i]} | x_t^{[i-1]}) = ext{Categorical}(x_t^{[i]} | p_t)$$
Diffusion-based language models competitive with autoregressive approaches.
### 11.4 Hybrid with VAE
Combine VAE's structured latent with diffusion's generative quality:
- VAE encodes to latent, diffusion learns latent distribution
- Often outperforms pure diffusion or VAE alone
## 12. Future Research Directions
### 12.1 Faster Sampling
Active research into:
- Distillation: Train student network for fewer steps
- Latent prediction: Predict final sample directly
- Consistency models: Learning teacher-student consistency
Target: Real-time generation (< 100ms) while maintaining quality.
### 12.2 Improved Guidance
Current guidance sometimes fails (e.g., multi-object scenes):
- Better guidance mechanisms for complex conditions
- Compositional guidance combining multiple concepts
- Adaptive guidance strength per image region
### 12.3 Interpretability
Limited understanding of what diffusion models learn:
- Probe representations at different timesteps
- Analyze attention patterns
- Explain failure modes (e.g., wrong number of objects)
### 12.4 Scaling Laws
Understand how diffusion model capabilities scale with:
- Model size (parameters)
- Data scale (training samples)
- Compute budget (training steps)
Develop efficient scaling strategies similar to transformer scaling laws.
## 13. Summary & Key Takeaways
Core Algorithm:
- Forward process: Gradually add noise to data over T steps
- Reverse process: Train network to predict noise and denoise iteratively
- Training: Minimize predicted vs actual noise, unbiased estimator of ELBO
Key Mathematical Insights:
- Noise prediction equivalent to score matching for Gaussian noise
- Closed-form x_t from x_0 enables efficient training
- Variance schedule critical for balancing early/late denoising
Practical Performance:
- DDPM: FID 3.17 on CIFAR-10, competitive with GANs
- Latent diffusion: 10x faster, enables high-resolution generation
- CLIP guidance: Enables text-conditional generation with high quality
Noise Schedules:
- Cosine generally outperforms linear
- Schedule sets difficulty curve: early vs late timesteps
- Weighting scheme (uniform vs min-SNR) affects convergence
Conditional Generation:
- Classifier-free guidance: Drop conditioning with probability 10-50%, enables guidance at test time
- Guidance scale w = 7-15 typical trade-off between quality and adherence
- Text conditioning via CLIP or similar encoders
Computational Efficiency:
- DDPM sampling: 1000 steps, 50-100s per image
- DDIM: 50 steps, 2-5s per image (10-20x speedup)
- Latent diffusion: Additional 10x speedup through compression
Main Limitations:
- Sampling still slower than GANs (though improving)
- Occasional semantic errors despite high FID
- Large memory footprint for high-resolution models
- Training can be unstable without careful tuning
Current Status:
Diffusion models have become the dominant generative modeling approach, achieving state-of-the-art results across image generation, conditional generation, and text-to-image tasks. Continued research focuses on faster sampling, improved quality at extreme resolutions, and better understanding of model behavior.
---
## Appendix: Practical Implementation Labs
### Lab 1: Gaussian Diffusion Forward Process
import torch
import torch.nn as nn
import numpy as np
class GaussianDiffusion:
def __init__(self, timesteps=1000, beta_start=0.0001, beta_end=0.02):
self.timesteps = timesteps
# Linear schedule (can be improved with cosine)
betas = torch.linspace(beta_start, beta_end, timesteps)
alphas = 1.0 - betas
alphas_cumprod = torch.cumprod(alphas, dim=0)
self.register_buffer('betas', betas)
self.register_buffer('alphas', alphas)
self.register_buffer('alphas_cumprod', alphas_cumprod)
self.register_buffer('sqrt_alphas_cumprod',
torch.sqrt(alphas_cumprod))
self.register_buffer('sqrt_one_minus_alphas_cumprod',
torch.sqrt(1.0 - alphas_cumprod))
def register_buffer(self, name, tensor):
setattr(self, name, tensor)
def q_sample(self, x_0, t, noise):
"""Sample x_t from q(x_t | x_0)"""
sqrt_alphas = self.sqrt_alphas_cumprod[t]
sqrt_one_minus_alphas = self.sqrt_one_minus_alphas_cumprod[t]
# Reshape for broadcasting
while len(sqrt_alphas.shape) < len(x_0.shape):
sqrt_alphas = sqrt_alphas.unsqueeze(-1)
sqrt_one_minus_alphas = sqrt_one_minus_alphas.unsqueeze(-1)
return sqrt_alphas * x_0 + sqrt_one_minus_alphas * noise
def q_posterior(self, x_0, x_t, t):
"""Posterior mean and variance q(x_{t-1} | x_t, x_0)"""
posterior_mean_coeff1 = (1 - self.alphas_cumprod[t-1]) / (1 - self.alphas_cumprod[t])
posterior_mean_coeff2 = self.betas[t] * torch.sqrt(self.alphas_cumprod[t-1]) / (1 - self.alphas_cumprod[t])
posterior_mean = posterior_mean_coeff1 * x_t + posterior_mean_coeff2 * x_0
posterior_variance = self.betas[t] * (1 - self.alphas_cumprod[t-1]) / (1 - self.alphas_cumprod[t])
return posterior_mean, posterior_variance
# Test
diffusion = GaussianDiffusion(timesteps=1000)
x_0 = torch.randn(4, 3, 32, 32)
t = torch.tensor([10, 100, 500, 999])
noise = torch.randn_like(x_0)
x_t = diffusion.q_sample(x_0, t, noise)
print(f"x_t shape: {x_t.shape}")### Lab 2: Simple UNet Denoising Model
import torch
import torch.nn as nn
class ResBlock(nn.Module):
def __init__(self, in_channels, out_channels, time_channels):
super().__init__()
self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)
self.time_emb = nn.Linear(time_channels, out_channels)
self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)
self.skip = nn.Conv2d(in_channels, out_channels, kernel_size=1) if in_channels != out_channels else nn.Identity()
self.norm1 = nn.GroupNorm(32, in_channels)
self.norm2 = nn.GroupNorm(32, out_channels)
def forward(self, x, t_emb):
h = self.norm1(x)
h = torch.relu(h)
h = self.conv1(h)
h = h + self.time_emb(t_emb)[:, :, None, None]
h = self.norm2(h)
h = torch.relu(h)
h = self.conv2(h)
return h + self.skip(x)
class SimpleUNet(nn.Module):
def __init__(self, channels=3, time_channels=128):
super().__init__()
self.time_emb = nn.Sequential(
nn.Linear(1, time_channels),
nn.RELU(),
nn.Linear(time_channels, time_channels)
)
self.down1 = nn.Sequential(
nn.Conv2d(channels, 64, 3, padding=1),
ResBlock(64, 64, time_channels)
)
self.down2 = nn.Sequential(
nn.MaxPool2d(2),
ResBlock(64, 128, time_channels)
)
self.middle = ResBlock(128, 128, time_channels)
self.up1 = nn.Sequential(
nn.Upsample(scale_factor=2),
ResBlock(128, 64, time_channels)
)
self.out = nn.Conv2d(64, channels, 3, padding=1)
def forward(self, x, t):
t_emb = self.time_emb(t.unsqueeze(-1).float())
h1 = self.down1(x)
h2 = self.down2(h1)
h = self.middle(h2, t_emb)
h = self.up1(h)
return self.out(h)
# Test
model = SimpleUNet(channels=3)
x = torch.randn(4, 3, 32, 32)
t = torch.randint(0, 1000, (4,))
output = model(x, t)
print(f"Output shape: {output.shape}")### Lab 3: Training Loop with Diffusion
import torch
import torch.optim as optim
from torch.utils.data import DataLoader
def train_diffusion(model, train_loader, num_epochs, device):
diffusion = GaussianDiffusion(timesteps=1000)
optimizer = optim.Adam(model.parameters(), lr=1e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs)
model.to(device)
for epoch in range(num_epochs):
epoch_loss = 0
for batch_idx, (images, _) in enumerate(train_loader):
images = images.to(device)
batch_size = images.shape[0]
# Sample random timesteps
t = torch.randint(0, 1000, (batch_size,), device=device)
# Sample noise
noise = torch.randn_like(images)
# Forward diffusion
x_t = diffusion.q_sample(images, t, noise)
# Predict noise
predicted_noise = model(x_t, t)
# MSE loss
loss = torch.mean((noise - predicted_noise) ** 2)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
epoch_loss += loss.item()
if batch_idx % 100 == 0:
print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}")
scheduler.step()
print(f"Epoch {epoch} Average Loss: {epoch_loss / len(train_loader):.4f}")
# Usage:
# train_diffusion(model, train_loader, num_epochs=10, device='cuda')### Lab 4: DDIM Sampling (Accelerated)
import torch
def ddim_sample(model, diffusion, x_T, num_steps=50, eta=0.0):
"""DDIM sampling with acceleration"""
device = x_T.device
batch_size = x_T.shape[0]
# Compute timestep schedule
timesteps = torch.linspace(0, 999, num_steps + 1, device=device).long()
timesteps = timesteps[::-1] # Reverse
x = x_T
for i, t in enumerate(timesteps[:-1]):
t_next = timesteps[i + 1]
# Get current alphas
alpha_t = diffusion.alphas_cumprod[t]
alpha_next = diffusion.alphas_cumprod[t_next]
# Predict noise
with torch.no_grad():
noise_pred = model(x, t.unsqueeze(0).expand(batch_size))
# DDIM update
sigma_t = eta * torch.sqrt((1 - alpha_next) / (1 - alpha_t) *
(1 - alpha_t / alpha_next))
x_0_pred = (x - torch.sqrt(1 - alpha_t) * noise_pred) / torch.sqrt(alpha_t)
x = (torch.sqrt(alpha_next) * x_0_pred +
torch.sqrt(1 - alpha_next - sigma_t ** 2) * noise_pred +
sigma_t * torch.randn_like(x))
return x
# Test
x_T = torch.randn(4, 3, 32, 32)
samples = ddim_sample(model, diffusion, x_T, num_steps=50)
print(f"Generated samples shape: {samples.shape}")