diffusion models denoising ddpm
# Diffusion Models: Denoising & DDPM
## Introduction & Motivation
Diffusion models: iterative denoising process. DDPM: Denoising Diffusion Probabilistic Models. Forward: add noise to data. Reverse: learn denoising network. Applications: image generation, audio synthesis, molecular design.
Motivation: Stable training alternative to GANs. Powerful generative model via diffusion.
Applications: Image generation, conditional synthesis.
---
## Core Concepts & Theory
### Diffusion Process (Forward)
Iteratively add Gaussian noise.
### Denoising Network
Learn reverse noise prediction.
### Score Matching
Gradient of log probability.
---
## Mathematical Formulation
Diffusion forward process:
$$q(x_t | x_0) = \sqrt{\alpha_t} x_0 + \sqrt{1-\alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$
Reverse process loss (DDPM):
$$L = \mathbb{E}_{t, x_0, \epsilon}[\|\epsilon - \epsilon_ heta(x_t, t)\|^2]$$
Sampling:
$$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}(x_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}} \epsilon_ heta(x_t, t)) + \sigma_t z$$
---
## Advanced Theory & Extensions
### Classifier-Free Guidance
Conditional generation without classifier.
### Latent Diffusion
Diffusion in latent space.
### Consistency Models
Faster sampling; one-step generation.
---
## Computational Considerations
Forward pass: O(T·model_size) where T = timesteps.
Sampling: O(T·model_size); typically T=1000.
Training: O(model_size) per iteration.
---
## Practical Implementation Strategies
### Time Embedding
Sinusoidal positional encoding for timesteps.
### Noise Schedule
Linear, cosine, or learned schedule for α_t.
### Guidance Scale
Control classifier-free guidance strength.
---
## Benchmark Datasets & Evaluation
CIFAR-10: Diffusion benchmark.
CelebA: High-resolution generation.
ImageNet: Large-scale synthesis.
---
## Key Challenges & Limitations
### Sampling Speed
Slow compared to GANs; T=1000 steps.
### Timestep Dependency
Model must handle varying noise levels.
### Guidance Trade-off
Balance fidelity and diversity.
---
## Hyperparameter Tuning
Noise schedule: Linear or cosine; affects convergence.
Timesteps T: 1000 typical; quality-speed tradeoff.
Guidance scale: 7.5-15 for conditional; conditional only.
---
## Real-World Applications & Case Studies
Text-to-Image: DALL-E, Stable Diffusion.
Image Inpainting: Fill missing regions.
Super-Resolution: Enhance low-res images.
---
## Integration with Other Methods
Diffusion + Classifier → conditional generation.
Diffusion + Quantization → efficient inference.
---
## Summary & Key Takeaways
Diffusion models via iterative denoising enable high-quality generative modeling through noise prediction networks.
Principles:
1. Forward: iterative noise addition.
2. Reverse: learn denoising.
3. Score matching: gradient prediction.
4. Classifier-free guidance: conditional control.
5. Efficiency: latent space diffusion.
---
---
## Appendix: Practical Labs
### Lab 1: Noise Schedule
import numpy as np
def cosine_noise_schedule(num_steps=1000):
"""Cosine noise schedule"""
s = 0.008
steps = np.arange(num_steps + 1)
alphas_cumprod = np.cos(((steps / num_steps) + s) / (1 + s) * np.pi * 0.5) ** 2
alphas_cumprod = alphas_cumprod / alphas_cumprod[0]
betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1])
betas = np.clip(betas, 0.0001, 0.9999)
alphas = 1 - betas
alphas_cumprod = np.cumprod(alphas)
return betas, alphas, alphas_cumprod
# Test
betas, alphas, alphas_cumprod = cosine_noise_schedule(1000)
assert len(betas) == 1000, "Beta schedule length"
assert np.all(betas >= 0) and np.all(betas <= 1), "Betas in [0,1]"
print("✓ Noise schedule working")
if __name__ == "__main__":
print("Lab 1: NoiseSchedule - PASSED")### Lab 2: Forward Process
import numpy as np
def forward_diffusion_step(x0, t, alphas_cumprod):
"""Single forward diffusion step"""
alpha_cumprod_t = alphas_cumprod[t]
# Add noise
noise = np.random.randn(*x0.shape)
xt = np.sqrt(alpha_cumprod_t) * x0 + np.sqrt(1 - alpha_cumprod_t) * noise
return xt, noise
# Test
np.random.seed(42)
x0 = np.random.randn(32, 3, 32, 32)
alphas_cumprod = np.linspace(1, 0.01, 1000)
xt, noise = forward_diffusion_step(x0, 500, alphas_cumprod)
assert xt.shape == x0.shape, "Shape preserved"
assert noise.shape == x0.shape, "Noise shape matches"
print("✓ Forward diffusion working")
if __name__ == "__main__":
print("Lab 2: ForwardDiffusion - PASSED")### Lab 3: Reverse Process Sampling
import numpy as np
def reverse_diffusion_step(xt, t, noise_pred, alphas, alphas_cumprod, betas):
"""Single reverse diffusion step"""
alpha_t = alphas[t]
alpha_cumprod_t = alphas_cumprod[t]
beta_t = betas[t]
# Posterior variance
posterior_var = beta_t * (1 - alphas_cumprod[t-1]) / (1 - alpha_cumprod_t)
# Reverse step
coeff = (1 - alphas_cumprod[t-1]) / (1 - alpha_cumprod_t)
x0_pred = (xt - np.sqrt(1 - alpha_cumprod_t) * noise_pred) / np.sqrt(alpha_cumprod_t)
mean = (xt - beta_t / np.sqrt(1 - alpha_cumprod_t) * noise_pred) / np.sqrt(alpha_t)
z = np.random.randn(*xt.shape)
xt_minus_1 = mean + np.sqrt(posterior_var) * z
return xt_minus_1
# Test
np.random.seed(42)
xt = np.random.randn(32, 3, 32, 32)
noise_pred = np.random.randn(32, 3, 32, 32)
alphas = np.linspace(1, 0.01, 1000)
alphas_cumprod = np.cumprod(alphas)
betas = 1 - alphas
xt_minus_1 = reverse_diffusion_step(xt, 500, noise_pred, alphas, alphas_cumprod, betas)
assert xt_minus_1.shape == xt.shape, "Shape preserved"
print("✓ Reverse diffusion working")
if __name__ == "__main__":
print("Lab 3: ReverseDiffusion - PASSED")### Lab 4: Diffusion Sampling
import numpy as np
def sample_diffusion(noise_pred_fn, num_steps=50, num_samples=4, img_size=32):
"""Sample from diffusion model"""
# Simple noise schedule
alphas_cumprod = np.linspace(1, 0.01, num_steps)
# Start from noise
xt = np.random.randn(num_samples, 3, img_size, img_size)
# Reverse steps
for t in range(num_steps - 1, 0, -1):
# Predict noise (simplified)
noise_pred = noise_pred_fn(xt, t)
# Simplified reverse step
alpha_cumprod_t = alphas_cumprod[t]
alpha_cumprod_prev = alphas_cumprod[t-1] if t > 0 else 1.0
x0_coeff = np.sqrt(alpha_cumprod_prev) / np.sqrt(alpha_cumprod_t)
noise_coeff = np.sqrt(1 - alpha_cumprod_prev) / np.sqrt(1 - alpha_cumprod_t)
xt = x0_coeff * xt - noise_coeff * noise_pred
return xt
# Test
np.random.seed(42)
def dummy_noise_pred(x, t):
return np.random.randn(*x.shape) * 0.01
samples = sample_diffusion(dummy_noise_pred, num_steps=50)
assert samples.shape == (4, 3, 32, 32), "Sample shape"
print("✓ Diffusion sampling working")
if __name__ == "__main__":
print("Lab 4: DiffusionSampling - PASSED")