Diffusion models generate data by learning to reverse a gradual noising process — progressively adding Gaussian noise to data during training, then learning to denoise step-by-step during generation, producing high-quality images, audio, and video that rival or exceed GANs.
What Are Diffusion Models?
- Definition: Generative models based on denoising process.
- Training: Learn to reverse gradual corruption by noise.
- Generation: Start from pure noise, iteratively denoise.
- Examples: Stable Diffusion, DALL-E, Midjourney, Sora.
Why Diffusion Works
- Stable Training: No adversarial dynamics (unlike GANs).
- Quality: State-of-the-art image generation.
- Flexibility: Conditional generation, inpainting, editing.
- Theory: Strong mathematical foundation.
Forward Process (Noising)
Gradual Corruption:
x_0 → x_1 → x_2 → ... → x_T
(data) (pure noise)
At each step:
x_t = √(α_t) × x_{t-1} + √(1-α_t) × ε
Where ε ~ N(0, I) is Gaussian noise
α_t follows a schedule (typically 0.9999 to 0.0001)
Closed Form to Any Step:
x_t = √(ᾱ_t) × x_0 + √(1-ᾱ_t) × ε
Where ᾱ_t = Π_{s=1}^t α_s (cumulative product)
Visual:
<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
<rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
<text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Diffusion Models — Denoising Score Matching</text>
<text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">learn to reverse gradual noise addition — generate images by iterative denoising</text>
<!-- Forward process (top) -->
<rect x="30" y="65" width="700" height="100" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
<text x="380" y="84" fill="#f87171" font-size="10" font-weight="600" text-anchor="middle">Forward Process q(x_t | x_{t-1}) — Add Noise (fixed, no learning)</text>
<!-- Image at various noise levels -->
<rect x="60" y="98" width="50" height="50" rx="4" fill="#1e3a5f" stroke="#60a5fa" stroke-width="0.8"/>
<text x="85" y="127" fill="#93c5fd" font-size="8" text-anchor="middle">x₀</text>
<text x="85" y="155" fill="#6b7684" font-size="6.5" text-anchor="middle">clean</text>
<path d="M112,123 L135,123" fill="none" stroke="#f87171" stroke-width="0.8"/>
<polygon points="133,120 139,123 133,126" fill="#f87171"/>
<text x="123" y="117" fill="#fca5a5" font-size="6">+ε</text>
<rect x="140" y="98" width="50" height="50" rx="4" fill="#1a2a3a" stroke="#475569" stroke-width="0.8"/>
<text x="165" y="127" fill="#8b98a5" font-size="8" text-anchor="middle">x₁</text>
<path d="M192,123 L215,123" fill="none" stroke="#f87171" stroke-width="0.8"/>
<polygon points="213,120 219,123 213,126" fill="#f87171"/>
<rect x="220" y="98" width="50" height="50" rx="4" fill="#1a1a2e" stroke="#475569" stroke-width="0.8"/>
<text x="245" y="127" fill="#8b98a5" font-size="8" text-anchor="middle">x_t</text>
<path d="M272,123 L295,123" fill="none" stroke="#f87171" stroke-width="0.8"/>
<polygon points="293,120 299,123 293,126" fill="#f87171"/>
<text x="285" y="117" fill="#fca5a5" font-size="6">+ε</text>
<rect x="300" y="98" width="50" height="50" rx="4" fill="#1a0a1a" stroke="#475569" stroke-width="0.8"/>
<text x="325" y="127" fill="#8b98a5" font-size="8" text-anchor="middle">...</text>
<path d="M352,123 L375,123" fill="none" stroke="#f87171" stroke-width="0.8"/>
<polygon points="373,120 379,123 373,126" fill="#f87171"/>
<rect x="380" y="98" width="50" height="50" rx="4" fill="#1a0a0a" stroke="#f87171" stroke-width="0.8"/>
<text x="405" y="127" fill="#fca5a5" font-size="8" text-anchor="middle">x_T</text>
<text x="405" y="155" fill="#6b7684" font-size="6.5" text-anchor="middle">pure noise</text>
<text x="530" y="110" fill="#8b98a5" font-size="8.5">q(x_t|x₀) = N(√ᾱ_t · x₀, (1-ᾱ_t)I)</text>
<text x="530" y="128" fill="#6b7684" font-size="8">T = 1000 steps (DDPM)</text>
<text x="530" y="146" fill="#6b7684" font-size="8">closed-form: sample x_t directly from x₀</text>
<!-- Reverse process (middle) -->
<rect x="30" y="175" width="700" height="100" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
<text x="380" y="194" fill="#34d399" font-size="10" font-weight="600" text-anchor="middle">Reverse Process p_θ(x_{t-1} | x_t) — Denoise (learned)</text>
<!-- Denoising -->
<rect x="380" y="208" width="50" height="50" rx="4" fill="#1a0a0a" stroke="#f87171" stroke-width="0.8"/>
<text x="405" y="237" fill="#fca5a5" font-size="8" text-anchor="middle">x_T</text>
<path d="M378,233 L355,233" fill="none" stroke="#34d399" stroke-width="0.8"/>
<polygon points="357,236 351,233 357,230" fill="#34d399"/>
<text x="366" y="227" fill="#6ee7b7" font-size="6">-ε_θ</text>
<rect x="300" y="208" width="50" height="50" rx="4" fill="#1a0a1a" stroke="#475569" stroke-width="0.8"/>
<text x="325" y="237" fill="#8b98a5" font-size="8" text-anchor="middle">...</text>
<path d="M298,233 L275,233" fill="none" stroke="#34d399" stroke-width="0.8"/>
<polygon points="277,236 271,233 277,230" fill="#34d399"/>
<rect x="220" y="208" width="50" height="50" rx="4" fill="#1a1a2e" stroke="#475569" stroke-width="0.8"/>
<text x="245" y="237" fill="#8b98a5" font-size="8" text-anchor="middle">x_t</text>
<path d="M218,233 L195,233" fill="none" stroke="#34d399" stroke-width="0.8"/>
<polygon points="197,236 191,233 197,230" fill="#34d399"/>
<rect x="140" y="208" width="50" height="50" rx="4" fill="#1a2a3a" stroke="#475569" stroke-width="0.8"/>
<text x="165" y="237" fill="#8b98a5" font-size="8" text-anchor="middle">x₁</text>
<path d="M138,233 L115,233" fill="none" stroke="#34d399" stroke-width="0.8"/>
<polygon points="117,236 111,233 117,230" fill="#34d399"/>
<rect x="60" y="208" width="50" height="50" rx="4" fill="#1e3a5f" stroke="#34d399" stroke-width="1.2"/>
<text x="85" y="237" fill="#6ee7b7" font-size="8" text-anchor="middle">x₀</text>
<text x="85" y="265" fill="#34d399" font-size="6.5" text-anchor="middle">generated!</text>
<text x="530" y="218" fill="#8b98a5" font-size="8.5">U-Net (or DiT) predicts noise ε_θ(x_t, t)</text>
<text x="530" y="236" fill="#8b98a5" font-size="8.5">loss = ||ε - ε_θ(x_t, t)||²</text>
<text x="530" y="254" fill="#6b7684" font-size="8">50 steps (DDIM) or 1–4 (distilled)</text>
<text x="530" y="270" fill="#6b7684" font-size="8">CFG: classifier-free guidance (scale=7.5)</text>
<!-- Architecture + variants -->
<rect x="30" y="288" width="345" height="140" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
<text x="202" y="308" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">Architecture: U-Net → DiT</text>
<text x="50" y="330" fill="#60a5fa" font-size="9" font-weight="600">U-Net (SD 1.5, SDXL):</text>
<text x="50" y="346" fill="#8b98a5" font-size="8">encoder-decoder + skip connections + cross-attn</text>
<text x="50" y="360" fill="#6b7684" font-size="7.5">text conditioning via CLIP text encoder</text>
<text x="50" y="382" fill="#a78bfa" font-size="9" font-weight="600">DiT (SD 3, FLUX, Sora):</text>
<text x="50" y="398" fill="#8b98a5" font-size="8">pure transformer on latent patches</text>
<text x="50" y="412" fill="#6b7684" font-size="7.5">scales better with compute, uses MM-DiT blocks</text>
<!-- Right: key models -->
<rect x="390" y="288" width="340" height="140" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
<text x="560" y="308" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">Diffusion Model Family</text>
<text x="410" y="330" fill="#f59e0b" font-size="8.5" font-weight="600">DALL·E 2 (2022)</text>
<text x="540" y="330" fill="#6b7684" font-size="8">CLIP + diffusion prior</text>
<text x="410" y="348" fill="#34d399" font-size="8.5" font-weight="600">Stable Diffusion (2022)</text>
<text x="540" y="348" fill="#6b7684" font-size="8">latent space diffusion (VAE)</text>
<text x="410" y="366" fill="#60a5fa" font-size="8.5" font-weight="600">SDXL / SD 3 (2024)</text>
<text x="540" y="366" fill="#6b7684" font-size="8">DiT + flow matching</text>
<text x="410" y="384" fill="#a78bfa" font-size="8.5" font-weight="600">FLUX (2024)</text>
<text x="540" y="384" fill="#6b7684" font-size="8">rectified flow + T5 text enc</text>
<text x="410" y="402" fill="#f87171" font-size="8.5" font-weight="600">Sora (2024)</text>
<text x="540" y="402" fill="#6b7684" font-size="8">video DiT, spacetime patches</text>
<text x="410" y="420" fill="#38bdf8" font-size="8.5" font-weight="600">consistency models</text>
<text x="540" y="420" fill="#6b7684" font-size="8">1-step generation (distilled)</text>
<text x="380" y="452" fill="#6b7684" font-size="11" text-anchor="middle">Diffusion models replaced GANs for image generation — iterative refinement beats adversarial training.</text>
</svg>
Reverse Process (Denoising)
Learning to Denoise:
Train neural network ε_θ to predict noise:
Loss = ||ε - ε_θ(x_t, t)||²
Given noisy image x_t and timestep t,
predict the noise ε that was added.
Generation (Sampling):
Start: x_T ~ N(0, I) (pure noise)
For t = T, T-1, ..., 1:
Predict noise: ε̂ = ε_θ(x_t, t)
Compute x_{t-1} using ε̂
Return: x_0 (generated sample)
Implementation Sketch
Training Loop:
import torch
import torch.nn.functional as F
def train_step(model, x_0, noise_scheduler):
# Sample random timesteps
t = torch.randint(0, T, (batch_size,))
# Sample noise
noise = torch.randn_like(x_0)
# Add noise to get x_t
x_t = noise_scheduler.add_noise(x_0, noise, t)
# Predict noise
predicted_noise = model(x_t, t)
# MSE loss
loss = F.mse_loss(predicted_noise, noise)
return loss
Sampling Loop:
@torch.no_grad()
def sample(model, noise_scheduler, shape):
# Start from pure noise
x = torch.randn(shape)
# Iteratively denoise
for t in reversed(range(T)):
# Predict noise
predicted_noise = model(x, t)
# Compute previous step
x = noise_scheduler.step(predicted_noise, t, x)
return x
Key Architectures
U-Net (Standard):
<svg viewBox="0 0 536 511" xmlns="http://www.w3.org/2000/svg" style="max-width:100%;height:auto" role="img"><rect x="0" y="0" width="536" height="511" rx="12" fill="#0d1117"/><g font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,"Liberation Mono",monospace" font-size="14"><text xml:space="preserve" x="20" y="31.7"><tspan fill="#6e7681">┌─────────────────────────────────────────────────────────┐</tspan></text><text xml:space="preserve" x="20" y="50.7"><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> Noisy Image + t </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="69.7"><tspan fill="#6e7681">└─────────────────────────────────────────────────────────┘</tspan></text><text xml:space="preserve" x="20" y="88.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="107.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">┌───▼───┐</tspan><tspan fill="#c9d1d9"> Encoder (downsampling)</tspan></text><text xml:space="preserve" x="20" y="126.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> Conv </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="145.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">└───┬───┘</tspan></text><text xml:space="preserve" x="20" y="164.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│──────────────────┐</tspan></text><text xml:space="preserve" x="20" y="183.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">┌───▼───┐</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="202.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> Conv </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="221.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">└───┬───┘</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="240.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│─────────────┐</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="259.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">┌───▼───┐</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="278.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9">Bottom </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="297.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">└───┬───┘</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="316.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="335.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">┌───▼───┐←────────┘</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> Decoder (upsampling)</tspan></text><text xml:space="preserve" x="20" y="354.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> Conv </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> skip </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="373.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">└───┬───┘</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="392.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan></text><text xml:space="preserve" x="20" y="411.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">┌───▼───┐←─────────────┘</tspan></text><text xml:space="preserve" x="20" y="430.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> Conv </tspan><tspan fill="#6e7681">│</tspan><tspan fill="#c9d1d9"> skip</tspan></text><text xml:space="preserve" x="20" y="449.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">└───┬───┘</tspan></text><text xml:space="preserve" x="20" y="468.7"><tspan fill="#c9d1d9"> </tspan><tspan fill="#6e7681">▼</tspan></text><text xml:space="preserve" x="20" y="487.7"><tspan fill="#c9d1d9"> Predicted Noise</tspan></text></g></svg>
DiT (Diffusion Transformer):
Modern alternative using transformers instead of U-Net:
- Used in Sora, recent SOTA models
- Better scaling properties
- Patch-based processing
Conditional Generation
Text-to-Image:
# Classifier-free guidance
def guided_sample(model, prompt, guidance_scale=7.5):
text_embeddings = encode_text(prompt)
for t in reversed(range(T)):
# Conditional prediction
noise_cond = model(x, t, text_embeddings)
# Unconditional prediction
noise_uncond = model(x, t, null_embedding)
# Guided prediction
noise = noise_uncond + guidance_scale * (noise_cond - noise_uncond)
x = denoise_step(x, noise, t)
return x
Popular Models
Model | Type | Open Source
-------------------|----------------|------------
Stable Diffusion | Text-to-image | Yes
DALL-E 3 | Text-to-image | No
Midjourney | Text-to-image | No
Sora | Text-to-video | No
Runway Gen-2 | Text-to-video | No
AudioLDM | Text-to-audio | Yes
Diffusion models are the dominant paradigm for generative AI — their stable training, high quality outputs, and flexibility for conditioning have made them the foundation of modern image, video, and audio generation systems.
<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
<rect width="760" height="470" fill="#0d1117"/>
<text x="380" y="30" fill="#e6edf3" font-size="20" font-weight="700" text-anchor="middle">Diffusion Forward Process (q)</text>
<text x="380" y="50" fill="#8b98a5" font-size="12" text-anchor="middle">Data is progressively corrupted by Gaussian noise over timesteps 0..T</text>
<defs>
<marker id="d1a" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#58a6ff"/></marker>
</defs>
<g>
<rect x="35" y="110" width="150" height="120" rx="10" fill="#161b22" stroke="#2ea043"/>
<text x="110" y="138" fill="#3fb950" font-size="13" font-weight="700" text-anchor="middle">x₀ (clean)</text>
<rect x="55" y="155" width="110" height="55" rx="6" fill="#0f1720" stroke="#3fb950"/>
<text x="110" y="187" fill="#8b98a5" font-size="11" text-anchor="middle">data manifold</text>
</g>
<g>
<rect x="215" y="110" width="150" height="120" rx="10" fill="#161b22" stroke="#1f6feb"/>
<text x="290" y="138" fill="#58a6ff" font-size="13" font-weight="700" text-anchor="middle">xₜ (mid noise)</text>
<circle cx="290" cy="182" r="38" fill="#0f1720" stroke="#58a6ff"/>
<g fill="#58a6ff" opacity="0.9">
<circle cx="270" cy="170" r="2"/><circle cx="304" cy="166" r="2"/><circle cx="286" cy="185" r="2"/>
<circle cx="310" cy="196" r="2"/><circle cx="274" cy="198" r="2"/><circle cx="292" cy="207" r="2"/>
</g>
</g>
<g>
<rect x="395" y="110" width="150" height="120" rx="10" fill="#161b22" stroke="#d29922"/>
<text x="470" y="138" fill="#fbbf24" font-size="13" font-weight="700" text-anchor="middle">xₜ' (high noise)</text>
<circle cx="470" cy="182" r="38" fill="#0f1720" stroke="#fbbf24"/>
<g fill="#fbbf24" opacity="0.9">
<circle cx="450" cy="164" r="2"/><circle cx="486" cy="162" r="2"/><circle cx="468" cy="174" r="2"/>
<circle cx="500" cy="181" r="2"/><circle cx="441" cy="183" r="2"/><circle cx="480" cy="191" r="2"/>
<circle cx="458" cy="201" r="2"/><circle cx="492" cy="204" r="2"/><circle cx="470" cy="210" r="2"/>
</g>
</g>
<g>
<rect x="575" y="110" width="150" height="120" rx="10" fill="#161b22" stroke="#f85149"/>
<text x="650" y="138" fill="#ff7b72" font-size="13" font-weight="700" text-anchor="middle">x_T (Gaussian)</text>
<circle cx="650" cy="182" r="38" fill="#0f1720" stroke="#ff7b72"/>
<g fill="#ff7b72" opacity="0.95">
<circle cx="621" cy="162" r="2"/><circle cx="643" cy="159" r="2"/><circle cx="664" cy="164" r="2"/>
<circle cx="679" cy="175" r="2"/><circle cx="624" cy="182" r="2"/><circle cx="649" cy="182" r="2"/>
<circle cx="670" cy="188" r="2"/><circle cx="637" cy="196" r="2"/><circle cx="656" cy="201" r="2"/>
<circle cx="682" cy="201" r="2"/><circle cx="626" cy="206" r="2"/>
</g>
</g>
<path d="M185 170H212" stroke="#58a6ff" stroke-width="2" marker-end="url(#d1a)"/>
<path d="M365 170H392" stroke="#58a6ff" stroke-width="2" marker-end="url(#d1a)"/>
<path d="M545 170H572" stroke="#58a6ff" stroke-width="2" marker-end="url(#d1a)"/>
<text x="200" y="162" fill="#8b98a5" font-size="10" text-anchor="middle">β₁</text>
<text x="380" y="162" fill="#8b98a5" font-size="10" text-anchor="middle">βₜ</text>
<text x="560" y="162" fill="#8b98a5" font-size="10" text-anchor="middle">β_T</text>
<rect x="35" y="275" width="690" height="150" rx="10" fill="#121820" stroke="#30363d"/>
<text x="380" y="301" fill="#e6edf3" font-size="13" font-weight="700" text-anchor="middle">Closed-form noising relation</text>
<text x="380" y="330" fill="#79c0ff" font-size="18" text-anchor="middle">xₜ = √(ᾱₜ) x₀ + √(1-ᾱₜ) ε, ε ~ N(0, I)</text>
<text x="380" y="356" fill="#8b98a5" font-size="12" text-anchor="middle">αₜ = 1-βₜ, ᾱₜ = Πᵢ₌₁ᵗ αᵢ</text>
<text x="380" y="382" fill="#8b98a5" font-size="12" text-anchor="middle">As t increases: signal-to-noise ratio decreases monotonically</text>
<text x="380" y="408" fill="#6e7681" font-size="11" text-anchor="middle">Training samples random t and learns to predict ε from noisy xₜ</text>
</svg>
<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
<rect width="760" height="470" fill="#0d1117"/>
<text x="380" y="30" fill="#e6edf3" font-size="20" font-weight="700" text-anchor="middle">Reverse Denoising Process (pθ)</text>
<text x="380" y="50" fill="#8b98a5" font-size="12" text-anchor="middle">Generation starts from noise and iteratively removes it using a learned UNet</text>
<defs>
<marker id="d2a" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#3fb950"/></marker>
</defs>
<rect x="30" y="95" width="700" height="250" rx="12" fill="#11161f" stroke="#30363d"/>
<g>
<rect x="50" y="125" width="120" height="90" rx="8" fill="#161b22" stroke="#f85149"/>
<text x="110" y="149" fill="#ff7b72" font-size="12" font-weight="700" text-anchor="middle">x_T</text>
<text x="110" y="168" fill="#8b98a5" font-size="10" text-anchor="middle">pure Gaussian</text>
<text x="110" y="185" fill="#8b98a5" font-size="10" text-anchor="middle">noise</text>
</g>
<g>
<rect x="205" y="125" width="140" height="90" rx="8" fill="#161b22" stroke="#1f6feb"/>
<text x="275" y="149" fill="#58a6ff" font-size="12" font-weight="700" text-anchor="middle">UNet εθ(xₜ,t,c)</text>
<text x="275" y="168" fill="#8b98a5" font-size="10" text-anchor="middle">predict noise</text>
<text x="275" y="185" fill="#8b98a5" font-size="10" text-anchor="middle">conditioned on prompt c</text>
</g>
<g>
<rect x="380" y="125" width="140" height="90" rx="8" fill="#161b22" stroke="#d29922"/>
<text x="450" y="149" fill="#fbbf24" font-size="12" font-weight="700" text-anchor="middle">Sampler Step</text>
<text x="450" y="168" fill="#8b98a5" font-size="10" text-anchor="middle">DDPM / DDIM /</text>
<text x="450" y="185" fill="#8b98a5" font-size="10" text-anchor="middle">DPM-Solver</text>
</g>
<g>
<rect x="555" y="125" width="150" height="90" rx="8" fill="#161b22" stroke="#2ea043"/>
<text x="630" y="149" fill="#3fb950" font-size="12" font-weight="700" text-anchor="middle">xₜ₋₁</text>
<text x="630" y="168" fill="#8b98a5" font-size="10" text-anchor="middle">less noisy sample</text>
<text x="630" y="185" fill="#8b98a5" font-size="10" text-anchor="middle">repeat until x₀</text>
</g>
<path d="M170 170H200" stroke="#3fb950" stroke-width="2" marker-end="url(#d2a)"/>
<path d="M345 170H375" stroke="#3fb950" stroke-width="2" marker-end="url(#d2a)"/>
<path d="M520 170H550" stroke="#3fb950" stroke-width="2" marker-end="url(#d2a)"/>
<path d="M630 220V255H110V220" stroke="#58a6ff" stroke-width="2" fill="none" stroke-dasharray="6 5" marker-end="url(#d2a)"/>
<text x="370" y="248" fill="#79c0ff" font-size="11" text-anchor="middle">loop t = T..1</text>
<rect x="70" y="285" width="620" height="45" rx="8" fill="#0f1720" stroke="#30363d"/>
<text x="380" y="313" fill="#8b98a5" font-size="11" text-anchor="middle">Classifier-free guidance: ε̂ = ε_uncond + w(ε_cond - ε_uncond)</text>
<text x="380" y="362" fill="#e6edf3" font-size="13" font-weight="700" text-anchor="middle">Sampling quality/speed trade-off</text>
<g fill="#8b98a5" font-size="11" text-anchor="middle">
<text x="165" y="388">DDPM: higher fidelity, more steps</text>
<text x="380" y="388">DDIM: deterministic, fewer steps</text>
<text x="600" y="388">DPM-Solver: fast high-quality ODE</text>
</g>
<text x="380" y="436" fill="#6e7681" font-size="11" text-anchor="middle">Final x₀ becomes generated image/audio/video sample</text>
</svg>
<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
<rect width="760" height="470" fill="#0d1117"/>
<text x="380" y="30" fill="#e6edf3" font-size="20" font-weight="700" text-anchor="middle">Latent Diffusion Pipeline (Text-to-Image)</text>
<text x="380" y="50" fill="#8b98a5" font-size="12" text-anchor="middle">Compress to latent space, denoise there, decode back to pixels</text>
<defs>
<marker id="d3a" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#58a6ff"/></marker>
</defs>
<rect x="25" y="90" width="710" height="320" rx="12" fill="#11161f" stroke="#30363d"/>
<rect x="45" y="120" width="130" height="78" rx="8" fill="#161b22" stroke="#2ea043"/>
<text x="110" y="144" fill="#3fb950" font-size="12" font-weight="700" text-anchor="middle">Prompt</text>
<text x="110" y="162" fill="#8b98a5" font-size="10" text-anchor="middle">"silicon wafer in</text>
<text x="110" y="176" fill="#8b98a5" font-size="10" text-anchor="middle">cleanroom"</text>
<rect x="210" y="120" width="130" height="78" rx="8" fill="#161b22" stroke="#1f6feb"/>
<text x="275" y="144" fill="#58a6ff" font-size="12" font-weight="700" text-anchor="middle">Text Encoder</text>
<text x="275" y="162" fill="#8b98a5" font-size="10" text-anchor="middle">CLIP / T5</text>
<text x="275" y="176" fill="#8b98a5" font-size="10" text-anchor="middle">context embeddings</text>
<rect x="375" y="120" width="145" height="78" rx="8" fill="#161b22" stroke="#a371f7"/>
<text x="447" y="144" fill="#d2a8ff" font-size="12" font-weight="700" text-anchor="middle">Latent UNet</text>
<text x="447" y="162" fill="#8b98a5" font-size="10" text-anchor="middle">cross-attention</text>
<text x="447" y="176" fill="#8b98a5" font-size="10" text-anchor="middle">denoising steps</text>
<rect x="555" y="120" width="150" height="78" rx="8" fill="#161b22" stroke="#d29922"/>
<text x="630" y="144" fill="#fbbf24" font-size="12" font-weight="700" text-anchor="middle">VAE Decoder</text>
<text x="630" y="162" fill="#8b98a5" font-size="10" text-anchor="middle">latent z → image x</text>
<text x="630" y="176" fill="#8b98a5" font-size="10" text-anchor="middle">pixel space</text>
<path d="M175 156H205" stroke="#58a6ff" stroke-width="2" marker-end="url(#d3a)"/>
<path d="M340 156H370" stroke="#58a6ff" stroke-width="2" marker-end="url(#d3a)"/>
<path d="M520 156H550" stroke="#58a6ff" stroke-width="2" marker-end="url(#d3a)"/>
<rect x="65" y="240" width="630" height="62" rx="8" fill="#0f1720" stroke="#30363d"/>
<text x="380" y="266" fill="#e6edf3" font-size="12" font-weight="700" text-anchor="middle">Why latent space?</text>
<text x="380" y="285" fill="#8b98a5" font-size="11" text-anchor="middle">4x-16x fewer spatial elements than pixel space → faster sampling and lower memory</text>
<rect x="65" y="316" width="300" height="74" rx="8" fill="#161b22" stroke="#2ea043"/>
<text x="215" y="340" fill="#3fb950" font-size="12" font-weight="700" text-anchor="middle">Training objective</text>
<text x="215" y="358" fill="#8b98a5" font-size="10" text-anchor="middle">Predict added noise ε</text>
<text x="215" y="374" fill="#8b98a5" font-size="10" text-anchor="middle">with MSE loss in latent space</text>
<rect x="395" y="316" width="300" height="74" rx="8" fill="#161b22" stroke="#1f6feb"/>
<text x="545" y="340" fill="#58a6ff" font-size="12" font-weight="700" text-anchor="middle">Inference controls</text>
<text x="545" y="358" fill="#8b98a5" font-size="10" text-anchor="middle">guidance scale, sampler, step count, seed</text>
<text x="545" y="374" fill="#8b98a5" font-size="10" text-anchor="middle">trade-off: diversity vs prompt fidelity</text>
<text x="380" y="436" fill="#6e7681" font-size="11" text-anchor="middle">Stable Diffusion class models are latent diffusion systems with text-conditioned denoising.</text>
</svg>
diffusiondenoisinggenerativestable diffusionunetnoise
Related Topics
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.