diffusion model

**Diffusion models** are the generative method behind most modern image, video, and audio synthesis — Stable Diffusion, DALL-E, Midjourney, Sora and their kin. The core idea is almost paradoxically simple: take a clean image and gradually destroy it with random noise until nothing is left, then train a neural network to reverse that process one small step at a time. Once the network knows how to remove a little noise, you can start from pure static and, step by step, denoise your way to a brand-new image that never existed. Diffusion has largely displaced GANs for high-fidelity generation because it is far more stable to train and covers the diversity of the data better.\n\n```svg\n\n```\n\n**The forward process just adds noise, and it has no parameters.** A fixed schedule corrupts a real image over many timesteps, adding a small amount of Gaussian noise at each one until the final step is indistinguishable from pure static. There is nothing to learn here — it is a mathematically defined destruction. Its only purpose is to manufacture training pairs: at every noise level, the model gets to see "here is the noisy version, here is the noise that was added."\n\n**The reverse process is the model, and it is what you train.** A neural network learns to look at a noisy image and predict the noise that should be removed to make it slightly cleaner. The training objective is strikingly plain — a mean-squared error between the true added noise and the network's predicted noise. Do this well at every noise level and the network has implicitly learned the structure of the entire data distribution.\n\n**Generation is iterative denoising from scratch.** To create a new image, you sample pure random noise and run the reverse network repeatedly, each pass stripping away a bit more noise, until a coherent image emerges. This is why diffusion sampling is slower than a single forward pass of a GAN — it takes many steps. Much recent research (DDIM, distillation, consistency and flow-matching models) is about cutting the number of steps from hundreds down to a handful without losing quality.\n\n**Conditioning is how you control the output.** Text-to-image models feed a prompt embedding into the denoiser so that every denoising step is nudged toward images matching the text. Classifier-free guidance amplifies this by contrasting the conditioned and unconditioned predictions, trading some diversity for much stronger prompt adherence — the guidance scale knob users tune. The same conditioning mechanism handles inpainting, image-to-image, and control signals.\n\n**Latent diffusion is what made it cheap enough to ship.** Running diffusion directly on megapixel images is enormously expensive. Latent diffusion first compresses images into a small latent space with a VAE, runs the whole noising/denoising process there, and only decodes back to pixels at the end. This cut the compute by orders of magnitude and is why Stable Diffusion could run on consumer GPUs. Modern systems increasingly replace the U-Net denoiser with a Transformer (a DiT), inheriting the scaling behavior of large Transformers.\n\n| Piece | Role | Learned? |\n|---|---|---|\n| Forward process | add noise on a fixed schedule | no — defined, not trained |\n| Reverse network (U-Net / DiT) | predict the noise to remove | yes — the entire model |\n| Sampler (DDPM, DDIM, ...) | run reverse steps to generate | no — an algorithm |\n| Conditioning + guidance | steer output toward a prompt | conditioning trained in |\n| VAE (latent diffusion) | compress to a cheap latent space | yes — separately trained |\n\nRead diffusion through a *learn-to-denoise* lens rather than a *paint-a-picture* lens: the model never learns to draw, it only ever learns the far easier task of estimating "how much noise is in this image and what does it look like." Generation is just that one humble skill applied over and over, starting from nothing but static — and the schedule, the conditioning, and the latent-space trick are the engineering that turns that skill into a controllable, affordable image generator.\n