Diffusion models have revolutionized generative AI — powering the images from DALL-E, Midjourney, and Stable Diffusion that have captivated the world. But how do these models actually work? The core idea is elegant: learn to reverse the process of gradually adding noise to data.
The Intuition: Destroy and Rebuild
Imagine taking a clear photograph and slowly adding random noise to it, one tiny step at a time. After thousands of steps, the original image is completely destroyed — just random static. A diffusion model learns to reverse this process: starting from pure noise, it gradually removes noise to reveal a coherent image. The forward process is fixed (adding noise according to a schedule); the reverse process is learned.
The Forward (Diffusion) Process
Starting from a real image x₀, noise is added in T steps according to a variance schedule β₁, β₂, …, β_T:
x_t = √(α_t) × x_{t-1} + √(1-α_t) × ε, where ε ~ N(0, I) and α_t = 1 – β_t.
As T → ∞ (typically T = 1,000 in practice), x_T approaches pure Gaussian noise.
The Reverse (Denoising) Process
The model — typically a U-Net — is trained to predict the noise added at each step. Given a noisy image x_t and the timestep t, the model predicts the noise ε that was added. The training objective is simple: minimize the mean squared error between the predicted noise and the actual noise added.
At inference time, we start from random noise and iteratively denoise it through all T steps — each step removing a small amount of noise — until a clean image emerges.
The U-Net Architecture
The U-Net is the workhorse of diffusion models. Its hourglass shape has:
- Contracting Path (Encoder): Downsampling layers that capture context and compress spatial information
- Bottleneck: The deepest layer representing the most abstract features
- Expanding Path (Decoder): Upsampling layers that reconstruct spatial detail, with skip connections from the encoder preserving fine-grained information
Self-attention layers at multiple resolutions allow the model to capture long-range dependencies — essential for generating coherent global structures.
Latent Diffusion: The Stable Diffusion Innovation
Stable Diffusion’s key innovation was moving the diffusion process from pixel space to latent space. A pre-trained Variational Autoencoder (VAE) compresses images into a much smaller latent representation (e.g., 512×512 pixels → 64×64 latent). The diffusion model operates on this compressed latent space, dramatically reducing computational cost while maintaining quality.
This is why Stable Diffusion can run on consumer GPUs — the diffusion happens in a space 48x smaller than pixel space. The VAE decoder then reconstructs the final image from the denoised latent.
Text Conditioning
Text-to-image diffusion models use cross-attention to inject text prompts into the generation process. A text encoder (typically CLIP) converts the prompt into embeddings. At each denoising step, the U-Net attends to these embeddings via cross-attention layers — the text guides what the image should contain.
Diffusion models represent one of the most elegant ideas in modern AI: sometimes the best way to create is to first learn how to destroy.