Diffusion models

Plain English

Imagine a photo printed on wet paper. Drop it in a bucket of water and stir. In a minute you see colored smears—almost no recognizable image left. A diffusion model learns the reverse trick: start from random static and slowly un-smear it into a clear picture.

Most popular text-to-image tools today use some version of this idea. You type a prompt; the model does not paint stroke by stroke like a human. It runs many small denoising steps, each one nudging the pixels toward something that matches your words and the patterns it learned from training images.

The word diffusion here means spreading noise through data over time—not chemical diffusion in a lab. Training teaches a neural network to predict and remove noise at each step so generation can walk backward from chaos to structure.

How it works

Training has two linked phases. In the forward process, the system takes a real image and adds a little Gaussian noise repeatedly until the image is nearly pure noise. In the reverse process, a neural network learns to undo one noise step at a time—given a noisy image and a timestep, predict a slightly cleaner version.

At generation time there is no starting photo—only random noise. The model iterates: denoise a bit, denoise again, maybe hundreds of times, conditioned on your text embedding. Classifier-free guidance (a common training trick) lets the model listen harder to the prompt by comparing conditional and unconditional predictions.

• Forward process: clean image → progressively noisier latents or pixels.
• Reverse process: noise → learned denoiser steps → structured output.
• Conditioning: text, layout hints, or other inputs steer each step.
• Latent diffusion: many systems denoise in a compressed space for speed, then decode to pixels.

Going deeper

Diffusion sits alongside other generative families. GANs (generative adversarial networks) pit two networks against each other; VAEs learn compressed representations; autoregressive image models predict tokens left-to-right. Diffusion often wins on sample diversity and training stability, at the cost of slower sampling because many steps are required.

Research focuses on fewer-step samplers, better text alignment, control nets for pose or edges, and safety filters. For connections to vision backbones, see /learn/cnns-and-vision; for multimodal LLM interfaces, see /learn/multimodality and /learn/image-and-video-generation.

Common misconceptions

Diffusion models copy training images pixel by pixel.

They learn statistical patterns. Outputs can resemble styles or subjects from training data without being exact copies—though memorization of specific images can still happen and is an active concern.

One denoise step is enough.

Quality usually comes from many iterative steps. Fewer-step shortcuts exist but trade speed against detail and coherence.

The model understands your prompt like a person.

It matches learned associations between text embeddings and visual features. Unusual wording, negation, and counting often fail in predictable ways.

Key facts

• Diffusion models learn to reverse a gradual noising process.
• Generation starts from random noise and runs many denoise iterations.
• Text and other conditions steer each step via embeddings and guidance.
• Latent diffusion operates in a compressed representation for efficiency.
• Sampling speed and prompt fidelity remain active engineering tradeoffs.