25 Diffusion Models

Introduction to Generative Modelling

In neural computation, an image is treated as a high-dimensional random variable $\mathbf{x}\in {\mathbb{R}}^{H\times W\times C}$. These exists a “complicated” underlying distribution of real images, $p(\mathbf{x})$, which we want to understand.
The goal of a Generative Model is to use a neural network to approximate this distribution:

$${\hat{p}}_{\theta }(\mathbf{x})\approx p(\mathbf{x})$$

Once trained, we can “sample” from ${\hat{p}}_{\theta }(\mathbf{x})$ to create new images that look like they belong to the original dataset. Currently, diffusion models are considered the state-of-the-art, when outperforming Generative Adversarial Networks (GANs) in image synthesis.

Denoising Diffusion Probabilistic Models (DDPM)

A DDPM operates via two distinct processes: Forward Diffusion and Reverse Denoising.

A. The Forward Diffusion (Fixed)

This process takes a clean data sample ${\mathbf{x}}_0$ and gradually “diffuses” it by adding random Gaussian noise over $T$ time steps.

• Transition Step: Each step adds a small amount of noise governed by a variance schedule ${\beta }_t$.
  • $q({\mathbf{x}}_t|{\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_t;\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1},{\beta }_t\mathbf{I})$
• The Shortcut (Diffusion Kernel): We do not need to calculate every step. We can sample ${\mathbf{x}}_y$ directly from the starting image ${\mathbf{x}}_0$.
  • Define ${\alpha }_t=1-{\beta }_t$ and ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$
  • Formula: $q({\mathbf{x}}_t|{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_t;\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0,(1-{\bar{\alpha }}_t)\mathbf{I})$
  • Sampling Equation: ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$, where $ϵ\sim \mathcal{N}(0,\mathbf{I})$

B. The Reverse Denoising Process (Generative)

This is the “generative” part where the model learns to undo the noise.

• It starts with pure noise ${\mathbf{x}}_T\sim \mathcal{N}(0,\mathbf{I})$ and iteratively produces a slightly cleaner image ${\mathbf{x}}_{t-1}$ from a noisy image ${\mathbf{x}}_t$
• The model learns the transition $p_{\theta }({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)=\mathcal{N}({\mathbf{x}}_{t-1};{\mu }_{\theta }({\mathbf{x}}_t,t),{\sigma }_t^2\mathbf{I})$

Model Training and Architecture

The network is trained as Denoising Auto-encoder or U-Net.

Network Design

• Architecture: Typically uses a U-net with ResNet blocks and self-attention layers.
• Input: The noisy image ${\mathbf{x}}_t$ and the current time step $t$
• Time Representation: $t$ is fed into the model via fully-connected layers so the network knows the current level of noise it is dealing with.

Training Algorithm

The goal is to predict the noise $ϵ$ that was added to the clean image.

1. Pick a clean image ${\mathbf{x}}_0\sim q({\mathbf{x}}_0)$
2. Pick a random time step $t\sim \text{Uniform}(\{1,\cdots ,T\})$
3. Pick random noise $ϵ\sim \mathcal{N}(0,\mathbb{I})$
4. Loss Function: Minimise the difference between the actual noise and the predicted noise:

$${\nabla }_{\theta }\Vert ϵ-{ϵ}_0(\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ,t)\Vert$$

Sampling (Image Generation)

To generate a new image, the model follows an iterative process starting from pure noise.

The Sampling Algorithm

1. Start with ${\mathbf{x}}_T\sim \mathcal{N}(0,\mathbf{I})$
2. For $t=T,\dots ,1$:
   • Sample $\mathbf{z}\sim \mathcal{N}(0,\mathbf{I})$ (if $t>1$, else $\mathbf{z}=0$)
   • Calculate the previous (cleaner) step:

$${\mathbf{x}}_{t-1}=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{1-{\alpha }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_{\theta }({\mathbf{x}}_t,t)\right)+{\sigma }_t\mathbf{z}$$

3. Return ${\mathbf{x}}_0$

The Generative Trilemma

Generative model face three competing requirements:

1. High Quality Samples: Sharpe, realistic images
2. Mode Coverage/Diversity: Capturing all variations in the data
3. Fast Sampling: Speed of generation

Tip

Key Insight: Diffusion models excel at High Quality and Diversity but struggle with Fast Sampling because they require hundreds of sequential iterations to produce one image.

Extensions and Latent Diffusion

To overcome limitations like sampling speed, several extensions exist:

Conditional Diffusion

Conditions (like text or labels) are incorporated into the U-Net

• Scalar/Class: Encoded as vector embeddings.
• Image: Channel-wise concatenation (used for colorisation or super-resoilution).
• Text: Cross-attention mechanism (e.g., Stable Diffusion)

Latent Diffusion Models (Stable Diffusion)

LDMs solve the speed problem by performing the diffusion process in a compressed latent space.

1. Encoder $(E)$: Compress the pixel-space image ${\mathbf{x}}_0$ into a lower-dimensional latent vector $z_0$
2. Latent Diffusion: The forward and reverse processes are applied only to $z$.
3. Decoder $(D)$: Decompress the final denoising latent $z_0$ back into a high-resolution image.