In machine learning, diffusion models, also known as diffusion probabilistic models or score-based generative models, are a class of latent variable generative models. A diffusion model consists of three major components: the forward process, the reverse process, and the sampling procedure.^[1] The goal of diffusion models is to learn a diffusion process that generates a probability distribution for a given dataset from which we can then sample new images. They learn the latent structure of a dataset by modeling the way in which data points diffuse through their latent space.^[2]

In the case of computer vision, diffusion models can be applied to a variety of tasks, including image denoising, inpainting, super-resolution, and image generation. This typically involves training a neural network to sequentially denoise images blurred with Gaussian noise.^[2][3] The model is trained to reverse the process of adding noise to an image. After training to convergence, it can be used for image generation by starting with an image composed of random noise for the network to iteratively denoise. Announced on 13 April 2022, OpenAI's text-to-image model DALL-E 2 is an example that uses diffusion models for both the model's prior (which produces an image embedding given a text caption) and the decoder that generates the final image.^[4] Diffusion models have recently found applications in natural language processing (NLP),^[5] particularly in areas like text generation^[6][7] and summarization.^[8]

Diffusion models are typically formulated as markov chains and trained using variational inference.^[9] Examples of generic diffusion modeling frameworks used in computer vision are denoising diffusion probabilistic models, noise conditioned score networks, and stochastic differential equations.^[10]

Denoising diffusion model

Non-equilibrium thermodynamics

Diffusion models were introduced in 2015 as a method to learn a model that can sample from a highly complex probability distribution. They used techniques from non-equilibrium thermodynamics, especially diffusion.^[11]

Consider, for example, how one might model the distribution of all naturally-occurring photos. Each image is a point in the space of all images, and the distribution of naturally-occurring photos is a "cloud" in space, which, by repeatedly adding noise to the images, diffuses out to the rest of the image space, until the cloud becomes all but indistinguishable from a Gaussian distribution ${\displaystyle N(0,I)}$. A model that can approximately undo the diffusion can then be used to sample from the original distribution. This is studied in "non-equilibrium" thermodynamics, as the starting distribution is not in equilibrium, unlike the final distribution.

The equilibrium distribution is the Gaussian distribution ${\displaystyle N(0,I)}$, with pdf ${\displaystyle \rho (x)\propto e^{-{\frac {1}{2))\|x\|^{2))}$. This is just the Maxwell–Boltzmann distribution of particles in a potential well ${\displaystyle V(x)={\frac {1}{2))\|x\|^{2))$ at temperature 1. The initial distribution, being very much out of equilibrium, would diffuse towards the equilibrium distribution, making biased random steps that are a sum of pure randomness (like a Brownian walker) and gradient descent down the potential well. The randomness is necessary: if the particles were to undergo only gradient descent, then they will all fall to the origin, collapsing the distribution.

Denoising Diffusion Probabilistic Model (DDPM)

The 2020 paper proposed the Denoising Diffusion Probabilistic Model (DDPM), which improves upon the previous method by variational inference.^[9]

Forward diffusion

To present the model, we need some notation.

• ${\displaystyle \beta _{1},...,\beta _{T}\in (0,1)}$ are fixed constants.
• ${\displaystyle \alpha _{t}:=1-\beta _{t))$
• ${\displaystyle {\bar {\alpha ))_{t}:=\alpha _{1}\cdots \alpha _{t))$
• ${\displaystyle {\tilde {\beta ))_{t}:={\frac {1-{\bar {\alpha ))_{t-1)){1-{\bar {\alpha ))_{t))}\beta _{t))$
• ${\displaystyle {\tilde {\mu ))_{t}(x_{t},x_{0}):={\frac ((\sqrt {\alpha _{t))}(1-{\bar {\alpha ))_{t-1})x_{t}+{\sqrt ((\bar {\alpha ))_{t-1))}(1-\alpha _{t})x_{0)){1-{\bar {\alpha ))_{t))))$
• ${\displaystyle N(\mu ,\Sigma )}$ is the normal distribution with mean ${\displaystyle \mu }$ and variance ${\displaystyle \Sigma }$, and ${\displaystyle N(x|\mu ,\Sigma )}$ is the probability density at ${\displaystyle x}$.
• A vertical bar denotes conditioning.

A forward diffusion process starts at some starting point ${\displaystyle x_{0}\sim q}$, where ${\displaystyle q}$ is the probability distribution to be learned, then repeatedly adds noise to it by

$${\displaystyle x_{t}={\sqrt {1-\beta _{t))}x_{t-1}+{\sqrt {\beta _{t))}z_{t))$$

${\displaystyle z_{1},...,z_{T))$

${\displaystyle N(0,I)}$

${\displaystyle x_{0))$

${\displaystyle \lim _{t}x_{t}|x_{0))$

${\displaystyle N(0,I)}$

The entire diffusion process then satisfies

$${\displaystyle q(x_{0:T})=q(x_{0})q(x_{1}|x_{0})\cdots q(x_{T}|x_{T-1})=q(x_{0})N(x_{1}|{\sqrt {\alpha _{1))}x_{0},\beta _{1}I)\cdots N(x_{T}|{\sqrt {\alpha _{T))}x_{T-1},\beta _{T}I)}$$

$${\displaystyle \ln q(x_{0:T})=\ln q(x_{0})-\sum _{t=1}^{T}{\frac {1}{2\beta _{t))}\|x_{t}-{\sqrt {1-\beta _{t))}x_{t-1}\|^{2}+C}$$

${\displaystyle C}$

${\displaystyle x_{1:T}|x_{0))$

$${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt ((\bar {\alpha ))_{t))}x_{0},(1-{\bar {\alpha ))_{t})I\right)}$$

$${\displaystyle x_{t-1}|x_{t},x_{0}\sim N({\tilde {\mu ))_{t}(x_{t},x_{0}),{\tilde {\beta ))_{t}I)}$$

${\displaystyle t}$

${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt ((\bar {\alpha ))_{t))}x_{0},(1-{\bar {\alpha ))_{t})I\right)}$

${\displaystyle N(0,I)}$

${\displaystyle x_{T))$

${\displaystyle N(0,I)}$

${\displaystyle x_{0}\sim q}$

For example, since

$${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt ((\bar {\alpha ))_{t))}x_{0},(1-{\bar {\alpha ))_{t})I\right)}$$

${\displaystyle x_{t}|x_{0))$

${\displaystyle x_{1},x_{2},...,x_{t-1))$

Derivation by reparameterization

We know ${\textstyle x_{t-1}|x_{0))$ is a gaussian, and ${\textstyle x_{t}|x_{t-1))$ is another gaussian. We also know that these are independent. Thus we can perform a reparameterization:

$${\displaystyle x_{t-1}={\sqrt ((\bar {\alpha ))_{t-1))}x_{0}+{\sqrt {1-{\bar {\alpha ))_{t-1))}z}$$

$${\displaystyle x_{t}={\sqrt {\alpha _{t))}x_{t-1}+{\sqrt {1-\alpha _{t))}z'}$$

where ${\textstyle z,z'}$ are IID gaussians.

There are 5 variables ${\textstyle x_{0},x_{t-1},x_{t},z,z'}$ and two linear equations. The two sources of randomness are ${\textstyle z,z'}$, which can be reparameterized by rotation, since the IID gaussian distribution is rotationally symmetric.

By plugging in the equations, we can solve for the first reparameterization:

$${\displaystyle x_{t}={\sqrt ((\bar {\alpha ))_{t))}x_{0}+\underbrace ((\sqrt {\alpha _{t}-{\bar {\alpha ))_{t))}z+{\sqrt {1-\alpha _{t))}z'} _{={\sqrt {1-{\bar {\alpha ))_{t))}z''))$$

where ${\textstyle z''}$ is a gaussian with mean zero and variance one.

To find the second one, we complete the rotational matrix:

$${\displaystyle {\begin{bmatrix}z''\\z'''\end{bmatrix))={\begin{bmatrix}{\frac {\sqrt {\alpha _{t}-{\bar {\alpha ))_{t))}{\sqrt {1-{\bar {\alpha ))_{t))))&{\frac {\sqrt {\beta _{t))}{\sqrt {1-{\bar {\alpha ))_{t))))\\?&?\end{bmatrix)){\begin{bmatrix}z\\z'\end{bmatrix))}$$

Since rotational matrices are all of the form ${\textstyle {\begin{bmatrix}\cos \theta &\sin \theta \\-\sin \theta &\cos \theta \end{bmatrix))}$, we know the matrix must be

$${\displaystyle {\begin{bmatrix}z''\\z'''\end{bmatrix))={\begin{bmatrix}{\frac {\sqrt {\alpha _{t}-{\bar {\alpha ))_{t))}{\sqrt {1-{\bar {\alpha ))_{t))))&{\frac {\sqrt {\beta _{t))}{\sqrt {1-{\bar {\alpha ))_{t))))\\-{\frac {\sqrt {\beta _{t))}{\sqrt {1-{\bar {\alpha ))_{t))))&{\frac {\sqrt {\alpha _{t}-{\bar {\alpha ))_{t))}{\sqrt {1-{\bar {\alpha ))_{t))))\end{bmatrix)){\begin{bmatrix}z\\z'\end{bmatrix))}$$

and since the inverse of rotational matrix is its transpose,

$${\displaystyle {\begin{bmatrix}z\\z'\end{bmatrix))={\begin{bmatrix}{\frac {\sqrt {\alpha _{t}-{\bar {\alpha ))_{t))}{\sqrt {1-{\bar {\alpha ))_{t))))&-{\frac {\sqrt {\beta _{t))}{\sqrt {1-{\bar {\alpha ))_{t))))\\{\frac {\sqrt {\beta _{t))}{\sqrt {1-{\bar {\alpha ))_{t))))&{\frac {\sqrt {\alpha _{t}-{\bar {\alpha ))_{t))}{\sqrt {1-{\bar {\alpha ))_{t))))\end{bmatrix)){\begin{bmatrix}z''\\z'''\end{bmatrix))}$$

Plugging back, and simplifying, we have

$${\displaystyle x_{t}={\sqrt ((\bar {\alpha ))_{t))}x_{0}+{\sqrt {1-{\bar {\alpha ))_{t))}z''}$$

$${\displaystyle x_{t-1}={\tilde {\mu ))_{t}(x_{t},x_{0})-{\sqrt ((\tilde {\beta ))_{t))}z'''}$$

Backward diffusion

The key idea of DDPM is to use a neural network parametrized by ${\displaystyle \theta }$. The network takes in two arguments ${\displaystyle x_{t},t}$, and outputs a vector ${\displaystyle \mu _{\theta }(x_{t},t)}$ and a matrix ${\displaystyle \Sigma _{\theta }(x_{t},t)}$, such that each step in the forward diffusion process can be approximately undone by ${\displaystyle x_{t-1}\sim N(\mu _{\theta }(x_{t},t),\Sigma _{\theta }(x_{t},t))}$. This then gives us a backward diffusion process ${\displaystyle p_{\theta ))$ defined by

$${\displaystyle p_{\theta }(x_{T})=N(x_{T}|0,I)}$$

$${\displaystyle p_{\theta }(x_{t-1}|x_{t})=N(x_{t-1}|\mu _{\theta }(x_{t},t),\Sigma _{\theta }(x_{t},t))}$$

${\displaystyle p_{\theta }(x_{0})}$

${\displaystyle q(x_{0})}$

Variational inference

The ELBO inequality states that ${\displaystyle \ln p_{\theta }(x_{0})\geq E_{x_{1:T}\sim q(\cdot |x_{0})}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_{0})]}$, and taking one more expectation, we get

$${\displaystyle E_{x_{0}\sim q}[\ln p_{\theta }(x_{0})]\geq E_{x_{0:T}\sim q}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_{0})]}$$

Define the loss function

$${\displaystyle L(\theta ):=-E_{x_{0:T}\sim q}[\ln p_{\theta }(x_{0:T})-\ln q(x_{1:T}|x_{0})]}$$

$${\displaystyle L(\theta )=\sum _{t=1}^{T}E_{x_{t-1},x_{t}\sim q}[-\ln p_{\theta }(x_{t-1}|x_{t})]+E_{x_{0}\sim q}[D_{KL}(q(x_{T}|x_{0})\|p_{\theta }(x_{T}))]+C}$$

${\displaystyle C}$

${\displaystyle p_{\theta }(x_{T})=N(x_{T}|0,I)}$

${\displaystyle E_{x_{0}\sim q}[D_{KL}(q(x_{T}|x_{0})\|p_{\theta }(x_{T}))]}$

${\displaystyle L(\theta )=\sum _{t=1}^{T}L_{t))$

${\displaystyle L_{t}=E_{x_{t-1},x_{t}\sim q}[-\ln p_{\theta }(x_{t-1}|x_{t})]}$

Noise prediction network

Since ${\displaystyle x_{t-1}|x_{t},x_{0}\sim N({\tilde {\mu ))_{t}(x_{t},x_{0}),{\tilde {\beta ))_{t}I)}$, this suggests that we should use ${\displaystyle \mu _{\theta }(x_{t},t)={\tilde {\mu ))_{t}(x_{t},x_{0})}$; however, the network does not have access to ${\displaystyle x_{0))$, and so it has to estimate it instead. Now, since ${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt ((\bar {\alpha ))_{t))}x_{0},(1-{\bar {\alpha ))_{t})I\right)}$, we may write ${\displaystyle x_{t}={\sqrt ((\bar {\alpha ))_{t))}x_{0}+{\sqrt {1-{\bar {\alpha ))_{t))}z}$, where ${\displaystyle z}$ is some unknown gaussian noise. Now we see that estimating ${\displaystyle x_{0))$ is equivalent to estimating ${\displaystyle z}$.

Therefore, let the network output a noise vector ${\displaystyle \epsilon _{\theta }(x_{t},t)}$, and let it predict

$${\displaystyle \mu _{\theta }(x_{t},t)={\tilde {\mu ))_{t}\left(x_{t},{\frac {x_{t}-{\sqrt {1-{\bar {\alpha ))_{t))}\epsilon _{\theta }(x_{t},t)}{\sqrt ((\bar {\alpha ))_{t))))\right)={\frac {x_{t}-\epsilon _{\theta }(x_{t},t)\beta _{t}/{\sqrt {1-{\bar {\alpha ))_{t)))){\sqrt {\alpha _{t))))}$$

${\displaystyle \Sigma _{\theta }(x_{t},t)}$

${\displaystyle \Sigma _{\theta }(x_{t},t)=\sigma _{t}^{2}I}$

${\displaystyle \sigma _{t}^{2}=\beta _{t}{\text{ or )){\tilde {\beta ))_{t))$

With this, the loss simplifies to

$${\displaystyle L_{t}={\frac {\beta _{t}^{2)){2\alpha _{t}(1-{\bar {\alpha ))_{t})\sigma _{t}^{2))}E_{x_{0}\sim q;z\sim N(0,I)}\left[\left\|\epsilon _{\theta }(x_{t},t)-z\right\|^{2}\right]+C}$$

$${\displaystyle L_{simple,t}=E_{x_{0}\sim q;z\sim N(0,I)}\left[\left\|\epsilon _{\theta }(x_{t},t)-z\right\|^{2}\right]}$$

Score-based generative model

Score-based generative model is another formulation of diffusion modelling. They are also called noise conditional score network (NCSN) or score-matching with Langevin dynamics (SMLD).^[13][14]

Score matching

The idea of score functions

Consider the problem of image generation. Let ${\displaystyle x}$ represent an image, and let ${\displaystyle q(x)}$ be the probability distribution over all possible images. If we have ${\displaystyle q(x)}$ itself, then we can say for certain how likely a certain image is. However, this is intractable in general.

Most often, we are uninterested in knowing the absolute probability of a certain image. Instead, we are usually only interested in knowing how likely a certain image is compared to its immediate neighbors — e.g. how much more likely is an image of cat compared to some small variants of it? Is it more likely if the image contains two whiskers, or three, or with some Gaussian noise added?

Consequently, we are actually quite uninterested in ${\displaystyle q(x)}$ itself, but rather, ${\displaystyle \nabla _{x}\ln q(x)}$. This has two major effects:

• One, we no longer need to normalize ${\displaystyle q(x)}$, but can use any ${\displaystyle {\tilde {q))(x)=Cq(x)}$, where ${\displaystyle C=\int {\tilde {q))(x)dx>0}$ is any unknown constant that is of no concern to us.
• Two, we are comparing ${\displaystyle q(x)}$ neighbors ${\displaystyle q(x+dx)}$, by ${\displaystyle {\frac {q(x)}{q(x+dx)))=e^{-\langle \nabla _{x}\ln q,dx\rangle ))$

Let the score function be ${\displaystyle s(x):=\nabla _{x}\ln q(x)}$; then consider what we can do with ${\displaystyle s(x)}$.

As it turns out, ${\displaystyle s(x)}$ allows us to sample from ${\displaystyle q(x)}$ using thermodynamics. Specifically, if we have a potential energy function ${\displaystyle U(x)=-\ln q(x)}$, and a lot of particles in the potential well, then the distribution at thermodynamic equilibrium is the Boltzmann distribution ${\displaystyle q_{U}(x)\propto e^{-U(x)/k_{B}T}=q(x)^{1/k_{B}T))$. At temperature ${\displaystyle k_{B}T=1}$, the Boltzmann distribution is exactly ${\displaystyle q(x)}$.

Therefore, to model ${\displaystyle q(x)}$, we may start with a particle sampled at any convenient distribution (such as the standard gaussian distribution), then simulate the motion of the particle forwards according to the Langevin equation

$${\displaystyle dx_{t}=-\nabla _{x_{t))U(x_{t})dt+dW_{t))$$

${\displaystyle x_{0))$

${\displaystyle x_{t))$

${\displaystyle q}$

${\displaystyle t\to \infty }$

Learning the score function

Given a density ${\displaystyle q}$, we wish to learn a score function approximation ${\displaystyle f_{\theta }\approx \nabla \ln q}$. This is score matching.^[15] Typically, score matching is formalized as minimizing Fisher divergence function ${\displaystyle E_{q}[\|f_{\theta }(x)-\nabla \ln q(x)\|^{2}]}$. By expanding the integral, and performing an integration by parts,

$${\displaystyle E_{q}[\|f_{\theta }(x)-\nabla \ln q(x)\|^{2}]=E_{q}[\|f_{\theta }\|^{2}+2\nabla ^{2}\cdot f_{\theta }]+C}$$

Annealing the score function

Suppose we need to model the distribution of images, and we want ${\displaystyle x_{0}\sim N(0,I)}$, a white-noise image. Now, most white-noise images do not look like real images, so ${\displaystyle q(x_{0})\approx 0}$ for large swaths of ${\displaystyle x_{0}\sim N(0,I)}$. This presents a problem for learning the score function, because if there are no samples around a certain point, then we can't learn the score function at that point. If we do not know the score function ${\displaystyle \nabla _{x_{t))\ln q(x_{t})}$ at that point, then we cannot impose the time-evolution equation on a particle:

$${\displaystyle dx_{t}=\nabla _{x_{t))\ln q(x_{t})dt+dW_{t))$$

${\displaystyle q}$

Continuous diffusion processes

Forward diffusion process

Consider again the forward diffusion process, but this time in continuous time:

$${\displaystyle x_{t}={\sqrt {1-\beta _{t))}x_{t-1}+{\sqrt {\beta _{t))}z_{t))$$

${\displaystyle \beta _{t}\to \beta (t)dt,{\sqrt {dt))z_{t}\to dW_{t))$

$${\displaystyle dx_{t}=-{\frac {1}{2))\beta (t)x_{t}dt+{\sqrt {\beta (t)))dW_{t))$$

${\displaystyle W_{t))$

Now, the equation is exactly a special case of the overdamped Langevin equation

$${\displaystyle dx_{t}=-{\frac {D}{k_{B}T))(\nabla _{x}U)dt+{\sqrt {2D))dW_{t))$$

${\displaystyle D}$

${\displaystyle T}$

${\displaystyle U}$

${\displaystyle D={\frac {1}{2))\beta (t)I,k_{B}T=1,U={\frac {1}{2))\|x\|^{2))$

Now the above equation is for the stochastic motion of a single particle. Suppose we have a cloud of particles distributed according to ${\displaystyle q}$ at time ${\displaystyle t=0}$, then after a long time, the cloud of particles would settle into the stable distribution of ${\displaystyle N(0,I)}$. Let ${\displaystyle \rho _{t))$ be the density of the cloud of particles at time ${\displaystyle t}$, then we have

$${\displaystyle \rho _{0}=q;\quad \rho _{T}\approx N(0,I)}$$

By Fokker-Planck equation, the density of the cloud evolves according to

$${\displaystyle \partial _{t}\ln \rho _{t}={\frac {1}{2))\beta (t)\left(n+(x+\nabla \ln \rho _{t})\cdot \nabla \ln \rho _{t}+\Delta \ln \rho _{t}\right)}$$

${\displaystyle n}$

${\displaystyle \Delta }$

Backward diffusion process

If we have solved ${\displaystyle \rho _{t))$ for time ${\displaystyle t\in [0,T]}$, then we can exactly reverse the evolution of the cloud. Suppose we start with another cloud of particles with density ${\displaystyle \nu _{0}=\rho _{T))$, and let the particles in the cloud evolve according to

$${\displaystyle dy_{t}={\frac {1}{2))\beta (T-t)y_{t}dt+\beta (T-t)\underbrace {\nabla _{y_{t))\ln \rho _{T-t}\left(y_{t}\right)} _{\text{score function ))dt+{\sqrt {\beta (T-t)))dW_{t))$$

${\displaystyle \partial _{t}\rho _{T-t}=\partial _{t}\nu _{t))$

Noise conditional score network (NCSN)

At the continuous limit,

$${\displaystyle {\bar {\alpha ))_{t}=(1-\beta _{1})\cdots (1-\beta _{t})=e^{\sum _{i}\ln(1-\beta _{i})}\to e^{-\int _{0}^{t}\beta (t)dt))$$

$${\displaystyle x_{t}|x_{0}\sim N\left(e^{-{\frac {1}{2))\int _{0}^{t}\beta (t)dt}x_{0},\left(1-e^{-\int _{0}^{t}\beta (t)dt}\right)I\right)}$$

${\displaystyle x_{0}\sim q,z\sim N(0,I)}$

${\displaystyle x_{t}=e^{-{\frac {1}{2))\int _{0}^{t}\beta (t)dt}x_{0}+\left(1-e^{-\int _{0}^{t}\beta (t)dt}\right)z}$

${\displaystyle x_{t}\sim \rho _{t))$

${\displaystyle t\geq 0}$

Now, define a certain probability distribution ${\displaystyle \gamma }$ over ${\displaystyle [0,\infty )}$, then the score-matching loss function is defined as the expected Fisher divergence:

$${\displaystyle L(\theta )=E_{t\sim \gamma ,x_{t}\sim \rho _{t))[\|f_{\theta }(x_{t},t)\|^{2}+2\nabla \cdot f_{\theta }(x_{t},t)]}$$

${\displaystyle f_{\theta }(x_{t},t)\approx \nabla \ln \rho _{t))$

${\displaystyle x_{T}\sim N(0,I)}$

${\displaystyle t=T}$

${\displaystyle t=0}$

$${\displaystyle x_{t-dt}=x_{t}+{\frac {1}{2))\beta (t)x_{t}dt+\beta (t)f_{\theta }(x_{t},t)dt+{\sqrt {\beta (t)))dW_{t))$$

The name "noise conditional score network" is explained thus:

• "network", because ${\displaystyle f_{\theta ))$ is implemented as a neural network.
• "score", because the output of the network is interpreted as approximating the score function ${\displaystyle \nabla \ln \rho _{t))$.
• "noise conditional", because ${\displaystyle \rho _{t))$ is equal to ${\displaystyle \rho _{0))$ blurred by an added gaussian noise that increases with time, and so the score function depends on the amount of noise added.

Their equivalence

DDPM and score-based generative model are equivalent.^[17] This means that a network trained using DDPM can be used as a NCSN, and vice versa.

We know that ${\displaystyle x_{t}|x_{0}\sim N\left({\sqrt ((\bar {\alpha ))_{t))}x_{0},(1-{\bar {\alpha ))_{t})I\right)}$, so by Tweedie's formula, we have

$${\displaystyle \nabla _{x_{t))\ln q(x_{t})={\frac {1}{1-{\bar {\alpha ))_{t))}(-x_{t}+{\sqrt ((\bar {\alpha ))_{t))}E_{q}[x_{0}|x_{t}])}$$

${\displaystyle \sum _{t}L_{simple,t))$

$${\displaystyle L_{simple,t}=E_{x_{0}\sim q;z\sim N(0,I)}\left[\left\|\epsilon _{\theta }(x_{t},t)-z\right\|^{2}\right]}$$

${\displaystyle x_{t}={\sqrt ((\bar {\alpha ))_{t))}x_{0}+{\sqrt {1-{\bar {\alpha ))_{t))}z}$

$${\displaystyle L_{simple,t}=E_{x_{0},x_{t}\sim q}\left[\left\|\epsilon _{\theta }(x_{t},t)-{\frac {x_{t}-{\sqrt ((\bar {\alpha ))_{t))}x_{0)){\sqrt {1-{\bar {\alpha ))_{t))))\right\|^{2}\right]=E_{x_{t}\sim q,x_{0}\sim q(\cdot |x_{t})}\left[\left\|\epsilon _{\theta }(x_{t},t)-{\frac {x_{t}-{\sqrt ((\bar {\alpha ))_{t))}x_{0)){\sqrt {1-{\bar {\alpha ))_{t))))\right\|^{2}\right]}$$

${\displaystyle \epsilon _{\theta }(x_{t},t)={\frac {x_{t}-{\sqrt ((\bar {\alpha ))_{t))}E_{q}[x_{0}|x_{t}]}{\sqrt {1-{\bar {\alpha ))_{t))))=-{\sqrt {1-{\bar {\alpha ))_{t))}\nabla _{x_{t))\ln q(x_{t})}$

Now, the continuous limit ${\displaystyle x_{t-1}=x_{t-dt},\beta _{t}=\beta (t)dt,z_{t}{\sqrt {dt))=dW_{t))$ of the backward equation

$${\displaystyle x_{t-1}={\frac {x_{t)){\sqrt {\alpha _{t))))-{\frac {\beta _{t)){\sqrt {\alpha _{t}(1-{\bar {\alpha ))_{t})))}\epsilon _{\theta }(x_{t},t)+{\sqrt {\beta _{t))}z_{t};\quad z_{t}\sim N(0,I)}$$

$${\displaystyle x_{t-dt}=x_{t}(1+\beta (t)dt/2)+\beta (t)\nabla _{x_{t))\ln q(x_{t})dt+{\sqrt {\beta (t)))dW_{t))$$

Main variants

Denoising Diffusion Implicit Model (DDIM)

The original DDPM method for generating images is slow, since the forward diffusion process usually takes ${\displaystyle T\sim 1000}$ to make the distribution of ${\displaystyle x_{T))$ to appear close to gaussian. However this means the backward diffusion process also take 1000 steps. Unlike the forward diffusion process, which can skip steps as ${\displaystyle x_{t}|x_{0))$ is gaussian for all ${\displaystyle t\geq 1}$, the backward diffusion process does not allow skipping steps. For example, to sample ${\displaystyle x_{t-2}|x_{t-1}\sim N(\mu _{\theta }(x_{t-1},t-1),\Sigma _{\theta }(x_{t-1},t-1))}$ requires the model to first sample ${\displaystyle x_{t-1))$. Attempting to directly sample ${\displaystyle x_{t-2}|x_{t))$ would require us to marginalize out ${\displaystyle x_{t-1))$, which is generally intractable.

DDIM^[18] is a method to take any model trained on DDPM loss, and use it to sample with some steps skipped, sacrificing an adjustable amount of quality. If we generate the Markovian chain case in DDPM to non-Markovian case, DDIM corresponds to the case that the reverse process has variance equals to 0. In other words, the reverse process (and also the forward process) is deterministic. When using less sampling steps, DDIM outperforms DDPM.

Latent diffusion model (LDM)

Since the diffusion model is a general method for modelling probability distributions, if one wants to model a distribution over images, one can first encode the images into a lower-dimensional space by an encoder, then use a diffusion model to model the distribution over encoded images. Then to generate an image, one can sample from the diffusion model, then use a decoder to decode it into an image.^[19]

The encoder-decoder pair is most often a variational autoencoder (VAE).

Classifier guidance

Suppose we wish to sample not from the entire distribution of images, but conditional on the image description. We don't want to sample a generic image, but an image that fits the description "black cat with red eyes". Generally, we want to sample from the distribution ${\displaystyle p(x|y)}$, where ${\displaystyle x}$ ranges over images, and ${\displaystyle y}$ ranges over classes of images (a description "black cat with red eyes" is just a very detailed class, and a class "cat" is just a very vague description).

Taking the perspective of the noisy channel model, we can understand the process as follows: To generate an image ${\displaystyle x}$ conditional on description ${\displaystyle y}$, we imagine that the requester really had in mind an image ${\displaystyle x}$, but the image is passed through a noisy channel and came out garbled, as ${\displaystyle y}$. Image generation is then nothing but inferring which ${\displaystyle x}$ the requester had in mind.

In other words, conditional image generation is simply "translating from a textual language into a pictorial language". Then, as in noisy-channel model, we use Bayes theorem to get

$${\displaystyle p(x|y)\propto p(y|x)p(x)}$$

${\displaystyle \nabla \ln p(x)}$

$${\displaystyle \nabla _{x}\ln p(x|y)=\nabla _{x}\ln p(y|x)+\nabla _{x}\ln p(x)}$$

${\displaystyle \nabla _{x}\ln p(x)}$

${\displaystyle \nabla _{x}\ln p(y|x)}$

With temperature

The classifier-guided diffusion model samples from ${\displaystyle p(x|y)}$, which is concentrated around the maximum a posteriori estimate ${\displaystyle \arg \max _{x}p(x|y)}$. If we want to force the model to move towards the maximum likelihood estimate ${\displaystyle \arg \max _{x}p(y|x)}$, we can use

$${\displaystyle p_{\beta }(x|y)\propto p(y|x)^{\beta }p(x)}$$

${\displaystyle \beta >0}$

${\displaystyle \beta }$

${\displaystyle \arg \max _{x}p(y|x)}$

This can be done simply by SGLD with

$${\displaystyle \nabla _{x}\ln p_{\beta }(x|y)=\beta \nabla _{x}\ln p(y|x)+\nabla _{x}\ln p(x)}$$

Classifier-free guidance (CFG)

If we do not have a classifier ${\displaystyle p(y|x)}$, we could still extract one out of the image model itself:^[21]

$${\displaystyle \nabla _{x}\ln p_{\beta }(x|y)=(1-\beta )\nabla _{x}\ln p(x)+\beta \nabla _{x}\ln p(x|y)}$$

${\displaystyle (x,y)}$

${\displaystyle (x,{\rm {None)))}$

${\displaystyle \nabla _{x}\ln p(x|y)}$

${\displaystyle \nabla _{x}\ln p(x)}$

Samplers

Given a diffusion model, one may regard it either as a continuous process, and sample from it by integrating a SDE, or one can regard it as a discrete process, and sample from it by iterating the discrete steps. The choice of the "noise schedule" ${\displaystyle \beta _{t))$ can also affect the quality of samples. In the DDPM perspective, one can use the DDPM itself (with noise), or DDIM (with adjustable amount of noise). The case where one adds noise is sometimes called ancestral sampling.^[22] One can interpolate between noise and no noise. The amount of noise is denoted ${\displaystyle \eta }$ ("eta value") in the DDIM paper, with ${\displaystyle \eta =0}$ denoting no noise (as in deterministic DDIM), and ${\displaystyle \eta =1}$ denoting full noise (as in DDPM).

In the perspective of SDE, one can use any of the numerical integration methods, such as Euler–Maruyama method, Heun's method, linear multistep methods, etc. Just as in the discrete case, one can add an adjustable amount of noise during the integration.

A survey and comparison of samplers in the context of image generation is in.^[23]

Rectified flow

Rectified flow is a method for learning transport maps between two distributions, offers a new perspective on understanding diffusion models and their ODE variants. Distinct from the complex SDE models, rectified flow is purely ODE-based, offering a straightforward and unified framework for generative and transfer modeling. Given the infinite possibilities of ODEs/SDEs to transfer data between two distributions, rectified flow specifically advocates for ODEs with solution paths that are straight lines. By learning straight flows, it provides a principled approach to learning ODEs with fast inference, effectively training one-step models with ODEs as intermediate steps. Rectified flow formulation is employed in Stable Diffusion 3. ^[24]

Given two distributions ${\displaystyle \pi _{0))$ and ${\displaystyle \pi _{1))$, probability flow models implicitly learns the transport map by constructing an ODE driven by a drift force in ${\displaystyle \mathbb {R} ^{d}\times [0,1]}$:

$${\displaystyle \mathrm {d} \mathbf {Z} _{t}=\mathbf {v} (\mathbf {Z} _{t},t)\,\mathrm {d} t,\quad t\in [0,1],\quad {\text{starting from ))\mathbf {Z} _{0}\sim \mathbf {\pi } _{0))$$

such that ${\displaystyle \mathbf {Z} _{1}\sim \pi _{1))$ when following the ODE starting from ${\displaystyle \mathbf {Z} _{0}\sim \pi _{0))$. Generally, for any time-differentiable process ${\displaystyle \mathbf {X} (t)}$, ${\displaystyle \mathbf {v} }$ can be estimated by solving:

$${\displaystyle \min _{\mathbf {v} }\int _{0}^{1}\mathbb {E} \left[\lVert ((\dot {\mathbf {X} ))_{t}-\mathbf {v} (\mathbf {X} _{t},t)}\rVert ^{2}\right]\,\mathrm {d} t.}$$

In rectified flow, by injecting strong priors that intermediate trajectories are straight, it can achieve both theoretical relevance for optimal transport ^[25] and computational efficiency, as ODEs with straight paths can be simulated precisely without time discretization.

Specifically, rectified flow seeks to match an ODE with the marginal distributions of the linear interpolation between points from distributions ${\displaystyle \pi _{0))$ and ${\displaystyle \pi _{1))$^[26]. Given observations ${\displaystyle \mathbf {X} _{0}\sim \pi _{0))$ and ${\displaystyle \mathbf {X} _{1}\sim \pi _{1))$, the canonical linear interpolation ${\displaystyle \mathbf {X} _{t}=t\mathbf {X} _{1}+(1-t)\mathbf {X} _{0},t\in [0,1]}$ yields a trivial case ${\displaystyle {\dot {\mathbf {X} ))_{t}=\mathbf {X} _{1}-\mathbf {X} _{0))$, which cannot be causally simulated without ${\displaystyle \mathbf {X} _{1))$. To address this, ${\displaystyle \mathbf {X} _{t))$ is "projected" into a space of causally simulatable ODEs, expressed as ${\displaystyle \mathrm {d} \mathbf {Z} _{t}=\mathbf {v} (\mathbf {Z} _{t},t)}$, by minimizing the least squares loss with respect to the direction ${\displaystyle \mathbf {X} _{1}-\mathbf {X} _{0))$:

$${\displaystyle \min _{\mathbf {v} }\int _{0}^{1}\mathbb {E} \left[\lVert {(\mathbf {X} _{1}-\mathbf {X} _{0})-\mathbf {v} (\mathbf {X} _{t},t)}\rVert ^{2}\right]\,\mathrm {d} t.}$$

The data pair ${\displaystyle (\mathbf {X} _{0},\mathbf {X} _{1})}$ can be any coupling of ${\displaystyle \pi _{0))$ and ${\displaystyle \pi _{1))$, typically independent (i.e., ${\displaystyle (\mathbf {X} _{0},\mathbf {X} _{1})\sim \pi _{0}\times \pi _{1))$) obtained by randomly combining observations from ${\displaystyle \pi _{0))$ and ${\displaystyle \pi _{1))$. This process ensures that the ${\displaystyle \mathbf {Z} _{t))$ trajectories closely mirror the density map of ${\displaystyle \mathbf {X} _{t))$ trajectories but reroute at intersections to ensure causality. This rectifying process is also known as Flow Matching ^[27], Stochastic Interpolation ^[28], and Alpha-Blending ^[29].

A distinctive aspect of rectified flow is its capability for "reflow", which straightens the trajectory of ODE paths. Denote the rectified flow ${\displaystyle {\boldsymbol {Z))^{0}=\{\mathbf {Z} _{t}:t\in [0,1]\))$ induced from ${\displaystyle (\mathbf {X} _{0},\mathbf {X} _{1})}$ as ${\displaystyle {\boldsymbol {Z))^{0}={\mathsf {Rectflow))((\mathbf {X} _{0},\mathbf {X} _{1}))}$. Recursively applying this ${\displaystyle {\mathsf {Rectflow))(\cdot )}$ operator generates a series of rectified flows ${\displaystyle {\boldsymbol {Z))^{k+1}={\mathsf {Rectflow))((\mathbf {Z} _{0}^{k},\mathbf {Z} _{1}^{k}))}$, starting with ${\displaystyle (\mathbf {Z} _{0}^{0},\mathbf {Z} _{1}^{0})=(\mathbf {X} _{0},\mathbf {X} _{1})}$, where ${\displaystyle {\boldsymbol {Z))^{k))$ is the ${\displaystyle k}$-th iteration of rectified flow induced from ${\displaystyle (\mathbf {X} _{0},\mathbf {X} _{1})}$. This "reflow" process not only reduces transport costs but also straightens the paths of rectified flows, making ${\displaystyle {\boldsymbol {Z))^{k))$ paths straighter with increasing ${\displaystyle k}$.

Rectified flow includes a nonlinear extension where linear interpolation ${\displaystyle \mathbf {X} _{t))$ is replaced with any time-differentiable curve that connects ${\displaystyle \mathbf {X} _{0))$ and ${\displaystyle \mathbf {X} _{1))$, given by ${\displaystyle \mathbf {X} _{t}=\alpha _{t}\mathbf {X} _{1}+\beta _{t}\mathbf {X} _{0))$. This framework encompasses DDIM and probability flow ODEs as special cases, with particular choices of ${\displaystyle \alpha _{t))$ and ${\displaystyle \beta _{t))$. However, in the case where the path of ${\displaystyle \mathbf {X} }$ is not straight, the reflow process no longer ensures a reduction in convex transport costs, and also no longer straighten the paths of ${\displaystyle \mathbf {Z} _{t))$ ^[26].