# A Comprehensive Survey on Knowledge Distillation of Diffusion Models

Weijian Luo<EMAIL_ADDRESS>
School of Mathematical Sciences
Peking University
Beijing, 100871, China

###### Abstract

Diffusion Models (DMs), also referred to as score-based diffusion models,
utilize neural networks to specify score functions. Unlike most other
probabilistic models, DMs directly model the score functions, which makes them
more flexible to parametrize and potentially highly expressive for
probabilistic modeling. DMs can learn fine-grained knowledge, i.e., marginal
score functions, of the underlying distribution. Therefore, a crucial research
direction is to explore how to distill the knowledge of DMs and fully utilize
their potential. Our objective is to provide a comprehensible overview of the
modern approaches for distilling DMs, starting with an introduction to DMs and
a discussion of the challenges involved in distilling them into neural vector
fields. We also provide an overview of the existing works on distilling DMs
into both stochastic and deterministic implicit generators. Finally, we review
the accelerated diffusion sampling algorithms as a training-free method for
distillation. Our tutorial is intended for individuals with a basic
understanding of generative models who wish to apply DM’s distillation or
embark on a research project in this field.

##  1 Introduction

###  1.1 ARMs: Model Explicit Likelihood Functions

The objective of deep generative modeling is to train neural-parametrized
models that can generate highly realistic data samples. Numerous deep
generative models have been proposed to achieve this objective, each from a
different perspective. In general, generative models aim to express and
approximate certain sufficient characteristics of the underlying data
distribution by minimizing probability divergences or metrics. The likelihood
function, which is the log density of the underlying distribution, is a
commonly used distributional characteristic. Auto-regressive models (ARMs)
(Graves, 2013; Van Den Oord et al., 2016; Jozefowicz et al., 2016) are
representative models that use neural networks to parametrize the log-
likelihood function, i.e., the logarithm of probability density functions, and
learn to match the underlying data’s log-likelihood. ARMs are trained using KL
divergence minimization. Let $p_{d}$ denote the underlying data distribution,
from which we only have access to consistent samples, i.e., $x\sim p_{d}$.
ARMs sum up a sequence of outputs of neural networks with strict orders to
explicitly express the conditional factorization of the model’s likelihood
function which writes

$\displaystyle
p_{\theta}(x)=f_{\theta}(x^{(1)})+\Sigma_{i=2}^{D}f_{\theta}(x^{(i)}|x^{(1)},...,x^{(i-1)}).$
(1)

Here $x^{(i)}$ represents the $i$-th coordinates of a point
$x\in\mathbb{R}^{D}$ following some strict order. If $x$ is continuous and in
a Euclidean space, the term $f_{\theta}(x^{(i)}|x^{(1)},...,x^{(i-1)})$ is
often implemented via the conditional distribution of Gaussian distribution
with mean and variance obtained from outputs of neural networks as

$\displaystyle
f_{\theta}(x^{(i)}|x^{(1)},...,x^{(i-1)})=\log\mathcal{N}(x^{(i)};\mu_{\theta}(x^{(1)},..,x^{(i-1)}),\Sigma_{\theta}(x^{(1)},..,x^{(i-1)})).$
(2)

Here $\mathcal{N}(x;\mu,\Sigma)$ represents the density functions of
multivariate distribution with mean $\mu$ and covariance matrix $\Sigma$. The
terms $\mu^{(i)}_{\theta}$ and $\Sigma^{(i)}_{\theta}$ are usually implemented
as the outputs of some neural networks. For discrete data, the conditional
distribution in equation 1 is usually implemented via a soft-max output of
some neural networks with the expression

$\displaystyle
f_{\theta}(x^{(i)}|x^{(1)},...,x^{(i-1)})=\operatorname{Softmax}(g_{\theta}(x^{(1)},..,x^{(i-1)})).$
(3)

Here $g_{\theta}(.)$ is a neural network whose output has the same dimension
as $x^{(i)}$’s discrete range.

The KL divergence between $p_{d}$ and $p_{\theta}$ is defined as

$\displaystyle\mathcal{D}_{KL}(p_{d},p_{\theta})\coloneqq\mathbb{E}_{x\sim
p_{d}}\big{[}\log p_{d}(x)-\log
p_{\theta}(x)\big{]}=\mathbb{E}_{p_{d}}\big{[}\log
p_{d}(x)\big{]}-\mathbb{E}_{p_{d}}\big{[}\log p_{\theta}(x)\big{]}.$ (4)

Since the first term of equation 4 does not depend on parameters $\theta$,
training ARMs by minimizing the KL divergence between $p_{d}$ and $p_{\theta}$
is equivalent to

$\displaystyle\theta^{*}=\arg\min_{\theta}\mathcal{D}_{KL}(p_{d},p_{\theta})=\arg\max_{\theta}\mathbb{E}_{p_{d}}\big{[}\log
p_{\theta}(x)\big{]}.$ (5)

The optimal $\theta^{*}$ in problem equation 5 is also called a maximum
likelihood estimation (MLE) of the parameter $\theta$. The term
$\mathcal{L}(\theta)=\mathbb{E}_{p_{d}}\big{[}\log p_{\theta}(x)\big{]}$ is
the expected likelihood. Two important points about ARMs need to be
emphasized. Firstly, ARMs explicitly model log-likelihood functions (equation
2 and equation 3), which limits the implementation’s flexibility. Secondly,
ARMs require a strict sequential order in their generating algorithm, which
makes sampling from ARMs computationally inefficient.

###  1.2 EBMs: From Explicit Likelihoods To Unnormalized Ones

ARMs use neural networks to directly express normalized likelihood functions
through the conditional factorization formula. However, this normalization is
too limiting to unleash the full potential of neural networks. For instance,
the implementation equation 2 of continuous-valued ARMs constrains the
conditional distribution to be a multivariate Gaussian distribution, which may
not hold for real-world data distributions. This constraint prevents ARMs from
fully matching the data distribution even if the neural networks have infinite
expressive capacity. Energy-based models (EBMs) (LeCun et al., 2006; Zhu et
al., 1998), on the other hand, overcome this normalization issue by using
unconstrained neural networks to express and match the data’s potential
functions, i.e., the logarithm of the unnormalized density functions.
Formally, the potential function of a distribution $p$ is defined as a family
of functions that satisfy the equation

$\displaystyle e^{E(x)}/\int e^{E(x)}dx=p(x).$ (6)

Intuitively, the potential function $E(x)$ in EBMs represents the logarithm of
the unnormalized part of the density $p(x)$. Since the potential function does
not need to be normalized itself, EBMs use neural networks $E_{\theta}$ to
express model potential functions. The most basic training strategy of EBMs is
to also use the maximum likelihood training equation 5. Specifically, since
the EBM-induced distribution has the form
$p_{\theta}(x)=e^{E_{\theta}(x)}/Z_{\theta}$, where $Z_{\theta}=\int
e^{E_{\theta}(x)}dx$ is the normalizing constant, which is often challenging
to evaluate and is viewed as intractable in most cases. The expected
likelihood of EBM-induced distribution writes

$\displaystyle\mathcal{L}(\theta)=\mathbb{E}_{p_{d}}\big{[}\log
p_{\theta}(x)\big{]}=\mathbb{E}_{p_{d}}\big{[}E_{\theta}(x)\big{]}-\log
Z_{\theta}.$ (7)

Taking the parameter gradient of expected likelihood equation 7, we have

$\displaystyle\frac{\partial}{\partial\theta}\mathcal{L}(\theta)$
$\displaystyle=\mathbb{E}_{p_{d}}\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}-\frac{\partial}{\partial\theta}\log
Z_{\theta}$ (8)
$\displaystyle=\mathbb{E}_{p_{d}}\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}-\frac{1}{Z_{\theta}}\frac{\partial}{\partial\theta}Z_{\theta}$
(9)
$\displaystyle=\mathbb{E}_{p_{d}}\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}-\frac{1}{Z_{\theta}}\frac{\partial}{\partial\theta}\int
e^{E_{\theta}(x)}dx$ (10)
$\displaystyle=\mathbb{E}_{p_{d}}\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}-\frac{1}{Z_{\theta}}\int\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}e^{E_{\theta}(x)}dx$
(11)
$\displaystyle=\mathbb{E}_{p_{d}}\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}-\int\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}\frac{e^{E_{\theta}(x)}}{Z_{\theta}}dx$
(12)
$\displaystyle=\mathbb{E}_{p_{d}}\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}-\mathbb{E}_{p_{\theta}}\big{[}\frac{\partial}{\partial\theta}E_{\theta}(x)\big{]}.$
(13)

To calculate the expected log-likelihood gradient with respect to the
parameter $\theta$, we need to obtain consistent samples from the EBM-induced
distribution $x\sim p_{\theta}$, which is an un-normalized distribution.
Fortunately, several MCMC algorithms are capable of generating such samples,
including those proposed by Robert et al. (1999), Hastings (1970), Roberts and
Rosenthal (1998), Xifara et al. (2014), and Neal et al. (2011). By combining
the gradient formula equation 8 with MCMC algorithms, it is possible to train
EBMs using maximum likelihood estimation. The key distinction between energy-
based models (EBMs) and auto-regressive models (ARMs) lies in how they utilize
neural networks. While EBMs employ unconstrained neural networks to model
potential functions, ARMs use neural networks as a component of explicit
conditional densities. This difference in approach allows EBMs to tap into the
full expressive power of neural networks by avoiding the constraints imposed
by normalization requirements.

###  1.3 SBMs and DMs: From Potentials to Scores

In the preceding sections, we introduced EBMs and their training and sampling
methods. To approximate data potential functions, EBMs use neural networks and
generate samples using MCMC algorithms with learned potentials. Among the
various MCMC methods available, Langevin dynamics (LD) or Langevin MC (Roberts
and Tweedie, 1996) is a preferred choice for its ease of implementation and
good performance even under weak conditions. Let $p(x)$ be a differentiable
density function, the LD is defined through a stochastic differential
equation,

$\displaystyle dX_{t}=\frac{1}{2}\nabla_{X_{t}}\log
p(X_{t})dt+dW_{t},~{}p^{(0)}=p_{0},~{}t\in[0,\infty].$ (14)

Langevin dynamics (LD) is notable for two reasons. Firstly, when $t\to\infty$,
the marginal distribution of LD can converge to $p(x)$ regardless of the
initial distribution $p_{0}$, given certain conditions on $p(x)$. Secondly,
the simulation of LD only requires the gradient of the potential function, or
the score function $\nabla_{x}\log p(x)$. Hence, even if the distribution
$p(x)$ is not normalized, using the potential function $\log p(x)$ for LD
still generates valid samples as the normalized distribution. Score-based
models (SBMs) take inspiration from LD and utilize neural networks to train a
neural score function $S_{\theta}$ that can match the underlying data
distribution. If the neural score function is trained well enough to match the
data score functions $S_{d}(x)\coloneqq\nabla_{x}\log p_{d}(x)$, samples from
the SBM can be obtained by replacing the score function $\nabla_{x}\log p(x)$
with the SBM’s neural score function $S_{\theta}(x)$ in the simulation of LD.

Now, let’s turn our attention to the training strategies of SBMs. Since
$S_{\theta}$ directly expresses the model’s underlying score functions, the
model’s potential function $p_{\theta}$ becomes intractable. Therefore, the
commonly used KL divergence for training EBMs and ARMs as explained in
previous sections does not work for training SBMs. Instead, SBMs minimize the
Fisher divergence, a probability divergence that only requires the model’s
score functions. Formally, the Fisher divergence between two distributions
$p(x)$ and $q(x)$ is defined as

$\displaystyle\mathcal{D}_{F}(p,q)\coloneqq\mathbb{E}_{x\sim
p}\|\nabla_{x}\log p(x)-\nabla_{x}\log q(x)\|^{2}_{2}.$ (15)

Let $p_{d}$ denote the data distribution and $S_{d}(x)=\nabla_{x}\log
p_{d}(x)$ represents the data score function. The Fisher divergence between
data and SBM-induced distribution $p_{\theta}$ writes

$\displaystyle\mathcal{D}_{F}(p_{d},p_{\theta})$
$\displaystyle=\mathbb{E}_{p_{d}}\frac{1}{2}\|S_{d}(x)-S_{\theta}(x)\|_{2}^{2}$
(16)
$\displaystyle=\mathbb{E}_{p_{d}}\big{[}\frac{1}{2}\|S_{d}(x)\|_{2}^{2}\big{]}+\mathbb{E}_{p_{d}}\big{[}\frac{1}{2}\|S_{\theta}(x)\|_{2}^{2}\big{]}-\mathbb{E}_{p_{d}}\big{[}\langle
S_{d}(x),S_{\theta}(x)\rangle\big{]}.$ (17)

In practice, the first term of Fisher divergence equation 16 does not depend
on the parameter $\theta$ and thus can be dropped. The third term is shown to
be equivalent to a data-score-free form as

$\displaystyle\mathbb{E}_{p_{d}}\big{[}\langle
S_{d}(x),S_{\theta}(x)\rangle\big{]}=-\mathbb{E}_{p_{d}}\sum_{d=1}^{D}\frac{\partial
s_{\theta}^{(d)}(x)}{\partial x^{(d)}}.$

Here the notation $s_{\theta}^{(d)}(x)$ denotes the $d$-th component of the
model score function and $x^{(d)}$ denotes the $d$-th component of the input
vector. So minimizing the Fisher divergence is equivalent to minimizing a
tractable objective

$\displaystyle\mathcal{L}_{SM}(\theta)\colon=\mathbb{E}_{p_{d}}\big{[}\frac{1}{2}\|S_{\theta}(x)\|_{2}^{2}+\mathbb{E}_{p_{d}}\sum_{d=1}^{D}\frac{\partial
s_{\theta}^{(d)}(x)}{\partial x^{(d)}}\big{]}.$ (18)

The optimization problem in equation equation 18 is known as Score Matching
(SM) (Hyvärinen, 2005). However, evaluating the gradient term
$\langle\nabla_{x},S_{\theta}(x)\rangle$ by taking the data gradient through a
neural network can be memory-intensive, which poses challenges when working
with high-dimensional data. To overcome this limitation, several approaches
have been proposed to improve the efficiency of SM (Song et al., 2019; Pang et
al., 2020; Vincent, 2011). Among them, the seminal work (Vincent, 2011)
proposed a so-called denoising score matching (DSM) objective that does not
require taking data gradient but instead with the objective

$\displaystyle\mathcal{L}_{DSM}(\theta)\colon=\mathbb{E}_{x\sim
p_{d},\tilde{x}\sim
p(\tilde{x}|x)}\|S_{\theta}(\tilde{x})-\nabla_{\tilde{x}}\log
p(\tilde{x}|x)\|^{2}_{2}.$ (19)

In the above paragraph, the objective function $\mathcal{L}_{DSM}(\theta)$ is
minimized using a perturbation kernel $p(\tilde{x}|x)$, which is efficient to
sample and has an explicit expression. One common choice for the perturbation
kernel is a Gaussian distribution
$\mathcal{N}(\tilde{x};x,\sigma^{2}\mathbf{I})$ with a noise variance
$\sigma^{2}$. By minimizing this objective, we can obtain an approximation of
the data distribution $\tilde{p}_{d}(\tilde{x})$, which is similar to the
original data distribution if the perturbation kernel is not too different
from the identity. However, using the LD for sampling can be problematic for
high-dimensional data because the data is concentrated around some low-
dimensional manifold embedded in high-dimensional space. As a solution, score-
based diffusion models use multiple or continuously-indexed perturbation
kernels to improve both learning and sampling. This approach can improve the
performance of SBMs by generating samples that are more accurate and
representative of the original data distribution.

###  1.4 Diffusion Models: Multi-Level SBMs

In contrast to SBMs that use a single score network, score-based diffusion
models (DMs) (Song et al., 2020d) employ a more advanced approach by utilizing
a multiple-level or continuous-indexed score network $S_{\theta}(x,t)$.
Additionally, instead of a single perturbation kernel, DMs use a family of
conditional transition kernels induced by stochastic differential equations to
perturb the data. Consider a forward diffusion SDE

$\displaystyle dX_{t}=F(X_{t},t)dt+G(t)dW_{t},~{}X_{0}\sim p_{d}^{(0)}=p_{d},$
(20)

where $W_{t}$ is a Wiener process. Let $p_{t}(x_{t}|x_{0})$ denote the
conditional transition kernel of the forward diffusion 20, and $p_{d}^{(t)}$
denote the marginal distribution at diffusion time $t$, initialized with
$p_{d}^{(0)}=p_{d}$. Two special forward diffusions, variance preserving (VP)
diffusion, and variance exploding (VE) diffusion (Song et al., 2020d) are
favored across diffusion model literature.

Figure 1: Forward and Reversed SDE of Diffusion Models. The figure is taken
from Song et al. (2020d).

#### VP Diffusion

The VP diffusion takes the form of

$\displaystyle
dX_{t}=-\frac{1}{2}\beta(t)X_{t}dt+\sqrt{\beta(t)}dW_{t},~{}t\in[0,T],$ (21)

where $\beta(t)$ is a pre-defined schedule function. The conditional
transition kernel of VP diffusion has an explicit expression

$\displaystyle
p(x_{t}|x_{0})=\mathcal{N}(x_{t};\sqrt{\alpha_{t}}x_{0};(1-\alpha_{t})\mathbf{I}),$
(22)

where $\alpha_{t}=e^{-\int_{0}^{t}\beta(s)ds}$. With formula equation 22, the
simulation of samples of time $t$ is efficient, requiring only a scaling of an
initial sample and adding a Gaussian noise

$\displaystyle
X_{t}=\sqrt{\alpha_{t}}X_{0}+\sqrt{1-\alpha_{t}}\epsilon,~{}X_{0}\sim p_{0}.$
(23)

$\epsilon\sim\mathcal{N}(0,\mathbf{I})$ is a standard normal vector of the
same size as $X_{0}$. With formula equation 22, obtaining $X_{t}$ is cheap
because we do not need a sequential simulation of the diffusion process.
Another advantage of VP diffusion is that under loose conditions, VP diffusion
can transport arbitrary initial distribution $p_{0}$ to a standard multi-
variate Gaussian distribution $\mathcal{N}(0,\mathbf{I})$. VP diffusion is
perhaps the most widely used forward diffusion across DM literature .

#### VE Diffusion

The VE diffusion takes the form

$\displaystyle dX_{t}=\sqrt{\frac{d\sigma^{2}(t)}{dt}}dW_{t},~{}t\in[0,T].$
(24)

$W_{t}$ is an independent Weiner process. The transition kernel of VE
diffusion writes

$\displaystyle p(x_{t}|x_{0})=\mathcal{N}(x_{t};x_{0},\sigma(t)\mathbf{I}).$
(25)

Similar to VP diffusion, the marginal samples of VE diffusion are cheap to be
drawn with

$\displaystyle X_{t}=X_{0}+\sigma(t)\epsilon,~{}X_{0}\sim p_{0}$ (26)

Here $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ is a standard Gaussian vector.

Both VE and VP diffusion processes have been successfully used in diffusion
models for various tasks. However, in recent years, several new diffusion
processes have been proposed that have either improved the performance of DMs
or have been designed for specific tasks.

#### Training Method

DMs minimize a weighted combination of DSM with perturbation kernels
$p_{t}(.|.)$ at each time $t$. More precisely, DMs training objective writes

$\displaystyle\mathcal{L}_{WDSM}(\theta)=\int_{t=0}^{T}w(t)\mathcal{L}_{DSM}^{(t)}(\theta)dt.$
(27) $\displaystyle\mathcal{L}_{DSM}^{(t)}(\theta)=\mathbb{E}_{x_{0}\sim
p_{d}^{(0)},x_{t}|x_{0}\sim
p_{t}(x_{t}|x_{0})}\|S_{\theta}(x_{t},t)-\nabla_{x_{t}}\log
p_{t}(x_{t}|x_{0})\|_{2}^{2}.$ (28)

By minimizing objective equation 27, continuous-indexed score network
$S_{\theta}(x,t)$ is capable of matching marginal score functions of forward
diffusion process equation 20.

In some literature, the DSM objective equation 28 is reformulated as a _noise-
prediction_ objective that trains DMs by learning to predict the added noise.
More precisely, consider the VP’s transition kernel equation 22 as an
instance, if $x_{t}$ and $x_{0}$ is obtained with re-parametrization technique
equation 23, then the gradient terms writes

$\displaystyle\nabla_{x_{t}}\log p_{t}(x_{t}|x_{0})$
$\displaystyle=\nabla_{x_{t}}\bigg{[}-\frac{1}{2(1-\alpha_{t})}\|x_{t}-\sqrt{\alpha_{t}}x_{0}\|_{2}^{2}\bigg{]}$
(29)
$\displaystyle=\frac{1}{1-\alpha_{t}}(x_{t}-\sqrt{\alpha_{t}}x_{0})=\frac{1}{1-\alpha_{t}}\sqrt{1-\alpha_{t}}\epsilon$
(30) $\displaystyle=\frac{1}{\sqrt{1-\alpha_{t}}}\epsilon$ (31)

Combining gradient term equation 29 and DSM objective equation 28, the DSM
objective for VP diffusion can be reformulated as

$\displaystyle\mathcal{L}_{DSM}(\theta)=$ $\displaystyle\mathbb{E}_{x_{0}\sim
p_{d},\epsilon\sim\mathcal{N}(0,\mathbf{I})}\frac{w(t)}{1-\alpha_{t}}\|\sqrt{1-\alpha_{t}}S_{\theta}(x_{t},t)-\epsilon\|_{2}^{2}$
(32) $\displaystyle=\mathbb{E}_{x_{0}\sim
p_{d},\epsilon\sim\mathcal{N}(0,\mathbf{I})}\frac{1}{1-\alpha_{t}}\|\epsilon_{\theta}(x_{t},t)-\epsilon\|_{2}^{2}$
(33)

The DSM can be reformulated using a noise-prediction objective where a neural
network $\epsilon_{\theta}(x_{t},t)$ is trained to predict the added noise
$\epsilon$ using the noised data sample $x_{t}$ and the score network
$S_{\theta}(x_{t},t)$. This reformulation involves a modified version of the
score network, represented as
$\epsilon_{\theta}(x_{t},t)\colon=\sqrt{1-\alpha_{t}}S_{\theta}(x_{t},t)$, and
a re-expression of the diffusion process as
$x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon$. A similar
formulation can also be applied to the VE diffusion process, which is
discussed in detail in the Appendix.

#### Sampling Strategy

The score network trained in DSM can be utilized in various applications, with
one of the most direct applications being the design of a sampling strategy
for approximating the underlying data distribution. The fundamental concept
behind this mechanism is the existence of a reversed SDE equation 34 that has
the same marginal distribution as the forward SDE equation 20,

$\displaystyle dX_{t}=[F(X_{t},t)-G^{2}(t)\nabla_{x_{t}}\log
p^{(t)}(X_{t})]dt+G(t)d\bar{W}_{t},t\in[T,0],X_{T}\sim p_{d}^{(T)}.$ (34)

Moreover, an ODE equation 35 is also found to share the same marginal
distribution

$\displaystyle dX_{t}=[F(X_{t},t)-\frac{1}{2}G^{2}(t)\nabla_{x_{t}}\log
p^{(t)}(X_{t})]dt.$ (35)

Both the reversed SDE 34 and ODE 35 rely on the true marginal score functions
$\nabla_{x_{t}}\log p_{d}^{(t)}(x_{t})$. However, by replacing the true
marginal score functions with the learned neural score functions
$S_{\theta}(x,t)$, generative SDEs and ODEs of DMs can be obtained. Moreover,
the concept of generative ODEs can be extended to neural continuous-time
normalizing flow models under certain circumstances. By using the learned
score functions $S_{\theta}(x,t)$, sampling from DMs can be achieved through
the numerical solutions of sampling SDEs or ODEs. Numerous practical
algorithms that use advanced numerical techniques have been developed to
improve the generative performance of DMs or enhance the sampling efficiency
with minimal loss of performance (Song et al., 2020b; Bao et al., 2022c; Liu
et al., ; Zhao et al., 2023).

#### Successes of DMs

Since the pioneering works by Sohl-Dickstein et al. (2015), Ho et al. (2020b),
and Song et al. (2020c), diffusion models (DMs) have emerged as the leading
approach for generative modeling, finding widespread use in various domains,
including neural image synthesis and editing (Nichol and Dhariwal, 2021;
Dhariwal and Nichol, 2021; Ramesh et al., 2022b; Saharia et al., 2022a;
Rombach et al., 2022), audio and molecule synthesis (Hoogeboom et al., 2022;
Chen et al., 2020), image segmentation (Baranchuk et al., 2021), and video or
3D object generation (Ho et al., 2022; Molad et al., 2023; Poole et al.,
2022b). DMs have shown remarkable performance improvements over time, as seen
in the steady trend of unconditional Frechet Inception Score (Heusel et al.,
2017) (FID) reductions on datasets such as CIFAR10, from 25.32 (Song and
Ermon, 2019) to 1.97 (Karras et al., ).

#### Diffusion Distillation

The concept of knowledge distillation (Hinton et al., 2015; Oord et al.,
2018), which aims to create smaller and more efficient models while
maintaining accuracy, has shown great success in various research domains. In
particular, distilling knowledge from pre-trained classifiers has resulted in
models with comparable accuracy, reduced model size, and improved inference
efficiency (Touvron et al., 2021). Given the success of diffusion models (DMs)
in numerous applications, there is a growing interest in distilling knowledge
from these models to create smaller and more efficient versions. One of the
key motivations for diffusion distillation is to significantly accelerate the
sampling speed, which is currently hindered by the large number of neural
function evaluations required. To improve the inference efficiency of DMs,
researchers are exploring ways to distill the learned knowledge from DMs to
efficient sampling mechanisms, such as a direct implicit generator or a fewer-
steps vector field. By doing so, they have been able to create student models
that have further improved inference efficiency with minimal performance loss.
Some distilled student models require less than 10 neural function evaluations
but still offer comparable generative performance to their larger
counterparts.

Diffusion distillation also serves as a means to establish connections between
DMs and other generative models, such as implicit generative models and
normalizing flows. Through knowledge transfer between DMs and other models,
researchers can study the micro-connections between them and explore their
potential for future generative modeling research.

This paper provides a comprehensive review of existing research on diffusion
distillation strategies. Our review is organized into three main categories:
diffusion-to-field (D2F) distillation (Section 2), diffusion-to-generator
(D2G) distillation (Section 3), and training-free (TF) distillation (Section
4). Each category contains studies that share similar settings and
methodologies. In addition to our categorization, we also discuss broader
topics in diffusion distillation throughout the rest of this survey.

##  2 Diffusion-to-Field Distillation

The D2F distillation approach aims to address the lack of efficiency in the
deterministic sampling method of DMs by distilling the generative ODE equation
35 into another generative vector field that requires fewer NFEs to generate
comparable samples. This approach can be categorized into two classes: output
distillation and path distillation. Output distillation aims to teach a
student vector field to replicate the output of the DM’s deterministic
sampling method. Path distillation, on the other hand, aims to produce a
student ODE that has better path properties than the teacher ODE. Both output
and path distillations can be used in combination to improve both the path
properties and simulation efficiency of the teacher ODE.

###  2.1 Output Distillation

To begin with, we first recap the generative ODE equation 35

$\displaystyle dX_{t}=[F(X_{t},t)-\frac{1}{2}G^{2}(t)\nabla_{X_{t}}\log
p^{(t)}(X_{t})]dt,~{}t\in[T,0],~{}p^{(T)}=p_{T}$

To make the discussion simpler, we consider the most naive Euler-Maruyama (EM)
discretization method that solves the ODE with sequential updates

$\displaystyle
X_{t_{i}}=X_{t_{i+1}}+[F(X_{t_{i=1}},t_{i+1})-\frac{1}{2}G^{2}(t_{i+1})\nabla_{X_{t_{i+1}}}\log
p^{(t_{i+1})}(X_{t_{i+1}})](t_{i}-t_{i+1}),~{}i=N,...,1$

In the context of sampling from DMs, using EM methods directly for numerical
solvers suffers from computational inefficiency. This is because the EM
discretization error increases significantly with the step size, leading to
poorly generated samples. To address this issue, an alternative approach
called output distillation has been proposed. It involves training a student
neural network to learn the output of the ODE using larger step sizes.
Specifically, assuming that the step size $\Delta t$ is not too small, a
student continuously-indexed neural mapping $S_{\phi}^{(stu)}(x,t)$ is trained
to approximate the change in the ODE’s output. This helps improve the
computational efficiency of sampling from DMs by reducing the number of NFEs
required for competitive sampling performance. More precisely, the student
continuously-indexed neural mapping $S_{\phi}^{(stu)}(x,t)$ is trained to
approximate the change of teacher ODE’s output between time $t$ and $t-\Delta
t$

$\displaystyle\Delta X\colon=X_{t-\Delta t}-X_{t}=\int_{s=t}^{t-\Delta
t}[F(X_{s},s)-\frac{1}{2}G^{2}(s)\nabla_{X_{s}}\log p^{(s)}(X_{s})]ds.$ (36)

The residual in equation 36 is tackled using numerical methods with
sufficiently small step sizes. In order to overcome the computational
inefficiency of this approach, a student network $S_{\phi}^{(stu)}$ is
introduced as a distilled time-dependent vector field that approximates the
non-linear residual in equation 36. By leveraging the expressive power of
neural networks, empirical studies have shown that properly designed
diffusion-to-field techniques can result in a new sampling ODE that requires
only one NFE but still yields comparable generative performance in terms of
Fretchet Inception Distance (FID) . It is worth noting that in the following
sections, we use $\theta$ to denote the parameters of teacher models (DMs) and
$\phi$ to represent those of student models unless otherwise specified.

Figure 2: Knowledge Distillation Strategy proposed in Luhman and Luhman
(2021).

Luhman and Luhman (2021) propose a knowledge distillation (KD) strategy to
distill a DDIM sampler to a Gaussian model that requires only one NFE when
sampling. DDIM is a typical deterministic sampling strategy for VP diffusion
models. Assume $\epsilon_{\theta}(x,t)$ is DM’s noise-prediction network as
discussed in equation 32, the DDIM sampler is a deterministic sampler that
sequentially updates

$\displaystyle
X_{t_{i-1}}=\sqrt{\frac{\alpha_{t_{i-1}}}{\alpha_{t_{i}}}}(X_{t_{i}}-\sqrt{1-\alpha_{t_{i}}}\epsilon_{\theta}(X_{t_{i}}))+\sqrt{1-\alpha_{t_{i-1}}}\epsilon_{\theta}(X_{t_{i}})$
(37)

DDIM is a deterministic sampling strategy that has been shown to significantly
reduce the number of NFEs required for generating samples, with as few as 100
NFEs being sufficient for maintaining good generative performance. In contrast
to the generative ODE method, DDIM is widely regarded as a superior
deterministic sampling strategy. In their work, Luhman and Luhman (2021)
propose to use a conditional Gaussian model as the student generative model

$\displaystyle
p_{stu}(x_{0}|x_{T})=\mathcal{N}(x_{0};f_{\phi}(x_{T}),\mathbf{I}).$ (38)

The neural network $f_{\phi}(.)$ has the same input and output dimensions as
the data. To implement it, the authors choose the architecture of $f_{\phi}$
to be the same as that of the score network in the teacher DM. Let
$\operatorname{DDIM}(.)$ denote the deterministic mapping induced by the DDIM,
they took

$\displaystyle
p_{teacher}(x_{0}|x_{t})=\mathcal{N}(x_{0};\operatorname{DDIM}(x_{T}),\mathbf{I})$

To train the student model, they propose to minimize the conditional KL
divergence between the student model and the DDIM sampler

$\displaystyle\mathcal{L}(\phi)$
$\displaystyle=\mathbb{E}_{x_{T}\sim\mathcal{N}(0,\mathbf{I})}\mathcal{D}_{KL}\bigg{[}p_{teacher}(x_{0}|x_{T}),p_{stu}(x_{0}|x_{T})\bigg{]}$
(39)
$\displaystyle=\mathbb{E}_{x_{T}\sim\mathcal{N}(0,\mathbf{I})}\mathcal{D}_{KL}\bigg{[}\frac{1}{2}\|f_{\phi}(x_{T})-\operatorname{DDIM}(x_{T})\|_{2}^{2}\bigg{]}$
(40)

The student model sampling strategy proposed by Luhman and Luhman (2021) is
simple. It involves drawing a Gaussian random variable $x_{T}$ from a normal
distribution with mean 0 and identity covariance matrix. They then obtain the
mean vector of the student model $x_{0}$ by passing $x_{T}$ through the neural
network $f_{\phi}$. This results in a one-NFE sampling model with an FID of
9.39, while the teacher generative ODE has an FID of 4.16. Although this
method provides a first step towards considering knowledge distillation for
diffusion models, it has a computational inefficiency as it requires
generating final outputs of DDIM or other ODE sampler, which consists of
hundreds of NFEs for computing a single batch of training data.

Figure 3: Progressive Distillation Strategy proposed in Salimans and Ho
(2022).

The progressive distillation (PD) strategy proposed by (Salimans and Ho, 2022)
aims to train a student neural network that requires half the number of NFEs
as the teacher model by learning the two-step prediction of the teacher DM’s
deterministic sampling strategy. The teacher diffusion model is discretized to
$N$ time-stamps, denoted by $\mathcal{T}={t_{i}}{i=0,1,...,N-1}$, while
$\mathcal{T}^{\prime}={0,2,4,...}$ represents $N/2$ even time-stamps of
$\mathcal{T}$. The student network is denoted by $f\phi(x,t)$, and the updates
of the DDIM method from a time-stamp $t_{j}$ to another time-stamp $t_{i}$ in
the teacher DM are denoted by $\operatorname{DDIM}(x,t_{j},t_{i})$ with $i\leq
j$. The PD trains the student network by minimizing

$\displaystyle\mathcal{L}(\phi)=\mathbb{E}_{x_{0}\sim p_{d},i\sim
Unif(\mathcal{T}^{\prime}),\epsilon\sim\mathcal{N}(0,\mathbf{I})}\|f_{\phi}(\tilde{x},t_{i})-\operatorname{DDIM}(\tilde{x},t_{i},t_{i-2})\|,$
(41)

where $\tilde{x}=\sqrt{\alpha_{t_{i}}}x_{0}+\sqrt{1-\alpha_{t_{i}}}\epsilon$
is the forward diffused data at time $t_{i}$ of VP diffusion. By minimizing
the PD objective equation 41, the student network learns to output a two-step
prediction of the teacher model, so the total NFEs are halved. After the
student model is trained to accurately predict the teacher model’s two-step
sampling strategy, it replaces the teacher model, and a new student model is
trained to further reduce the number of sampling steps by half. The authors of
the paper used the same UNet architecture as the teacher model’s score network
and the DDIM method as the initial teacher sampling strategy in their
implementation of the progressive distillation (PD) approach. Their results
showed that successive PD rounds can reduce the required vector fields to only
4 NFEs, making it 250 times more efficient than the teacher diffusion’s ODE
sampler, with a 5% drop in generative performance as measured by FID. Both PD
and KD use the teacher network’s architecture to distill a few-step sampling
method by learning from the many-step teacher sampling method. Both approaches
are implemented by minimizing the $L^{2}$ error between the multi-step
predictions of the teacher network and the single-step prediction of the
student network. The key difference between PD and KD is that PD progressively
reduces the number of required function evaluations, while KD directly trains
a one-step student model for the final prediction. Thus, KD can be seen as an
extreme PD method that reduces the teacher sampling strategies’ full time-
stamps to one in a single round.

A two-stage distillation strategy is proposed by Meng et al. (2022) to address
the challenge of distilling knowledge from classifier-free guided conditional
diffusion models like GLIDE (Nichol et al., 2021), DALL$\cdot$E-2 (Ramesh et
al., 2022a), Stable Diffusion (Rombach et al., 2021) and Imagen (Saharia et
al., 2022b). The key challenge is to transfer knowledge from the teacher DMs
while preserving the classifier-free guidance mechanism, which successfully
trains a single DM to learn both conditional and unconditional distributions.
The conditional knowledge is integrated into the DM through a conditional
input, while the unconditional knowledge is learned by replacing the
conditional inputs with a _None_ input. In the first stage of their strategy,
a student conditional diffusion model with a classifier-free guidance input is
trained to learn from the teacher diffusion model. The student model is
implemented as a neural network with learnable parameters, which takes in an
input $x$ and a conditional context input $c$. For simplicity, the notation
$c$ is dropped in the following discussion. The student model is trained to
match the output by minimizing the objective

$\displaystyle\mathcal{L}(\phi_{1})=\mathbb{E}_{w\sim p_{w},t\sim U[0,1],x\sim
p_{d}}\bigg{[}\lambda(t)\|f_{\phi_{1}}(x_{t},t,w)-\hat{x}^{w}_{\theta}(x_{t})\|_{2}^{2}\bigg{]}$
(42)

Here $\lambda(t)$ represents a pre-defined weighting function.
$\hat{x}^{w}_{\theta}(x_{t})=(1+w)\hat{x}_{c,\theta}(x_{t})-w\hat{x}_{\theta}(x_{t})$,
$x_{t}\sim p_{t}(x_{t}|x_{0})$ and $p_{w}=U[w_{min},w_{max}]$. Note that the
stage-one distillation only incorporates the classifier-free guidance through
an additional input $w$ for the student model, no reduction of NFEs and
efficiency-related distillation is applied. The second stage of distillation
employs the progressive diffusion (PD) strategy proposed by Salimans and Ho
(2022) to significantly reduce the number of diffusion steps of the previously
trained student model with classifier-free guidance inputs. The two-stage
distillation approach is used to distill both pixel-space and latent-space
classifier-free guided conditional diffusion models for various tasks,
including class-conditional generation, text-guided image generation, text-
guided image-to-image translation, and image inpainting. In fact, the
distilled students achieved even better Fréchet Inception Distance (FID)
scores than their teachers in the experiment of distilling pixel-space class-
conditional generation on the ImageNet-64x64 dataset.

In recent work, Sun et al. (2022) proposed a feature space distillation method
called Classifier-based Feature Distillation (CFD) for image data. The primary
motivation behind this approach was to address the challenge of directly
aligning pixels in the teacher’s output and a few-step student model’s output,
which they found to be too difficult to learn. Instead, they trained the
student networks to align with the multiple-step output of the teacher models
in the feature space extracted by a pre-trained classifier. This approach
follows a similar distillation strategy as the PD technique.

To achieve this, they proposed minimizing the KL divergence between the
predicted probability distribution (after $\operatorname{Softmax}$ function)
of the student model’s one-step outputs and the teacher model’s multiple-step
outputs. They also found that incorporating additional terms such as entropy
and diversity regularizations improved the distillation performance. They
employed the same diffusion models and student models as Salimans and Ho
(2022) and used DenseNet-201 (Huang et al., 2016) as the classifier in their
implementations. Their distillation approach resulted in a student model with
FID 3.80 on CIFAR10 with only 4 NFEs, which is lower than the DP’s
implementation in Salimans and Ho (2022).

The PD strategy was also applied by Huang et al. (2022) to distill fast text-
to-speech (TTS) generation models based on a pre-trained diffusion TTS model.
They introduced a variance predictor and a spectrogram denoiser to improve the
student model’s architecture, which was specifically designed for TTS
applications.

In contrast to mimicking the output of the generative ODE of diffusion models,
Song et al. (2023) proposes to minimize the difference of self-consistency
function of generative ODE to implement the output distillation. They randomly
diffuse a real data sample and simulate a few steps of generative ODE to
obtain another noisy data which lies on the same ODE path. They input the two
noisy samples into a student model and minimize the difference in the outputs
in order to ensure the self-consistency of generative ODE. They name their
proposed model the consistency model (CM). CM can be viewed as another output
distillation method that utilizes the self-consistent property of generative
ODE for distillation.

For output distillation, the student network is trained to minimize the
difference between its output and the output of the teacher ODE at the same
time points. The output distillation is particularly useful when the teacher’s
ODE has a relatively simple form, such as diagonal ODEs, or when the teacher’s
ODE is difficult to train. In these cases, the student network can be trained
to efficiently mimic the teacher ODE’s output with much fewer NFEs. The
trained student network can then be used for efficient sampling from the DMs.

It is worth noting that the output distillation only considers the output of
the ODE at discrete time points, and thus may not fully capture the path
properties of the DMs. Therefore, it may not be suitable for applications
where path properties are important, such as image synthesis and editing. For
such applications, path distillation can be used instead, which will be
discussed in the next section.

###  2.2 Path Distillation

Output distillation is a technique that helps improve the sampling strategy of
DMs by allowing the student neural network to mimic the multiple-step output
of teacher models. In contrast, path distillation aims to refine DMs’ sampling
strategy to potentially have better properties. Some researchers argue that
the forward (and reverse) diffusion process creates a curve in data space that
connects the data distribution and prior distribution. Thus, path distillation
focuses on refining the diffusion generative SDE or ODE to a straight version
that has a more efficient sampling path than the teacher models. The key
difference between path distillation and output distillation is that path
distillation is more concerned with refining an existing teacher model’s
sampling strategy, whereas output distillation focuses on teaching the student
to learn the skipped output of the teacher model, without changing the
sampling path and mechanism of the student model.

The Reflow method, introduced by Liu et al. (2022b), is a path distillation
approach that aims to improve generative speed by modifying a pre-trained
teacher neural ODE through a student model. The student model straightens the
path of the teacher model by minimizing a time average of $L^{2}$ loss between
its outputs and interpolations of data samples and corresponding outputs from
the pre-trained model. More precisely, let $p_{T}$ be an initial distribution

$\displaystyle\mathcal{L}(\phi)=\mathbb{E}_{x_{T}\sim p_{T},x_{0}\sim
p_{teacher}(x_{0}|x_{T}),t\sim
U[0,T]}\bigg{[}\|x_{T}-x_{0}-f_{\phi}(\frac{t}{T}x_{T}+\frac{T-t}{T}x_{0},t)\|_{2}^{2}\bigg{]}$
(43)

The term $p_{teacher}(x_{0}|x_{T})$ can be an arbitrary teacher model
regardless it is an SDE or ODE. In Reflow distillation proposed by Liu et al.
(2022b), if the teacher model is assumed to be a DDIM sampler, then
$p_{teacher}(x_{0}|x_{T})=\delta(x_{0}=\operatorname{DDIM}(x_{T}))$. Reflow
distillation has the advantage of being repeatable for several rounds, which
can further straighten the teacher model’s path. Additionally, Reflow can be
used in conjunction with progressive distillation as discussed in Section
LABEL:, where the Reflow strategy can be used first to straighten the teacher
model’s path, followed by progressive distillation to make the student model
more efficient. Finally, the authors provided numerical comparisons between
ODE paths of rectified and un-rectified models.

Figure 4: Reflow Strategy for path distillation. The figure is taken from Liu
et al. (2022b). Figure 5: Reflow Strategy for path distillation. The figure is
taken from Liu et al. (2022b).

In their work, Wu et al. (2022) applied the Reflow technique to the problem of
3D point cloud generation. They proposed a three-stage procedure that enables
the construction of a generative ODE capable of producing high-quality point
clouds. At their first training stage, they train a teacher generative ODE
$f_{\theta}$ by minimizing the objective

$\displaystyle\mathcal{L}(\theta)=\mathbb{E}_{x_{0}\sim
p_{d},x_{T}\sim\mathcal{N}(0,\mathbf{I}),t\sim
U[0,T]}\bigg{[}\|f_{\theta}(x_{t},t)-(x_{0}-x_{T})\|^{2}_{2}\bigg{]}$ (44)

where $x_{t}=(t/T)x_{0}+(T-t/T)x_{T}$ is the interpolated point at time $t$.
In the second stage, they applied the Reflow strategy to further _straighten_
the teacher model. In the third stage, they use a student model $f_{\phi}$
which finally distills the multiple-step teacher model into a single-step
student model by minimizing

$\displaystyle\mathcal{L}(\phi)=\mathbb{E}_{x_{T}\sim
p_{d}}\operatorname{Dist}\bigg{[}x_{T}+f_{\phi}(x_{T},T),x_{0}\bigg{]}$ (45)

Here $x_{0}\sim p_{teacher}(x_{0}|x_{T})$ is obtained from teacher model. The
term $\operatorname{Dist}(.,.)$ represents some distance function between two
points. In their implementation, they use the Chamfer distance for measuring
the distance between two cloud points.

Lee et al. (2023) proposed an objective that can result in a forward process
with much smaller curvations, to be more insensitive to truncation errors.
Chen and Lipman (2023), Li et al. (2023a) and Albergo and Vanden-Eijnden
(2022) further study the refinements of the diffusion model’s generative path.
Zheng et al. (2022) proposed another view for path distillation, which learns
a mapping operator which could generate the path. Fan and Lee (2023) also
proposes another path distillation method. They refine the generative path by
finetuning the teacher ODE to minimize some IPM according to the forward path.
Aiello et al. (2023) proposed to finetune the generative path by minimizing
the MMD between each marginal distribution of the generative path and the
data.

The diffusion-to-field distillation remains an active area of research in the
field of efficient diffusion models. The aim is to use the knowledge gained
from diffusion models and generative SDE/ODE to distill a student model with a
faster sampling speed while maintaining a comparable level of generative
performance to that of the teacher models.

##  3 Diffusion-to-generator Distillation

In contrast to the diffusion-to-field distillation discussed earlier,
diffusion-to-generator (D2G) distillation is another important category of
distillation methods. The primary goal of D2G distillation is to transfer the
distributional knowledge learned by the diffusion model into an efficient
generator. Unlike D2F distillation, which focuses on learning student models
with the same input and output dimensions, D2G distillation usually involves
implicit generators that may not have the same latent dimension as the data
space. Additionally, both deterministic and stochastic generators are
considered as student generators, depending on the specific application. The
training objective for D2G distillation typically takes a similar form as the
diffusion model’s training objective, as opposed to the commonly used mean
square error-based objective in D2F distillation.

###  3.1 Distill Deterministic Generator

There is increasing attention on distilling deterministic generators (e.g.
neural radius field) as student models for further applications of pre-trained
large-scale diffusion models. More specifically, the pre-trained text-to-image
diffusion models are found notably useful for learning neural radius fields
which have contents that are related to some given text prompts. The neural
radius field (NeRF) (Mildenhall et al., 2020) is a kind of 3D object which use
a multi-layer-perceptron (MLP) to map coordinates of a mesh grid to volume
properties such as color and density. Given the camera parameters such as the
angles of the views, the rendering algorithm outputs a 2D image that is a view
projection of the 3D NeRF scene. From a given view, the NeRF is viewed as a
deterministic 2D image generator whose MLP’s parameters are learnable.

Figure 6: Score Distillation Sampling method. The figure is taken from Poole
et al. (2022b).

The limited availability of data for constructing NeRFs has motivated
researchers to explore distillation methods to obtain NeRFs with contents
related to given text prompts. The pioneering work by Poole et al. (Poole et
al., 2022a) proposed a method called score distillation sampling (SDS) to
distill a 2D text-to-image diffusion model into 3D NeRFs. Unlike traditional
NeRF construction that requires images from multiple views of the target 3D
objects, text-driven construction of NeRF lacks both the 3D object and the
multiple views. The SDS method optimizes the NeRF by minimizing the diffusion
model’s loss function using NeRF-rendered images from a fixed view. To avoid
the computational expense of directly optimizing the diffusion model’s loss
function, the researchers propose to approximate the distillation objective by
omitting the Unet jacobian term. Specifically, the rendered NeRF image from a
fixed view with the learnable parameter $\phi$ is represented by $x=g(\phi)$,
and the forward diffusion sample is denoted as
$x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon$ as proposed in
equation 23. The trained text-conditional diffusion model is denoted by
$\epsilon_{\theta}(x,t,c)$, where $t$ and $c$ represent the time-stamp and
text prompt. The SDS uses the gradient to update parameter $\phi$ with

$\displaystyle\operatorname{Grad}(\phi)=\frac{\partial}{\partial\phi}\mathcal{L}(\phi)=\mathbb{E}_{t,\epsilon,x=g(\phi)}\bigg{[}w(t)(\epsilon_{\theta}(x_{t},t,c)-\epsilon)\frac{\partial
x}{\partial\phi}\bigg{]}$ (46)

The work by Poole et al. (2022a) achieved remarkable results in the task of
generating 3D NeRFs from text prompts, which has spurred further research on
the distillation of diffusion models into deterministic generators.

Several works have extended the SDS approach to other applications, as
reported in Wang et al. (2022); Lin et al. (2022); Deng et al. (2022); Xu et
al. (2022); Singer et al. (2023). In particular, Wang et al. (2022) has
conducted a thorough investigation of the tunable parameters of the SDS
algorithm for text-to-3D generation. Meanwhile, Lin et al. (2022) have
proposed a two-stage optimization strategy that further enhances the
performance of SDS on high-resolution images. The first stage of their
approach is similar to that of Poole et al. (2022a), which obtains a low-
resolution NeRF. The second stage up-scales the trained NeRF from the first
stage to a higher resolution and fine-tunes it for better performance.

The successful application of distillation to deterministic generators,
particularly in the domain of 3D generative modeling, has been widely
acknowledged. However, the research in developing better-performing and more
efficient distillation strategies is still largely unexplored, making it a hot
research topic that demands further exploration.

###  3.2 Distill Stochastic Generator

In this section, we will discuss distillation strategies for stochastic
generators, also known as implicit generative models. These models have been
widely used in generative modeling and differ from deterministic generators in
that they use neural transforms to map latent vectors to generate data samples
with randomness. Stochastic generators have shown great success in the past
decade (Goodfellow et al., 2014; Karras et al., 2019; Brock et al., 2018),
with their advantages being fast inference speed, low inference memory, and
lightweight models. Distilling diffusion models into stochastic generators has
been motivated by the need for extreme inference efficiency.

The work by Luhman and Luhman (2021) can be interpreted as a type of score
distillation for stochastic generators. Specifically, they employ a Unit,
which has the same neural architecture as the pre-trained diffusion model, to
map a latent vector and calculate the mean of a Gaussian output. The
dimensionality of their latent vector matches that of the output data, and
they assume unit variance for Gaussian conditional generation. To train the
student model, they minimize the conditional KL divergence between the
generator’s output and a deterministic sampler for the diffusion model. Since
they use conditional Gaussian output, minimizing KL divergence is equivalent
to minimizing the mean square error between the model’s mean and the sampler’s
output, as discussed in Section 2.

##  4 Accelerated Sampling Algorithms as Diffusion Distillation

The diffusion model has the unique characteristic of separating training and
sampling processes. During training, the diffusion model does not require
sampling, and during sampling, there is a high level of flexibility to design
and improve the process. Typically, the diffusion model is trained with all
discrete or continuous noise levels, but sampling from it does not necessarily
require querying all diffusion time levels. Recent research has shown that
using a small subset of diffusion time levels can significantly accelerate the
sampling process with much fewer NFEs. These accelerated sampling algorithms
can be viewed as generalized distillations of diffusion models, where the goal
is to train a big model but use a small one. We can further categorize these
algorithms into training-free and training-based accelerating algorithms as
two distinct categories of diffusion distillation.

To train a diffusion model, a neural score network is trained to match the
marginal score functions of a data diffusion process. As Song et al. (2020d)
pointed out, the reversed SDE equation 34 and ODE equation 35 share the same
marginal distribution as the forward diffusion and can be used for sampling if
simulated backward. Therefore, the reversed ODE or SDE serves as the starting
point for sampling from diffusion models. Formally, the most naive simulation
of reversed SDE uses an Euler-Maruyama discretization with a formula

$\displaystyle
X_{t+1}=X_{t}-\big{[}F(X_{t},t)-G^{2}(t)S_{\theta}(X_{t},t)\big{]}\Delta
t+G(t)\Delta t,t=T,...,1.$ (47)

Here $S_{\theta}(x,t)$ represents the trained score network, i.e. the teacher
model. The simple sampler presented in equation equation 47 updates a batch of
samples by simulating the reversed SDE, taking all diffusion time levels into
account sequentially. However, this sampling strategy that uses all-time
levels suffers from poor computational efficiency. In later sections, we will
introduce both training-based and training-free accelerated sampling
algorithms. These algorithms aim to improve the sampling speed of diffusion
models by utilizing a smaller subset of diffusion time levels.

###  4.1 Training-based Accelerating Algorithms

The selection of appropriate diffusion time levels is a crucial problem for
accelerating sampling algorithms.

Previous works have shown that it is possible to construct more efficient
samplers that have comparable performance to the naive sampler by using only a
subset of diffusion time levels. Watson et al. (2022) addressed this problem
by proposing to solve a dynamic programming problem with the Fréchet Inception
Distance (FID) metric as the target. They learned a scheduler model to choose
the subset of steps to be used in the sampling process, resulting in better
efficiency.

To improve the generative performance of diffusion models, Bao et al. (2022a)
proposed to train additional covariance networks. The idea is to capture the
full covariance matrix of the latent space distribution, instead of assuming
diagonal covariance as in the original diffusion model. By doing so, the model
can better capture the complex correlation structure of the data and generate
more realistic samples. The authors achieved promising results on various
datasets and showed that their method outperforms the baseline diffusion model
with diagonal covariance.

The approach proposed by Kim et al. (2022) involves using a likeliratio-
estimation model, which is essentially a discriminator, to learn the
difference between the ground truth score function and the model’s score
function. This difference is then used to obtain an unbiased score function by
combining it with DM’s score functions, after differentiating through the
learned diffusion. The resulting refined DM is expected to offer improved
generative performance even with fewer NFEs.

###  4.2 Training-free Accelerating Algorithms

The training-free algorithms aim to achieve comparable generative performance
with a much fewer number of used diffusion time levels. The most distinguished
part of training-free accelerating algorithms is to design faster sampling
algorithms with no training of new parametric models but only inferencing by
querying pre-trained DMs. Most of such algorithms center around different
numerical solvers of generative SDE or ODE.

To begin with, we start with the denoising diffusion probabilistic models
(DDPM) (Ho et al., 2020a). The DDPM implements a discretized version of VP
diffusion and uses a discretized sampling scheme with

$\displaystyle
x_{t_{i-1}}=\frac{1}{\alpha_{t_{i}}}(x_{t_{i}}-\frac{1-\alpha_{t_{i}}}{\sqrt{1-\bar{\alpha}_{t_{i}}}}\epsilon_{\theta}(x_{t_{i}},t_{i}))+\sigma_{t_{i}}z_{i}$
(48)

Here
$\alpha_{t_{i}}=1-\beta_{t_{i}},\bar{\alpha}_{t_{i}}=\prod_{j=1}^{i}\alpha_{t_{i}}\approx
e^{-\int_{s=0}^{t_{i}}e^{\beta(s)}ds}$ is a discretized inplementation of
integral. $\sigma_{t_{i}}$ is an arbitrary $\sigma$ levels and $z_{i}$ is an
independent Gaussian noise. The DDPM uses all $1000$ diffusion levels (i.e.
$\\{t_{1},...,t_{1000}\\}$) for sampling.

Song et al. (2020a) re-formulate the derivation of DDPM under a non-Markov
forward diffusion model and results in a new family of sampler

$\displaystyle
x_{t_{i-1}}=\sqrt{\alpha_{t_{i-1}}}\bigg{[}\frac{x_{t_{i}}-\sqrt{1-\alpha_{t_{i}}}\epsilon_{\theta}(x_{t_{i}},t_{i})}{\sqrt{\alpha_{t_{i}}}}\bigg{]}+\sqrt{1-\alpha_{t_{i-1}}-\sigma_{t_{i}}^{2}}\cdot\epsilon_{\theta}(x_{t_{i}},t_{i})+\sigma_{t_{i}}z_{t_{i}}$
(49)

The term $\alpha_{t_{i}}$ has the same meaning as the one in DDPM’s sampling
algorithm. The $\sigma_{t_{i}}$ is a free hyper-parameter that controls the
strength of randomness in the DDIM sampler. When
$\sigma=\sqrt{(1-\alpha_{t_{i-1}})/(1-\alpha_{t_{i}})}\sqrt{1-\alpha_{t_{i}}/\alpha_{t_{i-1}}}$,
the DDIM sampler coincides with the DDPM sampler. The sampler is deterministic
when $\sigma_{t_{i}}=0$. The DDIM sampler requires the same pre-trained DM as
the DDPM’s sampler and is shown to be able to remain the generative
performance by querying fewer diffusion time levels for sampling.

The work of Bao et al. (2022b) showed that there is an optimal reverse
variance $\sigma_{t_{i}}$ in the DDIM sampler family that minimizes the KL
divergence between the forward and reversed Markov chain of DDIM. Moreover,
they derived an explicit expression for this optimal variance with the form

$\displaystyle\sigma_{n}^{*2}=\lambda_{n}^{2}+\bigg{[}\sqrt{\frac{\bar{\beta}_{n}}{\alpha_{n}}}-\sqrt{\bar{\beta}_{n-1}-\lambda_{n}^{2}}\bigg{]}^{2}\cdot\bigg{[}1-\bar{\beta}_{n}\mathbb{E}_{q_{n}(x_{n})}\frac{\|\nabla_{x_{n}}\log
q_{n}(x_{n})\|^{2}}{d})\bigg{]}$ (50)

They claimed the DDIM sampler with such optimal variance leads to improved
sampling regardless of the pre-trained diffusion models. In practice, the
optimal variance is estimated with the Monte Carlo method with the help of
pre-trained DMs, with the form

$\displaystyle\Gamma_{n}=\frac{1}{M}\sum_{m=1}^{M}\frac{\|s_{n}(x_{n,m})\|^{2}}{d},x_{n,m}\sim^{iid}q_{n}(x_{n})$
(51)
$\displaystyle\hat{\sigma}^{2}_{n}=\lambda_{n}^{2}+\bigg{[}\sqrt{\frac{\bar{\beta}_{n}}{\alpha_{n}}}-\sqrt{\bar{\beta}_{n-1}-\lambda^{2}_{n}}\bigg{]}^{2}(1-\bar{\beta}_{n}\Gamma_{n}).$
(52)

Liu et al. (2022a),Zhang and Chen (2022) and Lu et al. (2022a) find out the
semi-linear structure of VP’s reversed ODE

$\displaystyle\frac{dX_{t}}{dt}=F(t)X_{t}-\frac{1}{2}G^{2}(t)\nabla_{X_{t}}\log
p^{(t)}(X_{t})$ (53)

where $F(t)=-\frac{1}{2}\beta(t)$ and $G(t)=\sqrt{\beta(t)}$ for VP diffusion.
They applied an exponential integrator technique on ODE that further
simplified VP’s reversed ODE as

$\displaystyle
x_{t}=\frac{\alpha_{t}}{\alpha_{s}}X_{s}-\alpha_{t}\int_{s}^{t}\frac{d\lambda_{\tau}}{d\tau}\frac{\sigma_{\tau}}{\alpha_{\tau}}\epsilon_{\theta}(X_{\tau},\tau)d\tau$
(54)

Where $\lambda_{t}\colon=\log(\alpha_{t}/\sigma_{t})$ is the log-SNR function.
Lu et al. (2022a) further proposed a change of variable trick and Taylor
expansion on a simplified algorithm and obtain higher-order solvers for VP
reversed ODE.

Higher order SDE solvers have been proposed by Jolicoeur-Martineau et al.
(2021) as an alternative to EM discretization, and they have demonstrated
improved sampling performance in terms of FID. Building on this work, Karras
et al. (2022) have suggested better neural preconditioning of DMs along with
the use of second-order Heun discretization for simulating reversed ODE and
SDE. Their method has set a new record for generative performance for DMs with
a score of 1.79 FID.

Other research works have also explored the use of advanced numerical solvers
for DMs. For example, Li et al. (2023b), Wizadwongsa and Suwajanakorn (2023),
and Lu et al. (2022b) have investigated the potential of different numerical
solvers to accelerate DM’s inference efficiency. The key idea of this
training-free accelerated sampling algorithm is to achieve faster sampling
without the need for additional training of parametric models. However, even
with the best-performing training-free methods, more than 10 NFEs are
typically required to achieve comparable generative performance. Zhao et al.
(2023) proposes a unified framework that introduces the prediction-correction-
type numerical solutions to analyze and design higher-order ODE solvers for
diffusion models. Their proposed UniPC framework includes many well-studied
solvers as their special cases.

##  5 Conclusion

This paper provides a comprehensive review of existing techniques for
knowledge distillation of diffusion models. Our review is organized into three
categories: diffusion-to-field distillation, diffusion-to-generator
distillation, and accelerating algorithms as diffusion distillation. For each
category, we discuss several landmark methodologies and their applications.
Given the growing popularity of diffusion models in recent years, knowledge
distillation of diffusion models has become an increasingly important research
area that can impact the efficient application of these models. Although
significant progress has been made in these areas, there is still much room
for improvement in the efficiency and effectiveness of knowledge distillation
of diffusion models.

## References

  * Aiello et al. (2023) Emanuele Aiello, Diego Valsesia, and Enrico Magli.  Fast inference in denoising diffusion models via mmd finetuning.  _ArXiv_ , abs/2301.07969, 2023.
  * Albergo and Vanden-Eijnden (2022) M. S. Albergo and Eric Vanden-Eijnden.  Building normalizing flows with stochastic interpolants.  _ArXiv_ , abs/2209.15571, 2022.
  * Bao et al. (2022a) Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang.  Estimating the optimal covariance with imperfect mean in diffusion probabilistic models.  In _International Conference on Machine Learning_ , 2022a.
  * Bao et al. (2022b) Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang.  Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models.  _ArXiv_ , abs/2201.06503, 2022b.
  * Bao et al. (2022c) Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang.  Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models.  _arXiv preprint arXiv:2201.06503_ , 2022c.
  * Baranchuk et al. (2021) Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models.  _arXiv preprint arXiv:2112.03126_ , 2021.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _ArXiv_ , abs/1809.11096, 2018.
  * Chen et al. (2020) Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  _arXiv preprint arXiv:2009.00713_ , 2020.
  * Chen and Lipman (2023) Ricky T. Q. Chen and Yaron Lipman.  Riemannian flow matching on general geometries.  2023\.
  * Deng et al. (2022) Congyue Deng, Chiyu Max Jiang, C. Qi, Xinchen Yan, Yin Zhou, Leonidas J. Guibas, and Drago Anguelov.  Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors.  _ArXiv_ , abs/2212.03267, 2022.
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Fan and Lee (2023) Ying Fan and Kangwook Lee.  Optimizing ddpm sampling with shortcut fine-tuning.  _ArXiv_ , abs/2301.13362, 2023.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  In _NIPS_ , 2014.
  * Graves (2013) Alex Graves.  Generating sequences with recurrent neural networks.  _arXiv preprint arXiv:1308.0850_ , 2013.
  * Hastings (1970) W Keith Hastings.  Monte carlo sampling methods using markov chains and their applications.  1970\.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Ho et al. (2020a) Jonathan Ho, Ajay Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  _ArXiv_ , abs/2006.11239, 2020a.
  * Ho et al. (2020b) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020b.
  * Ho et al. (2022) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  _arXiv preprint arXiv:2204.03458_ , 2022.
  * Hoogeboom et al. (2022) Emiel Hoogeboom, Vıctor Garcia Satorras, Clément Vignac, and Max Welling.  Equivariant diffusion for molecule generation in 3d.  In _International Conference on Machine Learning_ , pages 8867–8887. PMLR, 2022.
  * Huang et al. (2016) Gao Huang, Zhuang Liu, and Kilian Q. Weinberger.  Densely connected convolutional networks.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2261–2269, 2016.
  * Huang et al. (2022) Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022.
  * Hyvärinen (2005) Aapo Hyvärinen.  Estimation of non-normalized statistical models by score matching.  _J. Mach. Learn. Res._ , 6:695–709, 2005.
  * Jolicoeur-Martineau et al. (2021) Alexia Jolicoeur-Martineau, Ke Li, Remi Piche-Taillefer, Tal Kachman, and Ioannis Mitliagkas.  Gotta go fast when generating data with score-based models.  _ArXiv_ , abs/2105.14080, 2021.
  * Jozefowicz et al. (2016) Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu.  Exploring the limits of language modeling.  _arXiv preprint arXiv:1602.02410_ , 2016.
  * (27) Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  In _Advances in Neural Information Processing Systems_.
  * Karras et al. (2019) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8107–8116, 2019.
  * Karras et al. (2022) Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _ArXiv_ , abs/2206.00364, 2022.
  * Kim et al. (2022) Dongjun Kim, Yeongmin Kim, Wanmo Kang, and Il-Chul Moon.  Refining generative process with discriminator guidance in score-based diffusion models.  _ArXiv_ , abs/2211.17091, 2022.
  * LeCun et al. (2006) Yann LeCun, Sumit Chopra, Raia Hadsell, Aurelio Ranzato, and Fu Jie Huang.  A tutorial on energy-based learning.  2006\.
  * Lee et al. (2023) Sangyun Lee, Beomsu Kim, and Jong-Chul Ye.  Minimizing trajectory curvature of ode-based generative models.  _ArXiv_ , abs/2301.12003, 2023.
  * Li et al. (2023a) Lingxiao Li, Samuel Hurault, and Justin M. Solomon.  Self-consistent velocity matching of probability flows.  _ArXiv_ , abs/2301.13737, 2023a.
  * Li et al. (2023b) Shengmeng Li, Luping Liu, Zenghao Chai, Runnan Li, and Xu Tan.  Era-solver: Error-robust adams solver for fast sampling of diffusion probabilistic models.  _ArXiv_ , abs/2301.12935, 2023b.
  * Lin et al. (2022) Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  _ArXiv_ , abs/2211.10440, 2022.
  * (36) Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.  Pseudo numerical methods for diffusion models on manifolds.  In _International Conference on Learning Representations_.
  * Liu et al. (2022a) Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.  Pseudo numerical methods for diffusion models on manifolds.  _ArXiv_ , abs/2202.09778, 2022a.
  * Liu et al. (2022b) Xingchao Liu, Chengyue Gong, and Qiang Liu.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  _ArXiv_ , abs/2209.03003, 2022b.
  * Lu et al. (2022a) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _ArXiv_ , abs/2206.00927, 2022a.
  * Lu et al. (2022b) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _ArXiv_ , abs/2211.01095, 2022b.
  * Luhman and Luhman (2021) Eric Luhman and Troy Luhman.  Knowledge distillation in iterative generative models for improved sampling speed.  _ArXiv_ , abs/2101.02388, 2021.
  * Meng et al. (2022) Chenlin Meng, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  _ArXiv_ , abs/2210.03142, 2022.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European Conference on Computer Vision_ , 2020.
  * Molad et al. (2023) Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen.  Dreamix: Video diffusion models are general video editors.  _arXiv preprint arXiv:2302.01329_ , 2023.
  * Neal et al. (2011) Radford M Neal et al.  Mcmc using hamiltonian dynamics.  _Handbook of markov chain monte carlo_ , 2(11):2, 2011.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning_ , 2021.
  * Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Oord et al. (2018) Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al.  Parallel wavenet: Fast high-fidelity speech synthesis.  In _International conference on machine learning_ , pages 3918–3926. PMLR, 2018.
  * Pang et al. (2020) Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu.  Efficient learning of generative models via finite-difference score matching.  _ArXiv_ , abs/2007.03317, 2020.
  * Poole et al. (2022a) Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _ArXiv_ , abs/2209.14988, 2022a.
  * Poole et al. (2022b) Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022b.
  * Ramesh et al. (2022a) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _ArXiv_ , abs/2204.06125, 2022a.
  * Ramesh et al. (2022b) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022b.
  * Robert et al. (1999) Christian P Robert, George Casella, and George Casella.  _Monte Carlo statistical methods_ , volume 2.  Springer, 1999.
  * Roberts and Rosenthal (1998) Gareth O Roberts and Jeffrey S Rosenthal.  Optimal scaling of discrete approximations to langevin diffusions.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 60(1):255–268, 1998.
  * Roberts and Tweedie (1996) Gareth O Roberts and Richard L Tweedie.  Exponential convergence of langevin distributions and their discrete approximations.  _Bernoulli_ , pages 341–363, 1996.
  * Rombach et al. (2021) Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10674–10685, 2021.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Saharia et al. (2022a) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_ , 2022a.
  * Saharia et al. (2022b) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  _ArXiv_ , abs/2205.11487, 2022b.
  * Salimans and Ho (2022) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _ArXiv_ , abs/2202.00512, 2022.
  * Singer et al. (2023) Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman.  Text-to-4d dynamic scene generation.  _ArXiv_ , abs/2301.11280, 2023.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pages 2256–2265. PMLR, 2015.
  * Song et al. (2020a) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _ArXiv_ , abs/2010.02502, 2020a.
  * Song et al. (2020b) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020b.
  * Song and Ermon (2019) Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Song et al. (2019) Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon.  Sliced score matching: A scalable approach to density and score estimation.  In _Conference on Uncertainty in Artificial Intelligence_ , 2019.
  * Song et al. (2020c) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020c.
  * Song et al. (2020d) Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _ArXiv_ , abs/2011.13456, 2020d.
  * Song et al. (2023) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models, 2023.  URL https://arxiv.org/abs/2303.01469.
  * Sun et al. (2022) Wujie Sun, Defang Chen, Can Wang, Deshi Ye, Yan Feng, and Chun Chen.  Accelerating diffusion sampling with classifier-based feature distillation.  _ArXiv_ , abs/2211.12039, 2022.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pages 10347–10357. PMLR, 2021.
  * Van Den Oord et al. (2016) Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  In _International conference on machine learning_ , pages 1747–1756. PMLR, 2016.
  * Vincent (2011) Pascal Vincent.  A connection between score matching and denoising autoencoders.  _Neural Computation_ , 23:1661–1674, 2011.
  * Wang et al. (2022) Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  _ArXiv_ , abs/2212.00774, 2022.
  * Watson et al. (2022) Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi.  Learning fast samplers for diffusion models by differentiating through sample quality.  _ArXiv_ , abs/2202.05830, 2022.
  * Wizadwongsa and Suwajanakorn (2023) Suttisak Wizadwongsa and Supasorn Suwajanakorn.  Accelerating guided diffusion sampling with splitting numerical methods.  _ArXiv_ , abs/2301.11558, 2023.
  * Wu et al. (2022) Lemeng Wu, Dilin Wang, Chengyue Gong, Xingchao Liu, Yunyang Xiong, Rakesh Ranjan, Raghuraman Krishnamoorthi, Vikas Chandra, and Qiang Liu.  Fast point cloud generation with straight flows.  _ArXiv_ , abs/2212.01747, 2022.
  * Xifara et al. (2014) Tatiana Xifara, Chris Sherlock, Samuel Livingstone, Simon Byrne, and Mark Girolami.  Langevin diffusions and the metropolis-adjusted langevin algorithm.  _Statistics & Probability Letters_, 91:14–19, 2014.
  * Xu et al. (2022) Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang.  Neurallift-360: Lifting an in-the-wild 2d photo to a 3d object with 360° views.  _ArXiv_ , abs/2211.16431, 2022.
  * Zhang and Chen (2022) Qinsheng Zhang and Yongxin Chen.  Fast sampling of diffusion models with exponential integrator.  _ArXiv_ , abs/2204.13902, 2022.
  * Zhao et al. (2023) Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu.  Unipc: A unified predictor-corrector framework for fast sampling of diffusion models.  _arXiv preprint arXiv:2302.04867_ , 2023.
  * Zheng et al. (2022) Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar.  Fast sampling of diffusion models via operator learning.  _ArXiv_ , abs/2211.13449, 2022.
  * Zhu et al. (1998) Song-Chun Zhu, Ying Nian Wu, and David Mumford.  Filters, random fields and maximum entropy (frame): Towards a unified theory for texture modeling.  _International Journal of Computer Vision_ , 27:107–126, 1998.