bounds in terms of the $\infty$-norm resulting in the same $\psi$-near
optimality dimension but incurring an additional exponential factor in the
constant $C^{(\infty)}_{\epsilon\leq\eta}\leftarrow
C^{(2)}_{\epsilon\leq\eta}d^{d/2}$.

###### Proof.

Recall that
$||\cdot||_{\infty}\leq||\cdot||_{2}\leq\sqrt{d}||\cdot||_{\infty}$, therefore

$\displaystyle f(x^{*})-f(x)$
$\displaystyle\leq\sigma_{+2}||x-x^{*}||_{2}^{\alpha_{lo}}\leq\sigma_{+\infty}||x-x^{*}||_{\infty}^{\alpha_{lo}}$
(297) $\displaystyle f(x^{*})-f(x)$
$\displaystyle\geq\sigma_{-2}||x-x^{*}||_{2}^{\alpha_{hi}}\geq\sigma_{-\infty}||x-x^{*}||_{\infty}^{\alpha_{hi}}$
(298)

where $\sigma_{+\infty}=\sigma_{+2}d^{\alpha_{lo}/2}$ and
$\sigma_{-\infty}=\sigma_{-2}$. Then

$\displaystyle C^{(\infty)}_{\epsilon\leq\eta}$
$\displaystyle=\Big{(}\frac{\sigma_{+2}d^{\alpha_{lo}/2}}{\psi\sigma_{-2}^{\alpha_{lo}/\alpha_{hi}}}\Big{)}^{d/\alpha_{lo}}=\Big{(}\frac{\sigma_{+2}}{\psi\sigma_{-2}^{\alpha_{lo}/\alpha_{hi}}}\Big{)}^{d/\alpha_{lo}}d^{d/2}=C^{(2)}_{\epsilon\leq\eta}d^{d/2}.$
(299)

∎

Recall, these results apply when $f(x)\geq f(x^{*})-\eta$, i.e., when
$\epsilon\leq\eta$. Otherwise, we can upper bound the number of $\ell$-balls
by considering the entire set $\mathcal{X}$ which has volume $1$. First, we
will bound the constant associated with the volume of a $d$-sphere.

######  Lemma 34.

The volume of a $d$-sphere with radius $r$ and $d$ even is given by
$S_{d}r^{d}$ where $S_{d}^{-1}\leq\sqrt{2\pi d}\Big{(}\frac{d}{2\pi
e}\Big{)}^{d/2}$.

###### Proof.

First, we recall Stirling’s bounds on the factorial: $\sqrt{2\pi
n}(\frac{n}{e})^{n}e^{\frac{1}{12n+1}}<n!<\sqrt{2\pi
n}(\frac{n}{e})^{n}e^{\frac{1}{12n}}$. This will be useful for bounding the
Gamma function: $\Gamma(d)=(d-1)!$ for even $d$.

Given $d$ is even, we start with the exact formula for $S_{d}$:

$\displaystyle S_{d}^{-1}$ $\displaystyle=\frac{\Gamma(d/2+1)}{\pi^{d/2}}$
(300) $\displaystyle=\frac{(d/2)!}{\pi^{d/2}}$ (301)
$\displaystyle<\frac{\sqrt{2\pi(d/2)}(\frac{d/2}{e})^{d/2}e^{\frac{1}{12(d/2)}}}{\pi^{d/2}}$
(302)
$\displaystyle=\frac{\pi^{1/2}d^{1/2}(\frac{d}{2e})^{d/2}e^{\frac{1}{6d}}}{\pi^{d/2}}$
(303) $\displaystyle=\frac{\pi^{1/2}d^{(d+1)/2}e^{\frac{1}{6d}}}{(2\pi
e)^{d/2}}$ (304) $\displaystyle\leq\sqrt{2\pi d}\Big{(}\frac{d}{2\pi
e}\Big{)}^{d/2}.$ (305)

∎

######  Lemma 35 ($N_{\epsilon\geq\eta}\leq C_{\epsilon\geq\eta}$).

The number of disjoint $\ell$-balls that can pack into a set
$\mathcal{X}_{\epsilon\geq\eta}$, $N_{\epsilon\geq\eta}$, is upper bounded by
$C_{\epsilon\geq\eta}$ where
$C_{\epsilon\geq\eta}=S_{d}^{-1}\Big{(}\frac{\sigma_{+}}{\psi\eta}\Big{)}^{d/\alpha_{lo}}$
and $S_{d}$ is the volume constant for a $d$-sphere under a given norm.

###### Proof.

We can upper bound the number of $\ell$-balls needed to pack the entire space
as follows:

$\displaystyle N_{\epsilon\geq\eta}$
$\displaystyle\leq\frac{Vol(\mathcal{X})}{Vol(\mathcal{B}_{\ell})}$ (306)
$\displaystyle=\frac{1}{S_{d}r_{\ell}^{d}}$ (307)
$\displaystyle\leq\frac{1}{S_{d}\Big{(}\frac{\psi\eta}{\sigma_{+}}\Big{)}^{d/\alpha_{lo}}}$
(308)
$\displaystyle=S_{d}^{-1}\Big{(}\frac{\sigma_{+}}{\psi\eta}\Big{)}^{d/\alpha_{lo}}$
(309) $\displaystyle=C_{\epsilon\geq\eta}$ (310)

where $r_{l}$ was defined in equation (288). $S_{d}^{-1}$ has been upper
bounded for the $2$-norm in Lemma 34. For the $\infty$-norm,
$S_{d}^{-1}=2^{-d}$. ∎

######  Corollary 8.

If Assumption 1 is given in terms of the $2$-norm, these can be translated to
bounds in terms of the $\infty$-norm resulting in the same $\psi$-near
optimality dimension but incurring an additional exponential factor in the
constant
$C^{(\infty)}_{\epsilon\geq\eta}=\Big{(}\frac{\sigma_{+2}^{1/\alpha_{hi}}}{2\eta^{1/\alpha_{lo}}}\Big{)}^{d}C^{(\infty)}_{\epsilon\leq\eta}=\Big{(}\frac{\sigma_{+2}^{1/\alpha_{hi}}}{2\eta^{1/\alpha_{lo}}}\Big{)}^{d}C^{(2)}_{\epsilon\leq\eta}d^{d/2}$.

###### Proof.

$\displaystyle C^{(\infty)}_{\epsilon\geq\eta}$
$\displaystyle=2^{-d}\Big{(}\frac{\sigma_{+2}}{\psi\eta}\Big{)}^{d/\alpha_{lo}}d^{d/2}$
(311)
$\displaystyle=2^{-d}\eta^{-d/\alpha_{lo}}\sigma_{-2}^{d/\alpha_{hi}}\Big{(}\frac{\sigma_{+2}}{\psi\sigma_{-2}^{\alpha_{lo}/\alpha_{hi}}}\Big{)}^{d/\alpha_{lo}}d^{d/2}$
(312)
$\displaystyle=2^{-d}\eta^{-d/\alpha_{lo}}\sigma_{-2}^{d/\alpha_{hi}}C^{(\infty)}_{\epsilon\leq\eta}$
(313)
$\displaystyle=\Big{(}\frac{\sigma_{+2}^{1/\alpha_{hi}}}{2\eta^{1/\alpha_{lo}}}\Big{)}^{d}C^{(\infty)}_{\epsilon\leq\eta}.$
(314)

∎

If we further assume $\alpha=\alpha_{lo}=\alpha_{hi}=2$, then we can bound the
number of $\ell$-balls required with a constant, independent of $\epsilon$, as

$\displaystyle C$
$\displaystyle=\max\\{N_{\epsilon\leq\eta},N_{\epsilon\geq\eta}\\}$ (315)
$\displaystyle=\max\Big{\\{}\Big{(}\frac{\kappa}{\psi}\Big{)}^{d/2},\sqrt{2\pi
d}\Big{(}\frac{d\sigma_{max}}{2\pi e\psi\eta}\Big{)}^{d/2}\Big{\\}}$ (316)
$\displaystyle=\beta^{d/2}\psi^{-d/2}d^{\xi/2(d+1)}$ (317)

where $\beta=\kappa$, $\xi=0$ for $N_{\epsilon\leq\eta}$ and
$\beta=\frac{\sigma_{\max}(2\pi)^{1/d}}{2\pi e\eta}<\frac{2\sigma_{\max}}{\pi
e\eta}=\frac{2\kappa}{\pi er_{\eta}^{2}}<\frac{\kappa}{(2r_{\eta})^{2}}$ for
$d\geq 2$, $\xi=1$ for $N_{\epsilon\geq\eta}$. $N_{\epsilon\geq\eta}$
dominates for large $d$. The cross over occurs at

$\displaystyle\Big{(}\frac{\kappa}{\psi}\Big{)}^{d/2}$
$\displaystyle=\sqrt{2\pi d}\Big{(}\frac{d\sigma_{max}}{2\pi
e\psi\eta}\Big{)}^{d/2}$ (318) $\displaystyle\implies\frac{\kappa}{\psi}$
$\displaystyle=(2\pi d)^{1/d}\Big{(}\frac{d\sigma_{max}}{2\pi
e\psi\eta}\Big{)}$ (319) $\displaystyle\implies
r_{\eta}^{2}=\frac{\eta}{\sigma_{-}}$ $\displaystyle=(2\pi
d)^{1/d}\Big{(}\frac{d}{2\pi e}\Big{)}=z(d).$ (320)

where $r_{\eta}$ was defined in equation (287). As $d$ grows and $z(d)$
exceeds $r_{\eta}^{2}$, $N_{\epsilon\geq\eta}$ begins to dominate, therefore
we will upper bound $C$ as

$\displaystyle C$
$\displaystyle\leq\Big{(}\frac{\kappa}{\psi(2r_{\eta})^{2}}\Big{)}^{d/2}d^{\frac{1}{2}(d+1)}.$
(321)

Locality | $C$
---|---
$(*)$ $r_{\eta}^{2}\leq z(d)$ | $N_{\epsilon\geq\eta}\leq\Big{(}\frac{\kappa}{\psi(2r_{\eta})^{2}}\Big{)}^{d/2}d^{\frac{1}{2}(d+1)}=\Big{(}\frac{3\kappa b^{2}}{(2r_{\eta})^{2}}\Big{)}^{d/2}d^{d+\frac{1}{2}}$
$r_{\eta}^{2}>z(d)$ | $N_{\epsilon\leq\eta}\leq\Big{(}\frac{\kappa}{\psi}\Big{)}^{d/2}=\big{(}3\kappa b^{2}\big{)}^{d/2}d^{d/2}$
Table 4: Bounding Constants for $\ell(x,y)=||x-y||_{2}^{2}$,
$\psi=\nu/3=(3b^{2}d)^{-1}$ and $z(d)=(2\pi d)^{1/d}\Big{(}\frac{d}{2\pi
e}\Big{)}$ with smoothness radius $r_{\eta}$ and $\psi=\nu/3$. ($*$) indicates
the case that is more likely for difficult problems.

For convenience, we repeat the other relevant constants in Table 5.

$\ell(x,y)$ | $c$ | $\gamma$ | $\nu$
---|---|---|---
$\ell(x,y)=||x-y||_{2}^{2}$ | $d\big{(}\frac{b}{2}\big{)}^{2}$ | $b^{-2/d}$ | $d^{-1}b^{-2}$
Table 5: Bounding Constants

##  Appendix J D-BLiN

The regret bound for Doubling BLiN [Feng et al., 2022] was originally proved
assuming a standard normal distribution, however, the authors state their
proof can be easily adapted to any sub-Gaussian distribution, which includes
bounded random variables. This matches our setting with bounded payoffs, so we
repeat their analysis here for that setting.

######  Definition 6 (Global Arm Accuracy).

$\mathcal{E}\mathrel{\overset{\makebox[0.0pt]{\mbox{def}}}{=}}\Big{\\{}|\mu(x)-\hat{\mu}_{m}(C)|\leq
r_{m}+\sqrt{c_{1}\frac{\ln T}{n_{m}}},\,\,\forall\,1\leq m\leq
B_{stop}-1,\,\,\forall C\in\mathcal{A}_{m},\,\,\forall x\in C\Big{\\}}$.

Define: $n_{m}=c_{2}\frac{\ln T}{r_{m}^{2}}\implies r_{m}=\sqrt{c_{2}\frac{\ln
T}{n_{m}}}$.

######  Definition 7 (Elimination Rule).

Eliminate $C\in\mathcal{A}_{m}$ if $\hat{\mu}^{\max}_{m}-\hat{\mu}_{m}(C)\geq
2(1+\sqrt{c_{1}/c_{2}})r_{m}=2(\sqrt{c_{2}}+\sqrt{c_{1}})\sqrt{\frac{\ln
T}{n_{m}}}$ where
$\hat{\mu}^{\max}_{m}\mathrel{\overset{\makebox[0.0pt]{\mbox{def}}}{=}}\max_{C\in\mathcal{A}_{m}}\hat{\mu}_{m}(C)$.

######  Lemma 36.

$Pr[\mathcal{E}]\geq 1-2T^{-2(c_{1}/c^{2}-1)}$.

###### Proof.

Assume $y_{C,i}\in[a,b]$ with $c=b-a$ and
$\hat{\mu}(C)=\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}y_{C,i}$. Applying a Hoeffding
inequality gives

$\displaystyle
Pr\Big{[}|\hat{\mu}(C)-\mathbb{E}[\hat{\mu}(C)]|\geq\sqrt{c_{1}\frac{\ln
T}{n_{m}}}\Big{]}$ $\displaystyle\leq 2e^{-2c_{1}\ln T/c^{2}}$ (322)
$\displaystyle=2(e^{\ln T})^{-2c_{1}/c^{2}}$ (323)
$\displaystyle=2T^{-2c_{1}/c^{2}}\,\,\forall C.$ (324)

By Lipschitzness of $\mu$ we also have

$\displaystyle|\mathbb{E}[\hat{\mu}(C)]-\mu(x)|\leq r_{m},\,\,\forall x\in C.$
(325)

Then consider

$\displaystyle\sup_{x\in C}|\mu(x)-\hat{\mu}(C)|$ $\displaystyle=\sup_{x\in
C}|\mu(x)-\mathbb{E}[\hat{\mu}(C)]+\mathbb{E}[\hat{\mu}(C)]-\hat{\mu}(C)|$
(326) $\displaystyle\leq\sup_{x\in
C}\Big{(}|\mu(x)-\mathbb{E}[\hat{\mu}(C)]|+|\mathbb{E}[\hat{\mu}(C)]-\hat{\mu}(C)|\Big{)}$
(327) $\displaystyle=\sup_{x\in
C}|\mu(x)-\mathbb{E}[\hat{\mu}(C)]|+|\mathbb{E}[\hat{\mu}(C)]-\hat{\mu}(C)|$
(328) $\displaystyle\leq\sqrt{c_{1}\frac{\ln T}{n_{m}}}+r_{m}$ (329)

with probability $1-2T^{-2c_{1}/c^{2}}$. The first inequality follows by
triangle inequality and the second follows from equation (325) and considering
the complement of equation (324).

The complement of this result occurs with probability

$\displaystyle Pr\Big{[}\sup_{x\in C}|\mu(x)-\hat{\mu}(C)|\geq
r_{m}+\sqrt{c_{1}\frac{\ln T}{n_{m}}}\Big{]}$ $\displaystyle\leq
2T^{-2c_{1}/c^{2}}.$ (330)

At least $1$ arm is played in each cube $C\in\mathcal{A}_{m}$ for $1\leq m\leq
B_{stop}-1$, therefore, $|\mathcal{A}_{m}|\leq T$ must be true given the exit
condition of the algorithm. In addition, assume $B_{stop}\leq T$ ($B_{stop}$
will be defined such that this is true). Then a union bound over all $T^{2}$
events gives

$\displaystyle Pr\Big{[}$ $\displaystyle\exists
m\in[1,B_{stop}-1],C\in\mathcal{A}_{m}\,\,s.t.\,\,\sup_{x\in
C}|\mu(x)-\hat{\mu}(C)|\geq r_{m}+\sqrt{c_{1}\frac{\ln T}{n_{m}}}\Big{]}$
(331)
$\displaystyle\leq\sum_{m=1}^{B_{stop}-1}\sum_{C\in\mathcal{A}_{m}}Pr\Big{[}\sup_{x\in
C}|\mu(x)-\hat{\mu}(C)|\geq r_{m}+\sqrt{c_{1}\frac{\ln T}{n_{m}}}\Big{]}$
(332)
$\displaystyle\leq\sum_{m=1}^{B_{stop}-1}\sum_{C\in\mathcal{A}_{m}}2T^{-2c_{1}/c^{2}}$
(333) $\displaystyle\leq 2T^{-2c_{1}/c^{2}}T^{2}.$ (334)

Taking the complement of this event and noting that $\sup_{x\in
C}|\mu(x)-\hat{\mu}(C)|\leq r_{m}+\sqrt{c_{1}\frac{\ln
T}{n_{m}}}\implies|\mu(x)-\hat{\mu}(C)|\leq r_{m}+\sqrt{c_{1}\frac{\ln
T}{n_{m}}}\,\,\forall x\in C$ gives the desired result. ∎

######  Lemma 37 (Optimal Arm Survives).

Under event $\mathcal{E}$, the optimal arm
$x^{*}=\operatorname*{arg\,max}\mu(x)$ is not eliminated after the first
$B_{stop}-1$ batches.

###### Proof.

Let $C_{m}^{*}$ denote the cube containing $x^{*}$ in $\mathcal{A}_{m}$. Under
event $\mathcal{E}$, for any cube $C\in\mathcal{A}_{m}$ and $x\in C$, the
following relation shows that $C_{m}^{*}$ avoids the elimination rule in round
$m$:

$\displaystyle\hat{\mu}(C)-\hat{\mu}(C_{m}^{*})$
$\displaystyle\leq\Big{(}\mu(x)+r_{m}+\sqrt{c_{1}\frac{\ln
T}{n_{m}}}\Big{)}+\Big{(}-\mu(x^{*})+r_{m}+\sqrt{c_{1}\frac{\ln
T}{n_{m}}}\Big{)}$ (335) $\displaystyle=\underbrace{(\mu(x)-\mu(x^{*}))}_{\leq
0}+2r_{m}+2\sqrt{c_{1}\frac{\ln T}{n_{m}}}$ (336) $\displaystyle\leq
2\sqrt{c_{2}\frac{\ln T}{n_{m}}}+2\sqrt{c_{1}\frac{\ln T}{n_{m}}}$ (337)
$\displaystyle=2(\sqrt{c_{1}}+\sqrt{c_{2}})\sqrt{\frac{\ln T}{n_{m}}}$ (338)

where the first inequality follows from applying Lemma 36 to upper bound
$\hat{\mu}(C)$ and $\hat{\mu}(C_{m}^{*})$ individually. The remaining steps
use the optimality of $x^{*}$, the definition of $r_{m}$, and the elimination
rule. ∎

######  Lemma 38.

Under event $\mathcal{E}$, for any $1\leq m\leq B_{stop}$, any $C\in A_{m}$
and any $x\in C$, $\Delta_{x}$ satisfies

$\displaystyle\Delta_{x}\leq 4(1+\sqrt{c_{1}/c_{2}})r_{m-1}$ (339)

###### Proof.

For $m=1$, recall that $r_{m}$ is the side length of a cube
$C\in\mathcal{A}_{m}$, therefore, $\Delta_{x}\leq r_{m-1}\leq
4(1+\sqrt{c_{1}/c_{2}})r_{m-1}$ holds directly from the Lipschitzness of
$\mu$.

For $m>1$, let $C_{m-1}^{*}\in\mathcal{A}_{m-1}$ be the cube containing
$x^{*}$. From Lemma 37, this cube has not been eliminated under event
$\mathcal{E}$. For any cube $C\in\mathcal{A}_{m}$ and $x\in C$, it is clear
that $x$ is also in the parent of $C$, denoted $C_{par}$ ($x\in C\subset
C_{par}$). Then for any $x\in C$, it holds that

$\displaystyle\Delta_{x}=\mu(x^{*})-\mu(x)$
$\displaystyle\leq\Big{(}\hat{\mu}_{m-1}(C_{m-1}^{*})+r_{m-1}+\sqrt{c_{1}\frac{\ln
T}{n_{m-1}}}\Big{)}+\Big{(}-\hat{\mu}_{m-1}(C_{par})+r_{m-1}+\sqrt{c_{1}\frac{\ln
T}{n_{m-1}}}\Big{)}$ (340)
$\displaystyle=(\hat{\mu}_{m-1}(C_{m-1}^{*})-\hat{\mu}_{m-1}(C_{par}))+2(\sqrt{c_{1}}+\sqrt{c_{2}})\sqrt{\frac{\ln
T}{n_{m-1}}}$ (341)
$\displaystyle\leq(\hat{\mu}_{m-1}^{\max}-\hat{\mu}_{m-1}(C_{par}))+2(\sqrt{c_{1}}+\sqrt{c_{2}})\sqrt{\frac{\ln
T}{n_{m-1}}}$ (342) $\displaystyle\leq
2(\sqrt{c_{1}}+\sqrt{c_{2}})\sqrt{\frac{\ln
T}{n_{m-1}}}+2(\sqrt{c_{1}}+\sqrt{c_{2}})\sqrt{\frac{\ln T}{n_{m-1}}}$ (343)
$\displaystyle=4(\sqrt{c_{1}}+\sqrt{c_{2}})\sqrt{\frac{\ln T}{n_{m-1}}}$ (344)
$\displaystyle=4(1+\sqrt{c_{1}/c_{2}})r_{m-1}$ (345)

where we have applied Lemma 36 similarly as in Lemma 37 and also used the
definition of $r_{m-1}$. The last two inequalities use the fact that
$\hat{\mu}_{m-1}(C_{m-1}^{*})\leq\hat{\mu}_{m-1}^{\max}$ and $C_{par}$ was not
eliminated. ∎

######  Theorem 3.

With probability exceeding $1-2T^{-2(c_{1}/c^{2}-1)}$ , the $T$-step total
regret $R(T)$ of BLiN with Doubling Edge-length Sequence (D-BLiN) Feng et al.
[2022] satisfies

$\displaystyle R(T)\leq
8(1+\sqrt{c_{1}/c_{2}})(2c_{2}+1)\ln(T)^{\frac{1}{d_{z}+2}}T^{\frac{d_{z}+1}{d_{z}+2}}$
(346)

where $d_{z}$ is the zooming dimension of the problem instance. In addition,
D-BLiN only needs no more than $B^{*}=\frac{\log 2(T)-\log
2(\ln(T))}{d_{z}+2}+2$ rounds of communications to achieve this regret rate.

###### Proof.

Since $r_{m}=\frac{r_{m-1}}{2}\implies r_{m-1}=2r_{m}$ for the Doubling Edge-
length Sequence, Lemma 38 implies that every cube $C\in A_{m}$ is a subset of
$S(8(1+\sqrt{c_{1}/c_{2}})r_{m})$. Thus from the definition of zooming number
(Corollary 5 with appropriate condition), we have

$\displaystyle|\mathcal{A}_{m}|\leq N_{r_{m}}\leq C_{z}r_{m}^{-d_{z}}.$ (347)

Fix any positive number $B$. Also by Lemma 38, we know that any arm played
after batch $B$ incurs a regret bounded by $8(1+\sqrt{c_{1}/c_{2}})r_{B}$,
since the cubes played after batch $B$ have edge length no larger than
$r_{B}$. Then the total regret that occurs after batch $B$ is bounded by
$8(1+\sqrt{c_{1}/c_{2}})r_{B}T$ (where $T$ is an upper bound on the number of
arms).

Thus the regret can be bounded as

$\displaystyle R(T)$
$\displaystyle\leq\sum_{m=1}^{B}\sum_{C\in\mathcal{A}_{m}}\sum_{i=1}^{n_{m}}\Delta_{x_{C,i}}+8(1+\sqrt{c_{1}/c_{2}})r_{B}T$
(348)

where the first term bounds the regret in the first $B$ batches of D-BLiN, and
the second term bounds the regret after the first $B$ batches. If the
algorithm stops at batch $\tilde{B}<B$ , we define $\mathcal{A}_{m}=$ for any
$\tilde{B}<m\leq B$ and inequality equation (348) still holds.

By Lemma 38, we have $\Delta_{x_{C,i}}\leq 8(1+\sqrt{c_{1}/c_{2}})r_{m}$ for
all $C\in\mathcal{A}_{m}$. We can thus bound equation (348) by

$\displaystyle R(T)$ $\displaystyle\leq\sum_{m=1}^{B}|\mathcal{A}_{m}|\cdot
n_{m}\cdot 8(1+\sqrt{c_{1}/c_{2}})r_{m}+8(1+\sqrt{c_{1}/c_{2}})r_{B}T$ (349)
$\displaystyle\leq\sum_{m=1}^{B}N_{r_{m}}\cdot n_{m}\cdot
8(1+\sqrt{c_{1}/c_{2}})r_{m}+8(1+\sqrt{c_{1}/c_{2}})r_{B}T$ (350)
$\displaystyle=\sum_{m=1}^{B}N_{r_{m}}\cdot c_{2}\frac{\ln T}{r_{m}^{2}}\cdot
8(1+\sqrt{c_{1}/c_{2}})r_{m}+8(1+\sqrt{c_{1}/c_{2}})r_{B}T$ (351)
$\displaystyle=\sum_{m=1}^{B}N_{r_{m}}\cdot\frac{\ln T}{r_{m}}\cdot
8c_{2}(1+\sqrt{c_{1}/c_{2}})+8(1+\sqrt{c_{1}/c_{2}})r_{B}T$ (352)

where equation (350) uses equation (347), and equation (351) uses equality
$n_{m}=c_{2}\frac{\ln T}{r_{m}^{2}}$. Since $r_{m}=2^{-m+1}$ and
$N_{r_{m}}\leq r_{m}^{-d_{z}}\leq 2^{(m-1)d_{z}}$, we have

$\displaystyle R(T)$
$\displaystyle\leq\sum_{m=1}^{B}2^{(m-1)d_{z}}\cdot\frac{\ln T}{2^{-m+1}}\cdot
8c_{2}(1+\sqrt{c_{1}/c_{2}})+8(1+\sqrt{c_{1}/c_{2}})2^{-B+1}T$ (353)
$\displaystyle=8(1+\sqrt{c_{1}/c_{2}})\Big{[}c_{2}\ln
T\sum_{m=1}^{B}2^{(m-1)(d_{z}+1)}+2^{-B+1}T\Big{]}.$ (354)

Continuing we find

$\displaystyle R(T)$ $\displaystyle\leq 8(1+\sqrt{c_{1}/c_{2}})\Big{[}c_{2}\ln
T\sum_{m=1}^{B}2^{(m-1)(d_{z}+1)}+2^{-B+1}T\Big{]}$ (355)
$\displaystyle=8(1+\sqrt{c_{1}/c_{2}})\Big{[}c_{2}\ln
T\sum_{m=1}^{B}\big{(}2^{d_{z}+1}\big{)}^{m-1}+2^{-B+1}T\Big{]}$ (356)
$\displaystyle=8(1+\sqrt{c_{1}/c_{2}})\Big{[}c_{2}\ln
T\sum_{m=0}^{B-1}\big{(}2^{d_{z}+1}\big{)}^{m}+2^{-B+1}T\Big{]}$ (357)
$\displaystyle=8(1+\sqrt{c_{1}/c_{2}})\Big{[}c_{2}\ln
T\Big{(}\frac{2^{B(d_{z}+1)}-1}{2^{d_{z}+1}-1}\Big{)}+2^{-B+1}T\Big{]}\,\,\text{
via geometric series}$ (358) $\displaystyle\leq
8(1+\sqrt{c_{1}/c_{2}})\Big{[}c_{2}\ln
T\Big{(}\frac{2^{B(d_{z}+1)}}{2^{d_{z}+1}-1}\Big{)}+2^{-B+1}T\Big{]}$ (359)
$\displaystyle\leq 8(1+\sqrt{c_{1}/c_{2}})\Big{[}c_{2}\ln
T\Big{(}2\cdot\frac{2^{B(d_{z}+1)}}{2^{d_{z}+1}}\Big{)}+2^{-B+1}T\Big{]}$
(360) $\displaystyle=8(1+\sqrt{c_{1}/c_{2}})\Big{[}2c_{2}2^{(B-1)(d_{z}+1)}\ln
T+2^{-(B-1)}T\Big{]}.$ (361)

This inequality holds for any positive $B$. By choosing
$B^{*}=1+\frac{\log_{2}(\frac{T}{\ln T})}{d_{z}+2}$, we have

$\displaystyle R(T)$ $\displaystyle\leq
8(1+\sqrt{c_{1}/c_{2}})\Big{[}2c_{2}\Big{(}\frac{T}{\ln
T}\Big{)}^{\frac{(d_{z}+1)}{(d_{z}+2)}}\ln T+\Big{(}\frac{\ln
T}{T}\Big{)}^{\frac{1}{(d_{z}+2)}}T\Big{]}$ (362)
$\displaystyle=8(1+\sqrt{c_{1}/c_{2}})\Big{[}2c_{2}T^{\frac{(d_{z}+1)}{(d_{z}+2)}}\ln
T^{1-\frac{(d_{z}+1)}{(d_{z}+2)}}+T^{1-\frac{1}{(d_{z}+2)}}\ln
T^{\frac{1}{(d_{z}+2)}}\Big{]}$ (363)
$\displaystyle=8(1+\sqrt{c_{1}/c_{2}})\Big{[}2c_{2}T^{\frac{(d_{z}+1)}{(d_{z}+2)}}\ln
T^{\frac{1}{(d_{z}+2)}}+T^{\frac{(d_{z}+1)}{(d_{z}+2)}}\ln
T^{\frac{1}{(d_{z}+2)}}\Big{]}$ (364)
$\displaystyle=8(1+\sqrt{c_{1}/c_{2}})(2c_{2}+1)T^{\frac{(d_{z}+1)}{(d_{z}+2)}}\ln
T^{\frac{1}{(d_{z}+2)}}.$ (365)

∎

######  Corollary 9.

Setting $c_{1}=2c^{2}$ and $c_{2}=\Big{(}\frac{c^{2}}{2}\Big{)}^{1/3}$
simplifies Theorem 3 such that

$\displaystyle R(T)$ $\displaystyle\leq
8(1+(4c^{2})^{1/3})^{2}T^{\frac{(d_{z}+1)}{(d_{z}+2)}}\ln
T^{\frac{1}{(d_{z}+2)}}.$ (366)

with probability $1-2T^{-2}$.

###### Proof.

∎

##  Appendix K Experimental Setup and Details

Here we provide further details on the experiments.

###  K.1 GAMBIT

The seven methods from the gambit [McKelvey et al., 2016] library that we
tested on the $3$-player and $4$-player Blotto games are listed below (with
runtimes). Only gambit-enumpoly and gambit-enumpure are able to return any NE
for $3$-player Blotto within a $1$ hour time limit (and only pure equilibria).
And only gambit-enumpure returns any NE for the $4$-player game.

  * •

gambit-enumpoly [73 sec 3-player, timeout 4-player]

  * •

gambit-enumpure [72 sec 3-player, 45 sec 4-player]

  * •

gambit-gnm

  * •

gambit-ipa

  * •

gambit-liap

  * •

gambit-logit

  * •

gambit-simpdiv

###  K.2 Loss Visualization and Rank Test

Figure 5 and claims made in Section 5 analyze several classical matrix games.
We report the payoff matrices in standard row-player / column-player payoff
form below. All games are then shifted and scaled so payoffs lie in $[0,1]$
(i.e., first by subtracting the minimum and then scaling by the max).

RPS:

$\displaystyle\begin{bmatrix}0/0&-1/1&1/-1\\\ 1/-1&0/0&-1/1\\\
-1/1&1/-1&0/0\end{bmatrix}.$ (367)

Chicken:

$\displaystyle\begin{bmatrix}0/0&-1/1\\\ 1/-1&-3/-3\end{bmatrix}.$ (368)

Matching Pennies:

$\displaystyle\begin{bmatrix}1/-1&-1/1\\\ -1/1&1/-1\end{bmatrix}.$ (369)

Modified-Shapleys:

$\displaystyle\begin{bmatrix}1/-0.5&0/1&0.5/0\\\ 0.5/0&1/-0.5&0/1\\\
0/1&0.5/0&1/-0.5\end{bmatrix}.$ (370)

Prisoner’s Dilemma:

$\displaystyle\begin{bmatrix}-1/-1&-3/0\\\ 0/-3&-2/-2\end{bmatrix}.$ (371)

####  K.2.1 Loss on Familiar Games

Figure 5: Upper Bound ($\epsilon\leq f(\mathcal{L}^{\tau})$) Heatmap
Visualization. The first row examines the loss landscape for the classic anti-
coordination game of Chicken (Nash equilibria:
$(0,1),(1,0),(\nicefrac{{2}}{{3}},\nicefrac{{1}}{{3}})$) while the second row
examines the Prisoner’s dilemma (Unique Nash equilibrium: $(0,0)$).
Temperature increases for each plot moving to the right. For high
temperatures, interior (fully-mixed) strategies are incentivized while for
lower temperatures, nearly pure strategies can achieve minimum exploitability.
For zero temperature, pure strategy equilibria (e.g., defect-defect) are not
captured by the loss as illustrated by the bottom-left Prisoner’s Dilemma plot
with a constant loss surface.

###  K.3 Saddle Point Analysis

To generate Figure 2, we follow a procedure similar to the study of MNIST in
[Dauphin et al., 2014] (Section 3 of Supp.). Their recommended procedure
searches for critical points in two ways. The first repeats a randomized,
iterative optimization process $20$ times. They then sample one these $20$
trials at random, select a random point along the descent trajectory, and
search for a critical point (using Newton’s method) nearby. They repeat this
sampling process $100$ times. The second approach randomly selects a feasible
point in the decision set and searches for a critical point nearby (again
using Newton’s method). They also perform this $100$ times.

Our protocol differs from theirs slightly in a few respects. One, we use SGD,
rather than the saddle-free Newton algorithm to trace out an initial descent
trajectory. Two, we do not add noise to strategies along the descent
trajectory prior to looking for critical points. Thirdly, we minimize gradient
norm rather than use Newton’s method to look for critical points. Lastly, we
use different experimental hyperparameters. We run SGD for $1000$ iterations
rather than $20$ epochs and rerun SGD $100$ times rather than $20$. We sample
$1000$ points for each of the two approaches for finding critical points.

###  K.4 SGD on Classical Games

The games examined in Figure 3 were all taken from [Gemp et al., 2022]. Each
is available via open source implementations in OpenSpiel [Lanctot et al.,
2019] or GAMUT [Nudelman et al., 2004].

We compare against several other baselines, replicating the experiments in
[Gemp et al., 2022]. RM indicates regret-matching and FTRL indicates follow-
the-regularized-leader. These are, arguably, the two most popular scalable
stochastic algorithms for approximating Nash equilibria.
${}^{y}\text{QRE}^{auto}$ is a stochastic algorithm developed in [Gemp et al.,
2022].

For each of the experiments, we sweep over learning rates in log-space from
$10^{-3}$ to $10^{2}$ in increments of $1$. We also consider whether to run
SGD with the projected-gradient and whether to constrain iterates to the
simplex via Euclidean projection or entropic mirror descent [Beck and
Teboulle, 2003]. We then presented the results of the best performing
hyperparameters. This was the same approach taken in [Gemp et al., 2022].

##### Saddle Points in Blotto

To confirm the existence of saddle points, we computed the Hessian of
$\mathcal{L}(\bm{x_{10k}})$ for SGD ($s=\infty$), deflated the matrix by
removing from its eigenvectors all directions orthogonal to the simplex, and
then computed its top-$(n\bar{m}-n)$ eigenvalues. We do this because there
always exists a $n$-dimensional nullspace of the Hessian at zero temperature
that lies outside the tangent space of the simplex, and we only care about
curvature within the tangent space. Specificaly, at an equilibrium $\bm{x}$,
if we compute $z^{\top}Hess(\mathcal{L})z$ where $z$ is formed as a linear
combination of the vectors
$\\{[x_{1},0,\ldots,0]^{\top},\ldots,[0,\ldots,x_{n}]^{\top}$, then each block
$\tilde{B}_{kl}$ is identically zero at an equilibrium:
$\tilde{B}_{kl}x_{l}=\sqrt{\eta_{k}}[I-\frac{1}{m_{k}}\mathbf{1}\mathbf{1}^{\top}]H^{k}_{kl}x_{l}=\sqrt{\eta_{k}}\Pi_{T\Delta}(\nabla^{k}_{x_{k}})=0$.
By Lemma 17, this implies there is zero curvature of the loss in the direction
$z$: $z^{\top}Hess(\mathcal{L})z=0$.

###  K.5 BLiN on Artificial Game

To construct the $7$-player, $2$-action, symmetric, artifical game in Figure
4, we used the following coefficients (discovered by trial-and-error):

$\displaystyle\begin{bmatrix}0.09906873&0&0.23116037&0&0.62743528&0&0.19813746\\\
0&0.33022909&0&0.03302291&0&0.62743528&0\end{bmatrix}.$ (372)

The first row indicates the payoffs received when player $i$ plays action $0$
and the background population plays any of the possible joint actions (number
of combinations with replacement). For example, the first column indicates the
payoff when all background players play action $0$. The second column
indicates all background players play action $0$ except for one which plays
action $1$, and so on. The last column indicates all background players play
action $1$. These $2n$ scalars uniquely define the payoffs of a symmetric
game.

Given that this game only has two actions, we represent a mixed strategy by a
single scalar $p\in[0,1]$, i.e., the probability of the first action.
Furthermore, this game is symmetric and we seek a symmetric equilibrium, so we
can represent a full Nash equilibrium by this single scalar $p$. This reduces
our search space from $7\times 2=14$ variables to $1$ variable (and obviates
any need for a map $s$ from the unit hypercube to the simplex—see Lemma 25).


# EM Distillation for One-step Diffusion Models

Sirui Xie1,2,3 Zhisheng Xiao2 Diederik P. Kingma1 Tingbo Hou2
Ying Nian Wu3 Kevin Murphy1 Tim Salimans1 Ben Poole1 Ruiqi Gao1
1Google DeepMind 2Google Research 3UCLA

###### Abstract

While diffusion models can learn complex distributions, sampling requires a
computationally expensive iterative process. Existing distillation methods
enable efficient sampling, but have notable limitations, such as performance
degradation with very few sampling steps, reliance on training data access, or
mode-seeking optimization that may fail to capture the full distribution. We
propose EM Distillation (EMD), a maximum likelihood-based approach that
distills a diffusion model to a one-step generator model with minimal loss of
perceptual quality. Our approach is derived through the lens of Expectation-
Maximization (EM), where the generator parameters are updated using samples
from the joint distribution of the diffusion teacher prior and inferred
generator latents. We develop a reparametrized sampling scheme and a noise
cancellation technique that together stabilizes the distillation process. We
further reveal an interesting connection of our method with existing methods
that minimize mode-seeking KL. EMD outperforms existing one-step generative
methods in terms of FID scores on ImageNet-64 and ImageNet-128, and compares
favorably with prior work on distilling text-to-image diffusion models.

##  1 Introduction

Diffusion models [1, 2, 3] have enabled high-quality generation of images [4,
5, 6], videos [7, 8], and other modalities [9, 10, 11]. Diffusion models use a
forward process to create a sequence of distributions that transform the
complex data distribution into a Gaussian distribution, and learn the score
function for each of these intermediate distributions. Sampling from a
diffusion model reverses this forward process to create data from random noise
by solving an SDE, or an equivalent probability flow ODE [12]. Typically,
solving this differential equation requires a significant number of
evaluations of the score function, resulting in a high computational cost.
Reducing this cost to single function evaluation would enable applications in
real-time generation.

To enable efficient sampling from diffusion models, two distinct approaches
have emerged: (1) trajectory distillation methods [13, 14, 15, 16, 17, 18]
that accelerate solving the differential equation, and (2) distribution
matching approaches [19, 20, 21, 22, 23] that learn implicit generators to
match the marginals learned by the diffusion model. Trajectory distillation-
based approaches have greatly reduced the number of steps required to produce
samples, but continue to face challenges in the 1-step generation regime.
Distribution matching approaches can enable the use of arbitrary generators
and produce more compelling results in the 1-step regime, but often fail to
capture the full distribution due to the _mode-seeking_ nature of the
divergences they minimize.

In this paper, we propose EM Distillation (EMD), a diffusion distillation
method that minimizes an approximation of the _mode-covering_ divergence
between a pre-trained diffusion teacher model and a latent-variable student
model. The student enables efficient generation by mapping from noise to data
in just one step. To achieve Maximum Likelihood Estimation (MLE) of the
marginal teacher distribution for the student, we propose a method similar to
the Expectation-Maximization (EM) framework [24], which alternates between an
Expectation-step (E-step) that estimates the learning gradients with Monte
Carlo samples, and a Maximization-step (M-step) that updates the student
through gradient ascent. As the target distribution is represented by the pre-
trained score function, the E-step in the original EM that first samples a
datapoint and then infers its implied latent variable would be expensive. We
introduce an alternative MCMC sampling scheme that jointly updates the data
and latent pairs initialized from student samples, and develop a
reparameterized approach that simplifies hyperparameter tuning and improves
performance for short-run MCMC [25]. For the optimization in the M-step given
these joint samples, we discover a tractable linear noise term in the learning
gradient, whose removal significantly reduces variances. Additionally, we
identify a connection to Variational Score Distillation [9, 26] and Diff-
Instruct [22], and show how the strength of the MCMC sampling scheme can
interpolate between mode-seeking and mode-covering divergences. Empirically,
we first demonstrate that a special case of EMD, which is equivalent to the
Diff-Instruct [22] baseline, can be readily scaled and improved to achieve
strong performance. We further show that the general formulation of EMD that
leverages multi-step MCMC can achieve even more competitive results. For
ImageNet-64 and ImageNet-128 conditional generation, EMD outperforms existing
one-step generation approaches with FID scores of 2.20 and 6.0. EMD also
performs favorably on one-step text-to-image generation by distilling from
Stable Diffusion models.

##  2 Preliminary

###  2.1 Diffusion models and score matching

Diffusion models [1, 2], also known as score-based generative models [27, 3],
consist of a forward process that gradually injects noise to the data
distribution and a reverse process that progressively denoises the
observations to recover the original data distribution
$p_{\text{data}}({\mathbf{x}}_{0})$. This results in a sequence of noise
levels $t\in(0,1]$ with conditional distributions
$q_{t}({\mathbf{x}}_{t}|{\mathbf{x}}_{0})=\mathcal{N}(\alpha_{t}{\mathbf{x}}_{0},\sigma_{t}^{2}{\bm{I}})$,
whose marginals are $q_{t}({\mathbf{x}}_{t})$. We use a variance-preserving
forward process [3, 28, 29] such that $\sigma_{t}^{2}=1-\alpha_{t}^{2}$. Song
et al. [3] showed that the reverse process can be simulated with a reverse-
time Stochastic Differential Equation (SDE) that depends only on the time-
dependent score function $\nabla_{{\mathbf{x}}_{t}}\log
p_{t}({\mathbf{x}}_{t})$ of the marginal distribution of the noisy
observations. This score function can be estimated by a neural network
$s_{\bm{\phi}}({\mathbf{x}}_{t},t)$ through (weighted) denoising score
matching [30, 31]:

$\mathcal{J}({\bm{\phi}})=\operatorname{\mathbb{E}}\nolimits_{p_{\rm
data}({\mathbf{x}}_{0}),p(t),q_{t}({\mathbf{x}}_{t}|{\mathbf{x}}_{0})}\left[w(t)\lVert
s_{\bm{\phi}}({\mathbf{x}}_{t},t)-\nabla_{{\mathbf{x}}_{t}}\log
q_{t}({\mathbf{x}}_{t}|{\mathbf{x}}_{0})\rVert_{2}^{2}\right],$ (1)

where $w(t)$ is the weighting function and $p(t)$ is the noise schedule.

###  2.2 MCMC with Langevin dynamics

While solving the reverse-time SDE results in a sampling process that
traverses noise levels, simulating Langevin dynamics [32] results in a sampler
that converges to and remains at the data manifold of a target distribution.
As a particularly useful Markov Chain Monte Carlo (MCMC) sampling method for
continuous random variables, Langevin dynamics generate samples from a target
distribution $\rho({\mathbf{x}})$ by iterating through

${\mathbf{x}}^{i+1}={\mathbf{x}}^{i}+\gamma\nabla_{{\mathbf{x}}}\log\rho({\mathbf{x}}^{i})+\sqrt{2\gamma}{\mathbf{n}},$
(2)

where $\gamma$ is the stepsize,
${\mathbf{n}}\sim\mathcal{N}({\bm{0}},{\bm{I}})$, and $i$ indexes the sampling
timestep. Langevin dynamics has been widely adopted for sampling from
diffusion models [27, 3] and energy-based models [33, 34, 35, 36]. Convergence
of Langevin dynamics requires a large number of sampling steps, especially for
high-dimensional data. In practice, short-run variants with early termination
have been succesfully used for learning of EBMs [25, 37, 38].

###  2.3 Maximum Likelihood and Expectation-Maximization

Expectation-Maximization (EM) [24] is a maximum likelihood estimation
framework to learn latent variable models:
$p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})=p_{\bm{\theta}}({\mathbf{x}}|{\mathbf{z}})p({\mathbf{z}})$,
such that the marginal distribution
$p_{\bm{\theta}}({\mathbf{x}})=\int{p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})}d{\mathbf{z}}$
approximates the target distribution $q({\mathbf{x}})$. It originates from the
generic training objective of maximizing the log-likelihood function over
parameters:
$\mathcal{L}({\bm{\theta}})=\operatorname{\mathbb{E}}_{q({\mathbf{x}})}[\log
p_{\bm{\theta}}({\mathbf{x}})]$, which is equivalent to minimizing the forward
KL divergence
$D_{\mathrm{KL}}(q({\mathbf{x}})||p_{\bm{\theta}}({\mathbf{x}}))$ [39]. Since
the marginal distribution $p_{\bm{\theta}}({\mathbf{x}})$ is usually
analytically intractable, EM involves an E-step that expresses the gradients
over the model parameters $\theta$ with an expectation formula

$\begin{split}\nabla_{\bm{\theta}}\mathcal{L}({\bm{\theta}})&=\nabla_{{\bm{\theta}}}\operatorname{\mathbb{E}}\nolimits_{q({\mathbf{x}})}[\log
p_{{\bm{\theta}}}({\mathbf{x}})]=\operatorname{\mathbb{E}}\nolimits_{q({\mathbf{x}})p_{{\bm{\theta}}}({\mathbf{z}}|{\mathbf{x}})}[\nabla_{{\bm{\theta}}}\log
p_{{\bm{\theta}}}({\mathbf{x}},{\mathbf{z}})],\end{split}$ (3)

where
$p_{\bm{\theta}}({\mathbf{z}}|{\mathbf{x}})=\frac{p_{\bm{\theta}}({\mathbf{x}}|{\mathbf{z}})p({\mathbf{z}})}{p_{\bm{\theta}}({\mathbf{x}})}$
is the posterior distribution of ${\mathbf{z}}$ given ${\mathbf{x}}$. See
Appendix A for a detailed derivation. The expectation can be approximated by
Monte Carlo samples drawn from the posterior using e.g. MCMC sampling
techniques. The estimated gradients are then used in an M-step to optimize the
parameters. [40] learned generator networks with an instantiation of this EM
framework where E-steps leverage Langevin dynamics for drawing samples.

###  2.4 Variational Score Distillation and Diff-Intruct

Our method is also closely related to Score Distillation Sampling (SDS) [9],
Variational Score Distillation (VSD) [26] and Diff-Instruct [22], which have
been used for distilling diffusion models into a single-step generator [23,
41]. The generator produces clean images
${\mathbf{x}}_{0}=g_{\bm{\theta}}({\mathbf{z}})$ with
$p({\mathbf{z}})=\mathcal{N}({\bm{0}},{\bm{I}})$, and can be diffused to noise
level $t$ to form a latent variable model
$p_{{\bm{\theta}},t}({\mathbf{x}}_{t},{\mathbf{z}})=p_{{\bm{\theta}},t}({\mathbf{x}}_{t}|{\mathbf{z}})p({\mathbf{z}})$,
$p_{{\bm{\theta}},t}({\mathbf{x}}_{t}|{\mathbf{z}})=\mathcal{N}(\alpha_{t}g_{\bm{\theta}}({\mathbf{z}}),\sigma_{t}^{2}{\bm{I}})$.
This model is trained to match the marginal distributions
$p_{{\bm{\theta}},t}({\mathbf{x}}_{t})$ and $q_{t}({\mathbf{x}}_{t})$ by
minimizing their reverse KL divergence. Integrating over all noise levels, the
objective is to minimize $\mathcal{J}({\bm{\theta}})$ where

$\mathcal{J}({\bm{\theta}})=\operatorname{\mathbb{E}}\nolimits_{p(t)}[\tilde{w}(t)D_{\mathrm{KL}}(p_{{\bm{\theta}},t}({\mathbf{x}}_{t})||q_{t}({\mathbf{x}}_{t}))]=\operatorname{\mathbb{E}}\nolimits_{p(t)}\left[\tilde{w}(t)\int
p_{{\bm{\theta}},t}({\mathbf{x}}_{t})\log\frac{p_{{\bm{\theta}},t}({\mathbf{x}}_{t})}{q_{t}({\mathbf{x}}_{t})}d{\mathbf{x}}_{t}\right].$
(4)

The gradient for this objective in Eq. 4 can be written as

$\nabla_{\bm{\theta}}\mathcal{J}({\bm{\theta}})=\operatorname{\mathbb{E}}\nolimits_{p(t),p(\epsilon),p({\mathbf{z}})}[-\tilde{w}(t){(\underbrace{\nabla_{{\mathbf{x}}_{t}}\log
q_{t}({\mathbf{x}}_{t})}_{\text{teacher
score}}-\underbrace{\nabla_{{\mathbf{x}}_{t}}\log
p_{{\bm{\theta}},t}({\mathbf{x}}_{t})}_{\text{learned
}s_{\phi}({\mathbf{x}}_{t},t)})\nabla_{\bm{\theta}}\alpha_{t}g_{\bm{\theta}}({\mathbf{z}})}],$
(5)

where the teacher score is provided by the pre-trained diffusion model. In
SDS, $\nabla_{{\mathbf{x}}_{t}}\log p_{{\bm{\theta}},t}({\mathbf{x}}_{t})$ is
the known analytic score function of the Gaussian generator. In VSD and Diff-
Instruct, an auxiliary score network $s_{\phi}({\mathbf{x}}_{t},t)$ is learned
to estimate it. The training alternates between learning the generator network
$g_{\bm{\theta}}$ with the gradient update in Eq. 5 and learning the score
network $s_{\bm{\phi}}$ with the denoising score matching loss in Eq. 1.

##  3 Method

(a) ImageNet

(b) Text-to-image (SD embedding space)

(c) Text-to-image (SD image space)

Figure 1: Before and after MCMC correction. In (a)(b), the left columns are
${\mathbf{x}}=g_{\bm{\theta}}({\mathbf{z}})$, the right columns are updated
${\mathbf{x}}$ after 300 steps of MCMC sampling jointly on ${\mathbf{x}}$ and
${\mathbf{z}}$. (a) illustrates the effect of correction in ImageNet. Note
that the off-manifold images are corrected. (b) illustrates the correction in
the embedding space of Stable Diffusion v1.5, which are decoded to image space
in (c). Note the disentanglement of the cats and sharpness of the sofa. Zoom
in for better viewing.

###  3.1 EM Distillation

We consider formulating the problem of distilling a pre-trained diffusion
model to a deep latent-variable model
$p_{{\bm{\theta}},t}({\mathbf{x}}_{t},{\mathbf{z}})$ defined in Section 2.4
using the EM framework introduced in Section 2.3. For simplicity, we begin
with discussing the framework at a single noise level and drop the subscript
$t$. We will revisit the integration over all noise levels in Section 3.3.
Assume the target distribution $q({\mathbf{x}})$ is represented by the
diffusion model where we can access the score function
$\nabla_{\mathbf{x}}\log q({\mathbf{x}})$. Theoretically speaking, the
generator network $g_{\bm{\theta}}({\mathbf{z}})$ can employ any architecture
including ones where the dimensionality of the latents differs from the data
dimensionality. In this work, we reuse the diffusion denoiser parameterization
as in other work on one-step distillation:
$g_{\bm{\theta}}({\mathbf{z}})=\hat{{\mathbf{x}}}_{\bm{\theta}}({\mathbf{z}},t^{*})$,
where $\hat{{\mathbf{x}}}_{\bm{\theta}}$ is the ${\mathbf{x}}$-prediction
function inherited from the teacher diffusion model, and $t^{*}$ remains a
hyper-parameter.

Naive adoption of the EM algorithm involves two steps: (1) draw samples from
the target diffusion model $q({\mathbf{x}})$ and (2) sample the latent
variable ${\mathbf{z}}$ from $p_{\bm{\theta}}({\mathbf{z}}|{\mathbf{x}})$ with
e.g. MCMC techniques. Both steps can be highly non-trivial and computationally
expensive, so here we present an alternative approach to sampling the same
target distribution that avoids directly sampling from the pretrained
diffusion model, by instead running MCMC from the joint distribution of
$({\mathbf{x}},{\mathbf{z}})$. We initialize this sampling process using a
joint sample from the student: drawing ${\mathbf{z}}\sim p({\mathbf{z}})$ and
${\mathbf{x}}\sim p_{\theta}({\mathbf{x}}|{\mathbf{z}})$. This sampled
${\mathbf{x}}$ is no longer drawn from $q({\mathbf{x}})$, but ${\mathbf{z}}$
is guaranteed to be a valid sample from the posterior
$p_{\theta}({\mathbf{z}}|{\mathbf{x}})$. We then run MCMC to correct the
sampled pair towards the desired distribution:
$\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})\vcentcolon=q({\mathbf{x}})p_{\bm{\theta}}({\mathbf{z}}|{\mathbf{x}})=p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})\frac{q({\mathbf{x}})}{p_{\theta}({\mathbf{x}})}$
(see Fig. 1 for a visualization of this process). If $q({\mathbf{x}})$ and
$p_{\theta}({\mathbf{x}})$ are close to each other, then
$\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})$ is close to
$p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})$. In that case, initializing the
_joint_ sampling of $\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})$ with pairs
of $({\mathbf{x}},{\mathbf{z}})$ from
$p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})$ could significantly accelerate
both sampling of ${\mathbf{x}}$ and inference of ${\mathbf{z}}$. Assuming MCMC
converges, we can use the resulting samples to estimate the learning gradients
for EM:

$\begin{split}\nabla_{\bm{\theta}}\mathcal{L}({\bm{\theta}})&=\operatorname{\mathbb{E}}\nolimits_{\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})}\left[\nabla_{\bm{\theta}}\log
p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})\right]=\operatorname{\mathbb{E}}\nolimits_{\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})}\left[-\frac{\nabla_{\bm{\theta}}\lVert{{\mathbf{x}}}-\alpha
g_{\bm{\theta}}({\mathbf{z}})\rVert_{2}^{2}}{2\sigma^{2}}\right].\end{split}$
(6)

We abbreviate our method as EMD hereafter. To successfully learn the student
network with EMD, we need to identify efficient approaches to sample from
$\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})$.

###  3.2 Reparametrized sampling and noise cancellation

As an initial strategy, we consider Langevin dynamics which only requires the
score functions:

$\begin{split}\nabla_{{\mathbf{x}}}\log\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})&=\underbrace{\nabla_{{\mathbf{x}}}\log
q({\mathbf{x}})}_{\text{teacher score}}-\underbrace{\nabla_{{\mathbf{x}}}\log
p_{\bm{\theta}}({\mathbf{x}})}_{\text{learned
}s_{\phi}({\mathbf{x}})}+\underbrace{\nabla_{{\mathbf{x}}}\log
p_{\bm{\theta}}({\mathbf{x}}|{\mathbf{z}})}_{-\frac{{\mathbf{x}}-\alpha
g_{\bm{\theta}}({\mathbf{z}})}{\sigma^{2}}},\\\
\nabla_{{\mathbf{z}}}\log\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})&=\nabla_{{\mathbf{z}}}\log
p_{\bm{\theta}}({\mathbf{x}}|{\mathbf{z}})+\nabla_{{\mathbf{z}}}\log
p_{\bm{\theta}}({\mathbf{z}})=-\frac{{\mathbf{x}}-\alpha
g_{\bm{\theta}}({\mathbf{z}})}{\sigma^{2}}\nabla_{\mathbf{z}}g_{\bm{\theta}}({\mathbf{z}})-{\mathbf{z}}.\end{split}$
(7)

While we do not have access to the score of the student,
$\nabla_{\mathbf{x}}\log p_{\bm{\theta}}({\mathbf{x}})$, we can approximate it
with a learned score network $s_{\bm{\phi}}$ estimated with denoising score
matching as in VSD [26] and Diff-Instruct [22]. As will be covered in Section
3.3, this score network is estimated at all noise levels. The Langevin
dynamics defined in Eq. 7 can therefore be simulated at any noise level.

Running Langevin MCMC is expensive and requires careful tuning, and we found
this challenging in the context of diffusion model distillation where
different noise levels have different optimal step sizes. We leverage a
reparametrization of ${\mathbf{x}}$ and ${\mathbf{z}}$ to accelerate the joint
MCMC sampling and simplify step size tuning, similar to [36, 42].
Specifically, ${\mathbf{x}}=\alpha
g_{\bm{\theta}}({\mathbf{z}})+\sigma{\bm{\epsilon}}$ defines a deterministic
transformation from the pair of $({\bm{\epsilon}},{\mathbf{z}})$ to the pair
of $({\mathbf{x}},{\mathbf{z}})$, which enables us to push back the joint
distribution $\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})$ to the
$({\bm{\epsilon}},{\mathbf{z}})$-space. The reparameterized distribution is
given by:

$\rho_{\bm{\theta}}({\bm{\epsilon}},{\mathbf{z}})=\frac{q(\alpha
g_{\bm{\theta}}({\mathbf{z}})+\sigma{\bm{\epsilon}})}{p_{\bm{\theta}}(\alpha
g_{\bm{\theta}}({\mathbf{z}})+\sigma{\bm{\epsilon}})}p({\bm{\epsilon}})p({\mathbf{z}}).$
(8)

See Appendix B for a detailed derivation. We found that this parameterization
admits the same step sizes across noise levels and results in better
performance empirically (Table 1).

(a) ${\mathbf{x}}$ w/ accumulated noise

(b) ${\mathbf{x}}$ w/o accumulated noise

Figure 2: Images after 8-step Langevin updates with and without accumulated
noise.

Still, learning the student with these samples continued to present
challenges. When visualizing samples ${\mathbf{x}}$ produced by MCMC (see Fig.
2(a)), we found that samples contained substantial noise. While this makes
sense given the level of noise in the marginal distributions, we found that
this inhibited learning of the student. We identify that, due to the structure
of Langevin dynamics, there is noise added to ${\mathbf{x}}$ at each step that
can be linearly accumulated across iterations. By removing this accumulated
noise along with the temporally decayed initial $\epsilon$, we recover cleaner
${\mathbf{x}}$ samples (Fig. 2(b)). Since ${\mathbf{x}}$ is effectively a
regression target in Eq. 6, and the expected value of the noises is
${\bm{0}}$, canceling these noises reduces variance of the gradient without
introducing bias. This noise cancellation was critical to the success of EMD,
and is detailed in Appendix B and ablated in experiments (Fig. 3ab).

###  3.3 Maximum Likelihood across all noise levels

The derivation above assumes smoothing the data distribution with a single
noise level. In practice, the diffusion teachers always employ multiple noise
levels $t$, coordinated by a noise schedule $p(t)$. Therefore, we optimize a
weighted loss over all noise levels of the diffusion model, to encourage that
the marginals of the student network match the marginals of the diffusion
process at all noise levels:

$\begin{split}\nabla_{\bm{\theta}}\mathcal{L}({\bm{\theta}})&=\nabla_{{\bm{\theta}}}\operatorname{\mathbb{E}}\nolimits_{p(t),q_{t}({\mathbf{x}}_{t})}\left[\tilde{w}(t)\log
p_{{\bm{\theta}},t}({\mathbf{x}}_{t})\right]=\operatorname{\mathbb{E}}\nolimits_{p(t),\rho_{t}({\mathbf{x}}_{t},{\mathbf{z}})}\left[\tilde{w}(t)\nabla_{{\bm{\theta}}}\log
p_{{\bm{\theta}},t}({\mathbf{x}}_{t},{\mathbf{z}})\right],\end{split}$ (9)

where
$p_{{\bm{\theta}},t}({\mathbf{x}}_{t},{\mathbf{z}})=p_{{\bm{\theta}},t}({\mathbf{x}}_{t}|{\mathbf{z}})p({\mathbf{z}})$,
$p_{{\bm{\theta}},t}({\mathbf{x}}_{t}|{\mathbf{z}})=\mathcal{N}(\alpha_{t}g_{\bm{\theta}}({\mathbf{z}}),\sigma_{t}^{2}{\bm{I}})$,
with a shared generator $g_{\bm{\theta}}({\mathbf{z}})$ across all noise
levels. Empirically, we find $\tilde{w}(t)=\sigma_{t}^{2}/\alpha_{t}$ or
$\tilde{w}(t)=\sigma_{t}^{2}/\alpha_{t}^{2}$ perform well.

Denote the resulted distribution after $K$ steps of MCMC sampling with noise
cancellation as $\rho_{t}^{K}({\mathbf{x}}_{t}^{K},{\mathbf{z}}^{K})$, the
final gradient for the generator network $g_{\bm{\theta}}$ is

$\begin{split}\nabla_{\bm{\theta}}\mathcal{L}({\bm{\theta}})=&\operatorname{\mathbb{E}}\nolimits_{p(t),\rho_{t}^{K}({\mathbf{x}}_{t}^{K},{\mathbf{z}}^{K})}\left[-\tilde{w}(t)\frac{\nabla_{\bm{\theta}}\lVert{{\mathbf{x}}_{t}^{K}}-\alpha_{t}g_{\bm{\theta}}({\mathbf{z}}^{K})\rVert_{2}^{2}}{2\sigma_{t}^{2}}\right].\end{split}$
(10)

The final gradient for the score network $s_{\bm{\phi}}({\mathbf{x}}_{t},t)$
is

$\nabla_{\bm{\phi}}\mathcal{J}({\bm{\phi}})=\operatorname{\mathbb{E}}\nolimits_{p(t),p_{{\bm{\theta}},t}({\mathbf{x}}_{t},{\mathbf{z}})}\left[{w}(t)\nabla_{\bm{\phi}}\lVert
s_{\bm{\phi}}({\mathbf{x}}_{t},t)-\nabla_{{\mathbf{x}}_{t}}\log
p_{t}({\mathbf{x}}_{t}|g_{\bm{\theta}}({\mathbf{z}}))\rVert_{2}^{2}\right].$
(11)

Similar to VSD [26, 22], we employ alternating update for the generator
network $g_{\bm{\theta}}$ and the score network
$s_{\bm{\phi}}({\mathbf{x}}_{t},t)$. See summarization in Algorithm 1.

Input: Teacher score functions $\nabla_{{\mathbf{x}}_{t}}\log
q_{t}({\mathbf{x}}_{t})$, generator network $g_{\bm{\theta}}$, prior
$p({\mathbf{z}})$, score network $s_{\bm{\phi}}$, noise scheduler $p(t)$,
weighting functions $w(t)$ and $\tilde{w}(t)$, # of MCMC steps $K$, MCMC step
size $\gamma$.

Output: Generator network $g_{\bm{\theta}}$, score network $s_{\bm{\phi}}$.

while not converged do

Sampling a batch of $t$, ${\mathbf{z}}$, ${\bm{\epsilon}}$ from $p(t)$,
$p({\mathbf{z}})$, $\mathcal{N}({\bm{0}},{\bm{I}})$ to obtain
${\mathbf{x}}_{t}$

Updating $s_{\bm{\phi}}$ via Stochastic Gradient Descent with the batch
estimate of Eq. 11

Sampling ${\mathbf{x}}_{t}^{K}$ and ${\mathbf{z}}^{K}$ with
$({\bm{\epsilon}},{\mathbf{z}})\text{-corrector}({\mathbf{x}}_{0},{\bm{\epsilon}},{\mathbf{z}},t,\nabla_{{\mathbf{x}}_{t}}\log
q_{t}({\mathbf{x}}_{t}),g_{\bm{\theta}},s_{\bm{\phi}},K,\gamma)$

Updating $g_{\bm{\theta}}$ via Stochastic Gradient Ascent with the batch
estimate of Eq. 10

end while

Algorithm 1 EM Distillation

Input: ${\mathbf{x}}_{0}$, ${\bm{\epsilon}}$, ${\mathbf{z}}$, $t$, teacher
score function $\nabla_{{\mathbf{x}}_{t}}\log q_{t}({\mathbf{x}}_{t})$,
generator network $g_{\bm{\theta}}$, prior $p_{0}({\mathbf{z}})$, score
network $s_{\bm{\phi}}$, # of MCMC steps $K$, MCMC step size $\gamma$.

Output: ${\mathbf{x}}_{t}^{K}$, ${\mathbf{z}}^{K}$.

Sampling Langevin noise
${\mathbf{n}}^{1},{\mathbf{n}}^{2},...,{\mathbf{n}}^{K}$ from
$\mathcal{N}({\bm{0}},{\bm{I}})$, letting
${\bm{\epsilon}}^{0}={\bm{\epsilon}},{\mathbf{z}}^{0}={\mathbf{z}}$

for $i$ in $[1,K]$ do

Updating $({\bm{\epsilon}}^{i},{\mathbf{z}}^{i})$ with 1-step Langevin update
over scores Eq. 16, with ${\bm{\epsilon}}^{i}$ updated using
${\mathbf{n}}^{i}$

end for

Pushing $({\bm{\epsilon}}^{K},{\mathbf{z}}^{K})$ forward to
$({\mathbf{x}}_{t}^{K},{\mathbf{z}}^{K})$ and then canceling the noises in
${\mathbf{x}}_{t}^{K}$

Algorithm 2 $({\bm{\epsilon}},{\mathbf{z}})\text{-corrector}$

###  3.4 Connection with VSD and Diff-Instruct

In this subsection, we reveal an interesting connection between EMD and
Variational Score Distillation (VSD) [26, 22], i.e., although motivated by
optimizing different types of divergences, VSD [26, 22] is equivalent to EMD
with a special sampling scheme.

To see this, consider the 1-step EMD with stepsize $\gamma=1$ in
${\mathbf{x}}$, no update on ${\mathbf{z}}$ and noise cancellation

${\mathbf{x}}_{t}^{0}=\alpha_{t}g_{\bm{\theta}}({\mathbf{z}})+\sigma_{t}\epsilon,\quad{\mathbf{x}}_{t}^{1}=\alpha_{t}g_{\bm{\theta}}({\mathbf{z}})+\sigma_{t}^{2}\nabla_{{\mathbf{x}}}\log\frac{q({\mathbf{x}}_{t}^{0})}{p_{{\bm{\theta}},t}({\mathbf{x}}_{t}^{0})}\cancel{+\sqrt{2}\sigma{\mathbf{n}}^{1}}.$
(12)

Substitute it into Eq. 10, we have

$\begin{split}\nabla_{\bm{\theta}}\mathcal{L}({\bm{\theta}})&=\operatorname{\mathbb{E}}\nolimits_{p(t),p(\epsilon),p({\mathbf{z}})}\left[-\tilde{w}(t)\frac{\nabla_{\bm{\theta}}\lVert{{\mathbf{x}}}_{t}^{1}-\alpha_{t}g_{\bm{\theta}}({\mathbf{z}})\rVert^{2}_{2}}{2\sigma_{t}^{2}}\right]\\\
&=\operatorname{\mathbb{E}}\nolimits_{p(t),p(\epsilon),p({\mathbf{z}})}\left[\tilde{w}(t){(\nabla_{{\mathbf{x}}_{t}}\log
q_{t}({\mathbf{x}}_{t})-\nabla_{{\mathbf{x}}_{t}}\log
p_{{\bm{\theta}},t}({\mathbf{x}}_{t}))\nabla_{\bm{\theta}}\alpha_{t}g_{\bm{\theta}}({\mathbf{z}})}\right],\end{split}$
(13)

which is exactly the gradient for VSD (Eq. 5), up to a sign difference. This
insight demonstrates that, EMD framework can flexibly interpolate between
mode-seeking and mode-covering divergences, by leveraging different sampling
schemes from 1-step sampling in only ${\mathbf{x}}$ (a likely biased sampler)
to many-step joint sampling in $({\mathbf{x}},{\mathbf{z}})$ (closer to a
mixing sampler).

If we further assume the marginal $p_{\bm{\theta}}({\mathbf{x}})$ is a
Gaussian, then EMD update in Eq. 13 would resemble Score Distillation Sampling
(SDS) [9].

##  4 Related Work

Diffusion acceleration. Diffusion models have the notable issue of slowness in
inference, that motivates many research efforts to accelerate the sampling
process. One line of work focuses on developing numerical solvers [43, 12, 44,
45, 46, 47] for the PF-ODE. Another line of work leverages the concept of
knowledge distillation [48] to condense the sampling trajectory of PF-ODE into
fewer steps [13, 49, 15, 14, 50, 51, 18, 52, 53, 54, 55]. However, both
approaches have significant limitations and have difficulty in substantially
reducing the sampling steps to the single-step regime without significant loss
in perceptual quality.

Single-step diffusion models. Recently, several methods for one-step diffusion
sampling have been proposed, sharing the same goal as our approach. Some
methods fine-tune the pre-trained diffusion model into a single-step generator
via adversarial training [20, 21, 56], where the adversarial loss enhances the
sharpness of the diffusion model’s single-step output. Adversarial training
can also be combined with trajectory distillation techniques to improve
performance in few or single-step regimes [52, 57, 58]. Score distillation
techniques [9, 26] have been adopted to match the distribution of the one-step
generator’s output with that of the teacher diffusion model, enabling single-
step generation [22, 41]. Additionally, [23] introduces a regression loss to
further enhance performance. These methods achieve more impressive 1-step
generation, some of which enjoy additional merits of being data-free or
flexible in the selection of generator architecture. However, they often
minimizes over mode-seeking divergences that can fail to capture the full
distribution and therefore causes mode collapse issues. We discuss the
connection between our method and this line of work in Section 3.4.

##  5 Experiments

We employ EMD to learn one-step image generators on ImageNet 64$\times$64,
ImageNet 128$\times$128 [59] and text-to-image generation. The student
generators are initialized with the teacher model weights. Results are
compared according to Frechet Inception Distance (FID) [60], Inception Score
(IS) [61], Recall (Rec.) [62] and CLIP Score [63]. Throughout this section, we
will refer to the proposed EMD with $K$ steps of Langevin updates on
$({\mathbf{x}},{\mathbf{z}})$ as EMD-$K$, and we use EMD-1 to describe the
DiffInstruct/VSD-equivalent formulation with only one update in ${\mathbf{x}}$
as presented in Section 3.4.

###  5.1 ImageNet

We start from showcasing the effect of the key components of EMD, namely noise
cancellation, multi-step joint sampling, and reparametrized sampling. We then
summarize results on ImageNet 64$\times$64 with Karras et al. [47] as teacher,
and ImageNet 128$\times$128 with Kingma and Gao [29] as teacher.

#### Noise cancellation

During our development, we observed the vital importance of canceling the
noise after the Langevin update. Even though theoretically speaking our noise
cancellation technique does not guarantee reducing the variance of the
gradients for learning, we find removing the accumulated noise term from the
samples (including the initial diffusion noise ${\bm{\epsilon}}$) does give us
seemingly clean images empirically. See Fig. 2 for a comparison. These updated
${\mathbf{x}}^{K}$ can be seen as regression targets in Eq. 10. Intuitively
speaking, regressing a generator towards clean images should result in more
stable training than towards noisy images. Reflected in the training process,
canceling the noise significantly decreases the variance in the gradient (Fig.
3a) and boosts the speed of convergence (Fig. 3b). We also compare with
another setting where only the noise in the last step gets canceled, which is
only marginally helpful.

Figure 3: (a)(b) Gradient norms and FIDs for complete noise cancellation,
last-step noise cancellation and no noise cancellation. (c)(d) FIDs and
Recalls of EMD with different numbers of Langevin steps.

#### Multi-step joint sampling

We scrutinize the effect of multi-step joint update on
$({\bm{\epsilon}},{\mathbf{z}})$. Empirically, we find a constant step size of
Langevin dynamics across all noise levels in the
$({\bm{\epsilon}},{\mathbf{z}})$-space works well:
${\bm{\gamma}}=(\gamma_{{\bm{\epsilon}}},\gamma_{{\mathbf{z}}})=(0.4^{2},0.004^{2})$,
which simplifies the process of step size tuning. Fig. 1 shows results of
running this $({\bm{\epsilon}},{\mathbf{z}})$-corrector for 300 steps. We can
see that the $({\bm{\epsilon}},{\mathbf{z}})$-corrector removes visual
artifacts and improves structure. Fig. 3cd illustrates the relation between
the distilled generator’s performance and the number of Langevin steps per
distillation iteration, measured by FID and Recall respectively. Both metrics
show clear improvement monotonically as the number of Langevin steps
increases. Recall is designed for measuring mode coverage [62], and has been
widely adopted in the GAN literature. A larger number of Langevin steps
encourages better mode coverage, likely because it approximates the mode-
covering forward KL better. Sampling ${\mathbf{z}}$ is more expensive than
sampling ${\bm{\epsilon}}$, requiring back-propagation through the generator
$g_{\bm{\theta}}$. An alternative is to only sample ${\bm{\epsilon}}$ while
keeping ${\mathbf{z}}$ fixed, with the hope that if ${\mathbf{x}}$ does not
change dramatically with a finite number of MCMC updates, the initial
${\mathbf{z}}$ remains a good approximation of samples from
$\rho_{\bm{\theta}}({\mathbf{z}}|{\mathbf{x}})$. As shown in Fig. 3cd,
sampling ${\bm{\epsilon}}$ performs similarly to the joint sampling of
$({\bm{\epsilon}},{\mathbf{z}})$ when the number of sampling steps is small,
but starts to fall behind with more sampling steps.

#### Reparametrized sampling

As shown in Appendix B, the noise cancellation technique does not depend on
the reparametrization. One can start from either the score functions of
$({\mathbf{x}},{\mathbf{z}})$ in Eq. 7 or the score functions of
$({\bm{\epsilon}},{\mathbf{z}})$ in Eq. 16 to derive something similar. We
conduct a comparison between the two parameterizations for joint sampling,
$({\mathbf{x}},{\mathbf{z}})$-corrector and
$({\bm{\epsilon}},{\mathbf{z}})$-corrector.

Table 1: EMD-8 on ImageNet 64$\times$64, 100k steps of training

| FID ($\downarrow$) | IS ($\uparrow$)
---|---|---
$({\mathbf{x}},)/({\bm{\epsilon}},)$ | 2.829 | 62.31
$({\mathbf{x}},{\mathbf{z}})$ | 3.11 | 61.08
$({\bm{\epsilon}},{\mathbf{z}})$ | 2.77 | 62.98

For the $({\mathbf{x}},{\mathbf{z}})$-corrector, we set the step size of
${\mathbf{x}}$ as $\sigma_{t}^{2}\gamma_{\bm{\epsilon}}$ to align the
magnitude of update with the one of the
$({\bm{\epsilon}},{\mathbf{z}})$-corrector, and keep the step size of
${\mathbf{z}}$ the same (see Appendix B for details). This also promotes
numerical stability in $({\mathbf{x}},{\mathbf{z}})$-corrector by canceling
the $\sigma_{t}^{2}$ in the denominator of the term
${\nabla_{{\mathbf{x}}}\log
p_{\bm{\theta}}({\mathbf{x}}|{\mathbf{z}})}={-\frac{{\mathbf{x}}-\alpha
g_{\bm{\theta}}({\mathbf{z}})}{\sigma^{2}}}$ in the score function (Eq. 7). A
similar design choice was proposed in Song and Ermon [27]. Also note that
adjusting the step sizes in this way results in an equivalence between
$({\bm{\epsilon}},)$-corrector and $({\mathbf{x}},)$-corrector without
sampling in ${\mathbf{z}}$, which serves as the baseline for the joint
sampling.

Table 1 reports the quantitative comparisons with EMD-8 on ImageNet
64$\times$64 after 100k steps of training. While joint sampling with
$({\bm{\epsilon}},{\mathbf{z}})$-corrector improves over
$({\bm{\epsilon}},)$-corrector, $({\mathbf{x}},{\mathbf{z}})$-corrector
struggles to even match the baseline. Possible explanations include that the
space of $({\bm{\epsilon}},{\mathbf{z}})$ is more MCMC friendly, or it
requires more effort on searching for the optimal step size of ${\mathbf{z}}$
for the $({\mathbf{x}},{\mathbf{z}})$-corrector. We leave further
investigation to future work.

#### Comparsion with existing methods

We report the results from our full-fledged method, EMD-16, which utilizes a
$({\bm{\epsilon}},{\mathbf{z}})$-corrector with 16 steps of Langevin updates,
and compare with existing approaches. We train for 300k steps on ImageNet
64$\times$64, and 200k steps on ImageNet 128$\times$128\. Other
hyperparameters can be found in Appendix C. Samples from the distilled
generator can be found in Fig. 4. We also include additional samples in
Section D.1. We summarize the comparison with existing methods for few-step
diffusion generation in LABEL:tab:i64_all and LABEL:tab:i128_all for ImageNet
64$\times$64 and ImageNet 128$\times$128 respectively. Note that we also tune
the baseline EMD-1, which in formulation is equivalent to Diff-Instruct [22],
to perform better than their reported numbers. The improvement mainly comes
from a fine-grained tuning of learning rates and enabling dropout for both the
teacher and student score functions. Our final models outperform existing
approaches for one-step distillation of diffusion models in terms of FID
scores on both tasks. On ImageNet $64\times 64$, EMD achieves a competitive
recall among distribution matching approaches but falls behind trajectory
distillation approaches which maintain individual trajectory mappings from the
teacher.

Table 2: Class-conditional genreation on ImageNet 64$\times$64.

Method |  NFE ($\downarrow$) |  FID ($\downarrow$) |  Rec. ($\uparrow$)
---|---|---|---
Multiple Steps |
DDIM [12] | 50 | 13.7 |
EDM-Heun [47] | 10 | 17.25 |
DPM Solver [44] | 10 | 7.93 |
PD [13] | 2 | 8.95 | 0.65
CD [15] | 2 | 4.70 | 0.64
Multistep CD [18] | 2 | 2.0 | -
Single Step |
BigGAN-deep [64] | 1 | 4.06 | 0.48
EDM [47] | 1 | 154.78 | -
PD [13] | 1 | 15.39 | 0.62
BOOT [16] | 1 | 16.30 | 0.36
DFNO [17] | 1 | 7.83 | -
TRACT [14] | 1 | 7.43 | -
CD-LPIPS [15] | 1 | 6.20 | 0.63
Diff-Instruct [22] | 1 | 5.57 | -
DMD [23] | 1 | 2.62 | -
EMD-1 (baseline) | 1 | 3.1 | 0.55
EMD-16 (ours) | 1 | 2.20 | 0.59
Teacher | 256 | 1.43 | -

Table 3: Class-conditional generation on ImageNet 128$\times$128.

Method |  NFE ($\downarrow$) |  FID ($\downarrow$) |  IS ($\uparrow$)
---|---|---|---
Multiple Steps |
Multistep CD [18] | 8 | 2.1 | 164
Multistep CD [18] | 4 | 2.3 | 157
Multistep CD [18] | 2 | 3.1 | 147
Single Step |
CD [15] | 1 | 7.0 | -
EMD-1 (baseline) | 1 | 6.3 |  134 $\pm$ 2.75
EMD-16 (ours) | 1 | 6.0 |  140 $\pm$ 2.83
Teacher | 512 | 1.75 |  171.1 $\pm$ 2.7

Table 4: FID-30k for text-to-image generation in MSCOCO. † Results are
evaluated by Yin et al. [23].

Family | Method |  Latency ($\downarrow$) |  FID ($\downarrow$)
---|---|---|---
Unaccelerated |  DALL·E [65] | - | 27.5
DALL·E 2 [4] | - | 10.39
Parti-3B [66] | 6.4s | 8.10
Make-A-Scene [67] | 25.0s | 11.84
GLIDE [68] | 15.0s | 12.24
Imagen [5] | 9.1s | 7.27
eDiff-I [69] | 32.0s | 6.95
GANs |  StyleGAN-T [70] | 0.10s | 13.90
GigaGAN [71] | 0.13s | 9.09
Accelerated |  DPM++ (4 step)† [45] | 0.26s | 22.36
UniPC (4 step)† [72] | 0.26s | 19.57
LCM-LoRA (1 step)† [73] | 0.09s | 77.90
LCM-LoRA (4 step)† [73] | 0.19s | 23.62
InstaFlow-0.9B† [55] | 0.09s | 13.10
UFOGen [20] | 0.09s | 12.78
|  DMD (tCFG=3)† [23] | 0.09s | 11.49
| EMD-1 (baseline, tCFG=3) | 0.09s | 10.96
| EMD-1 (baseline, tCFG=2) | 0.09s | 9.78
| EMD-8 (ours, tCFG=2) | 0.09s | 9.66
Teacher |  SDv1.5† [6] | 2.59s | 8.78

Table 5: CLIP Score in high CFG regime.

Family | Method |  Latency ($\downarrow$) |  CLIP ($\uparrow$)
---|---|---|---
Accelerated |  DPM++ (4 step) [45]† | 0.26s | 0.309
UniPC (4 step)† [72] | 0.26s | 0.308
LCM-LoRA (1 step)† [73] | 0.09s | 0.238
LCM-LoRA (4 step)† [73] | 0.19s | 0.297
DMD† [23] | 0.09s | 0.320
EMD-8 (ours) | 0.09s | 0.316
Teacher |  SDv1.5 † [6] | 2.59s | 0.322

(a) ImageNet 64$\times$64 multi-class

(b) ImageNet 128$\times$128 multi-class

(c) ImageNet 128$\times$128 single-class

Figure 4: ImageNet samples from the distilled 1-step generator. Models are
trained class-conditionally with all classes. We provide single-class samples
in (c) to demonstrate good mode coverage.

###  5.2 Text-to-image generation

Figure 5: Text-to-image samples from the 1-step student model distilled from
Stable Diffusion v1.5.

We further test the potential of EMD on text-to-image models at scale by
distilling the Stable Diffusion v1.5 [6] model. Note that the training is
image-free and we only use text prompts from the LAION-Aesthetics-6.25+
dataset [74]. On this task, DMD [23] is a strong baseline, which introduced an
additional regression loss to VSD or Diff-Instruct to avoid mode collapse.
However, we find the baseline without regression loss, or equivalently EMD-1,
can be improved by simply tuning the hyperparameter $t^{*}$. Empirically, we
find it is better to set $t^{*}$ to intermediate noise levels, consistent with
the observation from Luo et al. [22]. Other hyperparameters can be found in
Appendix C.

We evaluate the distilled one-step generator for text-to-image generation with
zero-shot generalization on MSCOCO [75] and report the FID-30k in
LABEL:tab:t2i and CLIP Score in Table 5. Yin et al. [23] uses the guidance
scale of 3.0 to compose the classifer-free guided teacher score (we refer to
this guidance scale of teacher as tCFG) in the learning gradient of DMD, for
it achieves the best FID for DDIM sampler. However, we find EMD achieves a
lower FID at the tCFG of 2.0. Our method, EMD-8, trained on 256 TPU-v5e for 5
hours (5000 steps), achieves the FID=9.66 for one-step text-to-image
generation. Using a higher tCFG, similar to DMD, produces a model with
competitive CLIP Score. In Fig. 5, we include some samples for qualitative
evaluation. Additional qualitative results (Tables 13 and 14), as well as
side-by-side comparisons (Tables 9, 10, 11 and 12) with trajectory-based
distillation baselines [55, 73] and adversarial distillation baselines [21]
can be found in Section D.2.

##  6 Discussion and limitation

We present EMD, a maximum likelihood-based method that leverages EM framework
with novel sampling and optimization techniques to learn a one-step student
model whose marginal distributions match the marginals of a pretrained
diffusion model. EMD demonstrates strong performance in class-conditional
generation on ImageNet and text-to-image generation. Despite exhibiting
compelling results, EMD has a few limitations that call for future work.
Empirically, we find that EMD still requires the student model to be
initialized from the teacher model to perform competitively, and is sensitive
to the choice of $t^{*}$ (fixed timestep conditioning that repurposes the
diffusion denoiser to become a one-step genertor) at initialization. While
training a student model entirely from scratch is supported theoretically by
our framework, empirically we were unable to achieve competitive results.
Improving methods to enable generation from randomly initialized generator
networks with distinct architectures and lower-dimensional latent variables is
an exciting direction of future work. Although being efficient in inference,
EMD introduces additional computational cost in training by running multiple
sampling steps per iteration, and the step size of MCMC sampling can require
careful tuning. There remains a fundamental trade-off between training cost
and model performance. Analysis and further improving on the Pareto frontier
of this trade-off would be interesting for future work.

## Acknowledgement

We thank Jonathan Heek and Lucas Theis for their valuable discussion and
feedback. Sirui would like to thank Tao Zhu, Jiahui Yu, and Zhishuai Zhang for
their support in a prior internship at Google DeepMind.

## References

  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pages 2256–2265. PMLR, 2015.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Song et al. [2020a] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2020a.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in neural information processing systems_ , 35:36479–36494, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ho et al. [2022] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022.
  * Brooks et al. [2024] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.  Video generation models as world simulators.  2024\.  URL https://openai.com/research/video-generation-models-as-world-simulators.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Xu et al. [2022] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang.  Geodiff: A geometric diffusion model for molecular conformation generation.  _arXiv preprint arXiv:2203.02923_ , 2022.
  * Yang et al. [2023] Mengjiao Yang, KwangHwan Cho, Amil Merchant, Pieter Abbeel, Dale Schuurmans, Igor Mordatch, and Ekin Dogus Cubuk.  Scalable diffusion for materials generation.  _arXiv preprint arXiv:2311.09235_ , 2023.
  * Song et al. [2020b] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020b.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Berthelot et al. [2023] David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu.  Tract: Denoising diffusion models with transitive closure time-distillation.  _arXiv preprint arXiv:2303.04248_ , 2023.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  In _International Conference on Machine Learning_ , pages 32211–32252. PMLR, 2023.
  * Gu et al. [2023] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind.  Boot: Data-free distillation of denoising diffusion models with bootstrapping.  In _ICML 2023 Workshop on Structured Probabilistic Inference $\\{$$\backslash$&$\\}$ Generative Modeling_, 2023.
  * Zheng et al. [2023] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar.  Fast sampling of diffusion models via operator learning.  In _International Conference on Machine Learning_ , pages 42390–42402. PMLR, 2023.
  * Heek et al. [2024] Jonathan Heek, Emiel Hoogeboom, and Tim Salimans.  Multistep consistency models.  _arXiv preprint arXiv:2403.06807_ , 2024.
  * Xiao et al. [2021] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.  Tackling the generative learning trilemma with denoising diffusion gans.  In _International Conference on Learning Representations_ , 2021.
  * Xu et al. [2023] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou.  Ufogen: You forward once large scale text-to-image generation via diffusion gans.  _arXiv preprint arXiv:2311.09257_ , 2023.
  * Sauer et al. [2023] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial diffusion distillation.  _arXiv preprint arXiv:2311.17042_ , 2023.
  * Luo et al. [2023a] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang.  Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2023a.
  * Yin et al. [2024] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T Freeman, and Taesung Park.  One-step diffusion with distribution matching distillation.  In _CVPR_ , 2024.
  * Dempster et al. [1977] Arthur P Dempster, Nan M Laird, and Donald B Rubin.  Maximum likelihood from incomplete data via the em algorithm.  _Journal of the royal statistical society: series B (methodological)_ , 39(1):1–22, 1977.
  * Nijkamp et al. [2019] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu.  Learning non-convergent non-persistent short-run mcmc toward energy-based model.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Wang et al. [2024] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Kingma et al. [2021] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  _Advances in neural information processing systems_ , 34:21696–21707, 2021.
  * Kingma and Gao [2023] Diederik Kingma and Ruiqi Gao.  Understanding diffusion objectives as the elbo with simple data augmentation.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Hyvärinen and Dayan [2005] Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Vincent [2011] Pascal Vincent.  A connection between score matching and denoising autoencoders.  _Neural computation_ , 23(7):1661–1674, 2011.
  * Neal et al. [2011] Radford M Neal et al.  Mcmc using hamiltonian dynamics.  _Handbook of markov chain monte carlo_ , 2(11):2, 2011.
  * Xie et al. [2016] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu.  A theory of generative convnet.  In _International Conference on Machine Learning_ , pages 2635–2644. PMLR, 2016.
  * Xie et al. [2018] Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu.  Cooperative learning of energy-based model and latent variable model via mcmc teaching.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Gao et al. [2018] Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu.  Learning generative convnets via multi-grid modeling and sampling.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9155–9164, 2018.
  * Nijkamp et al. [2021] Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang, Song-Chun Zhu, and Ying Nian Wu.  Mcmc should mix: Learning energy-based model with neural transport latent space mcmc.  In _International Conference on Learning Representations_ , 2021.
  * Nijkamp et al. [2020] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu.  On the anatomy of mcmc-based maximum likelihood learning of energy-based models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 5272–5280, 2020.
  * Gao et al. [2020] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma.  Learning energy-based models by diffusion recovery likelihood.  _arXiv preprint arXiv:2012.08125_ , 2020.
  * Bishop [2006] Christopher M Bishop.  Pattern recognition and machine learning.  _Springer google schola_ , 2:1122–1128, 2006.
  * Han et al. [2017] Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu.  Alternating back-propagation for generator network.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31, 2017.
  * Nguyen and Tran [2023] Thuan Hoang Nguyen and Anh Tran.  Swiftbrush: One-step text-to-image diffusion model with variational score distillation.  _arXiv preprint arXiv:2312.05239_ , 2023.
  * Xiao et al. [2020] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat.  Vaebm: A symbiosis between variational autoencoders and energy-based models.  In _International Conference on Learning Representations_ , 2020.
  * Luhman and Luhman [2021] Eric Luhman and Troy Luhman.  Knowledge distillation in iterative generative models for improved sampling speed.  _arXiv preprint arXiv:2101.02388_ , 2021.
  * Lu et al. [2022a] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022a.
  * Lu et al. [2022b] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022b.
  * Bao et al. [2022] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang.  Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models.  _arXiv preprint arXiv:2201.06503_ , 2022.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Meng et al. [2023] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14297–14306, 2023.
  * Li et al. [2023] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren.  Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.  _arXiv preprint arXiv:2306.00980_ , 2023.
  * Luo et al. [2023b] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  _arXiv preprint arXiv:2310.04378_ , 2023b.
  * Kim et al. [2023] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon.  Consistency trajectory models: Learning probability flow ode trajectory of diffusion.  _arXiv preprint arXiv:2310.02279_ , 2023.
  * Zheng et al. [2024] Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham.  Trajectory consistency distillation.  _arXiv preprint arXiv:2402.19159_ , 2024.
  * Yan et al. [2024] Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng.  Perflow: Piecewise rectified flow as universal plug-and-play accelerator.  _arXiv preprint arXiv:2405.07510_ , 2024.
  * Liu et al. [2023] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al.  Instaflow: One step is enough for high-quality diffusion-based text-to-image generation.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Sauer et al. [2024] Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach.  Fast high-resolution image synthesis with latent adversarial diffusion distillation.  _arXiv preprint arXiv:2403.12015_ , 2024.
  * Lin et al. [2024] Shanchuan Lin, Anran Wang, and Xiao Yang.  Sdxl-lightning: Progressive adversarial diffusion distillation.  _arXiv preprint arXiv:2402.13929_ , 2024.
  * Ren et al. [2024] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao.  Hyper-sd: Trajectory segmented consistency model for efficient image synthesis.  _arXiv preprint arXiv:2404.13686_ , 2024.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  _Advances in neural information processing systems_ , 29, 2016.
  * Kynkäänniemi et al. [2019] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models.  _Advances in neural information processing systems_ , 32, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International conference on machine learning_ , pages 8821–8831. Pmlr, 2021.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2(3):5, 2022.
  * Gafni et al. [2022] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _European Conference on Computer Vision_ , pages 89–106. Springer, 2022.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Balaji et al. [2022] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al.  ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.  _arXiv preprint arXiv:2211.01324_ , 2022.
  * Karras et al. [2018] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks. arxiv e-prints.  _arXiv preprint arXiv:1812.04948_ , 2018.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10124–10134, 2023.
  * Zhao et al. [2024] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu.  Unipc: A unified predictor-corrector framework for fast sampling of diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Luo et al. [2023c] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao.  Lcm-lora: A universal stable-diffusion acceleration module.  _arXiv preprint arXiv:2311.05556_ , 2023c.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Hoogeboom et al. [2023] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans.  simple diffusion: End-to-end diffusion for high resolution images.  In _International Conference on Machine Learning_ , pages 13213–13232. PMLR, 2023.

##  Appendix A Expectation-Minimization

To learn a latent variable model
$p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})=p_{\bm{\theta}}({\mathbf{x}}|{\mathbf{z}})p_{\bm{\theta}}({\mathbf{z}})$,
$p_{\bm{\theta}}({\mathbf{x}})=\int{p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})}d{\mathbf{z}}$
from a target distribution $q({\mathbf{x}})$, the EM-like transformation on
the gradient of the log-likelihood function is:

$\begin{split}\nabla_{{\bm{\theta}}}\mathcal{L}({\bm{\theta}})&=\nabla_{{\bm{\theta}}}\operatorname{\mathbb{E}}\nolimits_{q({\mathbf{x}})}[\log
p_{{\bm{\theta}}}({\mathbf{x}})]\\\
&=\operatorname{\mathbb{E}}\nolimits_{p_{{\bm{\theta}}}({\mathbf{z}}|{\mathbf{x}})}[\operatorname{\mathbb{E}}\nolimits_{q({\mathbf{x}})}[\nabla_{{\bm{\theta}}}\log
p_{{\bm{\theta}}}({\mathbf{x}})]]\\\
&=\operatorname{\mathbb{E}}\nolimits_{q({\mathbf{x}})p_{{\bm{\theta}}}({\mathbf{z}}|{\mathbf{x}})}[\nabla_{{\bm{\theta}}}\log
p_{{\bm{\theta}}}({\mathbf{x}})+\nabla_{{\bm{\theta}}}\log
p_{{\bm{\theta}}}({\mathbf{z}}|{\mathbf{x}})]\\\
&=\operatorname{\mathbb{E}}\nolimits_{q({\mathbf{x}})p_{{\bm{\theta}}}({\mathbf{z}}|{\mathbf{x}})}[\nabla_{{\bm{\theta}}}\log
p_{{\bm{\theta}}}({\mathbf{x}},{\mathbf{z}})].\end{split}$ (14)

Line 3 is due to the simple equality that
$\operatorname{\mathbb{E}}\nolimits_{p_{{\bm{\theta}}}({\mathbf{z}}|{\mathbf{x}})}[\nabla_{{\bm{\theta}}}\log
p_{{\bm{\theta}}}({\mathbf{z}}|{\mathbf{x}})]=0$.

##  Appendix B Reparametrized sampling and noise cancellation

#### Reparametrization.

The EM-like algorithm we propose requires joint sampling of
$({\mathbf{x}},{\mathbf{z}})$ from
$\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})$. Similar to [36, 42], we
utilize a reparameterization of ${\mathbf{x}}$ and ${\mathbf{z}}$ to overcome
challenges in joint MCMC sampling, such as slow convergence and complex step
size tuning. Notice that ${\mathbf{x}}=\alpha
g_{\bm{\theta}}({\mathbf{z}})+\sigma{\bm{\epsilon}}$ defines a deterministc
mapping from $({\bm{\epsilon}},{\mathbf{z}})$ to
$({\mathbf{x}},{\mathbf{z}})$. Then by _change of variable_ we have:

$\begin{split}\rho_{\bm{\theta}}({\bm{\epsilon}},{\mathbf{z}})d{\bm{\epsilon}}d{\mathbf{z}}&=\rho_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})d{\mathbf{x}}d{\mathbf{z}}\\\
&=\frac{q({\mathbf{x}})}{p_{\bm{\theta}}({\mathbf{x}})}p_{\bm{\theta}}({\mathbf{x}},{\mathbf{z}})d{\mathbf{x}}d{\mathbf{z}}\\\
&=\frac{q(\alpha
g_{\bm{\theta}}({\mathbf{z}})+\sigma{\bm{\epsilon}})}{p_{\bm{\theta}}(\alpha
g_{\bm{\theta}}({\mathbf{z}})+\sigma{\bm{\epsilon}})}p_{\bm{\theta}}({\bm{\epsilon}},{\mathbf{z}})d{\bm{\epsilon}}d{\mathbf{z}}\\\
\Rightarrow\rho_{\bm{\theta}}({\bm{\epsilon}},{\mathbf{z}})&=\frac{q(\alpha
g_{\bm{\theta}}({\mathbf{z}})+\sigma{\bm{\epsilon}})}{p_{\bm{\theta}}(\alpha
g_{\bm{\theta}}({\mathbf{z}})+\sigma{\bm{\epsilon}})}p({\bm{\epsilon}})p({\mathbf{z}}),\end{split}$
(15)

where $p({\bm{\epsilon}})$ and $p({\mathbf{z}})$ are standard Normal
distributions.

The score functions become

$\begin{split}\nabla_{{\bm{\epsilon}}}\log\rho({\bm{\epsilon}},{\mathbf{z}})&=\sigma(\nabla_{{\mathbf{x}}}\log
q({\mathbf{x}})-\nabla_{{\mathbf{x}}}\log
p_{\bm{\theta}}({\mathbf{x}}))-{\bm{\epsilon}},\\\
\nabla_{{\mathbf{z}}}\log\rho({\bm{\epsilon}},{\mathbf{z}})&=\alpha(\nabla_{{\mathbf{x}}}\log
q({\mathbf{x}})-\nabla_{{\mathbf{x}}}\log
p_{\bm{\theta}}({\mathbf{x}}))\nabla_{\mathbf{z}}g_{\bm{\theta}}({\mathbf{z}})-{\mathbf{z}}.\end{split}$
(16)

#### Noise cancellation.

The single-step Langevin update on ${\bm{\epsilon}}$ is then:

$\begin{split}{\bm{\epsilon}}^{i+1}=(1-\gamma){\bm{\epsilon}}^{i}+\gamma\sigma\nabla_{{\mathbf{x}}}\log\frac{q({\mathbf{x}}^{i})}{p_{\bm{\theta}}({\mathbf{x}}^{i})}+\sqrt{2\gamma}{\mathbf{n}}^{i}.\end{split}$
(17)

Interestingly, we find the particular form of
$\nabla_{{\bm{\epsilon}}}\log\rho({\bm{\epsilon}},{\mathbf{z}})$ results in a
closed-form accumulation of multi-step updates

$\begin{split}{\bm{\epsilon}}^{i+1}={(1-\gamma)^{i+1}{\bm{\epsilon}}^{0}}+\gamma\sum\nolimits_{k=0}^{i}(1-\gamma)^{i-k}\sigma\nabla_{{\mathbf{x}}}\log\frac{q({\mathbf{x}}^{i})}{p_{\bm{\theta}}({\mathbf{x}}^{i})}+{\sum\nolimits_{k=0}^{i}(1-\gamma)^{i-k}\sqrt{2\gamma}{\mathbf{n}}^{k}},\end{split}$
(18)

which, after the push-forward, gives us

$\begin{split}{\mathbf{x}}^{i+1}=&\alpha
g({\mathbf{z}}^{i+1})+\underbrace{\gamma\sum\nolimits_{k=0}^{i}(1-\gamma)^{i-k}\sigma^{2}\nabla_{{\mathbf{x}}}\log\frac{q({\mathbf{x}}^{i})}{p_{\bm{\theta}}({\mathbf{x}}^{i})}}_{drift}\\\
&+\underbrace{(1-\gamma)^{i+1}\sigma{\bm{\epsilon}}^{0}+\sum\nolimits_{k=0}^{i}(1-\gamma)^{i-k}\sqrt{2\gamma}\sigma{\mathbf{n}}^{k}}_{noise},\end{split}$
(19)

As ${\mathbf{x}}^{i+1}$ is effectively a regression target in Eq. 6, and the
expected value of the $noise$ is ${\bm{0}}$, we can remove it without biasing
the gradient. Empirically, we find book-keeping the sampled noises in the MCMC
chain and canceling these noises after the loop significantly stabilize the
training of the generator network.

The same applies to the $({\mathbf{x}},{\mathbf{z}})$ sampling:

$\begin{split}{\mathbf{x}}^{i+1}=&{\mathbf{x}}^{i}+\gamma\sigma^{2}(\nabla_{\mathbf{x}}\log
q({\mathbf{x}}^{i})-\nabla_{\mathbf{x}}\log
p_{\bm{\theta}}({\mathbf{x}}^{i}))-\gamma({\mathbf{x}}^{i}-\alpha
g({\mathbf{z}}^{i}))+\sqrt{2\gamma}n^{i}\\\
=&\gamma\sum\nolimits_{k=1}^{i}(1-\gamma)^{i-k}\alpha
g({\mathbf{z}}^{k})+\gamma\sum\nolimits_{k=0}^{i}(1-\gamma)^{i-k}\sigma^{2}(\nabla_{\mathbf{x}}\log
q({\mathbf{x}}^{i})-\nabla_{\mathbf{x}}\log
p_{\bm{\theta}}({\mathbf{x}}^{i}))\\\ &+(1-\gamma)^{i}\alpha
g({\mathbf{z}}^{0})+\underbrace{(1-\gamma)^{i+1}\sigma{\bm{\epsilon}}^{0}+\sum\nolimits_{k=0}^{i}(1-\gamma)^{i-k}\sqrt{2\gamma}\sigma
n^{k}}_{noises}.\end{split}$ (20)

##  Appendix C Implementation details

###  C.1 ImageNet 64$\times$64

We train the teacher model using the best setting of EDM [47] with the ADM
UNet architecture [76]. We inherit the noise schedule and the score matching
weighting function from the teacher. We run the distillation training for 300k
steps (roughly 8 days) on 64 TPU-v4. We use
$({\bm{\epsilon}},{\mathbf{z}})$-corrector, in which both the teacher and the
student score networks have a dropout probability of 0.1. We list other
hyperparameters in Table 6. Instead of listing $t^{*}$, we list the
corresponding log signal-to-noise ratio $\lambda^{*}$.

Table 6: Hyperparameters for EMD on ImageNet 64$\times$64. $lr_{g}$ | $lr_{s}$ | batch size | Adam $b_{1}$ | Adam $b_{2}$ | $\gamma_{\bm{\epsilon}}$ | $\gamma_{\mathbf{z}}$ | $K$ | $\lambda^{*}$ | $\tilde{w}(t)$
---|---|---|---|---|---|---|---|---|---
$2\times 10^{-6}$ | $1\times 10^{-5}$ | 128 | 0.0 | 0.99 | $0.4^{2}$ | $0.004^{2}$ | $16$ | $3.2189$ | $\frac{\sigma_{t}^{2}}{\alpha_{t}^{2}}$

###  C.2 ImageNet 128$\times$128

We train the teacher model following the best setting of VDM++ [29] with the
‘U-ViT, L’ architecture [77]. We use the ‘cosine-adjusted’ noise schedule [77]
and ‘EDM monotonic’ weighting for student score matching. We run the
distillation training for 200k steps (roughly 10 days) on 128 TPU-v5p. We use
$({\bm{\epsilon}},{\mathbf{z}})$-corrector, in which both the teacher and the
student score networks have a dropout probability of 0.1. We list other
hyperparameters in Table 7.

Table 7: Hyperparameters for EMD on ImageNet 128$\times$128. $lr_{g}$ | $lr_{s}$ | batch size | Adam $b_{1}$ | Adam $b_{2}$ | $\gamma_{\bm{\epsilon}}$ | $\gamma_{\mathbf{z}}$ | $K$ | $\lambda^{*}$ | $\tilde{w}(t)$
---|---|---|---|---|---|---|---|---|---
$2\times 10^{-6}$ | $1\times 10^{-5}$ | 1024 | 0.0 | 0.99 | $0.4^{2}$ | $0.004^{2}$ | $16$ | $6$ | $\frac{\sigma_{t}^{2}}{\alpha_{t}}$

###  C.3 Text-to-image generation

We adopt the public checkpoint of Stable Diffusion v1.5 [6] as the teacher. We
inherit the noise schedule from the teacher model. The student score matching
uses the same weighting function as the teacher. We list other hyperparameters
in Table 8.

Table 8: Hyperparameters for EMD on Text-to-image generation. $lr_{g}$ | $lr_{s}$ | batch size | Adam $b_{1}$ | Adam $b_{2}$ | $\gamma_{\bm{\epsilon}}$ | $\gamma_{\mathbf{z}}$ | $K$ | $t^{*}$ | $\tilde{w}(t)$
---|---|---|---|---|---|---|---|---|---
$2\times 10^{-6}$ | $1\times 10^{-5}$ | 1024 | 0.0 | 0.99 | $0.3^{2}$ | $0.005^{2}$ | $8$ | $500$ | $\frac{\sigma_{t}^{2}}{\alpha_{t}}$

##  Appendix D Additional qualitative results

###  D.1 Additional ImageNet results

In this section, we present additional qualitative samples for our one-step
generator on ImageNet 64$\times$64 and ImageNet 128$\times$128 in Fig. 6 to
help further evaluate the generation quality and diversity.

(a) ImageNet 64$\times$64 Multi-class

(b) ImageNet 128$\times$128 Multi-class

(c) ImageNet 128$\times$128 Single-class (Left: Husky, right: Siamese)

Figure 6: Additional qualitative results for ImageNet

###  D.2 Additional text-to-image results

In this section, we present additional qualitative samples from our one-step
generator distilled from Stable Diffusion 1.5. In Table 9, 10, 11, and 12, we
visually compare the sample quality of our method with open-source competing
methods for few- or single-step generation. We also include the teacher model
in our comparison. We use the public checkpoints of
LCM111https://huggingface.co/latent-consistency/lcm-lora-sdv1-5 and
InstaFlow222https://huggingface.co/XCLiu/instaflow_0_9B_from_sd_1_5, where
both checkpoints share the same Stable Diffusion 1.5 as teachers. Note that
the SD-turbo results are obtaind from the public checkpoint
333https://huggingface.co/stabilityai/sd-turbo fine-tuned from Stable
Diffusion 2.1, which is different from our teacher model.

From the comparison, we observe that our model significantly outperforms
distillation-based methods including LCM and InstaFlow, and it demonstrates
better diversity and quality than GAN-based SD-turbo. The visual quality is
on-par with 50-step generation from the teacher model.

We show additional samples from our model on a more diverse set of prompts in
Table 13 and 14.

|
---|---
Teacher (50 step) | EMD (1 step)
|
LCM (1 step) | LCM (2 steps)
|
SD-Turbo (1 step) | InstaFlow (1 step)

Table 9: Prompt: Dog graduation at university.

|
---|---
Teacher (50 step) | EMD (1 step)
|
LCM (1 step) | LCM (2 steps)
|
SD-Turbo (1 step) | InstaFlow (1 step)

Table 10: Prompt: 3D animation cinematic style young caveman kid, in its
natural environment.

|
---|---
Teacher (50 step) | EMD (1 step)
|
LCM (1 step) | LCM (2 steps)
|
SD-Turbo (1 step) | InstaFlow (1 step)

Table 11: Prompt: An underwater photo portrait of a beautiful fluffy white
cat, hair floating. In a dynamic swimming pose. The sun rays filters through
the water. High-angle shot. Shot on Fujifilm X.

|
---|---
Teacher (50 step) | EMD (1 step)
|
LCM (1 step) | LCM (2 steps)
|
SD-Turbo (1 step) | InstaFlow (1 step)

Table 12: Prompt: A minimalist Teddy bear in front of a wall of red roses.

|  |  |
---|---|---|---
A close-up photo of a intricate beautiful natural landscape of mountains and
waterfalls.
|  |  |
A hyperrealistic photo of a fox astronaut; perfect face, artstation.
|  |  |
Large plate of delicious fried chicken, with a side of dipping sauce,
realistic advertising photo, 4k.
|  |  |
A DSLR photo of a golden retriever in heavy snow.
|  |  |
Masterpiece color pencil drawing of a horse, bright vivid color.
|  |  |
Oil painting of a wise old man with a white beard in the enchanted and magical
forest.
Table 13: Additional qualitative results of EMD. Zoom-in for better viewing. |  |  |
---|---|---|---
3D render baby parrot, Chibi, adorable big eyes. In a garden with butterflies,
greenery, lush
whimsical and soft, magical, octane render, fairy dust.
|  |  |
Dreamy puppy surrounded by floating bubbles.
|  |  |
A painting of an adorable rabbit sitting on a colorful splash.
|  |  |
Macro photo of a miniature toy sloth drinking a soda, shot on a light pastel
cyclorama.
|  |  |
A traditional tea house in a tranquil garden with blooming cherry blossom
trees.
|  |  |
Three cats having dinner at a table at new years eve, cinematic shot, 8k.
Table 14: Additional qualitative results of EMD. Zoom-in for better viewing.

  *[EMD]: EM Distillation
  *[MLE]: Maximum Likelihood Estimation
  *[EM]: Expectation-Maximization
  *[E-step]: Expectation-step
  *[M-step]: Maximization-step
  *[SDE]: Stochastic Differential Equation
  *[MCMC]: Markov Chain Monte Carlo