# Post-training Quantization with Progressive Calibration and Activation
Relaxing for Text-to-Image Diffusion Models

Siao Tang
Tsinghua University
Xin Wang
Tsinghua University
Corresponding author Hong Chen
Tsinghua University
Chaoyu Guan
Tsinghua University
Zewen Wu
Tsinghua University
Yansong Tang
Tsinghua University
Wenwu Zhu *
Tsinghua University

###### Abstract

Diffusion models have achieved great success due to their remarkable
generation ability. However, their high computational overhead is still a
troublesome problem. Recent studies have leveraged post-training quantization
(PTQ) to compress diffusion models. However, most of them only focus on
unconditional models, leaving the quantization of widely used large pretrained
text-to-image models, e.g., Stable Diffusion, largely unexplored. In this
paper, we propose a novel post-training quantization method PCR (Progressive
Calibration and Relaxing) for text-to-image diffusion models, which consists
of a progressive calibration strategy that considers the accumulated
quantization error across timesteps, and an activation relaxing strategy that
improves the performance with negligible cost. Additionally, we demonstrate
the previous metrics for text-to-image diffusion model quantization are not
accurate due to the distribution gap. To tackle the problem, we propose a
novel QDiffBench benchmark, which utilizes data in the same domain for more
accurate evaluation. Besides, QDiffBench also considers the generalization
performance of the quantized model outside the calibration dataset. Extensive
experiments on Stable Diffusion and Stable Diffusion XL demonstrate the
superiority of our method and benchmark. Moreover, we are the first to achieve
quantization for Stable Diffusion XL while largely maintaining the
performance.

##  1 Introduction

Diffusion models have achieved great success and demonstrated remarkable
ability in various generative tasks [9, 26, 21, 3, 22, 27], especially in
text-to-image generation [21, 3, 22]. However, one critical limitation of
diffusion models is the high computational cost, which mainly results from the
following two aspects. On the one hand, diffusion models typically require
multiple denoising steps to generate images, which is time-consuming. On the
other hand, the large network architecture of diffusion models usually
requires a great deal of time and memory, especially for foundation models
pretrained on large-scale datasets, e.g., Stable Diffusion [21] and Stable
Diffusion XL [20]. In this paper, we focus on the latter problem, i.e.,
compressing the diffusion models to reduce the computational cost.

Recent studies have explored quantization methods to compress diffusion models
[24, 11, 6, 30, 25], which mainly adopt Post-training Quantization (PTQ) [17,
12]. PTQ has no need for retraining or finetuning the network, making it more
resource-friendly than Quantization-aware Training (QAT) [28, 19]. Most of
these previous works focus on the unconditional diffusion models, leaving the
quantization of the widely used large pretrained text-to-image models, e.g.,
Stable Diffusion, largely unexplored. Only Q-diffusion [11] presents several
quantitative quantization results for the text-to-image diffusion model.
However, the results are limited because of the adopted inaccurate metrics
without considering the data distribution-gap problem.

In this paper, we investigate the largely unexplored text-to-image diffusion
model quantization problem, from both the method and the benchmark, two
perspectives.

On the one hand, about the method, previous works use the full-precision model
to obtain the calibration data for each timestep, and thus ignore the
accumulated quantization error of previous denoising steps, which may limit
their performance. Additionally, they neglect the sensitivity of image
fidelity or text-image matching to different denoising steps, which may
largely influence the quantization performance. On the other hand, about the
metrics, previous works use a prompt subset of the COCO [13] dataset to
perform the calibration and use the FID metric calculated with COCO images to
evaluate the model. However, this metric is inaccurate because there exists a
distribution gap between the generated images by the text-to-image diffusion
model (pretrained on large-scale datasets) and the COCO images. Additionally,
previous metrics only test on the COCO prompts, while ignoring the prompt-
generalization ability to generate images from various other styles of
prompts, which is important to evaluate the pretrained text-to-image diffusion
model in practical scenes.

To tackle the problems, we propose a novel quantization method PCR
(Progressive Calibration and Relaxing) and the QDiffBench benchmark for
quantizing text-to-image diffusion models. Specifically, the proposed PCR
method consists of a progressive calibration strategy and an activation
relaxing strategy. The progressive calibration progressively quantizes each
step with all the previous steps quantized, which can be aware of the
accumulated quantization error across steps. The activation relaxing strategy
relaxes a minor part of critical sensitive timesteps using higher bitwidth to
improve image fidelity or text-image matching, with negligible additional
cost. As for the benchmark, QDiffBench consists of a more accurate FID
calculation strategy and a prompt-generalization evaluation strategy. To deal
with the data distribution-gap problem, QDiffBench calculates the FID [8]
score between the images generated by the full-precision model and the
quantized model. This strategy uses the data in the same domain, making it
more accurate in evaluating the fidelity degradation caused by quantization.
Moreover, QDiffBench evaluates the generalization performance for the
quantized text-to-image diffusion model on the prompts which have quite
different styles from the calibration prompts.

High-resolution text-to-image generation is important in real-world
applications. Besides Stable Diffusion (512 x 512), we also conduct
experiments on Stable Diffusion XL (768 x 768) which is stronger and 4x
larger. Experimental results show that our method causes no CLIP score loss
for Stable Diffusion, and has comparable performance to the full-precision
model for Stable Diffusion XL.

In conclusion, our contributions are summarized as follows:

  1. 1.

We propose a novel PTQ quantization method PCR for text-to-image diffusion
models. It consists of a progressive calibration strategy and an activation
relaxing strategy.

  2. 2.

We propose a more comprehensive and effective benchmark QDiffBench for the
quantization of text-to-image diffusion models. It adopts a more accurate FID
scheme to evaluate the image fidelity, and emphasizes the prompt-
generalization ability of the quantized model.

  3. 3.

Extensive experiments on foundation diffusion models (i.e., Stable Diffusion
and Stable Diffusion XL) demonstrate the superiority of our proposed method
and benchmark.

  4. 4.

We are the first to achieve quantization on Stable Diffusion XL (3.5B
parameters) and generate high-quality quantized images with higher resolution
than 512 × 512.

##  2 Related Work

###  2.1 Diffusion Models

Diffusion models show remarkable performance in various tasks, such as text-
to-image generation [21], image editing [10], and text-video generation [31].
A large number of these works are based on the foundation text-to-image
diffusion models, e.g., Stable Diffusion. Therefore, we believe that the
quantization for large pretrained text-to-image diffusion models (not only on
unconditional models) is significant, which previous works rarely pay
attention to.

Diffusion models define a forward process that adds the noise to the real data
${x}_{0}$:

$\displaystyle\mathbf{x}_{t}=\alpha_{t}\mathbf{x}_{0}+\sigma_{t}\boldsymbol{\epsilon},\
\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),$

where $\alpha_{t}$ and $\sigma_{t}$ are functions of $t$ and the signal-to-
noise-ratio $\alpha_{t}/\sigma_{t}$ is strictly decreasing. Diffusion models
train a neural network ${\boldsymbol{\epsilon}_{\theta}}$ to predict the noise
of a noisy variable ${x}_{t}$, whose training objective is:

$\displaystyle\mathbf{E}_{\mathbf{x}_{0},\boldsymbol{\epsilon},t}[\left\|\boldsymbol{\epsilon}_{\theta}\left(\alpha_{t}\mathbf{x}_{0}+\sigma_{t}\boldsymbol{\epsilon},t\right)-\boldsymbol{\epsilon}\right\|_{2}^{2}],$

In the reverse process, diffusion models generate samples by gradually
denoising from a Gaussian noise
$\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ to $\mathbf{x}_{0}$,
along certain trajectories determined by sampling strategies [9, 14, 15].

###  2.2 Diffusion Model Acceleration

A critical limitation of diffusion models is the iterative multiple-step
sampling, which makes them much slower than their peers, e.g., GANs. To tackle
this problem, substantial works engage in reducing sampling steps [26, 14, 15,
23, 16]. Besides, there exists another limitation, i.e., large network
architecture. To tackle this limitation, quantization methods have been used
to compress and accelerate diffusion models. Specifically, Shang et al.
propose PTQ4DM [24] which collects calibration data in the denoising sampling
process rather than simulating the training process. They also propose that
the timesteps of calibration data should be generated from a skew-normal
distribution. Q-diffusion [11] proposes to collect calibration data at certain
timestep intervals and split quantization for shortcut layers whose input is a
concatenated skip-connection feature. PTQD [6] tries to mathematically correct
the quantization error in the sampling process. TDQ [25] proposes to train the
quantization parameters for every timestep individually, while ADP-DM [30]
proposes to train the group partitions of timesteps and corresponding
quantization parameters. Quantization is not only able to speed up the
sampling process, but also reduce memory footprint. However, all the existing
methods ignore the accumulated error across sampling steps. Additionally, none
of them tailor the design for the setting of text-to-image generation.

###  2.3 Post-training Quantization

Quantization [18] is one of the most significant techniques of model
compression. Quantization methods compress neural networks by compressing the
model weights and the activations to lower bits, e.g., converting 32/16-bit
float weights/activations into formats with INT4 or INT8 bit-width. The
quantization process can be formulated as:

$\displaystyle
w_{q}=\operatorname{clip}\left(\operatorname{round}\left(\frac{w}{s}\right)+z,q_{\min},q_{\max}\right),$

where $s$ is the scaling factor and $z$ is the zero-point. The corresponding
de-quantization process is:

$\displaystyle\hat{w}=s\cdot\left(w_{q}-z\right).$

Quantization methods are usually categorized into two classes, i.e., post-
training quantization (PTQ) [17, 12] and quantization-aware training (QAT)
[28, 19]. QAT methods involve training/finetuning, usually requiring massive
training data and costing many computation resources. In contrast, PTQ methods
only require a small subset of data (e.g., 128 images) for calibration, and
have no need for training, making them more resource-friendly and faster than
QAT methods. PTQ methods have been widely studied for LLMs [4, 32, 33, 2].
Recent studies have explored quantization methods for diffusion models [24,
11, 6, 30, 25], mostly focusing on PTQ.

Figure 1: Progressive calibration for activations. The yellow bar means the
activation has been quantized. We quantize the activations at step $t$ with
all the previous steps quantized, which considers the accumulated quantization
error across timesteps. Figure 2: The distribution shift caused by the
accumulated quantization error. We gather the statistics of sub feature maps
at timestep 1, 10, and 20.

##  3 Method

In this section, we present our method PCR, which consists of a progressive
calibration strategy that considers the accumulated quantization error across
timesteps, and an activation relaxing strategy which can effectively improve
performance with negligible costs. We also present the pseudo-code of PCR in
Algorithm 1 (in the Appendix).

###  3.1 Time-Accumulated Error Aware Progressive Calibration

Substantial studies [24, 11, 25, 30] point out that the distributions of
activation obviously vary over different time steps, which is a challenge for
quantizing diffusion models. To tackle this problem, some studies [25, 30]
adopt time-aware quantization, i.e., separately quantizing the activations for
each timestep, which has been shown to be effective. We also adopt the time-
aware strategy.

We collect the generated samples in the denoising process as the calibration
data, following [11, 24]. Previous works use the full-precision model to
obtain the calibration data for each timestep during the sampling process.
However, they are not aware of the accumulated quantization error across
timesteps in the iterative denoising process. The quantization errors in
previous sampling steps will shift the distribution of the activations in
subsequent steps, as shown in Fig. 2, which makes calibrating with the full-
precision model sub-optimal. To tackle this problem, we propose a progressive
calibration strategy for activations, which quantizes the activations at step
$t$ with all the previous steps $t+1,..,T$ quantized, illustrated in Fig. 1.

We further mathematically prove the necessity of considering the accumulated
quantization errors, as follows:

Notations. Let $\boldsymbol{\epsilon}_{\theta}$ be the denoising net and
$\mathbf{x}_{t}$ be the intermediate variable at timestep $t$. Let the mark
$\hat{..}$ denote the quantized term, and
$\hat{\boldsymbol{\epsilon}}_{\theta}\left(\hat{\mathbf{x}}_{t},t\right)=\boldsymbol{\epsilon}_{\theta}\left(\hat{\mathbf{x}}_{t},t\right)+\Delta_{t}$,
where $\boldsymbol{\epsilon}_{\theta}\left(\hat{\mathbf{x}}_{t},t\right)$ is
the full-precision predicted noise and
$\hat{\boldsymbol{\epsilon}}_{\theta}\left(\hat{\mathbf{x}}_{t},t\right)$ is
the quantized one. Without loss of generality, we use the DDPM sampler there,
i.e.,

$\displaystyle\mathbf{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t\right)\right)+\sigma_{t}\mathbf{z}.$

######  Theorem 1.

Through the multi-step denoising process, the upper bound of the ultimate
quantization error $\delta=\|\mathbf{x}_{0}-\mathbf{\hat{x}}_{0}\|$ can be
approximated as the linear combination of $\\{\|\Delta_{t}\|\\},\ t=1,..,T.$

Proof. For simplicity, we set $\sigma_{t}$ to 0. Consider the first step of
the sampling process, i.e., from $\mathbf{x}_{T}$ to $\mathbf{x}_{T-1}$:

$\displaystyle\mathbf{x}_{T-1}$
$\displaystyle=\frac{1}{\sqrt{\alpha_{T}}}\left(\mathbf{x}_{T}-\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_{T},T)\right).$

After quantization, it derives that:

$\displaystyle\hat{\mathbf{x}}_{T-1}$
$\displaystyle=\frac{1}{\sqrt{\alpha_{T}}}\left(\hat{\mathbf{x}}_{T}-\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\left(\boldsymbol{\epsilon}_{\theta}\left(\hat{\mathbf{x}}_{T},T\right)+\Delta_{T}\right)\right)$
$\displaystyle\overset{(a)}{=}\frac{1}{\sqrt{\alpha_{T}}}\left(\mathbf{x}_{T}-\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\left(\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{T},T\right)+\Delta_{T}\right)\right)$
$\displaystyle=\mathbf{x}_{T-1}-\frac{1}{\sqrt{\alpha_{T}}}\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\Delta_{T}.$
(1)

The equation (a) holds because $\hat{\mathbf{x}}_{T}=\mathbf{x}_{T}$ which is
sampled from $\mathcal{N}(\mathbf{0},\mathbf{I})$. Similarly, we next consider
the step from $\mathbf{x}_{T-1}$ to $\mathbf{x}_{T-2}$:

$\displaystyle\hat{\mathbf{x}}_{T-2}$
$\displaystyle=\frac{1}{\sqrt{\alpha_{T-1}}}\Bigg{(}\hat{\mathbf{x}}_{T-1}-\frac{1-\alpha_{T-1}}{\sqrt{1-\bar{\alpha}_{T-1}}}\Big{(}\boldsymbol{\epsilon}_{\theta}\left(\hat{\mathbf{x}}_{T-1},{T-1}\right)+\Delta_{T-1}\Big{)}\Bigg{)}$
$\displaystyle\overset{(b)}{\approx}\frac{1}{\sqrt{\alpha_{T-1}}}\Bigg{(}\hat{\mathbf{x}}_{T-1}-\frac{1-\alpha_{T-1}}{\sqrt{1-\bar{\alpha}_{T-1}}}\Big{(}\boldsymbol{\epsilon}_{\theta}\left({\mathbf{x}}_{T-1},{T-1}\right)$
$\displaystyle\qquad\qquad\qquad-\boldsymbol{\epsilon}_{\theta}^{\prime}(\mathbf{x}_{T-1},T-1)\frac{1}{\sqrt{\alpha_{T}}}\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\Delta_{T}+\Delta_{T-1}\Big{)}\Bigg{)}$
(2)

where (b) is derived from Taylor expansion, i.e.,

$\displaystyle\boldsymbol{\epsilon}_{\theta}\left(\hat{\mathbf{x}}_{T-1},{T-1}\right)$
$\displaystyle\approx\boldsymbol{\epsilon}_{\theta}\left({\mathbf{x}}_{T-1},{T-1}\right)$
$\displaystyle\quad-\boldsymbol{\epsilon}_{\theta}^{\prime}(\mathbf{x}_{T-1},T-1)\frac{1}{\sqrt{\alpha_{T}}}\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\Delta_{T}.$

Replacing $\hat{\mathbf{x}}_{T-1}$ with Sec. 3.1, Sec. 3.1 can be written as:

$\displaystyle\hat{\mathbf{x}}_{T-2}$
$\displaystyle\overset{(b)}{\approx}\frac{1}{\sqrt{\alpha_{T-1}}}\Bigg{(}\mathbf{x}_{T-1}-\frac{1}{\sqrt{\alpha_{T}}}\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\Delta_{T}$
$\displaystyle\qquad\qquad\qquad-\frac{1-\alpha_{T-1}}{\sqrt{1-\bar{\alpha}_{T-1}}}\Big{(}\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{T-1},{T-1}\right)$
$\displaystyle\qquad\qquad\qquad-\boldsymbol{\epsilon}_{\theta}^{\prime}(\mathbf{x}_{T-1},T-1)\frac{1}{\sqrt{\alpha_{T}}}\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\Delta_{T}+\Delta_{T-1}\Big{)}\Bigg{)}$
$\displaystyle=\mathbf{x}_{T-2}-\frac{1}{\sqrt{\alpha_{T}}}\frac{1}{\sqrt{\alpha_{T-1}}}\cdot\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\Delta_{T}$
$\displaystyle\quad+\frac{1}{\sqrt{\alpha_{T}}}\frac{1}{\sqrt{\alpha_{T-1}}}\cdot\frac{1-\alpha_{T}}{\sqrt{1-\bar{\alpha}_{T}}}\frac{1-\alpha_{T-1}}{\sqrt{1-\bar{\alpha}_{T-1}}}\boldsymbol{\epsilon}_{\theta}^{\prime}(\mathbf{x}_{T-1},T-1)\Delta_{T}$
$\displaystyle\quad-\frac{1}{\sqrt{\alpha_{T-1}}}\frac{1-\alpha_{T-1}}{\sqrt{1-\bar{\alpha}_{T-1}}}\Delta_{T-1}.$
(3)

Sec. 3.1 indicates $\mathbf{x}_{T-1}-\hat{\mathbf{x}}_{T-1}$ is the linear
combination of $\Delta_{T}$. Similarly, Sec. 3.1 proves
$\mathbf{x}_{T-2}-\hat{\mathbf{x}}_{T-2}$ is the linear combination of
$\Delta_{T}$ and $\Delta_{T-1}$. Iteratively, we can prove
$\mathbf{x}_{0}-\hat{\mathbf{x}}_{0}$ is the linear combination of
$\\{\Delta_{t}\\},\ t=1,..,T$. Then, we can easily derive that the upper bound
of $\delta=\|\mathbf{x}_{0}-\hat{\mathbf{x}}_{0}\|$ is the linear combination
of $\\{\|\Delta_{t}\|\\},\ t=1,..,T.$

######  Corollary 1.

To minimize the upper bound of the final quantization error
$\delta=\|\mathbf{x}_{0}-\hat{\mathbf{x}}_{0}\|$, we should minimize the
quantization error $\|\Delta_{t}\|$ at each timestep $t,\ t=1,..,T.$

Note that,
$\|\Delta_{t}\|=\|\hat{\boldsymbol{\epsilon}}_{\theta}\left(\hat{\mathbf{x}}_{t},t\right)-\boldsymbol{\epsilon}_{\theta}\left(\hat{\mathbf{x}}_{t},t\right)\|$
is related to the quantized input $\hat{\mathbf{x}}_{t}$ rather than the full-
precision input $\mathbf{x}_{t}$. Therefore, to minimize $\|\Delta_{t}\|$, we
should use the data generated with all its previous steps quantized as the
calibration data.

###  3.2 Time-wise Activation Relaxing

We observe the fidelity and semantics of the generated images by the text-to-
image diffusion models are prone to activation quantization. Moreover, we find
that the image fidelity is more sensitive to the timesteps near $x_{0}$, while
the text-image matching is more sensitive to the steps near $x_{T}$. Based on
this observation, we propose to relax a minor part of timesteps using higher
bitwidth to improve image fidelity or text-image matching. Let
${b_{t}},t=1,..,T$ denote the bit widths for activation quantization at step
$t$. For example, for the 8-bit activation quantization, to improve image
fidelity, we set $b_{1}$ to $b_{5}$ to a higher bit, e.g., 10-bit, and set the
others to 8-bit. To improve the text-image matching, we set $b_{T-4}$ to
$b_{T}$ to 10-bit. This strategy remarkably improves the performance, and only
requires negligible costs, because we only relax a small proportion of
timesteps. For instance, in 8-bit quantization, relaxing 5% steps to 10 bits
while keeping others the 8-bit actually results in the average 8.1 bits. In
our experiments, we obtain the following observation:

Stable Diffusion is sensitive to image fidelity degradation brought by
quantization, while Stable Diffusion XL is sensitive to text-image matching.

With the observation, we relax the steps near $x_{0}$ for Stable Diffusion. In
contrast, Stable Diffusion XL is more sensitive to text-image matching, which
might be the result of using massive cross-attention transformer blocks. Most
images generated by the quantized Stable Diffusion XL with previous methods
can not properly match the textual semantics, shown in Fig. 3. To tackle this
problem, we relax the timesteps near $x_{T}$ and can generate high-quality
images matching the text.

Figure 3: Stable Diffusion XL 768x768 generation results. PTQ4DM and
Q-diffusion fail to match the textual semantics, while our method with the
activation relaxing can generate high-quality images matching the prompt.

##  4 Benchmark

Previous works rarely test their methods on text-to-image generation tasks.
Only Q-diffusion [11] reports some quantitative results of Stable Diffusion.
However, we observe that there exist some problems in the way of calculating
FID used in Q-diffusion, which is caused by distribution-gap. Moreover, they
ignore the prompt-generalization ability of the foundation text-to-image
model. In this section, we discuss these issues of previous metrics and
propose a novel benchmark QDiffBench, which consists of a more accurate FID
strategy and a prompt-generalization evaluation strategy, to accurately and
comprehensively evaluate the quantization of text-to-image diffusion models.

###  4.1 FID to Full Precision Model

FID ((Frechet Inception Distance) [8] is an important metric to evaluate the
quality of generative models. FID can measure the fidelity of generated images
by calculating the distance between the generated images and the real images,
where a lower score indicates better performance. FID is also used to evaluate
text-to-image diffusion models [21]. For example, Rombach et al. [21] evaluate
latent text-to-image diffusion models on the COCO validation dataset. They
first generate images conditioned by COCO prompts, and calculate the FID score
between the generated images and real COCO images.

Previous studies also use the same way to evaluate quantized text-to-image
diffusion models [11], i.e., generating images with a quantized model and
calculating FID between the generated and the real ones. We name this strategy
“FID to COCO”. However, we demonstrate this strategy is inaccurate in the
context of quantization. For instance, as shown in Tab. 1, the quantized
models usually have similar or even lower “FID to COCO” scores than the full-
precision model. However, the generated images by the quantized models are
generally worse than the precision ones, taking Fig. 5 as an example. In
addition, Tab. 1 also shows the W4A8 models have better “FID to COCO” scores
than the W8A8 peers, which is also inconsistent with the visualized results,
as shown in the Appendix (Fig. 9). These phenomena prove the inaccuracy of
“FID to COCO”. The reason is that the distribution of the generated images by
text-to-image diffusion models obviously differs from the COCO images. The
manifestations of this distribution gap include style differences, fake
details in generated images, etc. We detailedly analyze the reasons and
manifestations of distribution-gap in the Appendix (Sec. 10). This gap makes
the distance to the COCO images inaccurate to measure the quality of the
images generated by quantized models. Therefore, the worse quantized models
can even have better “FID to COCO” scores. To tackle this problem, we propose
to directly compare the images generated by the quantized model with the
images generated by the full precision model, i.e., calculating the FID score
between them, as shown in Fig. 4. We name this strategy “FID to FP32”. This
strategy straightforwardly measures the fidelity degradation caused by the
quantization algorithm. It compares the generated images to the data in the
same domain, making it more accurate. Tab. 1 also shows our “FID to FP32’
metric is basically consistent with the CLIP score. Fig. 5 and the samples in
the Appendix also visually demonstrate our metric is consistent with human
cognition.

Figure 4: We calculate the FID to the images generated by the full-precision
model rather than the COCO images. COCO images are photorealism, while Stable
Diffusion can’t achieve this perfectly. Additionally, there are obvious fake
details in the generated plane images.

###  4.2 Prompt-Generalization Ability

An outstanding feature of text-to-image diffusion models is that they can
generate images of various styles and themes, as long as the user enters the
appropriate prompts. Although foundation diffusion models (e.g., Stable
Diffusion and Stable Diffusion XL) are pretrained on billions of image-text
pairs, researchers can usually only use a small set of data for quantization.
For example, previous studies use a subset of prompts from the COCO dataset to
perform calibration. In the evaluation, they only test the quantized models on
the COCO prompts. However, users always input prompts with a quite different
style from the calibration dataset, and generate images of various styles.
Therefore, to evaluate the prompt-generalization of the quantized models, we
propose to test them also on the unseen prompts that have a quite different
style from the calibration dataset. In our experiments, we evaluate the
quantized model on the dataset referred to as  Stable-Diffusion-Prompts [5],
which is a popular Stable Diffusion prompt dataset released on hugging face
[1]. It consists of about 80,000 user-created high-quality prompts that can
generate images of various styles. As shown in Fig. 6, the generated images by
this dataset have quite different styles from the calibration dataset (COCO
prompts in our settings), i.e.:

COCO prompts always generate realistic photographs, while Stable-Diffusion-
Prompts tends to generate artistic images.

We use the first 5,000 prompts of  Stable-Diffusion-Prompts with the token
number fewer than 30 as the prompt-generalization test set.

Figure 5: Comparison between the quantized models and the full-precision
model. Use Stable diffusion 512 x 512. Figure 6: Stable Diffusion 512 x 512
generation results on COCO prompts and Stable-Diffusion-Prompts.

##  5 Experiments

###  5.1 Experimental Setup

Datasets. Our experiments involve with two datasets, i.e., COCO [13] and
Stable-Diffusion-Prompts [5] (as discussed in Sec. 4.2). During the
calibration, we use a few prompts of the COCO [13] training dataset. During
the evaluation, on one hand, we use 5,000 prompts from the previously widely
used COCO [13] validation dataset. On the other hand, we further use 5,000
prompts in the Stable-Diffusion-Prompts dataset to evaluate the prompt-
generalization ability of the quantized model.

Metrics. We mainly adopt two metrics. The first metric we adopt is the “FID to
FP32”, which is the FID score between the quantized-model generated images and
full-precision-model generated images. Additionally, we also adopt the CLIP
score [7] to evaluate whether the generated images match the text prompt. For
the reference and the illustration of the problem in Sec. 4.1, we also present
the metric “FID to COCO” adopted by previous works, but it is not proper to be
used for comparison due to the distribution gap.

Baselines and implementation. We compare our methods with two state-of-the-art
methods Q-diffusion [11] and PTQ4DM [24]. For a fair comparison, we implement
all the baselines and our method in a unified framework based on diffusers
[29]. For quantization, we use block reconstruction [12] to adaptively round
[17] the weight by minimizing the mean squared errors between the quantized
and the full precision outputs, following previous works [11, 24, 6]. We use
the Stable Diffusion v1-4 (with resolution 512x512) and the Stable Diffusion
XL 1.0 (with resolution 768x768) checkpoints provided by hugging face [1]. We
use the default configuration, i.e., 50-step PNDM sampling for Stable
Diffusion and 50-step Euler sampling for Stable Diffusion XL, both with the
cfg scale set to 7.5. The calibration prompt number is 200 for Stable
Diffusion and 30 for Stable Diffusion XL. By default, we relax 20% steps near
$x_{0}$ for Stable Diffusion and 20% steps near $x_{T}$ for Stable Diffusion
XL. But we demonstrate relaxing only 5% steps also outperforms baselines by an
obvious margin, as discussed in Sec. 5.3. We use the package torch-fidelity to
calculate the FID scores, and use the model ViT/L-14 to calculate CLIP scores.

###  5.2 Main Results

Tab. 1 shows the results of Stable Diffusion on the COCO validation prompts,
which prove the effectiveness of our proposed method and benchmark.
Specifically, “FID to COCO” is the metric used in previous works, while “FID
to FP32” is our proposed metric that calculates the distance between the
quantized model and the full-precision 32-bit model. The quantized models have
similar or even lower “FID to COCO” than the full-precision model, which is
inaccurate, as discussed in Sec. 4.1. On the other hand, the results of the
CLIP score and our “FID to FP32” show our method reaches a new state-of-the-
art performance.

Method | Bits(W/A) | FID to COCO | FID to FP32$\downarrow$ | CLIP score$\uparrow$
---|---|---|---|---
FP32 | 32/32 | 27.50 | 0.00 | 26.46
Q-diffusion | 8/8 | 29.98 | 18.64 | 26.15
PTQ4DM | 8/8 | 28.00 | 14.60 | 26.33
PCR(Ours) | 8/8 | 25.83 | 8.35 | 26.47
Q-diffusion | 4/8 | 27.87 | 20.42 | 26.15
PTQ4DM | 4/8 | 25.64 | 17.73 | 26.25
PCR(Ours) | 4/8 | 22.04 | 14.25 | 26.48
PCR(Ours) | 8/32 | 27.58 | 2.95 | 26.47
PCR(Ours) | 4/32 | 24.29 | 12.45 | 26.50

Table 1: Quantization results on the COCO validation prompts for Stable
Diffusion v1-4. $\uparrow$ means higher metric is better, while $\downarrow$
means lower is better.

Method | Bits(W/A) | FID to COCO | FID to FP32$\downarrow$ | CLIP score$\uparrow$
---|---|---|---|---
FP32 | 32/32 | 27.28 | 0.00 | 26.44
Q-diffusion | 8/8 | 42.85 | 38.19 | 15.97
PTQ4DM | 8/8 | 42.33 | 38.65 | 15.98
PCR(Ours) | 8/8 | 26.16 | 12.00 | 24.05
Q-diffusion | 4/8 | 45.43 | 44.07 | 15.91
PTQ4DM | 4/8 | 43.37 | 46.45 | 15.92
PCR(Ours) | 4/8 | 24.27 | 18.27 | 23.85

Table 2: Quantization results on the COCO validation prompts for Stable
Diffusion XL 768 x 768.

#### The discussion about CLIP score.

Tab. 1 also reveals that the CLIP score sometimes is not able to effectively
distinguish the quality of quantized models. For example, the CLIP score of
PCR 4/8 is a little higher than PCR 8/8, even though the quality of PCR 8/8 is
obviously better, as shown in Fig. 9 and Fig. 10 in the Appendix. Tab. 5 also
indicates the similar circumstance. These phenomena support the significance
of our “FID to FP32” metric to measure the fidelity. Therefore, we should
combine the CLIP score and our “FID to FP32” score to comprehensively measure
the quality of quantized models.

To further validate our method for the larger model and higher resolution, we
experiment on Stable Diffusion XL, generating images with a resolution of 768
$\times$ 768\. Tab. 2 show the results of Stable Diffusion XL on COCO
validation prompts. Our method outperforms previous methods by an obvious
margin. Moreover, the visualized results in Fig. 3 show that previous methods
fail to generate images that can properly match the prompt, while our method
is able to generate high-quality images matching the text.

To assess the prompt generalization of our method, we evaluate the models on
the datasets Stable-Diffusion-Prompts with diverse style prompts. Tab. 3 show
the results of Stable Diffusion. Although our method has a slightly higher FID
score in the setting of W4/A8, its CLIP score obviously exceeds the baselines.
Tab. 4 show the results of Stable Diffusion XL. These results prove that our
method has better generalization performance on the unseen prompts outside the
calibration dataset.

Comparing the results evaluated on COCO with the results on Stable-Diffusion-
Prompts, we can observe some differences. For example, the gap between these
quantized Stable Diffusion XL models is obviously smaller on the Stable-
Diffusion-Prompts than the COCO. This may indicate the overfitting to the
calibration data, which also supports the necessity of the evaluation on the
unseen prompts outside the calibration data.

Fig. 7 shows the generated samples by Stable Diffusion XL using Stable-
Diffusion-Prompts. More samples can be found in the Appendix.

Method | Bits(W/A) | FID to FP32$\downarrow$ | CLIP score$\uparrow$
---|---|---|---
FP32 | 32/32 | 0.00 | 28.79
Q-diffusion | 8/8 | 16.22 | 27.19
PTQ4DM | 8/8 | 13.25 | 27.77
PCR(Ours) | 8/8 | 9.52 | 28.74
Q-diffusion | 4/8 | 17.43 | 27.31
PTQ4DM | 4/8 | 17.28 | 27.39
PCR(Ours) | 4/8 | 17.95 | 28.05

Table 3: Results on the Stable-Diffusion-Prompts dataset for Stable Diffusion
v1-4.

Method | Bits(W/A) | FID to FP32$\downarrow$ | CLIP score$\uparrow$
---|---|---|---
FP32 | 32/32 | 0.00 | 29.74
Q-diffusion | 8/8 | 22.81 | 20.01
PTQ4DM | 8/8 | 22.85 | 20.09
PCR(Ours) | 8/8 | 11.71 | 25.74
Q-diffusion | 4/8 | 24.98 | 19.93
PTQ4DM | 4/8 | 28.28 | 20.13
PCR(Ours) | 4/8 | 18.25 | 25.71

Table 4: Results on the Stable-Diffusion-Prompts dataset for Stable Diffusion
XL 768 x 768. Figure 7: Stable Diffusion XL 768x768 image generation using
Stable-Diffusion-Prompts.

###  5.3 Ablation Study

The effects of our components. In this section, we analyze the effects of
different components. We gradually add our components and test the
performance. Tab. 5 shows the results of Stable Diffusion tested on COCO
prompts, which verifies the effectiveness of our activation relaxing and
progressive calibration strategy.

The effects of the relaxing proportion. We explore how the proportion of the
relaxed steps affects the performance. Fig. 8 shows the score changes of
Stable Diffusion (SD) and Stable Diffusion XL (SDXL) with respect to the
proportion. The results show the gain from relaxing more steps decreases
rapidly as the proportion grows, which means the steps nearest the
$x_{0}/x_{T}$ are the most important to the image-fidelity/text-image-
matching. It also indicates relaxing only 5% steps can also obviously
outperform the baselines.

Method | Bits(W/A) | FID to FP32$\downarrow$ | CLIP score$\uparrow$
---|---|---|---
FP32 | 32/32 | 0.00 | 26.46
Base | 8/8 | 15.90 | 26.46
\+ Relax | 8/8 | 9.02 | 26.42
\+ Prog (PCR) | 8/8 | 8.35 | 26.47
Base | 4/8 | 25.29 | 26.21
\+ Relax | 4/8 | 15.63 | 26.35
\+ Prog (PCR) | 4/8 | 14.25 | 26.48

Table 5: Ablation results on the COCO validation prompts for Stable Diffusion
v1-4. “Relax” denotes the activation relaxing, and “Prog” denotes the
progressive calibration. Figure 8: The variation of the scores with respect to
the relaxing proportion, on the COCO validation prompts, W8/A8. The solid
lines denote the scores of PCR. The dashed lines parallel to the x-axis denote
the corresponding scores of the baselines.

##  6 Conclusion

Considering the significance of text-to-image generation, this paper aims to
advance the quantization of text-to-image diffusion models. Firstly, we
propose a quantization method PCR consisting of a progressive calibration
strategy and an activation relaxing strategy. Additionally, we point out the
limitations of previous metrics and propose a more effective benchmark
QDiffBench for the quantization of text-to-image diffusion models. Extensive
experiments prove the superiority of our method and benchmark.

## References

  * [1] https://huggingface.co/.
  * Bai et al. [2022] Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu.  Towards efficient post-training quantization of pre-trained language models.  _Advances in Neural Information Processing Systems_ , 35:1405–1418, 2022.
  * Chen et al. [2023] Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu.  Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation.  _arXiv preprint arXiv:2305.03374_ , 2023.
  * Frantar et al. [2022] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  Gptq: Accurate post-training quantization for generative pre-trained transformers.  _arXiv preprint arXiv:2210.17323_ , 2022.
  * [5] Gustavosta.  https://huggingface.co/datasets/Gustavosta/Stable-Diffusion-Prompts.
  * He et al. [2023] Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang.  Ptqd: Accurate post-training quantization for diffusion models.  _arXiv preprint arXiv:2305.10657_ , 2023.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7514–7528, 2021.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6007–6017, 2023.
  * Li et al. [2023] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer.  Q-diffusion: Quantizing diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 17535–17545, 2023.
  * Li et al. [2021] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu.  Brecq: Pushing the limit of post-training quantization by block reconstruction.  _arXiv preprint arXiv:2102.05426_ , 2021.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2021] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.  Pseudo numerical methods for diffusion models on manifolds.  In _International Conference on Learning Representations_ , 2021.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022.
  * Luhman and Luhman [2021] Eric Luhman and Troy Luhman.  Knowledge distillation in iterative generative models for improved sampling speed.  _arXiv preprint arXiv:2101.02388_ , 2021.
  * Nagel et al. [2020] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.  Up or down? adaptive rounding for post-training quantization.  In _International Conference on Machine Learning_ , pages 7197–7206. PMLR, 2020.
  * Nagel et al. [2021] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort.  A white paper on neural network quantization.  _arXiv preprint arXiv:2106.08295_ , 2021.
  * Nagel et al. [2022] Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort.  Overcoming oscillations in quantization-aware training.  In _International Conference on Machine Learning_ , pages 16318–16330. PMLR, 2022.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Salimans and Ho [2021] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations_ , 2021.
  * Shang et al. [2023] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan.  Post-training quantization on diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1972–1981, 2023.
  * So et al. [2023] Junhyuk So, Jungwon Lee, Daehyun Ahn, Hyungjun Kim, and Eunhyeok Park.  Temporal dynamic quantization for diffusion models.  _arXiv preprint arXiv:2306.02316_ , 2023.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2020b.
  * Tailor et al. [2020] Shyam A Tailor, Javier Fernandez-Marques, and Nicholas D Lane.  Degree-quant: Quantization-aware training for graph neural networks.  _arXiv preprint arXiv:2008.05000_ , 2020.
  * von Platen et al. [2022] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * Wang et al. [2023] Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu.  Towards accurate data-free quantization for diffusion models.  _arXiv preprint arXiv:2305.18723_ , 2023.
  * Wu et al. [2023] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7623–7633, 2023.
  * Xiao et al. [2023] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In _International Conference on Machine Learning_ , pages 38087–38099. PMLR, 2023.
  * Yao et al. [2022] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He.  Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.  _Advances in Neural Information Processing Systems_ , 35:27168–27183, 2022.

Supplementary Material

##  7 Additional Qualitative Results

We show more visualized results of quantized models on COCO prompts and
Stable-Diffusion-prompts, i.e., Figs. 9, 10 and 11. The COCO prompts tend to
generate realistic photographs, while Stable-Diffusion-Prompts tends to
generate artistic images. We also present the pseudo-code of our proposed PCR
method in Algorithm 1.

##  8 Additional Results of Unconditional Diffusion

We also validate our method in the unconditional scenarios, even though it is
not our major concern. We use the pretrained checkpoint CompVis/ldm-
celebahq-256 released on hugging face. We generate 50,000 images and calculate
the FID to the original training data, i.e., CelebA-HQ. The results are shown
in Tab. 6, which indicates our method is also effective for unconditional
diffusion models.

Method | Bits(W/A) |  FID $\downarrow$
---|---|---
FP32 | 32/32 | 12.24
Q-diffusion | 8/8 | 18.43
PTQ4DM | 8/8 | 17.18
PCR(Ours) | 8/8 | 13.22
Q-diffusion | 4/8 | 22.59
PTQ4DM | 4/8 | 22.15
PCR(Ours) | 4/8 | 19.38

Table 6: Results on the CelebA-HQ using the unconditional latent diffusion
model. Use DDIM 100-step sampler. Algorithm 1 Our proposed PCR algorithm

0: The full-precision model $W_{\theta}$, Sampling steps $T$

0: Relaxing timesteps list $R$, activation bitwidths $B$

0: The calibration prompts $\mathcal{P}$ selected from COCO

0: Quantized weight $\hat{W}_{\theta}$ and activation quantizers
$\\{q_{i}\\},i=1,...,T$

1: Calibrate the weight quantizers with $W_{\theta}$ and $\mathcal{P}$. Update
quantized weight $\hat{W}_{\theta}$

2: Activation relaxing:

3: for $i=1$ to $T$ do

4: if $i$ in $R$ then

5: set $B[i]$ to higher bit

6: end if

7: end for

8: Progressive Calibration:

9: for $t=T$ to $1$ do

10: generate calibration data at $t$ with $\hat{W}_{\theta}$, $\mathcal{P}$,
and updated quantizer $\\{q_{i}\\},i=t+1,...,T$ of previous steps

11: calibrate and update the quantizer at t, i.e., $q_{t}$, with bitwidth
$B[t]$

12: end for

Method | Bits(W/A) |  FID to FP32 $\downarrow$ |  CLIP score $\uparrow$
---|---|---|---
FP32 | 32/32 | 0.00 | 29.21
Q-diffusion | 8/8 | 16.30 | 27.98
PTQ4DM | 8/8 | 14.26 | 28.41
PCR(Ours) | 8/8 | 10.03 | 29.20
Q-diffusion | 4/8 | 18.77 | 27.84
PTQ4DM | 4/8 | 17.84 | 28.07
PCR(Ours) | 4/8 | 18.10 | 28.65

Table 7: Results on the DiffusionDB dataset for Stable Diffusion 512 x 512.

##  9 Additional Results on DiffusionDB datasets

To further evaluate the prompt-generalization ability of the model, we also
conducted experiments on the DiffusionDB dataset. Like Stable-Diffusion-
Prompts, DiffusionDB contains human-created prompts, which simulate the
situation in practical uses. The results are shown in Tab. 7, which indicates
our method performs excellently on this prompt-generalization dataset.

##  10 Further Analysis about Distribution Gap

We have pointed out the reason why “FID to COCO” is inaccurate is the data
distribution-gap between COCO images and diffusion-model-generated images. We
believe that the distribution-gap may result from two aspects: i) Large text-
to-image diffusion models are pretrained on the large-scale dataset, e.g.,
LAION-5B, whose distribution quite differs from COCO. ii) Additionally,
limited by the text-to-image generative ability of the model, generating high-
quality images that perfectly match the prompt can sometimes be challenging,
while text-image matching of COCO is good because of the human annotations.
Moreover, there are fake details more or less in the generated images,
especially for human faces, keyboards, and so on. We may need to try different
random seeds many times to produce a satisfying image. We demonstrate this
statement with Fig. 12. Row 1 and 2 show the distribution gap in the form of
style difference. Row 3 and 4 show that the generated images are not
satisfactory because of not matching the prompt well.

Figure 9: Stable Diffusion image generation using COCO prompts. Figure 10:
Stable Diffusion image generation using Stable-Diffusion-prompts. Figure 11:
Stable Diffusion XL image generation using COCO prompts. Figure 12: The
comparison between the COCO images and the Stable Diffusion generated images.
Row 1 and 2 show that COCO images are photorealism, while Stable Diffusion
can’t achieve this well. Row 3 and 4 show that Stable Diffusion can sometimes
be challenging to generate images with perfect text-image matching.