# Rich Human Feedback for Text-to-Image Generation

Youwei Liang Co-first authors, equal technical contributionThe work is done
during an internship at Google University of California San Diego  Junfeng
He11footnotemark: 1 Corresponding authors, equal leading contribution:
<EMAIL_ADDRESS>Google Research  Gang Li11footnotemark:
1††footnotemark:  Google Research  Peizhao Li22footnotemark: 2 Brandeis
University  Arseniy Klimovskiy  Google Research  Nicholas Carolan  Google
Research  Jiao Sun22footnotemark: 2 University of Southern California  Jordi
Pont-Tuset  Google Research  Sarah Young  Google Research  Feng Yang  Google
Research  Junjie Ke  Google Research  Krishnamurthy Dj Dvijotham  Google
Research  Katie Collins22footnotemark: 2 University of Cambridge  Yiwen Luo
Google Research  Yang Li  Google Research  Kai J Kohlhoff  Google Research
Deepak Ramachandran  Google Research  Vidhya Navalpakkam  Google Research

###### Abstract

Recent Text-to-Image (T2I) generation models such as Stable Diffusion and
Imagen have made significant progress in generating high-resolution images
based on text descriptions. However, many generated images still suffer from
issues such as artifacts/implausibility, misalignment with text descriptions,
and low aesthetic quality. Inspired by the success of Reinforcement Learning
with Human Feedback (RLHF) for large language models, prior works collected
human-provided scores as feedback on generated images and trained a reward
model to improve the T2I generation. In this paper, we enrich the feedback
signal by (i) marking image regions that are implausible or misaligned with
the text, and (ii) annotating which words in the text prompt are
misrepresented or missing on the image. We collect such rich human feedback on
18K generated images and train a multimodal transformer to predict the rich
feedback automatically. We show that the predicted rich human feedback can be
leveraged to improve image generation, for example, by selecting high-quality
training data to finetune and improve the generative models, or by creating
masks with predicted heatmaps to inpaint the problematic regions. Notably, the
improvements generalize to models (Muse) beyond those used to generate the
images on which human feedback data were collected (Stable Diffusion
variants).

##  1 Introduction

Text-to-image (T2I) generation models [42, 12, 56, 59, 41, 58, 17] are rapidly
becoming a key to content creation in various domains, including
entertainment, art, design, and advertising, and are also being generalized to
image editing [44, 4, 27, 50], video generation [23, 35, 53], among many other
applications. Despite significant recent advances, the outputs still usually
suffer from issues such as artifacts/implausibility, misalignment with text
descriptions, and low aesthetic quality [54, 30, 52]. For example, in the
Pick-a-Pic dataset [30], which mainly consists of images generated by Stable
Diffusion model variants , many images (_e.g_. Fig. 1) contain distorted
human/animal bodies (_e.g_. human hands with more than five fingers),
distorted objects and implausibility issues such as a floating lamp. Our human
evaluation experiments find that only $\sim$10% of the generated images in the
dataset are free of artifacts and implausibility. Similarly, text-image
misalignment issues are common too, _e.g_., the prompt is “a man jumping into
a river” but the generated image shows the man standing.

Existing automatic evaluation metrics for generated images, however, including
the well-known IS [43] and FID [20], are computed over distributions of images
and may not reflect nuances in individual images. Recent research has
collected human preferences/ratings to evaluate the quality of generated
images and trained evaluation models to predict those ratings [54, 30, 52],
notably ImageReward [54] or Pick-a-Pic [30]. While more focused, these metrics
still summarize the quality of one image into a single numeric score. In terms
of prompt-image alignment, there are also seminal single-score metrics such as
CLIPScore [19] and more recent question generation and answering pipelines
[24, 57, 10, 8]. While more calibrated and explainable, these are expensive
and complex models that still do not localize the regions of misalignment in
the image.

In this paper, we propose a dataset and a model of fine-grained multi-faceted
evaluations that are interpretable and attributable (_e.g_., to regions with
artifacts/implausibility or image-text misalignments), which provide a much
richer understanding of the image quality than single scalar scores. As a
first contribution, we collect a dataset of Rich Human Feedback on 18K images
(RichHF-18K), which contains (i) point annotations on the image that highlight
regions of implausibility/artifacts, and text-image misalignment; (ii) labeled
words on the prompts specifying the missing or misrepresented concepts in the
generated image; and (iii) four types of fine-grained scores for image
plausibility, text-image alignment, aesthetics, and overall rating. Equipped
with RichHF-18K, we design a multimodal transformer model, which we coin as
Rich Automatic Human Feedback (RAHF) to learn to predict these rich human
annotations on generated images and their associated text prompt. Our model
can therefore predict implausibility and misalignment regions, misaligned
keywords, as well as fine-grained scores. This not only provides reliable
ratings, but also more detailed and explainable insights about the quality of
the generated images. To the best of our knowledge, this is the first rich
feedback dataset and model for state-of-the-art text-to-image generation
models, providing an automatic and explainable pipeline to evaluate T2I
generation.

The main contributions of this paper are summarized below:

  1. 1.

The first Rich Human Feedback dataset (RichHF-18K) on generated images
(consisting of fine-grained scores, implausibility(artifact)/misalignment
image regions, and misalignment keywords), on 18K Pick-a-Pic images. The
RichHF-18K data set will be released soon.

  2. 2.

A multimodal Transformer model (RAHF) to predict rich feedback on generated
images, which we show to be highly correlated with the human annotations on a
test set.

  3. 3.

We further demonstrate the usefulness of the predicted rich human feedback by
RAHF to improve image generation: (i) by using the predicted heatmaps as masks
to inpaint problematic image regions and (ii) by using the predicted scores to
help finetune image generation models (like Muse [6]), _e.g_., via
selecting/filtering finetuning data, or as reward guidance. We show that in
both cases we obtain better images than with the original model.

  4. 4.

The improvement on the Muse model, which differs from the models that
generated the images in our training set, shows the good generalization
capacity of our RAHF model.

##  2 Related works

##### Text-to-image generation

Text-to-image (T2I) generation models have evolved and iterated through
several popular model architectures in the deep learning era. An early work is
the Generative Adversarial Network (GAN) [16, 3, 26], which trains a generator
for image generation and a discriminator to distinguish between real and
generated images, in parallel (also see [60, 55, 62, 38, 32, 47] among
others). Another category of generation models develops from variational auto-
encoders (VAEs) [29, 21, 48], which optimize evidence lower bound (ELBO) for
the likelihood of the image data.

Most recently, Diffusion Models (DMs) [46, 22, 36, 41] have emerged as the
SOTA for Image Generation [13]. DMs are trained to generate images
progressively from random noise, with the ability to capture more diversity
than GANs and achieve good sample quality [13]. Latent Diffusion Models [41]
are a further refinement that performs the diffusion process in a compact
latent space for more efficiency.

##### Text-to-image evaluation and reward models

There has been much recent work on evaluation of text-to-image models along
many dimensions [54, 30, 25, 52, 51, 37, 31, 9]. Xu et al. [54] collect a
human preference dataset by requesting users to rank multiple images and rate
them according to their quality. They trained a reward model ImageReward for
human preference learning, and proposed Reward Feedback Learning (ReFL) for
tuning diffusion models with the ImageReward model. Kirstain et al. [30] built
a web application to collect human preferences by asking users to choose the
better image from a pair of generated images, resulting in a dataset called
Pick-a-Pic with more than 500K examples generated by T2I models such as Stable
Diffusion 2.1, Dreamlike Photoreal 2.05, and Stable Diffusion XL variants.
They leveraged the human preference dataset to train a CLIP-based [39] scoring
function, called PickScore, to predict human preferences. Huang et al. [25]
proposed a benchmark called T2I-CompBench for evaluating text-to-image models,
which consists of 6,000 text prompts describing attribute binding, object
relationships, and complex compositions. They utilized multiple pretrained
vision-language models such as CLIP [39] and BLIP [34] to calculate multiple
evaluation metrics. Wu et al. [52, 51] collected a large scale dataset of
human choices on generated images and utilized the dataset to train a
classifier that outputs a Human Preference Score (HPS). They showed
improvement in image generation by tuning Stable Diffusion with the HPS.
Recently, Lee [31] proposed a holistic evaluation for T2I models with multiple
fine-grained metrics.

Despite these valuable contributions, most existing works only use binary
human ratings or preference ranking for construction of feedback/rewards, and
lack the ability to provide detailed actionable feedback such as implausible
regions of the image, misaligned regions, or misaligned keywords on the
generated images. One recent paper related to our work is Zhang et al. [61],
which collected a dataset of artifact regions for image synthesis tasks,
trained a segmentation-based model to predict artifact regions, and proposed a
region inpainting method for those regions. However, the focus of their work
is artifact region only, while in this paper, we collected rich feedback for
T2I generation containing not only artifact regions, but also misalignment
regions, misaligned keywords, and four fine-grained scores from multiple
aspects. To the best of our knowledge, this is the first work on heterogeneous
rich human feedback for text-to-image models.

##  3 Collecting rich human feedback

Figure 1: An illustration of our annotation UI. Annotators mark points on the
image to indicate artifact/implausibility regions (red points) or misaligned
regions (blue points) w.r.t the text prompt. Then, they click on the words to
mark the misaligned keywords (underlined and shaded) and choose the scores for
plausibility, text-image alignment, aesthetics, and overall quality
(underlined).

###  3.1 Data collection process

In this section, we discuss our procedure to collect the RichHF-18K dataset,
which includes two heatmaps (artifact/implausibility and misalignment), four
fine-grained scores (plausibility, alignment, aesthetics, overall), and one
text sequence (misaligned keywords).

For each generated image, the annotators are first asked to examine the image
and read the text prompt used to generate it. Then, they mark points on the
image to indicate the location of any implausibility/artifact or misalignment
w.r.t the text prompt. The annotators are told that each marked point has an
“effective radius” (1/20 of the image height), which forms an imaginary disk
centering at the marked point. In this way, we can use a relatively small
amount of points to cover the image regions with flaws. Lastly, annotators
label the misaligned keywords and the four types of scores for plausibility,
image-text alignment, aesthetic, and overall quality, respectively, on a
5-point Likert scale. Detailed definitions of image implausibility/artifact
and misalignment can be found in the supplementary materials.

We designed a web UI to facilitate data collection with the following
principles: 1) convenience for annotators to perform annotations, ideally
within a short time for an image-text pair and, 2) allowing annotators to
perform all annotations on the same UI, so that the fine-grained scores are
based on the annotated regions and keywords. To this end, we created the
interface as illustrated in Fig. 1. The main UI consists of an image displayed
on the left and a panel on the right, where the text prompt is shown at the
top of the panel. Annotators are asked to first click on the image to annotate
the artifact/implausible regions and misalignment regions, and then select the
misaligned keywords and the fine-grained scores on the right of the panel.

Figure 2: Histograms of the PaLI attributes of the images in the training set.
Figure 3: Histograms of the average scores of image-text pairs in the training
set. Figure 4: Counts of the samples with maximum differences of the scores in
the training set. Figure 5: Architecture of our rich feedback model. Our model
consists of two streams of computation: one vision and one text stream. We
perform self-attention on the ViT-outputted image tokens and the Text-embed
module-outputted text tokens to fuse the image and text information. The
vision tokens are reshaped into feature maps and mapped to heatmaps and
scores. The vision and text tokens are sent to a Transformer decoder to
generate a text sequence.

We created detailed annotation guidelines to instruct the annotators regarding
the annotation steps, interactions with the web UI, examples of different
types of implausibility, artifacts, and misalignment. All the annotators (27
in total) are trained with the annotation guidelines and calibrated, before
they perform the annotation in order to reduce annotation discrepancy and
improve quality. Our annotation took around 3,000 rater-hours in total. More
details about our data collection process are in the supplementary.

###  3.2 Human feedback consolidation

To improve the reliability of the collected human feedback on generated
images, each image-text pair is annotated by three annotators. We therefore
need to consolidate the multiple annotations for each sample. For the scores,
we simply average the scores from the multiple annotators for an image to
obtain the final score. For the misaligned keyword annotations, we perform
majority voting to get the final sequence of indicators of aligned/misaligned,
using the most frequent label for the keywords. For the point annotations, we
first convert them to heatmaps for each annotation, where each point is
converted to a disk region (as discussed in the last subsection) on the
heatmap, and then we compute the average heatmap across annotators. The
regions with clear implausibility are likely to be annotated by all annotators
and have a high value on the final average heatmap.

###  3.3 RichHF-18K: a dataset of rich human feedback

We select a subset of image-text pairs from the Pick-a-Pic dataset for data
annotation. Although our method is general and applicable to any generated
images, we choose the majority of our dataset to be photo-realistic images,
due to its importance and wider applications. Moreover, we also want to have
balanced categories across the images. To ensure balance, we utilized the PaLI
visual question answering (VQA) model [7] to extract some basic features from
the Pick-a-Pic data samples. Specifically, we asked the following questions
for each image-text pair in Pick-a-Pic. 1) Is the image photorealistic? 2)
Which category best describes the image? Choose one in ‘human’, ‘animal’,
‘object’, ‘indoor scene’, ‘outdoor scene’. PaLI’s answers to these two
questions are generally reliable under our manual inspection. We used the
answers to sample a diverse subset from Pick-a-Pic, resulting in 17K image-
text pairs. We randomly split the 17K samples into two subsets, a training set
with 16K samples and a validation set with 1K samples. The distribution of the
attributes of the 16K training samples is shown in Fig. 2. Additionally, we
collect rich human feedback on the unique prompts and their corresponding
images from the Pick-a-Pic test set as our test set. In total, we collected
rich human feedback on the 18K image-text pairs from Pick-a-Pic. Our
RichHF-18K dataset consists of 16K training, 1K validation, and 1K test
samples.

###  3.4 Data statistics of RichHF-18K

In this section, we summarize the statistics of the scores and perform the
annotator agreement analysis for the scores. We standardize the scores $s$
with the formula $(s-s_{\text{min}})/(s_{\text{max}}-s_{\text{min}})$
($s_{\text{max}}=5$ and $s_{\text{min}}=1$) so that the scores lie in the
range [0, 1].

The histogram plots of the scores are shown in Fig. 3. The distribution of the
scores is similar to a Gaussian distribution, while the plausibility and text-
image alignment scores have a slightly higher percentage of score 1.0. The
distribution of the collected scores ensures that we have a reasonable number
of negative and positive samples for training a good reward model.

To analyze the rating agreement among annotators for an image-text pair, we
calculate the maximum difference among the scores:
$\text{max}_{\text{diff}}=\max(\text{scores})-\min(\text{scores})$ where
scores are the three score labels for an image-text pair. We plot the counts
of the $\text{max}_{\text{diff}}$ for each score type in Fig. 4. We can see
that around 25% of the samples have perfect annotator agreement and around 85%
of the samples have good annotator agreement ($\text{max}_{\text{diff}}$ is
less than or equal to 0.25 after the standardization or 1 in the 5-point
Likert scale).

##  4 Predicting rich human feedback

###  4.1 Models

####  4.1.1 Architecture

The architecture of our model is shown in Fig. 5. We adopt a vision-language
model based on ViT [14] and T5X [40] models, inspired by the Spotlight model
architecture [33], but modifying both the model and pretraining datasets to
better suit our tasks. We use a self-attention module [49] among the
concatenated image tokens and text tokens, similar to PaLI [7], as our tasks
require bidirectional information propagation. The text information is
propagated to image tokens for text misalignment score and heatmap prediction,
while the vision information is propagated to text tokens for better vision-
aware text encoding to decode the text misalignment sequence. To pretrain the
model on more diverse images, we add the natural image captioning task on the
WebLI dataset [7] to the pretraining task mixture.

Specifically, the ViT takes the generated image as input and outputs image
tokens as high-level representations. The text prompt tokens are embedded into
dense vectors. The image tokens and embedded text tokens are concatenated and
encoded by the Transformer self-attention encoder in T5X. On top of the
encoded fused text and image tokens, we use three kinds of predictors to
predict different outputs. For heatmap prediction, the image tokens are
reshaped into a feature map and sent through convolution layers, deconvolution
layers, and sigmoid activation, and outputs implausibility and misalignment
heatmaps. For score prediction, the feature map is sent through convolution
layers, linear layers, and sigmoid activation, resulting in scalars as fine-
grained scores.

To predict the keyword misalignment sequence, the original prompt used to
generate the image is used as text input to the model. A modified prompt is
used as the prediction target for the T5X decoder. The modified prompt has a
special suffix (‘$\\_0$’) for each misaligned token, _e.g_., a yellow_0 cat if
the generated image contains a black cat and the word yellow is misaligned
with the image. During evaluation, we can extract the misaligned keywords
using the special suffix.

####  4.1.2 Model variants

We explore two model variants for the prediction heads of the heatmaps and
scores.

##### Multi-head

A straightforward way to predict multiple heatmaps and scores is to use
multiple prediction heads, with one head for each score and heatmap type. This
will require seven prediction heads in total.

##### Augmented prompt

Another approach is to use a single head for each prediction type, _i.e_.,
three heads for heatmap, score, and misalignment sequence, respectively. To
inform the model of the fine-grained heatmap or score type, we augment the
prompt with the output type. More specifically, we prepend a task string
(_e.g_., ‘implausibility heatmap’) to the prompt for each particular task of
one example and use the corresponding label as the training target. During
inference, by augmenting the prompt with the corresponding task string, the
single heatmap (score) head can predict different heatmaps (scores). As we
show in the experiments, this augmented prompt approach can create task-
specific vision feature maps and text encodings, which performs significantly
better in some of the tasks.

####  4.1.3 Model optimization

We train the model with a pixel-wise mean squared error (MSE) loss for the
heatmap prediction, and MSE loss for the score prediction. For misalignment
sequence prediction, the model is trained with teacher-forcing cross-entropy
loss. The final loss function is the weighted combination of the heatmap MSE
loss, score MSE loss, and the sequence teacher-forcing cross-entropy loss.

###  4.2 Experiments

(a) Image

(b) GT

(c) Our model

(d) ResNet-50

Figure 6: Examples of implausibility heatmaps

(a) Image

(b) GT

(c) Our model

(d) CLIP gradient

Figure 7: Examples of misalignment heatmaps. Prompt: A snake on a mushroom. | Plausibility | Aesthetics | Text-image Alignment | Overall
---|---|---|---|---
| PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC
ResNet-50 | 0.495 | 0.487 | 0.370 | 0.363 | 0.108 | 0.119 | 0.337 | 0.308
CLIP | $-$ | $-$ | $-$ | $-$ | 0.185 | 0.130 | $-$ | $-$
Our Model (multi-head) | 0.666 | 0.654 | 0.605 | 0.591 | 0.487 | 0.500 | 0.582 | 0.561
Our Model (augmented prompt) | 0.693 | 0.681 | 0.600 | 0.589 | 0.474 | 0.496 | 0.580 | 0.562
Table 1: Score prediction results on the test set. | All data | $GT=0$ | $GT>0$
---|---|---|---
| MSE $\downarrow$ | MSE $\downarrow$ | CC $\uparrow$ | KLD $\downarrow$ | SIM $\uparrow$ | NSS $\uparrow$ | AUC-Judd $\uparrow$
ResNet-50 | 0.00996 | 0.00093 | 0.506 | 1.669 | 0.338 | 2.924 | 0.909
Ours (multi-head) | 0.01216 | 0.00141 | 0.425 | 1.971 | 0.302 | 2.330 | 0.877
Ours (augmented prompt) | 0.00920 | 0.00095 | 0.556 | 1.652 | 0.409 | 3.085 | 0.913
Table 2: Implausibility heatmap prediction results on the test set. $GT=0$ refers to empty implausibility heatmap, _i.e_., no artifacts/implausibility, for ground truth. $GT>0$ refers to heatmaps with artifacts/implausibility, , for ground truth. | All data | $GT=0$ | $GT>0$
---|---|---|---
| MSE $\downarrow$ | MSE $\downarrow$ | CC $\uparrow$ | KLD $\downarrow$ | SIM $\uparrow$ | NSS $\uparrow$ | AUC-Judd $\uparrow$
CLIP gradient | 0.00817 | 0.00551 | 0.015 | 3.844 | 0.041 | 0.143 | 0.643
Our Model (multi-head) | 0.00303 | 0.00015 | 0.206 | 2.932 | 0.093 | 1.335 | 0.838
Our Model (augmented prompt) | 0.00304 | 0.00006 | 0.212 | 2.933 | 0.106 | 1.411 | 0.841
Table 3: Text misalignment heatmap prediction results on the test set. $GT=0$ refers to empty misalignment heatmap, _i.e_., no misalignment, for ground truth. $GT>0$ refers to heatmaps with misalignment, for ground truth. | Precision | Recall | F1 Score
---|---|---|---
Multi-head | 62.9 | 33.0 | 43.3
Augmented prompt | 61.3 | 34.1 | 43.9
Table 4: Text misalignment prediction results on the test set.

(a) Prompt: gamer playing league of legends at night.

Plausibility score.
GT: 0.333, Our model: 0.410
Overall score.
GT: 0.417, Our model: 0.457

(b) Prompt: An endless wavy ocean under a colorful night sky artistic painting
pastel.
Plausibility score.
GT: 1.0, Our model: 0.979
Overall score.
GT 1.0, Our model: 0.848

(c) Prompt: Mechanical bee flying in nature electronics motors wires buttons
lcd.
Text-image alignment score.
GT: 0.583, Our model: 0.408
Aesthetics score.
GT: 0.75, Our model: 0.722

(d) Prompt: anime fortnite character.

Text-image alignment score.
GT: 1.0, Our model: 0.897
Aesthetics score.
GT: 0.75, Our model: 0.713

Figure 8: Examples of ratings. “GT” is the ground-truth score (average score
from three annotators).

(a) Muse [6] before finetuning

(b) Muse [6] after finetuning

(c) LD [41] without guidance

(d) LD [41] after aesthetic guidance

Figure 9: Examples illustrating the impact of RAHF on generative models.
(a-b): Muse [6] generated images before and after finetuning with examples
filtered by plausibility scores, prompt: A cat sleeping on the ground using a
shoe as a pillow. (c-d): Results without and with aesthetic score used as
Classifier Guidance [2] on Latent Diffusion (LD) [41], prompt: a macro lens
closeup of a paperclip.

(a) Prompt: a baseball with the parthenon on its cover, sitting on the
pitcher’s mound

(b) Prompt: A photograph of a beautiful, modern house that is located in a
quiet neighborhood. The house is made of brick and has a large front porch. It
has a manicured lawn and a large backyard.

Figure 10: Region inpainting with Muse [6] generative model. From left to
right, the 4 figures are: original images with artifacts from Muse, predicted
implausibility heatmaps from our model, masks by processing (thresholding,
dilating) the heatmaps, and new images from Muse region inpainting with the
mask, respectively.

Preference | $\gg$ | $>$ | $\approx$ | $<$ | $\ll$
---|---|---|---|---|---
Percentage | 21.5% | 30.33% | 31.33% | 12.67% | 4.17%

Table 5: Human Evaluation Results: Finetuned Muse vs original Muse model
preference: Percentage of examples where finetuned Muse is significantly
better ($\gg$), slightly better ($>$), about the same ($\approx$), slightly
worse ($<$), significantly worse ($\ll$) than original Muse. Data was
collected from 6 individuals in a randomized survey.

####  4.2.1 Experimental setup

Our model is trained on the 16K RichHF-18K training samples, and the
hyperparameters were tuned using the model performance on the 1K RichHF-18K
validation set. The hyperparameters setup can be found in supplementary
material.

##### Evaluation metrics

For score prediction tasks, we report the Pearson linear correlation
coefficient (PLCC) and Spearman rank correlation coefficient (SRCC), which are
typical evaluation metrics for score predictions [28]. For heatmap prediction
tasks, a straightforward way to evaluate the results would be to borrow
standard saliency heatmap evaluation metrics such as NSS/KLD [5]. However,
these metrics cannot be applied directly in our case as all these metrics
assume the ground truth heatmap is not empty; yet in our case, empty ground
truth is possible (_e.g_., for artifact/implausibility heatmap, it means the
image does not have any artifact/implausibility). As such, we report MSE on
all samples and on those with empty ground truth, respectively, and report
saliency heatmap evaluation metrics like NSS/KLD/AUC-Judd/SIM/CC [5] for the
samples with non-empty ground truth. For the misaligned keyword sequence
prediction, we adopt the token-level precision, recall, and F1-score.
Specifically, the precision/recall/F1 scores are computed for the misaligned
keywords over all the samples.

##### Baselines

For comparison, we finetune two ResNet-50 models [18], with multiple fully
connected layers and deconvolution heads to predict the scores and heatmaps
accordingly. For the misalignment score, we use the CLIP score as a baseline.
For misalignment heatmap prediction, we use CLIP gradient [45] map as a
baseline.

####  4.2.2 Prediction result on RichHF-18K test set

##### Quantitative analysis

The experimental results of our model prediction on the four fine-grained
scores, the implausibility heatmap, misalignment heatmap, and misalignment
keyword sequence on our RichHF-18K test set are presented in Tab. 1, Tab. 2,
Tab. 3, and Tab. 4 respectively.

In both Tab. 1 and Tab. 3, the two variants of our proposed model both
significantly outperform ResNet-50 (or CLIP for text-image alignment score).
Yet, in Tab. 2, the multi-head version of our model performs worse than
ResNet-50, but our augmented prompt version outperforms ResNet-50. The main
reason might be that in the multi-head version, without augmenting the
prediction task in the prompt, the same prompt is used for all the seven
prediction tasks, and hence the feature maps and text tokens will be the same
for all tasks. It might not be easy to find a good tradeoff among these tasks,
and hence the performance of some tasks such as artifact/implausibility
heatmap became worse. However, after augmenting the prediction task into a
prompt, the feature map and text token can be adapted to each particular task
with better results. Additionally, we note that misalignment heatmap
prediction generally has worse results than artifact/implausibility heatmap
prediction, possibly because misalignment regions are less well-defined, and
the annotations may therefore be noisier.

##### Qualitative examples

We show some example predictions from our model for implausibility heatmap
(Fig. 6), where our model identifies the regions with artifact/implausibility,
and for misalignment heatmap (Fig. 7), where our model identifies the objects
that don’t correspond to the prompt. Fig. 8 shows some example images and
their ground-truth and predicted scores. More examples are in the
supplementary material.

##  5 Learning from rich human feedback

In this section, we investigate whether the predicted rich human feedback
(_e.g_., scores and heatmaps) can be used to improve image generation. To
ensure that the gains from our RAHF model generalize across generative model
families, we use Muse [6] as our target model to improve, which is based on a
masked transformer architecture and thus different from the Stable Diffusion
model variants in our RichHF-18K dataset.

##### Finetuning generative models with predicted scores

We first illustrate that finetuning with RAHF scores can improve Muse. First,
we generate eight images for each of the $12564$ prompts (the prompt set is
created via PaLM 2 [11, 1] with some seed prompts) using the pre-trained Muse
model. We predict RAHF scores for each image, and if the highest score for the
images from each prompt is above a fixed threshold, it will be selected as
part of our finetuning dataset. The Muse model is then finetuned with this
dataset. This approach could be viewed as a simplified version of Direct
Preference Optimization [15].

In Fig. 9 (a)-(b), we show one example of finetuning Muse with our predicted
plausibility score (threshold=0.8). To quantify the gain from Muse finetuning,
we used 100 new prompts to generate images, and asked 6 annotators to perform
side-by-side comparisons (for plausibility) between two images from the
original Muse and the fine-tuned Muse respectively. The annotators choose from
five possible responses (image A is significantly/slightly better than image
B, about the same, image B is slightly/significantly better than image A),
without knowledge of which model is used to generate the image A/B. The
results in Tab. 5 demonstrate that the finetuned Muse with RAHF plausibility
scores processes significantly fewer artifacts/implausibility than the
original Muse.

Moreover, in Fig. 9 (c)-(d), we show an example of using the RAHF aesthetic
score as Classifier Guidance to the Latent Diffusion model [41], similar to
the approach in Bansal et al. [2], demonstrating that each of the fine-grained
scores can improve different aspects of the generative model/results.

##### Region inpainting with predicted heatmaps and scores

We demonstrate that our model’s predicted heatmaps and scores can be used to
perform region inpainting to improve the quality of generated images. For each
image, we first predict implausibility heatmaps, then create a mask by
processing the heatmap (using thresholding and dilating). Muse inpainting [6]
is applied within the masked region to generate new images that match the text
prompt. Multiple images are generated, and the final image is chosen by the
highest predicted plausibility score by our RAHF.

In Fig. 10, we show several inpainting results with our predicted
implausibility heatmaps and plausibility scores. As shown, more plausible
images with fewer artifacts are generated after inpainting. Again, this shows
that our RAHF generalizes well to images from a generative model very
different from the ones whose images are used to train RAHF. More details and
examples can be found in the supplementary material.

##  6 Conclusions and limitations

In this work, we contributed RichHF-18K, the first rich human feedback dataset
for image generation. We designed and trained a multimodal Transformer to
predict the rich human feedback, and demonstrated some instances to improve
image generation with our rich human feedback.

While some of our results are quite exciting and promising, there are several
limitations to our work. First, the model performance on the misalignment
heatmap is worse than that on the implausibility heatmaps, possibly due to the
noise in the misalignment heatmap. It is somewhat ambiguous how to label some
misalignment cases such as absent objects on the image. Improving the
misalignment label quality is one of the future directions. Second, it would
be helpful to collect more data on generative models beyond Pick-a-Pic (Stable
Diffusion) and investigate their effect on the RAHF models. Moreover, while we
present three promising ways to leverage our model to improve T2I generation,
there is a myriad of other ways to utilize rich human feedback that can be
explored, _e.g_., how to use the predicted heatmaps or scores as a reward
signal to finetune generative models with reinforcement learning, and how to
use the predicted heatmaps as a weighting map, or how to use the predicted
misaligned sequences in learning from human feedback to help improve image
generation, etc. We hope RichHF-18K and our initial models inspire quests to
investigate these research directions in future work.

## References

  * Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Bansal et al. [2023] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models, 2023.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18392–18402, 2023.
  * Bylinskii et al. [2018] Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand.  What do different evaluation metrics tell us about saliency models?  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2018.
  * Chang et al. [2023] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al.  Muse: Text-to-image generation via masked generative transformers.  _arXiv preprint arXiv:2301.00704_ , 2023.
  * Chen et al. [2022] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model, 2022.
  * Cho et al. [2023a] Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang.  Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-Image Generation.  In _arXiv:2310.18235_ , 2023a.
  * Cho et al. [2023b] Jaemin Cho, Abhay Zala, and Mohit Bansal.  Visual programming for text-to-image generation and evaluation.  In _NeurIPS_ , 2023b.
  * Cho et al. [2023c] Jaemin Cho, Abhay Zala, and Mohit Bansal.  Visual programming for text-to-image generation and evaluation.  In _NeurIPS_ , 2023c.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Croitoru et al. [2023] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Fan et al. [2023] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.  Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gu et al. [2023] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Josh Susskind, and Navdeep Jaitly.  Matryoshka diffusion models, 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hessel et al. [2022] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  In _EMNLP_ , 2022.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Higgins et al. [2016] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  In _International conference on learning representations_ , 2016.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022.
  * Hu et al. [2023] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith.  Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.  In _ICCV_ , 2023.
  * Huang et al. [2023] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu.  T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.  _arXiv preprint arXiv:2307.06350_ , 2023.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _CVPR_ , 2023.
  * Ke et al. [2021] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang.  Musiq: Multi-scale image quality transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kirstain et al. [2023] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  _arXiv preprint arXiv:2305.01569_ , 2023.
  * Lee et al. [2023] Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang.  Holistic evaluation of text-to-image models.  In _Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.
  * Li et al. [2019] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr.  Controllable text-to-image generation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Li and Li [2023] Gang Li and Yang Li.  Spotlight: Mobile ui understanding using vision-language models with a focus.  In _International Conference on Learning Representations_ , 2023.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Ni et al. [2023] Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, and Martin Renqiang Min.  Conditional image-to-video generation with latent flow diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18444–18455, 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Otani et al. [2023] Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin’ichi Satoh.  Toward verifiable and reproducible human evaluation for text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14277–14286, 2023.
  * Qiao et al. [2019] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao.  Mirrorgan: Learning text-to-image generation by redescription.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1505–1514, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Roberts et al. [2022] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo.  Scaling up models and data with t5x and seqio.  _arXiv preprint arXiv:2203.17189_ , 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  _Advances in neural information processing systems_ , 29, 2016.
  * Sheynin et al. [2023] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman.  Emu edit: Precise image editing via recognition and generation tasks, 2023.
  * Simonyan et al. [2014] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings_ , 2014.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pages 2256–2265. PMLR, 2015.
  * Tao et al. [2022] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu.  Df-gan: A simple and effective baseline for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16515–16525, 2022.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023] S. Wang, C. Saharia, C. Montgomery, J. Pont-Tuset, S. Noy, S. Pellegrini, Y. Onoe, S. Laszlo, D. J. Fleet, R. Soricut, J. Baldridge, M. Norouzi, P. Anderson, and W. Chan.  Imagen editor and editbench: Advancing and evaluating text-guided image inpainting.  In _CVPR_ , 2023.
  * Wu et al. [2023a] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li.  Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.  _arXiv preprint arXiv:2306.09341_ , 2023a.
  * Wu et al. [2023b] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li.  Human preference score: Better aligning text-to-image models with human preference.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2096–2105, 2023b.
  * Xing et al. [2023] Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang.  A survey on video diffusion models.  _arXiv preprint arXiv:2310.10647_ , 2023.
  * Xu et al. [2023] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  In _Neural Information Processing Systems_ , 2023.
  * Xu et al. [2018] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1316–1324, 2018.
  * Yang et al. [2022] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  _ACM Computing Surveys_ , 2022.
  * Yarom et al. [2023] Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor.  What you see is what you read? improving text-image alignment evaluation, 2023.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2(3):5, 2022.
  * Zhang et al. [2023a] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon.  Text-to-image diffusion model in generative ai: A survey.  _arXiv preprint arXiv:2303.07909_ , 2023a.
  * Zhang et al. [2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 5907–5915, 2017.
  * Zhang et al. [2023b] Lingzhi Zhang, Zhengjie Xu, Connelly Barnes, Yuqian Zhou, Qing Liu, He Zhang, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, and Jianbo Shi.  Perceptual artifacts localization for image synthesis tasks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7579–7590, 2023b.
  * Zhu et al. [2019] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5802–5810, 2019.

Supplementary Material

##  7 Ethical conduct

Our data collection has been approved by an Institutional Review Board.

##  8 Data collection details

###  8.1 Image artifacts/implausibility definitions

  1. 1.

Distorted human/animal bodies/faces

    1. (a)

Distorted/combined faces and/or body parts (unless specified in the text
caption)

    2. (b)

Missing body parts (unless specified in the text caption)

    3. (c)

Additional body parts (unless specified in the text caption)

  2. 2.

Distorted objects (non human/animal)

    1. (a)

Distorted objects (e.g., furniture, vehicles, buildings) (unless specified in
the text caption)

  3. 3.

Distorted/Nonsensical text

    1. (a)

Text that is distorted, nonsensical, or misspelled (unless specified in the
text caption)

  4. 4.

Nonsensical Representations

    1. (a)

Representations that are unrealistic/nonsensical (unless specified in the text
caption), or difficult to understand

  5. 5.

Excessive blurriness/lack of sharpness

    1. (a)

The image contains excessive blurriness or quality that detracts from the
overall image (focus on one part of the image is OK)

    2. (b)

The image contains a lack of definition/sharpness that detracts from the
overall image

  6. 6.

Any other artifacts or implausibility not covered above

###  8.2 Text-image misalignment definitions and what-to-do

Since we require the annotators to mark the misaligned words in the text
prompt, we differentiate this part from Sec. 8.1 by including a what-to-do
under each definition.

  1. 1.

Something is missing: a human/animal/object specified in the text caption is
missing in the image

     * •

Click on that word of the human/animal/object in the text

  2. 2.

Incorrect attributes: an attribute (e.g., color) of an object specified in the
text is incorrect in the image

     * •

Click on that word of the attribute in the text and click on the region of the
object on the image

  3. 3.

Incorrect actions: an action specified in the text caption is not represented
in the image

     * •

Click on that word of the action in the text and click on the region of the
wrong actions on the image

  4. 4.

Incorrect numbers: counts of humans/animals/objects in the image do not match
those specified in the text

     * •

Click on the number in the text

  5. 5.

Incorrect position: the spatial position of two entities in the image does not
match that specified in the text

     * •

Click on the word of the position in the text

  6. 6.

Other: any other inconsistency between text and image

     * •

Click on the word of the inconsistency in the text

###  8.3 Additional details

##### Annotation guideline

To ensure the annotators understand the above definitions, we provide 4-10
examples for each definition of the annotation terms in the guideline. All of
our annotators can read English and thus understand the text prompts. In some
of the prompts, there are concepts or person names in the text prompts that
are uncommon and may cause confusion to the annotators. Therefore, we instruct
the annotators to do a quick search on the internet regarding any unfamiliar
concepts in the text prompts and skip samples with confusing prompts full of
strange concepts.

##### Dataset size

Since the Pick-a-Pic v1 dataset contains some images and/or prompts that are
inappropriate (_e.g_., containing nudity), we ask the annotators to mark these
images with a special flag and skip the annotation. We filter out these
inappropriate images and/or prompts during data post-processing. For this
reason, the total number of images in our final training set is around 300
short of 16K.

###  8.4 Discussions and limitations

We choose points over bounding boxes in our region annotation because we find
that points are much faster to mark and can have a reasonable coverage over
image regions with various shapes when we specify an effective radius for each
point as discussed in the main paper.

As a limitation in our region/heatmap annotations, we notice there is an over-
annotation issue in the artifacts/implausibility region annotation.
Specifically, our annotators tend to annotate more human faces and hands on
the images than necessary. One reason is that human faces and hands in the
Pick-a-Pic dataset indeed have more artifacts/implausibility than other parts.
Moreover, the annotators, as humans, may naturally pay more attention to human
faces and hands, resulting in over-annotation of these parts. Nevertheless,
the over-annotation issue is minor in our final dataset, as we strive to
provide feedback to the annotators frequently to make them less nitpicking
about human faces and hands.

Another limitation is the diversity of the subjects in the prompts/images. The
Pick-a-Pic dataset (and many others) is predominantly full of human, dog, and
cat subjects. For this reason, it is challenging to find a very diverse
dataset for annotation. We strive to make the subjects more diverse by using
balanced categories as indicated by the PaLI attributes (as in Fig. 2). We
didn’t choose more fine-grained categories for PaLI to test as there would be
an endless list of subjects we could consider. Therefore, we leave the goal of
annotating more diverse images/prompts in future works.

##  9 Experimental details

##### Hyperparameters

The main model components consist of a ViT B16 encoder for image encoding, a
T5 base encoder for mixing image and text tokens, and three predictors for
score, heatmap, and text misalignment, respectively. The ViT B16 encoder uses
a 16x16 patch size, 12 layers with 12 heads with a hidden dimension of 768,
wherein the MLP has a hidden dimension of 3072. The T5 base encoder uses 12
layers with 12 heads with a hidden dimension of 768, wherein the MLP has a
hidden dimension of 2048. The score predictor consists of four convolutional
layers with layer norm and ReLU activation, and the filter size, kernel size,
and strides are $[768,384,128,64],[2,2,2,2],[1,1,1,1]$, respectively. Three
dense layers of output sizes 2048, 1024, and 1, respectively, are used to
generate a scalar with ReLU activation for the first two layers and sigmoid
for the last. The heatmap predictor consists of two convolution layers with
filter size, kernel size, and stride as $[768,384],[3,3],[1,1]$, respectively.
It then uses four de-convolution layers to up-sample to the required output
size, with the filter size, kernel size, and stride as
$[768,384,384,192],[3,3,3,3],[2,2,2,2]$, respectively. Each de-convolution
layer is with two read-out convolution layers of kernel size 3 and stride 1.
Layer norm and ReLU are used for each layer. In the end, two read-out
convolution layers and a final sigmoid activation are used to generate the
heatmap prediction. The text predictor is implemented using a T5 base decoder
with 12 layers of 12 heads, MLP dimension 2048, and hidden dimension 768. The
output token length is 64.

We train the model on the datasets with a batch size of 256 for 20K
iterations. We utilize the AdamW optimizer with a base learning rate of 0.015.
We linearly increase the learning rate from 0 to the base learning rate in the
first 2000 iterations, and then decrease the learning rate with a reciprocal
square root scheduler w.r.t the number of iterations.

##### Image augmentations

For each image, we randomly crop it by sampling a bounding box with 80%-100%
width and 80%-100% height. The cropping is applied by 50% chance and otherwise
the original image is used. Note that we also crop the corresponding part of
the implausibility heatmap and misalignment heatmap to match the cropped
image. We then create an augmented version of the image by applying several
random augmentations including random brightness (max delta 0.05), random
contrast (random contrast factor between 0.8 and 1), random hue (max delta
0.025), random saturation (random saturation factor between 0.8 and 1) and
random jpeg noise (jpeg quality between 70 and 100). By 10% chance the
augmented version is used instead of the original image. We convert the image
to grayscale by 10% probability as the final image.

##  10 Additional qualitative examples

Fig. 11 provides more examples of artifacts/implausibility heatmaps. We can
see that our RAHF model can more accurately locate the positions of
artifacts/implausibility on various subjects such as human hands, animals,
vehicles, and concept arts.

(a) Image

(b) GT

(c) Our model

(d) ResNet-50

Figure 11: More examples of implausibility heatmaps

Fig. 12 provides more examples of misalignment heatmaps. We can see that our
RAHF model can more accurately locate the positions of misalignment on various
subjects such as animals, objects, and different outdoor scenes. For example,
our model can identify the subtle difference between the real handlebar of a
Segway and the one depicted in the image.

(a) Prompt: Photo of a cat eating a burger like a person

(b) Prompt: An abandoned Segway in the forest

(c) Prompt: inflatable pie floating down a river

(d) Prompt: A Red Pigeon Sat on a Branch Reflecting on Existence

Image GT Our model CLIP gradient

Figure 12: More examples of misalignment heatmaps.

Fig. 13 provides more examples of score predictions, where our RAHF model
predicts scores that are quite close to the ground truth score from human
evaluation.

Fig. 14 provides examples for the misaligned keywords prediction, which shows
that our RAHF model can predict the majority of the misaligned keywords marked
by human annotators.

Fig. 15 provides more examples of the comparison before and after finetuning
Muse with examples selected based on the predicted scores by our RAHF model
and examples of using RAHF model predicted overall score as Classifier
Guidance. We can see enhanced image quality of the generation from the
finetuned Muse model and the Latent Diffusion model, which highlights the
potential of improving T2I generation with our reward model.

Fig. 16 provides more examples of Muse inpainting with the predicted masks
(converted from heatmaps) by our RAHF model, where the inpainted regions are
significantly improved in plausibility.

(a) Prompt: Computer science students fighting with computer keyboards.
Plausibility score.
GT: 0.25, Our model: 0.236
Overall score.
GT: 0.5, Our model: 0.341

(b) Prompt: meditation under a rainbow during a thunderstorm.

Plausibility score.
GT: 0.5, Our model: 0.448
Overall score.
GT: 0.583, Our model: 0.505

(c) Prompt: A needle-felted palm tree.

Text-image alignment score.
GT: 0.75, Our model: 0.988
Aesthetics score.
GT: 0.75, Our model: 0.961

(d) Prompt: Renault Capture on a beach.

Text-image alignment score.
GT: 1.0, Our model: 0.877
Aesthetics score.
GT: 0.75, Our model: 0.720

(e) Prompt: all the letters of the greek alphabet.
Plausibility score.
GT: 0.167, Our model: 0.331
Overall score.
GT: 0.250, Our model: 0.447

(f) Prompt: a kittens in box.

Plausibility score.
GT: 0.75, Our model: 0.851
Overall score.
GT: 0.75, Our model: 0.855

(g) Prompt: monkey climbing a skyscraper.
Text-image alignment score.
GT: 0.833, Our model: 0.536
Aesthetics score.
GT: 0.583, Our model: 0.467

(h) Prompt: bread.

Text-image alignment score.
GT: 1.0, Our model: 0.975
Aesthetics score.
GT: 1.0, Our model: 0.984

Figure 13: Examples of ratings. “GT” is the ground-truth score (average score
from three annotators).

(a) The prompt is: Two cats watering roses in a greenhouse. The ground truth
labels two, watering, greenhouse as misaligned keywords and our model predicts
two, greenhouse as misaligned keywords. This text will be invisible, just to
occupy some spaces.

(b) The prompt is: A close up photograph of a fat orange cat with lasagna in
its mouth, shot on Leica M6. The ground truth labels fat, lasagna, Leica, M6
as misaligned keywords and our model predicts lasagna, Leica, M6 as misaligned
keywords.

Figure 14: Examples for text misalignment prediction.

(a) Muse before finetuning

(b) Muse after finetuning

(c) Muse before finetuning

(d) Muse after finetuning

(e) Muse before finetuning

(f) Muse after finetuning

(g) LD without guidance

(h) LD with overall guidance

Figure 15: More examples illustrating the impact of RAHF on generative models.
(a-f): Muse [6] generated images before and after finetuning with examples
filtered by plausibility scores. Prompt: (a-b): Three zebras are standing
together in a line. (c-d): An elephant scratching it’s neck on a post. (e-f):
Apples, lemons, grapes, oranges and other fruits in crates. (g-h): Results
without and with overall score used as Classifier Guidance [2] on Latent
Diffusion (LD) [41], prompt: Kitten sushi stained glass window sunset fog.

(a) Prompt: A 3d printed sculpture of a cat made of iron and plastic, with
arabic translation and ic gradients.

(b) Prompt: A 1960s slide out camper with a blonde, white and red color scheme

Figure 16: Region inpainting with Muse [6] generative model. From left to
right, the 4 figures are: original images with artifacts from Muse, predicted
implausibility heatmaps from our model, masks by processing (thresholding,
dilating) the heatmaps, and new images from Muse region inpainting with the
mask, respectively.