v#1

# Attention as Annotation: Generating Images and Pseudo-masks for Weakly
Supervised Semantic Segmentation with Diffusion

Ryota Yoshihashi, Yuya Otsuka, Kenji Doi, Tomohiro Tanaka

###### Abstract

Although recent advancements in diffusion models enabled high-fidelity and
diverse image generation, training of discriminative models largely depends on
collections of massive real images and their manual annotation. Here, we
present a training method for semantic segmentation that neither relies on
real images nor manual annotation. The proposed method attn2mask utilizes
images generated by a text-to-image diffusion model in combination with its
internal text-to-image cross-attention as supervisory pseudo-masks. Since the
text-to-image generator is trained with image-caption pairs but without pixel-
wise labels, attn2mask can be regarded as a weakly supervised segmentation
method overall. Experiments show that attn2mask achieves promising results in
PASCAL VOC for not using real training data for segmentation at all, and it is
also useful to scale up segmentation to a more-class scenario, i.e., ImageNet
segmentation. It also shows adaptation ability with LoRA-based fine-tuning,
which enables the transfer to a distant domain i.e., Cityscapes.

## Introduction

Generative modeling in images is a task to estimate distributions of natural
images and enable the sampling of new images from them. Although the
generation of natural-looking images has been a long-standing challenge,
recent diffusion models (Ho, Jain, and Abbeel 2020) brought rapid advancement
for stable training of high-fidelity image generative models. Particularly,
diffusion models trained with large-scale web-crawled image-text pairs
(Schuhmann et al. 2021) enabled highly controllable text-to-image synthesis
(text2img) (Ramesh et al. 2021; Rombach et al. 2022), which can reflect
prompting by free-form texts into the generated contents.

In contrast, the training of image-recognition models largely depends on
supervised learning using massive manual annotations on image collections
(Deng et al. 2009). An extreme case is semantic segmentation, where we need
pixel-wise class labeling in each training image to exploit the full
supervision. Weakly supervised learning (WS), a methodology to utilize coarser
signals such as boxes or image-level labels (Oquab et al. 2015), has been
extensively studied in segmentation to mitigate this labor. Although recent WS
methods have achieved preferable segmentation qualities in the benchmarks
(Everingham et al. 2015), complex techniques are often needed to acquire
pseudo-masks with satisfactory quality, which limits the generality of the
methods. Furthermore, WS can not mitigate the necessity to gather images that
are annotatable and aligned to the deployment (or test) domain.

Here, our idea is to leverage the advanced generative models to synthesize
images and corresponding pseudo-labels jointly. Generative models’ generation
processes may contain semantics of the images generated by themselves, which
are reusable for training another recognition model. We propose attn2mask, a
semantic-segmentation training method that exploits synthesized images and
internal attention maps from a text2img diffusion model as supervisory
signals, as illustrated in Fig. 1. Attn2mask can be regarded as a WS method
since it only requires image-text pairs in training the generative model and
no pixel-annotated image. However, attn2mask’s difference from the existing WS
method is that it requires no real images in training its segmentation model,
which reduces image collection labor once the text2img model is trained (or
downloaded from public model hubs).

Figure 1: Proposed method attn2mask uses generated images paired with pseudo-
mask computed from generative attentions per prompt token for semantic
segmentation.

While we demonstrate that a simple combination of off-the-shell Stable
Diffusion (Rombach et al. 2022) and a supervised segmentation method (Chen et
al. 2018) work unexpectedly well, we further investigate approaches for
improvements from two viewpoints: 1) robust segmentation training and 2)
domain adaptation. The former is based on the observation that the diffusion-
based pseudo-masks are sometimes not very accurate due to leaking into
backgrounds or missing parts of objects, and we find that a real-image-based
WS training method (Rong et al. 2023) can mitigate harmful effects from this,
even in training with synthesized images. For the latter, we develop an
unsupervised domain adaptation technique using test-domain images (but not
labels) for our image-mask generator, apart from the purely synthesized-based
training. This adaptation opens the possibility of exploiting generic image
generators in new domains where the generators are not intended.

In experiments, we verify attn2mask’s effectiveness by achieving 58.3 mIoU in
the PASCAL VOC segmentation task, which is accurate for not using any real
images or annotation. Attn2mask is also applicable to a larger-scale scenario
in ImageNet-S 1k-class segmentation, where it performs sufficiently close to
the semi-supervised BigDatasetGAN generator. Furthermore, an adaptation of
Stable Diffusion using a small amount of target-domain images is shown to be
beneficial for distant-domain transfer for the Cityscapes driving images,
although it is a bit apart from the spirit of real-image-free training.

Our contributions are summarized as follows: 1) We show that off-she-shelf
Stable Diffusion without any modification or fine-tuning can generate images
and pseudo-masks that have sufficient quality to train semantic-segmentation
models. 2) We improve the synthetic training results by incorporating a robust
training method using co-training. 3) We present a domain-adaptative image-
and-pseudo-mask generation method with Stable Diffusion exploiting LoRA to
make attn2mask applicable to distant-domain tasks.

Figure 2: Overview of the proposed method attn2mask.

## Related work

### Segmentation with diffusion models

Several studies have tried to leverage the diffusion models’ excellent image-
generation performance in segmentation. There are two lines of research:
reusing diffusion U-Net for segmentation tasks and generating training image-
label pairs for segmentation. For the former line, it was shown that denoising
diffusion probabilistic models (DDPMs) can be used as pre-trained feature
extractors for semantic segmentation (Baranchuk et al. 2022). ODISE (Xu et al.
2023) similarly fine-tunes a text2img diffusion model for segmentation, but it
enables phrase-based segmentation by reusing a text encoder in the text2img
model. This usage of text2img models as visual-linguistic backbones can be
seen as a generative counterpart of contrastive visual-linguistic foundation
models, i.g., CLIP (Radford et al. 2021), which also can be repurposed for
segmentation tasks (Rao et al. 2022). However, all of the aforementioned
methods rely on manually annotated pixel labels for training.

For the latter line, Grounded Diffusion (Li et al. 2023) trains image-and-mask
generation models by leveraging a pre-trained Mask R-CNN with pixel labels. We
refer to this kind of approaches semi-supervised mask generation for their
dependency on annotated training images on at least partial classes. DiffuMask
(Wu et al. 2023) is closer to ours in the sense of weakly supervised mask
generation, where masks are automatically acquired via text2img generation
training. The main difference between DiffuMask and our attn2mask is that
DiffuMask focuses on single-object generation but attn2mask supports multi-
object image generation and is usable for training multi-class segmentation
models. Note that these studies (Li et al. 2023; Wu et al. 2023) are
contemporary to ours.

### Training with generated images

The idea of exploiting free annotations with synthesized images dates back
before the large-scale generative models: for example, computer-graphics-based
dataset creation using game engines was examined (Richter et al. 2016). More
primitively, even segmentation of mathematically drawn contours can help real
segmentation as a pretraining task (Kataoka et al. 2022). However, such
rendered images tended to have gaps from real ones, which needed supervised
retraining or unsupervised adaptation techniques (Hoffman et al. 2018) to
utilize them fully to achieve preferable real-environment performances. In
contrast, we exploit modern generative models’ high-fidelity generation
ability that narrows the gap without any adaptation in at least generic
settings such as PASCAL VOC. Prior to the diffusion models, GAN-based image-
and-mask generation was also tried (Li et al. 2022a) but they still required
manual annotation for partial classes to fine-tune the GANs for mask
generation. In addition to segmentation, classification (Sariyildiz et al.
2023) and unsupervised representation learning (Tian et al. 2023) with
generated images have been studied, and they report performances gradually
approaching to real-dataset-based counterparts, which is motivating for
studies in synthetic training including ours.

### Weakly supervised semantic segmentation

The mainstream of weakly supervised semantic segmentation (WSSS) approaches
exploits image-level class labels. Two-stage training that first trains an
image-level classifier and then re-trains another segmentation model using the
classifier’s class-activation maps (CAM) (Zhou et al. 2016) as weak labels is
the most straightforward but still well-used strategy. Among the two-stage
methods, existing studies attempted techniques for better CAM generation (Wang
et al. 2020; Jo and Yu 2021; Li et al. 2021; Chen et al. 2022) and robust
segmentation training against weak labels (Liu et al. 2022; Li et al. 2022b;
Rong et al. 2023). From the viewpoint of WSSS, we replace classifier-based
CAMs with generative attentions combined with generated images. In addition,
we incorporate the latest robust segmentation-training method (Rong et al.
2023) , which we confirm to be beneficial in our training scenario.

### Unsupervised segmentation and text-based segmentation

Recently, segmentation studies diverged in various training strategies. For
example, unsupervised semantic segmentation does not use any form of
supervision in the downstream segmentation training (Yu et al. 2021; Cho et
al. 2021). Our attn2mask may be seen as unsupervised in the sense that it is
annotation-free in the downstream domains, but it operates under the stricter
constraint of unavailability of downstream-domain (real) images than usual
unsupervised methods. Text-based segmentation exploits large-scale vision-
language pretraining (Rao et al. 2022; Zhou, Loy, and Dai 2022; Shin, Xie, and
Albanie 2022), which often transfers well in the downstream segmentation tasks
without fine-tuning. Attn2mask is related to these methods in the usage of
vision-language pretraining except for the difference between generative
(i.e., Stable Diffusion) and discriminative (i.e., CLIP) pretraining.

## Proposed method Attn2mask

An overview of attn2mask is shown in Fig.2. It consists of a) the image-and-
pseudo-mask generation module and b) the segmentation module. We also apply
post-processing to the pseudo-masks before passing them to the segmentation
model to multi-class binary labels.

### Image-and-pseudo-mask generation

To extract semantic regions from diffusion-synthesized images, we need to
compute the contributions of each word in the prompts on each location in
images. Fortunately, image-text cross-attention, popularly used in most
diffusion models including Stable Diffusion, is naturally available for this
purpose. Transformer-style attention modules can be written as follows:

$\displaystyle\text{Transformer}(Q,K,V)$ $\displaystyle=$ $\displaystyle
A(Q,K)V,$ (1) $\displaystyle A(Q,K)$
$\displaystyle=\text{Softmax}(\frac{QK^{T}}{\sqrt{\text{dim}(K)}}),$ (2)

where Q, K, and V are the query, key, and value, each of which is a sequence
of vectors, respectively. In using this as text-to-image cross-attention, Q is
computed from the image embedding $\bm{f}\in\mathbb{R}^{W\times H\times C}$
whose size is $W\times H$ and dimensionality is $C$, and K and V are computed
from the text embedding $\bm{s}\in\mathbb{R}^{L\times D}$ whose length is $L$
and dimensionality is $D$. This makes the attention maps’ dimensionality

$\displaystyle A(Q(\bm{s}),K(\bm{f}))$ $\displaystyle\in$
$\displaystyle\mathbb{R}^{W\times H\times L},$ (3)

where the element at $(x,y,t)$ can be interpreted as the amount of
contribution from the $t$-th text token to the location in image coordinate
$(x,y)$.

From the attention maps $A$, we extract channels that are relevant to the
classes of interest to use them as supervisory masks. Given a token sequence
of the input prompt, we denote the positions of the relevant tokens of class
$c$ by $\tau_{c}=[t_{c1},t_{c2},...,t_{cn_{c}}]$, allowing multiple
occurrences of the relevant tokens for a single class. The “relevance” of
words and classes is set manually by rule-based matching, for example, listing
synonyms for a class’s display name.

$\displaystyle
A_{c}=\frac{1}{n_{c}}\sum_{t\in\tau_{c}}{A(Q(\bm{s}),K(\bm{f}))_{t}}$
$\displaystyle\in$ $\displaystyle\mathbb{R}^{W\times H},$ (4)

Here, we denote the operation to extract $t$-th channels of the attention map
by $A(\cdot,\cdot)_{t}$.

We add one more probability map for the background as a class defined by

$\displaystyle A_{0}(x,y)$ $\displaystyle=$ $\displaystyle
1.0-\max_{i\in[1,N]}A_{i}(x,y)-\beta,$ (5)

which represents the absence of activations for any other classes. Here,
$\beta$ is a hyperparameter to control the strength of the background prior;
larger $\beta$ brings smaller portions of the background area in labeling
results. The motivation behind this special treatment for the backgrounds is
that they tend to be generated implicitly without concrete indication in the
prompts, which makes class-word-based aggregation in Eq. 4 difficult.

After attention maps $A_{0},A_{1},...,A_{N}$ are gathered, we apply
binarization to them to acquire multi-class discrete pseudo-labels. We use the
densely connected conditional random field (dCRF) (Krähenbühl and Koltun
2011), a non-learning-based labeling algorithm. This is also useful to improve
mask quality by considering the color similarity and local smoothness in
labeling. These generation processes are iterated for a given set of prompts,
and generated images and masks are stored in the disk as a dataset.

### Training segmentation models

Once a dataset of synthesized images and pseudo-labels is generated, any
supervised segmentation models are trainable with it in general. However,
naive training of supervised learners may suffer since the attention-based
pseudo-masks are not always perfect in terms of accuracy. Robust learners that
are tolerant of label noises would help. Then, we adopt a robust training
algorithm from BECO (Rong et al. 2023), a recent WS segmentation method as
shown in Fig. 3.

BECO is an adaptive co-training method exploiting reliability maps of the
pseudo-labels: It performs supervised learning with full trust in the pseudo-
labels within regions where the pseudo-labels are confident and performs co-
training-based consistency regularization in regions where the pseudo-labels
are unreliable. In the original work, BECO was trained with reliability maps
generated by processing the classifier’s confidence score maps. We do not use
any classifier and hence simply binarize the attention map to use as the
reliability map as follows:

$\displaystyle R(x,y)$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{ll}1&\text{if
}\max_{i\in[0,N]}A_{i}(x,y)\geq r\\\ 0&\text{otherwise},\end{array}\right.$
(8)

where $r$ is a hyperparameter for the reliability threshold. Other parts in
BECO except the reliability map are used unchanged.

Figure 3: Our fully supervised and weakly supervised training schemes.

### Domain adaptive image-mask generation

While image-mask generation with off-the-shelf Stable Diffusion generally
performs satisfactorily, in out-of-domain applications, it may suffer from the
domain gap between the generative model and the downstream tasks. We notice
that this is the case for example, in applying attn2mask to driving images in
Cityscapes (Cordts et al. 2016).

To make attn2mask applicable to new domains, we develop an adaptation method
using unlabeled training images. Here, we repurpose the instantiated
generation technique DreamBooth (Ruiz et al. 2023) for domain adaptation. For
a brief review, DreamBooth learns an instance token (often denoted by [V])
from a relatively small set of instance images. After fine-tuning with the
instance-image set, [V] is usable as a new word in prompts to indicate the
instance after training. Using DreamBooth, we instead learn a domain token; by
training [V] not for a specific instance but for general training images in a
new domain, Stable Diffusion can adapt to the domain, while keeping the
ability to reflect other prompt words to the image contents.

In implementation, we especially use DreamBooth-LoRA (Hugging Face 2023a),
which trains a low-rank adapter (Hu et al. 2022) that is a small sub-network
instead of fine-tuning the whole network. This enables fast and stable
training with a relatively small target-domain training set.

Table 1: Segmentation performances on the PASCAL VOC 2012 val set.

Method | bg | aeroplane | bicycle | bird | boat | bottle | bus | car | cat | chair | cow | table
---|---|---|---|---|---|---|---|---|---|---|---|---
Real-supervised | 93.6 | 88.0 | 40.0 | 84.5 | 66.2 | 63.2 | 89.3 | 84.5 | 87.6 | 32.4 | 74.3 | 37.6
Attn2mask (ours) | 83.0 | 79.5 | 33.2 | 55.7 | 56.6 | 56.5 | 76.3 | 58.3 | 70.1 | 22.2 | 57.0 | 34.6
Attn2mask+BECO (ours) | 86.6 | 77.2 | 29.0 | 79.0 | 72.0 | 56.2 | 84.8 | 68.0 | 80.9 | 24.9 | 76.3 | 46.0

Method | dog | horse | motorbike | person | plant | sheep | sofa | train | tv | mean
---|---|---|---|---|---|---|---|---|---|---
Real-supervised | 83.0 | 73.8 | 78.1 | 86.1 | 56.5 | 76.4 | 45.0 | 82.8 | 71.5 | 71.2
Attn2mask (ours) | 64.7 | 64.2 | 55.3 | 9.0 | 17.2 | 57.7 | 18.7 | 63.2 | 38.9 | 51.1
Attn2mask+BECO (ours) | 73.6 | 71.3 | 65.9 | 19.8 | 22.5 | 67.8 | 7.4 | 68.7 | 46.1 | 58.3

## Experiments

We evaluate semantic-segmentation models trained with attn2mask on three
publicly available datasets, namely, PASCAL VOC (Everingham et al. 2015),
ImageNet-S (Gao et al. 2022), and Cityscapes (Cordts et al. 2016). We conduct
the main experiments in PASCAL VOC, which has been the de facto standard in
semantic-segmentation evaluation. While fully supervised learning in VOC seems
close to saturation in terms of accuracy, it is challenging enough to tackle
without full annotation and real images. Additionally, we evaluate attn2mask’s
generalizability to 1) a larger-scale segmentation task with ImageNet-S, which
was built on ImageNet-1k (Deng et al. 2009) and 2) a more task-specific
scenario with Cityscapes, a dataset toward urban auto-driving.

Table 2: Comparisons with real-image-based WSSS methods Method | #Synth. imgs | #Real imgs | mIoU
---|---|---|---
Attn2mask (ours) | 2,000 | 0 | 41.2
| 10,000 | 0 | 46.5
| 50,000 | 0 | 49.7
| 168,679 | 0 | 51.1
Attn2mask+BECO | 10,000 | 0 | 51.7
(ours) | 168,679 | 0 | 58.3
Pseudo-mask (base) | 0 | 10,582 | 52.7
ReCAM (base) | 0 | 10,582 | 54.8
Pseudo-mask (full) | 0 | 10,582 | 70.0
ReCAM (full) | 0 | 10,582 | 70.5
BECO (full) | 0 | 10,582 | 70.9
Table 3: Results in the ImageNet-S FG/BG segmentation evaluated with mIoU. |  | #Images |
---|---|---|---
Method | #Annotations | 10k | 100k
Real-Supervised | 10k | 82.1 | –
BigDatasetGAN | 5k | 74.1 | 74.4
Attn2mask (ours) | 0 | 63.7 | 66.0
Table 4: Results in Cityscapes Method | Adapt. | Acc. | mIoU
---|---|---|---
Attn2mask (ours) |  | 63.7 | 13.2
Attn2mask-LoRA (ours) | $\checkmark$ | $\bm{69.7}$ | $\bm{17.1}$
MaskCLIP |  | 35.9 | 10.0
MDC |  | – | 7.0
PiCIE |  | – | 9.7
ReCo |  | $\bm{74.6}$ | $\bm{19.3}$
MDC | $\checkmark$ | 47.9 | 7.1
PiCIE | $\checkmark$ | 40.7 | 12.3
STEGO | $\checkmark$ | 73.2 | 21.0
ReCo+ | $\checkmark$ | $\bm{83.7}$ | $\bm{24.2}$

### Dataset generation

We used Stable Diffusion 1.4 (Hugging Face 2023b) trained on LAION-2B
(Schuhmann et al. 2021) without modification except in the domain-adaptation
settings. We generated 512 $\times$ 512-resolution images. The attention maps
had smaller resolutions but were resized to match the image resolution. Stable
Diffusion’s U-Net had multiple cross-attention layers and they were averaged
after resizing. We used DDPM (Ho, Jain, and Abbeel 2020) sampler with 100
timesteps, and attention maps were taken at the 50th step. Generation of a
sample took around eight seconds with an NVIDIA V100 GPU. Post-processing was
conducted using a Python reimplementation of dCRF (Healthcare Intelligence
Laboratory 2017) with the default hyperparameters.

The prompt set for VOC was created from the image descriptions from COCO
captions (Chen et al. 2015). We curated captions that include VOC class names
and their major synonyms that we manually listed up, which made 168,679
captions in total. The prompt set for ImageNet-S was created by a template “a
photo of ${class_name} with a background”. The variable ${class_name} was
randomly chosen from the multiple-defined synonyms per class. This made the
number of possible varieties of the prompts significantly fewer than the
needed samples per class (around 100 images/class) for our experiments, and
thus we increased the number of samples by using multiple random seeds per
prompt. The suffix “with a background” helps to reduce the generation of full-
foreground images, which is less useful for training.

For Cityscapes, we used a variable-length template “a photo of
${class_names[0]}, ${class_names[1]}, ... , ${class_names[n]} in cityscape by
a dashboard camera”, where class_names is a random variable-length list of
Cityscapes class names. A number of the class names in Cityscapes were not
household words that may not be reflected in the generation, and we replaced
“terrain” with “sand” and “vegetation” with “plant”. We generated 12k samples.
In the adaptation experiments, we used a modified template “a photo of
${class_names[0]}, ${class_names[1]}, ... , ${class_names[n]} in sks cityscape
by a dashboard camera”, where “sks” was the reserved word for the domain token
[V] in DreamBooth fine-tuning. More details of this fine-tuning are seen in
Appendix.

### Configurations of our model

First, we implemented the simplest version of our proposed method with Stable
Diffusion and a fully supervised segmentation model, which we refer to as
attn2mask. We used DeepLabv3+ (Chen et al. 2018) with the ResNet50 (He et al.
2016) backbone from the mmsegmentation (MMSegmentation Contributors 2020)
codebase. Additionally, we replaced the segmentation training with WS
training, which we refer to as attn2mask+BECO; we used the official training
codes of BECO (Rong et al. 2023), but it also used DeepLabv3+-ResNet50 and
trained models from these two codebases were fully compatible. Reliability
threshold $r$ was set to 0.5. Finally, we fine-tuned Stable Diffusion using
DreamBooth-LoRA (Hugging Face 2023a) for transfer to Cityscapes, which we
refer to as attn2mask-LoRA. We set the base learning rate to 0.01, weight
decay to 1e-4, and total batch size to 32 on 2-GPU machines. We trained real-
supervised models for comparisons with the same settings.

### Evaluation protocol

For PASCAL VOC and Cityscapes, we followed the official protocol for
supervised learners, which is applicable without modification for WS learners
including ours. We used the mean intersection-over-union (mIoU) metric. For
ImageNet-S, we derived a foreground/background (FG/BG) segmentation task by
extracting masks that have the same class labels as their whole images. This
was intended to replicate the evaluation protocol of BigDatasetGAN (Li et al.
2022a), in which the authors built their own ImageNet-based segmentation
benchmark but it remains unpublished.

Figure 4: Visualized attention maps corresponding to each of prompt tokens. Table 5: Variations of post-processing and hyperparameters Post-processing | mIoU
---|---
Thresholding $\theta=64$ | 43.5
Thresholding $\theta=128$ | 44.6
Otsu | 41.2
dCRF $\beta=0$ | 44.3
dCRF $\beta=0.2$ | $\bm{46.5}$
Table 6: Comparisons with null-text inversion (NTI) Method | mIoU
---|---
Attn2mask | $\bm{51.1}$
NTI on the train set and retraining | 50.6
NTI on the val set | 47.3

### Results

Table 1 summarizes the results in VOC. Attn2mask marked mIoU of 51.1%, which
is of course worse than the results by supervised training with the real
images but surprisingly effective for not using real images or manual
annotation at all. In particular, attn2mask performs closely to the supervised
counterpart in relatively easy classes such as aeroplane (88.0% v.s. 79.5%),
bus (88.0% v.s. 79.5%), and horse (73.8% v.s. 64.2%) or contrarily difficult
classes where the supervised models do not work well, such as bicycle (40.0%
v.s. 33.2%) or table (37.6% v.s. 34.6%). We also confirmed that the WS
training with BECO benefits our synthetic training, which gained + 7.2%-point
improvement on mIoU. However, we also acknowledge that there exists
significant degradation of IoU in some classes such as person, plant, or sofa
compared with the real-image training. We observed that such classes tend to
be generated as parts of backgrounds without being explicitly prompted. This
made noise for training these classes, which is the major limitation of our
synthetic training.

Table 2 shows comparisons between attn2mask and real-image-based weakly
supervised methods. For comparisons, we selected Pseudo-mask matters (Li et
al. 2021) and ReCAM (Chen et al. 2022), which were CAM-based and reported
baselines that simply learned CAM with their networks. Interestingly,
attn2mask with full synthetic images performed similarly to the baseline weak
learners. However, there were gaps between the full configurations of the
real-weak learners and attn2mask; the WS training gained +7.2 (51.1
$\rightarrow$ 58.3) mIoU to attn2mask but over 15 points in the real-weak
learners (52.7 $\rightarrow$ 70.0 and 54.8 $\rightarrow$ 70.5). These
phenomena can be explained by that 1) the domain gaps between real and
synthesized images are less dominant when the supervisory signals are weak,
and 2) but they become non-negligible as the segmentation models fit tighter
with the weakly supervised algorithms.

Table 3 shows the result in ImageNet-S FG/BG segmentation. We compared
attn2mask with the same network trained with data from BigDatasetGAN (Li et
al. 2022a). BigDatasetGAN is a semi-supervised generator that relies on manual
annotation of generated images to fine-tune GAN for mask generation. Attn2mask
performed competitively without using pixel-label annotations at all, which is
encouraging by showing the ability to freely increase classes within the range
Stable Diffusion can generate.

Figure 5: Examples of our training data generated for VOC. Figure 6: Examples
of images and attention maps before and after LoRA-based adaptations in
Cityscapes.

Table 4 shows the results in Cityscapes. In contrast to segmenting object
images such as VOC, holistic scene understanding in Cityscapes is extremely
challenging for WSSS methods using image-level labels. Hence, unsupervised
segmentation methods (Cho et al. 2021; Hamilton et al. 2022; Shin, Xie, and
Albanie 2022) are rather intensively developed and we placed reference mIoU
values from them. Attn2mask performs similarly to the existing unsupervised
methods even with the significant domain gap between web-crawled training
images off Stable Diffusion and the driving images. We confirmed the LoRA-
based adaptation contributed to the segmentation accuracy, which is
encouraging for future development of domain-adaptive diffusion-based
segmentation methods. Attn2mask outperformed the strong unsupervised
segmentation methods MaskCLIP (Zhou, Loy, and Dai 2022) and PiCIE (Cho et al.
2021) in both settings with and without adaptation. The two methods that
outperformed attn2mask relied on test-time heavy computation; STEGO used test-
time clustering (Hamilton et al. 2022) and ReCo used retrieval of relevant
images for co-segmentation (Shin, Xie, and Albanie 2022).

### Variations of post-processing

As ablative analyses of post-processing, we tested alternatives of dCRF in the
post-processing. Table 5 summarizes the results measured in the attn2mask
10,000-training-data settings. The result shows the effectiveness of dCRF
compared with simply applying threshold value $\theta$ on the attention maps
ranging in $[0,255]$, or adaptive threshold with Otsu’s method (Otsu 1979).

### Stable-Diffusion attention on real images?

We witnessed the surprising effectiveness of Stable Diffusion’s synthesized
images and attention maps combined into a segmentation dataset, but raise a
new question: Is this effectiveness solely due to the quality of the attention
maps from Stable Diffusion? Do we necessarily need to combine them with
synthesized images? To answer these questions, we attempt to train
segmentation models with real images and Stable-Diffusion attentions projected
on them. For this purpose, we use null-text inversion (Mokady et al. 2022), an
inversion method for diffusion models that estimates generation trajectories
to generate a given image with a given prompt. The estimated trajectories,
i.e., latents at each time-step are used to compute attention maps, and then
we can use them as pseudo-labels for segmentation. However, this did not
outperform attn2mask despite its additional usage of real images, as shown in
Table 6. Using the ground-truth class labels as the prompts, we tried NTI-
based pseudo-label generation for VOC training images and direct NTI-based
inference of the validation images in combination with dCRF post-processing,
but both did not outperform attn2mask. This indicates that attentions from
Stable Diffusion are particularly good when combined with its own synthesized
images, which would support our design of attn2mask.

### Visualizations

Figure 4 visualizes examples of the attention maps corresponding to each
prompt token. We can see that Stable Diffusion distributes attention well in
response to the prompts even in the multi-object generation. This suggests
that visual grounding emerges in learning large-scale text2img generation
without explicit location-based supervision.

Figure 5 shows examples of generated VOC training images and labels by
attn2mask. In addition to the natural-looking object images, it is seen that
labels are highly accurate except for several missing detailed parts and over-
masking of backgrounds where contextual connections exist, i.g., shadows of
objects or waves around boats. We also noticed there exist severe failure
cases such as masks not covering whole objects (bottom left) or mismatches of
object classes and labels (bottom right).

Figure 6 visualizes synthesized images and attention maps before and after
DreamBooth-LoRA-based adaptation. Before adaptation, image contents and styles
were not well-aligned with Cityscapes and the naturality of images was also to
some extent harmed due to generation in a distant domain from the LAION
dataset. After adaptation, generated images look enough like the images in
Cityscapes, while attention maps were kept focused on the objects of interest.
It is interesting that the correspondence between attention maps and objects
is not corrupted after fine-tuning with the Cityscapes images that are not
paired with any captions, which might be because DreamBooth-LoRA reuses the
prior knowledge acquired in the original Stable Diffusion well.

## Conclusion

We presented a real-image-and-annotation-free semantic-segmentation method
attn2mask. Although synthetic training did not outperform real-image-based
counterparts to the extent we studied, attn2mask worked surprisingly well for
its purely real-image-free segmentation training, and the gaps between the
real and synthetic performances were non-negligibly narrow. We hope that this
work inspires future studies of generated-image-based methods in more various
tasks.

## References

  * Baranchuk et al. (2022) Baranchuk, D.; Voynov, A.; Rubachev, I.; Khrulkov, V.; and Babenko, A. 2022.  Label-Efficient Semantic Segmentation with Diffusion Models.  In _ICLR_.
  * Chen et al. (2018) Chen, L.-C.; Zhu, Y.; Papandreou, G.; Schroff, F.; and Adam, H. 2018.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.  In _ECCV_.
  * Chen et al. (2015) Chen, X.; Fang, H.; Lin, T.-Y.; Vedantam, R.; Gupta, S.; Dollár, P.; and Zitnick, C. L. 2015.  Microsoft COCO captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Chen et al. (2022) Chen, Z.; Wang, T.; Wu, X.; Hua, X.-S.; Zhang, H.; and Sun, Q. 2022.  Class re-activation maps for weakly-supervised semantic segmentation.  In _CVPR_ , 969–978.
  * Cho et al. (2021) Cho, J. H.; Mall, U.; Bala, K.; and Hariharan, B. 2021.  PiCIE: Unsupervised semantic segmentation using invariance and equivariance in clustering.  In _CVPR_ , 16794–16804.
  * Cordts et al. (2016) Cordts, M.; Omran, M.; Ramos, S.; Rehfeld, T.; Enzweiler, M.; Benenson, R.; Franke, U.; Roth, S.; and Schiele, B. 2016.  The cityscapes dataset for semantic urban scene understanding.  In _CVPR_ , 3213–3223.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  ImageNet: A large-scale hierarchical image database.  In _CVPR_ , 248–255.
  * Everingham et al. (2015) Everingham, M.; Eslami, S. A.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2015.  The pascal visual object classes challenge: A retrospective.  _IJCV_ , 111: 98–136.
  * Gao et al. (2022) Gao, S.; Li, Z.-Y.; Yang, M.-H.; Cheng, M.-M.; Han, J.; and Torr, P. 2022.  Large-scale Unsupervised Semantic Segmentation.  _IEEE TPAMI_.
  * Hamilton et al. (2022) Hamilton, M.; Zhang, Z.; Hariharan, B.; Snavely, N.; and Freeman, W. T. 2022.  Unsupervised semantic segmentation by distilling feature correspondences.  _ICLR_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _CVPR_ , 770–778.
  * Healthcare Intelligence Laboratory (2017) Healthcare Intelligence Laboratory. 2017.  SimpleCRF.  https://github.com/HiLab-git/SimpleCRF.
  * Ho, Jain, and Abbeel (2020) Ho, J.; Jain, A.; and Abbeel, P. 2020.  Denoising diffusion probabilistic models.  _NeurIPS_ , 33: 6840–6851.
  * Hoffman et al. (2018) Hoffman, J.; Tzeng, E.; Park, T.; Zhu, J.-Y.; Isola, P.; Saenko, K.; Efros, A.; and Darrell, T. 2018.  CyCADA: Cycle-consistent adversarial domain adaptation.  In _ICML_ , 1989–1998. Pmlr.
  * Hu et al. (2022) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022.  LoRA: Low-rank adaptation of large language models.  _ICLR_.
  * Hugging Face (2023a) Hugging Face. 2023a.  DreamBooth fine-tuning with LoRA.  https://huggingface.co/docs/peft/task˙guides/dreambooth˙lora.
  * Hugging Face (2023b) Hugging Face. 2023b.  Stable-Diffusion-v1-4 .  https://huggingface.co/CompVis/stable-diffusion-v1-4.
  * Jo and Yu (2021) Jo, S.; and Yu, I.-J. 2021.  Puzzle-CAM: Improved localization via matching partial and full features.  In _ICIP_ , 639–643. IEEE.
  * Kataoka et al. (2022) Kataoka, H.; Hayamizu, R.; Yamada, R.; Nakashima, K.; Takashima, S.; Zhang, X.; Martinez-Noriega, E. J.; Inoue, N.; and Yokota, R. 2022.  Replacing labeled real-image datasets with auto-generated contours.  In _CVPR_ , 21232–21241.
  * Krähenbühl and Koltun (2011) Krähenbühl, P.; and Koltun, V. 2011.  Efficient inference in fully connected CRFs with gaussian edge potentials.  _NeurIPS_ , 24.
  * Li et al. (2022a) Li, D.; Ling, H.; Kim, S. W.; Kreis, K.; Fidler, S.; and Torralba, A. 2022a.  BigDatasetGAN: Synthesizing ImageNet with pixel-wise annotations.  In _CVPR_ , 21330–21340.
  * Li et al. (2022b) Li, Y.; Duan, Y.; Kuang, Z.; Chen, Y.; Zhang, W.; and Li, X. 2022b.  Uncertainty estimation via response scaling for pseudo-mask noise mitigation in weakly-supervised semantic segmentation.  In _AAAI_ , volume 36, 1447–1455.
  * Li et al. (2021) Li, Y.; Kuang, Z.; Liu, L.; Chen, Y.; and Zhang, W. 2021.  Pseudo-mask matters in weakly-supervised semantic segmentation.  In _ICCV_ , 6964–6973.
  * Li et al. (2023) Li, Z.; Zhou, Q.; Zhang, X.; Zhang, Y.; Wang, Y.; and Xie, W. 2023.  Guiding Text-to-Image Diffusion Model Towards Grounded Generation.  _arXiv preprint arXiv:2301.05221_.
  * Liu et al. (2022) Liu, S.; Liu, K.; Zhu, W.; Shen, Y.; and Fernandez-Granda, C. 2022.  Adaptive early-learning correction for segmentation from noisy annotations.  In _CVPR_ , 2606–2616.
  * MMSegmentation Contributors (2020) MMSegmentation Contributors. 2020.  MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark.  https://github.com/open-mmlab/mmsegmentation.
  * Mokady et al. (2022) Mokady, R.; Hertz, A.; Aberman, K.; Pritch, Y.; and Cohen-Or, D. 2022.  Null-text Inversion for Editing Real Images using Guided Diffusion Models.  _arXiv preprint arXiv:2211.09794_.
  * Oquab et al. (2015) Oquab, M.; Bottou, L.; Laptev, I.; and Sivic, J. 2015.  Is object localization for free?-weakly-supervised learning with convolutional neural networks.  In _CVPR_ , 685–694.
  * Otsu (1979) Otsu, N. 1979.  A threshold selection method from gray-level histograms.  _IEEE transactions on systems, man, and cybernetics_ , 9(1): 62–66.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_ , 8748–8763. PMLR.
  * Ramesh et al. (2021) Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021.  Zero-shot text-to-image generation.  In _ICML_ , 8821–8831. PMLR.
  * Rao et al. (2022) Rao, Y.; Zhao, W.; Chen, G.; Tang, Y.; Zhu, Z.; Huang, G.; Zhou, J.; and Lu, J. 2022\.  DenseCLIP: Language-guided dense prediction with context-aware prompting.  In _CVPR_ , 18082–18091.
  * Richter et al. (2016) Richter, S. R.; Vineet, V.; Roth, S.; and Koltun, V. 2016.  Playing for data: Ground truth from computer games.  In _ECCV_ , 102–118. Springer.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 10684–10695.
  * Rong et al. (2023) Rong, S.; Tu, B.; Wang, Z.; and Li, J. 2023.  Boundary-Enhanced Co-Training for Weakly Supervised Semantic Segmentation.  In _CVPR_ , 19574–19584.
  * Ruiz et al. (2023) Ruiz, N.; Li, Y.; Jampani, V.; Pritch, Y.; Rubinstein, M.; and Aberman, K. 2023\.  DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 22500–22510.
  * Sariyildiz et al. (2023) Sariyildiz, M. B.; Alahari, K.; Larlus, D.; and Kalantidis, Y. 2023.  Fake it till you make it: Learning transferable representations from synthetic ImageNet clones.  In _CVPR_.
  * Schuhmann et al. (2021) Schuhmann, C.; Vencu, R.; Beaumont, R.; Kaczmarczyk, R.; Mullis, C.; Katta, A.; Coombes, T.; Jitsev, J.; and Komatsuzaki, A. 2021.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _NeurIPS workshop_.
  * Shin, Xie, and Albanie (2022) Shin, G.; Xie, W.; and Albanie, S. 2022.  ReCo: Retrieve and co-segment for zero-shot transfer.  _NeurIPS_ , 35: 33754–33767.
  * Tian et al. (2023) Tian, Y.; Fan, L.; Isola, P.; Chang, H.; and Krishnan, D. 2023.  StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners.  _arXiv preprint arXiv:2306.00984_.
  * Wang et al. (2020) Wang, Y.; Zhang, J.; Kan, M.; Shan, S.; and Chen, X. 2020.  Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation.  In _CVPR_ , 12275–12284.
  * Wu et al. (2023) Wu, W.; Zhao, Y.; Shou, M. Z.; Zhou, H.; and Shen, C. 2023.  Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models.  _arXiv preprint arXiv:2303.11681_.
  * Xu et al. (2023) Xu, J.; Liu, S.; Vahdat, A.; Byeon, W.; Wang, X.; and De Mello, S. 2023.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In _CVPR_ , 2955–2966.
  * Yu et al. (2021) Yu, P.; Xie, S.; Ma, X.; Zhu, Y.; Wu, Y. N.; and Zhu, S.-C. 2021.  Unsupervised foreground extraction via deep region competition.  _NeurIPS_ , 34: 14264–14279.
  * Zhou et al. (2016) Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. 2016.  Learning deep features for discriminative localization.  In _CVPR_ , 2921–2929.
  * Zhou, Loy, and Dai (2022) Zhou, C.; Loy, C. C.; and Dai, B. 2022.  Extract Free Dense Labels from CLIP.  In _ECCV_.