# Self-Supervised Learning from Non-Object Centric Images with a Geometric
Transformation Sensitive Architecture

Taeho Kim Jong-Min Lee∗
Department of Artificial Intelligence
Hanyang University
222, Wangsimni-ro, Seongdong-gu, Seoul, Republic of Korea
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Most invariance-based self-supervised methods rely on single object-centric
images (e.g., ImageNet images) for pretraining, learning features that
invariant to geometric transformation. However, when images are not object-
centric, the semantics of the image can be significantly altered due to
cropping. Furthermore, as the model becomes insensitive to geometric
transformations, it may struggle to capture location information. For this
reason, we propose a Geometric Transformation Sensitive Architecture designed
to be sensitive to geometric transformations, specifically focusing on four-
fold rotation, random crop, and multi-crop. Our method encourages the student
to be sensitive by predicting rotation and using targets that vary with those
transformations through pooling and rotating the teacher feature map.
Additionally, we use patch correspondence loss to encourage correspondence
between patches with similar features. This approach allows us to capture
long-term dependencies in a more appropriate way than capturing long-term
dependencies by encouraging local-to-global correspondence, which occurs when
learning to be insensitive to multi-crop. Our approach demonstrates improved
performance when using non-object-centric images as pretraining data compared
to other methods that train the model to be insensitive to geometric
transformation. We surpass DINO[Caron et al. (2021b)] baseline in tasks
including image classification, semantic segmentation, detection, and instance
segmentation with improvements of 4.9 $Top-1Acc$, 3.3 $mIoU$, 3.4 $AP^{b}$,
and 2.7 $AP^{m}$. Code and pretrained models are publicly available at:
https://github.com/bok3948/GTSA

##  1 Introduction

Invariance-based methods are one of the primary self-supervised learning
approaches for computer vision. These methods learn to be insensitive to
various transformations, such as rotations, flips, crops, color jittering,
blurring, and random grayscale, which provide an inductive bias that helps
with representation learning[Chen and He (2020), Bardes et al. (2022a),
Zbontar et al. (2021)].

Augmentations employed in self-supervised learning methods can be divided into
two categories: photometric transformations and geometric transformations.
Photometric transformations, such as color jittering, Gaussian blurring, and
grayscale conversion, involve changes to the appearance of an image, like
color, brightness, or contrast. geometric transformations, including random
crop, multi-crop, flip and rotation, deal with changes to the spatial
configuration of the image.

In the case of pretraining with non-object centric image, Learning invariant
features from crop-related geometric transformations can be problematic. This
is because cropped views may not always depict the same object[Purushwalkam
and Gupta (2020), Zhang et al. (2022)]. In contrast, object-centric images are
less prone to such issues due to their inherent focus on specific objects,
which remain semantically consistent across various augmentations. This
explains the significant performance drop observed when applying invariant
methods to non-object centric images[Purushwalkam and Gupta (2020), El-Nouby
et al. (2021)]. It can also be one of the reasons that, to obtain comparable
results with curated datasets, a considerably larger amount of uncurated data
is required[Goyal et al. (2021)].

Furthermore, when learning to be insensitive to geometric transformations,
there is a risk of not capturing location information, and dense prediction
models need to be sensitive to these transformations rather than insensitive.
Therefore, being insensitive to geometric transformations may not be
appropriate.

As mentioned earlier, training a model to be insensitive to geometric
transformations may lead to noise in learning. However, these transformations
can still be beneficial for representation learning since they prevent
pathological training behavior[Chen et al. (2020)] and provide diversity in
inputs. Therefore, we propose a method that focuses on training a model to be
sensitive with respect to those transformations, instead of insensitive to
those transformations.

To achieve this, we must provide a target that varies according to the input’s
geometric transformation during training. to create a target that varies with
cropping, we pool the overlapping region from the teacher’s feature map, which
can be seen as cropping, and provide it to the student as a target.
Additionally, to make the model sensitive to four-fold rotations, we rotate
the target feature map to align it appropriately with the student input, and
we include a prediction task to predict the degree of rotation of the
student’s input.

Furthermore, we use a patch correspondence loss in our approach. When learning
invariant features through multi-crop inputs, it will encourage global-to-
local correspondence[Caron et al. (2021b), Caron et al. (2021a)], resulting in
the capture of long-term dependencies. Our model uses an additional loss that
encourages correspondence between patch representations through cosine
similarity, allowing us to capture long-term dependencies [Bardes et al.
(2022b)]. Unlike encouraging correspondence between randomly selected crops,
our approach induces correspondence between those that are similar in feature
space, leading to more accurate correspondence.

Our experiments demonstrate that when using non-centric images as pretraining
data, it is more advantageous to train a model to be sensitive to geometric
transformations rather than insensitive. We significantly outperformed
prominent invariance-based methods in various tasks, including image
classification, semantic segmentation, detection, and instance segmentation.

##  2 Related Work

Non-Contrative Learning Non-contrastive learning methods aim to learn an
invariant bias towards transformations by training on different views of the
same image, without explicit negative samples[Garrido et al. (2022)]. In the
absence of negative samples, non-contrastive learning methods employ various
alternative approaches to prevent representation collapse. These include non-
contrastive losses that minimize redundancy across embeddings[Bardes et al.
(2022a),Zbontar et al. (2021)], clustering-based techniques that maximize the
entropy of the average embedding[Caron et al. (2019), Caron et al. (2021a),
Goyal et al. (2021), Assran et al. (2022)], centering and sharpening output
features[Caron et al. (2021b)], and heuristic strategies utilizing asymmetric
architectural design with stop-gradient, additional predictors, and momentum
encoders[Richemond et al. (2020), Chen and He (2020), Tian et al. (2021b)].
Our method belongs to the non-contrastive learning category and adopts an
asymmetric architectural design to prevent representation collapse.

Self-Supervised Learning with Uncurated Dataset Several self-supervised
pretraining methods have been proposed for uncurated datasets, such as the
clustering-based method presented in [Caron et al. (2021a), Tian et al.
(2021a)]. These methods have shown good performance even when using uncurated
datasets, and [Goyal et al. (2021)] demonstrated the scalability of their
method to larger datasets for increased performance. Additionally, [El-Nouby
et al. (2021)] showed that, given sufficient iterations, even a small non-
object centric dataset can yield results that are comparable to those obtained
using a larger, highly curated dataset.

However, our approach differs from other methods that aim to adapt to
uncurated datasets. While clustering-based techniques are used in these
methods, they still learn invariant representations to augmentations, whereas
our approach learns features that are sensitive to geometric transformations.
On the other hand, [Tian et al. (2021a)] aims to address the shift in the
distribution of image classes rather than object-centric bias.

Self-Supervised Methods that Learn to be Sensitive to Transformations Early
self-supervised learning methods, such as [Noroozi and Favaro (2017),
Yamaguchi et al. (2021)], train the model to be sensitive to transformations
by predicting the permutation or rotation applied to the input. As contrastive
learning has gained prominence in representation learning, the importance of
learning transformation-invariant representations has become increasingly
evident[Misra and van der Maaten (2019), Chen et al. (2020), He et al.
(2020)]. More recent work has utilized a hybrid approach that is sensitive to
some transformations and insensitive to others [Dangovski et al. (2022)].
Performance improvement has been achieved by training to be sensitive to four-
fold rotation while insensitive to other transformations. Similarly, our model
also learns to be sensitive to four-fold rotations and insensitive to other
photometric transformations. However, our method additionally becomes
sensitive to crop-related transformations.

Figure 1: The GTSA. The Geometric Transformation Sensitive Architecture
(GTSA). the teacher only receives global views only, while the student
receives both global and local views. The learning process is designed to
increase the similarity in overlapping regions and predicting four-fold
rotation. Additionally, to capture long-term dependencies, GTSA encourages
similarity between the teacher’s patch representations and the student’s patch
representations by matching these patch representations using cosine
similarity.

##  3 Methods

In this section, we describe the training procedure for our proposed GTSA
method, as illustrated in Figure 1. We adopt an asymmetric Teacher-Student
architecture, similar to those in [Richemond et al. (2020), Chen and He
(2020)]. The Teacher comprises an encoder and a projector, while the Student
includes an encoder, projector, and predictor. Following the multi-crop
strategy used in [Caron et al. (2021a), Caron et al. (2021b)], we feed only
the global view to the Teacher and both global and local views to the Student.
Our objectives involve maximizing the similarity between overlapping region
representations and similar patch representations and predicting rotation.

Inputs. Similar to [Ziegler and Asano (2022)], we utilized various
augmentation techniques including color jitter, random grayscale, random
Gaussian noise, Gaussian blur, random resize crop, and multi-crop.
Additionally, we employed four-fold rotation. For each image, we apply random
augmentations and generate G global views and L local views. The inputs are
$[x_{1}^{g},x_{2}^{g},\dots,x_{1}^{l},x_{2}^{l},\dots]$, where $g$ and $l$
indicate global and local view, respectively. $x$ represents batchified
images, and $x\in\mathbb{R}^{B\times C\times H\times W}$. Here, $B$ is the
batch size, $C$ represents the number of image channels, and $H$ and $W$
denote the image size.

Teacher and Student. Apart from the additional predictor attached to the
Student, the Teacher and Student share the same structural design. We utilized
the ViT[Dosovitskiy et al. (2021)] as the encoder and employed stacked CNN
blocks for the projector, each block comprising a convolution layer, layer
normalization[Ba et al. (2016)], GELU activation[Hendrycks and Gimpel (2020)]
and residual connection[He et al. (2015)]. The predictor has a similar
architecture to the projector but uses fewer CNN blocks in its composition. We
annotated the Predictor and Projector as H and U, respectively. Note that both
the projector and predictor do not reduce the spatial resolution of the
encoder’s output. The Teacher does not get updated through gradient descent;
instead, its weights follow the Student’s weights using exponential moving
average.[Tarvainen and Valpola (2018), He et al. (2019)]

Correspondence Region Pooling Operator. We introduce a correspondence region
pooling operator, denoted as $\Phi(\cdot)$. To be sensitive to crop-related
augmentations, the student must receive a target that reflects the crop
augmentation. The $\Phi(\cdot)$ operator serves this purpose by cropping
specific locations in the feature space. It extracts the overlapping portions
between the teacher view and student view in the feature space.

To accomplish this, we first calculate the overlap region bounding boxes in
the input space and scale them to match the feature spatial resolution. We
then apply the $\Phi(\cdot)$ operator to both the student and teacher feature
maps. We implement this operator using RoI-Align [He et al. (2018)].

Rotation Operator. To be sensitive to rotation, we propose a rotation
operator. This operator rotates the teacher output feature map according to
the input rotation. We denote this as $R(\cdot)$. By using this operator, the
student receives a target that reflects the input rotation.

Rotation Predictor. As we employ the rotation prediction pretext task, we
extract a vector from the student encoder output using a Global Average
Pooling layer[Lin et al. (2014)]. This vector is then input into the rotation
predictor, which generates logits for the rotation prediction pretext task.
The architecture of this process includes a linear layer, GELU activation, and
a normalization layer. We denote the rotation predictor as P.

Loss Function. We denote the output feature map of the student predictor as
$z$ and the output feature map of the teacher’s projection layer as
$\tilde{z}$. $z\in\mathbb{R}^{B\times D\times h_{s}\times w_{s}}$ and
$\tilde{z}\in\mathbb{R}^{B\times D\times h_{t}\times w_{t}}$, where $B$ is the
batch size, $D$ is the feature dimension, $h_{s}$ and $w_{s}$ represent the
spatial size of the student feature map, and $h_{t}$ and $w_{t}$ represent the
spatial size of the teacher feature map. We apply $\Phi$ to both $z$ and
$\tilde{z}$, and additionally apply $R$ to $\Phi(\tilde{z})$. Both $\Phi(z)$
and $R(\Phi(\tilde{z}))$ will have the same dimensions.
$\Phi(z),R(\Phi(\tilde{z}))\in\mathbb{R}^{B\times D\times h_{o}\times w_{o}}$.
We then compute the cosine similarity between them along feature dimension.
The equation is as follows:

$l(z,\tilde{z})=-\frac{1}{B\times
T}\sum_{i=1}^{B}\sum_{t=1}^{T}\frac{\Phi(z_{i})_{t}\cdot
R(\Phi(\tilde{z}_{i}))_{t}}{\lVert\Phi(z_{i})_{t}\rVert\lVert
R(\Phi(\tilde{z}_{i}))_{t}\rVert}$ (1)

where $i$ is the index of the batch, $t$ indicates the t-th spatial location
in the output of the operator and $T$ is $(h_{o}\times w_{o})$. the cosine
similarity is calculated as the dot product of the two vectors divided by the
product of their magnitudes.

For the Rotation prediction pretext task, we use rotation prediction loss,
which is annotated with $l_{rp}$. We design this loss using cross-entropy
loss. The equation is as follows:

$l_{rp}(\mathbf{y},\mathbf{\hat{y}})=-\frac{1}{B}\sum_{i=1}^{B}\mathbf{y}_{i}\cdot\log\mathbf{\hat{y}}_{i}$
(2)

where $\mathbf{y}_{i}$ is the target indicating the $i$-th sample rotation
angle. $\mathbf{\hat{y}}_{i}$ is the Rotation Predictor output probability
distribution of the $i$-th sample. In this case, as we use four-fold rotation,
the possible rotation angles are [0, 90, 180, 270] degrees.

Additionally, we use patch correspondence loss, which we annotate with
$l_{pc}$. We pair semantically similar patches from the teacher and student
patch representations based on their cosine similarity and make them more
alike using the $l_{pc}$ loss. However, as noise may exist in this process, we
do not encourage correspondence between all patches. Instead, we only
encourage the similarity of the top-$K$ most similar features among all
patches, the same as [Bardes et al. (2022b)]. This helps to ensure that we are
aligning semantically similar patches while avoiding pairing patches that are
too dissimilar.

$l_{pc}(z,\tilde{z})=-\frac{1}{B\times
K}\sum_{i=1}^{B}\sum_{p=1}^{K}\frac{z_{i,p}\cdot\tilde{z}_{i,\tilde{p}}}{\lVert
z_{i,p}\rVert\lVert\tilde{z}_{i,\tilde{p}}\rVert}$ (3)

Here, $z_{i,p}$ refers to the $p$-th patch representation of the i-th sample
from the student, and $\tilde{z}_{i,\tilde{p}}$ denotes The patch
representation that is the closest to $z_{i,p}$ among the patch
representations for the teacher’s i-th sample. $K$ represents total number of
matched pairs which are filtered.

To consider multi-crop scenarios, we use the following total loss function:

$\displaystyle\text{Loss}=$
$\displaystyle\frac{1}{G(L+G)-G}\sum_{g=1}^{G}\sum_{\begin{subarray}{c}l=1\\\
l\neq
g\end{subarray}}^{L+G}l(z_{l},\tilde{z}_{g})+\alpha*\frac{1}{G(L+G)-G}\sum_{g=1}^{G}\sum_{\begin{subarray}{c}l=1\\\
l\neq g\end{subarray}}^{L+G}l_{pc}(z_{l},\tilde{z}_{g})$
$\displaystyle+\beta*\frac{1}{L+G}\sum_{l=1}^{L+G}l_{rp}(\mathbf{y_{l}},\mathbf{\hat{y_{l}}})$
(4)

Here, $\alpha$ and $\beta$ are hyperparameters that control the impact of
$l_{pc}$ and $l_{rp}$, respectively. $G$ and $L$ denote the total number of
global views and local views, respectively.

Table 1: The iNaturalist 2019 image classification performance. performance was obtained by pretraining all models with 100 epochs on COCO train2017, followed by fine-tuning for 300 epochs using the same settings for all models. Method | Top-1 Acc | Top-5 Acc
---|---|---
rand init | 40.6 | 69.0
MoCo v3 | 48.8 | 76.3
DINO | 54.8 | 82.9
GTSA (Ours) | 59.7 | 85.7

##  4 Experiments

Experiments have three distinct subsections for a comprehensive explanation of
our approach.

In Section 4.1, we demonstrate the effectiveness of GTSA in learning high-
quality representations from non-object centric images. We pretrained our
model on the COCO train2017 dataset [Lin et al. (2015)], a collection composed
of non-object centric images. Then, we evaluated the performance of our model
by fine-tuning it on various downstream tasks, specifically Classification,
Detection, Instance Segmentation, and Semantic Segmentation. In addition, we
pretrained our model on the ADE20K train dataset [Zhou et al. (2018)], which
also consists of non-object centric images. However, due to the larger size of
other downstream task datasets compared to the pretraining dataset, we
restricted our evaluation to Semantic Segmentation on the ADE20K dataset.

In Section 4.2, we demonstrated that our model operates as intended,
showcasing its sensitivity to rotation and crop-related transformations.

In Section 4.3, we compiled the results from our ablation study. We
illustrated the effects of rotation prediction loss and patch correspondence,
and through visualization of encouraged patch pairs, we verified that our
method encourages correspondence even with patches that are far apart.

Pretrain Setup. We used the same hyperparameters as DINO, as much as possible.
Specifically, we set the batch size to 512, the global view size to 224x224,
and the local view size to 96x96. We also used a scheduler to start the
momentum at 0.996, just like DINO, and gradually increased it to 1. For the
optimizer, we employed AdamW[Loshchilov and Hutter (2019)] and set both
$\alpha$ and $\beta$ to 0.5. When pretraining with the ADE20K train dataset,
we changed the jitter strength, which is a hyperparameter used for color
jittering, from 1.0 to 0.2 and didn’t normalize the encoder’s output, and also
$\beta$ to 0.25. Apart from these differences, all other settings were the
same. Our default model is ViT-S/16, and we pretrained it for 100 epochs with
8 NVIDIA GeForce RTX 3090 GPUs.

Table 2: The COCO 2017 detection and instance segmentation performance. Performance was obtained by pretraining all models with 100 epochs on COCO train2017, followed by fine-tuning for a standard $1\times$ schedule using the same settings for all models. Method |  Detection |  | Instance Segmentation
---|---|---|---
|  | $AP\textsuperscript{b}$ | $AP\textsuperscript{b}_{50}$ | $AP\textsuperscript{b}_{75}$ | $AP\textsuperscript{m}$ | $AP\textsuperscript{m}_{50}$ | $AP\textsuperscript{m}_{75}$
rand init |  | 23.2 | 42.3 | 22.5 | 23.0 | 40.0 | 23.3
MoCo v3 |  | 29.6 | 50.1 | 30.4 | 28.3 | 47.7 | 29.2
DINO |  | 32.4 | 54.2 | 33.8 | 30.8 | 51.1 | 32.2
GTSA(ours) |  | 35.8 | 57.8 | 38.5 | 33.5 | 54.7 | 35.3
Table 3: The ADE20K image semantic segmentation performance. Performance was obtained by pretraining all models with 100 epochs on COCO train2017, followed by fine-tuning for a standard 40K iteration schedule. Method | aAcc | mIoU | mAcc
---|---|---|---
rand init | 64.5 | 12.1 | 16.1
MoCo v3 | 72.7 | 23.5 | 31.7
DINO | 74.7 | 27.3 | 35.9
GTSA (Ours) | 76.4 | 30.6 | 40.0

###  4.1 Fine-tuning

Baseline. We chose Dino and MoCo v3[Chen et al. (2021)] as our baseline
methods. These methods are highly relevant to our research as they, like us,
employ the Vision Transformer (ViT) as an encoder and focus on training the
model to be insensitive in a self-supervised manner. Thus, they provide a
compelling counterpoint to our approach which train the model to be sensitive
to geometric transformations.

The official codes from these baseline methods were leveraged in producing our
results. To ensure a fair and balanced comparison, all methods underwent
pretraining under the same conditions: 100 pretraining epochs and the use of
the ViT-S/16 encoder. Moreover, the fine-tuning process was executed in an
entirely same manner across all methods.

Image Classification. We compared our method with other self-supervised
methods in terms of image classification performance when fine-tuning on the
iNaturalist 2019 dataset [Horn et al. (2018)]. Table 1 shows that our method
outperforms other methods that learn only invariant features. We achieve a 4.9
and 10.9 higher accuracy compared to DINO and MoCo-v3, respectively, and a
19.1 accuracy improvement compared to random initialization.

Detection and Instance Segmentation. Table 2 shows the performance of our
method on COCO detection and instance segmentation tasks. GTSA outperforms
DINO and MoCo-v3 by 3.4 and 6.2 $AP^{b}$ in detection, and by 2.7 and 5.2
$AP^{m}$ in instance segmentation, respectively. All models were fine-tuned
using Mask R-CNN [He et al. (2018)] and FPN [Lin et al. (2017)] under the
standard 1x schedule.

Table 4: The performance of ADE20K image semantic segmentation when pretraining on the ADE20K train dataset. Performance was obtained by pretraining all models with 100 epochs on ADE20K train dataset, followed by fine-tuning for a standard 40K iteration schedule. Method | aAcc | mIoU | mAcc
---|---|---|---
rand init | 64.5 | 12.1 | 16.1
MoCo v3 | 66.3 | 13.6 | 19.0
DINO | 65.5 | 13.6 | 18.8
GTSA (Ours) | 68.4 | 16.2 | 22.6

Figure 2: Output variance comparison. This figure demonstrates how sensitive the DINO and our model are to input transformations. The y-axis represents the mean of output variance of the pretrained encoder. It demonstrates that our model is more sensitive to geometric augmentations than DINO. Table 5: Ablation Study: effect of $\mathbf{l_{pc}}$ and $\mathbf{l_{rc}}$ This table shows that the segmentation performance improves with the additional use of ${l_{pc}}$ and ${l_{rc}}$. loss function | aAcc | mIoU | mAcc
---|---|---|---
$l$ | 67.3 | 15.4 | 21.5
$l$ \+ $l_{pc}$ | 67.7 | 15.8 | 21.7
$l$ \+ $l_{pc}$ \+ $l_{rc}$ | 68.4 | 16.2 | 22.6

Figure 3: Ablation Study: Matched Pairs. The visualization of matched pairs
when computing patch $l_{pc}$. The visualization was generated by feeding
images from the COCO val2017 dataset into a GTSA model, which had been
previously trained on the COCO train2017 dataset.

Semantic Segmentation. Table 3 reports the performance of on ADE20K semantic
segmentation using the Acc, mIoU, and mAcc metrics with all methods are
pretrained with COCO train2017 dataset. While DINO achieves a 27.3 mIoU, GTSA
attains a higher performance of 30.6 mIoU, which is a 3.3 mIoU improvement.
Moreover, our method outperforms MoCo-v3 by 7.1 mIoU. All models were fine-
tuned using Semantic FPN [Kirillov et al. (2019)] under the standard 40k
iteration schedule, following the same approach as in [Yun et al. (2022)].

Table 4, also reports the performance of on ADE20K semantic segmentation but
differ in that in this table use ADE20K train dataset as pretraining data.
GTSA outperforms DINO and MoCo with inprovement 2.6 mIoU. fine-tuning setting
are all same to Table 3.

###  4.2 Proving sensitivity to geometric transformations

In this section, we present Figure 2 to showcase our model’s sensitivity to
various transformations. We designed this experiment by measuring the variance
in the output based on input transformations. Specifically, we generated ten
views with single type of augmentation and fed these views into a model
pretrained using the DINO method and another pretrained using our method. We
then measured the variance of the encoder output generated by global average
pooling. Here, we utilized the COCO val2017 dataset to compute the mean of
variance, which is denoted on the y-axis.

As shown in Figure 2, both DINO and GTSA learned to be invariant to color
jittering, resulting in a very low variance. However, for four-fold rotation
and crop-related transformations, GTSA exhibited a substantially higher
variance compared to DINO. The crop-related transformations was implemented
using a random resize crop, creating two global views and eight local views.
It is important to note that the exact same inputs were fed into both the DINO
and GTSA.

###  4.3 Ablation study.

In this section, we demonstrate the performance enhancement achieved through
$l_{pc}$ and $l_{rp}$, and we present a figure that visualizes matched pairs.
This illustrates that even distant patches are matched, confirming that
correspondence is encouraged over long distances.

As displayed in Table 5, we set $l$ as the baseline and showed performance
improvement with the addition of $l_{pc}$ and $l_{rp}$. For simplicity, we
pretrain on the ADE20K train dataset for 100 epochs and report the results for
Semantic Segmentation on the ADE20K dataset. We observed a 0.4 mIoU increase
upon adding $l_{pc}$, and an additional 0.4 mIoU increase when $l_{rp}$ was
incorporated.

Figure 3 visualizes matched pairs. We used GTSA, which was pretrained for 100
epochs on the COCO train2017 dataset, and input images from COCO val2017 to
obtain the matched pairs. From the left image, we can see that matching occurs
between parts that depict columns, even if they are not precisely the same
column. Similarly, in the right image, we see matching between two parts, both
depicting a wall, despite being located at a distance from each other. This
demonstrates that our method encourages the capture of long-term dependencies
as we intended.

##  5 Conclusion

We propose the Geometric Transformation Sensitive Architecture (GTSA) as a
self-supervised method designed for non-object centric images. Our approach
trains the model to be sensitive to geometric transformations, specifically
rotation and crop-related transformations, by utilizing targets that reflect
geometric transformations. Experimental results demonstrate that our method
outperforms other transformation-invariant methods when pretrained on non-
object centric images.

Limitations and Future Works. Our method does not learn to be sensitive to all
types of geometric transformations. Specifically, it is trained to be
sensitive to four-fold rotations and crop-related transformations. In the
future, we aim to explore its effectiveness when made sensitive to a broader
range of geometric transformations. Moreover, we will conduct research to
achieve superior performance on curated datasets.

## References

  * Assran et al. [2022] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas.  Masked siamese networks for label-efficient learning, 2022.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.
  * Bardes et al. [2022a] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning, 2022a.
  * Bardes et al. [2022b] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicregl: Self-supervised learning of local visual features, 2022b.
  * Caron et al. [2019] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features, 2019.
  * Caron et al. [2021a] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments, 2021a.
  * Caron et al. [2021b] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers, 2021b.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations, 2020.
  * Chen and He [2020] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15745–15753, 2020.
  * Chen et al. [2021] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers, 2021\.
  * Dangovski et al. [2022] Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačić.  Equivariant contrastive learning, 2022.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * El-Nouby et al. [2021] Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave.  Are large-scale datasets necessary for self-supervised pre-training?, 2021\.
  * Garrido et al. [2022] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun.  On the duality between contrastive and non-contrastive self-supervised learning, 2022.
  * Goyal et al. [2021] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski.  Self-supervised pretraining of visual features in the wild, 2021.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * He et al. [2018] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn, 2018.
  * He et al. [2019] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick.  Momentum contrast for unsupervised visual representation learning.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9726–9735, 2019.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning, 2020\.
  * Hendrycks and Gimpel [2020] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus), 2020.
  * Horn et al. [2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset, 2018.
  * Kirillov et al. [2019] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.  Panoptic feature pyramid networks, 2019.
  * Lin et al. [2014] Min Lin, Qiang Chen, and Shuicheng Yan.  Network in network, 2014.
  * Lin et al. [2015] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context, 2015.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection, 2017.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization, 2019.
  * Misra and van der Maaten [2019] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations, 2019.
  * Noroozi and Favaro [2017] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles, 2017.
  * Purushwalkam and Gupta [2020] Senthil Purushwalkam and Abhinav Gupta.  Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases, 2020.
  * Richemond et al. [2020] Pierre H. Richemond, Jean-Bastien Grill, Florent Altch’e, Corentin Tallec, Florian Strub, Andrew Brock, Samuel L. Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko.  Byol works even without batch statistics.  _ArXiv_ , abs/2010.10241, 2020.
  * Tarvainen and Valpola [2018] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, 2018.
  * Tian et al. [2021a] Yonglong Tian, Olivier J. Henaff, and Aaron van den Oord.  Divide and contrast: Self-supervised learning from uncurated data, 2021a.
  * Tian et al. [2021b] Yuandong Tian, Xinlei Chen, and Surya Ganguli.  Understanding self-supervised learning dynamics without contrastive pairs, 2021b.
  * Yamaguchi et al. [2021] Shin’ya Yamaguchi, Sekitoshi Kanai, Tetsuya Shioda, and Shoichiro Takeda.  Image enhanced rotation prediction for self-supervised learning, 2021\.
  * Yun et al. [2022] Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin.  Patch-level representation learning for self-supervised vision transformers, 2022.
  * Zbontar et al. [2021] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In _International Conference on Machine Learning_ , 2021.
  * Zhang et al. [2022] Tong Zhang, Congpei Qiu, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann.  Leverage your local and global representations: A new self-supervised learning strategy, 2022.
  * Zhou et al. [2018] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset, 2018.
  * Ziegler and Asano [2022] Adrian Ziegler and Yuki M. Asano.  Self-supervised learning of object parts for semantic segmentation, 2022\.


[fieldsource=url, match={
_}—{_}—
_, replace=_] [fieldsource=url, match={$
sim$}—{}̃—$
sim$, replace=~] [fieldsource=url, match={
26}, replace=26]

# Multi-Source Soft Pseudo-Label Learning with
Domain Similarity-based Weighting for Semantic Segmentation

Shigemichi Matsuzaki1, Hiroaki Masuzawa1, and Jun Miura1 1S. Matsuzaki, H.
Masuzawa, and J. Miura are with Department of Computer Science and
Engineering, Toyohashi University of Technology, Hibarigaoka 1-1, Tenpaku-cho,
Toyohashi, Aichi, Japan<EMAIL_ADDRESS>code:
https://github.com/ShigemichiMatsuzaki/MS2PL

###### Abstract

This paper describes a method of domain adaptive training for semantic
segmentation using multiple source datasets that are not necessarily relevant
to the target dataset. We propose a soft pseudo-label generation method by
integrating predicted object probabilities from multiple source models. The
prediction of each source model is weighted based on the estimated domain
similarity between the source and the target datasets to emphasize
contribution of a model trained on a source that is more similar to the target
and generate reasonable pseudo-labels. We also propose a training method using
the soft pseudo-labels considering their entropy to fully exploit information
from the source datasets while suppressing the influence of possibly
misclassified pixels. The experiments show comparative or better performance
than our previous work and another existing multi-source domain adaptation
method, and applicability to a variety of target environments.

##  I Introduction

Semantic segmentation based on deep neural networks (DNNs) has been used as a
common and strong tool for scene recognition of autonomous mobile agents.
However, efficiency of training remains a critical problem. Usually, networks
are trained on a large amount of manually labeled data collected via a
laborious annotation process. Although rich datasets are publicly available
for some actively studied scenes, e.g., urban scenes where autonomous driving
is a hot topic [1, 2], there are way fewer datasets for specific scenes, such
as greenhouses and unstructured scenes.

Domain adaptation (DA) is a task to adapt a model pre-trained on a source
dataset to a target dataset, and unsupervised domain adaptation (UDA) is a
problem setting where the target dataset has no ground truth labels.
Especially in autonomous driving community, UDA for semantic segmentation has
been actively studied as a promising approach to efficient training by
exploiting simulated photorealistic images generated by video games [3] or
dedicated simulators [4, 5]. However, again, it is not easy to collect
suitable source datasets for many other environments.

In our previous work [6], we proposed a method to train a semantic
segmentation model for greenhouse images using multiple publicly available
datasets of scenes not relevant but related to greenhouses, e.g., urban scenes
and unstructured outdoor scenes, as source datasets to overcome the
aforementioned problem. The method utilizes source datasets with different
scenes and label sets to train a model on the target dataset. The outputs from
each source model are converted to pixel-wise one-hot labels in the common
target label sets and merged based on unanimity of all the models (see Fig.
1). This allows reliable labels to be effectively extracted.

While such a strict criterion contributes to excluding possibly wrong labels
and results in good performance, it completely ignores information of many
pixels excluded from the training, leading to sparsity of valid labels. A
limitation of the method stemming from this problem is that the source
datasets must have at least one corresponding object class for all target
classes to enable unanimity-based label selection. This restricts the choice
of source datasets.

Figure 1: Top: Previous method [6] generates pseudo-labels using multiple
source models. A label is assigned only if all models agree with each other to
remove wrong labels. If a class is not included in a source, the class never
appears in the resulting pseudo-labels. Bottom: Proposed method. It generates
soft pseudo-labels by summing predicted scores weighted by inverse domain gap,
i.e., domain similarity. The degree of agreement of the source models are
represented by the inverse entropy of the soft pseudo-labels. It can also
involve labels not present in some source datasets.

In this paper, we extend our previous work [6] and propose a novel multi-
source pseudo-label generation method. Instead of selecting one-hot pseudo-
labels based on unanimity, we generate soft pseudo-labels which take a form of
class probability distribution on each pixel by integrating the outputs from
the source models. To integrate the outputs from the source models, we take
into account quantitatively evaluated domain similarities between the target
data and each source dataset so that the predictions of a model trained on a
dataset closer to the target is emphasized more. In training, we weight the
loss values of each pixel with a value inversely proportional to the entropy
of the soft pseudo-label.

The contributions of the paper are as follows:

  1. 1.

A soft pseudo-label generation method using multiple source datasets
considering the domain gap between the source datasets and the target dataset

  2. 2.

A training method using the soft pseudo-labels considering the entropy of the
labels

##  II Related Work

###  II-A Domain adaptation for semantic segmentation

DA is attracting attention as a method to workaround the necessity of manually
annotating the dataset of the target task. Specifically in training of DNNs
for semantic segmentation, there are two major approaches: domain alignment
and pseudo-label learning [7, 8]. The former is realized via minimizing
divergence metrics [9], adversarial learning [10, 11], etc. These two
approaches are not mutually exclusive and jointly used in many methods [12,
13].

Multi-source Domain Adaptation (MDA) for semantic segmentation has been
actively studied in the last few years. Zhao et al. [14] pioneered an MDA
method for semantic segmentation by extending the work by [10] to the multi-
source setting. He et al. [15] proposed to use multiple collaboratively
trained source models to generate pseudo-labels by simply summing their
predicted probabilities. This approach assumes that the source datasets are
equally similar to the target dataset, which is not the case in our problem
setting. In our previous work [6], we proposed a multi-source pseudo label
learning method specifically for training a model on greenhouse images
leveraging multiple publicly available datasets. Unlike [15], our previous
method showed effectiveness on transferring knowledge from source datasets
structurally dissimilar to the target dataset. However, the pseudo-labels
generated in [6] are based on unanimity of the source models, which inherently
discards a lot of pixels and is incapable of involving labels only in some of
the source datasets. Moreover, it does not consider similarities of the source
datasets with the target. The present work is an attempt to resolve the
limitation of the previous work and to gain better applicability of the
method.

###  II-B Domain gap evaluation

The domain gap, or domain shift, generally stems from the discrepancy between
data distributions of the two data domains. There have been several metrics to
measure the shift between two domains, such as Kulback-Leibler divergence
(KLD), Maximum Mean Discrepancy (MMD) [16], and
$\mathcal{H}\Delta\mathcal{H}$-divergence [17]. These metrics are often used
as an objective to minimize in domain adaptive tasks to learn domain-invariant
knowledge [18, 19].

Liu et al. [20] proposed a data-driven method of domain gap evaluation. While
the discrepancy metrics such as MMD are used as an objective of minimization
during adaptation, the method focuses on evaluating the discrepancy between
the source and the target dataset themselves. We employ this method to
evaluate relative domain gaps between different source datasets against the
target dataset.

##  III Preliminaries

###  III-A Notations

Formally, we assume $M$ labeled source datasets $S_{1},\cdots,S_{M}$ and an
unlabeled target dataset $S_{T}$. A source dataset $S_{i}$ is a set of $N_{i}$
input images $X_{i}=\\{x_{i,j}\\}^{N_{i}}_{j=1}$ and corresponding pixel-wise
semantic label maps $Y_{i}=\\{y_{i,j}\\}^{N_{i}}_{j=1}$ with $C_{i}$ classes.
The target dataset $S_{T}$ consists of a set of $N_{T}$ unlabeled images
$X_{T}=\\{x_{T,j}\\}^{N_{T}}_{j=1}$. Let $F\left(\cdot;\theta_{k}\right)$
denote a segmentation model with learnable weights $\theta_{k}$ trained on a
source dataset $S_{k}$. In addition, let
$\prescript{k}{}{p}_{i,j}\in\mathbb{R}^{H\times W\times C_{k}}$ and
${}^{k}f_{i,j}\in\mathbb{R}^{H\times W\times D}$ denote a pixel-wise object
probability and $D$-dimensional intermediate features produced by
$F\left(x_{i,j};\theta_{k}\right)$, respectively. For a tensor
$z\in\mathbb{R}^{H\times W\times C}$, $z^{\left(h,w,c\right)}$ denotes an
element at index $\left(h,w,c\right)$.

###  III-B Network architecture

The proposed method does not rely on a specific network architecture. It,
however, assumes two parallel segmentation decoders, namely main and auxiliary
branches, to enable uncertainty-based loss rectification [12, 6]. Let
$\prescript{k}{m}{p}_{i,j}$ and $\prescript{k}{a}{p}_{i,j}$ denote object
scores predicted by the main branch and the auxiliary branch, respectively.
The object probability $\prescript{k}{}{p}_{i,j}$ is specifically given as
follows [12]:

$\prescript{k}{}{p}_{i,j}=\text{Softmax}\left(\prescript{k}{m}{p}_{i,j}+w\prescript{k}{a}{p}_{i,j}\right)$
(1)

where $w$ is a weight parameter and is set to $0.5$ as in [12].

##  IV Multi-source soft pseudo-label generation
considering domain similarity

In this section, we describe the method of generating soft pseudo-labels
utilizing segmentation models pre-trained on the source datasets as the first
step of the proposed method.

###  IV-A Pseudo-label generation

First, we train semantic segmentation models using the source datasets.
Following [6], the models are trained using the ordinary cross entropy loss:

$L_{ce}\left(p,y\right)^{(h,w)}=-\sum_{c\in C}y^{\left(h,w,c\right)}\log
p^{\left(h,w,c\right)},$ (2)

where $p$ denotes the predicted pixel-wise probability distributions, and $y$
denotes a one-hot label map.

After pre-training, we generate pseudo-labels by integrating outputs on the
target images from the source models. The outputs are weighted with domain
similarity, i.e., inverse of domain gap between the source and the target
datasets. To evaluate relative domain gap, we employ a method by Liu et al.
[20] that uses the entropy of predictions as a measure of the domain gap. The
domain gap $\prescript{i}{}{G}_{j}$ between source dataset $S_{i}$ and a
target image $x_{T,j}$ is calculated as follows:

$\prescript{i}{}{G}_{j}=\frac{1}{\log{C_{i}}}E\left({}^{i}p_{T,j}\right),$ (3)

where $E\left(\cdot\right)$ is the entropy of the prediction defined as
follows:

$E\left({}^{i}p_{T,j}\right)=-\sum_{h,w}\sum^{C_{i}}_{c=1}{}^{i}p_{T,j}^{(h,w,c)}\log\left({}^{i}p_{T,j}^{(h,w,c)}\right).$
(4)

The larger $\prescript{i}{}{G}_{j}$ is, the farther the source dataset $S_{i}$
and the target image $x_{T,j}$ are. This method is based on an observation
that the large domain gap makes fuzzy classification scores [20] and leads to
a high entropy of the distribution. Since the value range of entropy depends
on the number of classes, the value is normalized by the maximum value, i.e.,
$\log{C_{i}}$.

Using the estimated domain gaps, soft pseudo-labels are generated as follows:

$\hat{y}_{j}^{(h,w)}=\text{Softmax}\left(\sum^{M}_{i=1}\frac{1}{\prescript{i}{}{G}_{j}}\psi_{i}\left({}^{i}p_{T,j}^{(h,w)}\right)\right),$
(5)

where $\psi_{i}:\mathbb{R}^{C_{i}}\rightarrow\mathbb{R}^{C_{T}}$ denotes a
function to convert a probability distribution in the source label space to
the target label space (see Fig. 2). The label mapping is heuristically
defined for each source as in [6].

Figure 2: Label conversion function $\psi$. The colors of the bars represent
the corresponding target labels. For each target class, the highest score
among a group of corresponding source classes is selected. The scores are then
normalized to form a probability distribution.

###  IV-B Analysis of the soft pseudo-labels

Fig. 3 shows examples of predictions from each source model, and the predicted
relative domain similarity (the inverse of domain gap) of the inputs, as well
as argmax and inverse label entropy of the generated soft pseudo-labels. The
first image of TUT Park dataset having a large area of plant and grass on both
sides was evaluated similar to the Forest dataset, which shares a similar
structure. In the second image, on the other hand, there is a large building,
which is seen more often in urban scenes, and it resulted in higher similarity
with Cityscapes. In a case where the similarity between the sources and the
target is difficult to evaluate due to large difference (e.g., greenhouse vs.
urban / outdoor scenes), the domain similarity serves as relative confidence
measures. For example, in the second example of Greenhouse A, the predictions
by CamVid and Forest models are fairly reasonable and the estimated
similarities are relatively high. Conversely, the Cityscapes model performs
poorly and exhibits lower similarity. From the observation above, it seems
reasonable to use the domain similarity scores as importance weights in
pseudo-label generation.

By merging the source predictions using the proposed method described above,
fairly accurate pseudo-labels are generated (as can be seen in the column of
“argmax”). In addition, entropy tends to be higher (a weight is lower) on
pixels on which all source models agree with each other. This observation
leads to our training method utilizing the inverse entropy of the soft pseudo-
labels, which is a soft alternative of the unanimity-based label selection
[6].

Notably in the examples of TUT Park dataset, the grass class is assigned in
the pseudo-labels though it is not present in CamVid dataset. In the previous
method [6], such flexible label assignment is not possible because of the
strict unanimity-based label selection. This characteristic allows for more
flexible choice of source datasets.

Figure 3: Predictions of the source models, domain similarity (inverse domain
gap), and pseudo-labels (labels with the maximum probability, and inverse of
label entropy). We used CamVid [2], Cityscapes [1], and Freiburg Forest [21]
as source datasets (descriptions about the datasets are in Table I). In
inverse entropy images, a darker pixel indicates a lower weight, i.e., higher
entropy.

##  V Network training

We train a target model using the soft pseudo-labels with the loss function
specifically tailored for the soft pseudo-labels. We also employ an existing
method [13] for training robust to misclassified pseudo-labels.

###  V-A Loss function

As a base classification loss, we employ symmetric cross-entropy (SCE) loss
[22], a variant of cross-entropy loss robust to noisy labels, following [12]:

$\displaystyle L_{sce}^{\left(h,w,c\right)}=\alpha
L_{ce}\left(\prescript{T}{}{p}^{\left(h,w,c\right)}_{T,j},\delta\left(\hat{y}_{j}^{(h,w)}\right)\right)$
$\displaystyle+\beta
L_{ce}\left(\delta\left(\hat{y}_{j}^{(h,w)}\right),\prescript{T}{}{p}^{\left(h,w,c\right)}_{T,j}\right),$
(6)

where $\alpha$ and $\beta$ are balancing parameters and set to $0.1$ and
$1.0$, respectively. $\delta\left(\cdot\right)$ denotes a function to convert
a soft label to one-hot label. In implementation, the one-hot label is clamped
to $[1e-4,1.0]$ to avoid numerical error [13].

Based on the observation in IV-B, we consider weighting loss values on each
pixel with the inverse value of label entropy. The pixel-wise weight is
calculated as follows:

$W^{(h,w)}_{j}=\exp{\left(-\lambda_{scale}\cdot
E\left(\hat{y}^{(h,w)}_{j}\right)\right)},$ (7)

where $\lambda_{scale}$ is a scaling parameter. Using eq. (7), we calculate a
weighted SCE loss:

$L_{w\\_sce}^{(h,w)}=W^{(h,w)}_{j}\cdot L_{sce}^{(h,w)}.$ (8)

This way, loss values on the pixels with low label entropy are weighted more,
and it results in a similar effect to the previous unanimity-based hard
pseudo-labels.

Following [6], we also introduce a loss rectification method based on the
pixel-wise uncertainty of the prediction proposed by [12] to suppress the
effect of pixels with high uncertainty, whose pseudo-labels are more likely to
be wrong. The uncertainty is estimated as Kulback-Leibler divergence between
the branches on each pixel as follows:

$L_{kld}^{\left(h,w\right)}=\sum_{c\in
C}\prescript{T}{m}{p}^{\left(h,w,c\right)}_{T,j}\log{\frac{\prescript{T}{m}{p}^{\left(h,w,c\right)}_{T,j}}{\prescript{T}{a}{p}^{\left(h,w,c\right)}_{T,j}}}.$
(9)

The rectified classification loss is defined as follows:

$L_{rect}^{(h,w)}=\exp{\left(-L_{kld}^{\left(h,w\right)}\right)}L_{w\\_sce}^{\left(h,w\right)}.$
(10)

Along with the classification loss, we add the entropy loss of the predictions
to force the network to clearly distinguish the classes, as used e.g., in
[11]:

$L_{ent}^{(h,w)}=E\left(\prescript{T}{}{p}^{(h,w)}_{T,j}\right),$ (11)

The overall loss is as follows:

$L_{all}=\sum_{h,w}\left(L_{rect}^{(h,w)}+\lambda_{ent}L_{ent}^{(h,w)}+\lambda_{kld}L_{kld}^{(h,w)}\right),$
(12)

where $\lambda_{ent}$ and $\lambda_{kld}$ denote balancing parameters.

###  V-B Prototype-based pseudo-label rectification

Along with the soft pseudo-labels, we employ ProDA [13] to make the training
process robust to noisy pseudo-labels. The core of the method is to save the
source model’s prediction as soft pseudo-labels
$\\{\hat{y}_{j}\\}_{j=1}^{N_{T}}$ and rectify them during training. The labels
are rectified based on the distance between the pixel feature and the
prototype, which is the representative feature for each class. See [13] for
details.

The feature-wise weights are calculated as follows:

$\omega^{(h,w,c)}_{j}=\frac{\exp\left(-\left\|\prescript{T}{}{\tilde{f}}^{(h,w)}_{j}-\eta^{(c)}\right\|/\tau\right)}{\sum_{c^{\prime}}\exp\left(-\left\|\prescript{T}{}{\tilde{f}}^{(h,w)}_{j}-\eta^{\left(c^{\prime}\right)}\right\|/\tau\right)},$
(13)

where $\eta^{(c)}$ denotes the prototype for class $c$, initialized as a mean
of the features predicted as $c$ and updated during training.
$\left\|\cdot\right\|$ denotes the Euclidean norm, and $\tau$ denotes a
temperature parameter that controls the degree of bias of the distribution.
$\prescript{T}{}{\tilde{f}}_{j}$ is a feature vector from a momentum encoder
[23], i.e., a model identical to $F\left(\cdot;\theta_{T}\right)$ whose
parameters are updated via exponential moving average (EMA).

Using $\omega_{j}$, the soft pseudo-labels are rectified as follows:

$\hat{y}^{\prime(h,w,c)}_{j}=\frac{\omega^{(h,w,c)}_{j}\hat{y}^{(h,w,c)}_{j}}{\sum_{c^{\prime}}\omega^{(h,w,c^{\prime})}_{j}\hat{y}^{(h,w,c^{\prime})}_{j}}\
$ (14)

In our method, the soft pseudo-labels are generated by the method described in
IV and rectified by eq. (14) in training.

##  VI Experiments

###  VI-A Experimental setting

####  VI-A1 Training environment

We used PyTorch implementation of ESPNetv2 [24] with modifications of adding
an auxiliary classification branch, and normalizing the features and
classification weights. The same architecture was used in the baseline
methods. The models were trained and evaluated on one NVIDIA Quadro RTX 8000
with 48GB RAM. The network is trained with an initial learning rate of
$2\times 10^{-2}$ and cyclic learning rate scheduling [25].

####  VI-A2 Datasets

As source datasets, we use CamVid (CV) [2], Cityscapes (CS) [1], and Freiburg
Forest (FR) [21]. As target datasets, we use Greenhouse A [6]; TUT Park [26],
images around an in-campus park; and Toyohashi Trail [26], images in
unstructured mountain paths in Toyohashi Nature Trail. The datasets are
summarized in Table I.

TABLE I: Datasets used in the experiments | Name | Train | Test | Description
---|---|---|---|---
Source |  Camvid (CV) [2] | 367 | 233 | Urban
Cityscapes (CS) [1] | 2970 | 500 | Urban
Freiburg Forest (FR) [21] | 230 | 136 | Unpaved outdoor
Target |  Greenhouse A [6] | 6689 | 100 | Tomato greenhouse
TUT Park [26] | 859 | 10 | Pavements in campus
Toyohashi Trail [26] | 9137 | 10 | Unpaved mountain

####  VI-A3 Baselines

In our comparative studies, we employ baseline methods as follows.

Training without DA As baselines without DA, we use two methods, namely
supervised and ensemble. In supervised, a segmentation model is trained with
the three source datasets with labels converted to the common target label set
using the label conversions. Ensemble merges the predicted object
probabilities from the source models in the same way as the proposed soft
pseudo-label generation. In other words, the soft pseudo-labels are directly
used as prediction results.

Single-source DA We also evaluate single-source DA methods. As source
datasets, we use CV, CS, and FR in Greenhouse A, and CS and FR in TUT Park
dataset. Pseudo-labels are generated from a model trained on each source
followed by label conversion $\psi_{i}$.

Multi-source DA We use our previous method [6] as a baseline of multi-source
DA, referred to as MSPL. We also use the method by He et al. [15], a state-of-
the-art MDA for semantic segmentation, hereafter referred to as MSDA_CL. We
used our own implementation of the method [15]. In multi-source DA, we
evaluate both double-source and triple-source settings. For double-source
training, we use CS and FR as source datasets.

###  VI-B Comparisons with the baselines

TABLE II: Comparison with the baselines on TUT Greenhouse A | Method | Class IoU | mIoU
---|---|---|---
|  Plant |  Artificial |  Ground
No adapt | Supervised | 75.4 | 64.8 | 56.7 | 65.6
Ensemble | 74.8 | 70.5 | 50.8 | 65.4
| CV | 71.0 | 62.7 | 50.7 | 61.5
Single | CS | 65.0 | 72.9 | 40.8 | 59.6
| FR | 67.2 | 52.9 | 37.8 | 52.7
|  MSDA_CL [15] | 70.9 | 57.9 | 52.8 | 60.5
Double |  MSPL [6] | 80.7 | 74.9 | 67.6 | 74.4
| Proposed | 79.9 | 76.4 | 57.4 | 71.2
|  MSDA_CL [15] | 70.5 | 62.1 | 43.9 | 58.9
Triple |  MSPL [6] | 80.7 | 78.2 | 72.6 | 77.2
| Proposed | 82.7 | 77.0 | 64.7 | 74.8

Greenhouse A dataset Table II shows the results of the baselines and the
proposed method. The performance of the proposed method did not reach those of
our previous method (MSPL). However, the proposed method resulted in the
second-best mean IoU and outperformed MSDA_CL [15]. In MSDA_CL, pseudo-labels
are generated by simply adding predicted scores from the source models,
followed by confidence-based label selection [7]. In contrast, the proposed
method considers domain similarity between the source datasets and the target.
Moreover, by loss weighting using the label entropy, the effect of noisy
pseudo-labels is suppressed, resulting in better performance. We further
evaluate the effect of considering domain similarity in pseudo-label
generation and label entropy weight in training in VI-C.

TABLE III: Comparison with the baselines on TUT Park | Method | Class IoU | mIoU
---|---|---|---
|  Plant |  Grass |  Artificial |  Ground |  Sky
No adapt | Supervised | 79.8 | 47.9 | 61.8 | 35.0 | 78.7 | 60.6
Ensemble | 82.8 | 37.5 | 68.2 | 23.9 | 77.3 | 57.9
Single | CS | 82.2 | 22.4 | 59.5 | 17.7 | 90.4 | 54.4
FR | 74.8 | 71.2 | 8.2 | 57.1 | 55.6 | 53.4
|  MSDA_CL [15] | 81.6 | 60.3 | 43.9 | 33.8 | 72.8 | 58.5
Double |  MSPL [6] | 79.7 | 65.3 | 45.4 | 44.2 | 76.9 | 62.3
| Proposed | 73.7 | 67.4 | 48.9 | 51.4 | 86.5 | 65.6
|  MSDA_CL [15] | 85.8 | 0.67 | 67.4 | 15.2 | 82.0 | 50.2
Triple |  MSPL [6] | 84.0 | 0.00 | 51.6 | 16.4 | 81.0 | 46.6
| Proposed | 83.5 | 62.0 | 58.9 | 37.2 | 88.2 | 65.9

TUT Park dataset In this target dataset, we use five object classes that
dominate the scenes, namely plant (e.g., trees), grass (e.g., ground
vegetation), artificial object, road, and sky. CS and FR have a distinction
between plant (vegetation in CS, and tree in FR) and grass (terrain in CS, and
grass in FR). CamVid, however, only has tree class that corresponds to plant
in the target dataset.

Table III shows the results. Interestingly, the proposed method trained with
three sources did not perform the best on any classes, but resulted in the
best mean IoU. While the baseline methods were biased towards a specific class
and poorly performed on others, the proposed method realized well-balanced
training. Notably, while MSPL by nature failed to capture the grass class due
to its absence in one of the source datasets (CV), the proposed method
successfully learned it. Unlike the previous method which excludes from
pseudo-labels the pixels on which the source models did not agree with each
other, our soft pseudo-labeling allowed for learning classes induced by only
part of source datasets.

TABLE IV: Comparison with the baselines on Toyohashi Trail | Method | Class IoU | mIoU
---|---|---|---
|  Plant |  Grass |  Artificial |  Ground |  Sky
No adapt | Supervised | 58.3 | 2.05 | 17.7 | 15.9 | 0.0 | 18.8
Ensemble | 69.0 | 1.34 | 0.14 | 0.28 | 0.0 | 14.1
Single | CS | 65.2 | 3.67 | 2.97 | 32.6 | 63.1 | 33.5
FR | 63.8 | 24.8 | 34.3 | 22.5 | 0.42 | 29.2
|  MSDA_CL [15] | 74.6 | 30.5 | 4.10 | 56.9 | 3.25 | 34.0
Double |  MSPL [6] | 70.4 | 9.36 | 14.7 | 50.2 | 0.75 | 29.1
| Proposed | 68.9 | 24.4 | 0.0 | 59.4 | 0.16 | 30.6
|  MSDA_CL [15] | 72.2 | 0.01 | 28.1 | 46.7 | 1.99 | 29.8
Triple |  MSPL [6] | 68.0 | 0.00 | 19.7 | 53.4 | 17.5 | 31.7
| Proposed | 71.3 | 33.4 | 19.6 | 59.7 | 1.06 | 37.0

Toyohashi Trail dataset We use the same label set as TUT Park dataset. Table
IV shows the comparative results. The proposed method resulted in the best
mean IoU. Although the domain gap estimation was highly unreliable due to
large domain gap between the all source datasets and the target datasets, and
thus it is difficult to give theoretical justification to the effect of
considering domain similarity, they show a better capability of the proposed
method to transfer knowledge even from source datasets that are very different
from the target dataset compared to the baselines.

###  VI-C Ablation studies

Next, we conducted an ablation study on loss weighting based on entropy of the
pseudo-labels during training, and the domain similarity-based integration of
source predictions in pseudo-label generation. We used TUT Park as the target
dataset, and trained the network using CV, CS, and FR.

Table V shows the results. Using domain similarity in pseudo-label generation
improved the performance in both training settings with and without the label
entropy weights. Fig. 4 shows an example of pseudo-label generated with and
without domain similarity weighting. While a large area of grass is wrongly
assigned with ground class when source predictions are simply summed as shown
in Fig. 4(c), part of the area is correctly classified when domain similarity
is incorporated as shown in Fig. 4(c). By considering domain similarity, the
prediction of a source model with larger domain similarity was emphasized and
pseudo-labels becomes more accurate. The effect of using domain similarity is
more evident when label entropy weight is not employed in training because the
improvement of pseudo-labels occurs on pixels where the source predictions are
fuzzy, which are suppressed in training by label entropy weight. We conclude
that incorporating domain similarity brings about positive effect on pseudo-
label generation.

TABLE V: Ablation study on training method Label entropy weight | Domain similarity | mIoU
---|---|---
|  | 60.7
| ✓ | 62.3
✓ |  | 65.8
✓ | ✓ | 65.9

((a)) Input

((b)) Ground truth

((c)) w/o domain similarity

((d)) w/ domain similarity

Figure 4: Comparison of pseudo-label generated with and without considering
domain similarity. (c) When domain similarity is not incorporated, a large
area of grass is misclassified as ground. (d) Some pixels misclassified in (c)
are classified correctly (inside the dotted ellipse).

##  VII Conclusion

We proposed a method of soft pseudo-label generation for training semantic
segmentation models on datasets of a variety of scenes without ground-truth
labels using not very relevant source datasets. Unlike our previous method [6]
using the unanimity criterion for pseudo-label selection, our method allows
for taking into account the domain similarity between each source dataset and
the target dataset, utilizing the information of the prediction certainty of
the source models, and involving target classes that does not have a
corresponding class in some source datasets, which increase the applicability
of the method to a variety of scenes.

## References

  * [1] Marius Cordts et al.  “The Cityscapes Dataset for Semantic Urban Scene Understanding”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ IEEE, 2016, pp. 3213–3223  DOI: 10.1109/CVPR.2016.350
  * [2] Gabriel J. Brostow et al.  “Semantic object classes in video: A high-definition ground truth database”  In _Pattern Recognition Letters_ 30.2, 2009, pp. 88–97  DOI: 10.1016/j.patrec.2008.04.005
  * [3] Stephan R. Richter et al.  “Playing for Data: Ground Truth from Computer Games”  In _European Conference on Computer Vision_ 9906, 2016, pp. 102–118  DOI: 10.1007/978-3-319-46475-6˙7
  * [4] German Ros et al.  “The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 3234–3243  DOI: 10.1109/CVPR.2016.352
  * [5] Alexey Dosovitskiy et al.  “CARLA: An Open Urban Driving Simulator”  In _Conference on Robot Learning_ PMLR, 2017, pp. 1–16  arXiv:1711.03938
  * [6] Shigemichi Matsuzaki et al.  “Multi-source pseudo-label learning of semantic segmentation for the scene recognition of agricultural mobile robots”  In _Advanced Robotics_ 36.19, 2022, pp. 1011–1029  DOI: 10.1080/01691864.2022.2109427
  * [7] Yang Zou et al.  “Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training”  In _European Conference on Computer Vision_ , 2018, pp. 297–313  DOI: 10.1007/978-3-030-01219-9˙18
  * [8] Yang Zou et al.  “Confidence Regularized Self-Training”  In _IEEE/CVF International Conference on Computer Vision_ IEEE, 2019, pp. 5981–5990  DOI: 10.1109/ICCV.2019.00608
  * [9] Kuniaki Saito et al.  “Maximum Classifier Discrepancy for Unsupervised Domain Adaptation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ IEEE, 2018, pp. 3723–3732  DOI: 10.1109/CVPR.2018.00392
  * [10] Judy Hoffman et al.  “CyCADA: Cycle-Consistent Adversarial Domain Adaptation”  In _International Conference on Machine Learning_ , 2018, pp. 1989–1998
  * [11] Tuan-Hung Vu et al.  “ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ IEEE, 2019, pp. 2512–2521  DOI: 10.1109/CVPR.2019.00262
  * [12] Zhedong Zheng and Yi Yang  “Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation”  In _International Journal of Computer Vision_ 129.4 Springer US, 2021, pp. 1106–1120  DOI: 10.1007/s11263-020-01395-y
  * [13] Pan Zhang et al.  “Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ IEEE, 2021, pp. 12409–12419  DOI: 10.1109/CVPR46437.2021.01223
  * [14] Sicheng Zhao et al.  “Multi-source Domain Adaptation for Semantic Segmentation”  In _Conference on Neural Information Processing Systems_ 32, 2019, pp. 7287–7300  arXiv: http://arxiv.org/abs/1910.12181
  * [15] Jianzhong He et al.  “Multi-Source Domain Adaptation with Collaborative Learning for Semantic Segmentation”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ IEEE, 2021, pp. 11003–11012  DOI: 10.1109/CVPR46437.2021.01086
  * [16] Arthur Gretton et al.  “A kernel two-sample test”  In _Journal of Machine Learning Research_ 13, 2012, pp. 723–773
  * [17] Shai Ben-David et al.  “A theory of learning from different domains”  In _Machine Learning_ 79.1-2, 2010, pp. 151–175  DOI: 10.1007/s10994-009-5152-4
  * [18] Mingsheng Long et al.  “Learning transferable features with deep adaptation networks”  In _International Conference on Machine Learning_ , 2015, pp. 97–105  arXiv:1502.02791
  * [19] Mingsheng Long et al.  “Unsupervised domain adaptation with residual transfer networks”  In _Conference on Neural Information Processing Systems_ , 2016, pp. 136–144  arXiv:1602.04433
  * [20] Xiaobin Liu and Shiliang Zhang  “Who is closer: A computational method for domain gap evaluation”  In _Pattern Recognition_ 122 Elsevier Ltd, 2022, pp. 108293  DOI: 10.1016/j.patcog.2021.108293
  * [21] Abhinav Valada et al.  “Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion”  In _International Symposium for Experimental Robotics_ , 2017, pp. 465–477  DOI: 10.1007/978-3-319-50115-4˙41
  * [22] Yisen Wang et al.  “Symmetric Cross Entropy for Robust Learning With Noisy Labels”  In _IEEE/CVF International Conference on Computer Vision_ 2019-Octob IEEE, 2019, pp. 322–330  DOI: 10.1109/ICCV.2019.00041
  * [23] Kaiming He et al.  “Momentum Contrast for Unsupervised Visual Representation Learning”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ IEEE, 2020, pp. 9726–9735  DOI: 10.1109/CVPR42600.2020.00975
  * [24] Sachin Mehta et al.  “ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ IEEE, 2019, pp. 9182–9192  DOI: 10.1109/CVPR.2019.00941
  * [25] Leslie N. Smith  “Cyclical Learning Rates for Training Neural Networks”  In _IEEE Winter Conference on Applications of Computer Vision_ IEEE, 2017, pp. 464–472  DOI: 10.1109/WACV.2017.58
  * [26] Yoshinobu Uzawa et al.  “Dataset Generation for Deep Visual Navigation in Unstructured Environments”  In _European Conference on Mobile Robots_ , 2023