# Uncertainty-aware Contrastive Distillation
for Incremental Semantic Segmentation

Guanglei Yang, Enrico Fini, Dan Xu, Paolo Rota, Mingli Ding∗, Moin Nabi,
Xavier Alameda-Pineda, Elisa Ricci∗ Guanglei Yang and Mingli Ding are with
School of Instrument Science and Engineering, Harbin Institute of Technology
(HIT), Harbin, China. E-mail<EMAIL_ADDRESS>∗Corresponding
author. Enrico Fini, Paolo Rota and Elisa Ricci are with the Department of
Information Engineering and Computer Science, University of Trento, Italy.
E-mail: {enrico.fini, paolo.rota<EMAIL_ADDRESS>author.
Dan Xu is with the Department of Computer Science and Engineering, Hong Kong
University of Science and Technology. E-mail<EMAIL_ADDRESS>Xavier
Alameda-Pineda is with the Perception Group, INRIA. E-mail: xavier.alameda-
<EMAIL_ADDRESS>Moin Nabi is with SAP AI Research, Berlin, Germany. E-mail:
<EMAIL_ADDRESS>Elisa Ricci is with Deep Visual Learning group at Fondazione
Bruno Kessler, Trento, Italy.

###### Abstract

A fundamental and challenging problem in deep learning is catastrophic
forgetting, i.e. the tendency of neural networks to fail to preserve the
knowledge acquired from old tasks when learning new tasks. This problem has
been widely investigated in the research community and several Incremental
Learning (IL) approaches have been proposed in the past years. While earlier
works in computer vision have mostly focused on image classification and
object detection, more recently some IL approaches for semantic segmentation
have been introduced. These previous works showed that, despite its
simplicity, knowledge distillation can be effectively employed to alleviate
catastrophic forgetting. In this paper, we follow this research direction and,
inspired by recent literature on contrastive learning, we propose a novel
distillation framework, Uncertainty-aware Contrastive Distillation (UCD). In a
nutshell, UCD is operated by introducing a novel distillation loss that takes
into account all the images in a mini-batch, enforcing similarity between
features associated to all the pixels from the same classes, and pulling apart
those corresponding to pixels from different classes. In order to mitigate
catastrophic forgetting, we contrast features of the new model with features
extracted by a frozen model learned at the previous incremental step. Our
experimental results demonstrate the advantage of the proposed distillation
technique, which can be used in synergy with previous IL approaches, and leads
to state-of-art performance on three commonly adopted benchmarks for
incremental semantic segmentation. The code is available at
https://github.com/ygjwd12345/UCD.

###### Index Terms:

Knowledge Distillation, Contrastive Learning, Incremental Learning, Semantic
Segmentation.

##  1 Introduction

As a longstanding, fundamental and challenging problem in computer vision,
semantic segmentation, i.e. the task of automatically assigning a class label
to each pixel of an image, has been an active research area for several
decades. In the recent years, we have witnessed substantial progress in this
area [1, 2, 3, 4, 5, 6], mostly thanks to the introduction of fully
convolutional networks [1] and to the integration of several other
architectural components, such as attention models [7, 8, 9] or multi-scale
fusion strategies [10, 11].

Figure 1: High-level view of our Uncertainty-aware Contrastive Distillation.
(Left) Given a minibatch of data sampled from the new task we use the old
model to generate pseudo-labels for the old classes. We then merge the pseudo-
labels with the ground truth for the current task in order to obtain the
extended semantic maps for all samples in the batch. (Right) Our contrastive
distillation approach operates by contrasting the features of the new and old
model based on the extended semantic map. Green arrows represent attraction,
red arrows represent repulsion. The contrast between features of the new model
with itself is depicted as a dashed arrow, while the contrast between new and
old model features is depicted as a solid arrow.

Despite the astonishing performance, state of the art deep architectures for
semantic segmentation still lack one fundamental capability: they cannot
update their learned model continuously as new data are made available without
losing previously learned knowledge. This is a matter of utmost importance in
several applications, such as autonomous driving or robot navigation. The
problem, also known as catastrophic forgetting [12], has been long studied in
the research community and several approaches, commonly referred as
Incremental Learning (IL) methods [13], have been developed over the years. In
computer vision, most research efforts in the area of IL have addressed
problems such as image classification [14, 15, 16] and object detection [17,
18]. However, so far, much less attention has been devoted to incremental
semantic segmentation.

Recently, due to the importance of it for several high level tasks, some
approaches have proposed to continuously update a deep network for computing
semantic segmentation maps [19, 20, 21, 22, 23]. While having their own
peculiarities, these methods develop from a common idea: overcoming
catastrophic forgetting by resorting on knowledge distillation [24], that is
aiming to preserve the output of a complex network when adopting a smaller
network. This idea can be transferred to IL [14] by enforcing that the
predictions of the network on the old tasks do not fluctuate much whilst
updating its parameters with samples of the new task. Despite their
simplicity, distillation-based techniques [19, 23] have proved effective for
alleviating catastrophic forgetting, even in the challenging task of semantic
segmentation.

One prominent limitation of previous distillation-based techniques for
semantic segmentation [19, 23] is that distillation is performed on a pixel-
basis, i.e. the output of the new model for a given pixel is only compared
with the output of the old model on the same pixel. This formulation is
derived as a trivial extension of existing IL methods for image classification
to segmentation. However, with this design choice, the crucial role of context
and the structured prediction nature of the task is severely overlooked.
Moreover, distilling on a pixel-basis does not explicitly take into account
the relationships between representations of pixels belonging to different
images. In this paper we propose a novel approach that allows to perform
knowledge distillation for incremental semantic segmentation while
contextualising the learned representation of a given pixel w.r.t. each and
every single pixel in a minibatch. Inspired from recent literature on
supervised contrastive learning [25], we introduce a constrastive distillation
loss that enforces similarity on the representations of the pixels of the same
class at batch level. Simultaneously, the contrastive distillation loss pushes
pixels of different classes far apart. In other words, our method brings
contrastive learning into distillation by considering all pixels of all images
of a mini batch as samples to be pulled together or pushed apart depending on
their class membership.

The key contribution of the present manuscript is to tie together the
advantages of knowledge distillation with those of contrastive learning. On
the one side, we distill knowledge from the previous task by pulling together
the representations of the current encoder to those of the past encoder for
pixels belonging to classes of the previous task. To do so, we leverage the
classifier of the previous task to provide pseudo-labels to the background of
the current task, enriching the information content of the segmentation map of
this latter task. On the other side, we use this enhanced segmentation map to
contrast pixel-level representations of the new and old representations. In
more detail, the new encoder is trained to pull together (push apart) pixel
representations of the same class (different classes), whether these
representations are extracted with the new or with the old task
encoder.111Obviously the old encoder is not trained during the current task.
Additionally, we exploit the noisy nature of the pseudo-labels by modeling
their uncertainty. More precisely, we re-weight the loss with the probability
of two pixels belonging to the same class (whatever this class is). We name
our method Uncertainty-aware Constrastive Distillation (UCD), and illustrate
the core idea of it in Fig. 1.

In summary, our contributions are the following:

  * •

We introduce a novel contrastive distillation loss that accounts for semantic
association among pixels of the new and old model.

  * •

We introduce an uncertainty estimation strategy for our contrastive learning
framework that leverages the joint probability of the pixels pairs belonging
to the same class and weights the strength of the distillation signal
accordingly.

  * •

We conduct an extensive evaluation on three publicly available benchmarks,
i.e. Pascal-VOC2012 [26], ADE20K [27] and Cityscapes [28], demonstrating that
our approach outperforms state-of-the-art IL methods for semantic
segmentation.

  * •

We demonstrate that UCD can be integrated effortlessly on top of other state
of the art methods, MiB [23] and PLOP [29], boosting their performance
significantly on all benchmarks.

##  2 Related Works

In this section, we review previous works on IL, specifically focusing on
research studies addressing the catastrophic forgetting issue for semantic
segmentation. As our paper also introduces a novel contrastive distillation
strategy, we also review recent works on contrastive learning and knowledge
distillation.

Incremental Learning. Enabling neural networks to learn continually is
currently one of the most challenging and fascinating problems in computer
vision. In the specific case of class incremental learning, the challenge is
to train a neural network on subsequent tasks, each of them including a
different set of categories, while avoiding catastrophic forgetting [12]. The
task information is assumed to be available at training time but unavailable
at test time. Existing works in the literature can be grouped into three main
categories: rehearsal-based methods [16, 30, 31, 32, 33], parameter isolation
approaches [34, 35] and distillation-based techniques [14, 36, 37]. Methods of
the first category leverage from data that have been retained from the
previous task or generated during the second task for counteracting
forgetting. Parameter isolation strategies exploit a reduced set of task-
specific parameters to incorporate knowledge about specific tasks.
Distillation-based methods use predictions of the model on old tasks to
transfer the knowledge from the old to the new model.

Regarding the latter category, recently, advanced distillation techniques have
been explored. For instance, Dhar et al. [38] proposed to penalize the changes
in attention maps. Differently, Douillard et al. [39] distilled the pooled
statistics of feature maps throughout the network, while Liu et al. [40]
developed an attentive feature distillation method in the context of object
detection. More recently, RECALL [41] employs a web crawler or a pre-trained
conditional GAN to replay images for the past semantic classes. However, none
of these works use contrastive learning to perform distillation. Unlike
Douillard et al. [39] and RECALL [41], which use pseudo labels generated by
the old model directly to supervise training new model, we propose an extended
semantic map to guide uncertainty-aware contrastive distillation.

Semantic Segmentation. Deep learning has enabled great advancements in
semantic segmentation [1, 42, 43]. Long et al. [1] were the first to introduce
fully convolutional networks (FCNs) for semantic segmentation, achieving
significant improvements over previous models. Dilated convolutions [42, 43]
were designed in order to increase the receptive field while learning deep
representations, further boosting performance. Recently, the new state-of-the-
art methods use different strategies to condition pixel-level annotations on
their global context, e.g. attention module [9] and multiple scales [44, 45].
On the other hand, [46] proposes a pixel-wise contrastive algorithm for
semantic segmentation in the fully supervised setting. Most of semantic
segmentation methods work in the offline setting, i.e. they assume that all
classes are available beforehand during the training phase. To the best of our
knowledge, the problem of incremental learning in semantic segmentation has
been addressed only in [19, 47, 20, 22]. Ozdemir et al. [22] proposed an
incremental learning approach for medical images, extending a standard image-
level classification method to segmentation and devising a strategy to select
relevant samples of old datasets for rehearsal. Tarasr et al. [20] introduced
a similar method for segmenting remote sensing data. Differently, Michieli
[19] considered incremental learning for semantic segmentation in a special
setting where labels are provided step by step. Moreover, they require the
novel classes to be never present as background in pixels of previous learning
steps. These requirements strongly limit the applicability of their method.
Here we follow a more principled formulation of the ICL problem in semantic
segmentation, like [23] .

Incremental Learning for Semantic Segmentation. Despite the flourishing
literature in IL, still very few works tackled the problem of semantic
segmentation [19, 20, 21, 22, 48, 23, 49]. For instance, in [21] Ozdemir et
al. proposed a method to extract prototypical examples for the old categories
to be used as a support while learning new categories. Michieli et al. [19,
50] introduced different distillation-based techniques working both on the
output logits and on intermediate features. The works in [22] and [20] also
considered a distillation framework to transfer knowledge between tasks,
introducing specific contributions associated to the considered problems in
remote sensing and medical imaging. Cermelli et al. [23] introduced a
distillation-based method which specifically accounts for the semantic
distribution shift of the background class. Very recently, Douillard et al.
[29] proposed an alternative mechanism to mitigate the background distribution
shift using pseudo-labels, together with multi-scale pooling distillation
adapted from [39]. Finally, Michieli et al. [51] proposed a framework based on
three components: prototypes matching, feature sparsity and an attraction-
repulsion mechanism at prototype-level.

In this work, we also resort to distillation to mitigate catastrophic
forgetting. However, we introduce a novel contrastive distillation loss which
successfully accounts for semantic dependencies among pixels and empirically
demonstrate that our contribution provides significantly improved performances
over state of the art methods. Importantly, our approach can be combined with
existing techniques [23, 29].

Contrastive Learning. Self-supervised learning has recently attracted
considerable attention in the computer vision community for its ability to
learn discriminative features using a contrastive objective [52, 46]. This
training scheme has been successfully employed in different fields such as
image and video classification [53, 25, 54], as well as natural language
processing [55, 56] and audio representation learning [57, 58, 59].
Contrastive learning consists in forcing the network to learn feature
representations by pushing apart different samples (negatives) or pulling
close similar ones (positives). The notion of similar and dissimilar samples
varies according to the task. For instance, in [53] an image and a simple
transformation of such image are considered positives, while in [60] positives
and negatives are generated using a momentum encoder. Contrastive learning has
also proved itself to be extremely competitive in supervised settings [25]. In
[25] the class labels are exploited to encourage proximity not only to
transformed versions of the same images but also to different images of the
same class, leading to competitive performance with respect to traditional
cross-entropy based classification methods. Very recently, in the context of
semantic segmentation, the advantage of contrastive pre-training was
demonstrated in [53] and a pixel-wise contrastive loss has been proposed [46]
showing promising results. SDR [51] firstly uses contrastive learning for
incremental semantic segmentation while it does not use the contrastive loss
in [46]. There is two main difference between our method and SDR: (i) we
compare feature vectors between themselves in the feature space, not with
prototypes, (ii) we use an uncertainty estimation mechanism to guide
contrastive learning.

##  3 Proposed Approach

In this section, we first provide a formalization of the problem of IL for
semantic segmentation. Then, we describe the proposed Uncertainty-aware
Contrastive Distillation (UCD).

###  3.1 Problem Definition and Notation

Let $\mathcal{I}$ denote the input image space, $I\in\mathcal{I}$ an image and
$\mathcal{Y}$ the label space. In semantic segmentation, given $I$ we want to
assign a label in $\mathcal{Y}$, representing its semantic category, to each
pixel of $I$. Pixels that are not assigned to other categories are assigned to
the background class B. The problem can be addressed within a supervised
framework defining a training set $\mathcal{T}=\\{I_{n},M_{n}\\}_{n=1}^{N}$ of
pairs of images $I_{n}$ and associated segmentation maps
$M_{n}\in\mathcal{Y}^{H\times W}$, where $H$ and $W$ denote the image height
and width respectively.

In the general IL setting, training is realized over multiple learning steps,
and this is also the case in IL for semantic segmentation. In the following
$k$ will denote the index of the incremental learning step. Each step is
associated to a different training set
$\mathcal{T}^{k}=\\{I^{k}_{n},M^{k}_{n}\\}_{n=1}^{N^{k}}$. At each learning
step $k$ we are interested in learning a model $\Phi^{k}$ able to segment the
classes of the label space of the $k$-th step, $\mathcal{Y}^{k}$. Importantly,
the training set with $N^{k}$ image-segmentation pairs is available only
during the training of the corresponding incremental step. Moreover,
$\Phi^{k}$ should be trained to mitigate catastrophic forgetting, that is to
be able to also recognise the semantic classes of the previous learning steps
$\cup_{k^{\prime}=1}^{k-1}\mathcal{Y}^{k^{\prime}}$ without having access to
$\cup_{k^{\prime}=1}^{k-1}(\mathcal{I}^{k^{\prime}},\mathcal{Y}^{k^{\prime}})$.
Following [23], we explore two settings for the splitting the datasets into
different tasks: (i) the overlapped setting, which only ensures label
separation across tasks
$\mathcal{Y}^{k}\cap\mathcal{Y}^{k^{\prime}}=\emptyset,\forall
k^{\prime}\in\\{1,...,k-1\\}$; and (ii) the disjoint setting, where in
addition to the label separation the same training image must not belong to
more than one task, formally: $\forall
I_{n}^{k}\in\mathcal{T}^{k},I_{n}^{k}\notin\cup_{k^{\prime}=1}^{k-1}\mathcal{T}^{k^{\prime}}$.

In this paper we assume that the model $\Phi^{k}$ is implemented by the
composition of a generic backbone $\psi_{\theta^{k}}$ with parameters
$\theta^{k}$ and a classifier $\phi_{\omega^{k}}$ with parameters
$\omega^{k}$. The features extracted with the backbone are denoted by
$f\in\mathbb{R}^{\frac{H}{16}\times\frac{W}{16}\times D}$, i.e.
$f^{k}=\psi_{\theta^{k}}(I)$, meaning that $f^{k}$ are extracted with the
backbone at the $k$-th step. The output segmentation map can be computed from
our pixel-wise predictor
$Y^{k}_{p}=\\{\arg\max(\operatorname{upsample}(\phi_{\omega^{k}}(f^{k}_{p})[p,c]))\\}_{p\in
I}$, where $\phi_{\omega^{k}}(f^{k}_{p})[p,c]$ is the probability for class
$c$ in pixel $p$ (which stands for a tuple $(H,W)$ where $H$ and $W$ are the
height and width of an image).

Figure 2: Overview of the proposed architecture at the incremental step $k$.
The grey blocks denote the network trained on old tasks, while the light blue
one indicates the network trained on old task. The proposed network is made of
a feature extractor and a classifier and it is trained on three losses. Our
uncertainty-aware contrastive distillation framework corresponds to the green
box. Downsample and Upsample denote the resolution decrease from $[{H},{W}]$
to $[\frac{H}{16},\frac{W}{16}]$ and the resolution increase
$[\frac{H}{16},\frac{W}{16}]$ to $[{H},{W}]$, respectively.

###  3.2 Overall Idea

Existing semantic segmentation frameworks are currently based on the use of
the cross-entropy loss to align the probability distributions predicted by the
new and old models in a pixel-wise fashion. However, this is suboptimal for
two reasons: (i) the cross-entropy loss compares pixel-wise predictions
independently and ignores relationships between pixels; (ii) due to the use of
the softmax, the cross-entropy loss only depends on the relationships among
logits and cannot directly supervise the learned feature representations.

A supervised contrastive framework [25] seems like a natural solution to
address these problems. In fact, as recently shown in [46], by employing a
contrastive loss, it is possible to explicitly model semantic relationships
between pixels and learn an embedding space by pulling representations of
pixel samples of the same class close and pushing those associated to
different classes apart.

In this paper, we propose to leverage from this idea and introduce a
distillation loss that does not operate in a pixel-wise fashion but rather
considers inter-pixel dependencies. In particular, we consider both
relationships among representations of the new model, and between
representations of the new and old models. We show that this enables better
knowledge transfer and mitigates catastrophic forgetting.

Figure 3: A detailed view of the k-th incremental learning step. The grey
modules correspond to the network learned at the previous incremental learning
step, which is frozen at step $k$ while the blue modules correspond to the
network trained at step $k$. $\tiny{U}⃝$ is an uncertainty estimation
operation. $\tiny{M}⃝$ is a merger operation.

The overall pipeline used in the $k$-th incremental learning step is
illustrated in Fig. 2. The two streams represent the backbone for the current
and previous learning steps respectively. Notably, at learning step $k$, the
backbone associated with the old tasks $\psi_{\theta^{k-1}}$ and the
classifier $\phi_{\omega^{k-1}}$ are frozen and used only during training.
When a new batch of data sampled from the current task $k$ is received, the
images are forwarded into both networks, producing two sets of features and
probability distributions. The old classifier outputs probabilities for old
classes only, while the new classifier also predicts the classes of the
current task. In order to train the new classifier we need to provide
supervision on all classes. Naturally, before introducing the contrastive
distillation loss we explain how to extend the semantic map using the pseudo-
labels for the old classes.

###  3.3 Extended Semantic Map

In this paper, we also make use of the pseudo-labels generated by the old
model to counteract catastrophic forgetting. The pseudo-labels are obtained as
follows:

$\tilde{M}^{k-1}_{n,p}=\arg\max(\phi_{\omega^{k-1}}(f^{k-1}_{n,p})).$ (1)

For each pixel in the image, $\tilde{M}^{k-1}$ contains the index of the most
likely class according to the old model. Note that, since the old model is
frozen during task $k$ it is not able to predict the classes in
$\mathcal{Y}^{k}$, in fact it will probably confuse them with the background.
However, since we have access to the ground truth for the current task, we can
use it to correct the pseudo-label. In practice, we superimpose the ground
truth (excluding the background) on top of the pseudo-labels, generating
$\bar{M}^{k}$, the Extended Semantic Map (ESM). This merger operation, denoted
as $\tiny{M}⃝$, can be summarised by the following equation:

$\bar{M}^{k}_{n,p}=\left\\{\begin{array}[]{ll}M^{k}_{n,p}&\text{ if
}M^{k}_{n,p}\neq 0,\\\ \tilde{M}^{k-1}_{n,p}&\text{otherwise}.\\\
\end{array}\right.$ (2)

With this simple mechanism, we are able to associate an ESM to each sample in
the batch, and virtually to the whole dataset, namely:
$\bar{\mathcal{T}}^{k}=\\{I^{k}_{n},\bar{M}^{k}_{n}\\}_{n=1}^{N^{k}}$. Note
that, since the pseudo-labels are estimated by a neural network, they might be
noisy. The uncertainty associated with the ESM will be handled in section 3.5,
but first we introduce the contrastive distillation loss.

Figure 4: Visualization of the pairwise similarity matrix employed for
efficient computation of the contrastive distillation loss. On the left side
of the matrix we have the anchor features that are being compared with the
contrast features on top. Blue and purple features and elements of the matrix
belong to the new and old tasks respectively. Grey and orange colors represent
the old and new feature extractors.

###  3.4 Contrastive Distillation Loss

With this extended semantic map, we can now construct a contrastive feature
distillation loss in the following way. For a given training image $I^{k}_{n}$
we can extract features with the previous and current backbones, and denote
them by $f^{k}_{n}$ and $f^{k-1}_{n}$ respectively. We must now interpret
these two tensors as a list of $\frac{H}{16}\times\frac{W}{16}$ features of
dimension $C$. At this point, we need to mask out the pixels that do not
contain interesting information. For $f^{k}_{n}$ we ignore the background
pixels (i.e. the pixels that do not belong to any of the new and old classes),
since they can contain multiple objects, making their representations very
heterogeneous. We do this at batch-level by collecting the following set of
indices:

$\mathcal{R}^{k}=\left\\{\left(n,p\right)\mid\bar{M}^{k}_{n,p}\neq 0,\,p\in
I^{k}_{n},n\in\mathcal{B}\right\\},$ (3)

where $p$ represents the coordinates of a pixel in an image and $\mathcal{B}$
is the current mini-batch. Similarly, for the features $f^{k-1}_{n}$ generated
by the old model we need to apply an analogous masking mechanism. However,
since the old model does not contain any knowledge about the new task, we need
to mask out the pixels that belong to the current task, in addition to the
background. As before, we gather the following indices:

$\mathcal{S}^{k}=\left\\{\left(n,p\right)\mid\bar{M}^{k}_{n,p}\notin\mathcal{Y}^{k}\cup\left\\{0\right\\},\,p\in
I^{k}_{n},n\in\mathcal{B}\right\\}.$ (4)

Apart from the reasons listed above, filtering the indices of the
representations of the old and new model into $\mathcal{R}^{k}$ and
$\mathcal{S}^{k}$ is also useful to reduce the computational footprint of our
contrastive distillation loss.

Figure 5: An intuitive visualization of Eq. 3, 4, 5 and 6

In order to mitigate catastrophic forgetting, we want to contrast the new and
old representations. Nonetheless, it is also critical that new features remain
consistent with themselves, especially for pixels of new classes. More
precisely, we examine all possible new-new pairs of features, i.e. both
features extracted from $\psi_{\theta^{k}}$, and all possible old-new pairs of
features, i.e. one feature extracted from $\psi_{\theta^{k}}$ and one from
$\psi_{\theta^{k-1}}$ (subject to their membership to $\mathcal{R}^{k}$ and
$\mathcal{S}^{k}$). To make them easier to grasp, we show an intuitive
visualization of $\mathcal{R}^{k}$ and $\mathcal{S}^{k}$ in Fig. 5. Moreover,
for each anchor feature we select, we need to construct a set of positives and
a set of negatives in order to perform the contrast. Given the index $a$ of an
anchor, we select as positives all the pixels that belong to the same class:

$\displaystyle G_{\mathcal{R}}^{k}(a)$
$\displaystyle=\left\\{f^{k}_{r}\mid\bar{M}^{k}_{a}=\bar{M}^{k}_{r},\,r\in\mathcal{R}^{k}\setminus\\{a\\}\right\\},$
(5) $\displaystyle G_{\mathcal{S}}^{k}(a)$
$\displaystyle=\left\\{f^{k-1}_{r}\mid\bar{M}^{k}_{a}=\bar{M}^{k}_{r},\,r\in\mathcal{S}^{k}\right\\},$
$\displaystyle G^{k}(a)$ $\displaystyle=G_{\mathcal{R}}^{k}(a)\cup
G_{\mathcal{S}}^{k}(a),$

where $G_{\mathcal{R}}^{k}(a)$ and $G_{\mathcal{S}}^{k}(a)$ are the positives
mined from the new and old features respectively, and $G^{k}(a)$ is the full
set of positives for $f_{a}$. Similarly for the negatives:

$\displaystyle H_{\mathcal{R}}^{k}(a)$
$\displaystyle=\left\\{f^{k}_{r}\mid\bar{M}^{k}_{a}\neq\bar{M}^{k}_{r},\,r\in\mathcal{R}^{k}\right\\},$
(6) $\displaystyle H_{\mathcal{S}}^{k}(a)$
$\displaystyle=\left\\{f^{k-1}_{r}\mid\bar{M}^{k}_{a}\neq\bar{M}^{k}_{r},\,r\in\mathcal{S}^{k}\right\\},$
$\displaystyle H^{k}(a)$ $\displaystyle=H_{\mathcal{R}}^{k}(a)\cup
H_{\mathcal{S}}^{k}(a)$

The positives $G^{k}(a)$ and negatives $H^{k}(a)$ are shown in Fig. 5 to
highlight the relationship among anchor, positives and negatives. Note that in
this second case we are selecting all the pixels that do not share the same
class with $a$ (negatives). We can now write the contrastive distillation loss
as follows:

$\mathcal{L}_{\textrm{cd}}=\frac{1}{|\mathcal{R}^{k}|}\sum_{a\in\mathcal{R}^{k}}\frac{-1}{|G^{k}(a)|}\sum_{f^{+}\in
G^{k}(a)}\mathcal{L}_{\textrm{lc}}\left(f_{a}^{k},f^{+},a\right),$ (7)

where $\mathcal{L}_{\textrm{lc}}$ is the log-contrast loss involving an anchor
feature $f_{a}^{k}$ generated by the new model, a positive feature $f_{g}$ and
the set of negatives for the anchor, namely:

$\mathcal{L}_{\textrm{lc}}=\log\frac{\displaystyle\exp\left({\delta(f_{a}^{k},f^{+})}/{\tau}\right)}{\displaystyle\sum_{f^{-}\in
H^{k}(a)}\exp\left({\delta(f_{a}^{k},f^{-})}/{\tau}\right)},$ (8)

where $\delta$ denotes the cosine similarity
$\delta\left(u,v\right)=\frac{u}{||u||}\cdot\frac{v}{||v||}$ and $\tau$ is the
temperature of the contrast. In practice to compute the contrastive
distillation loss efficiently using modern deep learning libraries we compute
a matrix multiplication over the whole batch to obtain the pairwise similarity
matrix (see Fig. 4). Each element in the matrix contains the cosine similarity
between two pixels. Then we sum over the rows using masks for positives and
negatives to obtain the loss value.

Overall, $\mathcal{L}_{\textrm{cd}}$ pushes the features of all same-class
pixels together, while pulling the pixels with different class apart. Very
importantly, the proposed loss considers all features within the same batch as
potential pairs, thus adding many more positive pairs, and regularising de
facto the learning. We recall that the feature extractor from the previous
learning step is frozen. Therefore, features $f^{k-1}$ are also frozen and
used to attract the new features $f^{k}$ towards a representation that encodes
the classes of the previous learning step, hence the name contrastive
distillation loss.

###  3.5 Uncertainty-aware Contrastive Distillation

As discussed in the previous section, the contrastive distillation loss
requires to have pseudo-labels of the semantic classes of the previous
incremental learning steps. In order to achieve this, we exploit the semantic
classifier learned at the previous step, that outputs a per-class probability
tensor $P^{k-1}\in[0,1]^{\frac{H}{16}\times\frac{W}{16}\times T^{k-1}}$,
$\sum_{l=1}^{T^{k-1}}P^{k-1}_{p,l}=1,\forall pcx$ (recall that $T^{k-1}$ is
the total number of classes until step $k-1$). This probability map is
extended with the ground-truth at learning step $k$ in the following way:

$\bar{P}_{p}^{k}=\left\\{\begin{array}[]{ll}[P_{p}^{k-1},0_{L^{k}}]&\text{ if
}M_{p}^{k}=0,\\\ \mathbb{I}_{M_{p}^{k}}&\text{otherwise},\\\
\end{array}\right.$ (9)

where, $0_{d}$ is the zero vector of dimension $d$ and $\mathbb{I}_{c}$ is a
one-hot vector at coordinate $c$. In other words, the new probability tensor
copies from the previous one when the ground-truth points to the background,
and is a one-hot vector with the ground-truth class otherwise.

Therefore, the extended probability tensor is $\bar{P}^{k}\in[0,1]^{H\times
W\times T^{k}}$. By arg-maxing this tensor over the classes, we obtain the
pseudo-labels in the extended semantic map $\bar{M}_{n}^{K}$. The direct
application of these pseudo-labels in the contrastive distillation loss
discussed in Eq. (7). Our intuition is that the arg-max operation takes a
hard-decision that does not account for the uncertainty. We have therefore
developed a principled strategy to account for the uncertainty in the
classification of each pixel using the previous backbone.

The extended probability map can used be also to produce an estimate of the
uncertainty of the pseudo-label. Indeed, one can compute the probability of
two pixels $a$ (anchor) and $g$ (positive) belonging to the same class at
incremental step $k$ using:

$\sigma_{a,g}^{k}=\bar{P}^{k}_{a}\cdot\bar{P}^{k}_{g}.$ (10)

This formulation naturally takes into account two important aspects that are
particularly relevant for contrastive learning: (i) the confidence of the
network in predicting a certain class; (ii) the alignment of the two
probability distributions. if the network is not confident (high entropy) or
the alignment is poor then $\sigma_{a,g}^{k}$ is low, reducing the strength of
the attraction. This mechanism intrinsically accounts for the uncertainty,
reducing the noise and therefore leading to better performance.

Finally, we can use this probability to modify $\mathcal{L}_{\textrm{cd}}$ in
(7) into $\mathcal{L}_{\textrm{ucd}}$ – the uncertainty-aware contrastive
distillation – defined as:

$\mathcal{L}_{\textrm{ucd}}=\frac{1}{|\mathcal{R}^{k}|}\sum_{a\in\mathcal{R}^{k}}\frac{-1}{|G(a)|}\sum_{f^{+}_{g}\in
G(a)}\sigma_{a,g}^{k}\mathcal{L}_{\textrm{lc}},$ (11)

where $g$ is the pixel index associated to $f^{+}_{g}$.

We also summary all step of in Fig 3 and explain more detail about contrastive
distillation with uncertainty (red block in Fig 3).

###  3.6 UCD as a generic framework

The proposed uncertainty-aware contrastive distillation framework can be
integrated seamlessly on top of virtually any other baseline. In the following
we describe how to integrate UCD with two state of the art methods.

MiB-UCD. Recently, Cermelli et al. [23] proposed Modeling the Background for
Incremental Learning (MiB), a distillation-based approach to learn the
predictor $\phi_{\theta^{k}}$ at step $k$ given a training set
$\mathcal{T}^{k}$ by distilling knowledge using the predictions of model
$\phi_{\theta^{k-1}}$ on the previous step. In particular, they propose to
minimize the following loss function:

$\mathcal{L}=\frac{1}{|\mathcal{T}^{k}|}\sum_{(I_{n}^{k},M_{n}^{k})\in\mathcal{T}^{k}}\left(\mathcal{L}_{\textrm{seg}}(I_{n}^{k},M_{n}^{k})+\lambda\mathcal{L}_{\textrm{kd}}(I_{n}^{k})\right)$
(12)

where $\lambda>0$ is a hyperparameter balancing the importance of two loss
terms $\mathcal{L}_{\textrm{seg}}(x,y)$ and $\mathcal{L}_{KD}(x)$. These are
defined in order to specifically take into account the role of the background
class B, which contains different semantic categories over subsequent time
steps. In particular, the loss $\mathcal{L}_{\textrm{seg}}$ is a revisited
cross-entropy loss defined as follows:

$\mathcal{L}_{\textrm{seg}}(I_{n}^{k},M_{n,p}^{k})=-\frac{1}{N}\sum_{p\in
I}\log\tilde{\phi}_{\omega^{k}}(f_{n}^{k})[p,M_{n,p}^{k}]\,,$ (13)

where $f_{n}^{k}=\psi_{\theta^{k}}(I_{n}^{k})$ and:

$\tilde{\phi}_{\omega^{k}}(f_{n}^{k})[p,c]=\begin{cases}\phi_{\omega^{k}}(f_{n}^{k})[p,c]\;\;&\text{if}\
c\neq 0\,,\\\
\sum_{q\in\mathcal{{Y}}^{t-1}}\phi_{\omega^{k}}(f_{n}^{k})[p,q]\;\;&\text{if}\
c=0\,.\end{cases}$ (14)

This loss is meant to compensate the fact that the training set
$\mathcal{T}^{k}$ might include also pixels associated to the categories
previously observed in the old task.

Furthermore, a distillation loss [24] is introduced in order to encourage
$\phi_{\omega^{k}}$ to produce probabilities close to the ones produced by
$\phi_{\omega^{k-1}}$. The distillation loss writes:

$\small\mathcal{L}_{\textrm{kd}}(I_{n}^{k})=-\frac{1}{|I|}\sum_{i\in
I}\sum_{c\in\mathcal{Y}^{k-1}}\phi_{\omega^{k-1}}(f_{n}^{k-1})[i,c]\log\hat{\phi}_{\omega^{k}}(f_{n}^{k})[i,c]\,,$
(15)

where:

$\hat{\phi}_{\omega^{k}}(f_{n}^{k})[p,c]=\begin{cases}{\phi}_{\omega^{k}}(x)[p,c]\;\;&\text{if}\
c\neq\mathtt{B}\\\
\sum_{q\in\mathcal{Y}^{k}}{\phi}_{\omega^{k}}(f_{n}^{k})[p,q]\;\;&\text{if}\
c=\mathtt{B}\,.\end{cases}$ (16)

This loss is intended to account for the fact that $\phi_{\theta^{k-1}}$ might
predict as background pixels of classes that we are currently trying to learn.
This aspect is crucial to perform a correct distillation of the old model into
the new one.

Nonetheless, distilling the output probabilities might not be sufficient in
order to defy catastrophic forgetting. For this reason, MiB seems to be a
natural candidate for enhancement using UCD. The overall loss MiB-UCD is:

$\mathcal{L}=\mathcal{L}_{\textrm{seg}}+\lambda_{\textrm{kd}}\mathcal{L}_{\textrm{kd}}+\lambda_{\textrm{ucd}}\mathcal{L}_{\textrm{ucd}},$
(17)

PLOP-UCD. A different approach for incremental learning in semantic
segmentation was explored in [29]. While MiB [23] performs distillation on
output probabilities, Pseudo-label and LOcal POD (PLOP) [29] proposes to use
intermediate representations to transfer richer information. This is achieved
using a technique called Pooled Output Distillation (POD) [39] that was
already shown to work well in classification settings. In addition, in order
to preserve details that would be neglected by the pooling, PLOP uses a multi-
scale distillation scheme. In practice, for every layer $l$ of the network,
pooled features (Local POD embeddings) are collected at several scales and
concatenated to obtain a feature vector $\mathcal{F}_{l}$. This process is
carried out on both old and current models, and then the concatenated features
are compared using an 2 loss:

$\mathcal{L}_{\textrm{pod}}=\frac{1}{L}\sum_{l=1}^{L}\left\|\mathcal{F}_{l}^{k}-\mathcal{F}_{l}^{k-1}\right\|^{2},$
(18)

where $L$ is the total number of layers of the network.

Besides Local POD, PLOP also introduces a simpler way of solving background
shift: a pseudo-labeling strategy for background pixels. Predictions of the
old model for background pixels are used as clues regarding their real class,
if they belong to any of the old classes. An argmax operation is applied on
every pixel, subject to a class-specific confidence threshold. In other words,
in the case of non-background pixels they copy the ground truth label;
otherwise, if the old model is confident enough, the most likely class
predicted by the old model is selected. The pseudo-label $S$ is then used in
the classification loss as follows:

$\mathcal{L}_{\text{pseudo }}=-\frac{\nu}{|I|}\sum_{p\in
I}\log\phi_{\omega^{k}}(f_{n}^{k})[p,S_{n,p}]\,,$ (19)

where $\nu$ is the ratio of accepted old classes pixels over the total number
of such pixels.

Combining these two ideas, PLOP was shown to be able to outperform MiB in the
overlapped setting, while its superiority on the harder disjoint setting is
still unclear. Nonetheless, we believe PLOP can take advantage of our
contrastive distillation loss. In fact, although it already performs
distilation of intermediate features, PLOP lacks two critical aspects: (i) it
does not take into account the relationships between pixel representations for
different images (ii) it does not use the pseudo-label for feature
distillation. Also, the way PLOP estimates the confidence of the pseudo-label
is very primitive and cursed with class-specific hyper-parameter tuning. On
the contrary, our UCD loss naturally accounts for the uncertainty in a more
elegant way. The overall loss of PLOP-UCD is the following:

$\mathcal{L}=\mathcal{L}_{\textrm{pseudo}}+\lambda_{\textrm{pod}}\mathcal{L}_{\textrm{pod}}+\lambda_{\textrm{ucd}}\mathcal{L}_{\textrm{ucd}},$
(20)

TABLE I: Mean IoU on the Pascal-VOC 2012 dataset for different incremental
class learning scenarios. $*$ means results come from re-implementation.
${\dagger}$ means a updated version. $\sharp$ means the conditional GAN model
pretrained on ImageNet [61] is used. UDC means uncertainty-aware contrastive
distillation. Best among table in bold, best among part in underlined.

Method | 19-1 | 15-5 | 15-1
---|---|---|---
Disjoint | Overlapped | Disjoint | Overlapped | Disjoint | Overlapped
1-19 | 20 | all | 1-19 | 20 | all | 1-15 | 16-20 | all | 1-15 | 16-20 | all | 1-15 | 16-20 | all | 1-15 | 16-20 | all
FT | 5.8 | 12.3 | 6.2 | 6.8 | 12.9 | 7.1 | 1.1 | 33.6 | 9.2 | 2.1 | 33.1 | 9.8 | 0.2 | 1.8 | 0.6 | 0.2 | 1.8 | 0.6
PI[62] | 5.4 | 14.1 | 5.9 | 7.5 | 14.0 | 7.8 | 1.3 | 34.1 | 9.5 | 1.6 | 33.3 | 9.5 | 0.0 | 1.8 | 0.4 | 0.0 | 1.8 | 0.4
EWC[15] | 23.2 | 16.0 | 22.9 | 26.9 | 14.0 | 26.3 | 26.7 | 37.7 | 29.4 | 24.3 | 35.5 | 27.1 | 0.3 | 4.3 | 1.3 | 0.3 | 4.3 | 1.3
RW[63] | 19.4 | 15.7 | 19.2 | 23.3 | 14.2 | 22.9 | 17.9 | 36.9 | 22.7 | 16.6 | 34.9 | 21.2 | 0.2 | 5.4 | 1.5 | 0.0 | 5.2 | 1.3
LwF[14] | 53.0 | 9.1 | 50.8 | 51.2 | 8.5 | 49.1 | 58.4 | 37.4 | 53.1 | 58.9 | 36.6 | 53.3 | 0.8 | 3.6 | 1.5 | 1.0 | 3.9 | 1.8
LwF-MC[16] | 63.0 | 13.2 | 60.5 | 64.4 | 13.3 | 61.9 | 67.2 | 41.2 | 60.7 | 58.1 | 35.0 | 52.3 | 4.5 | 7.0 | 5.2 | 6.4 | 8.4 | 6.9
ILT[19] | 69.1 | 16.4 | 66.4 | 67.1 | 12.3 | 64.4 | 63.2 | 39.5 | 57.3 | 66.3 | 40.6 | 59.9 | 3.7 | 5.7 | 4.2 | 4.9 | 7.8 | 5.7
ILT†[50] | 69.4 | 16.5 | 66.7 | 67.4 | 12.4 | 64.7 | 63.3 | 39.6 | 57.4 | 66.4 | 40.8 | 60.0 | 4.4 | 6.4 | 4.9 | 5.5 | 8.0 | 6.1
SDR[51] | 69.9 | 37.3 | 68.4 | 69.1 | 32.6 | 67.4 | 73.5 | 47.3 | 67.2 | 75.4 | 52.6 | 69.9 | 59.2 | 12.9 | 48.1 | 44.7 | 21.8 | 39.2
RECALL♯[41] | 65.2 | 50.1 | 65.8 | 67.9 | 53.5 | 68.4 | 66.3 | 49.8 | 63.5 | 66.6 | 50.9 | 64.0 | 66.0 | 44.9 | 62.1 | 65.7 | 47.8 | 62.7
UCD | 73.4 | 33.7 | 71.5 | 71.4 | 47.3 | 70.0 | 71.9 | 49.5 | 66.2 | 77.5 | 53.1 | 71.3 | 53.1 | 13.0 | 42.9 | 49.0 | 19.5 | 41.9
MiB[23] | 69.6 | 25.6 | 67.4 | 70.2 | 22.1 | 67.8 | 71.8 | 43.3 | 64.7 | 75.5 | 49.4 | 69.0 | 46.2 | 12.9 | 37.9 | 35.1 | 13.5 | 29.7
MiB+SDR[51] | 70.8 | 31.4 | 68.9 | 71.3 | 23.4 | 69.0 | 74.6 | 44.1 | 67.3 | 76.3 | 50.2 | 70.1 | 59.4 | 14.3 | 48.7 | 47.3 | 14.7 | 39.5
MiB+UCD | 74.3 | 28.4 | 72.0 | 73.7 | 34.0 | 71.7 | 73.0 | 46.2 | 66.3 | 78.5 | 50.7 | 71.5 | 53.3 | 14.4 | 43.5 | 51.9 | 13.1 | 42.2
PLOP∗ [29] | 75.1 | 38.2 | 73.2 | 75.0 | 39.1 | 73.2 | 66.5 | 39.6 | 59.8 | 74.7 | 49.8 | 68.5 | 49.0 | 13.8 | 40.2 | 65.2 | 22.4 | 54.5
PLOP+ UCD | 75.7 | 31.8 | 73.5 | 75.9 | 39.5 | 74.0 | 67.0 | 39.3 | 60.1 | 75.0 | 51.8 | 69.2 | 50.8 | 13.3 | 41.4 | 66.3 | 21.6 | 55.1
Joint | 77.4 | 78.0 | 77.4 | 77.4 | 78.0 | 77.4 | 79.1 | 72.6 | 77.4 | 79.1 | 72.6 | 77.4 | 79.1 | 72.6 | 77.4 | 79.1 | 72.6 | 77.4

TABLE II: Mean IoU on the Pascal-VOC 2012 dataset for more incremental class learning scenarios. $*$ means results come from re-implementation. $\sharp$ means the conditional GAN model pretrained on ImageNet [61] is used. UDC means uncertainty-aware contrastive distillation. Best among table in bold, best among part in underlined. Method | 10-10 | 10-1
---|---|---
Disjoint | Overlapped | Disjoint | Overlapped
1-10 | 11-20 | all | 1-10 | 11-20 | all | 1-10 | 11-20 | all | 1-10 | 11-20 | all
FT | 7.7 | 60.8 | 33.0 | 7.8 | 58.9 | 32.1 | 6.3 | 2.0 | 4.3 | 6.3 | 2.8 | 4.7
LwF[14] | 63.1 | 61.1 | 62.2 | 70.7 | 63.4 | 67.2 | 6.7 | 6.5 | 6.6 | 16.6 | 14.9 | 15.8
LwF-MC[16] | 52.4 | 42.5 | 47.7 | 53.9 | 43.0 | 48.7 | 6.9 | 1.7 | 4.4 | 11.2 | 2.5 | 7.1
ILT[19] | 67.7 | 61.3 | 64.7 | 70.3 | 61.9 | 66.3 | 14.1 | 0.6 | 7.5 | 16.5 | 1.0 | 9.1
RECALL♯[41] | 62.6 | 56.1 | 60.8 | 65.0 | 58.4 | 63.1 | 58.3 | 46.0 | 53.9 | 59.5 | 46.7 | 54.8
MiB[23] | 66.9 | 57.5 | 62.4 | 70.4 | 63.7 | 67.2 | 14.9 | 9.5 | 12.3 | 15.1 | 14.8 | 15.0
MiB+SDR[51] | 67.5 | 57.9 | 62.9 | 70.5 | 63.9 | 67.4 | 25.5 | 15.7 | 20.8 | 26.3 | 19.7 | 23.2
MiB+UCD | 65.0 | 58.7 | 61.9 | 71.8 | 65.2 | 68.5 | 33.1 | 26.1 | 30.6 | 33.7 | 26.5 | 31.1
PLOP∗ [29] | 61.8 | 53.1 | 57.5 | 65.0 | 58.8 | 61.9 | 39.5 | 13.9 | 26.7 | 40.0 | 21.7 | 30.8
PLOP+ UCD | 63.0 | 55.8 | 59.4 | 71.5 | 58.9 | 65.2 | 42.6 | 15.0 | 28.8 | 42.3 | 28.3 | 35.3
Joint | 78.6 | 76.0 | 77.4 | 78.6 | 76.0 | 77.4 | 78.6 | 76.0 | 77.4 | 78.6 | 76.0 | 77.4

(a) Image

(b) ILT [19]

(c) PLOP [29]

(d) PLOP+UCD

(e) MiB [23]

(f) MiB+UCD

(g) GT

Figure 6: Qualitative results on the VOC 2012 dataset (15-5 disjoint setting).
Figure 7: Qualitative results on the VOC 2012 dataset (15-1 overlapped
setting)

##  4 Experiments

In this section we first introduce the datasets used for testing UCD against
state of the art approaches. Our experimental results are then shown and
illustrated. Additional ablation experiments are also provided, aiming to show
the impact of every module included in the proposed architecture.

###  4.1 Datasets

PASCAL-VOC 2012. The Pascal VOC2012 dataset [26] contains 11,530
training/validation images of a variable size and with semantic labels
correspondint to 20 classes plus background. Following previous works [19, 17,
23, 20], we propose two different experimental settings: disjoint and
overlapped. In the first setting [19], each learning step contains a unique
set of images, whose pixels belong to classes seen either in the current or in
the previous learning steps. In other words, each image is seen in only one
learning step. On the contrary, in the overlapped setting [17], a few classes
appear on both tasks. Additionally, results are proposed in multiple
experiments, using different class-incremental procedures: the first adds only
one class during the new task (19-1), the second adds 5 classes at once (15-5)
and the third adds 5 classes one at the time averaging the final results
(15-1). Results are reported in terms of mIoU.

ADE20K. ADE20K [27] is a large-scale dataset including 150 classes. Unlike
Pascal-VOC 2012 it comprises also non-object classes such as sky, ground,
water and wall to name a few, resulting in a more fluid label set. The results
are presented using disjoint and overlapped sets following the experimental
setup proposed in [23, 29]: addition of 50 classes in one single step
(100-50), multi-step addition of 50 classes (100-10) and three steps of 50
classes each (50-50). Coherently with the VOC dataset, all classes that are
not considered in the ground truth of the current task fall into the
background macro-class. Results are presented using the mIoU metrics.

Cityscapes. The Cityscapes dataset [28] has been built for Autonomous Driving
applications. The release includes sets of images with different labeling
accuracy, and since this is the first time the dataset has been used for
Incremental Semantic Segmentation, we adopt the 3,475 finely annotated images,
divided into 2,975/500 images for training and validation respectively.
Following the setup of other datasets proposed in related works [23], results
are presented using the ovelapped setting where tasks are arranged following
two strategies: six classes added at once in the new task (13-6) and adding
one of the 6 classes at a time and average the final result (13-1). As in
previous cases the final results are reported in mIoU.

###  4.2 Network Details

The proposed method is implemented in PyTorch. The experiments are conducted
on two Nvidia RTX 6000 GPUs during training while one for testing. The feature
extractor used in this work includes a backbone network and a decoder. In
detail, the ResNet-101 [64] is chosen as backbone while features are decoded
using the Deeplab-v3 architecture [11]. The weights are initialized on
ImageNet [65] and trained using momentum and weight decay proposed in [11].
The initial learning rate is set to $10^{-2}$ for the first learning step,
then it is diminished for the following steps to $10^{-3}$ for PASCAL-VOC 2012
and Cityscapes, while it is kept unchanged ($10^{-2}$) for ADE20K. We train
the model with a batch-size of 24 for 30 epochs for Pascal-VOC 2012 and 60
epochs for ADE20K and Cityscapes in every learning step. The input image is
cropped to meet the input dimension of $512\times 512$ and transformed using
the augmentation protocol proposed in [11]. The temperature value of the
contrastive loss has been set to $0.07$ (additional details are shown in
Section 4.3.2), while the weighting parameters $\lambda_{ucd}$,
$\lambda_{pod}$and $\lambda_{kd}$ are set to $0.01$, $0.01$ and $10$
respectively.

###  4.3 Experimental Results

Results are presented in this section, first in terms overall accuracy against
the current state of the art (i.e. Tables I-III-V, secondly with an extensive
ablation study to weigh the importance of every single component proposed in
this work. As further reference, two additional experiments are reported to
serve as a lower and upper reference: in the first the new task is Fine Tuned
(FT) starting from the old model magnifying the catastrophic forgeting
phenomena typical of neural networks in this setting, while the second is a
joint training (Joint) of both tasks, therefore breaking the incremental
paradigm hypothesis.

TABLE III: Mean IoU on the ADE20K dataset for different incremental class
learning disjoint setting.$*$ means results come from re-implementation. UDC
means uncertainty-aware contrastive distillation. Best among table in bold,
best among part in underlined.

Method | 100-50 | 100-10 | 50-50
---|---|---|---
1-100 | 101-150 | all | 1-100 | 100-110 | 110-120 | 120-130 | 130-140 | 140-150 | all | 1-50 | 51-100 | 101-150 | all
FT | 0.0 | 24.9 | 8.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 16.6 | 1.1 | 0.0 | 0.0 | 22.0 | 7.3
LwF [14] | 21.1 | 25.6 | 22.6 | 0.1 | 0.0 | 0.4 | 2.6 | 4.6 | 16.9 | 1.7 | 5.7 | 12.9 | 22.8 | 13.9
LwF-MC [16] | 34.2 | 10.5 | 26.3 | 18.7 | 2.5 | 8.7 | 4.1 | 6.5 | 5.1 | 14.3 | 27.8 | 7.0 | 10.4 | 15.1
ILT [19] | 22.9 | 18.9 | 21.6 | 0.3 | 0.0 | 1.0 | 2.1 | 4.6 | 10.7 | 1.4 | 8.4 | 9.7 | 14.3 | 10.8
Inc.Seg [49] | 36.6 | 0.4 | 24.6 | 32.4 | 0.0 | 0.2 | 0.0 | 0.0 | 0.0 | 21.7 | 40.2 | 1.3 | 0.3 | 14.1
SDR [51] | 37.4 | 24.8 | 33.2 | 28.9 | - | - | - | - | - | 21.7 | 40.9 | - | - | 29.5
UCD | 40.4 | 27.3 | 36.0 | 28.6 | 13.0 | 13.1 | 9.2 | 10.7 | 16.1 | 23.2 | 39.3 | 25.3 | 19.1 | 27.9
MiB [23] | 37.9 | 27.9 | 34.6 | 31.8 | 10.4 | 14.8 | 12.8 | 13.6 | 18.7 | 25.9 | 35.5 | 22.2 | 23.6 | 27.0
MiB + SDR [51] | 37.5 | 25.5 | 33.5 | 28.9 | - | - | - | - | - | 23.2 | 42.9 | - | - | 31.3
MiB + UCD | 40.5 | 28.1 | 36.4 | 33.4 | 15.2 | 15.3 | 10.8 | 12.5 | 18.8 | 27.1 | 40.2 | 25.9 | 19.5 | 28.5
PLOP∗ [29] | 29.8 | 4.2 | 22.2 | 32.1 | 1.9 | 10.0 | 0.8 | 1.2 | 0.1 | 22.3 | 19.2 | 0.4 | 0.4 | 6.6
PLOP + UCD | 33.2 | 4.7 | 23.7 | 35.7 | 2.1 | 11.1 | 0.9 | 1.4 | 0.1 | 24.8 | 21.3 | 0.4 | 0.4 | 7.4
Joint | 44.3 | 28.2 | 38.9 | 44.3 | 26.1 | 42.8 | 26.7 | 28.1 | 17.3 | 38.9 | 51.1 | 38.3 | 28.2 | 38.9

TABLE IV: Mean IoU on the ADE20K dataset for different incremental class learning overlapped setting. $*$ means results come from PLOP. UDC means uncertainty-aware contrastive distillation. Best among table in bold.  method | 100-50 | 100-10 | 50-50
---|---|---|---
1-100 | 101-150 | mIoU | 1-100 | 101-150 | mIoU | 1-50 | 51-150 | mIoU
ILT∗[19] | 18.29 | 14.40 | 17.00 | 0.11 | 3.06 | 1.09 | 3.53 | 12.85 | 9.70
MiB∗[23] | 40.52 | 17.17 | 32.79 | 38.21 | 11.12 | 29.24 | 45.57 | 21.01 | 29.31
PLOP [29] | 41.87 | 14.89 | 32.94 | 40.48 | 13.61 | 31.59 | 48.83 | 20.99 | 30.40
PLOP+UCD | 42.12 | 15.84 | 33.31 | 40.80 | 15.23 | 32.29 | 47.12 | 24.12 | 31.79
TABLE V: Mean IoU on the Cityscapes dataset for different incremental class learning scenarios. UDC means uncertainty-aware contrastive distillation. Best among table in bold, best among part in underlined. method | 13-6 | 13-1
---|---|---
1-13 | 14-19 | mIoU | 1-13 | 14-19 | mIoU
FT | 0.0 | 33.4 | 10.6 | 0.0 | 7.4 | 2.3
ILT [19] | 52.3 | 18.3 | 41.5 | 4.2 | 9.1 | 5.8
PLOP [29] | 53.2 | 10.1 | 39.6 | 52.4 | 15.1 | 40.6
PLOP+UCD | 53.4 | 10.0 | 39.7 | 52.7 | 15.3 | 40.9
MiB [23] | 52.8 | 17.9 | 41.8 | 51.6 | 22.9 | 42.5
MiB+UCD | 53.0 | 18.6 | 42.1 | 52.2 | 23.4 | 43.1
Joint | 53.5 | 54.2 | 53.7 | 53.5 | 54.2 | 53.7

####  4.3.1 Comparison with state of the art methods

Comparison on VOC 2012. Table I shows the performance of UCD with respect to
competing method for class-incremental learning including PI [62], EWC [15],
RW [63], LwF [14], LwF-MC [16] and works specially designed for image
segmentation such as ILT [19], MiB [23], PLOP [29] and SDR [51]. As mentioned
earlier, fine tuning and joint training are also reported as a lower and upper
bound reference.

(a) Image

(b) ILT [19]

(c) PLOP [29]

(d) PLOP+UCD

(e) MiB [23]

(f) MiB+UCD

(g) GT

Figure 8: Qualitative results on the ADE 20K dataset (100-50 disjoint
setting).

Specifically, PLOP has been modified in order to not consider the background
class in the accuracy computation for sake of fairness in the results
comparison with the competing works. The training model proposed in this work
(i.e. UCD) can be applied on top of other methods therefore Table I shows
results of the best performing incremental learning algorithms for semantic
segmentation evaluated using the UCD procedure. Results show that UCD
generally improves the performances of other standalone models (i.e. MiB and
PLOP), in both overlapped and disjoint settings. However, UCD has a weaker
performance than MiB+SDR [51],mainly in disjoint setup of VOC 2012. The first
reason is that SDR uses attraction and repulsion losses with prototypes
(calculated with a running average), which is fundamentally different from the
contrastive loss. Averaging features into prototypes changes balancing between
the classes in the loss. Probably, this makes results of old classes perform
better because old classes already account for the majority of the total
classes. Another reason is that in some cases it is difficult for the network
to generate good extended semantic maps in the disjoint setting. This probably
affects UCD more than SDR, because SDR counteracts this noise by using running
averages and has additional regularization mechanisms. Another can notice that
neither of the two methods mentioned above is outperforming the other in all
settings, while adding UCD is generally beneficial. Moreover, we also compare
our with the SOTA on more incremental class learning scenarios in Table II.
For more challenging settings (10-10 and 10-1), UCD surpasses MiB+SDR and
PLOP. This reinforces the idea that UCD tends to perform better when the
number of old classes and new classes are more balanced. Qualitative results
are shown in Figure 6 and Figure 7 for 15-5 and 15-1 scenarios respectively.
Note that classes such as human, bike, and cow are included in the old task
but are still well represented in the resulting segmentation map when UCD is
involved. However, the improvements brought by UCD is larger for MiB than
PLOP, even in thecases where MiB was better than PLOP. One reason is, in most
of the settings, PLOP already achieves very good results, outperforming MiB by
a large margins. This makes it hard for UCD to produce an extra improvement
over PLOP. In addition, it seems PLOP is not well suited for the disjoint
setting and therefore makes UCD less effective as well. Finally, since PLOP
already has a feature distillation mechanism, it might be that the two feature
distillation losses take care of similar aspects of knowledge transfer.

Comparison on ADE20K. Quantitative results on ADE20K are reported in Table III
and Table IV. Following the previous works [23, 29, 66] the comparison is
performed under three different settings differentiating mainly in how the new
task data is handled: 100 classes in the old task and 50 classes at once for
the new one (100-50), same number of old classes and 50 classes in batches of
10 for 5 times (100-10), an old task of 50 classes and then two batches of 50
classes each as new task (50-50). In this dataset UCD still demonstrates to
improve results of base methods on the majority of the scenarios. In the
specific the classes between 120-140 are characterized by small objects such
as light, food, pots, dishes, hood and vase, resulting in an extremely
localized pixel contrastive that is probably not as efficient as for other
sets of classes. Figure 8 presents some qualitative results for the 100-50
disjoint setting. Large classes of the old task such as sky, building and car
are well recognized with the UCD method while smaller objects are penalized by
the smaller set of points to perform contrastive distillation.

Comparison on Cityscapes. In this work we introduce a new benchmark scenario
for incremental learning for semantic segmentation based on cityscapes
dataset. In the specific two settings, similar to those in previously treated
datasets, have been proposed: both considering 13 classes for the old task but
one testing 6 classes (i.e., car, truck, bus, train, motorcycle, bicycle) at
once for the new task (13-6) and the other adding once class at the time
(13-1) and averaging the final results over the remaining 6 classes.
Cityscapes does not allow test on a disjoint setting so we only propose the
overlapped one. Unlike ADE20K and VOC, the dataset shows a more coherent
scenario moving the performance of base methods toward the Joint upper bound.
However, also in this ideal scenario the contribute of UDC is beneficial in
almost all the settings. Some qualitative results of the (13-6) setting are
presented in Figure 9. According to 2nd-4th rows in Figure 9, adding UCD into
MiB’s and PLOP’s frame works effectively solves background over-fitting caused
by category forgetting.

(a) Image

(b) ILT [19]

(c) PLOP [29]

(d) PLOP+UCD

(e) MiB [23]

(f) MiB+UCD

(g) GT

Figure 9: Qualitative results on the Cityscapes dataset (13-6 overlapped
setting).

####  4.3.2 Ablation study

TABLE VI: Ablation study of the proposed method on the Pascal-VOC 2012 15-5
disjoint setup and ADE 20K 100-50 disjoint setup. “pixel contrastive” stands
for simply applying contrastive learning in the current task; “CD” means
contrastive distillation; ”UCD” means uncertainty-aware contrastive
distillation loss.

Method | ADE | VOC
---|---|---
1-100 | 101-150 | mIoU | 1-15 | 16-20 | mIoU
baseline | 37.5 | 27.0 | 34.0 | 65.4 | 41.4 | 59.4
\+ segment contrastive | 38.1 | 27.0 | 34.4 | 67.6 | 42.6 | 61.4
\+ pixel contrastive | 38.5 | 27.1 | 34.7 | 69.2 | 43.8 | 62.9
\+ CD | 39.9 | 27.7 | 35.8 | 72.6 | 43.5 | 65.3
\+ UCD | 40.5 | 28.1 | 36.4 | 73.0 | 46.2 | 66.3

To demonstrate the contribution of each component of our model, in this
section we show the results of two specific studies where these components are
added one by one in UCS and evaluated accordingly.

Figure 10: Mean IoU over time evolution with different temperature of
contrastive loss on VOC 2012 15-1 disjoint setup.  TABLE VII: Ablation study
of uncertainty-aware contrastive distillation loss. Old, new and bg mean the
features belong to old task, new task or background. Memory means GPU’s memory
consumption during training, measured by GB. Time means time consumption to
complete the training, measured by minutes.

Anchor | Contrast | Memory | Time | mIoU
---|---|---|---|---
New model | New model | Old model
Old | New | BG | Old | New | BG | Old | New | BG
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 63.4 | 147 | 66.9
$\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ |  | 46.5 | 108 | 66.5
$\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ |  | 41.8 | 78 | 66.3

TABLE VIII: Different temperature study of the proposed method on the Pascal-
VOC 2012 15-5 disjoint setup and ADE 20K 100-50 disjoint setup.

Temperature | ADE | VOC
---|---|---
1-100 | 101-150 | mIoU | 1-15 | 16-20 | mIoU
1 | 37.0 | 25.7 | 33.2 | 66.6 | 42.2 | 60.5
0.1 | 38.3 | 26.6 | 34.4 | 69.0 | 43.7 | 62.6
0.07 | 40.5 | 29.4 | 36.8 | 73.0 | 46.2 | 66.3
0.05 | 40.0 | 27.7 | 35.9 | 72.0 | 45.6 | 65.4
0.01 | 40.0 | 27.8 | 36.0 | 72.2 | 45.7 | 65.5

TABLE IX: Ablation study of $\lambda_{ucd}$ on the Pascal-VOC 2012 15-5 and
19-1 disjoint setup .

$\lambda_{ucd}$ | 15-5 | 19-1
---|---|---
1-15 | 16-20 | mIoU | 1-19 | 20 | mIoU
0 | 65.5 | 41.4 | 59.4 | 66.8 | 15.2 | 64.2
0.1 | 72.6 | 43.4 | 65.3 | 73.5 | 28.1 | 71.3
0.01 | 73.0 | 46.2 | 66.3 | 73.9 | 28.2 | 71.7
0.001 | 71.7 | 42.9 | 64.5 | 73.3 | 28.0 | 71.0
0.0001 | 68.4 | 42.0 | 61.7 | 69.7 | 21.5 | 67.3

Table VI shows the results of our ablation on different contrastive losses. We
start from a baseline model that includes only knowledge distillation at
class-probability level and cross-entropy loss as in [23]. In the subsequent
rows, several additional components are added on top of the baseline gradually
to investigate the performance effect of each component. Specifically, as
shown in the table, the pixel contrastive refers to a supervised contrastive
loss on the new task features, $f^{k}$. Differently, the segment contrastive
indicates the strategy using a contrastive loss applied by averaging the input
features for each pixel within the same class. The CD and UCD components
indicate the plugging of the proposed contrastive distillation and
uncertainty-aware contrastive distillation, respectively. Specifically, the
full contrastive distillation loss, $\mathcal{L}_{cd}$, is used for the
component of CD. Finally, the contrastive distillation loss with uncertainty
is used for the component of UCD. As can be observed from Table VI, our
results clearly demonstrate that the pixel contrastive loss outperforms the
segment contrastive loss on both datasets. This motivates our choice of
designing a contrastive distillation loss in UCD based on pixel contrastive
loss. In Table VI, CD outperforms the standard pixel-wise contrastive loss.
Moreover, the results also show that the proposed contrastive distillation
loss with uncertainty estimation and the mitigation of catastrophic forgetting
significantly performs better than the CD, with an improvement on mIoU by 0.5%
on ADE 20K 100-50 and by 1.0% on Pascal-VOC 2012 15-5, respectively. All these
ablation studies concretely verifies the effectiveness of the proposed
approach.

After ablation on different contrastive losses, we further make ablation study
on component of uncertainty-aware contrastive distillation loss in Tab. VII.
In detail, we run experiments with and without background class in the
contrastive loss. According to results in Tab. VII, We find the performance
without old class to be comparable. However, without background , the number
of pixels used in the contrastive loss is much smaller, resulting in GPU’s
memory consumption dropping significantly. Therefore we concluded that it is
better to remove the background in the contrastive distillation loss. For the
same reason, we also remove the old task feature of old model features in our
method.

We also provide a sensitivity study. The evaluation of the performance effect
of the temperature value $\tau$ and the number of learned classes is shown in
Table VIII and Figure. 10, respectively. It clearly shows the importance of
using a proper temperature value. Varying the value leads to certain variance
in the final performance. Nevertheless, the performance decreasing is
consistent with the increasing of the number of seen classes. Lastly, Fig. 11
shows the mIoU performance with the increase of the batch size on Pascal-VOC
2012 15-5 disjoint setup. This study confirms what found in previous other
tasks [53, 60], that is contrastive learning benefits the network optimization
with larger batch sizes, except for very small batch size such as 2 and 4. We
could also observe from the figure, our method steadily outperforms the main
competitor (i.e. MiB) at different batch-size values. Therefore, the
effectiveness of the proposed approach is independent of the batch size setup.
Meanwhile, performance does not drastically improve with larger batch size
while increasing the batch size also increases the memory usage. For fair
comparison with other methods like MiB and PLOP, the same batch size (24) has
been chosen for all the compared methods.

Figure 11: Mean IoU over time evolution with different batch size on VOC 2012
15-5 disjoint setup.

##  5 Conclusion

We presented a new uncertainty-aware contrastive distillation framework for
incremental semantic segmentation, leveraging from the complementarity of
pixel-wise contrastive learning and uncertainty estimation. Using our
approach, the model captures global semantic relationships between old and new
features through a contrastive formulation, significantly mitigating
catastrophic forgetting with respect to previous art. In order to perform
feature contrastive learning on the old classes, we used pseudo-labels, which
can be noisy. To address this issue we devised an uncertainty estimation
mechanism that leverages the joint probability of the pixel pairs belonging to
the same class. This makes the network focus on high-agreement pixel-pairs and
suppress the low-entropy pixel-pairs, leading to more effective distillation.
Our extensive experimental evaluation demonstrated outstanding performance of
our method in several datasets (VOC 2012, ADE20K and cityscape) and settings
(14 in total). Future works will focus on exploiting other continual learning
approaches departing from traditional distillation schemes, as well as
considering the segmentation problem in presence of domain shift [67].

## Acknowledgments

The support provided by China Scholarship Council (CSC) during a visit of
Guanglei Yang to University of Trento is acknowledged. This work has been also
supported by Caritro Foundation.

## References

  * [1] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _CVPR_ , 2015.
  * [2] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _ECCV_ , 2018.
  * [3] G. Lin, A. Milan, C. Shen, and I. Reid, “Refinenet: Multi-path refinement networks for high-resolution semantic segmentation,” in _CVPR_ , 2017.
  * [4] Z. Zhang, X. Zhang, C. Peng, X. Xue, and J. Sun, “Exfuse: Enhancing feature fusion for semantic segmentation,” in _ECCV_ , 2018.
  * [5] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _CVPR_ , 2017.
  * [6] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang, “Bisenet: Bilateral segmentation network for real-time semantic segmentation,” in _ECCV_ , 2018\.
  * [7] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _CVPR_ , 2018.
  * [8] L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille, “Attention to scale: Scale-aware semantic image segmentation,” in _CVPR_ , 2016.
  * [9] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _CVPR_ , 2019.
  * [10] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _TPAMI_ , 2017.
  * [11] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv_ , 2017.
  * [12] M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in _Psychology of learning and motivation_ , 1989.
  * [13] M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “Continual learning: A comparative study on how to defy forgetting in classification tasks,” _arXiv_ , 2019.
  * [14] Z. Li and D. Hoiem, “Learning without forgetting,” _TPAMI_ , 2017.
  * [15] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska _et al._ , “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the national academy of sciences_ , 2017.
  * [16] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in _CVPR_ , 2017.
  * [17] K. Shmelkov, C. Schmid, and K. Alahari, “Incremental learning of object detectors without catastrophic forgetting,” in _ICCV_ , 2017.
  * [18] X. Liu, H. Yang, A. Ravichandran, R. Bhotika, and S. Soatto, “Multi-task incremental learning for object detection,” in _CVPR_ , 2020.
  * [19] U. Michieli and P. Zanuttigh, “Incremental learning techniques for semantic segmentation,” in _ICCV Workshops_ , 2019.
  * [20] O. Tasar, Y. Tarabalka, and P. Alliez, “Incremental learning for semantic segmentation of large-scale remote sensing data,” _J-STARS_ , 2019.
  * [21] F. Ozdemir, P. Fuernstahl, and O. Goksel, “Learn the new, keep the old: Extending pretrained models with new anatomy and images,” in _MICCAI_ , 2018\.
  * [22] F. Ozdemir and O. Goksel, “Extending pretrained segmentation networks with additional anatomical structures,” _IJCARS_ , 2019.
  * [23] F. Cermelli, M. Mancini, S. R. Bulo, E. Ricci, and B. Caputo, “Modeling the background for incremental learning in semantic segmentation,” in _CVPR_ , 2020.
  * [24] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv_ , 2015.
  * [25] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” _NeurIPS_ , 2020\.
  * [26] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _IJCV_ , 2010.
  * [27] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in _CVPR_ , 2017.
  * [28] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _CVPR_ , 2016.
  * [29] A. Douillard, Y. Chen, A. Dapogny, and M. Cord, “Plop: Learning without forgetting for continual semantic segmentation,” _CVPR_ , 2021.
  * [30] H. Shin, J. K. Lee, J. Kim, and J. Kim, “Continual learning with deep generative replay,” in _NeurIPS_ , 2017.
  * [31] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in _CVPR_ , 2019.
  * [32] O. Ostapenko, M. Puscas, T. Klein, P. Jahnichen, and M. Nabi, “Learning to remember: A synaptic plasticity driven framework for continual learning,” in _CVPR_ , 2019.
  * [33] C. Wu, L. Herranz, X. Liu, J. van de Weijer, B. Raducanu _et al._ , “Memory replay gans: Learning to generate new categories without forgetting,” in _NeurIPS_ , 2018.
  * [34] A. Mallya and S. Lazebnik, “Packnet: Adding multiple tasks to a single network by iterative pruning,” in _CVPR_ , 2018.
  * [35] A. Mallya, D. Davis, and S. Lazebnik, “Piggyback: Adapting a single network to multiple tasks by learning to mask weights,” in _ECCV_ , 2018.
  * [36] F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in _ECCV_ , 2018.
  * [37] E. Fini, S. Lathuilière, E. Sangineto, M. Nabi, and E. Ricci, “Online continual learning under extreme memory constraints,” in _ECCV_ , 2020.
  * [38] P. Dhar, R. V. Singh, K.-C. Peng, Z. Wu, and R. Chellappa, “Learning without memorizing,” in _CVPR_ , 2019.
  * [39] A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “Podnet: Pooled outputs distillation for small-tasks incremental learning: Supplementary material,” _ECCV_ , 2020.
  * [40] X. Liu, H. Yang, A. Ravichandran, R. Bhotika, and S. Soatto, “Multi-task incremental learning for object detection,” _arXiv_ , 2020.
  * [41] A. Maracani, U. Michieli, M. Toldo, and P. Zanuttigh, “Recall: Replay-based continual learning in semantic segmentation,” _ICCV_ , 2021.
  * [42] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _arXiv_ , 2016.
  * [43] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” _arXiv_ , 2015.
  * [44] Y. Yuan, X. Chen, and J. Wang, “Object-contextual representations for semantic segmentation,” in _ECCV_ , 2020.
  * [45] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang _et al._ , “Deep high-resolution representation learning for visual recognition,” _TPAMI_ , 2020.
  * [46] W. Wang, T. Zhou, F. Yu, J. Dai, E. Konukoglu, and L. Van Gool, “Exploring cross-image pixel contrast for semantic segmentation,” _arXiv_ , 2021.
  * [47] Y. Wang, W. Zhou, T. Jiang, X. Bai, and Y. Xu, “Intra-class feature variation distillation for semantic segmentation,” in _ECCV_ , 2020.
  * [48] Y. Feng, X. Sun, W. Diao, J. Li, X. Gao, and K. Fu, “Continual learning with structured inheritance for semantic segmentation in aerial imagery,” _TGRS_ , 2021.
  * [49] S. Yan, J. Zhou, J. Xie, S. Zhang, and X. He, “An em framework for online incremental learning of semantic segmentation,” _arXiv_ , 2021.
  * [50] U. Michieli and P. Zanuttigh, “Knowledge distillation for incremental learning in semantic segmentation,” _CVIU_ , 2021.
  * [51] ——, “Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations,” _CVPR_ , 2021.
  * [52] G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann, “Contrastive adaptation network for unsupervised domain adaptation,” in _CVPR_ , 2019.
  * [53] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _ICML_ , 2020.
  * [54] M. Kim, J. Tack, and S. J. Hwang, “Adversarial self-supervised contrastive learning,” _NeurIPS_ , 2020.
  * [55] Y. Qu, D. Shen, Y. Shen, S. Sajeev, J. Han, and W. Chen, “Coda: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding,” _arXiv_ , 2020.
  * [56] T. Klein and M. Nabi, “Contrastive self-supervised learning for commonsense reasoning,” in _ACL_ , 2020.
  * [57] A. Saeed, D. Grangier, and N. Zeghidour, “Contrastive learning of general-purpose audio representations,” _arXiv_ , 2020.
  * [58] L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk, “Approximate nearest neighbor negative contrastive learning for dense text retrieval,” _arXiv_ , 2020.
  * [59] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” _arXiv_ , 2020.
  * [60] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _CVPR_ , 2020.
  * [61] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_ , 2009.
  * [62] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” _PLMR_ , 2017.
  * [63] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr, “Riemannian walk for incremental learning: Understanding forgetting and intransigence,” in _ECCV_ , 2018.
  * [64] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [65] S. Rota Bulò, L. Porzi, and P. Kontschieder, “In-place activated batchnorm for memory-optimized training of dnns,” in _CVPR_ , 2018.
  * [66] A. Iscen, J. Zhang, S. Lazebnik, and C. Schmid, “Memory-efficient incremental learning through feature adaptation,” _arXiv_ , 2020.
  * [67] M. Mancini, S. R. Bulò, B. Caputo, and E. Ricci, “Best sources forward: domain generalization through source-specific nets,” in _ICIP_ , 2018.

|  Guanglei Yang received the B.S. degree in instrument science and technology
from Harbin Institute of Technology (HIT), Harbin, China, in 2016. He is
currently pursuing the Ph.D degree in the School of Instrumentation Science
and Engineering, Harbin Institute of Technology(HIT), Harbin, China. He is
working at University of Trento as a visiting student from 2020 to now. His
research interests mainly include domain adaption, pixel-level prediction and
attention gate.
---|---
|  Enrico Fini is a Ph.D. student at the University of Trento. His research
focuses on continual learning and self-supervised learning. He received the
B.S. degree in computer engineering from the University of Parma, Italy, in
2015 and the M.S. degree in computer science and engineering from Politecnico
di Milano, Italy, in 2019. In 2018, He spent one year at the European Space
Astronomy Centre of the European Space Agency in Madrid, Spain, working on
machine learning for automatic sunspot detection.
---|---
|  Dan Xu is an Assistant Professor in the Department of Computer Science and
Engineering at HKUST. He was a Postdoctoral Research Fellow in VGG at the
University of Oxford. He was a Ph.D. in the Department of Computer Science at
the University of Trento. He was also a research assistant of MM Lab at the
Chinese University of Hong Kong. He received the best scientific paper award
at ICPR 2016, and a Best Paper Nominee at ACM MM 2018. He served as Area
Chairs of ACM MM 2020, WACV 2021 and ICPR 2020.
---|---
|  Paolo Rota is an assistant professor (RTDa) at University of Trento (in the
MHUG group), working on computer vision and machine learning. He received his
PhD in Information and Communication Technologies from the University of
Trento in 2015. Prior joining UniTN he worked as Post-doc at the TU Wien and
at the Italian Institute of Technology (IIT) of Genova. He is also
collaborating with the ProM facility of Rovereto on assisting companies in
inserting machine learning in their production chain.
---|---
|  Mingli Ding received the B.S., M.S. and Ph.D. degrees in instrument science
and technology from Harbin Institute of Technology (HIT), Harbin, China, in
1996, 1997 and 2001, respectively. He worked as a visiting scholar in France
from 2009 to 2010. Currently, he is a professor in the School of
Instrumentation Science and Engineering at Harbin Institute of Technology.
Prof. Ding’s research interests are intelligence tests and information
processing, automation test technology, computer vision, and machine learning.
He has published over 40 papers in peer-reviewed journals and conferences.
---|---
|  Moin Nabi is a Principal Research Scientist at SAP AI Research in Berlin.
Before that, he was a post-doctoral research fellow at the University of
Trento and a visiting researcher at the University of Washington. He received
his PhD degree from the Italian Institute of Technology in 2015. His research
lies at the intersection of machine learning, computer vision, and natural
language processing with an emphasis on learning Deep Neural Networks with
multimodal data.
---|---
|  Xavier Alameda-Pineda received M.Sc. degrees in mathematics (2008), in
telecommunications (2009) and in computer science (2010) and a Ph.D. in
mathematics and computer science (2013) from Université Joseph Fourier. Since
2016, he is a Research Scientist at Inria Grenoble Rhône-Alpes, with the
Perception team. He served as Area Chair at ICCV’17, of ICIAP’19 and of ACM
MM’19. He is the recipient of several paper awards and of the ACM SIGMM Rising
Star Award in 2018.
---|---
|  Elisa Ricci received the PhD degree from the University of Perugia in 2008.
She is an associate professor at the University of Trento and a researcher at
Fondazione Bruno Kessler. She has since been a post-doctoral researcher at
Idiap, Martigny, and Fondazione Bruno Kessler, Trento. She was also a visiting
researcher at the University of Bristol. Her research interests are mainly in
the areas of computer vision and machine learning. She is a member of the
IEEE.
---|---