accuracy of 27.55% falling behind its symmetric counterpart by 0.44%. In
general, the asymmetric and symmetric losses lead to improved accuracy scores
with respect to other methods. Moreover, we observe that all the methods
depict good forgetting resilience with their forgetting scores running around
$\approx$0.01%) except for EWC. This suggests us that vision transformers are
better incremental learners but require more training and tuning efforts to
achieve reasonable accuracies. This remark remains in accordance with prior
studies [Mirzadeh et al., 2022, Paul and Chen, 2021]. In the particular case
of EWC, we observe poor compatibility in terms of accuracy as well as
forgetting – with the scores falling behind finetuning at times. We suspect
that the method might not be less suited for ViTs due to its reliance on
exhaustive fisher information estimation.

###### Plasticity-stability tradeoff:

We compare the dilemma for various methods in Figure 4.10. With no
distillation, finetuning is prone to the worst trading of plasticity for
stability. Meanwhile, our asymmetric losses can be seen to be more plastic
with respect to their symmetric counterparts while depicting comparable
stability scores. This confirms our hypothesis regarding the nature of the
asymmetry keeping it from discarding older attention while favoring the
integration of new attention at the same time. Although, LwF with a last task
score of 47.74% on CIFAR100/10 and 32.0% on ImageNet/6, reports the best
plasticity among our approaches, it clearly lags behind the pooling-based
approaches at retaining stability. On the contrary, the (a)symmetric attention
losses and the symmetric functional loss perform similar with a last task
stability score of $\approx 0.23\%$ on ImageNet/6 and $\approx 53\%$ on
CIFAR100/10. EWC shows good plasticity but virtually zero stability. This
trend is in line with our previous comment on the limitation of EWC in Figure
4.9.

CIFAR100/10 (taw)
---
|  | Asym_Func
---
Spatial
| Sym_Func
---
Spatial
| Asym_Func
---
Intact
| Sym_Func
---
Intact
LwF
| Average Incr.
---
Accuracy
56.18% | 55.67% | 54.43% | 53.12% | 55.11%
| Last Task
---
Accuracy
57.26% | 56.92% | 56.04% | 54.59% | 55.93%
Table 4.7: Comparison of intact (no pooling), spatial (pooling along width and
height), and LwF.

#### Ablation study

Towards the end goal of evaluating the effectiveness of PAD losses, we ablate
the contribution of pooling on the CIFAR100/10 setting. In particular, we
consider distilling the attention maps when these are: (a) pooled along both
dimensions, i.e.,(A)sym_Func Spatial (see Equation 4.7), and (b) not pooled at
all, i.e., (A)sym_Func Intact. Distilling the intact maps of the latter
setting imply enhanced stability over their pooled counterparts. Our standard
accuracy and plasticity-stability measures across tasks can therefore be
deemed redundant in this setting. As a consequence, we choose to compare the
task-aware average incremental accuracy [Rebuffi et al., 2017] and the last
task accuracy across (a) and (b) while contrasting these with LwF as a strong
baseline. For further crisper observations, we limit our comparisons to the
functional setting. As shown in Table 4.7, we find that Asym_Func Spatial
consistently performs the best across both the metrics (with a gain of $>2\%$
over Sym_Func Intact in either metric). In general, distilling the intact
attention maps can be seen to be hurting the performance of the models as
their accuracy drop below that of the baseline LwF.

Figure 4.12: Mean and standard deviation of task-aware accuracy and forgetting
scores for the additional CIFAR100/20 and CIFAR100/50 settings (over 3 random
runs).

### Conclusion

In this work, we adapted and analyzed several continual learning methods to
counter forgetting in Vision Transformers mainly with the help of
regularization. We then introduced a novel PODNet-inspired regularization,
based on the attention maps of self-attention mechanisms which we termed as
Pooled Attention Distillation (PAD). Shedding light on its limitation at
learning new attention, we devised its asymmetric version that avoids
penalizing the addition of new knowledge in the model. We validated the
superior plasticity of the asymmetric loss on several benchmarks.

Besides the meticulous comparison of a range of regularization approaches,
i.e., functional (LwF), weight (EWC), and the proposed attention-based
regularization, we extended the application of PAD to the functional
submodules of ViTs. To this end, we investigated regularization in the
contextualized embeddings of ViTs. The latter exploration led us to discover
that the regularization of functional submodules can help achieve the best
overall performances while the regularization of their attentional
counterparts endow CL models with superior stability. Finally, we remarked the
low forgetting scores of vision transformers across the incremental tasks and
concluded that their enhanced generalization capabilities may endow them with
a natural inclination for incremental learning. By making our code open-
source, we hope to open the doors for future research along the direction of
efficient continual learning with transformer-based architectures.

### Additional Settings

We experiment on two further CIFAR100 settings with distinct cardinality of
base task classes:

  * •

CIFAR100/20 Base, with 20 base task classes followed by 8 incremental tasks
with 10 classes each,

  * •

CIFAR100/50 Base, with 50 base task classes followed by 5 incremental tasks
with 10 classes each.

Figure 4.13: Mean and standard deviation of task-aware plasticity-stability
scores for the additional CIFAR100/20 and CIFAR100/50 settings (over 3 random
runs).

The task aware accuracy and forgetting scores on these are shown in Figure
4.12. We find the PAD-based losses to consistently outperform other
regularization approaches with LwF being the closest tie. Along the direction
of plasticity-stability tradeoff (see Figure 4.13), we observe that: (a) the
attentional PAD losses retain better rigidity than their functional
counterparts, and (b) the asymmetric variants of PAD losses are more plastic
than their symmetric counterparts across these settings. These trends further
validate our hypotheses in sections 4 and 4, respectively.

### Task Agnostic Results

Figure 4.14 depicts the task-agnostic accuracy and forgetting scores for the
settings mentioned in the main section as well as in Section 4. Given the
contradictory terms of resource-scarce exemplar-free CL and data-hungry ViTs,
task-agnostic evaluations can be seen to be particularly challenging. The
further avoidance of heavier data augmentations in our training settings can
be seen to give rise to two major repercussions across the task-agnostic
accuracies: (a) the scores remain consistently low, and (b) the models show
smaller yet consistent variations in performances across all settings.

That said, we find functional PAD losses to be performing the best on all but
CIFAR100/50 setting. The larger proportion of base task classes in the latter
setting can be seen to be greatly benefiting the learning of LwF (the least
parameterized loss term). Further on the note of class proportions, we observe
that an equal spread of classes across the tasks can be seen to have a
smoothing effect on the variations of scores across different methods.

On the contrary, the CIFAR100/50 setting leads to low variability of task-
agnostic forgetting scores across the methods. This can again be attributed to
the fact that a very large first task better leverages the generalization
capabilities of ViTs thus making them better at avoiding forgetting over the
subsequent incremental steps. This further adds to our reasoning regarding the
natural resilience of ViTs to incremental learning settings. When compared
across methods, the attentional variants of PAD losses can be seen to display
the least amount of forgetting followed by their functional counterparts.

Figure 4.14: Mean and standard deviation of task-agnostic accuracy and
forgetting scores for CIFAR100/10, CIFAR100/20, CIFAR100/50, and ImageNet/6
settings (over 3 random runs). The larger proportion of base task classes (for
example, CIFAR100/50) gives rise to higher variations of accuracies and lower
variation of forgetting scores across methods – with the latter indicating the
inclination of ViTs towards better generalization and preservation of
knowledge.

###  4.3 Simpler is Better: off-the-shelf Continual Learning through
Pretrained Backbones

In this section we propose a simple baseline for continual learning that
leverages pretrained backbones. The approach devised is fast, since requires
no parameters updates and has minimal memory requirements (order of KBytes).
By providing such a simple baseline, and achieving strong performance on all
the major benchmarks used in literature, we follow the concerns raised in
Section 4.1 on the simplicity of the benchmarks used. Secondly, we show that
pretraining cause the network to generalize at a point where the incremental
learning of new tasks is very simple.

In particular, the ”training” phase reorders data and exploit the power of
pretrained models to compute a class prototype and fill a memory bank. At
inference time we match the closest prototype through a knn-like approach,
providing us the prediction. We will see how this naive solution can act as an
off-the-shelf continual learning system. In order to better consolidate our
results, and merge the above two works, we use the devised pipeline with CNN
and Vision Transformers. We will discover that thew latter have the ability to
produce features of higher quality. As a side note we discuss some extension
to the unsupervised realm.

In a nutshell, this simple pipeline raises the same questions raised by
previous works such as Prabhu et al. [2020] on the effective progresses made
by the CL community especially in the dataset considered and the usage of
pretrained models.

Until now, the CL community mainly focused in the analysis of catastrophic
forgetting in Convolutional Neural Networks (CNN) models. But, as can be seen
by some recent works, Vision Transformers (ViT) are asserting themselves as a
valuable alternative to CNNs for computer vision tasks, sometimes, achieving
better performances with respect to CNNs Chen et al. [2022]. The power of ViTs
lies in their less inductive bias Morrison et al. [2021] and in their
subsequent better generalization ability. Thanks to this ability ViTs are
naturally inclined continual learners, as pointed in Section 4.2.

Figure 4.15: Depiction of our simple baseline. Our pipeline does not perform
parameters updates and consumes few KBytes as memory bank.

In transformer literature, the usage of pretrained backbones is becoming a
must, in fact, training such systems requires extensive amount of data and
careful hyperparameters optimization. Using pretrained backbones is common
also in Computer Vision communities where CNNs are the main player. In CL
literature, the pretraining is frequent, but not constant. It is typically
carried on half of the analyzed dataset or through a big initial task that has
the objective of facilitating the learning of low level features. The very
best results, however, have been achieved when we do not skip pretraining.
This can be confirmed by the CVPR 2020 Continual Learning Challenge summary
report Lomonaco et al. [2022], where the authors noted that all the methods
proposed solutions leveraging pretrained backbones.

On top of that, simple baselines sometimes provide better results with respect
to overly engineered CL solutions, GDumb Prabhu et al. [2020] is such an
example. In the work, the authors showed superior performance against several
methods at the state-of-the-art through a system composed just by a memory
random sampler and a simple learner (CNN or MLP). From a practical point of
view, these methods often constitute a simple, clear, fast, intuitive and
efficient solution.

Following these lines, we explore a knn-like method to perform off-the-shelf
online continual learning leveraging the power of pretrained vision
transformers. Our system constitutes a simple and memory-friendly architecture
requiring zero parameters updates. Being our work one of the first using ViTs
in CL, we propose a robust baseline for future works and provide an extensive
comparison against CNNs.

In brevity, the contributions are the following:

  * •

We devise a simple pipeline composed by a pretrained feature extractor and an
incremental prototype bank. The latter is updated as new data is experienced.
The overall cost of the method is in the storage of a pretrained backbone and
few Kbytes for the memory bank.

  * •

We devise a baseline for future CL methodologies that will exploit pretrained
Vision Transformers or Resnets. The baseline is fast and does not require any
parameter update, yet achieving robust results in 200 lines of Python,
unlocking reproducibility too.

  * •

We provide a comparison for our pipeline between Resnets and Visual
Transformers. We discover that Vision Transformers produce more discriminative
features, appealing also for the CL setting.

  * •

In light of such results, we arise the same questions, as GDumb Prabhu et al.
[2020] does, in the progresses made by the CL community so far specifically in
the quality of the datasets and in the usage of pretrained backbones.

Algorithm 1 Off-the-shelf CL. “Training”

0: $t_{i},\phi,\mathcal{M}$

for $t_{i}\in\mathcal{T}$ do

$\mathcal{G}=\texttt{GroupByClass}(t_{i})$

for $g\in\mathcal{G}$ do

$f=\phi(g)$ Extract features

$p=\mu(f)$ Compute mean feature

$\mathcal{M}\leftarrow p$ Store prototype in memory

return $\mathcal{M}$

#### Related Works

Only recently few works considered self-attention models in continual
learning. Li et al. Li et al. [2022] proposed a framework for object detection
exploiting Swin Transformer Liu et al. [2021] as pretrained backbone for a
CascadeRCNN detector, the authors show that the extracted features generalize
better to unseen domains hence achieving lesser forgetting rates compared to
ResNet50 He et al. [2016] backbones. This also follows the conclusions made by
Paul and Chen Paul and Chen [2021] on the fact that vision transformers are
more robust learners with respect to CNNs.

Several methods in CL use pretrained backbones as feature extractors such as
in Hayes et al Hayes and Kanan [2020] or Aljundi et al. [2019b], Hocquet et
al. [2020] and sometimes the pretraining is carried on half (or a big portion)
the dataset considered, as in PODNet Douillard et al. [2020] or in Yu et al.
Yu et al. [2021]. For a more complete review on CL methodologies we point out
these recent surveys Parisi et al. [2019], Hadsell et al. [2020], Mundt et al.
[2020].

A similar study on pretraining for CL has been conducted by Mehta et al. Mehta
et al. [2021]. In particular, they study the impact on catastrophic forgetting
that a linear layer might accuse while using a pretrained backbone. Their
study focuses only on Resnet18 for vision tasks, but they also include NLP
tasks.

#### Method

###### Setting

Continual Learning characterizes the learning by introducing the notion of
subsequent tasks. In particular, the learning happens in an incremental
fashion, that is, the model incrementally experiences different training
sessions as time advances. Practically, a learning dataset is split in chunks
where each split is considered an incremental task containing data. CL being a
relatively new field, the community is still converging to a common setting
notation, but we focus on an online, task-agnostic NC-type scenario. Tat is,
the model forwards a pattern just once and does not have the task label at
test time. As a more fine grained specific we follow Lomonaco and Maltoni
[2017] categorization and use a NC-type scenario where each task contains a
disjoint group of classes.

More formally, given a dataset $\mathcal{D}$ and a set of $n$ disjoint tasks
$\mathcal{T}$ that will be experienced sequentially:

$\mathcal{T}=\left[t_{1},t_{2},\dots,t_{n}\right]$ (4.13)

each task $t_{i}=(C_{i},D_{i})$ represented by a set of classes
$Ct=c_{1}^{t},c_{2}^{t}\dots,c_{n^{t}}^{t}$ and training data $D_{t}$
(images). We assume that the classes of each task do not overlap i.e.
$C^{i}\bigcap C^{j}=\emptyset$ if $i\neq j$

###### “Training” Phase

In the training phase, given a task $t_{i}\in\mathcal{T}$, a feature extractor
$\phi$ and a memory bank as a dictionary $\mathcal{M}$, the procedure does the
following:

  1. 1.

First it performs batch reordering, that is, it groups the images of a given
task by their class

  2. 2.

After grouping, it forwards each new subset to the feature extractor $\phi$

  3. 3.

Given the feature representations of a group, it computes the mean of the
features to create a class prototype

  4. 4.

Updates the memory bank $\mathcal{M}$ by storing the each computed prototype

At the end of the training procedure for a given task $t_{i}$, we would have a
representative prototype vector for each class contained in $t_{i}$. As we
said, the prototype vector is computed as the mean feature representation of
the patterns of the same class. A depiction of the “training” phase is
reported in Figure 4.15, we also provide a pseudocode in Algorithm 1. We also
point out that there is not formal “training” of the network, in fact we do
not perform any parameter update, we simply exploit the pretrained models and
construct a knn-like memory system.

| Memory
---
KiB class
Params | Model | CIFAR100 | CIFAR10 | Core50 |  | Oxford
---
Flowers102
| Tiny
---
ImgNet200
2 KiB | 11.7M | resnet18 | 0.53 | 0.76 | 0.72 | 0.73 | 0.55
2 KiB | 21.8M | resnet34 | 0.55 | 0.81 | 0.74 | 0.67 | 0.62
8 KiB | 25.5M | resnet50 | 0.59 | 0.80 | 0.71 | 0.70 | 0.63
8 KiB | 60.1M | resnet152 | 0.67 | 0.89 | 0.72 | 0.66 | 0.76
0.75 KiB | 5.6M | ViT-T/16 | 0.36 | 0.63 | 0.49 | 0.54 | 0.24
3 KiB | 86.4M | ViT-B/16 | 0.64 | 0.87 | 0.74 | 0.95 | 0.63
0.75 KiB | 5.6M | DeiT-T/16 | 0.57 | 0.80 | 0.73 | 0.68 | 0.64
3 KiB | 86.4M | DeiT-B/16 | 0.68 | 0.90 | 0.80 | 0.74 | 0.79
Table 4.8: Off-the-shelf accuracy performance on different dataset benchmarks,
we both analyzed a CNN model and a ViT pretrained models.

###### Test Phase

After completing the training phase for a task $t_{i}$ the memory bank
$\mathcal{M}$ will be populated by the prototypes of the classes encoundered
so far. During this test phase, we simply use a knn-like approach. Given an
image $x$, the updated memory bank $\mathcal{M}$ and the feature extractor
$\phi$ we devise the test phase as follows:

  1. 1.

Forward the test image $x$ to the feature extractor $\phi$

  2. 2.

Compute a distance between the feature representation of the image and all
prototypes contained in $\mathcal{M}$

  3. 3.

We match the prototype with minimum distance and return its class

In a nutshell, we perform k-nn with k=1 over the feature representation of an
image, matching the class of the closes prototype in the bank. If the class
selected is the same of the test example we would have a hit, a miss
otherwise. Figure 4.15 reports a visual depiction of the test procedure. As
distance we use a simple $l^{2}$, but several tests have been made with cosine
similarity. Although the results with the cosine similarity are better, we opt
for the $l^{2}$ since provides the best speedup in the implementation through
Pytorch.

#### Experiments

It is suspected that Visual Transformers generalize better with respect to CNN
models. To this end, we compare CNNs models and ViTs models as feature
extractors. We selected four CNN models to compare against four attention-
based models. In particular, we selected DeiT-Base/15, DeiT-Tiny/15 Touvron et
al. [2021], ViT-Base/16 and ViT-Tiny/16 Dosovitskiy et al. [2021] as visual
transformers. While we opted for Resnet18/34/50/152 He et al. [2016] as CNN
models. We used the timm Wightman [2019] library to fetch the pretrained
models where all the models have been trained on ImageNet Deng et al. [2009]
and the continuum Douillard and Lesort [2021] library to create the
incremental setting for 5 datasets, namely CIFAR10/100, Core50,
OxfordFlowers102 and TinyImageNet200.

In all dataset benchmarks, we upscaled the images to $224\times 224$ pixels in
order to accommodate visual transformers which needs such imput dimension. We
apply such transformation to resnet data too for a fair comparison. In order
to match the closes prototype at test time, we used $l^{2}$ as preferred
measure.

The main results are reported in Table 4.8. The pipeline is extremely simple,
yet it achieves impressive performance as an off-the-shelf method, at cost of
a very small overhead to store the prototype memory. In fact, at the end of
the training phase, the memory bank translates only into few KBytes of
storage. Although this preliminary work only consider task-agnostic setting,
we remind that if at test time we are given the task label of the data, we can
recast the method to work in task-aware setting. In this case, performing the
test phase would be easier since the comparison of the test data will be
carried only on a subset of the prototypes. On the same line, one can see that
in Table 4.8 we do not report each dataset task split. In fact, our method
works for any dataset split since it just need any partition of the datasets
that respect a NC protocol i.e. as long as tasks are formed by images that can
be grouped in classes. We can also appreciate that transformer architectures
work best in all benchmarks, suggesting direct superior generalization
capabilities with respect to CNNs or, at least, more discriminative features.

Figure 4.16: Direct off-the-shelf extension of the baseline proposed to tackle
unspervised continual learning.

#### Discussion

In light of these results, we think that this work may be extended to be
considered as a baseline to assess the performance continual learning
methodologies using pretrained networks as feature extractors. In particular,
a thorough investigation should be carried by substituting the k-nn approach
with a linear classifier, this would allow also a better comparison between
resnets and visual transformers. However, we think that these preliminary
results are of interest to the Vision Transformer and CL research community.

We then raise some concerns with respect to the procedure and the benchmarks
used to assess new CL methodologies. As we can see, through a pretrained
model, we can achieve impressive results with respect to the current CL state-
of-the-art Parisi et al. [2019], Hadsell et al. [2020], Mundt et al. [2020].
This point have been also raised by GDumb Prabhu et al. [2020] where the
authors questioned the progresses by providing a very simple baseline.

Moreover, we can further extend this simple pipeline to be used in
unsupervised continual learning. Actually, the extension is straightforward.
In an unsupervised scenario the batch reordering step cannot be performed
since we are not allowed to know each data class label. To cope with this lack
of information one can substitute the step with any clustering algorithm such
as K-means (we tried it but with no luck) or a more sophisticated approach
such as autoencoders, self-organizing maps etc.. The test phase of the
unsupervised extension would be analogous to the supervised counterpart.

#### Conclusion

In this short ex[erimental segment we proposed a baseline for continual
learning methodologies that exploit pretrained Vision Transformers and
Resnets. We tackle online NC-type class-incremental learning scenario, the
most common one, even though, our pipeline can be extended to different
scenarios. Our off-the-shelf method is conceptually simple yet gives strong
results and can be implemented in 200 lines of Python therefore enhancing
reproducibility. To assess the performance of different backbones our pipeline
we compared Resnets models against Vision Transformers feature extractors
pretrained on the same dataset, and show that vision transformers provide more
powerful features. This suggests that Vision Transformers ability to encode
knowledge is is broader. Then we raise some questions about CL research
progress and note that with a pretrained model and a simple pipeline one can
achieve strong results and, therefore, new methodologies should drop the usage
of pretrained backbones when testing on such dataset benchmarks.

###  4.4 Unsupervised Semantic Discovery through Visual Patterns detection

So far, we directly investigated the impact of performance by altering
structural and data properties of object recognition frameworks. If we step
back a bit and consider a more broader vision about continual learning, we
understand that, in order to adapt to a changing environment, an artificial
agent should manifest also the ability to continuously discover new patterns,
in our case visual patterns.

We propose a smart pipeline that it is able to discover repetitive patterns in
an image, by means of a threshold parameter. That is, if we alter this
specific parameter, we are able to discover new semantic levels in a scene.
This work goes a bit in another direction from the dissection of current
continual learning methodologies treated in this thesis. Instead, it is a step
towards the ability to build a system able to incrementally explore.

To this end, we propose a new fast fully unsupervised method to discover
semantic patterns. Our algorithm is able to hierarchically find visual
categories and produce a segmentation mask. Through the modeling of what is a
visual pattern in an image, we introduce the notion of “semantic levels” and
devise a conceptual framework along with measures and a dedicated benchmark
dataset for future comparisons. Our algorithm is composed by two phases. A
filtering phase, which selects semantical hotsposts by means of an accumulator
space, then a clustering phase which propagates the semantic properties of the
hotspots on a superpixels basis. We provide both qualitative and quantitative
experimental validation, achieving optimal results.

While the vast majority of supervised object detection and segmentation
approaches leverage rich datasets with semantically labelled categories,
unsupervised methods cannot rely on such a luxury. Indeed they are expected to
infer from the image content itself what is a relevant object and which are
its boundaries. This is a daunting task, as relevance is totally domain-
specific and also highly subjective, especially when taking in account human
judgement, which exploits a lot of out-of-band information that cannot be
found in the sheer image data.

As a matter of fact, little effort have been put to investigate unsupervised
automatic approaches to detect and segment semantically relevant objects
without any additional information than the image or any a priori knowledge of
the context. This is due to the fact that a unique definition of what is a
relevant object (or, how we prefer to call it, a _visual category_) does not
actually exist.

This is especially true if we are seeking to set a formal definition that can
be adopted across all the domains in a consistent manner with respect to human
judgement.

Within this section, we try to address this problem by considering a visual
category each pattern which appearance is consistent enough across the image.
In other words, we consider something to be a relevant object if it appears
more than once, exhibiting consistent visual features in different parts of
the scene.

From a cognitive and perceptual point of view this makes a lot of sense. In
fact, it is easy to observe that if a human is presented with images
representing several different but recurring objects, even in a cluttered
scene, he does not need to know what the objects actually are representing in
order to be able to assign semantically-consistent labels to each of them. He
would even be able to label each pixel, defining the boundaries of the
objects.

As an example, if someone takes a look at a large bin of different (but to
some extent repeated) mechanical parts he never saw before, he is still able
to tell one part from the other by exploiting their coherent visual and
structural appearance. This ability is also preserved with slight changes in
scale, orientation or partial occlusion of the objects.

Since this automatic assignment to a visual category of recurrent object is
both well-defined and quite natural in humans, it is a very good candidate as
a rule for automatically detecting relevant objects in an unsupervised manner
that has good chances of being coherent with human judgement applied to the
same image.

To be fair, we must also underline the fact that, in order to define the
boundaries of a visual category and thus obtain a meaningful segmentation,
also the level of detail must be taken into account.

Figure 4.17: A real world example of unsupervised segmentation of a grocery
shelf. Our method can automatically discover both low-level coherent patterns
(brands, flavor images and logos) and high-level compound objects (multi-packs
and bricks) by controlling the semantical level of the detection and
segmentation process.

As an example, if we present to a human an image of a crowded road captured
from a side, and we ask him to segment visual categories according to
recurrent patterns, we could get slightly different results from different
people depending on their attention to details. Some people will segment cars
and trees. Other could consider the car body to be a different object from the
wheels ad branches from the tree trunk. The most picky could even separate
tires from wheel rims and segment out each single leaf. In practice semantic
consistency can happen at different scale when dealing with compound objects
presenting themselves internal self repetitions or made up of single parts
that are also present in other objects.

To address this aspect we also have to design a proper strategy to perform
visual category detection and interpretation at a particular scale, according
to the level of detail we want to express during the segmentation process. We
define this level of detail as _semantical level_. Semantical levels, of
course, do not map directly on specific high level concepts, such as whole
objects, large parts or minute components. Rather the semantic level will act
as a coarse degree of granularity of the segmentation process that will result
in a hierarchical split of segments as it changes.

These two definitions of _visual categories_ and _semantical levels_ , that
will be developed throughout the remainder of the work, are the two key
concepts driving our novel segmentation method.

The ability of our approach to leverage repetitions to capture the internal
representation in the real world and then extrapolates visual categories at a
specific semantical level is actually achieved through the combination of a
couple of standard techniques, slightly modified for the specific task, and of
a few key steps specifically crafted to make the process work in a consistent
way with respect to the cognitive process adopted by humans. This happens, for
instance, by seeking for highly relevant repetitive structural patterns,
called _semantical hotspots_ , characterized by a novel feature descriptor,
called _splash_. We do this through a scale-invariant method and with no
continuous geometrical constraints on the visual pattern disposition.

We also do not constrain ourselves to find only one visual pattern, which is
another very common assumption with other approaches in literature. Rather our
technique is designed from the start to be able to detect more patterns at
once, being able to assign to each of them a different visual category label,
corresponding to a different real world object or object part, according to
the selected semantical level.

Overall, with this study, we are offering to the community the following
contributions:

  * •

A new pipeline, including the definition of a specially crafted feature
descriptor, to capture semantical categories with the ability to
hierarchically span over semantical levels;

  * •

A specially crafted conceptual framework to evaluate unsupervised semantic-
driven segmentation methods through the introduction of the semantical levels
notion along with a new metric;

  * •

A new dataset consisting of a few hundredths labelled images that can be used
as a benchmark for visual repetition detection in general.

The remainder of the section is organized as follows. Section 4 describes the
related works with respect to feature extraction and automatic visual patterns
detection. Section 4 introduces our method, giving details on the overall
pipeline and on the implementation details. Section 4 presents an experimental
evaluation and comparison with similar approaches. Finally, the conclusions
are found in Section 4.

Code, dataset and notebooks used in this study will be made available for
public use.

### Related Works

Several works have been proposed to tackle visual pattern discovery and
detection. While the paper by Leung and Malik [Leung and Malik, 1996] could be
considered seminal, many other works build on their basic approach, working by
detecting contiguous structures of similar patches by knowing the window size
enclosing the distinctive pattern.

One common procedure in order to describe what a pattern is, consists to first
extract descriptive features such as SIFT to perform a clustering in the
feature space and then model the group disposition over the image by
exploiting geometrical constraints, as in [Pritts et al., 2014] and [Chum and
Matas, 2010], or by relying only on appearance, as in [Doubek et al., 2010,
Liu and Liu, 2013, Torii et al., 2015].

The geometrical modeling of the repetitions usually is done by fitting a
planar 2-D lattice, or a deformation of it [Park et al., 2009], through RANSAC
procedures as in [Schaffalitzky and Zisserman, ] [Pritts et al., 2014] or even
by exploiting the mathematical theory of crystallographic groups as in [Liu et
al., 2004]. Shechtman and Irani [Shechtman and Irani, 2007], also exploited an
active learning environment to detect visual patterns in a semi-supervised
fashion. For example Cheng et al. [Cheng et al., 2010] use input scribbles
performed by a human to guide detection and extraction of such repeated
elements, while Huberman and Fattal [Huberman and Fattal, 2016] ask the user
to detect an object instance and then the detection is performed by exploiting
correlation of patches near the input area.

Recently, as a result of the new wave of AI-driven Computer Vision, a number
of Deep Leaning based approaches emerged, in particular Lettry et al. [Lettry
et al., 2017] argued that filter activation in a model such as AlexNet can be
exploited in order to find regions of repeated elements over the image, thanks
to the fact that filters over different layers show regularity in the
activations when convolved with the repeated elements of the image. On top of
the latter work, Rodríguez-Pardo et al. [Rodríguez-Pardo et al., 2019]
proposed a modification to perform the texture synthesis step.

A brief survey of visual pattern discovery in both video and image data, up to
2013, is given by Wang et al. [Wang et al., 2014], unfortunately after that it
seems that the computer vision community lost interest in this challenging
problem. We point out that all the aforementioned methods look for only one
particular visual repetition except for [Liu and Liu, 2013] that can be
considered the most direct competitor and the main benchmark against which to
compare our results.

### Method Description

#### Features Localization and Extraction

We observe that any visual pattern is delimited by its contours. The first
step of our algorithm, in fact, consists in the extraction of a set
$\mathcal{C}$ of contour _keypoints_ indicating a position $\vec{c}_{j}$ in
the image. To extract keypoints, we opted for the Canny algorithm, for its
simplicity and efficiency, although more recent and better edge extractor
could be used [Liu et al., 2019] to have a better overall procedure.

A descriptor $d_{j}$ is then computed for each selected
$\vec{c}_{j}\in\mathcal{C}$ thus obtaining a _descriptor set_ $\mathcal{D}$.
In particular, we adopted the DAISY algorithm because of its appealing dense
matching properties that nicely fit our scenario. Again, here we can replace
this module of the pipeline with something more advanced such as [Ono et al.,
2018] at the cost of some computational time.

#### Semantic Hot Spots Detection

Figure 4.18: (a) A splash in the image space with center in the keypoint
$\vec{c}_{j}$. (b) $\mathcal{H}$, with the superimposed splash at the center,
you can note the different levels of the vote ordered by endpoint importance
i.e. descriptor similarity. (c) 3D projection showing the gaussian-like
formations and the thresholding procedure of $\mathcal{H}$. (d) Backprojection
through the set $\mathcal{S}$.

In order to detect self-similar patterns in the image we start by associating
the $k$ most similar descriptors for each descriptor $\vec{d}_{j}$. We can
visualize this data structure as a star subgraph with $k$ endpoints called
_splash_ “centered” on descriptor $\vec{d}_{j}$. Figure 4.18 (a) shows one.

Splashes potentially encode repeated patterns in the image and similar
patterns are then represented by similar splashes. The next step consists in
separating these splashes from those that encode noise only, this is
accomplished through an accumulator space.

In particular, we consider a $2$-D _accumulator space_ $\mathcal{H}$ of size
double the image. We then superimpose each splash on the space $\mathcal{H}$
and cast $k$ votes as shown in Figure 4.18 (b). In order to take into account
the noise present in the splashes, we adopt a gaussian vote-casting procedure
$g(\cdot)$. Similar superimposed splashes contribute to similar locations on
the accumulator space, resulting in peak formations (Figure 4.18 (c)). We
summarize the voting procedure as follows:

$\mathcal{H}_{\vec{w}}=\mathcal{H}_{\vec{w}}+g(\vec{w},\vec{h}^{(j)}_{i})$
(4.14)

where $\vec{h}^{(j)}_{i}$ is the $i$-th splash endpoint of descriptor
$\vec{d}_{j}$ in accumulator coordinates and $\vec{w}$ is the size of the
gaussian vote. We filter all the regions in $\mathcal{H}$ which are above a
certain _threshold_ $\tau$, to get a set $\mathcal{S}$ of the locations
corresponding to the peaks in $\mathcal{H}$. The $\tau$ parameter acts as a
coarse filter and is not a critical parameter to the overall pipeline. A
sufficient value is to set it to $0.05\cdot max(\mathcal{H})$. Lastly, in
order to visualize the semantic hotspots in the image plane we map splash
locations between $\mathcal{H}$ and the image plane by means of a
_backtracking structure_ $\mathcal{V}$.

In summary, the key insight here is that similar visual regions share similar
splashes, we discern noisy splashes from representative splashes through an
auxiliary structure, namely an accumulator. We then identify and backtrack in
the image plane the semantic hotspots that are candidate points part of a
visual repetition.

#### Semantic Categories Definition and Extraction

While the first part previously described acts as a filter for noisy keypoints
allowing to obtain a good pool of candidates, we now transform the problem of
finding visual categories in a problem of dense subgraphs extraction.

We enclose semantic hotspots in superpixels, this extends the semantic
significance of such identified points to a broader, but coherent, area. To do
so we use the SLIC [Achanta et al., 2012] algorithm which is a simple and one
of the fastest approaches to extract superpixels as pointed out in this recent
survey [Stutz et al., 2018]. Then we choose the cardinality of the
_superpixels_ $\mathcal{P}$ to extract. This is the second and most
fundamental parameter that will allow us to span over different semantic
levels.

Once the superpixels have been extracted, let $\mathcal{G}$ be an _undirected
weighted graph_ where each node correspond to a superpixel $p\in\mathcal{P}$.
In order to put edges between graph nodes (i.e. two superpixels), we exploit
the splashes origin and endpoints. In particular the strength of the
connection between two vertices in $\mathcal{G}$ is calculated with the number
of splashes endpoints falling between the two in a mutual coherent way. So to
put a weight of 1 between two nodes we need exactly 2 splashes endpoints
falling with both origin and end point in the two candidate superpixels.

With this construction scheme, the graph has clear dense subraphs formations.
Therefore, the last part simply computes a partition of $\mathcal{G}$ where
each connected component correspond to a cluster of similar superpixels. In
order to achieve such objective we optimize a function that is maximized when
we partition the graph to represent so. To this end we define the following
density score that given $G$ and a set $K$ of connected components captures
the optimality of the clustering:

$s(G,K)=\sum_{k\in K}\mu(k)-\alpha\left|K\right|$ (4.15)

where $\mu(k)$ is a function that computes the average edge weight in a
undirected weighted graph.

Algorithm 2 Semantic categories extraction algorithm

0: $G$ weighted undirected graph

$i=0$

$s^{*}=-\inf$

$K^{*}=\emptyset$

while $G_{i}$ is not fully disconnected do

$i=i+1$

Compute $G_{i}$ by corroding each edge with the minimum edge weight

Extract the set $K_{i}$ of all connected components in $G_{i}$

$s(G_{i},K_{i})=\sum_{k\in K_{i}}\mu(k)-\alpha\left|K_{i}\right|$

if $s(G_{i},K_{i})>s^{*}$ then

$s^{*}=s(G_{i},K_{i})$

$K^{*}=K_{i}$

return $s^{*},K^{*}$

The first term, in the score function, assign a high vote if each connected
component is dense. While the second term acts as a regulator for the number
of connected components. We also added a weighting factor $\alpha$ to better
adjust the procedure. As a proxy to maximize this function we devised an
_iterative algorithm_ reported in Algorithm 2 based on graph corrosion and
with temporal complexity of
$O(\left|E\right|^{2}+\left|E\right|\left|V\right|)$. At each step the
procedure corrupts the graph edges by the minimum edge weight of $G$. For each
corroded version of the graph that we call _partition_ , we compute $s$ to
capture the density. Finally the algorithm selects the corroded graph
partition which maximizes the $s$ and subsequently extracts the node groups.

In brevity we first enclose semantic hotspots in superpixels and consider each
one as a node of a weighted graph. We then put edges with weight proportional
to the number of splashes falling between two superpixels. This results in a
graph with clear dense subgraphs formations that correspond to superpixels
clusters i.e. semantic categories. The semantic categories detection
translates in the extraction of dense subgraphs. To this end we devised an
iterative algorithm based on graph corrosion where we let the procedure select
the corroded graph partition that filters noisy edges and let dense subgraphs
emerge. We do so by maximizing score that captures the density of each
connected component.

### Experiments

Figure 4.19: (top) Analysis of measures as the number of superpixels
$|\mathcal{P}|$ retrieved varies. The rightmost figure shows the running time
of the algorithm. We repeated the experiments with the noisy version of the
dataset but report only the mean since variation is almost equal to the
original one. (bottom) Distributions of the measures for the two semantic
levels, by varying the two main parameters $r$ and $|\mathcal{P}|$. Figure
4.20: Qualitative comparison between [Liu and Liu, 2013] [14], [Lettry et al.,
2017] [10] and our algorithm. Our method detects and segments more than one
pattern and does not constrain itself to a particular geometrical disposition.

#### Dataset

As we introduced in Section 4.4 one of the aims of this work is to provide a
better comparative framework for visual pattern detection. To do so we created
a public dataset by taking 104 pictures of store shelves. Each picture has
been took with a 5mpx camera with approximatively the same visual conditions.
We also rectified the images to eliminate visual distortions.

We manually segmented and labeled each repeating product in two different
semantic levels. In the first semantic level _products made by the same
company_ share the same label. In the second semantic level visual repetitions
consist in the _exact identical products_. In total the dataset is composed by
208 ground truth images, half in the first level and the rest for the second
one.

####  $\mu$-consistency

We devised a new measure that captures the semantic consistency of a detected
pattern that is a proxy of the average precision of detection.

In fact, we want to be sure that all pattern instances fall on similar ground
truth objects. First we introduce the concept of semantic consistency for a
particular pattern $\vec{p}$. Let $\vec{P}$ be the set of patterns discovered
by the algorithm. Each pattern $\vec{p}$ contains several instances
$\vec{p}_{i}$. $\vec{L}$ is the set of ground truth categories, each ground
truth category $\vec{l}$ contain several objects instances $\vec{l}_{i}$. Let
us define $\vec{t}_{p}$ as the vector of ground truth labels touched by all
instances of $\vec{p}$. We say that $\vec{p}$ is consistent if all its
instances $\vec{p}_{i},i=0\dots|\vec{p}|$ fall on ground truth regions sharing
the same label. In this case $\vec{t}_{p}$ would be uniform and we consider
$\vec{p}$ a good detection. The worst scenario is when given a pattern
$\vec{p}$ every $\vec{p}_{i}$ falls on objects with different label $\vec{l}$
i.e. all the values in $\vec{t}_{p}$ are different.

To get an estimate of the overall consistency of the proposed detection, we
average the consistency for each $\vec{p}\in\vec{P}$ giving us:

$\text{$\mu$-consistency}=\frac{1}{\left|\vec{P}\right|}\sum_{\vec{p}\in\vec{P}}\frac{\left|\operatorname{mode}\left(\vec{t}_{p}\right)\right|}{\left|\vec{t}_{p}\right|}$
(4.16)

#### Recall

The second measure is the classical recall over the objects retrieved by the
algorithm. Since our object detector outputs more than one pattern we average
the recall for each ground truth label by taking the best fitting pattern.

$\frac{1}{\left|\vec{L}\right|}\sum_{\vec{l}\in\vec{L}}\operatorname{max}_{\vec{p}\in\vec{P}}\text{
recall }(\vec{p},\vec{l})$ (4.17)

The last measure is the total recall, here we consider a hit if any of the
pattern falls in a labeled region. In general we expect this to be higher than
the recall.

We report the summary performances in Figure 4.20. As can be seen the
algorithm achieves a very high $\mu$-consistency while still able to retrieve
the majority of the ground truth patterns in both levels.

One can observe in Figure 4.19 an inverse behaviour between recall and
consistency as the number of superpixels retrieved grows. This is expected
since less superpixels means bigger patterns, therefore it is more likely to
retrieve more ground truth patterns.

In order to study the robustness we repeated the same experiments with an
altered version of our dataset. In particular for each image we applied one of
the following corruptions: Additive Gaussian Noise ($scale=0.1*255$), Gaussian
Blur ($\sigma=3$), Spline Distortions (grid affine), Brightness ($+100$), and
Linear Contrast ($1.5$).

#### Qualitative Validation

Firstly we begin the comparison by commenting on [Liu and Liu, 2013]. One can
observe that our approach has a significant advantage in terms of how the
visual pattern is modeled. While the authors model visual repetitions as
geometrical artifacts associating points, we output a higher order
representation of the visual pattern. Indeed the capability to provide a
segmentation mask of the repeated instance region together the ability to span
over different levels unlocks a wider range of use cases and applications.

As qualitative comparison we also added the latest (and only) deep learning
based methodology [Lettry et al., 2017] we found. This methodology is only
able to find a single instance of visual pattern, namely the most frequent and
most significant with respect to the filters weights. This means that the
detection strongly depends from the training set of the CNN backbone, while
our algorithm is fully unsupervised and data agnostic.

#### Quantitative Validation

We compared quantitatively our method against [Liu and Liu, 2013] that
constitutes, to the best of our knowledge, the only work developed able to
detect more than one visual pattern. We recreated the experimental settings of
the authors by using the Face dataset [Li et al., 2007] as benchmark achieving
$1.00$ precision vs. $0.98$ of [Liu and Liu, 2013] and $0.77$ in recall vs.
and $0.63$. We considered a miss on the object retrieval task, if more than
20% of a pattern total area falls outside from the ground truth. The parameter
used were $|\mathcal{C}|=9000$, $k=15$, $r=30$, $\tau=5$, $|\mathcal{P}|=150$.
We also fixed the window of the gaussian vote to be $11\times 11$ pixels
throughout all the experiments.

### Conclusions

With this study we introduced a fast and unsupervised method addressing the
problem of finding semantic categories by detecting consistent visual pattern
repetitions at a given scale. The proposed pipeline hierarchically detects
self-similar regions represented by a segmentation mask.

As we demonstrated in the experimental evaluation, our approach retrieves more
than one pattern and achieves better performances with respect to competitors
methods. We also introduce the concept of _semantic levels_ endowed with a
dedicated dataset and a new metric to provide to other researchers tools to
evaluate the consistency of their approaches.

#### Acknowledgments

We would like to express our gratitude to Alessandro Torcinovich and Filippo
Bergamasco for their suggestions to improve the work. We also thank Mattia
Mantoan for his work to produce the dataset labeling.

##  Chapter 5 Conclusions

In this thesis, we contributed spanned the dissection of continual learning by
providing several structural and data analyses. First we provide a gentle
introduction to the topic of continual learning starting by highlighting the
difference between natural and artificial models. Among the differences we
stress the importance of time, which is an essential component for developing
lifelong learning machines. Then, we informally introduce the main challenges
that continual learning systems must tackle. In particular, catastrophic
forgetting and the stability plasticity dilemma. To better provide an
intuition about these topics, we provided a visual example of catastrophic
forgetting in an autoencoder model, showing how distributional shifts in the
subsequent tasks result in the abrupt damage of past knowledge.

Later, we move on by giving a more formal definition of continual learning
settings prominently adopted in literature. We introduced the notions of
class-incremental, task-incremental, online/offline learning along with a
specification on other common settings in the field. Before moving on the
contributions we provided a small literature review on the state-of-the-art by
describing the main categories under which continual learning methods have
been grouped.

Finally, we move on the main contributions. First, we introduced a study on
the quality/quantity trade-off in rehearsal-based continual learning. Here, we
selected one of the most performant baselines, that is GDumb, and analyzed
several compression techniques when applied to the replay buffer. We
highlighted that the quantity of data is a far more important factor when
storing examplars in the replay buffer. We do so by considering different
compression schemes with extreme rates. Then, we moved into the second major
contribution which considers Visual Transformers in an incremental setting.
Here, besides being one of the first works on visual transformers for
continual learning, we provided a surgical investigation on regularization
methods for ViTs in the challenging setting of rehearsal-free CL. We compared
functional, weight and attentional regularizations, with the latter being a
regularization in the matrix of the self-attention mechanism. Attentional
regularizations provide comparable performance with respect to the other
methods. As second contribution we also introduced a loss inspired by a method
nowadays in vogue (PODNet) and devised an asymmetric variant. We show that the
introduction of the asymmetric variant allows achieving more plasticity to the
model when applied to different part of the mechanism of self-attention. Then,
we proposed a study on off-the-shelf continual learning exploiting fully
pretrained networks and, in particular, we proposed a simple baseline. The
baseline is composed by a feature extractor and a knn-like prototype memory.
The baseline is crafted to be performant in practical scenarios achieving
optimal results with a memory overhead of few KBytes. Moreover we discussed
its possible extension to the realm of unsupervised continual learning. We
then linked this preliminary discussion with the exploration of visual
categories. To do so we introduce another work tackling unsupervised pattern
discovery. In fact, the notion of discovery is naturally included into the
notion of lifelong learning: an agent capable of lifelong learning, surely
should possess the ability to autonomously discover new knowledge. We do so by
introducing a new unsupervised algorithm to perform unsupervised semantic
segmentation at different semantic scales.

#### Further Developings

With the several studies proposed, we want to highlight the directions where
it might be more fruitful to investigate further to build better Continual
Learning agents.

A first warning we raised regards the dataset usage to assess the performance
of CL algorithms. In particular, with Section 4.1 we see that extreme levels
of buffer data resize still provide good results in rehearsal systems,
suggesting that, perhaps, more realistic datasets should be included to devise
more useful solutions. This finding is also supported by Section 4.3 which
shows that tackling these benchmarks with a pretrained backbone is sufficient
to overcome quasi-optimally continual learning scenarios on 5 different
datasets. This also suggests that pretraining could be a great advantage, in
the generalization ability of the model, when building new CL algorithms.

To tackle the aforementioned point, the community can focus more on
unsupervised continual learning which is a natural and more challenging
problem extension. While keeping the same datasets we can now also leverage
pretrained backbones. While being appealing on its own, following this line is
also greatly encouraged by the fact that there are virtually no works on such
a topic.

With the study proposed in Section 4.2 we show that ViTs are naturally
inclined continual learners. We suspect that the less inductive bias carried
by such models might be the key that allows such models to perform better in
incremental scenarios. On another side, we see that the results obtained
without pretraining have difficulty achieving CNN performances so easily (we
can compare the results of Section 4.1 and Section 4.2). This calls for the
need to build less data-hungry models in line with the world’s fast-paced data
generation. Within Section 4.2 we also propose a new way to assess Continual
Learning methods. We think that the community still lacks of a principled way
to measure the stability-plasticity trade-off. With our introduction of the
two curves, we proposed an initial tentative to monitor the performance of a
system.

Last but not least, with the work of Section 4.4 we stress that autonomously
discovering new patterns should be a core ability of an intelligent system. In
fact, if an agent can explore the real world and find hierarchies of knowledge
without help, all it has to do to incrementally learn is to store such
knowledge in some kind of long-term memory repository which translates into a
compression problem.

## References

  * MAA [1997] Networks of spiking neurons: The third generation of neural network models.  _Neural Networks_ , 1997.
  * Achanta et al. [2012] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurélien Lucchi, Pascal Fua, and Sabine Süsstrunk.  SLIC superpixels compared to state-of-the-art superpixel methods.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2012.
  * Adel et al. [2020] Tameem Adel, Han Zhao, and Richard E. Turner.  Continual learning with adaptive weights (CLAW).  In _International Conference on Learning Representations, (ICLR)_ , 2020.
  * Agarap [2018] Abien Fred Agarap.  Deep learning using rectified linear units (relu).  _ArXiv_ , 2018.
  * Agarwal et al. [2020] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.  An optimistic perspective on offline reinforcement learning.  In _ICML_ , 2020.
  * Ahn et al. [2021] Hongjoon Ahn, Jihwan Kwak, Su Fang Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon.  Ss-il: Separated softmax for incremental learning.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 824–833, 2021.
  * Aljundi et al. [2017] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * Aljundi et al. [2018] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Aljundi et al. [2019a] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia.  Online continual learning with maximal interfered retrieval.  In _Advances in Neural Information Processing Systems (NeurIPS)_. Curran Associates, Inc., 2019a.
  * Aljundi et al. [2019b] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars.  Task-free continual learning.  In _CVPR_. Computer Vision Foundation / IEEE, 2019b.
  * Aljundi et al. [2019c] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019c.
  * Arani et al. [2022] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz.  Learning fast, learning slow: A general continual learning method based on complementary learning system.  _ICLR_ , 2022.
  * Banerjee et al. [2021] S. Banerjee, Vinay Kumar Verma, Toufiq Parag, Maneesh Kumar Singh, and Vinay P. Namboodiri.  Class incremental online streaming learning.  _NeurIPS_ , 2021.
  * Belouadah et al. [2021] Eden Belouadah, Adrian Popescu, and Ioannis Kanellos.  A comprehensive study of class incremental learning algorithms for visual tasks.  _Neural Networks_ , 2021.
  * Bennett et al. [1964] Edward L. Bennett, Marian Cleeves Diamond, David Krech, and Mark Richard Rosenzweig.  Chemical and anatomical plasticity of brain changes in brain through experience, demanded by learning theories, are found in experiments with rats.  _Science_ , 1964.
  * Betti et al. [2020] Alessandro Betti, Marco Gori, Simone Marullo, and Stefano Melacci.  Developing constrained neural units over time.  In _International Joint Conference on Neural Networks, IJCNN_. IEEE, 2020.
  * Biesialska et al. [2020] Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà.  Continual lifelong learning in natural language processing: A survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, 2020.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS_ , 2020.
  * Buzzega et al. [2020] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS_ , 2020.
  * Carpenter and Grossberg [1988] Gail A. Carpenter and Stephen Grossberg.  The art of adaptive pattern recognition by a self-organizing neural network.  _Computer_ , 1988.
  * Cermelli et al. [2020] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo.  Modeling the background for incremental learning in semantic segmentation.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Chaudhry et al. [2018] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Chaudhry et al. [2019a] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with A-GEM.  In _International Conference on Learning Representations, (ICLR)_ , 2019a.
  * Chaudhry et al. [2019b] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning, 2019b.
  * Chen et al. [2020] Hui Chen, Chao Tan, and Zan Lin.  Ensemble of extreme learning machines for multivariate calibration of near-infrared spectroscopy.  _Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy_ , 2020.
  * Chen et al. [2022] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong.  When vision transformers outperform resnets without pretraining or strong data augmentations.  _ICLR_ , 2022.
  * Cheng et al. [2010] Ming-Ming Cheng, Fang-Lue Zhang, Niloy J. Mitra, Xiaolei Huang, and Shi-Min Hu.  Repfinder: finding approximately repeated scene elements for image editing.  _ACM Trans. Graph._ , 2010.
  * Chiaro et al. [2020] Riccardo Del Chiaro, Bartlomiej Twardowski, Andrew D. Bagdanov, and Joost van de Weijer.  RATT: recurrent attention to transient tasks for continual image captioning.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS_ , 2020.
  * Chum and Matas [2010] Ondrej Chum and Jiri Matas.  Unsupervised discovery of co-occurrence in sparse high dimensional data.  In _The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010_ , 2010.
  * Clanuwat et al. [2018] Tarin Clanuwat, Mikel Bober-Irizar, A. Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha.  Deep learning for classical japanese literature.  _ArXiv_ , 2018.
  * Clevert et al. [2016] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  In Yoshua Bengio and Yann LeCun, editors, _4th International Conference on Learning Representations, ICLR_ , 2016.
  * de Masson d’Autume et al. [2019] Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama.  Episodic memory in lifelong language learning.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Delange et al. [2021] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: A large-scale hierarchical image database.  In _Society Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2009.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT_. Association for Computational Linguistics, 2019.
  * Dhar et al. [2019] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa.  Learning without memorizing.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations, ICLR_ , 2021.
  * Doty et al. [2021] Briar Doty, Stefan Mihalas, Anton Arkhipov, and Alex T. Piet.  Heterogeneous ‘cell types’ can improve performance of deep neural networks.  _bioRxiv_ , 2021.
  * Doubek et al. [2010] Petr Doubek, Jiri Matas, Michal Perdoch, and Ondrej Chum.  Image matching and retrieval by repetitive patterns.  In _20th International Conference on Pattern Recognition, ICPR 2010_ , 2010.
  * Douillard and Lesort [2021] Arthur Douillard and Timothée Lesort.  Continuum: Simple management of complex continual learning scenarios, 2021\.
  * Douillard et al. [2020] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle.  Podnet: Pooled outputs distillation for small-tasks incremental learning.  In _Proceedings of the IEEE European Conference on Computer Vision (ECCV)_ , 2020.
  * Douillard et al. [2021] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord.  Dytox: Transformers for continual learning with dynamic token expansion.  _CoRR_ , abs/2111.11326, 2021.
  * Ebrahimi et al. [2020] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach.  Uncertainty-guided continual learning with bayesian neural networks.  In _International Conference on Learning Representations, (ICLR)_ , 2020.
  * Gidaris and Komodakis [2018] Spyros Gidaris and Nikos Komodakis.  Dynamic few-shot visual learning without forgetting.  In _Conference on Computer Vision and Pattern Recognition, (CVPR)_. IEEE, 2018.
  * Goodfellow et al. [2014] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  In _NIPS_ , 2014.
  * Gupta et al. [2020] Pankaj Gupta, Yatin Chaudhary, Thomas A. Runkler, and Hinrich Schütze.  Neural topic modeling with continual lifelong learning.  In _International Conference on Machine Learning, (ICML)_. PMLR, 2020.
  * Hadsell et al. [2020] Raia Hadsell, D. Rao, Andrei A. Rusu, and Razvan Pascanu.  Embracing change: Continual learning in deep neural networks.  _Trends in Cognitive Sciences_ , 2020.
  * Hayes and Kanan [2020] Tyler L. Hayes and Christopher Kanan.  Lifelong machine learning with deep streaming linear discriminant analysis.  In _CVPR_ , 2020.
  * Hayes et al. [2019] Tyler L. Hayes, Nathan D. Cahill, and Christopher Kanan.  Memory efficient experience replay for streaming learning.  In _International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019_. IEEE, 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Conference on Computer Vision and Pattern Recognition, (CVPR)_. IEEE, 2016.
  * Hill et al. [2019] Felix Hill, Adam Santoro, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap.  Learning to make analogies by contrasting abstract relational structure.  In _7th International Conference on Learning Representations, ICLR_. OpenReview.net, 2019.
  * Hinton et al. [2015] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.  Distilling the knowledge in a neural network.  _ArXiv_ , 2015.
  * Hochreiter [1998] Sepp Hochreiter.  The vanishing gradient problem during learning recurrent neural nets and problem solutions.  _International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems_ , 1998.
  * Hocquet et al. [2020] Guillaume Hocquet, Olivier Bichler, and Damien Querlioz.  Ova-inn: Continual learning with invertible neural networks.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , 2020.
  * Hou et al. [2019] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Hsu et al. [2018] Yen-Chang Hsu, Y. Liu, and Z. Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines.  _ArXiv_ , abs/1810.12488, 2018.
  * Huang et al. [2006] Guang-Bin Huang, Lei Chen, and Chee Kheong Siew.  Universal approximation using incremental constructive feedforward networks with random hidden nodes.  _IEEE Trans. Neural Networks_ , 2006.
  * Huberman and Fattal [2016] Inbar Huberman and Raanan Fattal.  Detecting repeating objects using patch correlation analysis.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2016.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Francis R. Bach and David M. Blei, editors, _Proceedings of the 32nd International Conference on Machine Learning, ICML_ , JMLR Workshop and Conference Proceedings. JMLR.org, 2015.
  * Javed and Shafait [2018] Khurram Javed and Faisal Shafait.  Revisiting distillation and incremental classifier learning.  In _ACCV_ , 2018.
  * Johnson [1984] W. Johnson.  Extensions of lipschitz mappings into hilbert space.  _Contemporary mathematics_ , 1984.
  * Joseph et al. [2021] K. J. Joseph, Jathushan Rajasegaran, Salman Hameed Khan, Fahad Shahbaz Khan, Vineeth N. Balasubramanian, and Ling Shao.  Incremental object detection via meta-learning.  _IEEE transactions on pattern analysis and machine intelligence TPAMI_ , 2021.
  * Kingma and Welling [2014] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In Yoshua Bengio and Yann LeCun, editors, _2nd International Conference on Learning Representations, (ICLR)_ , 2014.
  * Kipf and Welling [2017] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In _5th International Conference on Learning Representations, ICLR 2017_. OpenReview.net, 2017.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 2017.
  * Knoblauch et al. [2020] Jeremias Knoblauch, Hisham Husain, and Tom Diethe.  Optimal continual learning has perfect memory and is np-hard.  In _ICML_ , 2020.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * Lama et al. [2017] Ramesh Kumar Lama, Jeonghwan Gwak, Jeong-Seon Park, and Sang-Woong Lee.  Diagnosis of alzheimer’s disease based on structural mri images using a regularized extreme learning machine and pca features.  _Journal of healthcare engineering_ , 2017.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 1998.
  * Lee et al. [2020] Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim.  A neural dirichlet process mixture model for task-free continual learning.  In _8th International Conference on Learning Representations, ICLR_ , 2020.
  * Lettry et al. [2017] Louis Lettry, Michal Perdoch, Kenneth Vanhoey, and Luc Van Gool.  Repeated pattern detection using CNN activations.  In _2017 IEEE Winter Conference on Applications of Computer Vision, WACV 2017_ , 2017.
  * Leung and Malik [1996] Thomas K. Leung and Jitendra Malik.  Detecting, localizing and grouping repeated scene elements from an image.  In _Computer Vision - ECCV’96, 4th European Conference on Computer Vision, Proceedings, Volume I_. Springer, 1996.
  * Li et al. [2022] Duo Li, Guimei Cao, Yunlu Xu, Zhanzhan Cheng, and Yi Niu.  Technical report for iccv 2021 challenge sslad-track3b: Transformers are better continual learners.  _arXiv preprint arXiv:2201.04924_ , 2022.
  * Li et al. [2007] Fei-Fei Li, Robert Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  _Comput. Vis. Image Underst._ , 2007.
  * Li and Hoiem [2017] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _transactions on pattern analysis and machine intelligence (PAMI)_ , 2017.
  * Linnainmaa [1976] Seppo Linnainmaa.  Taylor expansion of the accumulated rounding error.  _BIT Numerical Mathematics_ , 1976.
  * Liu and Liu [2013] Jingchen Liu and Yanxi Liu.  GRASP recurring patterns from a single view.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2013.
  * Liu et al. [2020a] Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer.  Generative feature replay for class-incremental learning.  In _Conference on Computer Vision and Pattern Recognition Workshops CVPRW_ , 2020a.
  * Liu et al. [2004] Yanxi Liu, Robert T. Collins, and Yanghai Tsin.  A computational model for periodic pattern perception based on frieze and wallpaper groups.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2004.
  * Liu et al. [2020b] Yaoyao Liu, Anan Liu, Yuting Su, Bernt Schiele, and Qianru Sun.  Mnemonics training: Multi-class incremental learning without forgetting.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020b.
  * Liu et al. [2019] Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Jia-Wang Bian, Le Zhang, Xiang Bai, and Jinhui Tang.  Richer convolutional features for edge detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  _Proc. ICCV_ , 2021.
  * Lomonaco and Maltoni [2017] Vincenzo Lomonaco and Davide Maltoni.  Core50: a new dataset and benchmark for continuous object recognition.  In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg, editors, _Proceedings of the 1st Annual Conference on Robot Learning_ , Proceedings of Machine Learning Research. PMLR, 2017.
  * Lomonaco et al. [2020] Vincenzo Lomonaco, Karan Desai, Eugenio Culurciello, and Davide Maltoni.  Continual reinforcement learning in 3d non-stationary environments.  In _Conference on Computer Vision and Pattern Recognition, (CVPR)_. IEEE, 2020.
  * Lomonaco et al. [2022] Vincenzo Lomonaco, Lorenzo Pellegrini, Pau Rodríguez López, Massimo Caccia, Qi She, Yu Chen, Quentin Jodelet, Ruiping Wang, Zheda Mai, David Vázquez, German Ignacio Parisi, Nikhil Churamani, Marc Pickett, Issam H. Laradji, and Davide Maltoni.  Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions.  _Artif. Intell._ , 2022.
  * Lopez-Paz and Ranzato [2017] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with warm restarts.  In _International Conference on Learning Representations, (ICLR)_ , 2017.
  * Maas et al. [2013] Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng.  Rectifier nonlinearities improve neural network acoustic models.  In _in ICML Workshop on Deep Learning for Audio, Speech and Language Processing_ , 2013.
  * Mai et al. [2022] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo J. Kim, and Scott Sanner.  Online continual learning in image classification: An empirical survey.  _Neurocomputing_ , 2022.
  * Mallya and Lazebnik [2018] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR_. Computer Vision Foundation / IEEE Computer Society, 2018\.
  * McCaffary [2021] David McCaffary.  Towards continual task learning in artificial neural networks: current approaches and insights from neuroscience.  _ArXiv_ , 2021.
  * McClelland et al. [1995] James L. McClelland, Bruce L. McNaughton, and Randall C. O’Reilly.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.  _Psychological review_ , 1995.
  * McCloskey and Cohen [1989] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_. Elsevier, 1989.
  * McCulloch and Pitts [1943] Warren S McCulloch and Walter Pitts.  A logical calculus of the ideas immanent in nervous activity.  _The bulletin of mathematical biophysics_ , 1943.
  * Mehta et al. [2021] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell.  An empirical investigation of the role of pre-training in lifelong learning.  _ArXiv_ , abs/2112.09153, 2021.
  * Michieli and Zanuttigh [2019] Umberto Michieli and Pietro Zanuttigh.  Incremental learning techniques for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCV-W)_ , 2019.
  * Mirzadeh et al. [2022] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Architecture matters in continual learning.  _arXiv preprint arXiv:2202.00275_ , 2022.
  * Morrison et al. [2021] Katelyn Morrison, Benjamin Gilby, Colton Lipchak, Adam Mattioli, and Adriana Kovashka.  Exploring corruption robustness: Inductive biases in vision transformers and mlp-mixers.  _CoRR_ , abs/2106.13122, 2021.
  * Mundt et al. [2020] Martin Mundt, Yong Won Hong, Iuliia Pliushch, and Visvanathan Ramesh.  A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning.  _ArXiv_ , abs/2009.01797, 2020.
  * Nair and Hinton [2010] Vinod Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In _ICML_ , 2010.
  * Olah et al. [2017] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  _Distill_ , 2017.  doi: 10.23915/distill.00007.  https://distill.pub/2017/feature-visualization.
  * Ono et al. [2018] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi.  Lf-net: Learning local features from images.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 31, NeurIPS_ , 2018.
  * Parisi et al. [2019] German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 2019.
  * Park et al. [2009] Minwoo Park, Kyle Brocklehurst, Robert T. Collins, and Yanxi Liu.  Deformed lattice detection in real-world images using mean-shift belief propagation.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2009.
  * Paul and Chen [2021] Sayak Paul and Pin-Yu Chen.  Vision transformers are robust learners.  _arXiv preprint arXiv:2105.07581_ , 2021.
  * Pomponi et al. [2020] Jary Pomponi, Simone Scardapane, Vincenzo Lomonaco, and Aurelio Uncini.  Efficient continual learning in neural networks with embedding regularization.  _Neurocomputing_ , 2020.
  * Pourkeshavarz and Sabokrou [2022] Mozhgan Pourkeshavarz and M. Sabokrou.  Zs-il: Looking back on learned experiences for zero-shot incremental learning.  _ICLR_ , 2022.
  * Prabhu et al. [2020] Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _Computer Vision - (ECCV)_ , Lecture Notes in Computer Science. Springer, 2020.
  * Pritts et al. [2014] James Pritts, Ondrej Chum, and Jiri Matas.  Rectification, and segmentation of coplanar repeated patterns.  In _2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2014.
  * Qureshi et al. [2016] Muhammad Naveed Iqbal Qureshi, Beomjun Min, Hang Joon Jo, and Boreom Lee.  Multiclass classification for the differential diagnosis on the adhd subtypes using recursive feature elimination and hierarchical extreme learning machine: structural mri study.  _PloS one_ , 2016.
  * Raghu et al. [2021] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  _ArXiv_ , 2021.
  * Rajasegaran et al. [2019] Jathushan Rajasegaran, Munawar Hayat, Salman H. Khan, Fahad Shahbaz Khan, and Ling Shao.  Random path selection for continual learning.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS_ , 2019.
  * Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In _Conference on Computer Vision and Pattern Recognition, (CVPR)_. IEEE, 2017.
  * Riesenhuber and Poggio [1999] Maximilian Riesenhuber and Tomaso A. Poggio.  Hierarchical models of object recognition in cortex.  _Nature Neuroscience_ , 1999.
  * Ring et al. [1994] Mark Bishop Ring et al.  _Continual learning in reinforcement environments_.  PhD thesis, 1994.
  * Ritter et al. [2018] Hippolyt Ritter, Aleksandar Botev, and David Barber.  Online structured laplace approximations for overcoming catastrophic forgetting.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Rodríguez-Pardo et al. [2019] Carlos Rodríguez-Pardo, Sergio Suja, David Pascual, Jorge Lopez-Moreno, and Elena Garces.  Automatic extraction and synthesis of regular repeatable patterns.  _Comput. Graph._ , 2019.
  * Rolnick et al. [2019] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS_ , 2019.
  * Rosenblatt [1958] Frank Rosenblatt.  The perceptron: a probabilistic model for information storage and organization in the brain.  _Psychological review_ , 1958.
  * Roy et al. [2021] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier.  Efficient content-based sparse attention with routing transformers.  _Trans. Assoc. Comput. Linguistics_ , 2021.
  * Rumelhart et al. [1986] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.  Learning representations by back-propagating errors.  _Nature_ , 1986.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  _Int. J. Comput. Vis._ , 2015.
  * Rusu et al. [2016] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _ArXiv_ , 2016.
  * Sabour et al. [2017] Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton.  Dynamic routing between capsules.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems NIPS_ , 2017.
  * [125] Frederik Schaffalitzky and Andrew Zisserman.  Geometric grouping of repeated elements within images.  In John N. Carter and Mark S. Nixon, editors, _Proceedings of the British Machine Vision Conference 1998, BMVC_.
  * Schmidhuber [2014] Jürgen Schmidhuber.  Deep learning in neural networks: An overview.  _CoRR_ , abs/1404.7828, 2014.
  * Shechtman and Irani [2007] Eli Shechtman and Michal Irani.  Matching local self-similarities across images and videos.  In _2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007)_. IEEE Computer Society, 2007.
  * Shim et al. [2021] Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang.  Online class-incremental continual learning with adversarial shapley value.  _AAAI_ , 2021.
  * Shin et al. [2017] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems NIPS_ , 2017.
  * Sodhani et al. [2020] Shagun Sodhani, Sarath Chandar, and Yoshua Bengio.  Toward training recurrent neural networks for lifelong learning.  _Neural Computation_ , 2020.
  * Stutz et al. [2018] David Stutz, Alexander Hermans, and Bastian Leibe.  Superpixels: An evaluation of the state-of-the-art.  _Comput. Vis. Image Underst._ , 2018.
  * Sun et al. [2020] Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee.  LAMOL: language modeling for lifelong language learning.  In _International Conference on Learning Representations, (ICLR)_ , 2020.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , 2015.
  * Tan et al. [2018] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu.  A survey on deep transfer learning.  In Vera Kurková, Yannis Manolopoulos, Barbara Hammer, Lazaros S. Iliadis, and Ilias Maglogiannis, editors, _International Conference on Artificial Neural Networks (ICANN)_ , Lecture Notes in Computer Science. Springer, 2018.
  * Tasic et al. [2018] Bosiljka Tasic, Zizhen Yao, Lucas T. Graybuck, Kimberly A. Smith, Thuc Nghi Nguyen, Darren Bertagnolli, Jeff Goldy, Emma Garren, Michael N. Economo, Sarada Viswanathan, Osnat Penn, Trygve E Bakken, Vilas Menon, Jeremy A. Miller, Olivia Fong, Karla E. Hirokawa, Kanan Lathia, Christine Rimorin, Michael Tieu, Rachael Larsen, Tamara Casper, Eliza Barkan, Matthew Kroll, Sheana E. Parry, Nadiya Shapovalova, Daniel Hirschstein, Julie Pendergraft, Heather Anne Sullivan, Tae Kyung Kim, Aaron Szafer, Nick Dee, Peter A. Groblewski, Ian R. Wickersham, Ali H. Cetin, Julie A. Harris, Boaz P. Levi, Susan M. Sunkin, Linda J. Madisen, Tanya L. Daigle, Loren L. Looger, Amy Bernard, John W. Phillips, Ed S. Lein, Michael J. Hawrylycz, Karel Svoboda, Allan R. Jones, Christof Koch, and Hongkui Zeng.  Shared and distinct transcriptomic cell types across neocortical areas.  _Nature_ , 2018.
  * Thrun [1995a] Sebastian Thrun.  _Explanation-based neural network learning a lifelong learning approach_.  PhD thesis, University of Bonn, Germany, 1995a.
  * Thrun [1995b] Sebastian Thrun.  Is learning the n-th thing any easier than learning the first?  In David S. Touretzky, Michael Mozer, and Michael E. Hasselmo, editors, _Advances in Neural Information Processing Systems, (NIPS)_. MIT Press, 1995b.
  * Thrun and Mitchell [1995] Sebastian Thrun and Tom M. Mitchell.  Lifelong robot learning.  _Robotics Auton. Syst._ , 15(1-2):25–46, 1995\.
  * Torii et al. [2015] Akihiko Torii, Josef Sivic, Masatoshi Okutomi, and Tomás Pajdla.  Visual place recognition with repetitive structures.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2015.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In Marina Meila and Tong Zhang, editors, _ICML_ , 2021.
  * van de Ven and Tolias [2019] Gido M. van de Ven and A. Tolias.  Three scenarios for continual learning.  _ArXiv_ , 2019.
  * van de Ven and Tolias [2018] Gido M. van de Ven and Andreas Savas Tolias.  Generative replay with feedback connections as a general strategy for continual learning.  _ArXiv_ , 2018.
  * van den Oord et al. [2016] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  In _The 9th ISCA Speech Synthesis Workshop_. ISCA, 2016.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems NIPS_ , 2017.
  * Wang et al. [2014] Hongxing Wang, Gangqiang Zhao, and Junsong Yuan.  Visual pattern discovery in image and video data: a brief survey.  _Wiley Interdiscip. Rev. Data Min. Knowl. Discov._ , 2014.
  * Wang et al. [2021] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia.  End-to-end video instance segmentation with transformers.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Wightman [2019] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * Wu et al. [2018] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Zhengyou Zhang, and Yun Raymond Fu.  Incremental classifier learning with generative adversarial networks.  _ArXiv_ , 2018.
  * Wu et al. [2019] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Raymond Fu.  Large scale incremental learning.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Xiao et al. [2021] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross B. Girshick.  Early convolutions help transformers see better.  _CoRR_ , abs/2106.14881, 2021.
  * Xu and Zhu [2018] Ju Xu and Zhanxing Zhu.  Reinforced continual learning.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, (NeurIPS)_ , 2018.
  * Yang et al. [2021] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao.  Focal self-attention for local-global interactions in vision transformers.  _ArXiv_ , abs/2107.00641, 2021.
  * Yoon et al. [2018] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In _6th International Conference on Learning Representations, ICLR_. OpenReview.net, 2018.
  * Yu et al. [2020a] Lu Yu, X. Liu, and J. Weijer.  Self-training for class-incremental semantic segmentation.  _ArXiv_ , 2020a.
  * Yu et al. [2020b] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer.  Semantic drift compensation for class-incremental learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR_ , 2020b.
  * Yu et al. [2021] Pei Yu, Yinpeng Chen, Ying Jin, and Zicheng Liu.  Improving vision transformers for incremental learning.  _ArXiv_ , abs/2112.06103, 2021.
  * Yun et al. [2019] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _IEEE/CVF International Conference on Computer Vision, (ICCV)_. IEEE, 2019.
  * Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _Proceedings of the 34th International Conference on Machine Learning_ , Proceedings of Machine Learning Research. PMLR, 2017.