# Clustering-based Domain-Incremental Learning

Christiaan Lamers 111Corresponding author.
NORCE
Grimstad, 4879, Norway
<EMAIL_ADDRESS>&René Vidal
Institute for Data Engineering and Science
University of Pennsylvania
Philadelphia, USA
<EMAIL_ADDRESS>&Nabil Belbachir
NORCE
Grimstad, 4879, Norway
<EMAIL_ADDRESS>&Niki van Stein
Leiden Institute of Advanced Computer Science
Leiden, 2333 CA, The Netherlands
<EMAIL_ADDRESS>&Thomas Bäck
Leiden Institute of Advanced Computer Science
Leiden, 2333 CA, The Netherlands
<EMAIL_ADDRESS>&Paris Giampouras
Johns Hopkins University
Baltimore, MD 21218, US
<EMAIL_ADDRESS>
###### Abstract

We consider the problem of learning multiple tasks in a continual learning
setting in which data from different tasks is presented to the learner in a
streaming fashion. A key challenge in this setting is the so-called
“catastrophic forgetting problem", in which the performance of the learner in
an “old task" decreases when subsequently trained on a “new task". Existing
continual learning methods, such as Averaged Gradient Episodic Memory (A-GEM)
and Orthogonal Gradient Descent (OGD), address catastrophic forgetting by
minimizing the loss for the current task without increasing the loss for
previous tasks. However, these methods assume the learner knows when the task
changes, which is unrealistic in practice. In this paper, we alleviate the
need to provide the algorithm with information about task changes by using an
online clustering-based approach on a dynamically updated finite pool of
samples or gradients. We thereby successfully counteract catastrophic
forgetting in one of the hardest settings, namely: domain-incremental
learning, a setting for which the problem was previously unsolved. We showcase
the benefits of our approach by applying these ideas to projection-based
methods, such as A-GEM and OGD, which lead to task-agnostic versions of them.
Experiments on real datasets demonstrate the effectiveness of the proposed
strategy and its promising performance compared to state-of-the-art methods.

This work is supported by the project ULEARN “Unsupervised Lifelong Learning”
and co-funded under the grant number 316080 of the Research Council of Norway.

##  1 Introduction

Continual learning can be described as the ability to continually learn over
time by accommodating new knowledge while retaining previously learned
experiences Thrun (1998); Parisi et al. (2019). We humans typically have no
problem with retaining old experiences while at the same time being able to
learn new tasks. For example: when a child learns to ride a bike, she does not
forget the previous experience of learning how to walk.

In sharp contrast, standard machine learning algorithms typically assume that
independent and identically distributed (i.i.d.) training examples of a task
are given and use Empirical Risk Minimization (ERM) to learn a model for the
task Vapnik (1999). While this approach can be naturally extended to the
setting in which samples arrive in an online fashion, when the task changes
the conditional distribution of the data given the task also changes. As a
consequence, the performance of the model on previously learned tasks
significantly degrades when trained on new tasks, a phenomenon known as
catastrophic forgetting.

Existing methods that deal with catastrophic forgetting often assume that the
moment the task changes and the identity of the task are known at training
time. For instance, Averaged Gradient Episodic Memory (A-GEM) Chaudhry et al.
(2018) and Orthogonal Gradient Descent (OGD) Farajtabar et al. (2020)
counteract catastrophic forgetting by solving a constrained optimization
problem for each task change, which ensures that the loss function: a)
decreases on the current task and b) does not increase on previous tasks. The
constraints on previous tasks are enforced by storing either _labeled data
samples_ (A-GEM) or _model gradients_ (OGD) from previous tasks as new tasks
incrementally arrive. Thus, knowledge of a task change is needed to both solve
the constrained optimization problem and update the pool of stored samples or
gradients. Moreover, both A-GEM and OGD use pool size that grows with the
number of tasks, making memory requirements prohibitive for a large number of
tasks. While such memory requirements could be reduced by maintaining a
constant and finite memory, this would inevitably lead to catastrophic
forgetting as the number of tasks grows.

The aforementioned weaknesses raise two critical questions:

  1. 1.

_Can we develop a memory and projection-based continual learning algorithm
that does not require knowledge of task boundaries?_

  2. 2.

_Can we address catastrophic forgetting more effectively for a large number of
tasks while maintaining a constant and finite amount of memory?_

Figure 1: After the task-incremental method is finished with the training on
task $T_{k}$, the memory (containing either labeled data samples in the case
of A-GEM or model gradients in the case of OGD) is updated. This method is
made domain-incremental by using an online clustering-based approach for
updating the memory while keeping its size fixed.

#### Paper contributions.

In this work, we address these questions by proposing an online clustering-
based approach that renders standard projection-based continual learning
algorithms task-agnostic. This approach successfully counteracts forgetting in
the setting of domain-incremental learning, a setting for which this problem
was previously unsolved van de Ven et al. (2022). The proposed approach is
generic and can be applied to different projection-based algorithms. To
showcase its merits, we focus on the A-GEM and OGD algorithms and propose two
new task-agnostics versions called Task Agnostic Averaged Gradient Episodic
Memory (TA-A-GEM) and Task Agnostic Orthogonal Gradient Descent (TA-OGD).
These algorithms reduce the amount of forgetting when training on different
tasks without the need to know any task boundaries and identities. This is
achieved by dynamically updating the pool of _labeled data samples_ (A-GEM) or
_model gradients_ (OGD) each time a new batch becomes available. In addition,
unlike A-GEM and OGD, which store a growing number of samples or gradients as
the number of tasks increases, leading to prohibitive memory requirements in
practical scenarios, the proposed TA-A-GEM and TA-OGD methods have constant
and finite memory requirements by keeping a finite number of samples or
gradients throughout the training process. To achieve this, TA-A-GEM and TA-
OGD leverage the structure of the training data, which are now grouped into
clusters of samples or gradients. Specifically, for each new batch, we first
uniformly draw samples or gradients from the current batch and use them to
initialize a predefined number of clusters using the samples or gradients as
the cluster centers. After initialization, new samples or gradients are
assigned to the cluster center with minimum $\ell_{2}$ distance. To keep a
constant memory, when the maximum cluster size is reached we remove less
informative cluster members and update the cluster center with the average of
the cluster members.

In short, this paper makes the following contributions:

  * •

We propose a generic clustering-based method for successfully extending
projection-based continual learning algorithms to a task-agnostic context. We
focus on two state-of-the-art projection-based algorithms i.e., A-GEM and OGD
showing that the proposed strategy enjoys the merits of memory and projection-
based methods Farajtabar et al. (2020); Lopez-Paz and Ranzato (2017); Doan et
al. (2020) without requiring knowledge of the task identity or task changes.

  * •

By leveraging the structure of the data from previously seen tasks, we can
retain the information needed to address catastrophic forgetting, such as
training data (A-GEM) or model gradients (OGD), while keeping the memory-size
finite via a simple and efficient clustering procedure. We thus depart from
the standard approach of OGD and A-GEM, which demand a growing amount of
memory as new tasks sequentially arrive, which is impractical in real-world
scenarios.

  * •

We provide extensive experimental results for different continual learning
settings on various datasets showing the promising performance of the proposed
task-agnostic algorithms (TA-A-GEM and TA-OGD) compared to state-of-the-art
methods.

##  2 Related Work

This section starts with an explanation of the three types of incremental
learning. It then reviews the stability-plasticity dilemma, which continual
learning methods have to face. Moreover, we present the main ideas of memory
and projection-based continual learning approaches to which class the proposed
TA-A-GEM and TA-OGD method belong and the main advances in task continual
learning. Finally, we review the recent works leveraging representation
learning for deriving efficient continual learning algorithms.

###  2.1 Domain-incremental learning

In continual learning, different tasks can arrive in sequence. The learner
must therefore learn new tasks incrementally. This is referred to as
incremental learning. Three types of incremental learning can be specified:
task-incremental learning, domain-incremental learning and class-incremental
learning van de Ven et al. (2022). In task-incremental learning, the task
identity is known to the learner during the training and testing phase. In
domain-incremental learning, the task identity is not known to the learner at
both training and testing time. In class-incremental learning, the learner
must learn to identify a growing number of classes. Since we focus on a
scenario where the number of classes is static and the task identity is not
known during training and testing, we focus on the domain-incremental setting.
Alleviating catastrophic forgetting in such a scenario is an important
unsolved challenge van de Ven et al. (2022).

###  2.2 The Stability-Plasticity Dilemma

The balancing act between being able to gain new knowledge while assuring old
knowledge is not lost is referred to as the stability-plasticity dilemma
Mermillod et al. (2013). Continual learning approaches can be categorized in
three major trends based on how the stability-plasticity dilemma is handled De
Lange et al. (2021); Parisi et al. (2019). The first trend is to use the
concept of regularization of synaptic plasticity, where the plasticity of
important weights is constrained in order to retain old skills, like the
Memory Aware Synapses used in a continual setting in Aljundi et al. (2019a).
Elastic Weight Consolidation (EWC) is a seminal work of this class. When a new
task arrives, EWC learns the optimal weights for this task, while penalizing
changes of the weights towards values that are far from the optimal ones for
the previous task Kirkpatrick et al. (2017). Several other variants of EWC
have appeared in the literature and we refer the readers to De Lange et al.
(2021) for a detailed review. The second trend is expansion Rusu et al.
(2016); Aljundi et al. (2017); Mehta et al. (2021); Douillard et al. (2022),
where a neural network is expanded by allocating new neural resources in order
to gain new skills, while leaving old neurons unchanged in order to retain old
skills. Finally, according to the third trend, which is repetition, old
information is repeatedly fed to the network, along with new information. This
can be implemented by applying a complementary learning system for integrating
old and new skills and applying experience replay, or by simply mixing old and
new data in the training step. In the literature, various approaches of the
so-called replay-based methods which rely on the principle of repetition have
come to the scene. These methods make use of memory resources and vary in the
strategy they follow Rebuffi et al. (2017); Lopez-Paz and Ranzato (2017); Shin
et al. (2017); Chaudhry et al. (2019); Aljundi et al. (2019b); van de Ven et
al. (2020); Koh et al. (2021); Ye and Bors (2022).

This paper uses the terms “replay-based” and “memory-based” interchangeably
because they represent similar concepts. Still, we tend to favor “replay-
based” when a method stores samples from the dataset and “memory-based” when
it stores different information. The proposed TA-A-GEM builds on A-GEM
Chaudhry et al. (2018), which stores samples from the training set, and can
thus be considered “replay based”. The proposed TA-OGD builds on OGD
Farajtabar et al. (2020), and thus, in principle, falls into the category of
memory-based methods since it stores gradients. At the same time, the proposed
TA-A-GEM and TA-OGD use a projected gradient step and, hence, are also a
projection-based approach. Note that this projection step implicitly
regularizes the weights; therefore, A-GEM and OGD bear similarities with the
regularization-based methods. Next, we elaborate on the specific class of
memory-based and projection-based continual learning algorithms.

###  2.3 Memory-based and Projection-based Continual Learning Methods

Over the last few years, several memory-based and projection-based methods
have been proposed in the literature, Lopez-Paz and Ranzato (2017); Farajtabar
et al. (2020). These make use of memory for storing information from the past,
which helps to update the model towards non-forgetting directions. The goal is
to address catastrophic forgetting by means of imposing certain constraints on
the weight-updating process. Many different approaches have appeared in the
literature over the last few years. In Lopez-Paz and Ranzato (2017), the
authors propose to update weights in directions that do not increase the loss
function values on samples of previously seen tasks. The resulting algorithm,
dubbed Gradient Episodic Memory (GEM), thus stores a predefined number of
gradients of the loss function corresponding to old tasks, Chaudhry et al.
(2018); Lopez-Paz and Ranzato (2017). These are then used for updating the
model by solving a constrained optimization problem. Orthogonal Gradient
Descent (OGD) Farajtabar et al. (2020) stores a growing number of gradients of
the model corresponding to old tasks’ samples. In the weight update step, it
projects its loss gradient to a direction that is orthogonal to all stored
gradients. Specifically, gradients of the loss are projected on the orthogonal
basis spanned by the stored gradients. In doing so, directions that increase
forgetting of past tasks are excluded when the model learns a new task. This
assumes however that the stored gradients remain relevant, even when the
weights of the model move during the training process, thus arriving at a
different point in the configuration space in which older tasks can have
different gradients. Averaged Gradient Episodic Memory (A-GEM) Chaudhry et al.
(2018) solves this problem by storing labeled data samples instead of
gradients. It projects the loss gradient orthogonal to a reference gradient
that is calculated at every training step from a subset of the stored labeled
data. Though showing promising performance in addressing catastrophic
forgetting, memory-based and projection-based methods suffer from two
fundamental weaknesses: a) they require the moment of task change to be
available in order to know when the memory should be updated, and b) memory
cost should either scale with the number of tasks, e.g., in OGD Farajtabar et
al. (2020), which is infeasible in real-world scenarios, or the stored data
per task will decrease as in the case of GEM Lopez-Paz and Ranzato (2017),
which also hinders the ability of the algorithm to address forgetting when it
encounters a large number of tasks.

###  2.4 Task Agnostic Continual Learning

Task boundaries and identities are rarely available in practical continual
learning applications. In light of this, various task-agnostic continual
learning methods have been proposed in the literature. In Harrison et al.
(2020), the authors propose an auxiliary mechanism to detect tasks while
counteracting forgetting. The resulting method operates in a task-agnostic
environment showing promising empirical performance. Several other approaches
have been proposed in the same spirit Caccia et al. (2020); He et al. (2019).
Another line of work hinges on online learning ideas completely neglecting
task identity or the need to know the moment of task change. In Zeno et al.
(2018), the authors propose Bayesian Gradient Descent (BGD), an online
variational Bayes approach in which model parameters with low variance are
considered more important for previous tasks and, thus, are less updated. The
opposite holds for parameters with high variance (hence high uncertainty). A
similar idea for task-free continual learning appeared in Aljundi et al.
(2019c). Namely, the authors modified the so-called Memory Aware Synapses
(MAS) algorithm in Aljundi et al. (2018), in order to operate in a task-
agnostic online learning setup. For, they use an importance weight regularizer
which penalizes changes to model parameters which negatively affect model
performance on prior tasks. Finally, in Jin et al. (2020) the authors propose
an online task-agnostic memory-based method. The main idea is to edit the
stored-in-memory gradients used for addressing forgetting by solving an
optimization problem in an online fashion. Recently, the idea of using self-
supervised representations for task-agnostic continual learning was proposed
in Pham et al. (2021), showing promising empirical performance.

Though the emergence of clustering in episodic memory has been recently
acknowledged in the child development literature Horn et al. (2021), to the
best of our knowledge, the proposed TA-A-GEM and TA-OGD are the first
algorithms that use online clustering for dynamically updating the memory of
continual learning methods. While we focus on A-GEM and OGD, the adopted
strategy could be applied to other memory-based and task-dependent continual
learning approaches for allowing them to operate in task-agnostic
environments.

###  2.5 Representation Learning

Representation learning aims to find insightful data representations by
exploiting their structure Ma et al. (2022). Recently, learned representations
have been at the heart of several continual learning algorithms. In Chaudhry
et al. (2020), the authors employed low-rank orthogonal subspace
representations of the model parameters formulating continual learning as an
optimization over the Stiefel manifold problem. The reported results showed
promising performance and the ability of the approach to counteract
forgetting. In Guo et al. (2022), holistic representations learned via a
mutual information maximization criterion were employed in the continual
learning setting. The method can learn feature representations of the current
task that are useful for the future tasks, hence leading to models that are
more robust to forgetting. In Doan et al. (2020), a variant of the projection-
based OGD method was proposed. The main idea is to perform principal component
analysis on the set of stored gradients of the model and keep only the most
informative principal components. However, the work in Doan et al. (2020),
still assumes that task changes are provided to the algorithms and batch
processing is utilized. Hence it is far from our proposed online clustering-
based task-agnostic algorithms.

##  3 Proposed Approach

We assume that the $n$ tasks $\\{T_{i}\\}_{i=1}^{n}$ arrive sequentially and
that during task $T_{k}$ the data from tasks $T_{i}$ for $i<k$ are not
presented to the learner. Each task consists of pairs of data points $(x,y)\in
T_{k}$, where $x\in\mathbb{R}^{d}$ is the input and $y$ is a label. Here we
assume that each task is a classification task and that all classification
tasks share the same classes $j=1,\dots,c$, where $c$ is the number of
classes. Therefore, we can represent $y\in\mathbb{R}^{c}$ as a one-hot class
encoding vector, i.e., $y_{j}=1$ when $j$ is the class label and $y_{j}=0$
otherwise. We denote the network model as $f(x;w)\in\mathbb{R}^{c}$, where
$w\in\mathbb{R}^{p}$ denotes the $p$-dimensional weights (parameters) of the
network and $f_{j}(x;w)$ is the $j$-th logit corresponding to the $j$-th
class. The model is trained to predict the class label for input $x$.

The proposed Task Agnostic Averaged Gradient Episodic Memory (TA-A-GEM) and
Task Agnostic Orthogonal Gradient Descent (TA-OGD) methods rely on the
forgetting counteracting mechanisms of Averaged Gradient Episodic Memory
(A-GEM) Chaudhry et al. (2018) and Orthogonal Gradient Descent (OGD)
Farajtabar et al. (2020), respectively. Next, we briefly describe the main
ideas behind A-GEM and OGD and refer the reader to the Appendix or Chaudhry et
al. (2018) and Farajtabar et al. (2020) for further details.

Both A-GEM and OGD assume the identity $k_{t}$ of the task $T_{k_{t}}$ at time
step $t$ is known. The empirical loss, during time step $t$, with a batch size
$|T_{k_{t}}|$, is given by,

$L_{t}(w)=\frac{1}{|T_{k}|}\sum_{(x,y)\in T_{k}}L_{(x,y)}(w),$ (1)

where the per sample loss $L_{(x,y)}(w)$ is assumed to be the cross-entropy,
which is defined as

$L_{(x,y)}(w)=-\sum_{j=1}^{c}y_{j}\log\left(\frac{\exp
f_{j}(x;w)}{\sum^{c}_{m=1}\exp f_{m}(x;w)}\right).$ (2)

Both A-GEM and OGD use a pool of samples to counteract the catastrophic
forgetting. The difference is that OGD stores network gradient, while A-GEM
stores training data.

###  3.1 Clustering-based Task Agnostic A-GEM (TA-A-GEM) and OGD (TA-OGD)

Figure 1 shows our strategy to convert a task-aware task-incremental
projection algorithm to a task-agnostic domain-incremental algorithm. Task-
incremental projection algorithms like A-GEM and OGD keep a pool of samples
from either the training data or model gradients, respectively. This pool of
samples is used to mitigate catastrophic forgetting of previous tasks through
projection. When the algorithm is finished with training on one task, it
stores samples from this task before it starts training on the new task. In
this way, it ensures that the samples in the pool are relevant for previous
tasks when addressing forgetting. However, this comes at the cost of requiring
to know the moment a task changes. In our approach, we make this process task-
agnostic by updating the pool of samples during the process of training, i.e.
the pool of samples is updated every time the model is trained on a batch.
This removes the need to know the moment the task changes but introduces the
problem that the size of the pool now grows more rapidly. However, our goal is
to keep the memory requirements constant in the number of tasks. Hence, a
strategy is necessary to decide which samples should be added to the pool and
which ones should be removed during the updating process. Our strategy aims to
select stored samples in a way that addresses forgetting all previous tasks in
the most efficient way while being constrained by constant and finite pool
size. Because we aim for a true task-agnostic setting, all tasks are made to
have the same label space, so the task identity can not be inferred from the
labels.

Next, we detail the proposed online clustering-based approach that consists of
the following four steps:

1) Initialization: We first set the number of clusters $Q$ and consider the
first $Q$ samples becoming available as the centers
$\boldsymbol{\mu}_{i},q=1,2,\dots,Q$ of these clusters.

2) Cluster assignment: A new sample $\mathbf{z}_{p}$ (corresponding to a
training sample in the case of A-GEM or gradient logit in the case OGD) is
assigned to the cluster $q^{\ast}$ that minimizes the $\ell_{2}$ norm i.e.,

$q^{\ast}=\mathrm{argmin}_{q\in\\{1,2,\dots,Q\\}}\|\mathbf{z}_{p}-\boldsymbol{\mu}_{q}\|^{2}_{2}$
(3)

3) Memory update: The size of each cluster is predefined, and once the maximum
size has been reached, for new samples to that assigned to that cluster an
equal number of older samples residing in the cluster should be removed. Note
that the process of accepting/rejecting new samples and deciding which “old”
samples to delete could be implemented using information-theoretic criteria or
rejection sampling-based ideas. Here, in an effort to simplify the approach
and make it computationally efficient, we follow a first-in-first-out (FIFO)
approach. This dictates that samples that arrived first in the cluster are the
first to be removed. Note that the strategy followed ensures that samples
corresponding to a task with information distinct from other tasks will not be
deleted from the pool. This will occur since these samples will “live” within
clusters that will not be updated and thus remain unaffected by the memory
updating process.222Empirical findings reported in the Appendix corroborate
our hypothesis.

3) Update of cluster means: Once samples are assigned to the clusters and the
memory has been updated, the cluster means are re-computed i.e.,

$\boldsymbol{\mu}_{q}=\frac{1}{P}\sum^{P}_{p=1}{\mathbf{z}^{q}_{p}},\;\;\;\;\;\;\;\;\;\;\forall
i=1,2,\dots,N,$ (4)

where $P$ denotes the size of the clusters and $\mathbf{z}^{q}_{p}$ the
$p_{th}$ element of cluster $q$. For the case of the task-agnostic version of
A-GEM, i.e., TA-A-GEM, we have
$\mathbf{z}_{p}\equiv\mathbf{x}_{p}\in\tilde{M}_{t}$ (where $t$ here denotes
the batch index) whereas for the task-agnostic OGD algorithm (TA-OGD)
$\mathbf{z}_{p}\equiv\nabla f_{j}(\mathbf{x}_{p},w^{*}_{i})$. Our clustering-
based strategy is depicted at Fig. 2, while a pseudo-code of the algorithm is
given in the Appendix.

A single or a different pool for each class? A possible complication that can
occur is that more similarity exists between samples of the same class that
are of a different task than between different classes of the same task. If
this happens, class information will be well represented in the pool, but task
information can be easily lost. Since class labels of the samples are
available, a way to get around that issue and disentangle the class from task
information is to use a different pool for each class. In that case, samples
are first assigned to a pool based on their class label. Then, the procedure
described above is independently followed for each pool. It is worth noting
that this is critically important for the task-agnostic version of A-GEM (TA-
A-GEM) since the pool contains training samples of different classes. Samples
corresponding to the same class but different tasks, e.g., a digit and its
rotated version might be close in the input space. As a result, if a single
pool is used, those two samples will be assigned to the same cluster, and
hence task information will be lost. This phenomenon is more likely not to be
observed in the case of TA-OGD since clustering takes place in the space of
model gradients, which are sufficiently separated for different tasks even for
samples corresponding to same classes.

The role of hyperparameters: The choice of hyperparameters, such as the number
of clusters $Q$ and their size, is important. A large number of clusters $Q$,
allows more task and class diversity to be stored in different clusters in
memory. The size of the clusters should be large enough so it can capture the
essence of a specific task. However, the size of $Q$ and the cluster size
should be kept as small as possible to reduce the memory footprint. A trade-
off can be made where $Q$ is large, and the cluster size is small versus using
a small $Q$ with a large cluster size. In addition, we follow an adaptive
strategy for the learning rate of the projected gradient step. Note that this
is a form of task detection that our method does not necessarily need. Our
focus is to create a truly task-agnostic method without any task detection.
Specifically, the learning rate $\eta^{t}$ at iteration $t$ decreases as
follows:

$\eta^{t}=a\eta^{t-1},$ (5)

where $a<0$, when the loss function is smoothly increasing for a given number
of iterations. This allows the algorithm to update the weights of the model
following a non-increasing path for the loss function. Moreover, when a sudden
increase is observed, then the learning rate is reset to its initial value
(therefore increases), i.e., $\eta^{t}=\eta_{ini}$. The reasoning behind this
rule is that spikes of the loss most likely imply task-change and therefore, a
higher learning rate can help to move fast along decreasing directions of the
loss corresponding to the new task. Empirical results on the effect of the
sampling rate, the number, and the size of clusters on the performance of the
proposed method, and more details on the adaptive updating process of learning
rate, are provided in Section 4 and Appendix.

Figure 2: The clustering mechanism to add training set samples / model
gradient samples to the memory by matching it to the closest cluster (pink
cluster), as used by TA-A-GEM / TA-OGD.

##  4 Experiments

We divide the experiments into two main classes: a) the disjoint tasks
experiment and b) the continuous change experiments. The task-aware methods
are notified of the task change, while the task-agnostic methods do not get
this information. In the continuous change experiments, discrete tasks still
exist, but task boundaries are no longer clearly defined. Details on the
experimental setting can be found in the Appendix. Since there is no clear
point that a task-aware method can be notified, only task-agnostic methods are
included in this experiment. For both methods, all tasks are made to have the
same label space, since it should not be possible to infer the task identity
from the labels. In cases where the label spaces are disjoint, the labels are
cast to the same label space. Since no task identity is provided during
training, the method is tested in a domain-incremental setting van de Ven et
al. (2022). Following empirical observations, we use the learning rate
scheduler described in Section 3.3 for the case of OGD and the proposed task-
agnostic version of it i.e., TA-OGD. The network used for training is a multi-
layer perceptron (MLP) with two hidden layers of 200 nodes. To compare the
performance of the tested methods, we use three metrics: a) The validation
accuracy, b) The average validation accuracy over all tasks trained on thus
far and c) The amount of forgetting. For an exact mathematical definition of
these quantities , we refer to the Appendix. To create separate tasks from
existing datasets, three task generation mechanisms are implemented: a) task
permutation, b) task rotation and c) class splitting. For the details of this
task generation, we refer to the Appendix.

###  4.1 Disjoint tasks experiment

Table 1 shows the results of the first class of experiments. It shows the
average accuracy over all tasks trained thus far, thereby capturing both the
ability to remember old tasks and the ability to learn new tasks. The average
accuracy was then averaged over 20 epochs, then over five runs. Plots of these
results can be found in the Appendix. Our proposed TA-OGD and TA-A-GEM
algorithms significantly outperform the state-of-the-art task-agnostic BGD
algorithm, Zeno et al. (2018), on the MNIST Deng (2012), Fashion MNIST Xiao et
al. (2017) and NOT MNIST datasets. Moreover, their performance is comparable
to BGD on CIFAR10 and SVHN. Focusing on MNIST, Fashion MNIST and NOT MNIST, we
observe that at the permutation experiments, no remarkable differences can be
seen among the methods. This can be explained by the fact that the baseline
SGD method shows little signs of forgetting in the first place. For the
rotation experiments, A-GEM is a clear winner, it is however not task-
agnostic. On MNIST and NOT MNIST, TA-OGD and TA-A-GEM are moderately effective
at mitigating forgetting. On Fashion MNIST however, TA-A-GEM is clearly the
best method among all the tested task-agnostic methods. We attained the most
remarkable results on the class split experiments. On MNIST, both TA-OGD and
TA-A-GEM clearly outperform the other task-agnostic methods. On Fashion MNIST,
TA-A-GEM’s performance is even on par with A-GEM, while on NOT MNIST, TA-OGD
takes the crown by performing on par with A-GEM, which is a task-aware method.

| MNIST | Fashion MNIST | NOT MNIST | CIFAR10 | SVHN
---|---|---|---|---|---
| perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class
SGD | 0.842 | 0.657 | 0.804 | 0.735 | 0.468 | 0.807 | 0.850 | 0.598 | 0.864 | 0.373 | 0.346 | 0.724 | 0.599 | 0.392 | 0.759
SGD lr adapt | 0.842 | 0.663 | 0.811 | 0.736 | 0.469 | 0.820 | 0.851 | 0.598 | 0.888 | 0.376 | 0.347 | 0.727 | 0.596 | 0.390 | 0.762
BGD | 0.883 | 0.682 | 0.790 | 0.765 | 0.507 | 0.802 | 0.856 | 0.633 | 0.875 | 0.385 | 0.357 | 0.718 | 0.591 | 0.417 | 0.763
TA-OGD | 0.871 | 0.705 | 0.857 | 0.749 | 0.516 | 0.893 | 0.845 | 0.625 | 0.937 | 0.328 | 0.343 | 0.731 | 0.547 | 0.393 | 0.773
TA-A-GEM | 0.876 | 0.688 | 0.878 | 0.746 | 0.604 | 0.931 | 0.853 | 0.602 | 0.884 | 0.365 | 0.343 | 0.726 | 0.605 | 0.399 | 0.772
OGD | 0.865 | 0.690 | 0.822 | 0.757 | 0.512 | 0.839 | 0.846 | 0.627 | 0.925 | 0.360 | 0.348 | 0.731 | 0.587 | 0.400 | 0.768
A-GEM | 0.884 | 0.806 | 0.952 | 0.761 | 0.706 | 0.934 | 0.854 | 0.740 | 0.947 | 0.360 | 0.356 | 0.741 | 0.552 | 0.451 | 0.827

Table 1: Average validation accuracy, averaged over all tasks trained thus
far, then averaged over all epochs, then averaged over five runs, for the
disjoint tasks experiments when using a MLP. Per column, the best result for
the task-agnostic methods are written in bold. In case a task-agnostic
method’s result is less optimal and not significantly different from the best
result, with a confidence of 99%, it is also written in bold. The results for
the task-aware methods OGD and A-GEM are given for context. Since these
algorithms benefit from knowing task identities and changes, we just use them
here as baselines for indicating the best performance we can achieve.

###  4.2 Continuous task change experiment

The results of the continuous change experiments are extremely similar to the
results in the disjoint tasks experiments. They can be found in the Appendix.
These experiments show that the proposed TA-OGD and TA-A-GEM fare just as well
in the challenging setting where task boundaries are blurred.

###  4.3 Effectiveness of the clustering-based procedure

In order to demonstrate the benefits obtained by the proposed clustering-based
approach, we compared the performance of TA-A-GEM with and without clustering.
To deactivate clustering we skipped the cluster assignment step and new
samples were randomly to allocated clusters. Similarly to our approach, an
equal number of old samples of update clusters is removed to keep the memory
size constant. For this experiment, a MLP was trained on Fashion MNIST, with
the task split segmentation. All settings are the same as in the disjoint
tasks experiments.

Figure 3 and 4 show the content of each cluster during training time. Each
horizontal line corresponds to a cluster. Each task is associated with a
unique color, which represents the oldest task information that is present in
the cluster. The horizontal line changes color the moment that the last
information of the oldest task disappears from the cluster. Then, the second
oldest task information becomes the new oldest task information. The moment
that a new task starts -not available to the algorithms- is indicated by a
black vertical line. As it can be observed in Figs 3 and 4, clustering helps
in keeping a greater variety of task information in the gradient pool, with
samples from Task 0 or Task 1 still being present in clusters even after the
end of training on samples from Task 4. On the other hand, the use of random
cluster assignment results in information of old task being almost immediately
lost after a task change, thus illustrating the merits of our proposed
clustering-based approach.

Figure 3: Clustering helps us address forgetting: samples from old tasks
remain in the pool even after the of training on Task 4 ends. Figure 4: When
using random cluster assignment, the information of old tasks is almost
immediately lost, once a new tasks starts.

##  5 Conclusions and future directions

In an effort to counteract catastrophic forgetting in a task-agnostic setting,
we proposed a clustering-based strategy to make task-aware projection methods
task-agnostic with constant memory requirements. By leveraging the structure
in the sampled data (in the case of TA-A-GEM) and model gradients (in the case
of TA-OGD), we can effectively counteract catastrophic forgetting without
providing knowledge of a task change and the need of a growing amount of
memory. Extensive experimental results provided in section 4.3 and the
Appendix show the benefits of our clustering-based method. As a future
direction, we aspire to explore more sophisticated, yet computationally
efficient, methods for the clustering and memory update step. Our goal is also
to illustrate the merits of our method on larger networks such as a ResNet He
et al. (2016), or more complicated datasets such as ImageNet Deng et al.
(2009). It is worth noting that our proposed method is generic hence we also
intend to inquire its application as an off-the-shelf tool to other
projection-based methods.

## References

  * Thrun [1998] Sebastian Thrun.  _Lifelong Learning Algorithms_ , pages 181–209.  Springer US, Boston, MA, 1998.  ISBN 978-1-4615-5529-2.  doi:10.1007/978-1-4615-5529-2_8.  URL https://doi.org/10.1007/978-1-4615-5529-2_8.
  * Parisi et al. [2019] German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113:54–71, 2019.  ISSN 18792782.  doi:10.1016/j.neunet.2019.01.012.  URL https://doi.org/10.1016/j.neunet.2019.01.012.
  * Vapnik [1999] Vladimir N Vapnik.  An overview of statistical learning theory.  _IEEE transactions on neural networks_ , 10(5):988–999, 1999.
  * Chaudhry et al. [2018] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with A-GEM.  _arXiv preprint arXiv:1812.00420_ , 2018.
  * Farajtabar et al. [2020] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li.  Orthogonal gradient descent for continual learning.  In _International Conference on Artificial Intelligence and Statistics_ , volume 108, pages 3762–3773. PMLR, 26–28 Aug 2020.  URL https://proceedings.mlr.press/v108/farajtabar20a.html.
  * van de Ven et al. [2022] Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias.  Three types of incremental learning.  _Nature Machine Intelligence_ , 4(12):1185–1197, 2022.
  * Lopez-Paz and Ranzato [2017] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Doan et al. [2020] Thang Doan, Mehdi Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier.  A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix.  130, 2020.  URL http://arxiv.org/abs/2010.04003.
  * Mermillod et al. [2013] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin.  The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects, 2013.
  * De Lange et al. [2021] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(7):3366–3385, 2021.
  * Aljundi et al. [2019a] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars.  Task-free continual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11254–11263, 2019a.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114(13):3521–3526, 2017.
  * Rusu et al. [2016] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Aljundi et al. [2017] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.  Expert gate: Lifelong learning with a network of experts.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3366–3375, 2017.
  * Mehta et al. [2021] Nikhil Mehta, Kevin Liang, Vinay Kumar Verma, and Lawrence Carin.  Continual learning using a bayesian nonparametric dictionary of weight factors.  In _International Conference on Artificial Intelligence and Statistics_ , pages 100–108. PMLR, 2021.
  * Douillard et al. [2022] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord.  Dytox: Transformers for continual learning with dynamic token expansion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9285–9295, 2022.
  * Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  Icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 2001–2010, 2017.
  * Shin et al. [2017] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  _Advances in neural information processing systems_ , 30, 2017.
  * Chaudhry et al. [2019] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning.  _arXiv preprint arXiv:1902.10486_ , 2019.
  * Aljundi et al. [2019b] Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars.  Online continual learning with maximally interfered retrieval.  _arXiv:1908.04742_ , 2019b.
  * van de Ven et al. [2020] Gido M van de Ven, Hava T Siegelmann, and Andreas S Tolias.  Brain-inspired replay for continual learning with artificial neural networks.  _Nature communications_ , 11(1):1–14, 2020.
  * Koh et al. [2021] Hyunseo Koh, Dahyun Kim, Jung-Woo Ha, and Jonghyun Choi.  Online continual learning on class incremental blurry task configuration with anytime inference.  _arXiv preprint arXiv:2110.10031_ , 2021.
  * Ye and Bors [2022] Fei Ye and Adrian G Bors.  Task-free continual learning via online discrepancy distance learning.  _Advances in Neural Information Processing Systems_ , 35:23675–23688, 2022.
  * Harrison et al. [2020] James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone.  Continuous meta-learning without tasks.  _Advances in Neural Information Processing Systems_ , 2020-December(NeurIPS), 2020.  ISSN 10495258.
  * Caccia et al. [2020] Massimo Caccia, Pau Rodríguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, and Laurent Charlin.  Online fast adaptation and knowledge accumulation (OSAKA): A new approach to continual learning.  _Advances in Neural Information Processing Systems_ , 2020-December(NeurIPS), 2020.  ISSN 10495258.
  * He et al. [2019] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu.  Task agnostic continual learning via meta learning.  _arXiv preprint arXiv:1906.05201_ , 2019.
  * Zeno et al. [2018] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry.  Task agnostic continual learning using online variational bayes.  _arXiv preprint arXiv:1803.10123_ , 2018.
  * Aljundi et al. [2019c] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars.  Task-free continual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11254–11263, 2019c.
  * Aljundi et al. [2018] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 139–154, 2018.
  * Jin et al. [2020] Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren.  Gradient Based Memory Editing for Task-Free Continual Learning.  (NeurIPS):1–22, 2020.  URL http://arxiv.org/abs/2006.15294.
  * Pham et al. [2021] Quang Pham, Chenghao Liu, and Steven Hoi.  Dualnet: Continual learning, fast and slow.  _Advances in Neural Information Processing Systems_ , 34:16131–16144, 2021.
  * Horn et al. [2021] Sebastian S Horn, Ute J Bayen, and Martha Michalkiewicz.  The development of clustering in episodic memory: A cognitive-modeling approach.  _Child development_ , 92(1):239–257, 2021.
  * Ma et al. [2022] Yi Ma, Doris Tsao, and Heung-Yeung Shum.  On the principles of parsimony and self-consistency for the emergence of intelligence.  _Frontiers of Information Technology & Electronic Engineering_, 23(9):1298–1323, 2022.
  * Chaudhry et al. [2020] Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip Torr.  Continual learning in low-rank orthogonal subspaces.  _Advances in Neural Information Processing Systems_ , 33:9900–9911, 2020.
  * Guo et al. [2022] Yiduo Guo, Bing Liu, and Dongyan Zhao.  Online continual learning through mutual information maximization.  In _International Conference on Machine Learning_ , pages 8109–8126. PMLR, 2022.
  * Deng [2012] Li Deng.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * Xiao et al. [2017] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Farquhar and Gal [2018] Sebastian Farquhar and Yarin Gal.  Towards robust evaluations of continual learning.  _arXiv preprint arXiv:1805.09733_ , 2018.

##  Appendix A Averaged Gradient Episodic Memory

A-GEM uses a projected gradient descent-type approach for updating the weights
of the model. It first calculates the gradient of the loss function $\nabla
L_{t}(w)\in\mathbb{R}^{p}$ and then compares this to a reference gradient
$\nabla L_{ref}(w)\in\mathbb{R}^{p}$, of the loss function $L_{ref}(w)$ as
defined by

$L_{ref}(w)=\frac{1}{|\tilde{M}|}\sum_{(x,y)\in\tilde{M}}L_{(x,y)}(w),$ (6)

where $\tilde{M}$ is a randomly selected batch of data sampled from the
episodic memory $M$ which contains label training data from tasks seen before
time $t$ i.e., $M=\cup_{k<t}M_{k}$, where $M_{k}$ is the episodic memory for
task $k$. When the inner product between the gradient of the loss function
$\nabla L_{t}(w)$ and the reference gradient $\nabla L_{ref}(w)$ is non-
negative, the gradient update boils down to a vanilla gradient descent step
without projections. However, when this inner product is less than zero, the
weights of the model are updated using a projected gradient-type update, i.e.,

$w\leftarrow w-\eta\tilde{g}$ (7)

where $\eta$ denotes the step size and $\tilde{g}$ is defined as,

$\tilde{g}=\nabla L_{t}(w)-\frac{\nabla L_{t}(w)^{\top}\nabla
L_{ref}(w)}{\nabla L_{ref}(w)^{\top}\nabla L_{ref}(w)}\nabla L_{ref}(w).$ (8)

Note that $\tilde{g}$ is now orthogonal to the reference gradient $\nabla
L_{ref}(w)$; thus, if $\nabla L_{ref}(w)$ successfully represents a non-
forgetting direction of previously sees tasks, then the update rule of the
weights of the model in (7), decreases the loss on the current task without
incurring any forgetting of previous tasks Chaudhry et al. (2018).

Algorithm 1 shows pseudo-code for A-GEM Chaudhry et al. (2018). It uses both a
loss gradient $\nabla L(w)$ and a reference gradient $\nabla L_{ref}(w)$ that
is computed using a sub sample of the episodic memory, namely $\tilde{M}$. If
the dot product between the loss gradient and the reference gradient is
greater or equal to zero, gradient descent proceeds as normal. However, if it
is smaller than zero, the loss gradient is projected onto the reference
gradient and the resulting component is subtracted from the loss gradient. The
weights $w$ are then updated using this newly obtained gradient and the step
size $\eta$. After the training on task $k$ finishes, $s$ samples are taken
from the data of task $T_{k}$ and added to the episodic memory $M$.

Algorithm 1 Averaged Gradient Episodic Memory

Input: Task sequence $T_{1},T_{2},T_{3},...$ learning rate $\eta$
Output: Model

1:Initialize $M\leftarrow\\{\\};w\leftarrow w_{0}$

2:for Task ID $k=1,2,3,...$ do

3: repeat

4: $L(w)\leftarrow\frac{1}{|T_{k_{t}}|}\sum_{(x,y)\in T_{k_{t}}}L_{(x,y)}(w)$

5:
$L_{ref}(w)\leftarrow\frac{1}{|\tilde{M}|}\sum_{(x,y)\in\tilde{M}}L_{(x,y)}(w)$

6: if $\nabla L(w)^{\top}\nabla L_{ref}(w)\geq 0$ then

7: $\tilde{g}\leftarrow\nabla L(w)$

8: else

9: $g_{proj}\leftarrow\frac{\nabla L(w)^{\top}\nabla L_{ref}(w)}{\nabla
L_{ref}(w)^{\top}\nabla L_{ref}(w)}\nabla L_{ref}(w)$

10: $\tilde{g}\leftarrow\nabla L(w)-g_{proj}$

11: end if

12: $w\leftarrow w-\eta\tilde{g}$

13: until converge

14: for $i=\\{1,...,s\\}$ do

15: $(\mathbf{x},y)\sim T_{k}$

16: $M\leftarrow M\cup(\mathbf{x},y)$

17: end for

18:end for

##  Appendix B Orthogonal Gradient Descent

Orthognal Gradient Descent (OGD) departs from A-GEM, which uses the gradient
of the loss function, and performs projections leveraging gradients of the
output $f(x;w)$ of the model denoted as $\nabla f(x;w)\in\mathbb{R}^{p\times
c},x\in T_{k_{t}}$. We also denote as

$\nabla f_{j}(x;w)\in\mathbb{R}^{p},(x,y)\in T_{k_{t}}$ (9)

the gradient associated with the output logit $j$:

$f_{j}(x;w)\in\mathbb{R},(x,y)\in T_{k_{t}},$ (10)

i.e. the $j$-th output of the final layer, where $j$ again is the class label.
For simplicity, from now on, we refer to (9) with the term model gradient.

OGD uses projected gradient-type updates in order to update the model
parameters for a new task in directions that minimally increase the loss for
samples from previously seen tasks. Specifically, after training on task $k$
is finished, an update takes place that stores the relevant model gradients
for task $k$ in the model gradient pool.

Assume our model has been trained on task $T_{1}$ using SGD and denote as
$w^{\ast}_{1}$ the weights learned through the training process. The gradient
pool after the training on task $T_{1}$ is the set $\mathcal{S}_{1}$, defined
as

$\mathcal{S}_{1}=\\{\nabla f_{j}(x;w^{\ast}_{1})\mid j:y_{j}=1,(x,y)\in
T_{1}\\}.$ (11)

When training on the next task $T_{2}$, rather than training using standard
SGD, OGD first projects the gradient of the loss for each batch onto the
orthogonal complement of the subspace spanned by all elements of the gradient
pool $\mathcal{S}_{1}$. More specifically, the updates of the weights of OGD
for given batches of task $T_{2}$ have the following form,

$w\leftarrow w-\eta\hat{g}$ (12)

where $\eta$ is the stepsize and
$\hat{g}=g-\sum_{u\in\mathcal{S}_{1}}\mathrm{proj}_{u}(g)$ with $g$ denoting
the gradient $\nabla L_{t}(w)$ of the loss $L_{t}(w)$ corresponding to the
batch $T_{k_{t}}$. When training on $T_{2}$ ends with weights $w_{2}^{*}$, OGD
augments the gradient pool $\mathcal{S}_{1}$ with model gradients for $T_{2}$
and so on. After training on task $T_{k-1}$, the gradient pool is redefined as

$\\!\\!\\!\\!\\!\mathcal{S}_{1:k-1}\\!=\\!\\{\nabla f_{j}(x;w^{\ast}_{i})\mid
j:y_{j}=1,1\leq i<k,(x,y)\in T_{i}\\}.\\!\\!$ (13)

Hence, when training on task $T_{k}$ using the update in (12), we have that
$\forall_{u\in\mathcal{S}_{1:k-1}},~{}\hat{g}\perp u$.

The stored model gradients are interpreted as the directions of most change on
the previous task. Thus, by projecting orthogonal to these, the resulting
projected gradient steps move the weights towards the region of low error on
the new task (Task B), while minimally changing the performance on previous
tasks (Task A).

Algorithm 2 shows pseudo-code for OGD Farajtabar et al. (2020). The loss
gradient $\nabla L(w)$ is projected on an orthogonal basis $S$ and the
resulting component is subtracted from the loss gradient $\nabla L(w)$. The
weights $w$ are then updated using this newly obtained gradient and the step
size $\eta$. After the training on task $T_{k}$ is finished, samples $(x,y)$
from the task $T_{k}$ are used to calculated the model gradient $f_{j}(x;w)$,
which represents the gradient of the $j$th output of the network, where $j$
corresponds to the label $y$. This model gradient is then added to the basis
$S$, but it is first made orthogonal to all elements in $S$, to assure that
$S$ remains an orthogonal basis.

Algorithm 2 Orthogonal Gradient Descent

Input: Task sequence $T_{1},T_{2},T_{3},...$ learning rate $\eta$
Output: Model

1:Initialize $S\leftarrow\\{\\};w\leftarrow w_{0}$

2:for Task ID $k=1,2,3,...$ do

3: repeat

4: $L(w)\leftarrow\frac{1}{|T_{k_{t}}|}\sum_{(x,y)\in T_{k_{t}}}L_{(x,y)}(w)$

5: $g_{proj}\leftarrow\sum_{\mathbf{v}\in S}\frac{\nabla
L(w)^{\top}\mathbf{v}}{\mathbf{v}^{\top}\mathbf{v}}\mathbf{v}$

6: $\tilde{g}\leftarrow\nabla L(w)-g_{proj}$

7: $w\leftarrow w-\eta\tilde{g}$

8: until converge

9: for $(x,y)\in T_{k}$ and $j\in[1,c]$ s.t. $y_{j}=1$ do

10: $f_{proj}\leftarrow\sum_{\mathbf{v}\in S}\frac{\nabla
f_{j}(x;w)^{\top}\mathbf{v}}{\mathbf{v}^{\top}\mathbf{v}}\mathbf{v}$

11: $u\leftarrow\nabla f_{j}(x;w)-f_{proj}$

12: $S\leftarrow S\cup\\{u\\}$

13: end for

14:end for

##  Appendix C Clustering Mechanism

Algorithm 3 shows the clustering mechanism used by TA-A-GEM and TA-OGD, when
adding a sample $\mathbf{z}$ to the pool. First, a pool $C$ is initialized as
an empty set. The maximum number of clusters is denoted with $Q$. The maximum
cluster size is $P$. The fist $Q$ samples $\mathbf{z}$ initialize a cluster
$q$, be being added to it and by setting the cluster’s mean
$\boldsymbol{\mu}_{q}$ to $\mathbf{z}$. After all clusters are initialized,
the $\ell_{2}$ norm is used to determine the cluster $q^{\ast}$, that has its
mean closest to $\mathbf{z}$. $\mathbf{z}$ is then added to this cluster. The
size of $q^{\ast}$ is kept below the maximum cluster size $P$, by removing its
oldest member $\mathbf{z}^{q^{\ast}}_{\mathrm{oldest}}$. The mean of the
cluster $q^{\ast}$, namely $\boldsymbol{\mu}_{q^{\ast}}$, is recalculated by
taking the average of all its members.

Algorithm 3 Clustering Mechanism

1:$C\leftarrow\\{\\}$

2:$Q\leftarrow 100$

3:$P\leftarrow 3$

4:procedure Add($\mathbf{z}$)

5: if $|C|<Q$ then

6: $q\leftarrow\\{\mathbf{z}\\}$

7: $C\leftarrow C\cup q$

8: $\boldsymbol{\mu}_{q}\leftarrow\mathbf{z}$

9: else

10:
$q^{\ast}\leftarrow\mathrm{argmin}_{q\in\\{1,2,\dots,Q\\}}\|\mathbf{z}-\boldsymbol{\mu}_{q}\|^{2}_{2}$

11: $q^{\ast}\leftarrow q^{\ast}\cup\mathbf{z}$

12: if $|q^{\ast}|>P$ then

13: $q^{\ast}\leftarrow
q^{\ast}\setminus\mathbf{z}^{q^{\ast}}_{\mathrm{oldest}}$

14: end if

15:
$\boldsymbol{\mu}_{q^{\ast}}\leftarrow\frac{1}{|q^{\ast}|}\sum^{|q^{\ast}|}_{p=1}{\mathbf{z}^{q^{\ast}}_{p}}$

16: end if

17:end procedure

##  Appendix D Metric definition

The accuracy $A_{l}$ is defined as $A_{l}=\frac{M_{k_{l}}}{N_{k_{l}}}$, where
$M_{k_{l}}$ is the number of correctly classified samples in dataset
$T_{k_{l}}$ and $N_{k_{l}}$, is the total number of samples corresponding to
task $T_{k_{l}}$. Note that the index $k_{l}$ is the index of samples from
task $k$ at epoch $l$. The forgetting $F_{l}$ on the validation accuracy is
calculated from the validation accuracy $A_{l}$, at epoch $l$ using,

$F_{l}=\max(\\{A_{i}|i\leq l\\})-A_{l}$ (14)

##  Appendix E Learning Rate Scheduler

Algorithm 4 shows the pseudo-code of the learning rate scheduler as used by
the TA-OGD method. It lowers the learning rate, by multiplying it with a
Factor $<1$, when the learning process stagnates, i.e. when the loss value
reaches a plateau. It increases the learning rate, by resetting it to LR init,
when a large spike in the loss value overshoots the Best loss value by more
than the Reset Threshold value. The Patience value, together with the counters
N_Stagnant and N_Spike, make sure that the learning rate scheduler does not
react to irrelevant outlier loss values. The Min LR determines a minimum
learning rate, which the learning rate should never fall below. The values
mentioned in the pseudo-code are used in the experiments.

Algorithm 4 Learning Rate Scheduler

1:Factor : $0.9999$

2:Min LR : $10^{-5}$

3:LR init: $10^{-3}$

4:Patience : $5$

5:Best : $\infty$

6:N_Stagnant : $0$

7:N_Spike : $0$

8:Threshold: $10^{-4}$

9:Reset Threshold: $1$

10:procedure Step(Loss, LR)

11: if Loss $<$ Best $*$ ($1-$ Threshold) then

12: Best $\leftarrow$ Loss

13: N_Stagnant $\leftarrow$ $0$

14: else

15: N_Stagnant $\leftarrow$ N_Stagnant $+1$

16: end if

17: if Loss $>$ Best + Reset Threshold then

18: N_Spike $\leftarrow$ N_Spike $+1$

19: else

20: N_Spike $\leftarrow$ $0$

21: end if

22: if N_Spike $>$ Patience then

23: LR $\leftarrow$ LR init

24: Best $\leftarrow$ Loss

25: N_Spike $\leftarrow$ $0$

26: end if

27: if N_Stagnant $>$ Patience then

28: LR $\leftarrow$ max(LR $*$ Factor, Min LR)

29: N_Stagnant $\leftarrow$ $0$

30: end if

31: return LR

32:end procedure

##  Appendix F Experiments

Two main classes of experiments are described in the main paper, the “Disjoint
tasks experiments” and the “Continuous change experiments”. In this section,
figures and results are given, for which no space existed in the main paper.

###  F.1 Best practices

In order to test the effectiveness of a continual learning method, different
tasks need to be presented to it over time. These tasks can be synthesized
from standard datasets, but it is crucial that the produced tasks give a good
representation of a real world continual learning setting. In designing our
experiment, we abide by all the desiderata described in Farquhar and Gal
(2018), i.e., 1) cross-task resemblances between tasks, 2) a shared output
head, 3) no test-time assumed task labels, 4) no unconstrained retraining on
old tasks, and 5) the use of more than just two tasks. By conforming to these,
we ensure that the continual learning task is not trivially easy and thus a
method does not seem to be more effective than it would truly be in real-case
scenarios. Note that for the case of the experiments on permutation for task
generation, we “violate" the cross-task resemblances since such a requirement
is impossible to be enforced.

###  F.2 Task generation details

Separate tasks were created from existing datasets by means of permutation,
rotation and class splitting. The task permutation mechanism picks one
permutation that shuffles all pixels at random per task and then applies this
one permutation to all images in the dataset. The task rotation mechanism
rotates every image in one task with a given number of degrees. We chose to
increment the number of degrees by 20 for each new task. The class splitting
mechanism splits the dataset on disjoint sets of labels. As subsets we chose
the labels (0,1), (2,3), (4,5), (6,7) and (8,9).

To make the tasks more disjoint, for the task permutation and task rotation,
the data was first split into disjoint subsets (one for each task). After
this, the permutation or rotation was applied. For class splitting, the
different tasks are already completely disjoint, so no extra separation step
was required. Note that we do not use head swapping, meaning that we have one
fixed output layer for all tasks. For the class split scenario, only two
classes are given to the network during one task. Therefore we chose to give
the network just two output nodes, instead of ten. To accommodate this, we
changed the labels. All even labels were set to $0$ and all odd labels were
set to $1$.

###  F.3 Environment

Each experiment instance ran on a single NVIDIA Tesla T4 card with 15 Gb of
video memory. Up to fifteen NVIDIA Tesla T4 cards were available in parallel,
together with 32 physical processor cores and 252 GB of RAM.

###  F.4 Disjoint tasks experiments

Figures 5, 6, 7, 8 and 9 show the average validation accuracy, the validation
accuracy on the first task and the forgetting on the validation accuracy
during training for different task split methods, using the MNIST, Fashion
MNIST, NOT MNIST, CIFAR10 and SVHN datasets. Table 2 and 3 summarize the
results on the validation accuracy on the first task and the forgetting on the
validation accuracy on the first task, respectively. A summary for the average
validation accuracy is included in the main paper.

Specific settings for the task splitting can be seen in table 7. The settings
specific to TA-A-GEM and TA-OGD can be seen in table 8 and 9, respectively.
All results depicted are the average of five independent runs.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 5: Results of the “Disjoint tasks experiments” on the MNIST dataset.
(a) Average validation accuracy, (b) Validation accuracy of the first task,
(c) Forgetting on validation accuracy of the first task, averaged over all
tasks trained thus far, then averaged over five runs, depicted by a solid line
plot with $\pm$ one standard deviation as a shaded area. From left to right:
task separation by permutation, task separation by rotation and task
separation by class split.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 6: Results of the “Disjoint tasks experiments” on the Fashion MNIST
dataset. (a) Average validation accuracy, (b) Validation accuracy of the first
task, (c) Forgetting on validation accuracy of the first task, averaged over
all tasks trained thus far, then averaged over five runs, depicted by a solid
line plot with $\pm$ one standard deviation as a shaded area. From left to
right: task separation by permutation, task separation by rotation and task
separation by class split.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 7: Results of the “Disjoint tasks experiments” on the NOT MNIST
dataset. (a) Average validation accuracy, (b) Validation accuracy of the first
task, (c) Forgetting on validation accuracy of the first task, averaged over
all tasks trained thus far, then averaged over five runs, depicted by a solid
line plot with $\pm$ one standard deviation as a shaded area. From left to
right: task separation by permutation, task separation by rotation and task
separation by class split.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 8: Results of the “Disjoint tasks experiments” on the CIFAR10 dataset.
(a) Average validation accuracy, (b) Validation accuracy of the first task,
(c) Forgetting on validation accuracy of the first task, averaged over all
tasks trained thus far, then averaged over five runs, depicted by a solid line
plot with $\pm$ one standard deviation as a shaded area. From left to right:
task separation by permutation, task separation by rotation and task
separation by class split.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 9: Results of the “Disjoint tasks experiments” on the SVHN dataset. (a)
Average validation accuracy, (b) Validation accuracy of the first task, (c)
Forgetting on validation accuracy of the first task, averaged over all tasks
trained thus far, then averaged over five runs, depicted by a solid line plot
with $\pm$ one standard deviation as a shaded area. From left to right: task
separation by permutation, task separation by rotation and task separation by
class split.

| MNIST | Fashion MNIST | NOT MNIST | CIFAR10 | SVHN
---|---|---|---|---|---
| perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class
SGD | 0.691 | 0.419 | 0.643 | 0.682 | 0.254 | 0.657 | 0.825 | 0.408 | 0.707 | 0.364 | 0.288 | 0.692 | 0.527 | 0.231 | 0.622
SGD lr adapt | 0.693 | 0.429 | 0.669 | 0.686 | 0.255 | 0.684 | 0.827 | 0.408 | 0.761 | 0.365 | 0.288 | 0.699 | 0.520 | 0.231 | 0.635
BGD | 0.812 | 0.431 | 0.606 | 0.716 | 0.270 | 0.646 | 0.813 | 0.425 | 0.711 | 0.365 | 0.297 | 0.682 | 0.596 | 0.261 | 0.623
TA-OGD | 0.779 | 0.487 | 0.721 | 0.686 | 0.302 | 0.831 | 0.804 | 0.429 | 0.866 | 0.332 | 0.283 | 0.708 | 0.473 | 0.235 | 0.637
TA-A-GEM | 0.858 | 0.465 | 0.748 | 0.734 | 0.625 | 0.931 | 0.830 | 0.415 | 0.738 | 0.347 | 0.285 | 0.695 | 0.536 | 0.236 | 0.640
OGD | 0.737 | 0.461 | 0.645 | 0.688 | 0.306 | 0.699 | 0.766 | 0.426 | 0.817 | 0.293 | 0.285 | 0.703 | 0.414 | 0.229 | 0.634
A-GEM | 0.830 | 0.654 | 0.95 | 0.735 | 0.640 | 0.934 | 0.834 | 0.605 | 0.906 | 0.338 | 0.308 | 0.725 | 0.443 | 0.292 | 0.757

Table 2: Validation accuracy of the first task, averaged over all epochs, then
averaged over five runs, for the disjoint tasks experiments when using a MLP.
Per column, the best result for the task-agnostic methods are written in bold.
In case a task-agnostic method’s result is less optimal and not significantly
different from the best result, with a confidence of 99%, it is also written
in bold. The results for the task-aware methods OGD and A-GEM are given for
context. Since these algorithms benefit from knowing task identities and
changes, we just use them here as baselines for indicating the best
performance we can achieve.

| MNIST | Fashion MNIST | NOT MNIST | CIFAR10 | SVHN
---|---|---|---|---|---
| perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class
SGD | 0.213 | 0.485 | 0.356 | 0.099 | 0.527 | 0.329 | 0.056 | 0.473 | 0.270 | 0.060 | 0.136 | 0.209 | 0.068 | 0.364 | 0.337
SGD lr adapt | 0.209 | 0.473 | 0.330 | 0.091 | 0.521 | 0.302 | 0.053 | 0.471 | 0.215 | 0.056 | 0.134 | 0.201 | 0.064 | 0.353 | 0.322
BGD | 0.115 | 0.496 | 0.393 | 0.104 | 0.550 | 0.340 | 0.081 | 0.469 | 0.266 | 0.065 | 0.133 | 0.212 | 0.032 | 0.367 | 0.328
TA-OGD | 0.123 | 0.415 | 0.278 | 0.081 | 0.464 | 0.154 | 0.070 | 0.444 | 0.109 | 0.086 | 0.135 | 0.193 | 0.091 | 0.330 | 0.319
TA-A-GEM | 0.049 | 0.442 | 0.251 | 0.058 | 0.167 | 0.057 | 0.053 | 0.468 | 0.239 | 0.078 | 0.139 | 0.207 | 0.078 | 0.378 | 0.322
OGD | 0.165 | 0.441 | 0.354 | 0.089 | 0.470 | 0.286 | 0.113 | 0.453 | 0.160 | 0.128 | 0.137 | 0.198 | 0.170 | 0.354 | 0.323
A-GEM | 0.074 | 0.250 | 0.049 | 0.046 | 0.142 | 0.052 | 0.048 | 0.276 | 0.072 | 0.086 | 0.116 | 0.176 | 0.153 | 0.303 | 0.202

Table 3: Forgetting on validation accuracy of the first task, averaged over
five runs, then averaged over all epochs, for the disjoint tasks experiments
when using a MLP. Per column, the best result for the task-agnostic methods
are written in bold. In case a task-agnostic method’s result is less optimal
and not significantly different from the best result, with a confidence of
99%, it is also written in bold. The results for the task-aware methods OGD
and A-GEM are given for context. Since these algorithms benefit from knowing
task identities and changes, we just use them here as baselines for indicating
the best performance we can achieve.

###  F.5 Continuous change experiments

In the Continuous change experiments, tasks slowly change into the next task
halfway through the training process. For the first 10 epochs (0-9), the
network is trained on purely the current task. For epoch 10 to 19, for every
batch, slowly more and more samples of the next task are mixed in. So the
first batch of epoch 10 will contain 100% samples from the current tasks and
0% of the next task, while the last batch of epoch 19 will contain 0% samples
of the current task and 100% samples of the next task. These percentages
gradually change by linear interpolation. Table 4 shows the result of the
continuous change experiments. In these experiments, tasks gradually change
into the next task. This is accomplished by first loading two datasets $A$ and
$B$, where dataset $A$ represent the current task and dataset $B$ represents
the next task. For each epoch, $A$ is randomly shuffled and $B$ is randomly
shuffled. A fraction $q\in(0,1)$ determines how many samples of $A$ and how
many samples of $B$ should be mixed into one batch. A batch of size $z$ is
taken from both $A$ and $B$. From the batch of $A$, the first $\lfloor
q*z\rfloor$ samples are selected. From the batch of $B$, we select the last
$\lceil(1-q)*z\rceil$ samples. Since datasets $A$ and $B$ are shuffled every
epoch, this is equivalent to random sampling without replacement. The selected
samples are joined in a mixed batch. This mixed batch is then shuffled again,
so no distinction between the tasks can be made. This mixed batch is then used
for training.

For this experiment, training commences for each pair of datasets for 20
epochs. For the first 10 epochs (0-9), $q$ is set to $1.0$. This means that
for these epochs, only the first of the datasets is presented to the learner.
For the next 10 tasks (10-19), $q$ gradually changes for each batch from $1.0$
to $0.0$. This means that for the first batch of epoch 10, $q$ equals $1.0$
and for the last batch of epoch 19, $q$ equals $0.0$. Between these batches,
$q$ changes linearly.

| MNIST | Fashion MNIST | NOT MNIST | CIFAR10 | SVHN
---|---|---|---|---|---
| perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class
SGD | 0.841 | 0.647 | 0.816 | 0.738 | 0.471 | 0.814 | 0.849 | 0.588 | 0.866 | 0.378 | 0.348 | 0.728 | 0.597 | 0.396 | 0.755
SGD lr adapt | 0.859 | 0.654 | 0.819 | 0.748 | 0.487 | 0.821 | 0.854 | 0.592 | 0.893 | 0.380 | 0.349 | 0.731 | 0.592 | 0.394 | 0.772
BGD | 0.884 | 0.678 | 0.808 | 0.765 | 0.515 | 0.812 | 0.860 | 0.621 | 0.865 | 0.390 | 0.358 | 0.719 | 0.605 | 0.419 | 0.760
TA-OGD | 0.875 | 0.706 | 0.912 | 0.748 | 0.522 | 0.902 | 0.846 | 0.622 | 0.937 | 0.335 | 0.343 | 0.735 | 0.567 | 0.392 | 0.792
TA-A-GEM | 0.876 | 0.670 | 0.87 | 0.745 | 0.594 | 0.929 | 0.850 | 0.592 | 0.876 | 0.369 | 0.346 | 0.715 | 0.606 | 0.403 | 0.765

Table 4: Average validation accuracy, averaged over all tasks trained thus
far, then averaged over all epochs, then averaged over five runs, for the
continuous change experiments when using a MLP. Per column, the best result is
written in bold. In case a result is less optimal and not significantly
different from the best result, with a confidence of 99%, it is also written
in bold.

Figures 10, 11, 12, 13 and 14 show the average validation accuracy during
training for different task split methods, using the MNIST, Fashion MNIST, NOT
MNIST, CIFAR10 and SVHN datasets, for the “Continuous change” experiments.
Table 5 and 6 summarize the results on the validation accuracy on the first
task and the forgetting on the validation accuracy on the first task,
respectively. Again, all results are the average of five independent runs.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 10: Results of the “Continuous change experiments” on the MNIST
dataset. (a) Average validation accuracy, (b) Validation accuracy of the first
task, (c) Forgetting on validation accuracy of the first task, averaged over
all tasks trained thus far, then averaged over five runs, depicted by a solid
line plot with $\pm$ one standard deviation as a shaded area. From left to
right: task separation by permutation, task separation by rotation and task
separation by class split.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 11: Results of the “Continuous change experiments” on the Fashion MNIST
dataset. (a) Average validation accuracy, (b) Validation accuracy of the first
task, (c) Forgetting on validation accuracy of the first task, averaged over
all tasks trained thus far, then averaged over five runs, depicted by a solid
line plot with $\pm$ one standard deviation as a shaded area. From left to
right: task separation by permutation, task separation by rotation and task
separation by class split.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 12: Results of the “Continuous change experiments” on the NOT MNIST
dataset. (a) Average validation accuracy, (b) Validation accuracy of the first
task, (c) Forgetting on validation accuracy of the first task, averaged over
all tasks trained thus far, then averaged over five runs, depicted by a solid
line plot with $\pm$ one standard deviation as a shaded area. From left to
right: task separation by permutation, task separation by rotation and task
separation by class split.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 13: Results of the “Continuous change experiments” on the CIFAR10
dataset. (a) Average validation accuracy, (b) Validation accuracy of the first
task, (c) Forgetting on validation accuracy of the first task, averaged over
all tasks trained thus far, then averaged over five runs, depicted by a solid
line plot with $\pm$ one standard deviation as a shaded area. From left to
right: task separation by permutation, task separation by rotation and task
separation by class split.

(a) Average validation accuracy

(b) Validation accuracy of the first task

(c) Forgetting on validation accuracy of the first task

Figure 14: Results of the “Continuous change experiments” on the SVHN dataset.
(a) Average validation accuracy, (b) Validation accuracy of the first task,
(c) Forgetting on validation accuracy of the first task, averaged over all
tasks trained thus far, then averaged over five runs, depicted by a solid line
plot with $\pm$ one standard deviation as a shaded area. From left to right:
task separation by permutation, task separation by rotation and task
separation by class split.

| MNIST | Fashion MNIST | NOT MNIST | CIFAR10 | SVHN
---|---|---|---|---|---
| perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class
SGD | 0.702 | 0.414 | 0.657 | 0.659 | 0.267 | 0.665 | 0.810 | 0.395 | 0.694 | 0.359 | 0.289 | 0.707 | 0.490 | 0.230 | 0.626
SGD lr adapt | 0.785 | 0.420 | 0.672 | 0.693 | 0.278 | 0.684 | 0.825 | 0.404 | 0.769 | 0.361 | 0.289 | 0.717 | 0.479 | 0.228 | 0.658
BGD | 0.812 | 0.426 | 0.645 | 0.702 | 0.297 | 0.663 | 0.804 | 0.409 | 0.675 | 0.366 | 0.299 | 0.695 | 0.588 | 0.264 | 0.636
TA-OGD | 0.819 | 0.476 | 0.803 | 0.682 | 0.317 | 0.842 | 0.789 | 0.422 | 0.851 | 0.325 | 0.284 | 0.719 | 0.457 | 0.228 | 0.681
TA-A-GEM | 0.842 | 0.442 | 0.726 | 0.729 | 0.600 | 0.926 | 0.813 | 0.398 | 0.712 | 0.331 | 0.286 | 0.704 | 0.509 | 0.233 | 0.634

Table 5: Validation accuracy of the first task, averaged over all epochs, then
averaged over five runs, for the continuous change experiments when using a
MLP. Per column, the best result is written in bold. In case a result is less
optimal and not significantly different from the best result, with a
confidence of 99%, it is also written in bold.

| MNIST | Fashion MNIST | NOT MNIST | CIFAR10 | SVHN
---|---|---|---|---|---
| perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class | perm | rot | class
SGD | 0.192 | 0.478 | 0.342 | 0.104 | 0.492 | 0.319 | 0.066 | 0.475 | 0.281 | 0.046 | 0.127 | 0.185 | 0.057 | 0.312 | 0.325
SGD lr adapt | 0.107 | 0.470 | 0.327 | 0.067 | 0.478 | 0.300 | 0.049 | 0.464 | 0.207 | 0.041 | 0.124 | 0.174 | 0.053 | 0.299 | 0.290
BGD | 0.106 | 0.492 | 0.355 | 0.110 | 0.512 | 0.323 | 0.088 | 0.480 | 0.302 | 0.058 | 0.132 | 0.190 | 0.027 | 0.351 | 0.310
TA-OGD | 0.073 | 0.416 | 0.196 | 0.067 | 0.429 | 0.142 | 0.077 | 0.441 | 0.124 | 0.069 | 0.127 | 0.171 | 0.050 | 0.283 | 0.268
TA-A-GEM | 0.055 | 0.454 | 0.273 | 0.047 | 0.172 | 0.061 | 0.064 | 0.473 | 0.265 | 0.078 | 0.131 | 0.147 | 0.064 | 0.335 | 0.322

Table 6: Forgetting on validation accuracy of the first task, averaged over
five runs, then averaged over all epochs, for the continuous change
experiments when using a MLP. Per column, the best result is written in bold.
In case a result is less optimal and not significantly different from the best
result, with a confidence of 99%, it is also written in bold.

Task generation | Mechanism | Parameters
---|---|---
Permutation | Number of tasks | 10
Rotation | Rotation angles | [0, 20, 40, 60, 80, 100, 120, 140, 160, 180]
Class split | Subset labels | [[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]

Table 7: Summary of the settings used for the task split for the “Disjoint
tasks” experiment and the “Continuous change” experiment

Parameter name | Value | Explanation
---|---|---
Learning rate | $10^{-3}$ | The same as for TA-OGD
Batch size | 10 | The same batch size as TA-OGD is chosen.
Sampling rate | $1$ | The number of labeled data points sampled per batch, so in this case, one labeled data point is sampled every batch.
Number of pools | 10 or 2 | The number of separate pools; one for each class, so 10 classes for permutation and rotation and 2 for class split.
Number of clusters | 10 or 50 | Each pool contains 10 clusters in case there are 10 classes, or 50 clusters in case there are 2 classes.
Cluster size | 3 | The maximum number of sampled data points per cluster, if this number is exceeded, the oldest sample is removed.

Table 8: TA-A-GEM uses these parameters. Their values were hand picked to
maximize the average accuracy.

Parameter name | Value | Explanation
---|---|---
Learning rate | $10^{-3}$ | The same as in the OGD paper of Farajtabar et al. Farajtabar et al. (2020)
Batch size | 10 | TA-OGD needs small batch sizes. Larger batch sizes will not yield good results.
Sampling rate | $1$ | The number of gradients sampled per batch, so in this case, one gradient is sampled every batch.
Number of pools | 1 | TA-OGD uses one pool for all sampled model gradients.
Number of clusters | 99 | The number of distinct model gradient clusters.
Cluster size | 3 | The maximum number of model gradients per cluster, if this number is exceeded, the oldest model gradient is removed.

Table 9: TA-OGD uses these parameters. Their values were hand picked to
maximize the average accuracy.

##  Appendix G Sampling Rate

For simplicity, we simply always add one sample per batch and set a maximum
pool size. Once the pool is full, for every sample that is added, one is
removed. If we remove one sample at random however, samples that are relevant
to previous tasks quickly disappear from the pool. In this case, the pool
itself suffers from catastrophic forgetting. The easiest way to fix this is to
lower the sampling rate, i.e. the frequency at which samples are added to the
pool. In pre-experiments, we found out that setting the sampling rate to
$0.01$ gave optimal results in negating the forgetting. This effectively means
that one sample was added to the pool every $100$ batches. As it turned out,
this gave the algorithm enough time to take a decent amount of samples for a
task, while it did not take so many samples as to completely eliminate samples
from older tasks from the pool. We then realized however, that this “optimal”
sampling rate was completely dependent on the fact that every task was
presented to the model for $20$ epochs. It thus completely depends on the
knowledge that a task will change after $20$ epochs. In a truly task-agnostic
setting, this information is not known. We therefore consider this method that
solely relies on the “optimal” sampling rate to be a trivial case that is
useless in a task-agnostic setting.

##  Appendix H Clustering versus Random sampling

Figure 18 demonstrates that using clustering is a necessity. It shows that
without the use of clustering (TA-A-GEM random), TA-A-GEM can only counteract
the forgetting with an “optimal” sampling rate (sr) of $0.01$, that is
completely dependent on the frequency of task change; once every $20$ epochs
in this case. When clustering is introduced (TA-A-GEM), a wide variety of
sampling rates, $0.01$, $0.1$ and $1$ can effectively negate forgetting.

To further investigate the effect of clustering on the model gradient pool,
the content of the pool is plotted over time for TA-A-GEM, that uses
clustering and TA-A-GEM with random cluster assignment. In the latter, a newly
sampled data point is added to a randomly selected cluster instead of being
added to the cluster that has its mean closest to it (as is the case in the
standard clustering-based TA-A-GEM approach). We test the performance of the
two methods for three different values of the sampling rate i.e., 0.01, 0.1
and 1. Figures 15, 16 and 17 show the content of the gradient pool for TA-A-
GEM (left) and TA-A-GEM with random cluster assignment (right), for a sampling
rate of 0.01, 0.1 and 1. Looking at the cluster contents of TA-A-GEM, it can
be seen that the amount of task variety is highly affected by the sampling
rate. The sampling rate can therefore be tweaked to an optimal value, in this
case 0.01, given that a new task arrives every 20 epochs. This is a trivial
case that we consider not to be truly task-agnostic, since it is tweaked on 20
epochs. By using clustering, TA-A-GEM manages to keep a wide variety of task
information in the pool for all sampling rates. It therefore improves on the
trivial case, which arises by fine-tuning the sampling rate.

Figure 15: The horizontal lines depict the content of clusters when using
class split as a task segmentation method on Fashion MNIST with a sampling
rate of 0.01. Each task is associated with a unique color. The color of the
vertical lines represents the oldest task information that is present in the
cluster. The moment that a new task starts is indicated by a black vertical
line. The clusters with index 0 to 49 correspond to the pool of class label 0.
The clusters with index 50 to 99 correspond to the pool of class label 1.
Left: Clustering helps in keeping a greater variety of task information in the
gradient pool, but this effect is hardly noticable due to the optimal sampling
rate being used. Right: Using random cluster assignment can result in a good
amount of variety, but it is completely dependent on the optimal sampling rate
of 0.01. but it is still just as effective at counteracting forgetting, as
compared to clustering. This is only ideal when every task presents itself for
exactly 20 epochs each time. For this, the frequency of task switching need to
be known, thereby destroying the task-agnostic nature of the method.

Figure 16: The content of clusters using when using class split as a task
segmentation method on Fashion MNIST with a sampling rate of 0.1. Left: The
use of clustering assures a good variety of task information is kept in the
pool. Right: Information from previous tasks is quickly lost with this
sampling rate setting.

Figure 17: The content of clusters when using class split as a task
segmentation method on Fashion MNIST with a sampling rate of 1. Left: Even
with a high sampling rate, the clustering assures a good variety of task
information in the pool. Right: Information from previous tasks is immediately
lost after the moment of task change. Figure 18: TA-A-GEM without its
clustering mechanism (TA-A-GEM random) can only successfully negate forgetting
with an “optimal” sampling rate (sr) of 0.01, that can only be known if the
task change frequency is known, thus undoing its task-agnostic nature. On the
other hand, when clustering is introduced to TA-A-GEM (TA-A-GEM), a wide
variety of sampling rates (sr) (0.01, 0.1 and 1) can effectively reduce the
amount of forgetting.