# Toward Sustainable Continual Learning: Detection and Knowledge Repurposing
of Similar Tasks

Sijia Wang  Duke University  Yoojin Choi  SoC R&D Samsung Semiconductor Inc.
Junya Chen  Duke University  Mostafa El-Khamy  SoC R&D Samsung Semiconductor
Inc.  Ricardo Henao  Duke University

###### Abstract

Most existing works on continual learning (CL) focus on overcoming the
catastrophic forgetting (CF) problem, with dynamic models and replay methods
performing exceptionally well. However, since current works tend to assume
exclusivity or dissimilarity among learning tasks, these methods require
constantly accumulating task-specific knowledge in memory for each task. This
results in the eventual prohibitive expansion of the knowledge repository if
we consider learning from a long sequence of tasks. In this work, we introduce
a paradigm where the continual learner gets a sequence of mixed similar and
dissimilar tasks. We propose a new continual learning framework that uses a
task similarity detection function that does not require additional learning,
with which we analyze whether there is a specific task in the past that is
similar to the current task. We can then reuse previous task knowledge to slow
down parameter expansion, ensuring that the CL system expands the knowledge
repository sublinearly to the number of learned tasks. Our experiments show
that the proposed framework performs competitively on widely used computer
vision benchmarks such as CIFAR10, CIFAR100, and EMNIST.

##  1 Introduction

Human intelligence is distinguished by the ability to learn new tasks over
time while remembering how to perform previously experienced tasks. Continual
learning (CL), an advanced machine learning paradigm requiring intelligent
agents to continuously learn new knowledge while trying not to forget past
knowledge, has a pivotal role in machines imitating human-level intelligence
[1]. The main problem in continual learning is catastrophic forgetting (CF) of
previous knowledge when new tasks, observed over time, are incorporated to the
model via training or (parameter) expansion.

In general, continual learning methods can be categorized into three major
groups based on: ($i$) regularization, ($ii$) replay, and ($iii$) dynamic
(expansion) models. Approaches based on regularization [2, 3, 4] alleviate
forgetting by constraining the updates of parameters that are important for
previous tasks, by adding a regularization term to the loss function
penalizing changes between parameters for current and previous tasks. While
this type of approach excels at keeping a low memory footprint, the ability to
remember previous knowledge eventually declines, especially, in scenarios with
long sequences of tasks [5]. Methods based on replay keep a memory bank to
store either a subset of exemplars to represent each class and task [6, 7, 8],
or generative models for pseudo-replay [9, 10]. Approaches based on dynamic
models [11, 12, 13, 14, 15, 16] allow the architecture (or components of it)
to expand over time. Specifically, [15, 16] leverage modular networks and
study a practical continual learning scenario with 100-task long sequences.
Nevertheless, these parameter expansion approaches face the challenges of
keeping model growth under control (ideally sublinear). This problem
associated with parameter expansion in CL systems is critical and requires
further attention.

This paper discusses CL under a task continual learning (TCL) setting, _i.e._
, that in which data arrives sequentially in groups of _tasks_. For works
under this scenario [17, 18], the assumption is usually that once a new task
is presented, all of its data becomes readily available for batch (offline)
training. In this setting, a _task_ is defined as an individual training phase
with a new collection of data that belongs to a new (never seen) group of
classes, or in general, a new domain. Further, TCL also (implicitly) requires
a task identifier during training. However, in practice, when the model has
seen enough tasks, a newly arriving batch of data becomes increasingly likely
to belong to the same group of classes or domain of a previously seen task.
Importantly, most existing works on TCL fail to acknowledge this possibility.
Moreover and in general, the task definition or identifier may not be
available during training, e.g., the model does not have access to the task
description due to (user) privacy concerns. In such case mostly concerning
dynamic models, the system has to treat every task as new, thus constantly
learning new sets of parameters regardless of task similarity or overlap. This
clearly constitutes a suboptimal use of resources (predominantly memory),
especially as the number of tasks experienced by the CL system grows.

This study investigates the aforementioned scenario and makes it an endeavor
to create a memory-efficient CL system which though focused on image
classification tasks, is general and in principle can be readily used toward
other applications or data modality settings. We provide a solution for
dynamic models to identify similar tasks when no task identifier is provided
during the training phase. To the best of our knowledge, the only work that
also discusses the learning of a continual learning system with mixed similar
and dissimilar tasks is [19], which proposes a task similarity function to
identify previously seen similar tasks, which requires training a reference
model every time a new task becomes available. Alternatively, in this work, we
identify similar tasks _without_ the need for training a new model, by
leveraging a task similarity metric, which in practice results in high task
similarity identification accuracy. We also discuss memory usage under
challenging scenarios where longer, more realistic, sequences of more than 20
tasks are used.

To summarize, our contributions are listed below:

  * •

We propose a new framework for an under-explored and yet practical TCL setting
in which we seek to learn a sequence of mixed similar and dissimilar tasks,
while preventing (catastrophic) forgetting and repurposing task-specific
parameters from a previously seen similar task, thus slowing down parameter
expansion.

  * •

The proposed TCL framework is characterized by a task similarity detection
module that determines, without additional learning, whether the CL system can
reuse the task-specific parameters of the model for a previous task or needs
to instantiate new ones.

  * •

Our task similarity detection module shows remarkable performance on widely
used computer vision benchmarks, such as CIFAR10 [20], CIFAR100 [20], EMNIST
[21], from which we create sequences of 10 to 100 tasks.

##  2 Related Work

TCL in Practical Scenarios Task continual learning (TCL) being an intuitive
imitation of human learning process constitutes one of the most studied
scenarios in CL. Though TCL systems have achieved impressive performance [22,
23], previous works have mainly focused on circumventing the problems
associated with CF. Historically, task sequences have been restricted to no
more than 10 tasks and strong assumptions have been imposed so that all the
tasks in the sequence are unique and classes among tasks are disjoint [10,
24]. The authors of [25] rightly argue that currently discussed CL settings
are oversimplified, and more general and practical CL forms should be
discussed to advance the field. It is not until recently that solutions have
been proposed for longer sequences and more practical CL scenarios.
Particularly, [19] proposed CAT, which learns from a sequence of mixed similar
and dissimilar tasks, thus enabling knowledge transfer between future and past
tasks detected to be similar. To characterize the tasks, a set of task-
specific masks, i.e., binary matrices indicating which parameters are
important for a given task [22], are trained along other model parameters.
Specifically, these masks are activated and parameters associated to them are
finetuned once the current task is identified as “similar” by a task
similarity function, or otherwise held fixed by the masking parameters to
protect them from changing, hence preventing CF. Alternatively, [15]
introduces a modular network, which is composed of neural modules that are
potentially shared with related tasks. Each task is optimized by selecting a
set of modules that are either freshly trained on the new task or borrowed
from similar past tasks. These similar tasks are detected by a task-driven
prior. [15, 16] evaluate their approach with a real-world CL benchmark CtrL
that contains 100 tasks with class overlap between tasks. The data-driven
prior used in [15] for recognizing similar tasks is a very simple approach,
which leaves space for improved substitutes yielding slower parameter growth.
In summary, these works serve as a reminder that identifying similar tasks in
TCL settings is in general a hard problem that deserves more attention.

Computational Efficiency in CL The first and foremost step for tackling
computational inefficiencies is to identify the framework components that are
most computationally consuming. For instance, to avoid storing real images for
replay, [26] proposes to keep lower-dimensional feature embeddings of images
instead. Another way to tackle the issue is through limiting trainable
parameters in the model architecture itself by partitioning a convolutional
layer into a backbone and task-specific adapting modules, like in [27, 28].
Similarly, Filter Atom Swapping in [29] decompose filters in neural network
layers and ensemble them with filters of tasks in the past. We can also make
use of task relevancy or similarity. For instance, approaches with model
structures similar to Progressive Network [14], where tasks are optimized in
search of neural modules to use. Further, [15] is proposed to optimize search
space through task similarity.

##  3 Background

###  3.1 Problem Setting

We consider the TCL scenario for image classification tasks, where we seek to
incrementally learn a sequence of tasks
$\mathcal{T}_{1},\mathcal{T}_{2},\ldots,\mathcal{T}_{t-1}$ and denote the
collection of tasks currently learned as ${\cal T}=\\{{\cal
T}_{1},\ldots,{\cal T}_{t-1}\\}$. The underlying assumption is that, as the
number of tasks in $\mathcal{T}$ grows, the current task $\mathcal{T}_{t}$
will eventually have a corresponding similar task
$\mathcal{T}^{sim}_{t}\in\mathcal{T}$. Let the set of all dissimilar tasks be
$\mathcal{T}^{dis}_{t}={\cal T}\backslash\mathcal{T}_{t}^{sim}$. We define
similar and dissimilar tasks as follows.

Similar and Dissimilar tasks: Consider two tasks $A$ and $B$, which are
represented by datasets
$D_{A}=\\{\mathbf{x}_{i}^{A},y^{A}_{i}\\}^{n_{A}}_{i=1}$ and
$D_{B}=\\{\mathbf{x}_{i}^{B},y^{B}_{i}\\}^{n_{B}}_{i=1}$, where
$y_{i}^{A}\in\\{\mathcal{Y}^{A}\\}$ and $y_{i}^{B}\in\\{\mathbf{Y}^{B}\\}$. If
predictors (_e.g._ , images) in
$\\{\mathbf{x}_{i}^{A}\\}^{n_{A}}_{i=1}\sim\mathcal{P}$ and
$\\{\mathbf{x}_{i}^{B}\\}^{n_{B}}_{i=1}\sim\mathcal{P}$, and labels
$\\{\mathcal{Y}^{A}\\}=\\{\mathcal{Y}^{B}\\}$, indicating that data from $A$
and $B$ belong to the same group of classes, and their predictors are drawn
from the same distribution $\mathcal{P}$, then we say A and B are _similar_
tasks, otherwise, $A$ and $B$ are deemed _dissimilar_. Notably, when both
tasks share the same distribution $\mathcal{P}$, but have different label
spaces, they are considered dissimilar.

The main objective in this work is to identify $\mathcal{T}^{sim}_{t}$ among
$\mathcal{T}$ without training a new model (or learning parameters) for
$\mathcal{T}_{t}$, by leveraging a task similarity identification function,
that will enable the system to reuse the parameters of $\mathcal{T}^{sim}_{t}$
when identification of a previously seen task is successful. Alternatively,
the system will instantiate new parameters for the dissimilar task. As a
result, the system will attempt to learn parameters for the set of unique
tasks, which in a long sequence of tasks is assumed to be smaller than the
sequence length. In practice, in order to handle memory efficiently, we do not
have to instantiate completely different sets of parameters for every unique
task, but rather we define _global_ and _task-specific_ parameters as a means
to further control model growth. For this purpose, we leverage the efficient
feature transformations for convolutional models described below.

###  3.2 Task-Specific Adaptation via Feature Transformation Techniques

The efficient feature transformation (EFT) framework in [30] proposed that
instead of finetuning all the parameters in a well-trained (pretrained) model,
one can instead partition the network into a (global) _backbone_ model and
_task-specific_ feature transformations. Note that similar ideas have also
been explored in [28, 29, 31, 32, 33]. Given a trained _backbone_
convolutional neural network, we can transform the convolutional feature maps
$F$ for each layer into task-specific feature maps $W$ by implementing small
convolutional transformations.

In our setting, only $W$ is learned for task $\mathcal{T}_{t}$ as a means to
reduce the parameter count, and thus the memory footprint, required for new
tasks. Specifically, the feature transformation involves two types of
convolutional kernels, namely, $\omega^{s}\in\mathbb{R}^{3\times 3\times a}$
for capturing spatial features within groups of channels and
$\omega^{d}\in\mathbb{R}^{1\times 1\times b}$ for capturing features across
channels at every location in $F$, where $a$ and $b$ are hyperparameters
controlling the size of each feature map groups.

The transformed feature maps $W=W^{s}+\gamma W^{d}$ are obtained from
$W^{s}=[W^{s}_{0:a-1}|\cdots|W^{s}_{(K-a):K}]$ and
$W^{d}=[W^{s}_{0:b-1}|\cdots|W^{s}_{(K-b):K}]$ via

$\displaystyle W^{s}_{ai:(ai+a-1)}$
$\displaystyle=[\omega^{s}_{i,1}*F_{ai:(ai+a-1)}|\cdots|w^{s}_{i,a}*F_{ai:(ai+a-1)}],$
$\displaystyle i$ $\displaystyle\in\\{0,\cdots,\frac{K}{a}-1\\}$ (1)
$\displaystyle W^{d}_{bi:(bi+b-1)}$
$\displaystyle=[\omega^{d}_{i,1}*F_{bi:(bi+b-1)}|\cdots|w^{d}_{i,b}*F_{bi:(bi+b-1)}],$
$\displaystyle i$ $\displaystyle\in\\{0,\cdots,\frac{K}{b}-1\\}$ (2)

where the feature maps $F\in\mathbb{R}^{M\times N\times K}$ and $W\in\mathbb{R}^{M\times N\times K}$ have spatial dimensions $M$ and $N$, $K$ is the number of feature maps, | is the concatenation operation, $K/a$ and $K/b$ are the number of groups into which $F$ is split for each feature map, $\gamma\in\\{0,1\\}$ indicates whether the point-wise convolutions $\omega^{d}$ are employed, and $W^{s}_{ai:(ai+a-1)}\in\mathbb{R}^{M\times N\times a}$, and $W^{d}_{bi:(bi+b-1)}\in\mathbb{R}^{M\times N\times b}$ are slices of the transformed feature map. In practice, we set $a\ll K$ and $b\ll K$ so that the amount of trainable parameters per task is substantially reduced. For instance, using a ResNet18 backbone, $a=8$, and $b=16$ results in 449k parameters per new tasks, which is $3.9\%$ the size of the backbone. As empirically demonstrated in [30], EFT preserves the remarkable representation learning power of ResNet models while significantly reducing the number of trainable parameters per task.

###  3.3 Task Continual Learning: Mixture Model Perspective

One of the key components when trying to identify similar tasks is to decide
whether $\\{\mathbf{x}_{i}^{A}\\}^{n_{A}}_{i=1}$ and
$\\{\mathbf{x}_{i}^{B}\\}^{n_{B}}_{i=1}$ originate from the same distribution
$\mathcal{P}$. However, though conceptually simple, it is extremely
challenging in practice, particularly when predictors are complex instances
such as images. Intuitively, for a sequence of tasks ${\cal T}=\\{{\cal
T}_{1},\ldots,{\cal T}_{t-1}\\}$, with corresponding data
$D_{1},\ldots,D_{t-1}$, $D_{t}=\\{\mathbf{x}_{i},y_{i}\\}^{n_{t}}_{i=1}$
consisting of $n_{t}$ instances, where $\mathbf{x}_{i}$ is an image and
$y_{i}$ is its corresponding label, we can think of data instances
$\mathbf{x}$ from the collection of all unique tasks as a mixture model
defined as

$\displaystyle
p(\mathbf{x})=\pi_{*}p(\mathbf{x}|\phi_{*})+\sum_{j=1}^{t-1}\pi_{j}p(\mathbf{x}|\phi_{j}),$
(3)

from which we can see that $\pi_{j}$ is the probability that $\mathbf{x}$
belongs to task ${\cal T}_{j}$ and $p(\mathbf{x}|\phi_{j})$ is the likelihood
of $\mathbf{x}$ under the distribution for task ${\cal T}_{j}$ parameterized
by $\phi_{j}$. Further, $\pi_{*}$ and $\phi_{*}$ denote the hypothetical
probability and parameters for a new _unseen_ task $*$, _i.e._ , distinct from
$\\{p(\mathbf{x}|\phi_{j})\\}_{j=1}^{t-1}$. The formulation in (3) which is
reminiscent of a Dirichlet Process Mixture Model (DPMM) [34, 35], can in
principle be used to estimate _a posteriori_ that $p(D_{t}\in{\cal T}_{*})$ by
evaluating (3), which assumes that parameters
$\\{\pi_{*},\pi_{1},\ldots,\pi_{t-1}\\}$ and
$\\{\phi_{*},\phi_{1}\ldots,\phi_{t-1}\\}$ are readily available.
Unfortunately, though for the collection of existing tasks ${\cal T}$ we can
effectively estimate $\pi_{j}$ and $p(\mathbf{x}|\phi_{j})$, using generative
models (_e.g._ , variational autoencoers), the parameters for a new task
$\phi_{*}$ and $p(\mathbf{x}|\phi_{*})$ are much more difficult to estimate
because $i$) if we naively build a generative model for the new dataset
$D_{t}$ to obtain $\phi_{t}$ and then evaluate (3), it is almost guaranteed
that $D_{t}$ corresponding to ${\cal T}_{t}$ will be more likely under
$p(\mathbf{x}|\phi_{*}=\phi_{t})$ than any other existing task distribution
$\\{p(\mathbf{x}|\phi_{j})\\}_{j=1}^{t-1}$, and alternatively, $ii$) if we set
$p(\mathbf{x}|\phi_{*})$ to some prior distribution, _e.g._ a pretrained
generative model, it will be most definitely never selected, especially in
scenarios with complex predictors such as images. In fact, in early stages of
development we empirically verified this being the case using both pretrained
generative models admitting (marginal) likelihood estimation [36, 37] and
anomaly detection models based on density estimators [38, 39].

In Section 4, we will show how to leverage (3) to identify new tasks in the
context of TCL without using data from ${\cal T}_{t}$ for learning nor
specifying a prior distribution for $p(\mathbf{x}|\phi_{*})$.

###  3.4 Estimating the Association between Predictors and Labels

The mixture model perspective for TCL in (3) offers a compelling way to
compare the distribution of predictors for different tasks, however, it does
not provide any insights about the strength of the association between
predictors and labels for dataset $D_{t}$ corresponding to task ${\cal
T}_{t}$. Further, [40] showed that overparameterized neural network
classifiers can attain zero training error, regardless of the strength of
association between predictors and labels, and in an extreme case, even for
randomly labeled data. However, it is clear that a model trained with random
labels will not generalize.

So motivated, [41] first studied the properties of suitably labeled data that
control generalization ability, and proposed a generalization bound on the
test error (empirical risk) for arbitrary overparameterized two-layer neural
network classifiers with rectified linear units (ReLU). Importantly, unlike
previous works on generalization bounds for neural networks [42, 43, 44],
their bound can be effectively calculated _without_ training the network or
making assumptions about its size (number of hidden units). Their _complexity
measure_ , which is shown in (4), is set to directly quantify the strength of
the association between data and labels without learning. More precisely, for
dataset $D=\\{\mathbf{x}_{i},y_{i}\\}_{i=1}^{n}$ of size $n$, the
generalization bound in (4) is an upper bound on the (test) error conditioned
on $D$,

$\displaystyle
S(D)=\sqrt{\frac{2\mathbf{y}^{\mathsf{T}}\mathbf{H}^{-1}\mathbf{y}}{n}},$ (4)

where $\mathbf{y}=(y_{1},\cdots,y_{n})^{\mathsf{T}}$ and matrix
$\mathbf{H}\in\mathbb{R}^{n\times n}$, which can be seen as a Gram matrix for
a ReLU activation function is defined as

$\displaystyle H_{ik}$
$\displaystyle=\mathbb{E}_{\mathbf{w}\sim\mathcal{N}(\mathbf{0},\mathbf{I})}\big{[}\mathbf{x}_{i}^{\top}\mathbf{x}_{k}\mathbb{I}\\{\mathbf{w}^{\top}\mathbf{x}_{i}\geq
0,\mathbf{w}^{\top}\mathbf{x}_{k}\geq 0\\}\big{]}$ (5)
$\displaystyle=\frac{\mathbf{x}^{\top}_{i}\mathbf{x}_{k}(\pi-\arccos{(\mathbf{x}^{\top}_{i}\mathbf{x}_{k})})}{2\pi},\
\ \forall i,k\in 1,\ldots,n,$

where $H_{ik}$ is the $(i,k)$-th entry of $\mathbf{H}$,
$\mathcal{N}(0,\mathbf{I})$ denotes the standard Gaussian distribution,
$\mathbf{w}$ is a weight vector in the first layer of the two-layer neural
network, and $\mathbb{I}\\{\cdot\\}$ the an indicator function. Empirically,
[41] showed that the complexity measure in (4) can distinguish between strong
and weak associations between predictors and labels. Effectively, weak
associations tend to be consistent with _randomly labeled_ data, thus unlikely
to generalize.

In Section 4 we will leverage (4) as a metric for similar task detection,
which we will recast a measure to quantify the association between the labels
for a given task and the features from encoders learned from previously seen
tasks.

Figure 1: SDR Framework: We start with a pretrained feature extractor backbone
$E_{0}(\cdot)$ and models for three dissimilar tasks kept in a knowledge
repository, so the following tasks have some tasks to compare to at first.
Autoencoding pairs $G(\cdot)$ and $E(\cdot)$ are used for building the
distributional consistency estimator and analyzing predictor-label association
in the similar task identification module. The similar task identification
module uses (6) and (8) in the middle block with training dataset $D_{t}$ for
each new task $\mathcal{T}_{t}$ in the CL sequences. If there is a similar
task in the knowledge repository, then simply map the new task to the
identified previous task, reuse the feature extractor, $E(\cdot)$, and only
learn a new classification head $f_{t}(\cdot)$ for the current task;
alternatively, we learn a new encoder, decoder and classification head. At the
end of the learning for each task, the knowledge repository is updated with
the newly trained parameters. $C$ denotes the number of unique tasks in the
repository, which is smaller than the total number of tasks seen so far.

##  4 Detection and Repurposing of Similar Tasks

We now propose the Similar task Detection and Repurposing (SDR) framework for
task continual learning, which though tailored to image classification tasks,
can be in principle reused for other modalities, _e.g,_ , text and structured
data. Specifically, the framework consists of the following two components:
($i$) task-specific encoders $\\{E_{j}(\cdot)\\}^{t-1}_{j=1}$ for
$\mathcal{T}=\\{{\cal T}_{j}\\}_{j=1}^{t-1}$; and ($ii$) a mechanism for
similar task identification structured as two separate but complementary
components, namely, a measure of distributional similarity between the current
task ${\cal T}_{t}$ and each of all previous tasks ${\cal T}$, and a
predictor-to-label association similarity, each of which leverage the mixture
model perspective and the complexity measure introduced in Sections 3.3 and
3.4, respectively. For the former we use task-specific generative models
$\\{G_{j}(\cdot)\\}^{t-1}_{j=1}$ specified as variational autoencoders that
admit (marginal) likelihood estimation via the evidence lower bound (ELBO)
[45, 46], and for the latter we use the measure $S(\cdot)$ in (4). The
complete framework described below is briefly illustrated in 1 and presented
as Algorithm 1 in the Supplementary Material (SM). In a nutshell, similar task
identification results in one of two outcomes, namely, a previously seen task
is identified as similar to ${\cal T}_{t}$, in which case, we will use their
corresponding encoder _as is_ , but will learn a task-specific classification
head. Alternatively, if ${\cal T}_{t}$ is deemed as a new unseen task, we will
learn both a task-specific encoder, decoder (generator), and classification
head. The classification head for ${\cal T}_{t}$ is defined as
$p(y|\mathbf{x})=f_{t}(E_{t}(\mathbf{x}))$ and specified as a fully connected
network.

###  4.1 Task-specific Encoder

Following the specifications for the efficient feature transformations in
[30], we first specify a pretrained backbone encoder $E_{0}(\cdot)$, then for
each new unseen task, we adapt $E_{0}(\cdot)$ into ${E}_{t}(\cdot)$ using an
EFT module that is learned together (end-to-end) with the task-specific
classification head $p(y|\mathbf{x})=f_{t}(E_{t}(\mathbf{x}))$ in a supervised
fashion using $D_{t}$, while keeping $E_{0}(\cdot)$ fixed.

###  4.2 Similar Task Identification

As previously briefly described, the procedure to identify similar tasks for a
new task ${\cal T}_{t}$ among existing ${\cal T}=\\{{\cal
T}_{j}\\}_{j=1}^{t-1}$, amounts to estimate the distributional similarity of
the predictors and the strength of the association between labels and
predictors.

Provided a sequence of tasks ${\cal T}$ with corresponding encoders and
generators $\\{E_{j}(\cdot)\\}^{t-1}_{j=1}$ and
$\\{G_{j}(\cdot)\\}^{t-1}_{j=1}$, respectively, we can use (3) to estimate the
likelihood that predictors in $D_{t}$ are consistent with the generator for a
previously seen task. Specifically,

$\displaystyle p({\cal P}_{t}={\cal P}_{j})=\mathbb{E}_{\mathbf{x}_{i}\in
D_{t}}[p(u_{i}=j|\mathbf{x}_{i})]=\frac{1}{n}\sum_{i=1}^{n}\frac{\pi_{j}p(\mathbf{x}_{i}|\phi_{j})}{\sum_{j^{{}^{\prime}}}\pi_{j^{{}^{\prime}}}p(\mathbf{x}_{i}|\phi_{j^{{}^{\prime}}})},$
(6)

where in a slight abuse of notation we let $p({\cal P}_{t}={\cal P}_{j})$
indicate the probability that ${\cal P}_{t}$ and ${\cal P}_{j}$ have the same
predictor distributions. In practice, we use the expected lower bound (ELBO)
of a variational autoencoder pair $E_{m}(\cdot)$ and $G_{m}(\cdot)$ to
approximate the (marginal) likelihood $p(\mathbf{x}_{i}|\phi_{j})$
parameterized by $\phi_{j}$, which encapsulates the parameters of both
$E_{j}(\cdot)$ and $G_{j}(\cdot)$, _i.e._ , encoder and generator,
respectively. However, $p({\cal P}_{t}={\cal P}_{j})$ though useful for
previously seen tasks ${\cal T}$, does not help to identify new unseen tasks
that will be consistent with a hypothetical distribution ${\cal P}_{*}$
specified as $p(\mathbf{x}_{i}|\phi_{*})$ and parameterized by $\phi_{*}$.
Intuitively, given a new unseen task ${\cal T}_{t}$, $p({\cal P}_{t}={\cal
P}_{j})$ is likely to be a drawn from a uniform distribution in $t-1$
dimensions, so when $p({\cal P}_{t}={\cal P}_{j})\approx 1/(t-1)$, for
$j=1,\ldots,t-1$, the predictors for the new task are likely from a new unseen
task. Alternatively, when $p({\cal P}_{t}={\cal P}_{j})\to 1$, for some task
${\cal T}_{j}$, the predictors in $D_{t}$ are likely to be consistent in
distribution to ${\cal P}_{j}$, _i.e._ , to predictors from dataset $D_{j}$.

Importantly, the distributional consistency estimator in (6) only estimates
the probability that predictors in $D_{t}$ are consistent in distribution to
that of $D_{j}$. However, we still need to estimate the likelihood that
encoder $E_{j}(\cdot)$ built from $D_{j}$ is strongly associated with the
labels of interest in $D_{t}$ without building a classification model for
$D_{t}$ using $E_{j}(\cdot)$. For this purpose, we leverage the complexity
measure in (4), which we write below in terms of the task-specific encoders.

Following the construction of $\mathbf{H}$ as defined in (5), we start by
redefining it in terms of the task-specific encoders. Specifically, given
encoders $\\{E_{j}(\cdot)\\}^{t-1}_{j=1}$ and dataset
$D_{t}=\\{\mathbf{x}_{i}^{t},y^{t}_{i}\\}^{n_{t}}_{i=1}$ for task ${\cal
T}_{t}$, we extract features for the predictors in $D_{t}$ using all available
encoders, _i.e._ ,
$\\{\\{E_{j}(\mathbf{x}_{i}^{t})\\}^{n_{t}}_{i=1}\\}^{t-1}_{j=1}$. For
convenience, let $\mathbf{e}^{j,t}_{i}=E_{j}(\mathbf{x}^{t}_{i})$. We can
rewrite the Gram matrix $\mathbf{H}^{j,t}$ corresponding to encoder $j$ and
dataset $D_{t}$ in (5) as

$\displaystyle\mathbf{H}^{j,t}_{ik}$
$\displaystyle=\frac{(\mathbf{e}^{j,t}_{i})^{\top}\mathbf{e}^{j,t}_{k}(\pi-\arccos((\mathbf{e}^{j,t}_{i})^{\top}\mathbf{e}^{j,t}_{k}))}{2\pi},$
(7)

which is a multi-class extension of the complexity measure for binary
classification tasks introduced in [41]. Note that the Gram matrix
$\mathbf{H}^{j,t}\in\mathbb{R}^{n_{t}\times n_{t}}$ has the same formulation
as for when $\mathbf{y}^{t}=(y^{t}_{1},\cdots,y^{t}_{n})\in\\{0,1\\}^{c\times
n}$, where $c$ is the number of classes. Let
$\mathbf{A}^{j,t}=\mathbf{y}^{\mathsf{T}}(\mathbf{H}^{j,t})^{-1}\mathbf{y}\in\mathbb{R}^{c\times
c}$, the similarity metric $S$ between tasks $\mathcal{T}_{t}$ and
$\mathcal{T}_{j}$ via encoder $E_{j}(\cdot)$ is defined as:

$S(j,t)=\sqrt{\frac{2||{\mathbf{A}^{j,t}}^{\top}\mathbf{A}^{j,t}||^{2}_{2}}{n_{t}}},$
(8)

where $||\cdot||^{2}_{2}$ is the Frobenius Norm, $n_{t}$ is the size of
$D_{t}$. The proof for (7) and (8) can be found in the SM A.3.

In practice, we obtain feature sets $\\{e^{j,t}_{i}\\}^{n_{t}}_{i=1}$ with all
available encoders $\\{E_{j}(\cdot)\\}^{t-1}_{j=1}$ for task collection ${\cal
T}$, and calculate $S(j,t)$ for a new task $D_{t}$. Then we select
$\mathcal{T}_{a}$ with $a={\rm argmin}_{j}\ \\{S(j,t)\\}^{t-1}_{j=1}$.
Further, for each data point $i$ in $D^{t}$, we obtain
$p(u_{i}=j|\mathbf{x}_{i})$, for $j=1,\ldots,t-1$ using the ELBO to evaluate
$p({\bf x}_{i}|\phi_{j})$. Then we can estimate the probability of consistency
between $\mathcal{T}_{t}$ and $\mathcal{T}_{j}$ via (6) and select
$\mathcal{T}_{b}$ with $b={\rm argmax}_{j}\
\\{p(\mathcal{P}_{t}=\mathcal{P}_{j})\\}^{t-1}_{j=1}$. Finally, if
$\mathcal{T}_{a}=\mathcal{T}_{b}$ then $\mathcal{T}_{a}$ is a similar task,
otherwise, $\mathcal{T}_{t}$ is a dissimilar task. The SDR framework is
illustrated in Figure 1, from which we see that for the collection of
currently learned tasks, we keep a _knowledge repository_ containing (for the
tasks seen so far ${\cal T}$) $E_{0}(\cdot)$,
$\\{E_{j}(\cdot)\\}_{j=1}^{t-1}$, $\\{G_{j}(\cdot)\\}_{j=1}^{t-1}$,
$\\{f_{t}(\cdot)\\}_{j=1}^{t-1}$, _i.e._ , the backbone encoder, task-specific
encoder-decoder pairs, and the classification heads, respectively. For a more
detailed view of the SDR framework see Algorithm 1 in the SM.

##  5 Experiments

We test our method on several image benchmark datasets, including EMNIST,
CIFAR10, CIFAR100. Next we describe how to construct dissimilar and similar
tasks.

For CIFAR100, we split all the classes into 20 tasks $\\{T_{k}\\}^{20}_{k=1}$
with 5 classes for each task; then we divide data for each class evenly into
two splits for every task to create the final dataset
$\\{T_{k,r}\\}^{2}_{r=1},k\in[1,20]$. For instance, $T_{1,1}$ and $T_{1,2}$
have the exact same classes, but the images come from different splits of the
same classes. In general, $T_{k,r},\forall r$ are similar tasks for the same
$k$.

We can create CL task sequences for CIFAR10 and EMNIST in the same way as for
CIFAR100. CIFAR10 has 10 classes in total, so we can evenly split the entire
dataset into 5 tasks $\\{T_{k}\\}^{5}_{k=1}$ with 2 classes for each task, and
then create the mixed sequence dataset $\\{T_{k,r}\\}^{2}_{r=1},k\in[1,5]$.
For EMNIST (balanced version), to create a 100-task sequence, we divide 47
classes into 10 tasks $\\{T_{k,r}\\}^{10}_{r=1},k\in[1,10]$ with 5 classes for
$\\{T_{k,r}\\}^{10}_{r=1},k\in[1,9]$, and 2 classes for
$\\{T_{k,r}\\}^{10}_{r=1},k=10$. Additional details of the datasets can be
found in Tables 3 and 4 in the SM A.2.

###  5.1 Model Architecture and Experimental Details

We start the CL experiments with a _backbone_ $E_{0}(\cdot)$ and the models
for three dissimilar tasks trained in the repository. The system seeks to
identify if a task is new or seen to the CL system for $N$ new tasks, with
$N+3$ being the total number of tasks. Different task ordering in the sequence
may lead to different results for the system. To show the robustness of our
method to the effects of task ordering, all the experiments are repeated with
five randomly permuted task sequences.

We use a pretrained ResNet18 [47] model on ImageNet [48] as our global
backbone model $E_{0}(\cdot)$, and for each new task $\mathcal{T}_{t}$, we
train a set of task-specific parameters for $E_{t}(\cdot)$, $G_{t}(\cdot)$ and
$f_{t}(\cdot)$. The autoencoding pair for the distributional consistency
estimator is implemented with a standard variational autoencoder (VAE) [49].
To further reduce memory footprint, we also apply the feature transformation
approach discussed in Section 4.1 to the convolutional layers in VAE models
for the experiments with CIFAR10 and CIFAR100. We start with a VAE model
pretrained on ImageNette [50], a subset of ImageNet, and for each new task
$\mathcal{T}_{t}$, similar to $E_{t}(\cdot)$, we train a set of task-specific
parameters for each $G_{t}(\cdot)$. More architecture and experimental details
can be found in the SM A.1.

We evaluate our method using five performance characteristics. Due to the
nature of supervised learning tasks, we measure the overall average accuracy
(Acc) after learning all the tasks in the sequence. For each task, we report
the test-set accuracy. The highlight of our work is the similar task detection
module. Therefore, it is crucial to show how well the framework can identify
correctly whether a task is new or similar to a previously learned task. We
analyze the ratio of the number of correct identification (short for correct)
to $N$ new tasks. As for incorrect predictions (identification mistakes), we
also evaluate the percentage of times that the system misses (does not
recognize) the similar task previously seen in the past, and that the
incorrect identification (short for incorrect) to reuse to $N$ new tasks.
Further, we report the overall memory usage (in MB) by the end of training.

###  5.2 Baselines

The CL baseline methods to which we compare are: ($a$) Single Model per task:
Training a model separately for each task without finetuning, which means it
will not suffer from CF. ($b$) Optimal: The optimal situation for our method,
where the system always selects the right model to reuse for tasks in the
sequence or trains a new set of VAE parameters and classifier for the new
task. This serves as a strong upper bound for the proposed approach. ($c$)
Finetune: Sequentially training a single model on all the tasks without
handling forgetting issues. ($d$) New-Head: A single model with a shared
feature extractor for all the tasks and a new classification head for each
individual task. ($e$) Online EWC [5]: Training with a regularization term
added to the loss function for a single model, so that the changes to
important parameters are penalized during training for later tasks.

($f$) Experience Replay [7]: Finetuning with the subset of exemplars (10 for
each class) saved for each task for rehearsal purpose. ($g$) Deep Generative
Replay (DGR) [51]: Training a generative model to replay the images. Following
[51], the replayed images were labeled with the most likely category predicted
by a copy of the main model stored after training on the previous task, i.e.,
hard targets. ($h$) HAT [22]: Learning an attention mask over the parameters
in backbone to prevent forgetting. For fair comparison, we implement HAT with
a wider version of the original AlexNet model used in [22]. ($i$) CAT [19]:
This method considers a similar CL setting as ours. Additional baseline
architecture details can be found in the SM A.1.

Table 1: Task similarity prediction performance during continual learning. Results are averages over five randomly permuted CL sequences. | Correct | Mistake
---|---|---
SDR (EFT for $E\&G$) | Miss | Incorrect
CIFAR10 | 88.6$\%$ | 2.9$\%$ | 8.6$\%$
CIFAR100 | 89.7$\%$ | 4.3$\%$ | 5.9$\%$
EMNIST | 93.8$\%$ | 0.2$\%$ | 6.0$\%$
SDR (EFT for $E\&G$) |  |  |
CIFAR10 | 97.1$\%$ | 2.9$\%$ | 0.0$\%$
CIFAR100 | 88.1$\%$ | 3.8$\%$ | 8.1$\%$

###  5.3 Results

We first show the performance of task similarity prediction module in Table 1.
As a key component in our system, it demonstrates high prediction accuracy
across all benchmark datasets. The performance remains competitive for CIFAR10
and CIFAR100 even after we further reduce the memory usage by applying EFT
feature transformation to the generators $G(\cdot)$. We also note that,
according to our experimental results, the ImageNette-pretrained VAE is not
suitable as a backbone for EMNIST data, thus we do not report the
corresponding results.

Figure 2: Examples from the same permuted CIFAR100 task sequence. The left
y-axis shows $p(\mathcal{P}_{t}=\mathcal{P}_{j})$ of the current task with
previous tasks. The right y-axis shows the value of $S(j,t)$ for each previous
task $j$ with current task $t$. The x-axis represents all the previously
identified dissimilar tasks by the system. On the right, we show a list of the
20 superclass labels, the indexes of which correspond to $k$ in task
$T_{k,r}$.

In Figure 2, we also present two task similarity prediction examples from the
same permuted sequence of CIFAR100. The left figure shows the case where the
system successfully identifies $\mathcal{T}_{t}$ as a new task. Moreover, the
figure on the right shows the case where the system successfully identifies
$\mathcal{T}_{t}$ as a similar task from the past. Task $T_{1,2}$ images are
the most similar to task $T_{3,2}$ based on
$p(\mathcal{P}_{t}=\mathcal{P}_{j})$. Task $T_{1,2}$ consists of images in
superclass “fish” from CIFAR100 while task $T_{3,2}$ includes images from
superclass “aquatic mammals”. Both tasks have a substantial amount of images
with a blue ocean background, and dolphins and whales look like sharks.
Regardless, our similar task identification module realizes that features
extracted with task $T_{3,2}$ are not the most consistent with the labels
given the current learning knowledge. Concerning task $T_{19,2}$, from all 22
tasks kept in the knowledge repository, the system recognizes that the images
for the current task are the most similar to $T_{19,1}$. Hence, the features
extracted with the model for task $T_{19,1}$ can describe the labels the best.

We also compare SDR with several CL baselines in Table 2, in terms of
continual learning performance and memory usage. Though the naive Single Model
per task achieves highest average accuracy for all datasets, its memory
consumption is comparatively too high. While maintaining a comparable accuracy
performance to Single Model per task and Optimal, we successfully reduce the
memory footprint. This advantage is even more noticeable when the number of
tasks increases and more similar tasks are in the sequence, as revealed in the
EMNIST experiment. Notably, since finetuning with wrongly identified tasks on
previous encoder $E(\cdot)$ may lead to overall deterioration of predictive
performance, we freeze the parameters for tasks in the repository, which
yields no negative backward transfer. In practice, if desired, we can trade-
off memory and accuracy by initializing a new set of parameters weights from a
previous similar task, which will lead to positive forward transfer. However,
this is left as interesting future work.

Table 2: Comparing SDR with continual learning baselines. Results are averages over five randomly permuted CL sequences. Method | CIFAR10 | (10 tasks) |  | CIFAR100 | (40 tasks) |  | EMNIST | (100 tasks)
---|---|---|---|---|---|---|---|---
|  Acc($\%$) | Mem(MB) |  |  Acc($\%$) | Mem(MB) |  |  Acc($\%$) | Mem(MB)
Optimal (EFT for $E$) | 88.45 | 76.2 |  | 68.30 | 169.2 |  | 97.41 | 107.0
Optimal (EFT for $E$ and $G$) | 88.45 | 64.7 |  | 68.30 | 114.0 |  | - | -
Single Model per task | 92.28 | 453.0 |  | 72.97 | 1812.0 |  | 97.39 | 4530.0
Finetune | 50.92 | 45.3 |  | 21.94 | 45.3 |  | 32.84 | 45.3
New-Head | 56.10 | 45.3 |  | 23.22 | 45.7 |  | 36.93 | 46.3
ER | 78.22 | 55.7 |  | 24.83 | 61.2 |  | 43.36 | 71.0
DGR | 63.70 | 95.0 |  | 21.79 | 95.0 |  | 25.16 | 95.0
Online EWC | 79.54 | 140.4 |  | 43.31 | 140.4 |  | 89.89 | 140.4
CAT | 79.52 | 210.8 |  | 50.20 | 220.8 |  | 93.27 | 200.0
HAT | 86.63 | 104.7 |  | 37.09 | 108.7 |  | 93.76 | 114.7
SDR (EFT for $E$) | 87.25 | 77.4 |  | 67.17 | 176.8 |  | 96.26 | 109.0
SDR (EFT for $E$ and $G$) | 88.64 | 65.4 |  | 66.94 | 119.2 |  | - | -

##  6 Conclusion and Future Work

In this study, we explored a practical TCL context where tasks are not always
distinct from each other. We propose a mechanism that allows for the
identification of previously seen tasks to avoid repetitive training; as a
means to address memory issues with parameter expansion methods for TCL. The
proposed mechanism is analyzed across several image benchmarks in terms of
identifying reusable modules (encoders and generators) from previous
experience. Our experimental results showed promising performance even without
any regularization or special modulation to the predictive models. For future
work, we intend to explore the possibility of integrating our task similarity
detection module in even more realistic Cl settings, where there is class
overlap between tasks.

## References

  * [1] Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick.  Neuroscience-inspired artificial intelligence.  Neuron, 95(2):245–258, 2017.
  * [2] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114:3521 – 3526, 2017.
  * [3] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon.  Uncertainty-based continual learning with adaptive regularization.  In NeurIPS, volume 32, 2019.
  * [4] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner.  Variational continual learning.  In ICLR, 2018.
  * [5] Jonathan Schwarz, Wojciech M. Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In ICML, volume abs/1805.06370, 2018.
  * [6] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, G. Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In CVPR, pages 5533–5542, 2017.
  * [7] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne.  Experience replay for continual learning.  In NeurIPS, 2019.
  * [8] David Isele and Akansel Cosgun.  Selective experience replay for lifelong learning.  In AAAI, 2018.
  * [9] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  In NIPS, 2017.
  * [10] Gido M. van de Ven and Andreas Savas Tolias.  Generative replay with feedback connections as a general strategy for continual learning.  ArXiv, abs/1809.10635, 2018.
  * [11] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In ICML, 2019.
  * [12] Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim.  A neural dirichlet process mixture model for task-free continual learning.  In ICLR, 2019.
  * [13] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In ICLR, 2018.
  * [14] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  ArXiv, abs/1606.04671, 2016.
  * [15] Tom Véniat, Ludovic Denoyer, and Marc’Aurelio Ranzato.  Efficient continual learning with modular networks and task-driven priors.  In ICLR, volume abs/2012.12631, 2021.
  * [16] Oleksiy Ostapenko, Pau Rodríguez López, Massimo Caccia, and Laurent Charlin.  Continual learning via local module composition.  In NeurIPS, volume abs/2111.07736, 2021.
  * [17] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines, 2018.
  * [18] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
  * [19] Zixuan Ke, Bing Liu, and Xingchang Huang.  Continual learning of a mixed sequence of similar and dissimilar tasks.  In NeurIPS, volume 33, pages 18493–18504, 2020.
  * [20] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  pages 32–33, 2009.
  * [21] Gregory Cohen, Saeed Afshar, Jonathan C. Tapson, and André van Schaik.  Emnist: an extension of mnist to handwritten letters.  In CVPR, volume abs/1702.05373, 2017.
  * [22] Joan Serrà, Dídac Surís, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In ICML, volume abs/1801.01423, 2018.
  * [23] Prakhar Kaushik, Alex Gain, Adam Kortylewski, and Alan Loddon Yuille.  Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping.  ArXiv, abs/2102.11343, 2021.
  * [24] Gido M. van de Ven and Andreas Savas Tolias.  Three scenarios for continual learning.  ArXiv, abs/1904.07734, 2019.
  * [25] Ameya Prabhu, Philip Torr, and Puneet Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In ECCV, August 2020.
  * [26] Ahmet Iscen, Jeffrey O. Zhang, Svetlana Lazebnik, and Cordelia Schmid.  Memory-efficient incremental learning through feature adaptation.  ECCV, abs/2004.00713, 2020.
  * [27] Sakshi Varshney, Vinay Kumar Verma, Srijith P K, Lawrence Carin, and Piyush Rai.  Cam-gan: Continual adaptation modules for generative adversarial networks.  In NeurIPS, 2021.
  * [28] Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, and Lawrence Carin.  Gan memory with no forgetting.  In NeurIPS, volume 33, pages 16481–16494, 2020.
  * [29] Zichen Miao, Ze Wang, Wei Chen, and Qiang Qiu.  Continual learning with filter atom swapping.  In International Conference on Learning Representations, 2022.
  * [30] Vinay Kumar Verma, Kevin J Liang, Nikhil Mehta, Piyush Rai, and Lawrence Carin.  Efficient feature transformations for discriminative and generative continual learning.  In CVPR, pages 13860–13870, 2021.
  * [31] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  ArXiv, abs/1704.04861, 2017.
  * [32] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville.  Film: Visual reasoning with a general conditioning layer.  In AAAI, 2018.
  * [33] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In CVPR, pages 7132–7141, 2018.
  * [34] Samuel J. Gershman and David M. Blei.  A tutorial on bayesian nonparametric models.  Journal of Mathematical Psychology, 56:1–12, 2011.
  * [35] David M Blei and Michael I Jordan.  Variational inference for dirichlet process mixtures.  Bayesian analysis, 1(1):121–143, 2006.
  * [36] Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  In ICLR, 2017.
  * [37] Christopher P. Burgess, Irina Higgins, Arka Pal, Loïc Matthey, Nicholas Watters, Guillaume Desjardins, and Alexander Lerchner.  Understanding disentangling in $\beta$-VAE.  In NIPS, 2017.
  * [38] Eric Pauwels and Onkar Ambekar.  One class classification for anomaly detection: Support vector data description revisited.  In ICDM, pages 25–39, 08 2011.
  * [39] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft.  Deep one-class classification.  In ICML, pages 4393–4402. PMLR, 2018.
  * [40] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  In CACM, volume 64, pages 107–115. ACM New York, NY, USA, 2021\.
  * [41] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang.  Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks.  In ICML. PMLR, 2019.
  * [42] Gintare Karolina Dziugaite and Daniel M. Roy.  Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data.  In UAI, volume abs/1703.11008, 2017.
  * [43] Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz.  Non-vacuous generalization bounds at the imagenet scale: a pac-bayesian compression approach.  In ICLR, 2019.
  * [44] Xingguo Li, Junwei Lu, Zhaoran Wang, Jarvis D. Haupt, and Tuo Zhao.  On tighter generalization bound for deep neural networks: Cnns, resnets, and beyond.  In ICLR, volume abs/1806.05159, 2019.
  * [45] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In ICLR, volume abs/1312.6114, 2014.
  * [46] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe.  Variational inference: A review for statisticians.  Journal of the American Statistical Association, 112:859 – 877, 2016.
  * [47] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, pages 248–255. Ieee, 2009.
  * [49] Dor Bank, Noam Koenigstein, and Raja Giryes.  Autoencoders.  ArXiv, abs/2003.05991, 2020.
  * [50] Jeremy Howard.  Imagenette.
  * [51] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  In NIPS, 2017.
  * [52] Simon Shaolei Du, Xiyu Zhai, Barnabás Póczos, and Aarti Singh.  Gradient descent provably optimizes over-parameterized neural networks.  ICLR, abs/1810.02054, 2019.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]  The continual learning
scenario we discuss in the paper is just one practical case of real-world
settings. In Section 6, we claimed that we are going to explore even more
practical settings where there is class overlap between tasks.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [No]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [Yes]
The assumptions of theoretical results can be found in Section 3.

    2. (b)

Did you include complete proofs of all theoretical results? [Yes]  It is shown
in the SM A.3.

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [No]
The code will be released upon acceptance.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]  Model architecture and experimental details,
including data description, hyperparameter selection are provided in the SM
A.2 and A.1.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes]  The corresponding performance is provided
in the SM.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [No]  We have not,
however, these details will be included in the final version.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [No]  The code reference is not
mentioned in the main paper, but we will provide the license of the assets in
our code base acknowledgement section.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]  The CL tasks sequences will be released with the source code.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [No]  The data we are using in the paper is open-
source.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [No]  There is no personally
identifiable information or offensive content in the data we are using.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Appendix

###  A.1 Experimental Details

In all the experiments except for methods CAT and HAT, we use a pretrained
ResNet18 as the backbone of the classification models. To adapt the same
pretrained backbone model for EMNIST, which contains images of one channel, we
first transform the grey-scale images into three-channel images. For a fair
comparison, we adopt the efficient feature transformation technique in
Optimal, Finetune, New-Head, and SDR. Serving as a benchmark, Single Model per
task is trained with a full ResNet18 for each task. They all use the
pretrained ResNet18 on ImageNet as a backbone. For CAT, we show the results
with a 2 fully connected layer network used in the original paper. Following
the setting in CAT paper [19], the embeddings for the hidden and final layer
in the knowledge base have a dimension of 2000; the task ID embeddings also
have 2000 dimensions. For HAT, we switched to a wide version of AlexNet model
as backbone. Specifically, each layer in the wide version has twice the number
of nodes compared to each layer in the original version [22]. For DGR, we
follow the setting in [10] for the generative model, which is a symmetric VAE
with 3 fully connected layers (fc1000 - fc1000 - fc100). As for the VAE models
in SDR, an architecture of 4 convolutional layers and 1 linear layer is used
for both encoder and decoder.

All the classification models (when feature extractors are newly trained) are
trained with a batch size of 128 for 100 epochs, with a starting learning rate
of 0.01 reduced with a factor of 0.1 at epoch 50. The only data
augmentation/transformation used for the datasets (except for EMNIST, which is
converted from one-channel to three-channels) is normalization. When the
feature extractor is reused, the classification head only needs to be trained
for 2 epochs with a learning rate of 0.001. As for VAE models trained in SDR,
they are trained with a batch size of 64 and a learning rate of 0.0001, for
2000 epochs with early stopping criteria.

Here we provide a detailed view of the SDR framework in 1.

Algorithm 1 SDR Framework

Data: $D_{t}=\\{\mathbf{x}_{i}^{t},y^{t}_{i}\\}^{n_{t}}_{i=1},t\in[1,N]$;
(Only $D_{t}$ available at time $t$)

Result: $\\{E_{m}\\}^{C}_{m=1}$, $\\{G_{m}\\}^{C}_{m=1}$, with $C$ being the
number of unique tasks recognized by the system

_Starting with:_ $E_{0}$, $\\{E_{m}\\}\leftarrow\\{E_{m}\\}^{3}_{m=1}$;
$G_{0}$ (might or might not be available),
$\\{G_{m}\\}\leftarrow\\{G_{m}\\}^{3}_{m=1}$ for  _$t\leftarrow 1$ to $N$_ do

for  _$j\leq t-1$_ do

for  _$i\leftarrow 1$ to $n^{t}$_ do

$p(u_{i}=j|\mathbf{x}_{i})$ ;

/* Distributional Consistency Estimator */

end for

$p(\mathcal{P}_{t}=\mathcal{P}_{j})=\mathbb{E}_{\mathbf{x}_{i}\in\mathcal{D}_{t}}[p(u_{i}=j|\mathbf{x}_{i})]$

end for

$\mathcal{T}_{a}=\arg\max_{j}\\{p(\mathcal{P}_{t}=\mathcal{P}_{j})\\}^{t-1}_{=1}$
for  _$j\leq t-1$_ do

$\mathbf{H}^{j,t}_{ik}=\frac{{\mathbf{e}^{j,t}_{i}}^{\top}\mathbf{e}^{k,t}_{k}(\pi-\arccos({\mathbf{e}^{j,t}_{i}}^{\top}\mathbf{e}^{j,t}_{k}))}{2\pi}$,
$\mathbf{A}^{j,t}=\mathbf{y}^{\mathsf{T}}(\mathbf{H}^{j,t})^{-1}\mathbf{y}$
$S(j,t)\leftarrow\sqrt{\frac{2||{\mathbf{A}^{j,t}}^{\top}\mathbf{A}^{j,t}||^{2}_{2}}{n_{t}}}$
;

/* Predictor-label Association Analysis */

end for

$\mathcal{T}_{b}=\arg\min_{j}\\{S(j,t)\\}^{t-1}_{j=1}$ if
_$\mathcal{T}_{a}==\mathcal{T}_{b}$_ then
reuse feature encoder and VAE model from task $\mathcal{T}_{a}$ for task
$\mathcal{T}_{t}$,
only train a new classification head $f_{t}$ for $\mathcal{T}_{t}$

$E_{t}\leftarrow E_{a}$ $G_{t}\leftarrow G_{a}$

else train new models for task $\mathcal{T}_{t}$

repeat

$\mathbf{B}\leftarrow$ random sampled mini-batch from $D_{t}$
$\mathbf{g}=\nabla_{\theta_{E_{t}}}\mathcal{L}(\phi_{E_{t}},\mathbf{B})$
$\theta_{E_{t}}\leftarrow AdamOptimizer(\theta_{E_{t}},\mathbf{g})$

until  _$\theta_{E_{t}}$ converges_;

repeat

$\mathbf{B}\leftarrow$ random sampled mini-batch from $D_{t}$
$\mathbf{g}=\nabla_{\psi_{G_{t}},\phi_{G_{t}}}\mathcal{L}(\psi_{G_{t}},\phi_{G_{t}},\mathbf{B})$
$\psi_{G_{t}},\phi_{G_{t}}\leftarrow
AdamOptimizer(\psi_{G_{t}},\phi_{G_{t}},\mathbf{g})$

until  _$\psi_{G^{t}},\phi_{G^{t}}$ converges_;

end if

end for

###  A.2 Dataset Details

In Table 3, we provide the number of tasks in the sequences created with each
dataset, the number of classes for each task, and the size of each data split.

Table 3: Details of the Datasets Dataset | # tasks | # classes/tsk | training/tsk | validation/tsk | testing/tsk
---|---|---|---|---|---
CIFAR10 | 10 | 2 | 4500 | 500 | 1000
CIFAR100 | 40 | 5 | 1125 | 125 | 250
EMNIST | 100 | 5 or 2 | 1080 or 432 | 120 or 48 | 200 or 80
Table 4: Details for Classes in Each Task Dataset | $k$ | Classes
---|---|---
CIFAR10 | 1 | [’airplane’, ’bird’]
| 2 | [’automobile’, ’truck’]
| 3 | [’cat’, ’dog’]
| 4 | [’deer’, ’horse’]
| 5 | [’frog’, ’ship’]
CIFAR100 | 1 | [’aquarium fish’, ’flatfish’, ’ray’, ’shark’, ’trout’]
| 2 | [’orchid’, ’poppy’, ’rose’, ’sunflower’, ’tulip’]
| 3 | [’beaver’, ’dolphin’, ’otter’, ’seal’, ’whale’]
| 4 | [’bottle’, ’bowl’, ’can’, ’cup’, ’plate’]
| 5 | [’bear’, ’leopard’, ’lion’, ’tiger’, ’wolf’]
| 6 | [’apple’, ’mushroom’, ’orange’, ’pear’, ’sweet pepper’]
| 7 | [’hamster’, ’mouse’, ’rabbit’, ’shrew’, ’squirrel’]
| 8 | [’baby’, ’boy’, ’girl’, ’man’, ’woman’]
| 9 | [’maple tree’, ’oak tree’, ’palm tree’, ’pine tree’, ’willow tree’]
| 10 | [’bicycle’, ’bus’, ’motorcycle’, ’pickup truck’, ’train’]
| 11 | [’clock’, ’keyboard’, ’lamp’, ’telephone’, ’television’]
| 12 | [’bed’, ’chair’, ’couch’, ’table’, ’wardrobe’]
| 13 | [’bee’, ’beetle’, ’butterfly’, ’caterpillar’, ’cockroach’]
| 14 | [’bridge’, ’castle’, ’house’, ’road’, ’skyscraper’]
| 15 | [’cloud’, ’forest’, ’mountain’, ’plain’, ’sea’]
| 16 | [’camel’, ’cattle’, ’chimpanzee’, ’elephant’, ’kangaroo’]
| 17 | [’fox’, ’porcupine’, ’possum’, ’raccoon’, ’skunk’]
| 18 | [’crab’, ’lobster’, ’snail’, ’spider’, ’worm’]
| 19 | [’crocodile’, ’dinosaur’, ’lizard’, ’snake’, ’turtle’]
| 20 | [’lawn mower’, ’rocket’, ’streetcar’, ’tank’, ’tractor’]
EMNIST | 1 | [’0’, ’1’, ’2’, ’3’, ’4’]
| 2 | [’5’, ’6’, ’7’, ’8’, ’9’]
| 3 | [’A’, ’B’, ’D’, ’E’, ’F’]
| 4 | [’G’, ’H’, ’N’, ’Q’, ’R’]
| 5 | [’T’, ’a’, ’b’, ’c’, ’d’]
| 6 | [’e’, ’f’, ’g’, ’h’, ’i’]
| 7 | [’j’, ’k’, ’l’, ’m’, ’n’]
| 8 | [’o’, ’p’, ’q’, ’r’, ’s’]
| 9 | [’t’, ’u’, ’v’, ’w’, ’x’]
| 10 | [’y’, ’z’]

###  A.3 Proof

In Section A.3.1 to A.3.3, we provide theoretical proofs for our similarity
metric, while in Section A.3.4 we show empirical proofs.

####  A.3.1 Setting

Given a two-layer ReLU activated neural network with $m$ neurons in the hidden
layer and $c$ neurons in the output layer:

$\displaystyle
f_{\mathbf{W},\mathbf{A}}(\mathbf{x})=\frac{1}{\sqrt{m}}\sum^{m}_{r=1}\mathbf{a}_{r}\sigma(\mathbf{w}_{r}^{\top}\mathbf{x}),$
(9)

where $\mathbf{x}\in\mathbb{R}^{d}$ is an input,
$\mathbf{W}=(\mathbf{w}_{1},\cdots,\mathbf{w}_{m})\in\mathbb{R}^{d\times m}$
are the weight vectors in the first layer and
$\mathbf{A}=(\mathbf{a}_{1},\cdots,\mathbf{a}_{m})^{\top}\in\mathbb{R}^{c\times
m}$ are weight vectors in the second layer, with $c$ being the number of
classes.

We train the neural network by _randomly initialized gradient descent_ (GD) on
the quadratic loss over data. In particular, we first initialize the
parameters randomly:111For each $r\in[1,m]$, randomly pick sum $s_{r}$ of the
$r$-th row $\mathbf{a}_{r}$ from $\text{unif}(\\{-1,1\\})$. For the first
$c-1$ neurons, we randomly sample $a_{z,r}\sim\text{unif}([-1,1]),\forall
z=1,\cdots,c-1$, the last neuron is set to satisfy the column sum, i.e.,
$a_{c,r}=s_{r}-\sum^{c-1}_{z=1}a_{z,r}$.

$\displaystyle\mathbf{w}_{r}(0)\sim\mathcal{N}(\mathbf{0},\kappa^{2}\mathbf{I}),\qquad\sum^{c}_{z=1}a_{z,r}\sim\text{unif}(\\{-1,1\\}),\qquad\forall
r\in[m],$ (10)

where $0<\kappa\leq 1$ controls the magnitude of initialization.

We want to minimize quadratic loss of

$\displaystyle\mathcal{L}(\mathbf{W})=\frac{1}{2N}\sum^{N}_{i=1}\sum^{c}_{z=1}(y_{iz}-f_{\mathbf{W},\mathbf{a}_{z}}(\mathbf{x}_{i}))^{2},$
(11)

through GD by fixing the second layer $\mathbf{A}$ and only optimizing the
first layer $\mathbf{W}$

$\displaystyle\frac{\partial\mathcal{L}(\mathbf{W})}{\partial\mathbf{w}_{r}}=\frac{1}{2N}\sum^{c}_{z=1}a_{z,r}\sum^{N}_{i=1}(f_{\mathbf{W},\mathbf{a}_{z}}(\mathbf{x}_{i})-y_{iz})\mathbb{I}\\{\mathbf{w}_{r}^{\top}\mathbf{x}_{i}\geq
0\\}\mathbf{x}_{i},\qquad\forall r\in[m].$ (12)

Define $\hat{y}_{iz}=f_{\mathbf{W},\mathbf{a}_{z}}(\mathbf{x}_{i})$, i.e., the
network’s prediction on the $i$-th input that belongs to class $z$, and (12)
can be rewritten as:

$\displaystyle\frac{\partial\mathcal{L}(\mathbf{W})}{\partial\mathbf{w}_{r}}=\frac{1}{2N}\sum^{c}_{z=1}a_{z,r}\sum^{N}_{i=1}(\hat{y}_{iz}-y_{iz})\mathbb{I}_{r,i}\mathbf{x}_{i},\qquad\forall
r\in[m],$ (13)

where
$\mathbb{I}_{r,i}=\mathbb{I}\\{\mathbf{\mathbf{w}^{\top}_{r}\mathbf{x}_{i}}\geq
0\\}$. We also define $\mathbf{Z}$ as:

$\displaystyle\mathbf{Z}=\frac{1}{\sqrt{m}}\begin{bmatrix}\mathbb{I}_{1,1}(\sum^{c}_{z=1}a_{z,1})\mathbf{x}_{1}&\cdots&\mathbb{I}_{1,1}(\sum^{c}_{z=1}a_{z,1})\mathbf{x}_{n}\\\
\vdots&\ddots&\vdots\\\
\mathbb{I}_{m,1}(\sum^{c}_{z=1}a_{z,m})\mathbf{x}_{1}&\cdots&\mathbb{I}_{m,n}(\sum^{c}_{z=1}a_{z,m})\mathbf{x}_{n}\end{bmatrix}$
(14)

With this notation we have a more compact form of the gradient (12):

$\displaystyle\text{vec}\left(\mathbf{\nabla(\mathcal{L}(\mathbf{W}))}\right)=\mathbf{Z}(\hat{\mathbf{y}}-\mathbf{y}).$
(15)

$\mathbf{H}=\mathbf{Z}^{\top}\mathbf{Z}$,222$\mathbf{H}$ corresponds to
$\mathbf{H}^{\infty}$ in [41], which denotes the Gram matrix when GD
converges, here we abuses the notation to be consist with the main paper since
we initialize the second layer with
$\sum^{c}_{z=1}a_{z,r}\sim\text{unif}(\\{-1,1\\})$

$\displaystyle\mathbf{H}_{ij}$
$\displaystyle=\frac{\mathbf{x}_{i}\mathbf{x}_{j}}{m}\sum^{m}_{r=1}\mathbb{I}_{r,1}\mathbb{I}_{r,2}(\sum^{c}_{z=1}a_{z,r})^{2}$
(16)
$\displaystyle=\frac{\mathbf{x}_{i}\mathbf{x}_{j}}{m}\sum^{m}_{r=1}\mathbb{I}_{r,i}\mathbb{I}_{r,j},\qquad\forall
i,j\in[n]$ (17)

and thus we prove that we can obtain the Gram matrix $\mathbf{H}$ the same
form for multi-classification setting as in binary-classification settings,
and complete the proof of (7).

####  A.3.2 Rademacher Complexity and Generalization

Generalization error measures how accurate an algorithm is for predicting
outcome values for previously unseen data. Specifically, given a function
$f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ and a loss function
$\mathit{l}:\mathbb{R}\times\mathbb{R}\rightarrow\mathbb{R}$, generalization
error can be defined as the gap between the _population loss_ over data
distribution $\mathcal{P}$ and the _empirical loss_ over $n$ samples
$\mathcal{D}=\\{(\mathbf{x}_{i},y_{i})\\}^{n}_{i=1}$ from $\mathcal{P}$:

$\displaystyle\mathcal{L}_{\mathcal{P}}(f)-\mathcal{L}_{\mathcal{D}}(f)=\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{P}}[\mathit{l}(f(\mathbf{x},y)]-\frac{1}{n}\sum^{n}_{i=1}\ell(f(\mathbf{x}_{i}),y_{i})$
(18)

######  Definition A.1 ([41])

Given $n$ samples $\mathcal{D}$, the empirical Rademacher complexity of a
function class $\mathcal{F}$ (mapping from $\mathbb{R}^{d}$) is defined as:

$\displaystyle\mathcal{R}_{\mathcal{D}}(\mathcal{F})=\frac{1}{n}\mathbb{E}_{\boldsymbol{\epsilon}}\bigg{[}\sup_{f\in\mathcal{F}}\sum^{n}_{i=1}\epsilon_{i}f(\mathbf{x}_{i})\bigg{]},$

where $\boldsymbol{\epsilon}=(\epsilon_{1},\cdots,\epsilon_{n})^{\top}$
contains $i.i.d$ random variables drawn from the Rademacher distribution unif
({1, -1}).

Rademacher complexity directly gives an upper bound on generalization error:

######  Theorem A.1 ([41])

Suppose the loss function $\mathit{l}(\cdot,\cdot)$ is bounded in $[0,c]$ and
is $\rho$-Lipschitz in the first argument. Then with probability at least
$1-\delta$ over sample $\mathcal{D}$ of size $n$:

$\displaystyle\sup_{f\in\mathcal{F}}\\{\mathcal{L}_{\mathcal{P}}(f)-\mathcal{L}_{\mathcal{D}}(f)\\}\leq
2\rho\mathcal{R}_{\mathcal{D}}(\mathcal{F})+3c\sqrt{\frac{\log(2/\delta)}{2n}}$
(19)

Therefore, as long as we can bound the Rademacher complexity of a certain
function class that contains our learned predictor, we can obtain a
generalization bound.

####  A.3.3 Proof for Complexity Measure under Multi-class Scenario

In this section, we show proof to the complexity measure (8) under multi-class
classification settings. We note that most of the related proofs of complexity
measure (generalization bound) under binary classification setting are
provided in [41].

######  Theorem A.2 ([52, 41])

Assume $\lambda_{0}=\lambda_{\min}(\mathbf{H}^{\infty})>0$. For
$\delta\in(0,1)$, if
$m=\Omega\left(\frac{n^{6}}{\lambda_{0}^{4}\kappa^{2}\delta^{3}}\right)$ and
$\eta=O\left(\frac{\lambda_{0}}{n^{2}}\right)$, then with probability at least
$1-\delta$ over the random initialization, we have:

  * •

$\mathcal{L}(\mathbf{W}(0))=O(n/\delta)$;

  * •

$\mathcal{L}(\mathbf{W}(k+1))\leq\left(1-\frac{\eta\lambda_{0}}{2}\right)\mathbf{\mathcal{L}}(\mathbf{W}(k)),\
\forall k\geq 0$.

Analysis of the Auxiliary Sequence
$\left\\{\widetilde{\mathbf{W}}(k)\right\\}_{k=0}^{\infty}$ 333$k$ denotes the
optimization step in the proof section Now we give a proof of
$\left\|\widetilde{\mathbf{W}}(\infty)-\widetilde{\mathbf{W}}(0)\right\|_{F}=\sqrt{\left\|\mathbf{y}^{\top}\mathbf{H}(0)^{-1}\mathbf{y}\right\|_{F}}$
as an illustration for the proof of Lemma A.1. Define
$\mathbf{v}(k)=\mathbf{Z}(0)^{\top}\text{vec}\left(\widetilde{\mathbf{W}}(k)\right)\in\mathbb{R}^{n}$.
Then from (20) we have $\mathbf{v}(0)=\mathbf{0}$ and
$\mathbf{v}(k+1)=\mathbf{v}(k)-\eta\mathbf{H}(0)(\mathbf{v}(k)-\mathbf{y})$,
yielding
$\mathbf{v}(k)-\mathbf{y}=-(\mathbf{I}-\eta\mathbf{H}(0))^{k}\mathbf{y}$.
Plugging this back to (20) we get
$\text{vec}\left(\widetilde{\mathbf{W}}(k+1)\right)-\text{vec}\left(\widetilde{\mathbf{W}}(k)\right)=\eta\mathbf{Z}(0)(\mathbf{I}-\eta\mathbf{H}(0))^{k}\mathbf{y}$.
Then taking a sum over step $k=0,1,\ldots$ we have

$\displaystyle\text{vec}\left(\widetilde{\mathbf{W}}(\infty)\right)-\text{vec}\left(\widetilde{\mathbf{W}}(0)\right)$
$\displaystyle=\sum_{k=0}^{\infty}\eta\mathbf{Z}(0)(\mathbf{I}-\eta\mathbf{H}(0))^{k}\mathbf{y}$
$\displaystyle=\mathbf{Z}(0)\mathbf{H}(0)^{-1}\mathbf{y}.$

The desired result thus follows:444Note that we have
$\mathbf{H}(0)\approx\mathbf{H}^{\infty}$ from standard concentration. See
Lemma C.3 in [41]

$\displaystyle\left\|\widetilde{\mathbf{W}}(\infty)-\widetilde{\mathbf{W}}(0)\right\|_{F}^{2}$
$\displaystyle=\mathbf{y}^{\top}\mathbf{H}(0)^{-1}\mathbf{Z}(0)^{\top}\mathbf{Z}(0)\mathbf{H}(0)^{-1}\mathbf{y}$
$\displaystyle=\mathbf{y}^{\top}\mathbf{H}(0)^{-1}\mathbf{y}.$
$\displaystyle\left\|\widetilde{\mathbf{W}}(\infty)-\widetilde{\mathbf{W}}(0)\right\|_{F}$
$\displaystyle=\sqrt{\left\|\mathbf{y}^{\top}\mathbf{H}(0)^{-1}\mathbf{y}\right\|_{F}}$

Lemma A.1’s proof is based on the careful characterization of the trajectory
of $\left\\{\mathbf{W}(k)\right\\}_{k=0}^{\infty}$ during GD. In particular,
we bound its _distance to initialization_ as follows:

######  Lemma A.1 ([41])

555$m$ is the number of parameters in the first layer, $\eta$ is the learning
rate

Suppose
$m\geq\kappa^{-2}\mathrm{poly}\left(n,\lambda_{0}^{-1},\delta^{-1}\right)$ and
$\eta=O\left(\frac{\lambda_{0}}{n^{2}}\right)$. Then with probability at least
$1-\delta$ over the random initialization, we have for all $k\geq 0$:

  * •

$\left\|\mathbf{w}_{r}(k)-\mathbf{w}_{r}(0)\right\|_{2}=O\left(\frac{n}{\sqrt{m}\lambda_{0}\sqrt{\delta}}\right)$
$(\forall r\in[m])$, and

  * •

$\left\|\mathbf{W}(k)-\mathbf{W}(0)\right\|_{F}\leq\sqrt{\|\mathbf{y}^{\top}\left(\mathbf{H}^{\infty}\right)^{-1}\mathbf{y}\|_{F}}+O\left(\frac{n\kappa}{\lambda_{0}\delta}\right)+\frac{\mathrm{poly}\left(n,\lambda_{0}^{-1},\delta^{-1}\right)}{m^{1/4}\kappa^{1/2}}$.

The bound on the movement of each $\mathbf{w}_{r}$ was proved in [52]. The
bound on $\left\|\mathbf{W}(k)-\mathbf{W}(0)\right\|_{F}$ corresponds to the
_total movement of all neurons_. The main idea is to couple the trajectory of
$\left\\{\mathbf{W}(k)\right\\}_{k=0}^{\infty}$ with another simpler
trajectory $\left\\{\widetilde{\mathbf{W}}(k)\right\\}_{k=0}^{\infty}$ defined
as:

$\displaystyle\widetilde{\mathbf{W}}(0)=\,$ $\displaystyle\mathbf{0},$
$\displaystyle\text{vec}\left(\widetilde{\mathbf{W}}(k+1)\right)=\,$
$\displaystyle\text{vec}\left(\widetilde{\mathbf{W}}(k)\right)$ (20)
$\displaystyle-\eta\mathbf{Z}(0)\left(\mathbf{Z}(0)^{\top}\text{vec}\left(\widetilde{\mathbf{W}}(k)\right)-\mathbf{y}\right).$

######  Lemma A.2 ([41])

Given $R>0$, with probability at least $1-\delta$ over the random
initialization ($\mathbf{W}(0),\mathbf{A})$, simultaneously for every $B>0$,
the following function class

$\displaystyle\mathcal{F}^{\mathbf{W}(0),\mathbf{A}}_{R,B}=\\{f_{\mathbf{W},\mathbf{A}}:\left\|\mathbf{w}_{r}-\mathbf{w}_{r}(0)\right\|_{2}\leq
R\,(\forall r\in[m]),$
$\displaystyle\left\|\mathbf{W}-\mathbf{W}(0)\right\|_{F}\leq B\\}$

has empirical Rademacher complexity bounded as:

$\displaystyle\mathcal{R}_{\mathcal{D}}\left(\mathcal{F}^{\mathbf{W}(0),\mathbf{A}}_{R,B}\right)=\frac{1}{n}\mathbb{E}_{\boldsymbol{\epsilon}\in\\{\pm
1\\}^{n}}\left[\sup_{f\in\mathcal{F}^{\mathbf{W}(0),\mathbf{A}}_{R,B}}\sum_{i=1}^{n}\epsilon_{i}f(\mathbf{x}_{i})\right]$
$\displaystyle\leq\,$
$\displaystyle\frac{B}{\sqrt{2n}}\left(1+\left(\frac{2\log\frac{2}{\delta}}{m}\right)^{1/4}\right)+\frac{2R^{2}\sqrt{m}}{\kappa}+R\sqrt{2\log\frac{2}{\delta}}.$

For the proof of Lemma A.2, the only difference compared to Lemma 5.4 in [41]
is $a_{r}$ in [41] becomes $\sum^{c}_{z=1}\mathbf{a}_{r,z}$ in our setting.

Finally, combining Lemma A.1 with A.2, we are able to conclude that the neural
network found by GD belongs to a function class with Rademacher complexity at
most
$\sqrt{\|\mathbf{y}^{\top}(\mathbf{H}^{\infty})^{-1}\mathbf{y}\|_{F}/(2n)}$
(plus negligible errors). This gives us the generalization bound in Theorem
A.3 using Rademacher complexity.

###### Definition A.2

A distribution $\mathcal{P}$ over $\mathbb{R}^{d}\times\mathbb{R}$ is
$(\lambda_{0},\delta,n)$-non-degenerate, if for $n$ i.i.d. samples
$\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{n}$ from $\mathcal{P}$, with probability
at least $1-\delta$ we have
$\lambda_{\min}(\mathbf{H}^{\infty})\geq\lambda_{0}>0$.

######  Theorem A.3 ([41])

Fix a failure probability $\delta\in(0,1)$. Suppose our data
$\mathcal{D}=\left\\{(\mathbf{x}_{i},y_{i})\right\\}_{i=1}^{n}$ are i.i.d.
samples from a $(\lambda_{0},\delta/3,n)$-non-degenerate distribution
$\mathcal{P}$, and
$\kappa=O\left(\frac{\lambda_{0}\delta}{n}\right),m\geq\kappa^{-2}\mathrm{poly}\left(n,\lambda_{0}^{-1},\delta^{-1}\right)$.
Consider any loss function $\ell:\mathbb{R}\times\mathbb{R}\to[0,1]$ that is
$1$-Lipschitz in the first argument such that $\ell(y,y)=0$. Then with
probability at least $1-\delta$ over the random initialization and the
training samples, the two-layer neural network $f_{\mathbf{W}(k),\mathbf{A}}$
trained by GD for
$k\geq\Omega\left(\frac{1}{\eta\lambda_{0}}\log\frac{n}{\delta}\right)$
iterations has population loss
$L_{\mathcal{P}}(f_{\mathbf{W}(k),\mathbf{A}})=\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{P}}\left[\ell(f_{\mathbf{W}(k),\mathbf{A}}(\mathbf{x}),y)\right]$
bounded as:

$L_{\mathcal{P}}(f_{\mathbf{W}(k),\mathbf{A}})\leq\sqrt{\frac{\left\|2\mathbf{y}^{\top}\left(\mathbf{H}^{\infty}\right)^{-1}\mathbf{y}\right\|_{F}}{n}}+O\left(\sqrt{\frac{\log\frac{n}{\lambda_{0}\delta}}{n}}\right).$
(21)

####  A.3.4 Empirical Proof for Using Complexity Measure as our Similarity
Metric

In Figures 3, 4, 5, we intent to show that datasets encoded with feature
extractors trained on similar tasks obtain relatively smaller values compared
to those trained on dissimilar tasks. In the heatmaps, the y-axis represents
the current tasks, while the x-axis shows the previous tasks. The value of our
similarity metric is color-coded for each cell in the heatmaps. The cells with
smaller values are concentrated on the right diagonals (in the light blue and
green colored rectangle regions), which implies that the similarity metric
values are the smallest when current tasks and previous tasks are similar.

Figure 3: Similarity Metric Results for CIFAR10 Figure 4: Similarity Metric
Results for CIFAR100 Figure 5: Similarity Metric Results for EMNIST