# Role of nucleon-nucleon correlation in transport coefficients and
gravitational-wave-driven $r$-mode instability of neutron stars

X. L. Shang  Institute of Modern Physics, Chinese Academy of Sciences, Lanzhou
730000, China  School of Physics, University of Chinese Academy of Sciences,
Beijing 100049, China  P. Wang  National Astronomical Observatories, Chinese
Academy of Sciences, Beijing 100012, China  W. Zuo  Institute of Modern
Physics, Chinese Academy of Sciences, Lanzhou 730000, China  School of
Physics, University of Chinese Academy of Sciences, Beijing 100049, China  J.
M. Dong<EMAIL_ADDRESS>Institute of Modern Physics, Chinese Academy of
Sciences, Lanzhou 730000, China  School of Physics, University of Chinese
Academy of Sciences, Beijing 100049, China

###### Abstract

The thermal conductivity and shear viscosity of dense nuclear matter, along
with the corresponding shear viscosity timescale of canonical neutron stars
(NSs), are investigated, where the effect of Fermi surface depletion (i.e.,
the $Z$-factor effect) induced by the nucleon-nucleon correlation are taken
into account. The factors which are responsible for the transport
coefficients, including the equation of state for building the stellar
structure, nucleon effective masses, in-medium cross sections, and the
$Z$-factor at Fermi surfaces, are all calculated in the framework of the
Brueckner theory. The Fermi surface depletion is found to enhance the
transport coefficients by several times at high densities, which is more
favorable to damping the gravitational-wave-driven $r$-mode instability of
NSs. Yet, the onset of the $Z$-factor-quenched neutron triplet superfluidity
provides the opposite effects, which can be much more significant than the
above mentioned $Z$-factor effect itself. Therefore, different from the
previous understanding, the nucleon shear viscosity is still smaller than the
lepton one in the superfluid NS matter at low temperatures. Accordingly, the
shear viscosity cannot stablize canonical NSs against $r$-mode oscillations
even at quite low core temperatures $10^{6}$ K.

As a class of compact objects, neutron stars (NSs) with typical mass $M\sim
1.4M_{\odot}$ and radii $R\sim 10$ km, contain extreme neutron-rich matter at
supranuclear density in their interiors. Interestingly, they have many extreme
features that cannot be produced in terrestrial laboratories, such as
extremely strong magnetic field, superstrong gravitational field, extremely
high density, superfluid matter and superprecise spin period HPY , suggesting
their importance for fundamental physics. These intriguing features have drawn
great interest for researchers of various branches of contemporary physics and
astronomy since the discovery of pulsars (rapidly rotating NSs) in 1967.

Due to the dense matter with large isospin asymmetry inside NSs, a great deal
of attention has been paid to the recent astronomical observations that can be
used to uncover the knowledge of the NS interior. For instance, the
observations of stellar cooling enables one to constrain the equation of state
(EOS) of dense matter, superfluidity and transport properties, in combination
with indispensable theoretical analysis CAS1 ; Page1 ; CAS2 ; CAS3 ; CAS4 ;
CAS5 ; CAS6 . Moreover, a rapidly rotating NS is regarded as a gravitational
wave source due to $r$-mode instability. The $r$-mode is a non-radial
oscillation mode with Coriolis force as restoring force, which leads to the
gravitational wave radiation in rapidly rotating NSs due to the Chandrasekhar-
Friedmann-Schutz instability CFS1 ; CFS2 ; CFS3  and thus prevents the NSs
from reaching their Kepler rotational frequency Kep1 ; Kep2 . The
gravitational radiation is in turn able to excite $r$ modes in NS core and
hence enhances their oscillation amplitudes, and it is particularly
interesting from the perspective of the gravitational wave observations with
ground-based facilities. The gravitational wave signal from the $r$-mode
oscillation, if detectable in the future, could help one to probe the dense
matter properties inside NSs.

The reliable knowledge about transport coefficients of dense matter is crucial
for understanding the stellar thermal evolution and $r$-mode-instability
induced gravitational radiation. The thermal conductivity which measures the
ability to conduct the heat, is an important input for modeling NS cooling
Cool1 ; Cool2 . The shear viscosity is the primary damping mechanism that
hinders the gravitational-wave-driven $r$-mode instability of rapidly rotating
NSs at low temperatures ($<10^{9}$ K) FI1979 ; CL1987 ; IV2012 . These two
transport coefficients have been calculated by several authors based on the
formulism derived by Abrikosov and Khalatnikov (AK) from the Landau kinetic
equations for a multicomponent systems AK , where the required in-medium
nucleon-nucleon cross sections is obtained by employing the correlated basis
function method and the Brueckner-Hartree-Fock (BHF) approach with realistic
nucleon-nucleon interactions Benhar2007 ; Benhar2010 ; Zhang2012 ; Baldo2013 .
In the present work, within the AK framework, we calculate the transport
coefficients by adopting the Brueckner theory with the inclusion of the effect
of Fermi surface depletion. The bulk viscosity is expected to become the
dominant dissipation mechanism for newborn NSs with rather high temperatures
($T>10^{10}$ K), and we do not consider this situation here.

It is well-known that, the momentum distribution for a perfect Fermi gas
follows a right-angle distribution at zero-temperature, namely the well-known
Fermi-Dirac distribution. Yet, owing to the short-range repulsive core and
tensor interaction (collectively referred to as short-range correlation in
some references), the system deviates from the typical profile of an ideal
degenerate Fermi gas featured by a high-momentum tail SRC11 ; SRC12 ; SRC13 ;
SRC14 , and as a result a Fermi surface depletion may appear. The $Z$-factor
measures such a Fermi surface depletion. The correlation between nucleons or
its induced $Z$-factor has far-reaching impact on many issues such as nuclear
structure Science2008 ; Science2014 , superfluidity of dense nuclear matter
Dong-SRC1 ; Dong-SRC2 ; BAL , NS cooling Dong-SRC2  and the European Muon
Collaboration effect Nature2018 ; EMC2 , highlighting its fundamental
importance in nuclear physics and NS physics. For instance, Dong et al. have
shown that the superfluid gap of $\beta$-stable neutron star matter is
strongly quenched by the $Z$ factor within the generalized BCS theory Dong-
SRC1 ; Dong-SRC2 . The neutrino emissivity for NS cooling due to direct Urca,
modified Urca processes are also reduced by the $Z$-factor, and therefore the
cooling rates of young NSs are considerably slowed Dong-SRC2 .

Figure 1: (a) Energy per particle in symmetric matter, pure neutron matter,
and $\beta$-stable matter as a function of nucleonic density from the BHF
approach. The square shows the position of calculated saturation point. (b)
Density-dependent effective mass at Fermi surfaces for three different nuclear
matter configurations.

In this work, the roles of the $Z$-factor in the thermal conductivity and
shear viscosity are clarified based on the AK formulism. The neutron triplet
superfluidity in NS core quenched by the $Z$-factor effect is introduced to
examine its effects on the viscosity of $\beta$-stable NS matter. Then we
calculate the shear viscosity timescale and gravitation-wave-driven $r$-mode
growth timescale of canonical NSs to explore whether the shear viscosity is
sufficiently strong to damp the $r$-mode instability. The required in-medium
cross sections and nucleon effective masses to calculate transport
coefficients, and the $Z$-factor at the Fermi surface, together with the EOS
to establish the NS structure, are all obtained in an unified framework, i.e.,
the Brueckner theory with AV18 two-body interaction plus a microscopic three-
body force baldo ; zuo . We should stress here that in the calculation the
exact treatment of total momentum is adopted to obtain more reliable results
shangbhf .

The $Z$-factor that measures the effect of Fermi surface depletion is given by

$Z(k)=\left[1-\frac{\partial\Sigma(k,\omega)}{\partial\omega}\right]_{\omega=\epsilon(k)}^{-1}$
(1)

with the single-particle energy $\epsilon(k)$. Where $\Sigma(k,\omega)$ is the
self-energy versus momentum $k$ and energy $\omega$. The $Z$ factor at the
Fermi surface, labeled $Z_{F}$ ($0<Z_{F}<1$), is equal to the discontinuity of
the occupation number at the Fermi surface, according to the Migdal-Luttinger
theorem Migdal1960 . Once the nucleon-nucleon correlation is included, the
nucleon momentum distribution is given as

$\displaystyle n(k)=\int\frac{d\omega}{2\pi}S(k,\omega)n^{0}(\omega)$ (2)

at finite temperature $T$ KG1962 , where $\omega$ is the energy.
$n^{0}(\omega)=1/[1+\exp(\frac{\omega-\mu}{k_{B}T})]$ is the well-known Fermi-
Dirac distribution function under temperature $T$ and chemical potential
$\mu$. The spectral function $S(k,\omega)$ can be expressed as baldo

$\displaystyle S(k,\omega)\approx Z_{F}\delta(\omega-\epsilon(k_{F})),k\approx
k_{F},$ (3)

when momentum $k$ is extremely close to the Fermi momentum $k_{F}$.
Consequently, the momentum distribution near the Fermi surface is approximated
by Dong-SRC2

$\displaystyle n(x)\approx Z_{F}n^{0}(x),k\approx k_{F},$ (4)

with $x=(\epsilon(k)-\mu)/(k_{B}T)$. Hereafter we take $x$ as variable in the
Fermi-Dirac distribution for convenience. We stress that this approximation is
only valid when $k$ is extremely close to the Fermi surface. The nucleon-
nucleon correlation quenches the occupation probability by a factor $Z_{F}$ at
Fermi surface $k_{F}$, and thus it hinders particle transitions around the
Fermi surface.

To embody the effects of nucleonic Fermi surface depletion in the calculation
of the kinetic coefficients, we extend the Landau kinetic equation by
including the $Z$-factor in the collision integral. In the AK framework, at
temperature $T$, the collision integral without the $Z$-factor effect takes
the form of PRB35

$\displaystyle I_{1i}^{0}$ $\displaystyle=$
$\displaystyle-\frac{m_{i}^{*}k_{B}^{2}T^{2}}{8\pi^{4}\hbar^{6}}\int\int
dx_{2}dx_{3}n^{0}(x_{1})n^{0}(x_{2})[1-n^{0}(x_{3})]$ (5)
$\displaystyle\times$
$\displaystyle[1-n^{0}(x_{1}+x_{2}-x_{3})]\sum_{j}m_{j}^{*2}\int\int\frac{d\Omega}{4\pi}\frac{d\phi_{2}}{2\pi}$
$\displaystyle\times$
$\displaystyle\frac{W_{ij}(\theta,\phi)\beta_{ij}}{1+\delta_{ij}}[\psi(\bm{p}_{1})+\psi(\bm{p}_{2})-\psi(\bm{p}_{3})-\psi(\bm{p}_{4})],$

where $m^{*}$ is the effective mass of nucleon $i$ or $j$. And the small
quantities $\psi(\bm{p})$ measures the departure from equilibrium state. Here
the nucleon-nucleon scattering is limited to the Fermi surface. For
convenience, one can assume $1$ and $3$ ($2$ and $4$) are the same component,
i.e., $|\bm{p}_{1}|=|\bm{p}_{3}|=p_{i}$ ($|\bm{p}_{2}|=|\bm{p}_{4}|=p_{j}$).
And the transition probability $W_{ij}$ from two quasiparticle state
$|\bm{p}_{1},\bm{p}_{2}\rangle$ to state $|\bm{p}_{3},\bm{p}_{4}\rangle$,
depends only on $\theta$ and $\phi$ ($d\Omega=\sin\theta d\theta d\phi$),
where $\theta$ is the angle between $\bm{p}_{1}$ and $\bm{p}_{2}$, and $\phi$
is the angle between the $\bm{p}_{1}$-$\bm{p}_{2}$ plane and
$\bm{p}_{3}$-$\bm{p}_{4}$ plane.
$\beta_{ij}=p_{j}/(p_{i}^{2}+p_{j}^{2}+2p_{i}p_{j}\cos\theta)^{1/2}$ reduces
to $[2\cos(\theta/2)]^{-1}$ for $i=j$. $\phi_{2}$ is the azimuthal angle of
$\bm{p}_{2}$ with respect to $\bm{p}_{1}$. The factor $(1+\delta_{ij})^{-1}$
takes into account double counting of the final states in the case of like
particles.

Due to the temperature $T$ we discussed is several orders of magnitude lower
than the nucleonic Fermi temperatures (the nucleons are strong degenerate),
the main contribution to the above integral comes from the very narrow regions
of momentum space near the corresponding Fermi surfaces $k_{F}$, just as the
calculation of neutrino emissivity in Ref. Y2001 . If the $Z$-factor effect is
included, in the above collision integral, $1-n^{0}(x)$ (and $n^{0}(x)$)
representing the unoccupied (and occupied) state due to the temperature,
should be replaced by $n(x)|_{T=0}-n(x)=Z_{F}[1-n^{0}(x)]$ (and
$Z_{F}n^{0}(x)$) when the $Z$-factor effect is included. The collision
integral is just attribute to thermal excitations of particles located in a
very narrow region of $\sim k_{B}T$ close to their Fermi surfaces, and the
state with $|\epsilon(k)-\epsilon(k_{F})|\gg k_{B}T$ plays no role for the
collision integral because the thermal energy $k_{B}T$ is too low to excite
those states. Therefore, the high momentum tail makes no contribution to the
collision integral, just as the influence of the Fermi surface depletion on
neutrino emissivity processes discussed in detail in Ref. Dong-SRC2 .
Consequently, the collision integral turns into

$\displaystyle I_{1i}$ $\displaystyle=$
$\displaystyle-\sum_{j}\frac{Z_{Fi}^{2}Z_{Fj}^{2}m_{i}^{*}m_{j}^{*2}k_{B}^{2}T^{2}}{8\pi^{4}\hbar^{6}}\int\int
dx_{2}dx_{3}n^{0}(x_{1})n^{0}(x_{2})$ (6) $\displaystyle\times$
$\displaystyle[1-n^{0}(x_{3})][1-n^{0}(x_{1}+x_{2}-x_{3})]\int\int\frac{d\Omega}{4\pi}\frac{d\phi_{2}}{2\pi}$
$\displaystyle\times$
$\displaystyle\frac{W_{ij}\beta_{ij}}{1+\delta_{ij}}[\psi(\bm{p}_{1})+\psi(\bm{p}_{2})-\psi(\bm{p}_{3})-\psi(\bm{p}_{4})].$

Moreover, the driving term of the Landau kinetic equation, which is
proportional to $\frac{\partial n}{\partial x}$ at equilibrium state, provides
a $Z_{F}$ as well. Therefore, one can include the $Z$-factor effect in the
calculation of the transport coefficients by adopting $Z_{F}$ both in the
collision integral and the driving term by following the derivations in Ref.
PRB35 . For example, the collision integral reduces to a simple formula of
$I_{1i}=Z_{F}^{4}I_{1i}^{0}$ for pure neutron matter. One should note that the
momentum (energy) flux corresponding to the the shear viscosity (thermal
conductivity) also includes $\frac{\partial n}{\partial x}$, Consequently, the
shear viscosity (thermal conductivity) is given by $\eta=\eta^{0}/Z_{F}^{2}$
($\kappa=\kappa^{0}/Z_{F}^{2}$) for pure neutron matter, where $\eta^{0}$
($\kappa^{0}$) is the corresponding transport coefficient without the
inclusion of the $Z$-factor effect.

Within the BHF approach, the EOSs of symmetric nuclear matter ($\beta=0$),
pure neutron matter ($\beta=1$), and $\beta$-stable matter, where
$\beta=(\rho_{n}-\rho_{p})/(\rho_{n}+\rho_{p})$ denotes the isospin asymmetry
with the neutron (proton) number densities $\rho_{n}$ ($\rho_{p}$), are
displayed in Fig. 1(a). The solid square shows the calculated saturation point
of symmetric matter which is marginally in agreement with the empirical value
due to the introducing of three-body force. The proton fraction in
$\beta$-stable matter is determined by the density-dependent symmetry energy,
i.e., the isospin-dependent part of the EOS. The EOSs for pure neutron matter
and $\beta$-stable matter show a distinct difference that becomes more and
more visible at high densities, indicating the non-negligible proton fraction
in NS matter. The NS interior is assumed to be composed of nucleons, electrons
and possible muons. With the conditions of electric neutrality and
$\beta$-equilibrium, the fractions of leptons (electrons and muons as
degenerate ideal gas) and their contributions to the energy density
$\varepsilon(\rho)$ and pressure $p(\rho)$ can be determined uniquely. With
the obtained $\varepsilon(\rho)$ and $p(\rho)$ of the core matter and the EOS
from Baym, Pethick, and Sutherland (BPS) BPS  for crust matter as inputs, the
stellar structure, e.g., the density profile $\rho(r)$ of a static and
spherically symmetric NS, is achieved by solving the Tolman-Oppenheimer-Volkov
(TOV) equation. The established stellar structure is essential for the final
estimation of the shear viscosity timescale and $r$-mode growth time scale of
NSs.

The nucleonic effective mass $m^{*}$ is defined from the single-particle
energy $\epsilon(p)$ by the relation
$m^{\ast}=k_{F}\left(\partial\epsilon(k)/\partial k\right)^{-1}|_{k=k_{F}}$.
It reduces the density of states at the Fermi surface with respect to non-
interacting Fermi gas since it is usually smaller than the free mass. As Ref.
Baldo2013 ; shangems , the rearrangement contribution of three-body force is
not included here. The calculated effective mass with the BHF approximation
are presented in Fig. 1(b). The neutron effective mass of pure neutron matter
is not much different from that of $\beta$-stable matter, but is distinctly
larger than that of symmetric matter at the same density.

We calculate the in-medium differential sections within the BHF method for
symmetric matter, pure neutron matter and $\beta$-stable matter, taking the
neutron-neutron scattering at density of $\rho=0.34$ fm-3 (twice the
saturation density) as an example, as shown in Fig. 2. The free-space cross
section is also shown for comparison. The in-medium effect leads to a
noticeable suppression of the cross sections, as other calculations within
microscopic nuclear many-body approaches, suggesting the important role of the
medium effect. Our calculated differential cross sections as functions of
center-of-mass scattering angle (and also the total cross sections versus
center-of-mass energy $E_{\text{c.m.}}$) have the same shape as that in Ref.
Baldo2013  for density $\rho=0.35$ fm-3, although different three-body forces
are used. We would like to stress that, the inclusion of the three-body force
increases the cross section at high $E_{\text{c.m.}}$, which is in agreement
with the conclusion of Ref. Baldo2013 , but disagrees with the results in Ref.
Zhang2012 ; Zhang2007 .

Figure 2: (a) Differential cross sections of neutron-neutron scattering in
symmetric matter, pure neutron matter, and $\beta$-stable matter, taking
$\rho=0.34$ fm-3 and center-of-mass energy $E_{\text{c.m.}}=75$ MeV as an
example. (b) The corresponding total cross sections versus $E_{\text{c.m.}}$.

Figure 3 exhibits the calculated $Z_{F}$ at Fermi surfaces for three different
nuclear matter configurations by employing the Brueckner theory where the
self-energy is expanded to the 2nd-order, i.e.,
$\Sigma=\Sigma_{1}+\Sigma_{2}$. The momentum distribution featured by a high
momentum tail and vacant position below the Fermi surface, is illustrated in
the inset. The behavior of $Z_{F}$ for symmetric matter is consistent with the
result in Refs. Dong-SRC2 ; shangzz . The $Z$-factor is caused by the short-
range repulsion core and tensor force. The tensor force is dominant at low
densities while the short-range repulsion is dominant at high densities. The
nonmonotonic behavior of $Z_{F}$ for symmetric matter and $\beta$-stable
matter displayed in Fig. 3 is exactly the results of competition between these
two effects, and the $Z_{F}$ is small both at very low and very high
densities. On the other hand, the $Z_{F}$ exhibits a strong isospin
dependence. At a given total nucleon density, the $Z_{F}$ of symmetric matter
is smaller obviously than that of pure neutron matter, that is, the
correlation in the former is stronger than that in the later, because the
${}^{3}SD_{1}$ tensor interaction component between neutrons and protons is
quite strong in symmetric matter but is completely absent in pure neutron
matter. Namely the pure neutron matter is much closer to the ideal degenerate
Fermi gas, as pointed out in Ref. Science2014 . The results displayed in Fig.
3 will be applied in the following calculations of transport coefficients.

Figure 3: Density-dependent $Z$-factor at Fermi surfaces in symmetric matter,
pure neutron matter, and $\beta$-stable matter. The inset presents a schematic
illustration of the Fermi surface depletion induced by the nucleon-nucleon
correlation.

When combining all the results that have been discussed above, we can now
compute the density-dependent shear viscosity under various temperatures
stemming from nucleon-nucleon collisions. The phase space is quenched in Eq.
(2) because of the depletion of Fermi surface, and therefore the thermal
conductivity $\kappa$ and shear viscosity $\eta$ are increased. The calculated
temperature-independent combinations $\kappa T$ and $\eta T^{2}$ versus
density are plotted in Fig. 4, respectively, without and with the inclusion of
the $Z$-factor effect. The lepton (electron and muon) shear viscosity
$\eta_{e\mu}$ and thermal conductivity $\kappa_{e\mu}$ mediated by collisions
of leptons with charged particles in electrically neutral NS matter, are taken
from Ref. Shternin2008 . Since the nucleon shear viscosity $\eta_{N}$ is
mediated by nucleon-nucleon collisions via strong nuclear force, the
$\eta_{N}$ and $\eta_{e\mu}$ can be treated independently. Yet, the
$\eta_{e\mu}$ ($\kappa_{e\mu}$) has different temperature-dependent behavior
as $\eta_{N}$ ($\kappa_{N}$). So here we show three cases: $T=10^{7}$,
$10^{8}$, and $10^{9}$ K. The relation between $\eta_{N}$ and $\eta_{e\mu}$ is
temperature dependent, that is, $\eta_{N}$ becomes more and more important as
temperature decreases. The proton contribution to the shear viscosity can be
neglected safely since the proton contribution is just $15\%$ even at high
density of $\rho=0.6$ fm-3.

Figure 4: Thermal conductivity $\kappa$ (upper panel) and shear viscosity
$\eta$ (lower panel) of nucleons and leptons as a function of density in
symmetric matter, pure neutron matter, and $\beta$-stable matter. The
nucleonic $\kappa_{N}$ and $\eta_{N}$ are calculated with the help of BHF
approach without and with the inclusion of $Z$-factors.

The $Z$-factor effect enhances the nucleonic $\kappa$ and $\eta$ for the three
nuclear matter configurations, in particular at high densities. For example,
at the density of $\rho=0.6$ fm-3, the $\kappa_{N}$ and $\eta_{N}$ can be
enhanced by about three to four times by the $Z$-factor effect. The nucleonic
thermal conductivity is much larger than the lepton ones for all densities of
NS matter and temperatures of interest. Yet, the situation is different for
shear viscosity. Without the $Z$-factor effect ($Z=1$), the primary
contribution to the shear viscosity $\eta=\eta_{N}+\eta_{e\mu}$ comes from the
lepton scattering which is just exceeds by nucleon scattering at low
densities, in agreement with the conclusion of Ref. Baldo2013 . Once the
$Z$-factor is taken into account, the $\eta_{N}$ and $\eta_{e\mu}$ become
comparable at intermediate densities, and the $\eta_{N}$ is about four times
larger than $\eta_{e\mu}$ at crust-core transition density $\rho\approx 0.08$
fm-3.

It is widely believed that superfluidity plays a crucial role in NS dynamics,
such as NS cooling and the observed pulsar glitch. It draw wide attention in
communities of nuclear physics and NS physics in particular after the rapid
cooling of the NS in Cassiopeia A was observed. The strong nuclear force
provides several attractive channels between nucleons in which superfluidity
is possible sh1 ; sh2 ; sh3 . The neutrons dripped out from the neutron-rich
nuclei in NS inner crust, are expected to be paired in a ${}^{1}S_{0}$ singlet
state with energy gap of $\sim 1.5$ MeV Lombardo2001 . The proton gas is so
dilute that the proton ${}^{1}S_{0}$ superconductivity (superfluidity of
charged particles) may survive until deep inside the star but the neutron
${}^{1}S_{0}$ superfluidity vanishes because the nuclear interaction in the
${}^{1}S_{0}$ channel becomes repulsive at short distances for high neutron
density. Nevertheless, at high density, neutron-neutron coupling in the
${}^{3}PF_{2}$ anisotropic pairing state could appear owing to the attractive
component of the nuclear interaction in this coupling channel. The coupling
between the ${}^{3}P_{2}$ and ${}^{3}F_{2}$ states is attributed to tensor
force. This neutron ${}^{3}PF_{2}$ superfluidity is of great interest because
it was employed to explain the rapid cooling of the NS in Cassiopeia A Page1 .
However, the superfluidity may reduced significantly by the nucleon-nucleon
correlation Dong-SRC1 ; Dong-SRC2 ; shangbcs . By performing fittings with
several parameters, the density-dependent gap for the neutron ${}^{3}PF_{2}$
superfluidity of $\beta$-stable matter is given by Dong2020

$\displaystyle\Delta_{n}(\rho)$ $\displaystyle=$
$\displaystyle(0.943\rho-0.050)\exp\left[-\left(\frac{\rho}{0.177}\right)^{1.665}\right],$
(7)

with a peak value of about 0.04 MeV at $\rho=0.17$ fm-3. The proton
${}^{1}S_{0}$ superfluid gap exists in a rather narrow region and is much
smaller than the neutron ${}^{3}PF_{2}$ superfluid gap as stressed in Dong2020
. In addition, the proton fraction is much smaller than the neutron one for
$\beta$-stable NS matter. Therefore, we do not consider it in the present
work. Here we only focus on the effects of neutron triplet superfluidity on
shear viscosity. As mentioned in Ref. Andersson2005 , we introduce a
suppression factor to estimate the nucleon shear viscosity via
$\eta^{(\text{SF})}_{N}\approx R_{n}\eta_{N}$, where $R_{n}$ is written as
Andersson2005

$\displaystyle R_{n}$ $\displaystyle\simeq$
$\displaystyle\left[0.9543+\sqrt{0.04569^{2}+(0.6971y)^{2}}\right]^{3}$ (8)
$\displaystyle\cdot\exp\left[0.1148-\sqrt{0.1148^{2}+4y^{2}}\right]$

with $y=\Delta(T)/T$. $\Delta(T)$ is the temperature-dependent energy gap, and
the critical temperature is $T_{c}=0.57\Delta(T=0)$. The $\eta_{N}$ due to
neutron-neutron scattering drops exponentially because of sharp decrease of
the number of momentum carriers near the Fermi surface.

Figure 5: Shear viscosity stemming from nucleon-nucleon scattering as a
function of density in $\beta$-stable matter with the inclusion of neutron
triplet superfluidity.

The $\eta T^{2}$ of each component as a function of density under different
temperatures $T$ in the presence of neutron ${}^{3}PF_{2}$ superfluidity are
displayed in Fig. 5. If the core temperatures of NSs are higher than $\sim
2\times 10^{8}$ K, the neutron ${}^{3}PF_{2}$ superfluidity disappears. The
neutrons in stellar core becomes superfluid as soon as the NS cools below the
critical temperatures, and accordingly the neutron-neutron scattering is
strongly depressed and the main contribution to the shear viscosity comes from
electron scattering processes. As a result, the $Z$-factor-quenched superfluid
effect plays an opposite role compared with the $Z$-factor effect itself, and
intriguingly it can be much more significant. For instance, at the temperature
$T=5\times 10^{7}$ K, the nucleon shear viscosity $\eta_{N}$ is reduced by
about six orders of magnitude at $\rho=0.17$ fm-3, and this suppression is
stronger at lower temperatures. It was concluded in other references such as
IV2012  that, at low temperatures $T<10^{7}$K, the contribution to the shear
viscosity from the neutron scattering is more important than the lepton
scattering. However, the $\eta_{e\mu}$ is still larger than $\eta_{N}$ in the
presence of such neutron triplet superfluidity. For example, at temperature
$T=10^{7}$ K, the $\eta_{N}$ of the nucleon scattering can be neglected at
density $\rho<0.5$ fm-3 in superfluid matter.

Table 1: The calculated shear viscosity time scale $\tau_{\eta}$, compared with gravitation-radiation-driven $r$-mode time scale $\tau_{\text{GW}}=196$ s for canonical neutron stars rotating at 716 Hz. The results with and without the neutron triplet superfluidity (SF) are listed, and the weights of the nucleon contribution are present in the brackets. Temperature (K) |  $\tau_{\eta}^{\text{nSF}}$(s) |  $\tau_{\eta}^{\text{SF}}$(s)
---|---|---
$10^{6}$ | 402 (66%) | 1200 (0%)
$10^{7}$ |  $2.99\times 10^{4}$ (50%) |  $4.38\times 10^{4}$ (9%)
$10^{8}$ |  $2.05\times 10^{6}$ (34%) |  $2.26\times 10^{6}$ (27%)
$10^{9}$ |  $1.38\times 10^{8}$ (23%) |  $1.07\times 10^{8}$ (23%)

After the stellar structure is established by solving the TOV equation with
the BHF EOS as an input, the time scales of shear viscosity and of
gravitation-radiation-driven growth of $r$-mode for $1.4M_{\odot}$ canonical
NSs are calculated. The overall time scale is
$1/\tau=-1/\tau_{\text{GW}}+1/\tau_{\eta}$, and if angular-velocity-dependent
$\tau_{\text{GW}}$ is smaller than temperature-dependent $\tau_{\eta}$, the
$r$-mode amplitude will exponentially grow, resulting in $r$-mode instability.
The equation of $1/\tau=0$ determines the critical frequency in frequency-
temperature space, above which is the usually referred to as the $r$-mode
instability window Andersson2001 ; Haskell2015 .

Table I lists the calculated shear viscosity $\tau_{\eta}$ and $r$-mode growth
time scale $\tau_{\text{GW}}$ for canonical NSs. In non-superfluid NSs, the
nucleon-nucleon scattering is indeed the dominant dissipation mechanism at low
temperatures. If the superfluid effect is included, the situation is
completely opposite. The $\eta_{N}$ becomes less and less important and even
negligible as temperature decreases. The $\tau_{\eta}$ is enlarged because of
the superfluid effect, indicating weaker shear viscosity damping. It is
generally believed that the $r$-mode instability limits the rotating angular
velocity of accretion millisecond pulsars. At present, the fastest spinning
pulsar is PSR J1748-2446ad spinning at 716 Hz PSR716 , and its corresponding
$r$-mode growth time scale $\tau_{\text{GW}}$ is 196 s if
$M_{\text{TOV}}=1.4M_{\odot}$ is assumed. At low temperatures $T=10^{6}$ K,
the shear viscosity $\tau_{\eta}$ is 402 Hz for nonsuperfluid NS core matter
which is comparable with the $\tau_{\text{GW}}$, and the weight of nucleonic
contribution is as large as $66\%$. However, if the superfluidity is taken
into account, the nucleon-nucleon scattering does not contribute to the
$\tau_{\eta}$ at such low temperature, and the $\tau_{\eta}$ is much larger
than the $\tau_{\text{GW}}$ and hence the shear viscosity is not much help to
damp the $r$-mode instability. Some authors proposed that the viscous
dissipation at the viscous boundary layer of perfectly rigid crust and fluid
core is the primary damping mechanism. However, it is questioned if the core-
crust boundary is defined by a continuous transition from non-uniform matter
to uniform matter through "nuclear pasta" phases PP1998  and consequently the
viscous boundary layer is smeared out Gearheart .

Figure 6: The calculated $r$-mode instability critical curves without the
superfluidity (SF) and $Z$-factor, with Z-factor only, with both the
$Z$-factor and neutron triplet superfluidity, are shown for comparison.

In order to more clearly reveal the roles of the $Z$-factor and superfluid
effects on the $r$-mode instability, the calculated $r$-mode instability
critical curves are presented in Fig. 6. The $Z$-factor effect is conducive to
damping the gravitational-wave-driven $r$-mode growth of NSs, in particular at
low temperatures. However, the neutron triplet superfluidity plays an opposite
role and is more significant. At temperatures higher than $\sim 10^{8}$ K,
both of the two effects are weak, which is because the neutron-neutron
scattering contributes secondary to shear viscosity and the superfluidity is
almost vanishes at such temperatures. The core temperature of NSs in low mass
X-ray binaries are estimated to be $(1\sim 5)\times 10^{8}$ K Ho2012  and
$10^{7}\sim 10^{8}$ K if the direct Urca process opens Dong2020 , therefore
the shear viscosity cannot be expected to stablize NSs against $r$-mode
oscillations in practical situation. Additional damping mechanisms perhaps is
required.

In summary, the $Z$-factor effects on the thermal conductivity and shear
viscosity have been calculated based on the AK framework, where the $Z$-factor
at Fermi surfaces ($Z_{F}$), the in-medium cross sections, nucleon effective
masses, and the EOS of NS matter, are calculated by using the Brueckner theory
with the two-body AV18 interaction plus microscopic three-body force. The
nucleon-nucleon correlations, induced by the effects of short-range repulsion
and tensor component of nuclear force, gives rise to the Fermi surface
depletion, i.e., the $Z$-factor effect. The calculated $Z_{F}$ of neutrons and
protons at Fermi surfaces presents a strong isospin dependence due to the
strong neutron-proton ${}^{3}SD_{1}$ tensor interaction. The two transport
coefficients are enlarged by several times for symmetric matter, pure neutron
matter and $\beta$-stable matter. The nucleonic thermal conductivity
$\kappa_{N}$ is much more important than lepton ones for different densities
and temperatures that we considered here, whether or not this $Z$-factor
effect is included. As temperature decreases, the nucleon shear viscosity
$\eta_{N}$ becomes more and more important with respect to the lepton
contribution $\eta_{e\mu}$. If we take into account the $Z$-factor effect, the
$\eta_{N}$ may become comparable with $\eta_{e\mu}$ at intermediate densities,
and larger than $\eta_{e\mu}$ at low densities. As concluded in the previous
works Dong-SRC1 ; Dong-SRC2 , the $Z$-factor effect suppresses the proton
${}^{1}S_{0}$ and neutron ${}^{3}PF_{2}$ superfluidity strongly, and the
proton ${}^{1}S_{0}$ superfluidity almost vanishes. Contrary to the role of
$Z$-factor itself, neutron superfluidity is able to reduce the shear viscosity
significantly (by several orders of magnitude) when the temperature drops
below the critical temperature. As a result, the contribution to the shear
viscosity from the lepton scattering is still more important than that from
the nucleon scattering at low temperature for the densities of interest in
superfluid matter. Finally, the shear viscosity time scales $\tau_{\eta}$
along with the time scales $\tau_{\text{GW}}$ of $r$-mode growth due to the
emission of gravitational waves for canonical NSs are calculated. At low
temperatures, the nucleon-nucleon scattering indeed contributes mainly to the
shear viscosity time scale $\tau_{\eta}$. However, if the $Z$-factor-quenched
superfluidity is present, it is less important and even negligible. In a word,
the appearance of superfluidity is not favorable to damping the $r$-mode
instability of NSs. The calculated $\tau_{\eta}$ is much larger than the
$\tau_{\text{GW}}$ and hence the shear viscosity is not able to damp the
$r$-mode instability even for very cold NSs with core temperature of $10^{6}$
K. The present work stretches our understanding of the $r$-mode instability of
pulsar physics.

This work was supported by the National Natural Science Foundation of China
(Grants No. 11775276, 11975282), the Strategic Priority Research Program of
Chinese Academy of Sciences (Grant No. XDB34000000), the Youth Innovation
Promotion Association of Chinese Academy of Sciences (Grant No. Y201871), the
Continuous Basic Scientific Research Project (Grant No. WDJC-2019-13), the
Leading Innovation Project (Grant No. LC 192209000701), and the Continuous
Basic Scientific Research Project (Grant No. WDJC-2019-13).

## References

  * (1) P. Haensel, A. Y. Potekhin, D. G. Yakovlev, Neutron Stars 1, (Springer, 2006).
  * (2) P. S. Shternin, et al., Mon. Not. Roy. Astron. Soc. 412 (2011) L108.
  * (3) D. Page, M. Prakash, J. M. Lattimer, A. W. Steiner, Phys. Rev. Lett. 106 (2011) 081101.
  * (4) A. Sedrakian, Astron. Astrophys. 555 (2013) L10.
  * (5) D. Blaschke, H. Grigorian, D. N. Voskresensky, F. Weber, Phys. Rev. C 85 (2012) 022802(R).
  * (6) W. G. Newton, K. Murphy, J. Hooker, B.-A. Li, Astrophys. J. 779 (2013) L4.
  * (7) A. Bonanno, M. Baldo, G. F. Burgio, V. Urpin, Astron. Astrophys. 561 (2014) L5.
  * (8) W. C. G. Ho, K. G. Elshamouty, C. O. Heinke, A. Y. Potekhin, Phys. Rev. C 91 (2015) 015806.
  * (9) S. Chandrasekhar, Astrophys. J. 161 (1970) 561.
  * (10) J. L. Friedmann, B. F. Schutz, Astrophys. J. 221 (1978) 937; 222 (1978) 281.
  * (11) L. Lindblom, B. J. Owen, S. M. Morsink, Phys. Rev. Lett. 80 (1998) 4843\.
  * (12) L. Bildsten, Astrophys. J. 501 (1998) L89.
  * (13) N. Andersson, K. D. Kokkotas, N. Stergioulas, Astrophys. J. 516 (1999) 307.
  * (14) D. Page, U. Geppert, F. Weber, Nucl. Phys. A 777 (2006) 497.
  * (15) D. G. Yakovlev, C. J. Pethick, Annu. Rev. Astron. Astrophys. 42 (2004) 169.
  * (16) E. Flowers, N. Itoh, Astrophys. J. 230 (1979) 847.
  * (17) C. Cutler, L. Lindblom, Astrophys. J. 314 (1987) 234.
  * (18) I. Vidana, Phys. Rev. C 85 (2012) 045808.
  * (19) A. A. Abrikosov, I. M. Khalatnikov, Sov. Phys. JETP 5 (1957) 887; Rep. Prog. Phys. 22 (1959) 329.
  * (20) O. Benhar, M. Valli, Phys. Rev. Lett. 99 (2007) 232501.
  * (21) O. Benhar, A. Polls, M. Valli, I. Vidana, Phys. Rev. C 81 (2010) 024305\.
  * (22) H. F. Zhang, U. Lombardo, W. Zuo, Phys. Rev. C 82 (2010) 015805.
  * (23) P. S. Shternin, M. Baldo, P. Haensel, Phys. Rev. C 88 (2013) 065803.
  * (24) J. P. Jeukenne, A. Lejeune, C. Mahaux, Phys. Rep. 25 (1976) 83.
  * (25) A. Ramos, A. Polls, W. H. Dickhoff, Nucl. Phys. A 503 (1989) 1.
  * (26) B. E. Vonderfecht, W. H. Dickhoff, A. Polls, A. Ramos, Nucl. Phys. A 555 (1993) 1.
  * (27) P. Yin, J. Dong, W. Zuo, Chin. Phys. C 41 (2017) 114102.
  * (28) R. Subedi, et al., Science 320 (2008) 1476.
  * (29) O. Hen, et al., Science 346 (2014) 614.
  * (30) J. M. Dong, U. Lombardo, W. Zuo, Phys. Rev. C 87 (2013) 062801(R).
  * (31) J. M. Dong, U. Lombardo, H. F. Zhang, W. Zuo, Astrophys. J. 817 (2016) 6.
  * (32) Bao-An Li, Bao-Jun Cai, Lie-Wen Chen, Jun Xu, Prog. Part. Nucl. Phys. 99 (2018) 29.
  * (33) O. Hen, G. A. Miller, E. Piasetzky, L. B. Weinstein, Rev. Mod. Phys. 89 (2017) 045002.
  * (34) The CLAS Collaboration, Nature 560 (2018) 617.
  * (35) M. Baldo, I. Bombaci, G. Giansiracusa, U. Lombardo, C. Mahaux, and R. Sartor, Phys. Rev. C 41 (1990) 1748 ; Nucl. Phys. A 545 (1992) 741\.
  * (36) W. Zuo, I. Bombaci, U. Lombardo, Phys. Rev. C 60 (1999) 024605.
  * (37) X. L. Shang, J. M. Dong, W. Zuo, P. Yin, U. Lombardo, (unpublished).
  * (38) A. B. Migdal, Sov. Phys. JETP 5 (1957) 333; J. M. Luttinger, Phys. Rev. 119 (1960) 1153.
  * (39) L. P. Kadanoff, G. Baym, Quantum Statistical Mechanics, (New York, 1962).
  * (40) R. H. Anderson, C. J. Pcthick, and K. F. Quader, Phys. Rev. B 35 (4) (1987) 1620.
  * (41) D. G. Yakovlev, A. D. Kaminker, O. Y. Gnedin, P. Haensel, Phys. Rep. 354 (2001) 1.
  * (42) G. Baym, C. J. Pethick, P. Sutherland, Astrophys. J. 170 (1971) 299; G. Baym, H. A. Bethe, C. J. Pethick, Nucl. Phys. A175 (1971) 225.
  * (43) X. L. Shang, A. Li, Z. Q. Miao, G. F. Burgio, H. J. Schulze, Phys. Rev. C 101 (2020) 065801.
  * (44) H. F. Zhang, Z. H. Li, U. Lombardo, P. Y. Luo, F. Sammarruca, and W. Zuo, Phys. Rev. C (2007) 054001.
  * (45) Z. X. Yang, X. L. Shang, G. C. Yong, W. Zuo, Y. Gao, Phys. Rev. C 100 (2019) 054325.
  * (46) P. S. Shternin, D. G. Yakovlev, Phys. Rev. D 78 (2008) 063006.
  * (47) D. J. Dean and M. Hjorth-Jensen, Rev. Mod. Phys. 75 (2003) 607.
  * (48) S. Frauendorf and A. O. Macchiavelli, Prog. Part. Nucl. Phys. 78 (2014) 24.
  * (49) X. L. Shang, W. Zuo, Phys. Rev. C 88 (2013) 025806.
  * (50) U. Lombardo, H.-J. Schulze, Physics of Neutron Star Interiors, edited by D. Blaschke, N. K. Glendenning, and A. Sedrakian, Lecture Notes in Physics Vol. 578, (Springer-Verlag, Berlin and Heidelberg, 2001), pp. 30¨C54.
  * (51) X. H. Fan, X. L. Shang, J. M. Dong, W. Zuo, Phys. Rev. C 99 (2019) 065804.
  * (52) J. M. Dong, (unpublished).
  * (53) N. Andersson, G. L. Comer, K. Glampedakis, Nucl. Phys. A 763 (2005) 212\.
  * (54) N. Andersson, K. D. Kokkotas, Int. J. Mod. Phys. D 10 (2001) 381.
  * (55) B. Haskell, Int. J. Mod. Phys. E 24 (2015) 1541007.
  * (56) J. W. T. Hessels, et al., Science 311 (2006) 1901.
  * (57) C. Pethick, A. Y. Potekhin, Phys. Lett. B 427 (1998) 7.
  * (58) M. Gearheart, W. G. Newton, J. Hooker, B. Li, Mon. Not. Roy. Astron. Soc. 418 (2011) 2343.
  * (59) W. C. G. Ho, N. Andersson, B. Haskell, Phys. Rev. Lett. 107 (2011) 101101\.


# Disentangled Continual Learning: Separating Memory Edits from Model Updates

Sebastian Dziadzio
University of Tübingen
<EMAIL_ADDRESS>
Çağatay Yıldız
University of Tübingen
Gido M. van de Ven
KU Leuven  Tomasz Trzciński
IDEAS NCBR, Warsaw University of Technology, Tooploox  Tinne Tuytelaars
KU Leuven  Matthias Bethge
University of Tübingen

###### Abstract

The ability of machine learning systems to learn continually is hindered by
catastrophic forgetting, the tendency of neural networks to overwrite existing
knowledge when learning a new task. Existing continual learning methods
alleviate this problem through regularisation, parameter isolation, or
rehearsal, and are typically evaluated on benchmarks consisting of a handful
of tasks. We propose a novel conceptual approach to continual classification
that aims to disentangle class-specific information that needs to be memorised
from the class-agnostic knowledge that encapsulates generalization. We store
the former in a buffer that can be easily pruned or updated when new
categories arrive, while the latter is represented with a neural network that
generalizes across tasks. We show that the class-agnostic network does not
suffer from catastrophic forgetting and by leveraging it to perform
classification, we improve accuracy on past tasks over time. In addition, our
approach supports open-set classification and one-shot generalization. To test
our conceptual framework, we introduce Infinite dSprites, a tool for creating
continual classification and disentanglement benchmarks of arbitrary length
with full control over generative factors. We show that over a sufficiently
long time horizon all major types of continual learning methods break down,
while our approach enables continual learning over hundreds of tasks with
explicit control over memorization and forgetting.

##  1 Introduction

A machine learning system designed for continual learning must not only adapt
to the current task, but also improve its performance on past tasks and build
representations that facilitate the learning of future tasks. The latter two
requirements are known as _backward_ and _forward_ transfer. The path to
meeting these requirements is obstructed by _catastrophic forgetting_ , the
inability to preserve existing knowledge upon learning new information.

As noted in early studies [24], catastrophic forgetting is caused by
_destructive model updates_ , where adjustments to model parameters, made
through gradient descent, focus solely on the current task’s objective and can
potentially impair performance on past tasks. To mitigate this issue,
continual learning methods employ strategies such as (i) regularization, which
aims to preserve existing knowledge by limiting the plasticity of selected
network weights [15, 17, 26, 36], (ii) parameter isolation or dynamic
architectures, which effectively solve each task with a dedicated model [33,
6], or (iii) replay, which augments the training data with stored samples from
past tasks [30, 4, 12, 32].

Most continual learning methods are evaluated on image classification
benchmarks in which a discriminative model is transferred across tasks that
typically involve disjoint sets of classes. We argue that this purely
discriminative learning framework is not conducive to positive forward or
backward transfer. Supervised classification networks tend to preserve only
the features that are relevant for predicting the output labels in the
training data [35, 11]. In a continual learning setting, these features
transfer poorly to future tasks with a completely different set of labels.
Conversely, gradient updates with respect to current task’s objective do not
encourage preserving features relevant to previous tasks.

Figure 1: Schematic illustration of our disentangled learning framework, which
consists of three main modules transferred across tasks: (i) an equivariant
network that estimates the parameters of the normalization transformation,
(ii) a normalization module that outputs a standardized version of the input
image based on the predicted parameters, and (iii) a buffer that stores class-
specific exemplars. During training, we minimize the loss between the output
of the normalization module and the corresponding class exemplar. At test
time, we return the label of the exemplar that is closest to the normalized
input.

Based on these observations, we propose an alternative paradigm for continual
learning centered around the idea of transferring modules that learn the
general aspects of the problem (for example identity-preserving transforms
that act similarly on all objects, such as illumination changes). We
hypothesize that destructive model updates can be avoided by separating two
objectives: (i) _generalization_ , or learning class-agnostic transforms that
successfully transfer to past and future tasks, and (ii) _memorization_ of
class-specific information. Crucially, our framework resolves the catastrophic
forgetting issue by _disentangled learning_ , that is, having a separate
update procedure for the generalization model and the memory buffer (please
note the difference from _learning disentangled representations_). This
separation allows us to maintain, prune, and expand task-specific knowledge
stored in the memory buffer while continuously training the generalization
model. By focusing on learning the universal transformations, we can not only
avoid destructive gradient updates, but efficiently accumulate knowledge over
time.

To demonstrate our proposed idea of learning universal transformations, we
introduce _infinite dSprites (idSprites)_ , a continual learning benchmark
generator inspired by the dSprites dataset [23]. It allows for procedurally
generating a virtually infinite progression of random two-dimensional shapes.
Similar to [23], we generate each unique shape in every combination of
orientation, scale, and position (see Fig. 2). Most importantly, by providing
the ground truth values of individual factors of variation (FoVs), idSprites
enables us to learn general transformations, thereby separating generalization
from memorization and testing our main hypothesis. We hope that by releasing
idSprites as a Python package we will encourage the research community to test
their methods on our benchmark.

Section 3.2 describes an implementation of our disentangled learning framework
in the context of class-incremental continual learning. Our proof of concept
consists of an _equivariant network_ that learns to regress the parameters of
an affine transform that maps any input into its canonical form, a
_normalisation module_ that applies the predicted affine transformation to the
input, and an _exemplar buffer_ that stores a single exemplar per class.
Figure 1 shows the main components of our framework.

#### Contributions

We summarize the most important contributions of this work below:

  * •

We introduce a new framework for generating continual classification and
disentanglement benchmarks that for the first time allows testing continual
learning methods over thousands of tasks. We will open-source our software
package upon acceptance.

  * •

We propose a novel continual learning paradigm based on learning symmetry
transformations, which circumvents catastrophic forgetting by separating
gradient-based model updates from explicit memory edits.

  * •

We demonstrate that as the number of tasks grows, regularization-based
continual learning methods quickly break down and replay-based methods either
deteriorate in performance or become impractical due to extensive use of
memory and compute.

  * •

We show that our approach exhibits significant forward and backward transfer,
strong open-set classification performance, and excellent zero-shot
generalisation. It can learn over hundreds of tasks with a constant
computational budget and a slowly growing memory footprint.

##  2 Two problems with the current approach to class-incremental continual
learning

We begin by addressing two aspects of current continual learning research that
motivate our contributions.

#### Benchmarking

Continual learning datasets are typically limited to just a few tasks and, at
most, a few hundred classes. In contrast, humans can learn and recognise
thousands of novel objects throughout their lifetime. We argue that as the
continual learning community we should focus more on scaling the number of
tasks in our benchmarks. We show that when tested over hundreds of tasks
standard methods inevitably fail: the effect of regularization decays over
time, adding more parameters quickly becomes unpractical, and storing and
replaying old examples causes a rapid increase in both memory and compute.
Moreover, to tackle individual sub-problems in continual learning, such as the
influence of task similarity on forgetting, the role of disentangled
representations, and the influence of hard task boundaries, we need to be able
to flexibly create datasets that let us isolate these problems. We should also
step away from the static training and testing stages and embrace streaming
settings where the model can be evaluated an any point. Finally, to advance
beyond classification tasks, we need richer ground truth data than just class
labels.

These observations motivated us to create a novel evaluation protocol. Taking
inspiration from object-centric disentanglement libraries [19, 8], we
introduce the infinite dSprites (idSprites) framework that allows for
procedurally generating virtually infinite streams of data while retaining
full control over the number of tasks, their difficulty and respective
similarity, and the nature of boundaries between them. We hope that it will be
a useful resource to the community and unlock new research directions in
continual learning.

#### Invariant representations

Continual learning methods are usually benchmarked on class-incremental
setups, where a classification problem is split into several tasks to be
solved sequentially [37]. Note that the classification learning objective is
_invariant_ to identity-preserving transformations of the object of interest,
such as rotation, change of lighting, or perspective projection.
Unsurprisingly, the most successful discriminative learning architectures,
from AlexNet [16] to ResNet [9], learn only features that are relevant to the
classification task [35, 11] and discard valuable information about universal
transformations, symmetries, and compositionality. In doing so, they entangle
the particular class information with the knowledge about generalization
mechanisms and represent both in the weights of the model. When a new task
arrives, there is no clear way to update the two separately.

In this paper, we reframe the problem and recognize that the information about
identity-preserving transformations, typically discarded, is important for
transfer across tasks. For instance, a change in illumination affects objects
of various classes similarly. Understanding this mechanism would lead to
better generalization on future classes. Consequently, we suggest that
modeling these transformations is key to achieving positive forward and
backward transfer in continual classification. Symmetry transformations, or
equivariances, offer a structured framework for this kind of modeling, which
we elaborate on in the subsequent section.

##  3 Methods

In this section, we describe two important contributions of this work: a
software package for generating arbitrarily long continual learning benchmarks
and a conceptual disentangled learning framework accompanied by an example
implementation. We would like to emphasize that this work aims to provide a
new perspective on knowledge transfer in continual learning, and to propose
new benchmarks for evaluating continual learning methods. Our implementation
serves as a proof of concept, spotlighting the potential of equivariance
learning, and is not intended as a practical method for general use.

Figure 2: A batch of images from the Infinite dSprites dataset containing four
distinct shapes. Each shape is shown in all combinations of four factors of
variation (horizontal and vertical position, orientation, and scale) with two
possible values per factor.

###  3.1 Infinite dSprites

We introduce idSprites, a novel framework inspired by dSprites [23], designed
for easy creation of arbitrarily long continual learning benchmarks. A single
idSprites benchmark consists of $T$ tasks, where each task is an $n$-fold
classification of procedurally generated shapes. Similar to dSprites, each
shape is observed in all possible combinations of the following FoVs: color,
scale, orientation, horizontal position, and vertical position. Figure 2 shows
an example batch of images with four FoVs and two values per factor (in
general, our implementation allows for arbitrary granularity). The _canonical
form_ corresponds to a scale of 1, orientation of 0, and horizontal and
vertical positions of 0.5. We only use a single color in our experiments for
simplicity and to save computation.

The shapes are generated by first randomly sampling the number of vertices
from a discrete uniform distribution over a closed integer interval
$\llbracket a,b\rrbracket$, then constructing a regular polygon on a unit
circle, randomly perturbing the polar coordinates of each vertex, and finally
connecting the perturbed vertices with a closed spline of the order randomly
chosen from $\\{1,3\\}$. All shapes are then scaled and centered so that their
bounding boxes are the same size and their centers of mass align in the
canonical form. We also make orientation identifiable by painting one half of
the shape black.

The number of tasks $T$, the number of shapes per task $n$, the vertex number
interval $\llbracket a,b\rrbracket$, the exact FoV ranges, and the parameters
of noise distributions for radial and angular coordinates are set by the user,
providing the flexibility to control the length and difficulty of the
benchmark. The framework also provides access to the ground truth values of
the individual FoVs. We will release idSprites as a Python package and hope it
will unlock new research directions in continual classification, transfer
learning, and continual disentanglement.

###  3.2 Disentangled learning

With our procedural benchmark generator, we can test continual learning
methods over time frames an order of magnitude longer than those covered by
existing datasets. As previously mentioned, we hypothesize that to learn
efficiently over such time horizons, we need to clearly distinguish between
the generalization mechanism that needs to be learned and the class-specific
information that has to be memorized. We start by observing that human
learning is likely characterized by such separation. Take face recognition,
for example. A child is able to memorize the face of its parent but can still
get confused by an unexpected transformation, as evidenced by countless online
videos of babies failing to recognize their fathers after a shave. Once we
learn the typical identity-preserving transformations that a face can undergo,
we need only to memorize the particular features of any new face to instantly
generalize over many transformations, such as facial expression, lighting,
three-dimensional rotation, scale, perspective projection, or a change of
hairstyle. Note that while we encounter new faces every day, these transforms
remain consistent and affect every face similarly.

Inspired by this observation, we aim to disentangle generalization from
memorization by explicitly separating the learning module from the memory
buffer in our model design. The memory buffer stores a single exemplar image
of each encountered shape. We assume these are given to us by an oracle
throughout training, but it would be possible to bootstrap the buffer with a
few initial exemplars. The equivariance learning module is a neural network
designed to learn the general transformations present in the data.

#### Our implementation and training objective

We tackle the class-incremental scenario [37]: at each task $n$ we observe a
set of data points $D_{n}=\\{(x_{n,m},y_{n,m})\\}_{m=1}^{N_{n}}$, where
$x_{n,m}$ is an image and $y_{n,m}$ is a vector containing the values of the
generative factors and the class label. Each data point belongs to one of
$C_{n}$ distinct classes. At the start of each task we add the current task
exemplars $E_{n}=\\{\hat{x}_{i}\\}_{i=1}^{C_{n}}$ to the exemplar buffer. We
then train a neural network to output a transformation matrix $\hat{\theta}$
that approximates the ground truth affine transformation matrix $\theta$
mapping each input image to its respective exemplar: $\hat{x}=T_{\theta}(x)$.
The network is trained continually to minimize the MSE loss between
$\hat{\theta}$ and $\theta$. At test time, we use the trained network to
regress an affine transformation matrix mapping a previously unseen image to
its exemplar. The normalization module applies the predicted transform to the
input image using grid sampling with border values for out-of-bound grid
locations. The image is then classified through nearest neighbour lookup. Our
normalization module is differentiable and therefore allows for
backpropagating a loss computed in image space. In practice, we found that
this additional reconstruction loss does not improve much over direct FoV
supervision.

Figure 3: From top to bottom: inputs, outputs of the normalization module,
closest exemplars.

#### Discussion

The disentangled learning approach has a number of advantages. First, by
learning transformations instead of class boundaries, we reformulate a
challenging class-incremental classification scenario as a domain-incremental
FoV regression learning problem [37]. Since the transformations affect every
class in the same way, they are easier to learn in a continual setting. We
show that this approach is not only less prone to forgetting but exhibits
significant forward and backward transfer. In other words, the knowledge about
regressing FoVs is efficiently accumulated over time. Second, the exemplar
buffer is a fully explainable representation of memory that can be explicitly
edited: we can easily add a new class or completely erase a class from memory
by removing its exemplar. Finally, we show experimentally that our method
generalises instantly to new shapes with just a single exemplar and works
reliably in an open-set classification scenario. Figure 3 illustrates the
stages of the classification mechanism: five input images, their corresponding
normalization network outputs, and closest exemplars from the buffer.

##  4 Related work

###  4.1 Continual learning

Continual learning literature typically focuses on catastrophic forgetting in
supervised classification. Parameter isolation methods use dedicated
parameters for each task by periodically extending the architecture while
freezing already trained parameters [33] or by relying on isolated sub-
networks [6]. Regularization approaches aim to preserve existing knowledge by
limiting the plasticity of the network. Functional regularization methods
constrain the network output through knowledge distillation [17] or by using a
small set of anchor points to build a functional prior [26, 36]. Weight
regularization methods [39] directly constrain network parameters according to
their estimated importance for previous tasks. In particular, Variational
Continual Learning (VCL) [25] derives the importance estimate by framing
continual learning as sequential approximate Bayesian inference. Most methods
incorporate regularization into the objective function, but it is also
possible to implement it using constrained optimization [22, 2, 10, 13].
Finally, replay methods [30, 4, 12, 32] retain knowledge through rehearsal.
When learning a new task, the network is trained with a mix of new samples
from the training stream and previously seen samples drawn from the memory
buffer. A specific case of this strategy is generative replay [34, 3], where
the rehearsal samples are produced by a generative model trained to
approximate the data distribution for each class. Many continual learning
methods are hybrid systems that mix and match the above techniques.

###  4.2 Benchmarking continual learning

Established continual learning benchmarks primarily involve splitting existing
computer vision datasets into discrete, non-overlapping segments to study
continual supervised classification. Notable examples in this domain include
split MNIST [39], split CIFAR [39], and split MiniImageNet [4, 1], along with
their augmented counterparts, such as rotated MNIST [22], and permuted MNIST
[15]. More recently, contributions from Lomonaco and Maltoni [20], Verwimp et
al. [38] and Roady et al. [31] have enriched the field with dataset designed
specifically for continual learning, such as CORe50, CLAD, and Stream-51,
which comprise temporally correlated images with diverse backgrounds and
environments.

##  5 Experiments

In this section, we evaluate standard continual learning methods and our
disentangled learning framework on a benchmark generated using idSprites. The
benchmark consists of 200 classification tasks. For each task, we randomly
generate 10 shapes and create an image dataset showing the shapes in all
combinations of 4 FoVs with 8 possible values per factor. This gives us 40,960
samples per task, which we then randomly split into training, validation, and
test sets with a 98:1:1 ratio. The test set is being accumulated at each task
until it reaches the maximum limit of 50,000 samples, at which point we employ
reservoir sampling to include new classes while keeping the test set bounded
and balanced. After training on each task, we report test accuracy on the
accumulated test set. To ensure a reliable comparison, we use the same
backbone (ResNet-18 [9]) for every method. We also make sure that all models
are trained until convergence.

We aim to position our approach among existing continual learning methods, as
well as understand its generalization properties. More concretely, the
experiments answer the following questions:

  * •

(Section 5.1) How does our approach compare to regularisation-based continual
learning baselines?

  * •

(Section 5.2) How does our approach compare to replay-based continual learning
baselines? How is the performance of replay affected by memory buffer size and
computational budget?

  * •

(Section 5.3) Do we need equivariance or is learning invariant representations
enough?

  * •

(Section 5.4) Is our approach able to generalise instantly to previously
unseen shapes?

  * •

(Section 5.5) Can our approach perform open-set classification, i.e.
distinguish between new and previously seen shapes?

  * •

(Section 5.6) Can we use our approach in the online learning scenario, where
each sample from the training stream is observed only once?

###  5.1 Regularization methods

In this section, we compare our method to standard regularization methods:
Learning without Forgetting (LwF) [17] and Synaptic Intelligence (SI) [39]. We
use implementations from Avalanche [21]. We provide details of the
hyperparameter choice in the supplementary material. As shown in Fig. 4, such
regularization methods are ill-equipped to deal with the class-incremental
learning scenario and perform no better than naive fine-tuning.

Figure 4: A comparison of our approach to standard regularization methods
(LwF, SI) and naive fine-tuning. The plot for our method is an average of 5
runs.

###  5.2 Replay-based methods

Replay-based methods retain a subset of past training data that is then mixed
with the current training data for every task. While this can be a viable
strategy to preserve accuracy over many tasks, it results in ever-growing
memory and computation requirements, unless the buffer size is bounded. In
this section, we investigate the effect of buffer size on performance for
standard experience replay with reservoir sampling. While there are replay-
based methods that improve on this baseline, we are interested in
investigating the fundamental limits of rehearsal over long time horizons and
strip away the confounding effects of data augmentation, generative replay,
sampling strategies, pseudo-rehearsal etc. Figure 5 shows test accuracy for
experience replay with different buffer sizes. Storing enough past samples
lets the model maintain high test accuracy, but even with a buffer of 20,000
images the performance eventually starts to deteriorate. Note that after 200
tasks a balanced buffer will only contain 10 samples per class.

#### A note on implementation

In an attempt to make the replay baseline stronger, we first add the data from
the current task to the buffer and then train the model exclusively on the
buffer, effectively discounting the influence of the current task over time
[28]. A more standard version of experience replay would mix current and past
data in equal proportions in each mini-batch, likely leading to diminished
performance on previous tasks. The supplementary material includes a
comparison to this other replay baseline, as well as to a version of
experience replay with no memory constraint but with a compute budget matching
our approach.

Figure 5: A comparison of our approach to experience replay with different
buffer sizes. The plot for our method is an average of 5 runs.

###  5.3 Do we need equivariance?

By training a network to regress the value of each FoV, we learn a
representation that is equivariant to affine transformations. However, we
could also take advantage of the shape labels to learn an invariant
representation that we could then use to perform classification. In this
section, we directly compare equivariant and invariant learning to demonstrate
further that learning an equivariant representation is the key to achieving
effective continual learning within our framework.

Our baseline for invariant representation learning is based on SimCLR [5], a
simple and effective contrastive learning algorithm that aims to learn
representations invariant to data augmentations. To adapt SimCLR to our
problem, we introduce two optimization objectives. The first objective pulls
the representation of each training point towards the representation of its
exemplar while repelling all other training points. The second objective
encourages well-separated exemplar representations by pushing the
representations of all exemplars in the current task away from each other. We
observed that the first training objective alone is sufficient, but including
the second loss term speeds up training. For each task, we train the baseline
until convergence. At test time, the class labels are assigned through nearest
neighbor lookup in the representation space. Similar to our method, we store a
single exemplar per class.

Figure 6 shows test accuracy for both methods over time. The performance of
the contrastive learning baseline decays over time, but not as rapidly as
naive fine-tuning. Note that in contrast to our method, invariant learning
could benefit from storing more than one exemplars per class. The
supplementary material provides an exact formulation of the contrastive
objective and implementation details.

Figure 6: A comparison of our approach to an invariant contrastive learning
baseline. The plot for our method is an average of 5 runs.

###  5.4 One-shot generalization

To evaluate whether the learned regression network can generalize to unseen
classes, we perform a one-shot learning experiment. Here, the model had to
normalize and classify transformed versions of shapes it had not previously
encountered.

Since the returned class label depends on the exemplars in the buffer, we
consider two variants of the experiment, corresponding to generalized and
standard one-shot learning. In the first one, we keep the training exemplars
in the buffer and add new ones. In the second, the buffer is cleared before
including novel exemplars. We also introduce different numbers of test
classes. The classification accuracies are presented in Table 1. As expected,
keeping the training exemplars in the buffer and adding more test classes
makes the task harder. Nevertheless, the accuracy stays remarkably high,
showing that the equivariant network has learned a correct and universal
mechanism that works even for previously unseen shapes. This is the essence of
our framework.

Figure 7: Precision-recall curve for the open-set classification task. Note that classifying all inputs as unseen (or seen) leads to perfect precision (or recall) scores. Table 1: One-shot generalization accuracy of our framework on previously unseen shapes. As expected, keeping the training exemplars in the buffer (first row), as well as including more test classes (left to right) result in slightly worse accuracy. The training set contains 2000 distinct classes. Reset buffer? |  | Test classes
---|---|---
|  10 |  100 |  1000
No |  |  0.96 |  0.95 |  0.93
Yes |  |  1.0 |  0.98 |  0.95

###  5.5 Open-set classification

Next, we investigate how well our proposed framework can detect novel shapes.
This differs from the one-shot generalization task because we do not add the
exemplars corresponding to the novel shapes to the buffer. Instead of
modifying the learning setup, we use a simple heuristic based on an empirical
observation that our model can almost perfectly normalize any input—we
classify the input image as unseen if we can’t find an exemplar that matches
the normalized input significantly better than others.

Denoting the equivariant network output by $\hat{\theta}$ and the two best
candidate exemplars by $c_{1}$ and $c_{2}$, we classify the test input as
novel if
$\|c_{1}-T_{\hat{\theta}}(x)\|_{2}^{2}>\sigma\|c_{2}-T_{\hat{\theta}}(x)\|_{2}^{2}$.
Figure 7 shows the precision-recall curve for different values of $\sigma$,
with an overall area under curve of 0.92. We also present qualitative results
in Fig. 8.

Figure 8: Some of the incorrectly classified unseen shapes in the open-set
recognition task. Top: outputs of the normalization module. Bottom: closest
training exemplars.

###  5.6 Online vs. offline

In all previous experiments, we applied our method in batch mode: we performed
multiple training passes over the data for each task. However, efficiently
learning from streaming data might require observing each training sample only
once to make sure computation is not becoming a bottleneck. This is why we
test our method in the online learning regime and compare it to two batch
learning scenarios. The results are shown in Fig. 9. Unsurprisingly, training
for multiple epochs results in better and more robust accuracy on past tasks;
it is however worth noting that our method still improves over time in the
online learning scenario. It is possible that, given enough tasks, all three
curves would converge.

Figure 9: Cumulative test accuracy of our method in the online scenario (every
data point is only seen once) and two offline scenarios (3 and 5 training
epochs per task). Each plot is an average of five runs.

##  6 Discussion

In the last decade, continual learning research has made progress through
parameter and functional regularization, rehearsal, and architectural
strategies that mitigate forgetting by preserving important parameters or
compartmentalizing knowledge. As pointed out in a recent survey [37], the best
performing continual learners are based on storing or synthesizing samples.
Such methods are typically evaluated on sequential versions of standard
computer vision datasets such as MNIST or CIFAR-100, which often involve only
a small number of learning tasks, discrete task boundaries, and fixed data
distributions. As such, the benchmarks do not match the lifelong nature of
real-world learning tasks.

Our work is motivated by the hypothesis that state-of-the-art continual
learners and their predecessors, would inevitably fail when trained in a true
lifelong fashion akin to humans. To test our claim, we introduced _idSprites_
dataset, consisting of procedurally generated shapes and their affine
transformation. To our knowledge, this is the first class-incremental
continual learning benchmark that allows generating hundreds or thousands of
tasks. While acknowledging the relatively simplistic nature of our dataset, we
believe any _lifelong_ learner must solve _idSprites_ before tackling more
complicated, real-world datasets. Nevertheless, our empirical findings
highlight that all standard methods are doomed to collapse, and memory buffers
can only defer the ultimate end.

Updating synaptic connections in the human brain upon novel experiences does
not interfere with the general knowledge accumulated throughout life. Inspired
by this insight, we propose our _disentangled learning_ framework, which
splits the continual learning problem into (i) sequentially training a network
that models the general aspects of the problem that apply to all instances
(equivariances) and (ii) memorizing class-specific information relevant to the
task (exemplars). This separation enables _disentangled model updating_ ,
which allows for continually learning equivariant representations without
catastrophic forgetting and explicitly updating class-specific information
without harming information corresponding to other classes. As demonstrated
experimentally, such a separation exhibits successful forward and backward
transfer and achieves impressive one-shot generalization and open-set
recognition performance.

#### Limitations

With this work, we aim to bring a fresh perspective and chart a novel research
direction in continual learning. To demonstrate our framework, we stick to a
simple dataset and include the correct inductive biases in our learning
architecture. We acknowledge that when applied to natural images, our approach
would suffer from a number of issues, which we list below, along with some
mitigation strategies.

  * •

Real-world data does not come with perfect supervision signals, hindering the
learning of equivariant networks. As a remedy, one might employ equivariant
architectures as an inductive bias [7] or weakly supervise the learning, e.g.
with image-text pairs [29].

  * •

Obtaining class exemplars for real-world data is not straightforward, which
makes training the normalization network difficult. A potential solution is to
maintain multiple exemplars per class.

  * •

It is not clear that we can separate generalization and memorization for any
continual learning problem. We plan to investigate this question on a real-
world dataset.

#### Acknowledgements

This work was supported by the German Federal Ministry of Education and
Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A. This research utilized
compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST
37/1057-1 FUGG. We thank the International Max Planck Research School for
Intelligent Systems (IMPRS-IS) for supporting SD. This work was supported by
the National Centre of Science (Poland) Grants No. 2020/39/B/ST6/01511 and
2022/45/B/ST6/02817.

## References

  * Aljundi et al. [2019a] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia.  Online continual learning with maximal interfered retrieval.  In _Advances in Neural Information Processing Systems 32_ , pages 11849–11860. Curran Associates, Inc., 2019a.
  * Aljundi et al. [2019b] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  _Advances in neural information processing systems_ , 32, 2019b.
  * Atkinson et al. [2018] Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins.  Pseudo-recursal: Solving the catastrophic forgetting problem in deep neural networks.  _arXiv preprint arXiv:1802.03875_ , 2018.
  * Chaudhry et al. [2019] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning.  _arXiv preprint arXiv:1902.10486_ , 2019.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Fernando et al. [2017] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra.  Pathnet: Evolution channels gradient descent in super neural networks.  _arXiv preprint arXiv:1701.08734_ , 2017.
  * Finzi et al. [2021] Marc Finzi, Max Welling, and Andrew Gordon Wilson.  A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups.  In _International conference on machine learning_ , pages 3318–3328. PMLR, 2021.
  * Gondal et al. [2019] Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer.  On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset.  In _Advances in Neural Information Processing Systems 32 (NeurIPS 2019)_ , pages 15714–15725. Curran Associates, Inc., 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hess et al. [2023] Timm Hess, Tinne Tuytelaars, and Gido M van de Ven.  Two complementary perspectives to continual learning: Ask not only what to optimize, but also how.  _arXiv preprint arXiv:2311.04898_ , 2023.
  * Higgins et al. [2022] Irina Higgins, Sébastien Racanière, and Danilo Rezende.  Symmetry-based representations for artificial and biological general intelligence.  _Frontiers in Computational Neuroscience_ , 16:836498, 2022.
  * Isele and Cosgun [2018] David Isele and Akansel Cosgun.  Selective experience replay for lifelong learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Kao et al. [2021] Ta-Chu Kao, Kristopher Jensen, Gido van de Ven, Alberto Bernacchia, and Guillaume Hennequin.  Natural continual learning: success is a journey, not (just) a destination.  _Advances in neural information processing systems_ , 34:28067–28079, 2021.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences of the United States of America_ , 114(13):3521–3526, 2017.  MAG ID: 2560647685.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Li and Hoiem [2017a] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(12):2935–2947, 2017a.
  * Li and Hoiem [2017b] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(12):2935–2947, 2017b.
  * Locatello et al. [2019] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In _international conference on machine learning_ , pages 4114–4124, 2019.
  * Lomonaco and Maltoni [2017] Vincenzo Lomonaco and Davide Maltoni.  Core50: a new dataset and benchmark for continuous object recognition.  In _Proceedings of the 1st Annual Conference on Robot Learning_ , pages 17–26. PMLR, 2017.
  * Lomonaco et al. [2021] Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido M Van de Ven, et al.  Avalanche: an end-to-end library for continual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 3600–3610, 2021.
  * Lopez-Paz and Ranzato [2017] David Lopez-Paz and Marc’ Aurelio Ranzato.  Gradient Episodic Memory for Continual Learning.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Matthey et al. [2017] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner.  dsprites: Disentanglement testing sprites dataset.  https://github.com/deepmind/dsprites-dataset/, 2017.
  * McCloskey and Cohen [1989] Michael McCloskey and Neal J. Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of Learning and Motivation Vol. 24_ , pages 109–165. Academic Press, 1989.
  * Nguyen et al. [2018] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner.  Variational continual learning.  In _International Conference on Learning Representations_ , 2018.
  * Pan et al. [2020] Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Turner, and Mohammad Emtiyaz E Khan.  Continual Deep Learning by Functional Regularisation of Memorable Past.  In _Advances in Neural Information Processing Systems_ , pages 4453–4464. Curran Associates, Inc., 2020.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Prabhu et al. [2020] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pages 524–540. Springer, 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Roady et al. [2020] Ryne Roady, Tyler L. Hayes, Hitesh Vaidya, and Christopher Kanan.  Stream-51: Streaming classification and novelty detection from videos.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2020.
  * Rolnick et al. [2019] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2019.
  * Rusu et al. [2016] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Shin et al. [2017] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  _Advances in neural information processing systems_ , 30, 2017.
  * Tishby and Zaslavsky [2015] Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  In _2015 ieee information theory workshop (itw)_ , pages 1–5. IEEE, 2015.
  * Titsias et al. [2020] Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh.  Functional regularisation for continual learning with gaussian processes.  In _International Conference on Learning Representations_ , 2020.
  * van de Ven et al. [2022] Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias.  Three types of incremental learning.  _Nature Machine Intelligence_ , 4(12):1185–1197, 2022.
  * Verwimp et al. [2023] Eli Verwimp, Kuo Yang, Sarah Parisot, Lanqing Hong, Steven McDonagh, Eduardo Pérez-Pellitero, Matthias De Lange, and Tinne Tuytelaars.  Clad: A realistic continual learning benchmark for autonomous driving.  _Neural Networks_ , 161:659–669, 2023.
  * Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _International conference on machine learning_ , pages 3987–3995. PMLR, 2017.

Supplementary Material

##  1 Experiment details

All models employ the same ResNet-18 [9] backbone and are trained using the
Adam optimizer [14] with default PyTorch [27] parameter values
($\lambda=0.001$, $\beta_{1}=0.9$, $\beta_{2}=0.999$). For our method we
always report the average test accuracy over 5 runs with 5 epochs per task.
For other methods, due to computational constraints, we only report the
accuracy of a single run.

###  1.1 Regularization methods

We ran a grid search to set the loss balance weight $\lambda_{o}$ in LwF [18]
and the strength parameter $c$ in SI [39], but found the choice of
hyperparameter did not influence the result. For the run shown in Figure Fig.
4, we used $\lambda_{o}=0.1$ and $c=0.1$.

###  1.2 Replay methods

For the experiments presented in the main text, we use a modified version of
experience replay, inspired by the approach of GDumb [28]. At the beginning of
each task $t$, we add all the training data from the current task to the
buffer. If the buffer is full, we employ reservoir sampling so that the memory
buffer contains an equal number of examples of every class seen so far,
including the classes in task $t$. We then train the model on the memory
buffer until convergence and report test accuracy.

Here we present results for a different rehearsal approach. For each task $t$,
we extend every mini-batch of the training set with an equal number of samples
chosen randomly (with replacement) from the buffer. We train the model for
five epochs per task. Figure 1 shows a comparison of this training protocol to
our method. Surprisingly, this way of doing replay yields better test set
accuracy than the replay baseline we used in the main text, despite putting a
disproportionate weight on the current task.

Figure 1: A comparison of our approach to experience replay with different
buffer sizes.

Figure 2 shows a comparison of our method to experience replay with no limit
on the buffer size. Here we also mix the samples from the current task with
randomly chosen buffer samples and train for five epochs per task.

Figure 2: A comparison of our approach to experience replay with an unbounded
buffer.

###  1.3 Contrastive baseline

The optimization objective of the contrastive baseline consists of two
components. The first one ensures that each sample in the batch is pulled
towards its exemplar and pushed away from all the other samples in the mini-
batch that belong to a different class. Using the terminology from [5], a
sample $x_{i}$ and its corresponding exemplar $\hat{x}_{i}$ constitute a
_positive pair_. Denoting the class of a sample as $c(x)$, the mini-batch size
as $N$, and the representation of $x_{i}$ and $\hat{x}_{i}$ as $z_{i}$ and
$\hat{z}_{i}$, respectively, the first component of the loss is:

$l_{1}(x_{i})=-\log\frac{\exp(sim(z_{i},\hat{z_{i}})/\tau)}{\sum_{k=1}^{N}\mathbbm{1}_{[c(k)\neq
c(i)]}\exp(sim(z_{i},z_{k})/\tau)},$ (1)

where $sim(z_{i},z_{j})$ is the cosine similarity between $z_{i}$ and $z_{j}$.

The second component encourages well-separated exemplar representations by
pushing apart the representations of all the exemplars in the current mini-
batch. Denoting the number of distinct classes in the mini-batch as $C$, we
have:

$l_{2}(x_{i})=-\log\frac{\exp(1/\tau)}{\sum_{k=1}^{C}\mathbbm{1}_{[k\neq
i]}\exp(sim(\hat{z_{i}},\hat{z_{k}})/\tau)}.$ (2)

The final mini-batch loss is
$\mathcal{L}=\sum_{i=1}^{N}l_{1}(x_{i})+l_{2}(x_{i})$.