# DSI++: Updating Transformer Memory with New Documents

Sanket Vaibhav Mehta1, Jai Gupta2, Yi Tay2, Mostafa Dehghani2, Vinh Q. Tran2,
Jinfeng Rao2, Marc Najork2, Emma Strubell1,2, Donald Metzler2

$1$Carnegie Mellon University, $2$Google Research
Work performed during an internship at Google Research.

###### Abstract

Differentiable Search Indices (DSIs) encode a corpus of documents in the
parameters of a model and use the same model to map queries directly to
relevant document identifiers. Despite the strong performance of DSI models,
deploying them in situations where the corpus changes over time is
computationally expensive because reindexing the corpus requires re-training
the model. In this work, we introduce DSI++, a continual learning challenge
for DSI to incrementally index new documents while being able to answer
queries related to both previously and newly indexed documents. Across
different model scales and document identifier representations, we show that
continual indexing of new documents leads to considerable forgetting of
previously indexed documents. We also hypothesize and verify that the model
experiences forgetting events during training, leading to unstable learning.
To mitigate these issues, we investigate two approaches. The first focuses on
modifying the training dynamics. Flatter minima implicitly alleviate
forgetting, so we optimize for flatter loss basins and show that the model
stably memorizes more documents (+12%). Next, we introduce a generative memory
to sample pseudo-queries for documents and supplement them during continual
indexing to prevent forgetting for the retrieval task. Extensive experiments
on novel continual indexing benchmarks based on Natural Questions (NQ) and MS
MARCO demonstrate that our proposed solution mitigates the forgetting by a
significant margin. Concretely, it improves the average Hits@10 by $+21.1\%$
over competitive baselines for NQ and requires $6$ times fewer model updates
compared to re-training the DSI model for incrementally indexing five corpora
in a sequence.

##  1 Introduction

Differentiable Search Indices (DSIs; Tay et al. (2022)) represent a new
modeling paradigm for information retrieval tasks using sequence-to-sequence
learning. Specifically, DSIs leverage Transformer memory (Vaswani et al.,
2017) to encode all of the information in a corpus of documents and then use
that memory to answer user queries directly, thereby simplifying the retrieval
process. DSIs achieve this functionality by jointly optimizing for indexing
(or memorization) and retrieval tasks. The indexing task requires learning a
mapping from document content to its identifier, typically represented by
integers or short strings (document identifiers, abbreviated docids). Then,
the retrieval task necessitates mapping user queries to relevant docids. Apart
from its simplicity and end-to-end differentiable nature, DSI significantly
outperforms state-of-the-art “retrieve-and-rank” methods based on dual-
encoders (Ni et al., 2022).

Despite the remarkable performance of DSI models, there remain open questions
about their applicability in the practical setting of dynamic corpora.
Consider the realistic scenario wherein new documents are continually added to
the indexed corpus. Updating the index in dual-encoder based methods requires
computing embeddings for new documents, followed by re-indexing all document
embeddings (Karpukhin et al., 2020). In contrast, index construction using a
DSI involves training a Transformer model. Therefore, the model must be re-
trained from scratch every time the underlying corpus is updated, thus
incurring prohibitively high computational cost compared to dual-encoders. In
this work, we aim to address this issue by devising methods for effective
incremental indexing using Transformer memory without re-training the DSI
model from scratch.

Lifelong (or continual) learning (Thrun, 1995; Parisi et al., 2019) is a
biologically-inspired machine learning paradigm that deals with continuous
learning of new tasks by preserving past knowledge and using it to efficiently
learn new concepts. Based on this paradigm, we propose DSI++ (DSI + new
documents), a continual learning challenge for DSI to incrementally index new
documents while maintaining the ability to answer user queries related to both
previously and newly indexed documents. To enable DSI++, we introduce novel
benchmarks constructed from existing Natural Questions (Kwiatkowski et al.,
2019) and MS MARCO (Nguyen et al., 2016) datasets, simulating the continual
addition of documents to the system. To the best of our knowledge, there is no
prior work studying incremental learning for DSI.

(a) Explicit forgetting during continual indexing

(b) Implicit forgetting during memorization

Figure 1: (a) Indexing accuracy of $D_{0},D_{1},$ and $D_{2}$ document corpora
visualized as we continuously index new documents (averaged over $3$ runs). We
observe that continual indexing of new documents leads to severe forgetting of
the previously memorized documents. (b) Cumulative histogram of forgetting
events, i.e., events when individual documents go from being classified
correctly to incorrectly, over the course of memorization of the initial
$D_{0}$ corpus (T5-Base w/ Adafactor optimizer).

A naive solution for DSI++ is to continuously fine-tune the model with an
indexing objective over new documents. However, Figure 1a shows that continual
indexing of new documents leads to catastrophic forgetting of the previously
memorized documents (more details in §2.1), a common phenomenon in
connectionist networks wherein learning of the new concepts interferes with
the previously acquired knowledge (McCloskey & Cohen, 1989; French, 1999).
Furthermore, when we investigate the learning dynamics of the DSI model during
memorization (Figure 1b, we observe a significant number of documents (approx.
$88\%$) experience forgetting events after they have been memorized.
Concretely, a forgetting event (Toneva et al., 2019) is when a prediction for
an individual document goes from correct docid to incorrect one over the
course of learning. Therefore, implicit forgetting during memorization and
explicit forgetting from continual indexing of new documents are two key
challenges to overcome for successfully implementing a DSI++ system.

To reduce forgetting during memorization, we propose explicitly optimizing for
flatter loss basins using Sharpness-Aware Minimization (SAM; Foret et al.
(2021)). Recent works have shown that geometrical properties of the minima
play a vital role in forgetting, especially models in flatter loss basins tend
to undergo less forgetting while lifelong learning from task sequences
(Mirzadeh et al., 2020; Mehta et al., 2021). Next, we introduce a generative
memory to sample pseudo-queries for already indexed documents and use them to
alleviate forgetting of the retrieval task during incremental indexing of the
new documents. Also, the generative memory enables continual semi-supervised
learning of the retrieval task by generating pseudo-queries for an incoming
batch of the new documents. Our main contributions can be summarized as
follows:

  * •

We introduce DSI++, a continual learning challenge for the recently proposed
Differentiable Search Indices (DSI) paradigm. To enable DSI++ evaluations, we
create two benchmarks based on existing Natural Questions and MS MARCO
datasets. To understand the severity of the forgetting phenomenon across
multiple scenarios, we analyze a suite of pre-trained models (T5-Base,
T5-Large, T5-XL) and different document identifier representations
(unstructured atomic, naively structured, and semantically structured).

  * •

We hypothesize and verify that the DSI model experiences forgetting events
throughout memorization. To alleviate these, we propose modifying training
dynamics to promote flatter minima using SAM and show that the model stably
memorizes $+12\%$ documents.

  * •

We propose generative memory-based experience rehearsal approach to alleviate
explicit forgetting during continual indexing and improve the average Hits@1
by $+25.0\%$ and Hits@10 by $+21.1\%$ over competitive baselines for MS MARCO
and NQ, respectively.

##  2 DSI++: Continual learning challenge for DSI

###  2.1 Problem setup

We focus on a setup where we receive an initial corpus of documents,
$D_{0}=\\{d_{1},\cdots,d_{n}\\}$, and user queries corresponding to a subset
of them, $R_{0}=\\{<q_{j},j>,\forall j\in\mathcal{Y}_{D}\\},$ where $D\subset
D_{0}$. DSI paradigm involves two tasks: (i) memorization task where the goal
is to learn an indexer $f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}$, a text-
to-text model like T5 (Raffel et al., 2020) parameterized by
$\theta\in\mathbb{R}^{P}$, that takes document tokens ($x\in\mathcal{X}$) as
input and maps it to a document identifier (docid) $j\in\mathcal{Y}$, and (ii)
retrieval task where the goal is to use the same indexer $f_{\theta}$ to
directly map a user query $q$ to a relevant docid $j\in\mathcal{Y}$. Following
Tay et al. (2022), two different prompts are used to differentiate between
these tasks.

Tay et al. (2022) discuss several variants for representing docids –
unstructured atomic and structured string docids, where each document is
assigned a unique token and tokenized string, respectively. Under the unified
text-to-text format, both of the above tasks are cast as generation tasks,
i.e., decoding one unique token (unstructured atomic) or decoding a tokenized
string sequentially, one token at a time (naively/ semantically structured).

In the dynamic corpus scenario, we simulate the arrival of new documents by
updating the initial corpus $D_{0}$ with a sequence of batches
$D_{1},D_{2},...$, with $D_{1}$ arriving first, followed by $D_{2}$, and so
on. In DSI++, we have access to the new batch of documents $D_{i}$, but we do
not have any queries related to these documents.

Goal: Learn a DSI++ system that incrementally indexes $D_{1},D_{2},\cdots$ in
$f_{\theta}$ while being able to answer queries related to previously as well
as additionally indexed documents.

###  2.2 Benchmark for DSI++.

To enable research on DSI++, we introduce two benchmarks constructed from the
Natural Questions (NQ; Kwiatkowski et al. (2019)) and MS MARCO (Nguyen et al.,
2016) datasets. The NQ dataset consists of Wikipedia articles and
corresponding natural language questions. Similar to Tay et al. (2022), we
consider Wikipedia articles for memorization and the retrieval task as
identifying the Wikipedia article that answers the given question. We use the
original NQ train split to construct train($80\%$)/ validation($20\%$) splits
and use NQ validation as a test split for our setup. We randomly sample $50K$
unique articles to constitute the initial $D_{0}$ corpus. Next, we construct
five corpora ($D_{1},\cdots,D_{5}$), each containing $10K$ unique articles, to
sequentially add them to the DSI model. Corresponding to articles in each of
these corpora, we filter queries from original NQ train/ validation splits to
construct $R_{i}^{train},R_{i}^{val},R_{i}^{test}$ ($\forall
i\in\\{0,\cdots,5\\}$) splits. We use $R_{0}$ to train the DSI model for the
retrieval task, and use $R_{i}^{test}$ to evaluate on previously and newly
indexed articles.

The full MS MARCO dataset has approx. $500K$ passage-query training pairs and
$6,980$ validation pairs. Similar to the benchmark created from the NQ
dataset, we randomly sample $50K$ unique passages to constitute the initial
$D_{0}$ corpus, and five more corpora, each with $10K$ passages. See Table 2
(in the Appendix A.1) for exact dataset statistics for NQ and MS MARCO.

###  2.3 Evaluation Metrics

For DSI evaluation, we report indexing accuracy for memorization task and
Hits@k ($k\in\\{1,10\\}$) metric for retrieval task. Indexing accuracy and
Hits@k are defined as the proportion of documents correctly memorized and the
proportion of correct documents ranked in the top k predictions, respectively.
Now, we formally define metrics to summarize the model performance as we
incrementally index new documents. Let $P_{n,o}$ denote the performance (e.g.,
indexing accuracy) on corpus $D_{o}$ after training on corpus $D_{n}$.
Following prior works (Lopez-Paz & Ranzato, 2017; Riemer et al., 2019), we
compute the average performance ($A_{n}$), forgetting ($F_{n}$) and learning
performance ($LA_{n}$) metrics after indexing the corpus $D_{n}$.

The term $F_{n}$ (also known as backward transfer) refers to the effect of
indexing the corpus $D_{n}$ on the performance of all previously indexed
documents $D_{o}$, where $0\leq o<n$. $LA_{n}$ (or forward transfer) measures
the model’s ability to learn when presented with a new corpus $D_{n}$, and is
defined as the average performance over the new corpora $D_{1},\cdots,D_{n}$.
When the $D_{n}^{\text{th}}$ corpus is incrementally indexed, $A_{n}$,
$F_{n}$, and $LA_{n}$ are defined as follows:

$\displaystyle\centering A_{n}=\frac{1}{n+1}\sum_{o=0}^{n}P_{n,o};\quad
F_{n}=\frac{1}{n}\sum_{o=0}^{n-1}\max_{o^{\prime}\in\\{0,\cdots,n-1\\}}(P_{o^{\prime},o}-P_{n,o});\quad
LA_{n}=\frac{1}{n}\sum_{o=1}^{n}P_{o,o}\@add@centering$ (1)

Figure 2: Systematic study about forgetting and forward transfer when
incrementally indexing new corpus of documents across different model sizes
(T5-Base, T5-Large, T5-XL) and docid representations. We use atomic docids by
default and denote (N)/(S) for naively/ semantically structured docids.
$\uparrow$ indicates higher is better, $\downarrow$ indicates lower is better.
We observe that by increasing the model scale, the average $A_{n}$ and
learning $LA_{n}$ performance improves. However, forgetting $F_{n}$ is severe
across all model scales. Moreover, we observe that naively structured docids,
T5-Base(N), underperform unstructured atomic docids, T5-Base, across all
metrics - indexing accuracy, Hits@1, (see Figure 5 in Appendix A.2 for Hits@10
results). Imbuing the docid space with semantic (S) structure alleviates the
forgetting compared to an arbitrary/ naive (N) structure.

###  2.4 Case study: Catastrophic Forgetting and Forward Transfer

After introducing the DSI++ problem setup, benchmark, and evaluation metrics,
we study the behavior of the DSI model as new documents are continuously added
to the system. Concretely, we are interested in investigating the following
for continual training of the DSI model with indexing objective on new
documents – (Q1) How severe is the forgetting for the originally indexed
documents?, (Q2) How does continual updating of the DSI model over a sequence
of corpora affect the forgetting?, (Q3) How does the updated DSI model perform
on newly indexed documents, especially the retrieval task?, (Q4) How do
different docid representation strategies affect forgetting?, and (Q5) How
does the DSI model scale affect forgetting? Figure 2 visualizes results
(averaged over $3$ runs) on the validation split of DSI++ and help us
convincingly answer these questions.

Forgetting (or negative backward transfer). From Figure 2, we see that T5-Base
model with atomic docid representation (blue line plots), undergo significant
forgetting and this trend holds across all DSI evaluation metrics - indexing
accuracy, Hits@1, and Hits@10 (see 5 in Appendix A.2). For the originally
indexed $D_{0}$ corpus, indexing accuracy and Hits@1 drop by approx. $25$ and
$20$ points, respectively. Further, as we continue indexing the sequence of
$D_{1},\cdots,D_{5}$ corpora, we see that forgetting becomes even more severe.
For example after continual indexing the $D_{5}$ corpus, $F_{5}$ (forgetting)
for indexing accuracy increases to $75$. These results provide evidence to
answer (Q1) & (Q2) that the DSI model undergoes severe forgetting under
continual indexing of new documents.

Forward transfer. To answer (Q3), we visualize the learning performance
($LA_{n}$) for all DSI metrics for sequential indexing. From Figure 2, we see
$LA_{n}$ increases for indexing accuracy, suggesting that the DSI model is
plastic enough to index new documents. However, from Figure 2, we see a
declining trend for Hits@1. Due to the continuous indexing updates, the
underlying DSI model drifts and becomes less effective for the retrieval task.
These findings hint at an approach that replays indexing and retrieval tasks
during continual learning (hence our proposed method in §4).

Docid representations. For studying (Q4), we consider unstructured atomic,
naively(N) structured, and semantically(S) structured docid representations.
From Figure 2, we see that T5-Base(N) underperforms T5-Base by a significant
margin. For example, average performance $A_{0}$ for Hits@1 metric is approx.
$30$ and $39$ for naive and atomic docids, respectively. Furthermore, as the
naively structured approach treats unstructured docids as tokenizable strings
as opposed to dedicated unique tokens in the case of atomic docids, they are
relatively more prone to interference from new docids (see $F_{n}$ subplot for
indexing accuracy). Imbuing semantic structure to the naive docid space helps
to reduce forgetting however still underperforms unstructured atomic docids.

Model scale. As atomic docids are superior to naive docids, we only consider
atomic docids for answering (Q5). From Figure 2, we observe that larger models
outperform their smaller counterparts in terms of the average performance
$A_{n}$ and the learning performance $LA_{n}$ (T5-XL $>$ T5-Large $>$
T5-Base). However, empirically we report that forgetting $F_{n}$ is severe
across all model scales, without any clear best performer, and therefore, we
focus on T5-Base for the rest of our experimentation.

##  3 Implicit Forgetting during Memorization: SAM

(a) Indexing accuracy during memorization

(b) Cumulative histogram of forgetting events

Figure 3: Investigating the effectiveness of SAM for alleviating implicit
forgetting in the T5-Base model. (a) We observe serious fluctuations in the
indexing accuracy in the case of the Adafactor optimizer, thereby, suggesting
unstable memorization. SAM leads to relatively stable memorization of the
documents. (b) A forgetting event (Toneva et al., 2019) is defined when an
individual document goes from being classified correctly to incorrectly over
the course of memorization. SAM increases the percentage of examples
experiencing zero forgetting events by absolute $12\%$ over Adafactor.

Memorization (or indexing) is a primary task in the DSI paradigm (Tay et al.,
2022) where the goal is to learn a neural corpus indexer that takes document
content as input and maps it to a document identifier (docid). Under
unstructured atomic docid representation strategy, each docid is assigned a
unique token / class label. Now given the large number of documents in the
corpus (even more than a million), memorization constitutes an instance of
challenging extreme classification setting (Bengio et al., 2019). Furthermore,
for every class, we have only one labeled example (i.e., document and its
identifier), making this task setup rare. Motivated by this largely unexplored
setup, we investigate the learning dynamics for the memorization task over the
course of training.

Forgetting events. In Figure 3a, we visualize the indexing accuracy for
T5-Base model, optimized with Adafactor (Shazeer & Stern, 2018), and note that
the model performance fluctuates a lot over the course of training, thereby
suggesting unstable memorization. We hypothesize that the model continuously
undergoes the forgetting phenomenon wherein subsequent mini-batch updates
interfere with the previously memorized documents. To differentiate this
phenomenon from forgetting due to adding new documents, we refer to earlier
one as implicit forgetting and the latter as explicit forgetting. To quantify
instability during memorization, we compute forgetting event (Toneva et al.,
2019) statistics. Forgetting event is defined when an individual document goes
from being classified correctly (mapped to correct docid) to incorrectly over
the course of memorization. In Figure 3b, we plot the cumulative histogram of
forgetting events and see that almost $88\%$ of the documents undergo
forgetting at least once, thus, validating our hypothesis about implicit
forgetting.

Flatter minima and forgetting. Mirzadeh et al. (2020) shows that during
sequential learning of tasks, geometric properties of the minima for each task
affect forgetting. Specifically, Mirzadeh et al. (2020) derives an upper bound
for forgetting in terms of the maximum eigenvalue $\lambda_{1}^{max}$ of the
Hessian for task loss at minima. Therefore, by lowering $\lambda_{1}^{max}$
(or alternatively promoting flatter minima), one can mitigate forgetting of
previous tasks. Further, Mehta et al. (2021) shows that pre-trained
initialization implicitly alleviate forgetting as they prefer flatter minima
and explicitly optimizing for the flatter loss basins using Sharpness-Aware
Minimization (Foret et al., 2021) (SAM) further lessens forgetting. Based upon
these observations, we hypothesize that modifying the training dynamics of
memorization task using SAM should alleviate implicit forgetting.

Sharpness-Aware Minimization. For the loss function $f$, SAM seeks to find the
parameters $w$ that lie in the neighborhood with uniformly low loss regions by
optimizing the following minimax objective:
$\min_{w}\max_{||\epsilon||_{2}\leq\rho}f(w+\epsilon)$, where the maximization
region is defined to be a $\ell^{p}$ ball with radius $\rho$ for $p=2$. Foret
et al. (2021) estimates the gradient of the inner maximization by employing
first-order approximation as follows:
$\nabla_{w}\max_{||\epsilon||_{2}\leq\rho}f(w+\epsilon)\approx\nabla_{w}f(w)\big{\rvert}_{w+\mathbf{\hat{\epsilon}(w)}}$,
where $\mathbf{\hat{\epsilon}(w)}=\rho\nabla_{w}f(w)/||\nabla_{w}f(w)||_{2}$.
For complete details about this derivation, we defer readers to (Foret et al.,
2021). Building on this work, Bahri et al. (2022) demonstrates successful
applicability of SAM for language model generalization, especially in pre-
trained T5 models. We mainly follow (Bahri et al., 2022) to set our hyper-
parameters: $\rho=0.15$, batch size=$32$ for inner maximization step.

SAM alleviates Implicit forgetting. We investigate the applicability of SAM
for alleviating implicit forgetting phenomenon. Concretely, we use pre-trained
T5-Base model to memorize $D_{0}$ corpus, containing $50$K unique documents.
We compare the performance of the SAM optimizer with the vanilla Adafactor
optimizer. In Figure 3a, we see that SAM outperforms Adafactor in terms of the
overall indexing accuracy. Further, we note that SAM undergoes less severe
fluctuations during the course of training, thus, hinting at lesser
forgetting. To bolster this claim, in Figure 3b, we see that SAM has
significant more percentage of documents corresponding to lower cumulative
number of forgetting events. For example, SAM stably (with zero forgetting
events) memorizes $+12\%$ more documents compared to Adafactor. We also note
that SAM ($35.9\pm 2.2$) outperforms Adafactor ($32.5\pm 6.4$) when evaluated
on the retrieval task (Hits@1) corresponding to $D_{0}$. Therefore, we set SAM
to be our default optimizer for rest of the experiments.

Discussion. Mehta et al. (2021) show that explicitly optimizing for flatter
loss basins using SAM leads to lesser forgetting, especially in task-
incremental learning settings where data undergoes a clear distributional
shift. We extend this work to the new DSI paradigm and convincingly
demonstrate that SAM helps with the stable memorization of documents. Our
results generalize the earlier findings even to the settings where data does
not undergo a clear distributional shift (i.e., memorization task). Although
SAM helps stably memorize documents, there is still room for improvement, and
our work invites more future work in this direction.

##  4 Explicit Forgetting: Generative Memory

DSI paradigm consists of two tasks – memorization and retrieval. In the
previous section, we showcase that SAM alleviates implicit forgetting by
stably memorizing documents. In this section, we focus on the forgetting
phenomenon that arises from the continual indexing of new documents,
specifically in the context of the retrieval task. Through our systematic
study (in §2.4), we show that irrespective of the model scale and docid
representations, DSI models undergo severe forgetting. Moreover, we observe
that the learning performance $LA_{n}$ keeps declining for the retrieval task
(see Figures 2 and 5 for Hits@1 and Hits@10, respectively). This observation
suggests that as we continuously update the DSI model with the indexing
objective, the model starts to forget the retrieval task. In DSI, both
memorization and retrieval tasks return docid for the given input, so by
setup, we can assume access to the contents of the previous documents and
continue indexing both old and new documents with the hope to reduce
forgetting of the retrieval task. However, in Figure 4, we see that the model
still undergoes forgetting (more discussion in §5.3).

Episodic memory. According to the Complementary Learning Systems (CLS)
(McClelland et al., 1995) theory, humans rely on an episodic memory – a module
that stores past experiences to rehearse/ revisit them and thereby retain
previously learned knowledge. Based on this motivation, memory-based
approaches (Sodhani et al., 2022) for continual learning use a subset of
previous task data to regularize the future task learning while minimizing
forgetting. Experience Replay (ER) (Chaudhry et al., 2019b) is one such
approach that samples previous task data (from episodic memory) to co-train
with the current task. ER outperforms other sophisticated memory-based
approaches like Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017),
Averaged GEM (Chaudhry et al., 2019a), even with a sparse replay. Based upon
this line of work, one approach for DSI++ is to retain ground-truth queries
for the retrieval task in an episodic memory and use them to co-train with
incremental indexing task. However, in DSI++, we do not have access to ground-
truth queries for an incoming batch of new documents. Even if one retains
queries for the initial $D_{0}$ corpus, we show in Table 1 that such a method
suffers from forward transfer to newly indexed documents.

Generative memory. Recent years have seen significant progress in the
capabilities of the generative language models (Raffel et al., 2020; Brown et
al., 2020). Motivated by the success of these models and in-applicability of
the episodic memory for DSI++, we pose a questions – instead of retaining the
ground-truth queries, can we learn a parametric model to generate such queries
given a document? Concretely, we propose to train a query generator model to
sample queries for previously seen documents, and supplement them during
incremental indexing. Since we use the generator model to sample queries for
sparse experience replay, hence our proposed method – generative memory.
Moreover, generative memory is also used to generate pseudo-queries for
incoming batch of new documents, thus, enabling continual semi-supervised
learning of the retrieval task.

##  5 Experimentation

Added | Method | Eval corpus = $D_{0}$ | Eval corpus = $D_{1}$
---|---|---|---
corpus |  | (Catastrophic forgetting) | (Forward transfer)
|  | Index acc. | Hits@1 | Hits@10 | Index acc. | Hits@1 | Hits@10
Natural Questions (NQ) – $|D_{0}|=50K$, $|D_{1}|=10K$
$D_{0}$ | - | $81.8_{1.2}$ | $35.9_{2.2}$ | $66.9_{0.9}$ | - | - | -
$D_{1}$ | cl($D_{1}$) | $52.4_{3.5}$ | $19.2_{3.9}$ | $43.6_{5.7}$ | $96.5_{0.0}$ | $31.7_{6.4}$ | $55.6_{4.9}$
cl($U_{1}=D_{0}\cup D_{1}$) | $78.2_{0.5}$ | $28.9_{8.9}$ | $59.0_{7.9}$ | $91.8_{0.4}$ | $34.0_{2.4}$ | $60.2_{1.9}$
cl($U_{1}$)+epsmem($D_{0}$) | $77.8_{0.5}$ | $22.9_{1.5}$ | $51.4_{0.5}$ | $93.1_{0.0}$ | $13.1_{2.1}$ | $39.6_{3.1}$
cl($U_{1}$)+genmem($D_{0}$) | $77.8_{0.3}$ | $26.0_{6.9}$ | $54.9_{8.3}$ | $93.0_{0.5}$ | $8.6_{4.8}$ | $31.6_{11.8}$
cl($U_{1}$)+epsmem($D_{1}$) | $53.2_{3.1}$ | $7.7_{2.1}$ | $26.0_{2.0}$ | $96.5_{0.0}$ | $48.3_{2.3}$ | $70.7_{1.9}$
cl($U_{1}$)+genmem($D_{1}$) | $50.1_{0.8}$ | $7.0_{1.2}$ | $23.1_{2.2}$ | $96.5_{0.0}$ | $57.7_{1.5}$ | $76.7_{0.9}$
cl($U_{1}$)+genmem($U_{1}$) | $78.2_{0.3}$ | $18.4_{2.8}$ | $47.5_{3.9}$ | $92.1_{0.3}$ | $48.5_{6.1}$ | $73.8_{2.9}$
train from scratch | $78.7_{0.6}$ | $35.9_{1.4}$ | $66.4_{0.0}$ | $79.2_{0.3}$ | $32.9_{1.8}$ | $63.9_{1.2}$
MS MARCO – $|D_{0}|=50K$, $|D_{1}|=10K$
$D_{0}$ | - | $99.4_{0.2}$ | $78.2_{0.2}$ | $95.0_{0.1}$ | - | - | -
$D_{1}$ | cl($D_{1}$) | $46.7_{18.6}$ | $68.0_{2.0}$ | $87.3_{1.3}$ | $99.8_{0.0}$ | $36.1_{9.5}$ | $65.8_{6.9}$
cl($U_{1}$) | $99.4_{0.0}$ | $76.5_{0.7}$ | $94.2_{0.3}$ | $99.8_{0.0}$ | $35.3_{4.1}$ | $64.4_{3.3}$
cl($U_{1}$)+genmem($U_{1}$) | $99.3_{0.1}$ | $73.7_{0.2}$ | $93.9_{0.3}$ | $99.8_{0.0}$ | $80.6_{1.0}$ | $95.5_{0.1}$
train from scratch | $99.5_{0.0}$ | $75.0_{0.2}$ | $93.9_{0.1}$ | $99.6_{0.0}$ | $73.4_{1.3}$ | $93.4_{0.9}$
MS MARCO (full) – $|D_{0}|=8M$, $|D_{1}|=842K$
$D_{0}$ | - | $99.4$ | $16.3$ | $46.8$ | - | - | -
$D_{1}$ | cl($D_{1}$) | $0.0$ | $0.1$ | $0.6$ | $97.9$ | $18.2$ | $40.5$
cl($U_{1}$)+genmem($U_{1}$) | $20.4$ | $7.3$ | $31.3$ | $86.6$ | $31.6$ | $65.8$
Table 1: Comparing performance on incremental indexing of $D_{1}$ corpus
across different methods - cl($D_{1}$): continue fine-tuning with indexing
task on $D_{1}$, cl($U_{1}$): continue fine-tuning on the updated corpus
$U_{1}$, cl($U_{1}$)+epsmem($D$): continual indexing of $U_{1}$ along with ER
of queries for $D$, cl($U_{1}$)+genmem($D$): continual indexing of $U_{1}$
along with ER of pseudo-queries for $D$. We observe that continual indexing on
the updated corpus cl($U_{1}$) reduces forgetting compared to just indexing
new corpus cl($D_{1}$) in both NQ and MS MARCO datasets. Next, ER with either
$D_{0}$ or $D_{1}$ hurts forward transfer or forgetting. Our proposed approach
of augmenting pseudo-queries for all documents along with continual indexing,
cl($U_{1}$)+genmem($U_{1}$), alleviates forgetting of $D_{0}$ corpus and
improves forward transfer to $D_{1}$ corpus. We also show that our proposed
solution reduces forgetting of $D_{0}(=8M)$ passages while incrementally
indexing in a large corpus setting, MS MARCO (full) containing $8.9M$
passages.

###  5.1 Implementation details

We utilize the pre-trained T5-Base (Raffel et al., 2020) model to initialize
all models and randomly initialize the additional parameters for atomic docid
tokens. While indexing $D_{0}$ corpus, we train all the models for a maximum
of $1$M steps with a warmup of $100$K steps. During continual indexing of
other corpora, we train for a maximum of $100$K steps with a warmup of $100$
steps. For the rest of hyper-parameters, we follow Tay et al. (2022) – set a
learning rate to $0.001$, batch size to $128$, and input sequence length to
$32$. We evaluate models after every $5$K steps and retain the checkpoint
yielding the best performance. For the initial training with $D_{0}$ corpus,
we co-train on indexing and retrieval tasks, therefore we use the average of
all DSI metrics (indexing accuracy, Hits@1, and Hits@10) for model selection.
For the continual learning experiments, we have access to only indexing
accuracy for all involved corpora and so we use it for model selection. To
train a parametric model for generative memory, we utilize the retrieval
dataset $R_{0}$, which corresponds to the $D_{0}$ corpus. We set the maximum
sequence length for document contents to $1024$, target length for generated
queries to $32$, batch size to $128$, train for a maximum of $100$K steps, and
use BLUE for model selection. We use beam decoding to generate pseudo-queries.
We tune the learning rate amongst $\\{0.001,0.0005\\}$ and linear warmup
amongst $\\{1K,10K\\}$. For all our experiments, we use T5X (Roberts et al.,
2022) framework along with $4$-$8$ TPUv4 chips for training the models.

###  5.2 Methods

We compare our proposed generative memory-based approach with the following
methods:

  * •

continual indexing, cl($\mathbf{D_{n}}$). The DSI model is sequentially fine-
tuned with the indexing objective on the incoming corpus of documents $D_{n}$.

  * •

continual indexing with all seen documents, cl($\mathbf{U_{n}}$). The DSI
model is continuously fine-tuned with the indexing objective on the updated
corpus $U_{n}$ ($=\bigcup_{i=0}^{n}D_{i}$). Also, we sample documents from old
($\bigcup_{i=0}^{n-1}D_{i}$) and new ($D_{n}$) corpora in equal proportion.

  * •

continual experience replay using generative memory, genmem($\mathbf{D_{n}}$).
In this method, the proposed generative memory model is used to sample pseudo-
queries corresponding to the corpus $D_{n}$. Next, these pseudo-queries are
used for (sparse) experience replay of the retrieval task samples.

  * •

continual experience replay using episodic memory, epsmem($\mathbf{D_{n}}$).
In this method, ground-truth queries corresponding to the $D_{n}^{th}$ corpus
are used for experience replay of the retrieval task.

  * •

train from scratch, (no cl). In this method, the DSI model is trained from
scratch every time a new corpus, $D_{n}$, is added. This method corresponds to
non-continual learning setup and is computationally expensive.

###  5.3 Results

In this section, we revisit some of the questions, (Q1)-(Q3), raised in our
case study (see §2.4) to investigate effectiveness of our proposed generative
memory-based approach. Towards answer these questions, in Table 1, we report
the performance of the DSI model on $D_{0}$ (to study forgetting phenomenon)
and $D_{1}$ corpora (to answer forward transfer question) after continual
indexing on $D_{1}$ for both NQ and MS MARCO datasets. In Figures 4 and 6 (NQ)
and Figure 7 (MS MARCO), we report overall performance across DSI metrics as
we continuously update the model with the sequence of five corpora
($D_{1},\cdots,D_{5}$).

Does generative memory alleviate forgetting of old documents? In Table 1, for
the NQ dataset, we report Hits@1 to be $35.9$ for the model after training on
$D_{0}$. We see that continually indexing both $D_{0}$ and $D_{1}$ corpora
(cl($U_{1}$) - $28.9$), significantly reduce forgetting the retrieval task
(Hits@1) over just indexing the new corpora $D_{1}$ (cl($D_{1}$) - $19.2$).
Next, we look at the performance of the ER approaches when augmented with the
continual indexing of all documents. We see that both episodic memory
(cl($U_{1}$)+epsmem($D_{0}$) - $22.9$), and generative memory
(cl($U_{1}$)+genmem($D_{0}$) - 26.0) reduce forgetting compared to cl($D_{1}$)
when we replay (pseudo-)queries corresponding to $D_{0}$ corpus. Moreover,
generative memory outperforms episodic memory, without retaining original
queries. Although from Table 1, we see generative memory,
cl($U_{1}$)+genmem($U_{1}$), underperforms cl($U_{1}$), from Figures 4 and 6,
we see that generative memory, cl($U_{5}$)+genmem($U_{5}$), outperforms
cl($U_{5}$) both in terms of average performance $A_{n}$ and forgetting
$F_{n}$ over five sequential updates. These results convincingly show that the
ER with generative memory significantly alleviates forgetting the retrieval
task compared to considered baselines.

Figure 4: Investigating the effectiveness of generative memory in mitigating
forgetting when continuously indexing new corpus $D_{n}$ (T5-Base model and
atomic docids representation) for the NQ dataset. $\uparrow$ indicates higher
is better, $\downarrow$ indicates lower is better. We observe that continual
indexing of old and new documents cl($U_{n}$) helps to alleviate forgetting of
older documents when evaluated on retrieval tasks. However, average Hits@10
($A_{n}$) still undergo $23$ points drop after sequential updates
($D_{0}\rightarrow D_{1}\cdots\rightarrow D_{5}$). Generative memory enables
sparse replaying of pseudo-queries for old documents and continual semi-
supervised learning with new documents. We observe that augmenting generative
memory during continual indexing not only reduces the forgetting ($F_{n}$) but
also improves average Hits@10 ($A_{n}$) by $+21.1\%$ over considered baselines
(see Figure 6 for Hits@1 results and Figure 7 for MS MARCO results in the
Appendix A.2).

Does generative memory enable forward transfer to new documents? One of the
goals of DSI++ is to enable answering queries related to newly indexed
documents. Towards this goal, in Table 1, for the NQ dataset, we look at the
retrieval task performance (Hits@1) for $D_{1}$ after incrementally indexing
$D_{1}$. To compare different methods, we consider a baseline in the form of
ER with ground-truth queries for $D_{1}$ (cl($U_{1}$)+epsmem($D_{1}$) - 48.3).
We see that without any fine-tuning on the retrieval task for $D_{1}$,
incremental learning with indexing objective shows impressive forward transfer
(or zero-shot gains, cl($D_{1}$) - 31.7 and cl($U_{1}$) - 34.0). Moreover, ER
with generative memory outperforms supervised baseline
(cl($U_{1}$)+genmem($D_{1}$) - 57.7). However, we notice that replaying
queries corresponding to either $D_{0}$ or $D_{1}$ hurt forward transfer to
$D_{1}$ (cl($U_{1}$)+genmem($D_{0}$) - 8.6) or amplify forgetting of $D_{0}$
(cl($U_{1}$)+genmem($D_{1}$) - 7.0), respectively. These results suggest that
the memory module should include (pseudo-)queries corresponding to both old
and new documents. From Figure 4, we see that continual indexing method
cl($U_{n}$) has a downward trend for $LA_{n}$ (Hits@10), therefore, eventually
forgetting the retrieval task. On the other hand, ER with generative memory is
relatively constant, providing evidence against forgetting. In summary, we
show that ER with the generative memory improves the overall performance for
the retrieval task, reducing forgetting of previously indexed documents and
enabling forward transfer to newly indexed documents.

Does generative memory generalize to different datasets? In Table 1, for the
MS MARCO dataset, we report Hits@1 to be $78.2$ after training on $D_{0}$
passages. We see that continually indexing both $D_{0}$ and $D_{1}$ corpora
(cl($U_{1}$) - $76.5$ and cl($U_{1}$)+genmem($U_{1}$) - $73.7$), significantly
reduce forgetting the retrieval task (Hits@1) over just indexing the new
corpora $D_{1}$ (cl($D_{1}$) - $68.0$). Next, we look at the retrieval task
performance (Hits@1) for $D_{1}$ after incrementally indexing $D_{1}$. We see
that without any fine-tuning on the retrieval task for $D_{1}$, incremental
learning with indexing objective shows impressive forward transfer
(cl($D_{1}$) - 36.1 and cl($U_{1}$) - 35.3). Moreover, ER with generative
memory, cl($U_{1}$)+genmem($U_{1}$) - 80.6, performs far superior to just
incremental indexing objective. Similar to the results with the NQ dataset, we
show that ER with generative memory, cl($U_{n}$)+genmem($U_{n}$), improves the
overall performance for the retrieval task, reducing forgetting of previously
indexed documents and enables forward transfer to new documents compared to
continual indexing of all documents, cl($U_{n}$). We show that our results
hold across two datasets – Natural Questions and MS MARCO, thus, showcasing
the generalizability of our approach.

Investigating the effectiveness of the generative memory with the scale of a
corpus. We conduct experiments with a full MS MARCO passage retrieval dataset
containing approx. $8.9M$ passages. We construct two corpora – $D_{0}=8M$ and
$D_{1}=841,823$ passages. We train the DSI model (T5-Base with unstructured
atomic docid representations) using $D_{0}$ passages and incremental add
$D_{1}$ passages. In Table 1, we report our results for MS MARCO (full). We
see that continual fine-tuning with the indexing task on $D_{1}$, cl($D_{1}$),
completely forgets the retrieval task for $D_{0}$ passages (Hits@1 goes to
$0.1$ from $16.3$). However, the generative memory-based approach
significantly reduces forgetting (Hits@1 of $7.3$). Moreover, generative
memory enables continual semi-supervised learning by augmenting pseudo-queries
for $D_{1}$ passages and thereby improving forward transfer (Hits@1 of $31.6$
as compared to $18.2$ for cl($D_{1}$)). We convincingly demonstrate that our
proposed solution also reduces forgetting in a large corpus setting.

Investigating sparsity of experience replay (ER) on forgetting. ER with
generative memory co-trains the indexing and pseudo-labeled retrieval tasks.
Tay et al. (2022) introduces a mixing ratio r to define the ratio of indexing
to retrieval samples. The mixing ratio is inversely related to the sparsity of
ER, i.e., higher $r$ (more indexing samples) corresponds to sparse updates
from pseudo-labeled retrieval samples. Following (Tay et al., 2022), we
consider $r=\\{2,32\\}$ for our analysis. From Figure 4, we see that $r=32$
(sparse replay) slightly outperforms $r=2$ in terms of average performance,
forgetting and learning accuracy. These results suggest that even sparse
regularization updates from ER positively influence the backward and forward
transfer in DSI++.

Analyzing index construction time for DSI++. Index construction using DSI
involves training a Transformer model. DSI++ setup enables incremental
updating of the indexer. In Figures 4, 6, and 7, we show that our proposed
incremental indexer updating method outperforms the “train from scratch”
baseline in terms of the average performance ($A_{n}$). Furthermore, in the
case of the NQ dataset, we require $350$K training steps to index $D_{0}$
corpus of $50$K documents. If we index an additional $D_{1}$ to $D_{5}$
corpora ($10$K each) by re-training the DSI model every time, the total number
of steps would be around $1.75$M. On the other hand, our proposed approach
requires just above $300$K additional updates to incrementally index all five
corpora, almost $6$ times fewer updates. We show that our approach outperforms
re-training the model from scratch in terms of overall performance and is
computationally efficient.

##  6 Related Work

We review relevant prior works along two dimensions – application setups
related to DSI++ and continual learning methods to alleviate forgetting and
enable forward transfer.

Language models (LMs) as knowledge bases (KBs). Petroni et al. (2019) shows
that pre-trained BERT (Devlin et al., 2019) models capture relational
knowledge comparable to that of the KBs constructed using off-the-shelf
techniques. Concretely, these models can be used to extract factual knowledge
about relations between entities by providing a prompt to predict missing
words in a cloze-style template (e.g., “New Delhi is the capital of  ”). In
the same vein, Roberts et al. (2020) demonstrates that pre-trained T5 (Raffel
et al., 2020) models can be employed to answer open-domain questions without
access to any external knowledge or context. However, unlike structured KBs,
it is non-trivial to update knowledge stored implicitly in the weights of
these models. Therefore, Zhu et al. (2020) introduces an experimentation setup
where the task is to update facts stored within the pre-trained models and
proposes a constrained optimization method, similar to Elastic Weight
Consolidation (Kirkpatrick et al., 2017), to alleviate catastrophic
forgetting. With similar motivation, (Dhingra et al., 2022) introduces a
diagnostic dataset to probe LMs for facts that change over time and suggests
jointly modeling text with its timestamp for improved memorization of seen
facts. Recent works have been investigating efficient ways to localize and
edit facts stored with the LMs (AlKhamissi et al., 2022) using finetuning (Zhu
et al., 2020; Dhingra et al., 2022), hyper-networks (De Cao et al., 2021;
Mitchell et al., 2022), and direct editing (Meng et al., 2022). Although a
crucial line of work around updating facts in the pre-trained LMs, using
prompting as our probing mechanism only provides a lower bound estimate of the
knowledge contained in these models (Jiang et al., 2020). On the other hand,
we explicitly focus on the memorization task in DSI++. This task helps us to
answer questions related to catastrophic forgetting more convincingly rather
than bounded by the mechanism of how we probe these models.

Optimization-based approaches for continual learning encode the necessary
inductive biases required to enable continual learning by modifying the
training dynamics. Flatter minima are shown to alleviate forgetting (Mirzadeh
et al., 2020). Further, Mehta et al. (2021) showed that explicitly optimizing
for flatter loss basins using Sharpness-Aware Minimization (SAM; Foret et al.
(2021)) reduces forgetting. Building on these works, we show that flatter
minima induced by SAM reduce implicit forgetting during memorization, thereby
leading to more stable memorization (see §3).

Memory-based (aka data-based regularization) approaches for continual learning
constrain the parameter updates based on the previous task examples sampled
from memory. Sparse experience replay using episodic memory (Chaudhry et al.,
2019b) is a prominent approach and in §4, we discuss its limitations of it for
DSI++. Next, Shin et al. (2017); Sun et al. (2020) learns a parametric model
to reconstruct the examples for seen tasks. However, in DSI++, we do not get
to see queries for the new documents. Therefore, we use a parametric memory to
generate pseudo-queries for already indexed (older) documents as well as an
incoming batch of new documents, thus, enabling us to leverage unlabeled data
(in the form of new documents) for continual semi-supervised learning. On the
other hand, Sun et al. (2020) assume that the incoming data are fully labeled,
which is not applicable in DSI++ (we do not get to see queries for the new
documents). Furthermore, Sun et al. (2020) shows that using a parametric model
underperforms episodic memory. In our work, we do not generate example pairs
$(x,y)$ but rather generate pseudo-queries ($y$), similar to concurrent works
(Zhuang et al., 2022; Bonifacio et al., 2022). We show that our approach
outperforms episodic memory. Lastly, in the context of pseudo-query
generation, neural models are prone to hallucinate additional content not
supported by the input documents, and future works can study methods to filter
out noisy pseudo-queries (Mehta et al., 2022) during incremental indexing.

Test time adaptation approaches for continual learning use episodic memory at
the inference time to alter the model weights before making predictions
(Rebuffi et al., 2017; Sprechmann et al., 2018; de Masson D’Autume et al.,
2019; Wang et al., 2020). Updating the DSI indexer for every user query is
computationally expensive, so we focus on continual learning methods during
training time. Apart from continual learning focused approaches, retrieval
augmented generation (Guu et al., 2020; Izacard & Grave, 2021; Borgeaud et
al., 2021) family of approaches retrieve auxiliary passages/documents to
enhance pre-trained language models. Basically, these approaches alter test-
time predictions of the generative models by augmenting their input with
relevant passages retrieved from external retrievable memory. Moreover, by
using the external retrievable memory, one is explicitly disabling the updates
to the employed pre-trained (and retrieval) model. Such approaches do not
faithfully assess the fundamental challenge of learning continually,
specifically catastrophic forgetting. On the other hand, our work focuses on
the recently introduced DSI paradigm (Tay et al., 2022), where information in
the document corpus is encoded into the model parameters. Therefore, any
updates to the underlying corpus necessitate updating the model parameters
hence, undergoing severe forgetting. Our work tackles a more challenging setup
for studying the forgetting phenomenon in detail. However, retrieval-augmented
generation-based methods do not analyze the forgetting phenomenon at all, only
looking at overall performance metrics. We agree that continual learning is
broader than catastrophic forgetting, however, in this work we decide to study
the forgetting phenomenon in detail on one of the challenging setups, if not
the most challenging.

Parameter isolation-based approaches for continual learning assign different
dedicated subsets of the model parameters to each task to prevent forgetting
(De Lange et al., 2021). While learning a new task, these methods either
freeze a subset of the parameters corresponding to older tasks or dynamically
add new parameters per new task. At the prediction time, these methods
typically require task identity to activate the corresponding subset of
parameters for inference. In the DSI paradigm, we are given user queries at
the inference time, and the goal is to predict relevant document identifiers.
Now during incremental indexing, if we consider every new document corpus as a
new task then a typical parameter isolation-based approach would require
corpus identity for every user query at the test-time, defeating the whole
purpose of the DSI paradigm. Due to this, the parameter isolation-based
approaches in their current form are rendered less useful for DSI++.
Nevertheless, we believe that by masking the weights for the already indexed
corpus, one is explicitly disabling the updates to the underlying DSI model;
therefore, parameter isolation-based methods would be robust to forgetting and
future works should explore them for DSI++.

##  7 Conclusion

DSI++ presents a novel direction for exploration of DSI models to solve one of
the critical requirements for them to be usable in most production setups
where new documents are added to the corpus continuously. Though DSI models
are prone to catastrophic forgetting while we index new documents, we have
shown through extensive experiments that our two proposed solutions to
optimize for the flatter loss basins using SAM and using generative memory
alleviate forgetting to a significant extent. With this work, we lay down
foundations for further research in this space, so that the strategies here do
not just benefit DSI, but the continual learning community in general.

## Ethics Statement

Training large models is expensive, and has a detrimental impact on the
environment (Strubell et al., 2019). Continual learning on top of existing
models is cheaper and better compared to re-training from scratch since it
requires a much smaller number of steps. With DSI++, we aim to reduce the need
to re-train DSI models from scratch whenever a new set of documents is added
to the corpus thereby making it cheaper and better for the environment.

## References

  * AlKhamissi et al. (2022) Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad.  A review on language models as knowledge bases.  _arXiv preprint arXiv:2204.06031_ , 2022.
  * Bahri et al. (2022) Dara Bahri, Hossein Mobahi, and Yi Tay.  Sharpness-aware minimization improves language model generalization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 7360–7371, 2022\.
  * Bengio et al. (2019) Samy Bengio, Krzysztof Dembczynski, Thorsten Joachims, Marius Kloft, and Manik Varma.  Extreme classification (dagstuhl seminar 18291).  In _Dagstuhl Reports_ , volume 8. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2019.
  * Bonifacio et al. (2022) Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira.  Inpars: Data augmentation for information retrieval using large language models.  _arXiv preprint arXiv:2202.05144_ , 2022.
  * Borgeaud et al. (2021) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al.  Improving language models by retrieving from trillions of tokens.  _arXiv preprint arXiv:2112.04426_ , 2021.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chaudhry et al. (2019a) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-GEM.  In _International Conference on Learning Representations_ , 2019a.
  * Chaudhry et al. (2019b) Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato.  On tiny episodic memories in continual learning.  _arXiv preprint arXiv:1902.10486_ , 2019b.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 6491–6506, 2021.
  * De Lange et al. (2021) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(7):3366–3385, 2021.
  * de Masson D’Autume et al. (2019) Cyprien de Masson D’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama.  Episodic memory in lifelong language learning.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT (1)_ , 2019.
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen.  Time-aware language models as temporal knowledge bases.  _Transactions of the Association for Computational Linguistics_ , 10:257–273, 2022.
  * Foret et al. (2021) Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.  Sharpness-aware minimization for efficiently improving generalization.  In _International Conference on Learning Representations_ , 2021.
  * French (1999) Robert M French.  Catastrophic forgetting in connectionist networks.  _Trends in cognitive sciences_ , 3(4):128–135, 1999.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang.  Retrieval augmented language model pre-training.  In _International Conference on Machine Learning_ , pp. 3929–3938. PMLR, 2020.
  * Izacard & Grave (2021) Gautier Izacard and Édouard Grave.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pp. 874–880, 2021.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig.  How can we know what language models know.  _Transactions of the Association for Computational Linguistics_ , 8:423–438, 2020.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 6769–6781, 2020.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466, 2019.
  * Lopez-Paz & Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30, 2017.
  * McClelland et al. (1995) James L McClelland, Bruce L McNaughton, and Randall C O’Reilly.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.  _Psychological review_ , 102(3):419, 1995.
  * McCloskey & Cohen (1989) Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pp. 109–165. Elsevier, 1989.
  * Mehta et al. (2021) Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell.  An empirical investigation of the role of pre-training in lifelong learning.  _arXiv preprint arXiv:2112.09153_ , 2021.
  * Mehta et al. (2022) Sanket Vaibhav Mehta, Jinfeng Rao, Yi Tay, Mihir Kale, Ankur Parikh, and Emma Strubell.  Improving compositional generalization with self-training for data-to-text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 4205–4219, 2022\.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov.  Locating and editing factual associations in GPT.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Mirzadeh et al. (2020) Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh.  Understanding the role of training regimes in continual learning.  _Advances in Neural Information Processing Systems_ , 33:7308–7320, 2020.
  * Mitchell et al. (2022) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning.  Fast model editing at scale.  In _International Conference on Learning Representations_ , 2022.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng.  Ms marco: A human generated machine reading comprehension dataset.  In _CoCo@ NIPs_ , 2016.
  * Ni et al. (2022) Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 1864–1874, 2022.
  * Parisi et al. (2019) German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113:54–71, 2019.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2463–2473, 2019.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67, 2020.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pp. 2001–2010, 2017.
  * Riemer et al. (2019) Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, , and Gerald Tesauro.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  In _International Conference on Learning Representations_ , 2019.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer.  How much knowledge can you pack into the parameters of a language model?  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 5418–5426, 2020.
  * Roberts et al. (2022) Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo.  Scaling up models and data with t5x and seqio.  _arXiv preprint arXiv:2203.17189_ , 2022.
  * Shazeer & Stern (2018) Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pp. 4596–4604. PMLR, 2018.
  * Shin et al. (2017) Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  _Advances in neural information processing systems_ , 30, 2017.
  * Sodhani et al. (2022) Shagun Sodhani, Mojtaba Faramarzi, Sanket Vaibhav Mehta, Pranshu Malviya, Mohamed Abdelsalam, Janarthanan Janarthanan, and Sarath Chandar.  An introduction to lifelong supervised learning.  _arXiv preprint arXiv:2207.04354_ , 2022.
  * Sprechmann et al. (2018) Pablo Sprechmann, Siddhant Jayakumar, Jack Rae, Alexander Pritzel, Adria Puigdomenech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell.  Memory-based parameter adaptation.  In _International Conference on Learning Representations_ , 2018.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in nlp.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 3645–3650, 2019.
  * Sun et al. (2020) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee.  {LAMAL}: {LA}nguage modeling is all you need for lifelong language learning.  In _International Conference on Learning Representations_ , 2020.
  * Tay et al. (2022) Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler.  Transformer memory as a differentiable search index.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Thrun (1995) Sebastian Thrun.  Is learning the n-th thing any easier than learning the first?  _Advances in neural information processing systems_ , 8, 1995.
  * Toneva et al. (2019) Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon.  An empirical study of example forgetting during deep neural network learning.  In _International Conference on Learning Representations_ , 2019.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2020) Zirui Wang, Sanket Vaibhav Mehta, Barnabás Póczos, and Jaime G Carbonell.  Efficient meta lifelong-learning with limited memory.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 535–548, 2020.
  * Zhu et al. (2020) Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar.  Modifying memories in transformer models.  _arXiv preprint arXiv:2012.00363_ , 2020.
  * Zhuang et al. (2022) Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang.  Bridging the gap between indexing and retrieval for differentiable search index with query generation.  _arXiv preprint arXiv:2206.10128_ , 2022.

##  Appendix A Appendix

###  A.1 Dataset

Dataset | #D | Natural Questions (NQ) | MS MARCO
---|---|---|---
|  | #Train | #Validation | #Test | #Train | #Validation | #Test
$R_{0}$ | 50K | 53.8K | 13.5K | 3.9K | 2M | 25.0K | 3.6K
$R_{1}$ | 10K | 10.7K | 2.7K | 809 | 400K | 5.1K | 762
$R_{2}$ | 10K | 10.6K | 2.7K | 787 | 400K | 5.1K | 770
$R_{3}$ | 10K | 10.7K | 2.7K | 727 | 400K | 4.9K | 734
$R_{4}$ | 10K | 10.9K | 2.7K | 772 | 400K | 4.9K | 730
$R_{5}$ | 10K | 10.7K | 2.7K | 847 | 400K | 4.9K | 660
Table 2: DSI++ dataset statistics for NQ and MS MARCO: memorization and
retrieval tasks.

###  A.2 Additional Results

Figure 5: Systematic study about forgetting and forward transfer when
incrementally indexing new corpus of documents across different model sizes
(T5-Base, T5-Large, T5-XL) and docid representations. We use atomic docids by
default and denote (N)/(S) for naively/semantically structured string docids.
$\uparrow$ indicates higher is better, $\downarrow$ indicates lower is better.
We observe that by increasing the model scale, the average $A_{n}$ and
learning $LA_{n}$ performance improves. However, forgetting $F_{n}$ is severe
across all model scales. Moreover, we observe that naive string docids (N)
underperforms atomic docids across Hits@10 metric. Similar to Figure 2,
imbuing the docid space with semantic (S) structure alleviates the forgetting
compared to an arbitrary/ naive (N) structure. Figure 6: Investigating the
effectiveness of generative memory in mitigating forgetting when continuously
indexing new corpus $D_{n}$ (T5-Base model and atomic docids representation)
for the NQ dataset. $\uparrow$ indicates higher is better, $\downarrow$
indicates lower is better. We observe that continual indexing of old and new
documents cl($U_{n}$) help to alleviate forgetting of older documents when
evaluated on retrieval tasks. However, average Hits@1 ($A_{n}$) still undergo
$19$ points drop after sequential updates ($D_{0}\rightarrow
D_{1}\cdots\rightarrow D_{5}$). We observe that by augmenting generative
memory during continual indexing not only reduces the forgetting ($F_{n}$) but
also improves average Hits@1 ($A_{n}$) by $+17.3\%$ over continual indexing.
Figure 7: Investigating the effectiveness of generative memory in mitigating
forgetting when continuously indexing new corpus $D_{n}$ (T5-Base model and
atomic docids representation) for the MS MARCO dataset. $\uparrow$ indicates
higher is better, $\downarrow$ indicates lower is better. We observe that
continual indexing of old and new documents cl($U_{n}$) helps to alleviate
forgetting of older documents when evaluated on retrieval tasks. However,
average Hits@10 ($A_{n}$) still undergo $25.0$ points drop after sequential
updates ($D_{0}\rightarrow D_{1}\cdots\rightarrow D_{5}$). Generative memory
enables sparse replaying of pseudo-queries for old documents and continual
semi-supervised learning with new documents. We observe that augmenting
generative memory during continual indexing not only reduces the forgetting
($F_{n}$) but also improves average Hits@10 ($A_{n}$) by $+23.0\%$ over
considered baselines.