#  ACL 2022
On Continual Model Refinement in Out-of-Distribution Data Streams

Bill Yuchen Lin† Sida Wang‡ Xi Victoria Lin‡
Robin Jia† Lin Xiao‡ Xiang Ren† Wen-tau Yih‡
‡ Facebook AI Research †University of Southern California
{yuchen.lin,robinjia,xiangren}<EMAIL_ADDRESS>The work was done when Bill was an intern at FAIR.

###### Abstract

Real-world natural language processing (NLP) models need to be continually
updated to fix the prediction errors in out-of-distribution (OOD) data streams
while overcoming catastrophic forgetting. However, existing continual learning
(CL) problem setups cannot cover such a realistic and complex scenario. In
response to this, we propose a new CL problem formulation dubbed continual
model refinement (CMR). Compared to prior CL settings, CMR is more practical
and introduces unique challenges (boundary-agnostic and non-stationary
distribution shift, diverse mixtures of multiple OOD data clusters, error-
centric streams, etc.). We extend several existing CL approaches to the CMR
setting and evaluate them extensively. For benchmarking and analysis, we
propose a general sampling algorithm to obtain dynamic OOD data streams with
controllable non-stationarity, as well as a suite of metrics measuring various
aspects of online performance. Our experiments and detailed analysis reveal
the promise and challenges of the CMR problem, supporting that studying CMR in
dynamic OOD streams can benefit the longevity of deployed NLP models in
production. 111Our code and data are available at the project website —
https://cmr-nlp.github.io/.

##  1 Introduction

Fine-tuning large pre-trained language models (LMs) has become the de facto
standard for training models of a variety of tasks in natural language
processing (NLP). These success stories are usually in places where the
training and testing data are drawn from the same distribution. However, in
real-world scenarios, a _deployed_ model (e.g., a question answering service)
often encounters examples that are out of the training distribution (i.e.,
out-of-distribution, OOD). Such distribution shift often leads to a high error
rate. In practice, it is highly preferred to continually refine deployed
models whenever new errors are reported and annotated, in order to reduce
their further negative impacts.

In spite of its importance, the challenge of continually refining a model over
OOD data streams has been underexplored. Prior work in continual learning (CL)
has primarily focused on task-incremental settings with boundary-aware data
streams. These CL methods are usually evaluated on simple models and data
(e.g., image classification with MNIST) Aljundi et al. (2019).

It is not clear to what extent they can efficiently refine a model in
boundary-agnostic streams for a complex language task (e.g., reading
comprehension) with modern LMs. In addition, there is no existing evaluation
protocol for comprehensively comparing the collection of applicable methods
for such a practical and complex problem. Traditional CL paradigms mainly
focus on incrementally learning a model from a data stream with a sequence of
distinct tasks with explicit delineation, which is rather unrealistic in real-
world NLP applications.

To address these research questions, we propose a novel CL formulation named
continual model refinement (CMR), which aims to efficiently update a model for
error correction in an out-of-distribution data stream without
catastrophically forgetting its acquired knowledge over time. In contrast to
prior CL setups, CMR targets learning a model of a particular task (e.g.,
question answering) from its prediction errors in dynamic OOD data streams.
Instead of assuming that the streams are drawn from a fixed unseen
distribution, we study CMR under a more general and realistic scenario, where
the underlying distribution of OOD data streams is non-stationary across time
steps without clear boundaries while being diverse at every time step.

Figure 1: The continual model refinement (CMR) problem. We offline train a
model $f_{0}$ and it may encounter many error cases $E_{t}$ when it is tested
on a stream of query examples $Q_{t}$ over time which is drawn from ever-
changing unseen distributions. A CMR method $g$ aims to fix the error cases
over time by refining $f_{t}$ without catastrophic forgetting.

In this paper, we focus on studying whether existing methods can address CMR
and how we should benchmark and analyze their performance. We first formulate
the CMR problem with several basic metrics covering multiple desiderata for a
CMR method: the ability to instantly fix known errors, the retention of
previously acquire knowledge from upstream/online data, and the generalization
to unseen OOD data (Sec. 2). Then, we propose a general method to create the
dynamic data streams of the aforementioned characteristics and evaluation
metrics to benchmark CMR methods, yielding a comprehensive evaluation protocol
for CMR (Sec. 3). We employ and extend several suitable methods from the CL
literature to study the CMR problem, which is based on parameter
regularization or memory replay (Sec. 4).

We have conducted a comprehensive analysis with extensive experimental
results, which reveal many interesting, non-trivial findings (Section 5). For
example, we find that even though replay methods are generally better than
regularization-based methods, EWC Kirkpatrick et al. (2017), a typical
regularization method, achieves the best score in generalizing to unseen OOD
data. We also find that a simple variant of ranking criteria in conditional
replay methods achieves more stable results. Moreover, we find that different
CMR methods have orthogonal improvements and our positive initial results
suggest that integrating regularization terms for replay methods is a
promising future direction to develop advanced CL methods to address CMR.

Figure 2:  The proposed five basic metrics for evaluating continual model
refinement methods.

##  2 Continual Model Refinement

In this section, we formally introduce the proposed continual learning setup,
continual model refinement (CMR). We first define the notations and describe
the learning objectives that are also illustrated in Fig. 1, then we design a
few basic evaluation metrics for assessing CMR methods, and finally, we
briefly discuss the unique challenges compared to other CL formulations.

###  2.1 Problem Formulation

#### Upstream learning.

Suppose that we want to build a question answering (QA) model. To do this, we
usually need to offline fine-tune a large pre-trained LM with the existing QA
data we have now. Formally, we denote a dataset with $D=\\{(x_{i},y_{i})\\}$,
consisting of the examples are drawn from an upstream distribution
$\mathcal{U}$, i.e., $D\sim\mathcal{U}$. The fine-tuned LM is named upstream
model $f_{0}$.

#### Query streams.

After the model $f_{0}$ is deployed in production, it is common to see ever-
changing distribution shifts in real-world data. We use
$\\{Q_{1},\dots,Q_{T}\\}$ to denote the arriving examples grouped in $T$
episodes and call this sequence of datasets as a query stream. We discuss our
method of creating such challenging query streams for evaluating CMR in Sec.
3.2 and Alg. 1.

#### Error streams.

In real-world scenarios, the size of $Q_{t}$ can be very large even in a short
period of time, and it is unrealistic to assume that we can annotate all of
them to refine the model $f_{t-1}$. A common practice is to only annotate the
ones that are reported as prediction errors or bugs. Motivated by this, we use
$E_{t}$ to denote the examples in $Q_{t}$ that are predicted incorrectly by
$f_{t-1}$. This thus forms an evolving, dynamic stream of prediction errors
$\\{E_{1},\dots,E_{T}\\}$, where $E_{t}=\\{~{}(x,y)\in
Q_{t}~{}~{}|~{}~{}f_{t-1}(x)\neq y~{}\\}$.

#### Learning objectives.

To improve the user satisfaction over time, we need a continual model
refinement (CMR) method $g$ that can efficiently take the model $f_{t-1}$ and
$E_{t}$ as input and then output a refined model $f_{t}$ for processing future
examples. We expect $f_{t}$ to output correct answers for the known errors
$E_{t}$ immediately while maintaining its correct predictions on previous
questions that are answered correctly. We also want the refined models to keep
their generalization ability to unseen future data in the stream. Sec. 2.2
shows the metrics to assess a CMR method $g$ toward these goals.

###  2.2 Basic Evaluation Metrics

We use five metrics to describe the desiderata for CMR methods and assess them
quantitatively, which are illustrated in Figure. 2. We show how to use these
metrics for benchmarking in a comprehensive yet concise way in Sec. 3.3.

####  $\bullet$ Error-fixing rates (EFR).

To assess the responsiveness of the error-fixing methods, we look at how many
errors can be fixed right away. We define the instant error-fixing rate at
time step $t$ as:

$\displaystyle\centering{\operatorname{\textbf{EFR}}(t)=\vcentcolon\operatorname{\textbf{Acc}}(f_{t},E_{t})=\vcentcolon\frac{|\\{(x,y)\in
E_{t}~{}|~{}f_{t}(x)=y\\}|}{|E_{t}|}.}\@add@centering$

####  $\bullet$ Knowledge retention (UKR&OKR).

We define two metrics below to assess how much knowledge acquired from
upstream or online data streams that the model maintains over time:

$\displaystyle\centering{\operatorname{\textbf{UKR}}(t)=\vcentcolon\operatorname{Acc}(f_{t},D)\text{~{}~{}and~{}~{}}\operatorname{\textbf{OKR}}(t)=\vcentcolon\operatorname{Acc}(f_{t},Q_{<t}),}\@add@centering$

where $Q_{<t}$ = $\bigcup^{t-1}_{i=1}Q_{i}$. We down-sample $D$ and $Q_{<t}$
and compute periodically for efficiency.

####  $\bullet$ Cumulative success rates (CSR).

To monitor the model performance on incoming query examples, we compute a
running average of success rates at past time steps:
$\operatorname{\textbf{CSR}}(t)=\vcentcolon 1-|E_{<t}|/|Q_{<t}|$. This can
assess the adaptability of a CMR method.

####  $\bullet$ Knowledge generalization (KG).

As we only have a finite number of episodes for experiments, to assess the
model performance in the future episodes, we test the models with a held-out
set of test examples, $H$, that are drawn from the same underlying
distributions which are used to create the query stream. That is,
$\operatorname{\textbf{KG}}(t)=\vcentcolon\operatorname{Acc}(f_{t},H)$.

###  2.3 Unique Challenges of CMR

Without loss of generality, we suppose that $Q_{t}\sim\mathcal{O}_{t}$, where
$\\{\mathcal{O}_{t}\\}$ denotes an ever-changing series of unseen
distributions. Typical task-incremental CL problem setups such as LAMOL Sun et
al. (2020) and CLIF Jin et al. (2021) consider $Q_{t}$ and $Q_{t+1}$ are
sampled from two distinct tasks. Therefore, the distribution shifts are sudden
(i.e., $\mathcal{O}_{t}$ and $\mathcal{O}_{t+1}$ does not share any
overlapping components).

Also, in conventional CL formulations, the past distribution will never be
revisited, which is rather unrealistic in real-world applications. They do not
have the concept of “error stream” either. Instead, the proposed CMR
formulation is essentially a boundary-agnostic CL problem in non-stationary
data streams, where the distribution shifts are more dynamic, unpredictable,
and diverse, yielding a more realistic yet challenging CL setup.

Figure 3: The left and middle figures are two query streams of the QA task
with different non-stationarity ($\beta=\\{0.1,0.5\\}$) while sharing other
arguments ($T$=50, $b$=64, $\alpha$=0.9, $\gamma$=0.8). We use the blue color
and ‘*’ to denote the in-distribution data cluster (i.e., $V_{0}$), the
percentage of which decay over time. The distribution of incoming query
examples dynamically shifts over time — with larger $\beta$, adjacent episodes
are more likely to share the same major OOD clusters which takes the $\gamma$
of the total OOD data. To encourage diversity, the other $1-\gamma$ OOD
examples are sampled from the remaining clusters. The right figure is the
error stream if we do not refine the upstream model (i.e., $f_{t}\equiv
f_{0}$) and test it on the middle query stream.

##  3 A Comprehensive Evaluation Protocol

We provide a comprehensive evaluation protocol for studying continual model
refinement in OOD streams. This section first briefly describes our selected
task and datasets (Sec. 3.1), then focuses on our proposed method to sample
non-stationary OOD data streams (Sec. 3.2), and finally, illustrate how we use
the basic metrics to benchmark various CMR methods in a comprehensive yet
concise way.

###  3.1 Datasets

In this paper, we mainly use extractive question answering (i.e., machine
reading comprehension) to evaluate and analyze CMR methods, while one could
also study the CMR problem in any NLP tasks with the proposed protocol. We use
the MRQA-19 benchmark Fisch et al. (2019) which consists of 6 datasets sharing
the same formats.

We use the SQuAD Rajpurkar et al. (2016) as the upstream data for offline
training the base LM, and use the other five parts as the OOD data for
continual learning: NQ Kwiatkowski et al. (2019), HotpotQA Yang et al. (2018),
SearchQA Dunn et al. (2017) and TriviaQA Trischler et al. (2017). This is
because SQuAD is more commonly used for deploying models in production and the
real-world QA examples from online users can be more similar to the
distribution of NQ and SearchQA.

###  3.2 Creating Dynamic OOD Data Streams

Here we discuss how to create a realistic ever-changing series of
distributions (i.e., $\\{\mathcal{O}_{t}\\}$ in Sec. 2.3) for creating query
streams $\\{Q_{t}\\}$.

#### Background.

A common practice in CL to create a controllable non-stationary data stream is
to control the context-switching probability. For example, OSAKA Caccia et al.
(2020), as a representative method, uses a Markov chain to sample a sequence
of tasks with a constant transition probability and then sample the examples
from the selected task at each time step. Despite its simplicity, this method
is nevertheless limited to the cases where query stream $Q_{t}$ can only be
drawn from a single distribution, which can be unrealistic.

Instead, it is common that the online data at a time step are from multiple
underlying OOD clusters, each of which has a different distribution, thus
yielding a more diverse and challenging environment for continual model
refinement. Also, it is often that in the early stage of the model deployment,
the query streams still contain examples of the upstream distribution
$\mathcal{U}$, and the proportion of such in-distribution examples will decay
over time.

Input Data Clusters: $V_{0},V_{1},\dots,V_{N}$

Configuration Arguments: $T,b,(\alpha,\beta,\gamma)$.

Output: A query stream $\\{Q_{1},Q_{2},\dots,Q_{T}\\}$

foreach  _$t$ in $\operatorname{range}(1,T)$_ do

$b_{u}=\lfloor b*\alpha^{t-1}\rceil$; $b_{o}=b-b_{u}$ ;
$b_{o}^{\prime}=\lfloor b_{o}*\gamma\rceil$ ;

$c_{t}\sim P(c|c_{t-1};\beta)$

/* The prob. of switching the major OOD data cluster is $1-\beta$, i.e.,
$P(c_{t}\neq c_{t-1})=1-\beta$ */

$V_{\neq c_{t}}=\bigcup_{\\{k\in[1,N]|k\neq c_{t}\\}}V_{k}$

$Q_{t}{~{}\xleftarrow[]{}}\operatorname{sample}(V_{0},b_{u})$

/* $V_{0}\sim\mathcal{U}$; from upstream distribution */

$Q_{t}\text{~{}+=~{}}\operatorname{sample}(V_{c_{t}},b_{o}^{\prime})$

/* from the current major OOD data cluster */

$Q_{t}\text{~{}+=~{}}\operatorname{sample}(V_{\neq
c_{t}},b_{o}-b_{o}^{\prime})$

/* from non-major data clusters */

$\operatorname{\textbf{assert}}~{}|Q_{t}|=b$

Algorithm 1 Sampling query streams with controllable non-stationarity from
multiple data clusters.

#### Our proposed method.

Motivated by these practical considerations, we propose a novel sampling
algorithm to control the dynamics of query streams, aiming to encourage
diversity and model the decaying upstream distribution. We consider that there
are $N$ underlying data clusters, $\\{{V}_{1},\dots,{V}_{N}\\}$, each of which
corresponds to an unseen distribution, and we have ${V}_{0}\sim\mathcal{U}$
which is a data set sampled from the upstream distribution.

Our key motivation is to sample the target $Q_{t}$ from three sources: the in-
distribution data cluster $V_{0}$, the data of a major OOD cluster
$V_{c_{t}}$, and the mix of other remaining OOD data clusters $V_{\neq
c_{t}}$. As shown in Alg. 1, we have three key configuration arguments
$(\alpha,\beta,\gamma)$ for controlling the dynamics of the query stream: 1)
$\alpha$ is the decaying factor for the proportion of in-distribution data, 2)
$\beta$ is the transition probability of the Markov chain for deciding the
index of the major OOD cluster $c_{t}$, and 3) $\gamma$ is to control the
diversity by adding data from remaining OOD clusters; $T$ is the number of
episodes and $b$ is size of $Q_{t}$. Fig. 3 shows examples of query streams
and associated error streams.

Figure 4:  The studied methods for continual model refinement (Section 4).

###  3.3 Benchmarking CMR Methods

#### Overall measurement.

Recall that there are five basic metrics in Section 2.2, namely EFR (instnat
error-fixing rate), UKR (upstream knowledge retention), OKR (online knowledge
retention), CSR (cumulative success rate) and KG (knowledge generalization).
To have a comprehensive yet concise analysis of CMR methods, we report the
average and final values of these metrics. Specifically, we use
$\overline{\text{{X}}}$ to denote the average scores in the metric X (e.g.,
X=UKR) over all time steps, and X${}^{\text{(T)}}$ to denote the score at the
final time step. Reporting both can help us quickly assess the trend of
performance of $f_{t}$ in addition to its final performance. Besides these
fine-grained scores, we also provide an overall evaluation criterion (OEC) by
taking the average of the four scores except for the EFRs222Note that we
report $\overline{\text{{EFR}}}$ scores separately because it computes on the
method-specific errors unlike other metrics that test on same examples for all
CMR methods. , i.e., $\text{OEC}=\operatorname{average}(\text{UKR, OKR, CSR,
KG})$.

#### Validation/testing streams.

To evaluate CMR methods (introduced later in Sec. 4), we use the method in
Alg. 1 to sample multiple streams under the same configurations (i.e.,
$T,b,\alpha,\beta,\gamma$ and $\\{V_{i}\\}$) and then split them as validation
streams and testing streams. The validation streams are used to pick the best
hyper-parameters of each CMR method (e.g., the $\lambda$ of regularization-
based methods and the size of $R_{t}$ in replay methods) and then they are
evaluated on the same set of testing streams.

##  4 Methods

We first introduce our base LM and then illustrate several typical continual
learning methods with our extensions to make them applicable to the CMR
problem. We discuss other relevant yet not suitable methods in Related Work
(Sec. 6). We use Figure 4 to illustrate these CMR methods.

###  4.1 Base Model & Continual Fine-Tuning

#### Base model.

Pretrained text-to-text language models, such as BART Lewis et al. (2020) and
T5 Raffel et al. (2020), are commonly used for studying a wide range of NLP
tasks. This is because they are generally applicable to tasks that can be
formulated as a text-to-text problem, and that they show better generalization
potential Ye et al. (2021); Wei et al. (2021); Sanh et al. (2021). We thus
employ the text-to-text formats to pre-process all data in our experiments and
use BART-base as the base model. We find the BART-base model is a great fit to
support our extensive experiments for its relatively smaller size and
comparable upstream performance versus its alternatives. Thus, we use it for
our experiments to ensure the scalability of our analysis and the generality
of our findings. Note that we do not aim to offline train a perfect upstream
model $f_{0}$ with the upstream dataset $D$. Instead, we focus on the CMR
methods that can continually refine a given upstream model.

#### Continual fine-tuning.

The most straightforward method is to always use a vanilla optimizer (e.g.,
Adam Kingma and Ba (2015)) to fine-tune $f_{t-1}$ with a small learning rate
on $E_{t}$ for a few epochs, aiming to minimize the loss $L_{\text{Error}}(t)$
of fine-tuned model $f_{t}$ on $E_{t}$. Such refined models $f_{t}$ should be
able to output correct outputs for these known errors. This method may overfit
these errors and thus forget previously acquired knowledge. We introduce a few
regularization methods next.

###  4.2 Regularization-based methods

A common solution to preventing forgetting is to add a temporal regularization
term to the loss for continual fine-tuning:
$\mathcal{L}_{\text{total}}(t)=\mathcal{L}_{\text{Error
}}(t)+\lambda\mathcal{L}_{\text{Reg }}(t)$, so that the parameter changes from
$f_{t-1}$ to $f_{t}$ are restricted to avoid over-fitting.

#### Online L2Reg.

We use an intuitive regularization term by computing the L2 distance between
the parameters. That is,

$\mathcal{L}_{\text{L2Reg
}}(t)=\sum_{i}\left(\theta_{t}^{i}-{\theta}_{t-1}^{i}\right)^{2},$

where $\theta_{t}$ is the parameters of $f_{t}$. This regularization term
mitigates the forgetting issue by applying a penalty for every parameter
change.

#### Online EWC.

Elastic weight consolidation Kirkpatrick et al. (2017) is a typical
regularization method for CL. Unlike L2Reg which gives an equal penalty to
every parameter change, EWC produces a weighted penalty such that the
parameters that are more important to the previous tasks will have larger
penalty weights, leading the parameter changes to find an overlapping space
where both previous knowledge and new knowledge can be stored in the
parameters. In particular, it efficiently estimates the Fisher Information
Matrices ${F}_{ii}^{(t)}$ and use them for consolidating the weighted penalty:

$\mathcal{L}_{\text{EWC
}}(t)=\sum_{j=1}^{t-1}\left(\frac{1}{2}\sum_{i}{F}_{ii}^{(j)}\left(\theta_{t}^{i}-{\theta}_{t-1}^{i}\right)^{2}\right).$

We here employ an extension of EWC by keeping a running sum of $F_{ii}$ to
avoid the growth of computation cost in the online setting.

###  4.3 Replay Methods

The other significant group of CL methods is based on replaying past examples,
as follows:

#### Experience replay.

ER Rolnick et al. (2019) is a simple yet effective replay method that stores
the previous examples into a growing memory module $M$. Then, we periodically
(every $k$ time steps) sample a small subset of the memory $R_{t}$ as
additional training examples for model refinement. It uses a two-stage
process: fine-tune $f_{t-1}$ on $R_{t}$ to get $f_{t-1}^{\prime}$ and then
fine-tune $f_{t-1}^{\prime}$ on $E_{t}$ to get $f_{t}$.

#### Maximally interfered replay (MIR).

Instead of randomly selecting $R_{t}$ from $M$, MIR Aljundi et al. (2019) aims
to replay the most forgettable examples, conditioning on the current
information: $f_{t-1}$ and $E_{t}$. It samples a small candidate pool
$C\subset M$ and then ranks the examples in $C$ by their “interference
scores.” Finally, the $R_{t}$ of MIR is the subset of $C$ with the largest
scores. To compute interference scores, we first fine-tune $f_{t-1}$ on
$E_{t}$ to get a virtual model $\hat{f_{t}}$. Then, we compute the loss of
$f_{t-1}$ and $\hat{f_{t}}$ on each example in $C$ to get the interference
scores (i.e., the loss delta):

$\operatorname{score}(x_{i},y_{i})=\vcentcolon\operatorname{loss}(\hat{f_{t}}(x_{i}),y_{i})-\operatorname{loss}(f_{t-1}(x_{i}),y_{i}).$

#### MaxLoss replay.

Inspired by Jiang et al. (2019) and Kawaguchi and Lu (2020) that show learning
with the examples with largest losses can enhance the learning efficiency, we
propose a variant of the MIR by redefining the scoring function to
$\operatorname{score}^{\prime}(x_{i},y_{i})=\vcentcolon\operatorname{loss}(\hat{f_{t}}(x_{i}),y_{i})$
and call it MaxLoss, which takes the examples that have largest losses on the
virtual model $\hat{f_{t}}$ (instead of the largest delta in MIR).

#### Extension for CMR.

(1) Bi-Memory: There are two types of knowledge that we want to maintain in
CMR: the knowledge acquired in upstream and online learning respectively.
Considering that the upstream data is much larger than the incoming errors, it
is thus not reasonable to use a single memory module as in other CL problems.
We thus use two separate memory modules $M_{u}$ and $M_{o}$ where the upstream
memory is $M_{u}=D$ and the online memory $M_{o}$ grows by adding $E_{t}$. (2)
Mixed-Tuning: Instead of following the two-stage method of using $R_{t}$, we
choose to mix $R_{t}$ and $E_{t}$ for fine-tuning $f_{t-1}$. Both
modifications are supported by their better empirical results.

Methods $\downarrow$ Metrics $\rightarrow$ | $\overline{\text{{EFR}}}$ | $\overline{\text{{UKR}}}$ | $\overline{\text{{OKR}}}$ | $\overline{\text{{CSR}}}$ | $\overline{\text{{KG}}}$ | $\overline{\text{{OEC}}}$ | UKR${}^{\text{(T)}}$ | OKR${}^{\text{(T)}}$ | CSR${}^{\text{(T)}}$ | KG${}^{\text{(T)}}$ | OEC${}^{\text{(T)}}$
---|---|---|---|---|---|---|---|---|---|---|---
Frozen Upstream ($f_{t}\equiv f_{0}$) | 0.00 | 80.27 | 43.69 | 44.95 | 31.25 | 50.04 | 80.27 | 36.13 | 35.44 | 31.25 | 45.77
⚫ Continual Fine-Tuning | 97.36 | 72.05 | 83.87 | 55.93 | 45.68 | 64.38 | 66.21 | 77.73 | 53.48 | 48.91 | 61.58
◼ Online L2Reg. | 97.18 | 73.47 | 85.37 | 57.27 | 47.12 | 65.81 | 71.09 | 83.59 | 54.50 | 51.17 | 65.09
▲ Online EWC | 97.49 | 73.38 | 86.09 | 56.17 | 47.34 | 65.75 | 68.55 | 85.74 | 53.67 | 53.28 | 65.31
✚ Exp. Replay (k=3) | 97.07 | 75.30 | 87.29 | 56.02 | 47.61 | 66.55 | 72.46 | 87.30 | 54.08 | 52.66 | 66.63
Experience Replay (k=1) | 96.72 | 78.91 | 89.38 | 57.80 | 47.17 | 68.31 | 78.13 | 86.52 | 55.33 | 52.73 | 68.18
◆ MaxLoss (k=3,c=256) | 97.43 | 75.43 | 86.89 | 57.14 | 46.70 | 66.54 | 75.00 | 84.77 | 55.11 | 51.33 | 66.55
MaxLoss (k=1,c=256) | 96.54 | 78.16 | 89.86 | 57.78 | 46.63 | 68.11 | 77.54 | 89.26 | 55.47 | 50.94 | 68.30
MaxLoss (k=1,c=512) | 97.41 | 75.57 | 87.09 | 56.80 | 46.45 | 66.48 | 77.54 | 89.65 | 55.88 | 52.81 | 68.97
MaxLoss (k=1,c=1024) | 96.63 | 77.61 | 89.82 | 58.13 | 47.10 | 68.17 | 80.66 | 91.02 | 55.88 | 50.78 | 69.59
$\mathbin{\rotatebox[origin={c}]{30.0}{$\text{\char 115}$}}$ MIR (k=3,c=256) | 97.08 | 75.92 | 87.13 | 56.91 | 47.22 | 66.79 | 75.78 | 87.50 | 54.53 | 51.80 | 67.40
MIR (k=1,c=256) | 96.59 | 77.84 | 89.77 | 58.35 | 47.28 | 68.31 | 79.49 | 90.43 | 55.91 | 51.25 | 69.27
MIR (k=1,c=512) | 96.96 | 77.86 | 89.41 | 58.13 | 46.40 | 67.95 | 79.69 | 89.45 | 55.50 | 50.08 | 68.68
MIR (k=1,c=1024) | 96.71 | 77.47 | 87.83 | 57.98 | 46.87 | 67.54 | 78.13 | 87.89 | 55.73 | 50.70 | 68.11
MIR(1,256)+OnlineL2Reg | 96.15 | 79.10 | 90.41 | 59.80 | 47.90 | 69.30 | 79.49 | 90.04 | 57.45 | 52.66 | 69.91
Offline Refining ($f_{0}\rightarrow f_{T}$) | 95.62 | - | - | - | - | - | 83.78 | 93.75 | 93.81 | 56.17 | 81.88
Table 1: Results (%) in multiple metrics: EFR=Error-Fixing Rate;
UKR/OKR=Upstream/Online Knowledge Retention; CSR=Cumulative Success Rate;
KG=Knowledge Generalization. OEC is the average of the last four. Column names
with bars are the average of all periods. The ones with ‘${}^{\text{(T)}}$’
are the scores at the final step. The underlined methods are matched with the
legends in Figure 5. $k$ is the replay interval (the smaller the more
frequent), and $c$ is the size of the candidate pool.  Figure 5: The curves of
four key metrics over time of selected CMR methods in Table 1. The $x$-axis is
the time step.

##  5 Evaluation & Analysis

We first present the setup in Sec. 5.1, and report our main results in Table 1
and Figure 5, which we use to discuss our key findings in Sec. 5.2 to 5.5.

###  5.1 Setup

#### Reference range.

To get a reference range of the performance, we set up two reference methods.
1) FrozenUpstream: We always use the upstream model (i.e., $f_{t}\equiv
f_{0}$) for inference at every time step. 2) OfflineRefining: We combine all
the errors of $f_{0}$ as $E_{\leq T}$ and then offline fine-tune the model
$f_{0}$ with $D^{\prime}+E_{\leq T}$, where $D^{\prime}$ is a subset of $D$,
to directly get the final refined model $f_{T}$ .

#### Hyper-parameters.

We here use a normal configuration of the streams (i.e., $T$=100, $b$=64,
$\alpha$=0.9, $\beta$=0.5, $\gamma$=0.8) for studying the CMR methods and
discuss other extreme configurations briefly in Sec. 5.5 and more in Appendix.
To select the optimal hyper-parameters of each method (e.g., the learning
rate, training epochs, method-specific arguments, etc.), we use grid search
and pick the ones with the best overall score on validation streams.

###  5.2 Main Results and Findings

We report the results in Table 1 & Figure 5, and organize our findings by
answering a coherent list of analysis questions: (Q1-Q7).

####  (Q1) Can we fix errors without forgetting?

From the $\overline{\text{{EFR}}}$ column, we can see that all methods can
achieve a 95+% instant error-fixing rate, meaning that they can indeed quickly
fix most of the known errors. However, they tend to forget the previously
fixed errors and even examples that are correctly predicted before in the
query stream. An oracle method that does not forget the previously acquired
knowledge would have an OKR${}^{\text{(T)}}$ of nearly 100%, while the
OKR${}^{\text{(T)}}$ of the continual fine-tuning method is only $77.7\%$. The
issue of forgetting both online and upstream knowledge in the continual fine-
tuning baseline is quite serious. Notably, its OKR${}^{\text{(T)}}$ is much
lower than its $\overline{\text{{OKR}}}$ (83.87$\rightarrow$77.73), and
similarly for UKR${}^{\text{(T)}}$ and $\overline{\text{{UKR}}}$
(72.05$\rightarrow$66.21). The curves in Figure 5 also suggest that the
forgetting issue can be increasingly more serious over time, and it does not
show any trend to diminish after $T$. This confirms that studying the CMR
problem is of great importance for enhancing deployed NLP models.

####  (Q2) How well do CMR methods mitigate the forgetting issue?

All tested CMR methods can indeed mitigate forgetting without lowering down
the EFRs, but they behave quite differently. The regularization methods (i.e.,
Online L2Reg and Online EWC) are better at improving OKRs rather than UKRs,
while replay methods enhance both OKRs and UKRs quite well. For example,
MaxLoss can achieve the best OKR${}^{\text{(T)}}$($91.0\%$) while having a
UKR${}^{\text{(T)}}$that is even slightly better than the FrozenUpstream model
(80.6 vs 80.3).

Moreover, we find that MaxLoss and MIR have great potential to continually
improve knowledge retention in the future. From both curves in Fig. 5 and
Table 1 (i.e., the comparisons between
$\overline{\text{{UKR}}}$/$\overline{\text{{OKR}}}$ and
UKR${}^{\text{(T)}}$/OKR${}^{\text{(T)}}$), we can see they tend to have
better scores in the later stages, but the retention scores of regularization-
based methods are decreasing over time. We have a detailed discussion on
replay-based methods in Q4.

####  (Q3) Can refined models generalize to unseen OOD data?

Recall that CSRs evaluate the incoming yet not touched examples over time in
the stream and the KGs evaluate the held-out examples that are not in the
stream. Both metrics thus test on OOD examples that are unseen to the refined
model at that time. Compared to the FrozenUpstream baseline, we see all
methods have large performance gains (from 30% to 50+% in
CSR${}^{\text{(T)}}$and KG${}^{\text{(T)}}$). The “MIR w/ Online L2Reg” even
achieves the best CSR${}^{\text{(T)}}$ and it is significantly better than
others, showing that learning with replay effectively improves the
generalization ability.

From the $\overline{\text{{KG}}}$ and KG${}^{\text{(T)}}$ columns of these CMR
methods (and Fig. 5), we can see that refined models are increasingly more
generalizable to held-out unseen data over time as well. However, the
differences among these methods in these two metrics are not obvious, although
they are all better than the continual fine-tuning baseline. Interestingly,
the regularization method OnlineEWC gets the best score of
KG${}^{\text{(T)}}$, even though its CSR${}^{\text{(T)}}$ is worse than
others. This suggests that learning with replay might hurt the held-out
knowledge generalization, but regularization could maintain a better
generalization ability in the long run.

###  5.3 Analysis on Memory Replaying

####  (Q4) How should we replay the memory?

We find that increasing the replay frequency (i.e., setting a smaller replay
interval $k$) can largely improve the overall performance for ER, MaxLoss, and
MIR. This is expected as there are more fine-tuning steps over the retrieved
data.

However, the reason for such improvement varies among them. Increasing the
replay frequency primarily benefits ER’s UKR${}^{\text{(T)}}$, but not for
other metrics, and it even causes a lower OKR${}^{\text{(T)}}$. Instead,
MaxLoss and MIR also benefit from larger OKR${}^{\text{(T)}}$ (MaxLoss: 84.77
$\rightarrow$ 89.26; MIR: 87.50 $\rightarrow$ 90.43). This suggests that
conditional replay methods can get more important stored memory to replay than
ER’s random selections.  Thus, it is promising to develop more advanced
conditional replay methods for CMR.

####  (Q5) Are larger buffer sizes always better for conditional replay
methods?

Larger buffer sizes (i.e., c=256 $\rightarrow$ 512 $\rightarrow$ 1024) can
increase MaxLoss’s UKR${}^{\text{(T)}}$ and OKR${}^{\text{(T)}}$with a large
margin and thus produce better overall scores. However, MIR with larger buffer
sizes suffers from decreasing UKR${}^{\text{(T)}}$ and OKR${}^{\text{(T)}}$.
This indicates that that delta of loss as the ranking criteria is less stable
than using the virtual loss itself (i.e., MaxLoss).

This finding conflicts with the MIR experiments on MNIST-based task-aware
streams Aljundi et al. (2019). We thus conjecture it is because our streams
are more complex and the loss landscapes of the task are significantly
different from the toy datasets used for evaluation in many prior CL works
(e.g., image classification over shuffled MNIST).

Figure 6:  The differences between refined models produced by different CMR
methods in terms of their predictions for the same inputs at two time spans
(10-20 and 30-40). The darker cells have large discrepancy.

###  5.4 Orthogonal Improvement for CMR

####  (Q6) Do different CMR methods produce similar refined models?

We use Figure 6 to visualize the differences among the refined models produced
by selected CMR methods in two different periods. We can see the refined
models by continual fine-tuning (CFT) and regularization methods are more
similar to each other, and all replay methods are quite distinct from other
methods. Also, the divergence among different methods rapidly increases from
$t=[10,20]$ to $t=[30,40]$. Therefore, we believe that the improvement of
these CMR methods is orthogonal to each other, especially between
regularization and replay methods.

####  (Q7) Can we integrate regularization and replay methods?

Inspired by Fig. 6 and findings in (Q3), we add an initial experiment by
combining the MIR and OnlineL2Reg and show its performance in Table 1.
Interestingly, we indeed observe this combination produces a noticeable
improvement over both MIR and OnlineL2Reg, yielding the state-of-the-art
performance in OEC${}^{\text{(T)}}$ scores. To the best of our knowledge,
there is little prior work that has studied the effect of integrating
regularization in (conditional) replay methods, and our initial results
suggest that this is a very promising direction for future research.

###  5.5 Additional Analysis

Our above analysis is based on the results of a normal stream configuration
(i.e., $\alpha$=0.9, $\beta$=0.5, $\gamma$=0.8), but can such tuned hyper-
parameters of CMR methods directly apply to streams of extreme configurations?
In Table 2, we briefly compare the gain of the previous CMR methods in terms
of their OCE${}^{\text{(T)}}$ improvement over the vanilla FrozenUpstream
baseline under a few extreme settings of We find that, in general, all replay
methods are still better than continual fine-tuning and Online EWC. ER shows
more stable results in extreme settings (e.g., $\beta$= 0.1 or 0.9) but MIR
and MaxLoss (MxLs) are more sensitive to the non-stationarity yet less
sensitive to the diversity.

Stream Dynamics | CFT | EWC | ER | MxLs | MIR
---|---|---|---|---|---
$\alpha$=0.9, $\beta$=0.5, $\gamma$=0.8 | 15.81 | 19.54 | 20.86 | 20.78 | 21.63
$\alpha$=0.9, $\beta$=0.1, $\gamma$=0.8 | 23.40 | 24.14 | 26.32 | 26.05 | 26.04
$\alpha$=0.9, $\beta$=0.9, $\gamma$=0.8 | 18.61 | 19.38 | 20.78 | 19.51 | 20.60
$\alpha$=0.9, $\beta$=0.5, $\gamma$=0.5 | 19.97 | 20.10 | 21.97 | 23.01 | 22.04
$\alpha$=0.9, $\beta$=0.5, $\gamma$=0.2 | 17.37 | 16.22 | 19.15 | 20.60 | 19.45
Table 2: The gain of OEC${}^{\text{(T)}}$over the Frozen Upstream baseline for
each method under different stream dynamics.

##  6 Related Work

#### Continual Learning for NLP.

Recently, continual learning (or lifelong learning) has drawn attention in the
NLP field Biesialska et al. (2020); Sun et al. (2020); Wang et al. (2019);
Huang et al. (2021); Jin et al. (2021). However, most of these works follow
the traditional task-incremental, boundary-aware, never-revisiting CL setup,
which is not directly beneficial to most of the real-world scenarios of
deployed NLP models. For example, the CLIF formulation Jin et al. (2021)
focuses on learning over a sequence of different NLP tasks with few-shot data
so that the trained model can generalize better to unseen tasks. In contrast,
the proposed CMR in this work is a particularly novel CL setup where we focus
on continually refining a model with its prediction errors in OOD data
streams, thus yielding a boundary-agnostic, dynamically non-stationary
environment for CL methods to work. Such fundamental differences between CMR
and traditional CL setups make it difficult to directly apply many CL methods
that are based on boundary-aware streams, especially for those who require
learning task representations.

#### CMR vs. OSAKA

The OSAKA Caccia et al. (2020) problem is similar to the CMR in that we both
focus on CL in non-stationary boundary-agnostic data streams. However, it does
not consider the distribution diversity inside each time step or the decay of
upstream distribution in the online setting. Our sampling method (Alg. 1)
fills the gap and yields a more realistic CL setup. In addition, the data
streams of CMR are always the prediction errors of the latest model, thus
producing a naturally evolving and adversarial environment for CL methods to
explore. Moreover, the experiments of OSAKA are limited to simple networks and
tasks such as MNIST, but our work uses pre-trained Transformer LMs and the QA
task, and thus we believe our analysis and findings are more useful for the
NLP community and beyond.

#### Model Refinement.

Model refinement has recently become an emerging topic in NLP, but existing
works have mainly been limited to offline editing time-sensitive factual
knowledge in pre-trained LMs Zhu et al. (2020); De Cao et al. (2021); Mitchell
et al. (2021). In contrast, our work studies the model refinement in an online
continual learning setting and for downstream NLP tasks such as reading
comprehension and natural language inference. Jang et al. (2021) attempt to
study the knowledge editing problem at a larger scale, but its problem
formulation only contains two time-steps, thus being significantly different
from CMR. Dhingra et al. (2021) propose a simple method to jointly model text
with its timestamp so that the trained language models can be calibrated when
new knowledge arrives, while CMR focuses on the error cases from OOD data
streams where the timestamps have little correlation with the skills we want
the deployed model to learn. Besides, Yao et al. (2021) propose a method of
learning from explanations to fix prediction errors, which shares similar
high-level motivation but has few direct connections to our focus in this
work.

##  7 Conclusion & Future Directions

In this paper, we propose a novel continual learning formulation named
continual model refinement (CMR). The CMR problem aims to efficiently fix
prediction errors when learning in out-of-distribution data streams without
catastrophically forgetting the acquired knowledge. For studying such a
realistic and complex problem, we presented a dedicated evaluation protocol
with a general method to create non-stationary, diverse OOD data streams for
analysis. Also, we design multiple evaluation metrics to deliver a
comprehensive yet concise measurement of CMR methods.

The proposed CMR problem with our comprehensive analysis opens up a range of
new opportunities for studying continual learning problems that are closer to
real-world applications for the NLP community and beyond. For example, based
on our results and analysis about (Q3) and (Q6), we find that it is promising
to study how we can integrate both regularization methods and replay methods
for mitigating the forgetting issue while improving the generalization
ability. The analysis about (Q5) suggests that developing more stable ranking
criteria is also important to conditional replay methods (e.g., our simple
extension MaxLoss can outperform MIR under specific settings). Developing CMR
methods of which the configurations can generalize to diverse types of streams
is also an important challenge. We release our codebase and processed datasets
for supporting the reproducibility of our experiments and future research.

## References

  * Aljundi et al. (2019) Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. 2019.  Online continual learning with maximally interfered retrieval.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , Red Hook, NY, USA. Curran Associates Inc.
  * Biesialska et al. (2020) Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020.  Continual lifelong learning in natural language processing: A survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6523–6541, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Caccia et al. (2020) Massimo Caccia, Pau Rodríguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, and Laurent Charlin. 2020.  Online fast adaptation and knowledge accumulation (OSAKA): a new approach to continual learning.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6491–6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dhingra et al. (2021) Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2021.  Time-aware language models as temporal knowledge bases.  _ArXiv preprint_ , abs/2106.15110.
  * Dunn et al. (2017) Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017.  Searchqa: A new q&a dataset augmented with context from a search engine.  _ArXiv preprint_ , abs/1704.05179.
  * Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019\.  MRQA 2019 shared task: Evaluating generalization in reading comprehension.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 1–13, Hong Kong, China. Association for Computational Linguistics.
  * Huang et al. (2021) Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021.  Continual learning for text classification with information disentanglement based regularization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2736–2746, Online. Association for Computational Linguistics.
  * Jang et al. (2021) Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021.  Towards continual knowledge learning of language models.  _ArXiv preprint_ , abs/2110.03215.
  * Jiang et al. (2019) Angela H. Jiang, Daniel L.-K. Wong, Giulio Zhou, David G. Andersen, Jeff Dean, Gregory R. Ganger, Gauri Joshi, Michael Kaminsky, Michael A. Kozuch, Zachary Chase Lipton, and Padmanabhan Pillai. 2019.  Accelerating deep learning by focusing on the biggest losers.  _ArXiv preprint_ , abs/1910.00762.
  * Jin et al. (2021) Xisen Jin, Bill Yuchen Lin, Mohammad Rostami, and Xiang Ren. 2021.  Learn continually, generalize rapidly: Lifelong knowledge accumulation for few-shot learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 714–729, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kawaguchi and Lu (2020) Kenji Kawaguchi and Haihao Lu. 2020.  Ordered SGD: A new stochastic optimization framework for empirical risk minimization.  In _The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]_ , volume 108 of _Proceedings of Machine Learning Research_ , pages 669–679. PMLR.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114:3521 – 3526\.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Mitchell et al. (2021) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021.  Fast model editing at scale.  _ArXiv preprint_ , abs/2110.11309.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rolnick et al. (2019) David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. 2019.  Experience replay for continual learning.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 348–358.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021.  Multitask prompted training enables zero-shot task generalization.
  * Sun et al. (2020) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020.  LAMOL: language modeling for lifelong language learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
  * Wang et al. (2019) Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019.  Sentence embedding alignment for lifelong relation extraction.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 796–806, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021.  Finetuned language models are zero-shot learners.  _ArXiv preprint_ , abs/2109.01652.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yao et al. (2021) Huihan Yao, Ying Chen, Qinyuan Ye, Xisen Jin, and Xiang Ren. 2021.  Refining neural networks with compositional explanations.  _ArXiv preprint_ , abs/2103.10415.
  * Ye et al. (2021) Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021.  CrossFit: A few-shot learning challenge for cross-task generalization in NLP.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7163–7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhu et al. (2020) Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020.  Modifying memories in transformer models.  _ArXiv preprint_ , abs/2012.00363.

##  Appendix A Implementation Details

###  A.1 Upstream Learning

We use the huggingface’s implementation of Transformer architectures for
running BartForCondtionalGeneration. Note that we choose to use this seq2seq
head instead of the BartForQuestionAnswering for the seq2seq version can
support a much wider range of NLP tasks as long as they can be converted in to
text-to-text formats (e.g,. CrossFit Ye et al. (2021) and FLAN Wei et al.
(2021)). Also, we find that the results using seq2seq formats is comparable to
using the span extraction for reading comprehension (at least for SQuAD).
Therefore, we choose to use seq2seq format to encourage the generality of our
released codebase. We use BART-base for all our experiments and here we
present the final hyper-parameters we used for upstream learning: lr=5e-5,
train_bsz=64, pred_bsz=64, num_epochs=30.

We have also tried to use BART-Large for running our experiments and analysis.
Our preliminary results show that our general findings still hold such as Q1
to Q3. But running BART-Large causes around 5 times slower speed for our
experiments. Considering the scale of our grid search and our analysis as well
as the negative impact to the environment, we choose to focus on using BART-
base for all our experiments and analysis. We believe future works for CMR can
also benefit from this due to the fact that using BART-base can help them
quickly analyze the performance. Also, as we seek to test different CMR
methods instead of different base LMs, we think using BART-base can represent
a reasonable scope of similar LMs that are widely used in the community such
as T5-base, etc.

###  A.2 Details for CMR Methods

#### Datasets.

We refer to the MRQA 2019 homepage for more detailed statistics of each
dataset: https://github.com/mrqa/MRQA-Shared-Task-2019. Particularly, we use
the SQuAD-train as $D$, the upstream data, and SQuAD-dev as $V_{0}$ (the
upstream data cluster); Also, other devs as $\\{V_{1},\dots,V_{N}\\}$ as the
OOD data clusters. We also tried to use NQ as the upstream data and it shows a
similar performance trend as we discussed in the main table.

#### Validation/Test Streams

We sampled 32 validation streams and 8 test streams for all our experiments,
shown in Table 1. We searched the hyperparameters (hps) of all CMR methods on
the set of validation streams and then pick the best one for each method by
measuring their average of OEC${}^{\text{(T)}}$and EFR${}^{\text{(T)}}$on the
sum of all validation streams. The results are based on the average of all
test streams, where for each stream we run each method with 5 different random
seeds, yielding 40 rounds of experiments for each CMR method (a reason why we
choose to use BART-base).

#### Continual Fine-Tuning

There are two major hps: the learning rate and the num_epochs, we searched
over $\\{1e-5,2e-5,\dots,5e-5\\}$ and $\\{5,10,15,20,30\\}$ for the num_epochs
at each episode. We use the mini-batch size of $8$ for fine-tuning the
$f_{t-1}$ on $E_{t}$ at each time step. Our final choices are lr=3e-5 and
num_epochs=20.

#### OnlineL2Reg

There is one additional hp: the $\lambda$, the weight of the L2 penalty. We
searched it from $\\{1,5,10,20,50,100\\}$ on top of the hps of the CFT and
finally decide to use $\lambda=10$.

#### OnlineEWC

Please refer to the original paper Kirkpatrick et al. (2017) for the details
of the online version. Therefore, we also have two hps $\lambda_{\text{EWC}}$
and $\gamma_{\text{EWC}}$, which we searched over $\\{1,5,10\\}$ and
$\\{1.0,0.95,0.9,0.8\\}$. We finally use $5$ and $0.9$ for their best
performance.

#### Replay Methods

For all replay methods, we first search them with the best hps using CFT and
then run them together with the same size of replay examples $|R_{t}|=32$
which we found perform the best. Their $k$ and $c$ are compared in Table 1.

We leave more details of the MIR and MaxLoss implementation in our codebase.

###  A.3 Computational Cost

Replay methods (with best searched hps) are slightly more expensive than
continual learning methods. Online L2Reg needs to store the weight of the
previous model checkpoint and compute the L2 distance, and OnlineEWC is more
expensive than OnlineL2Reg because computing the Fisher also needs a virtual
model learning step and storing the running sum of the previously stored
matrices. The replay based methods store all raw data in memory. ER is the
most cheap because it does not need any local adaptation (i.e., virtual model
update) for ranking. MIR and MaxLoss are almost equally expensive for ranking,
and they both use the same lr and epochs of CFT for virtual learning.


# US-China perspectives on extreme AI risks and global governance

Akash R. Wasil111Author correspondence<EMAIL_ADDRESS>
Georgetown University
Tim Durgin
Independent

###### Abstract

The United States and China will play an important role in navigating safety
and security challenges relating to advanced artificial intelligence. We
sought to better understand how experts in each country describe safety and
security threats from advanced artificial intelligence, extreme risks from AI,
and the potential for international cooperation. Specifically, we compiled
publicly-available statements from major technical and policy leaders in both
the United States and China. We focused our analysis on advanced forms of
artificial intelligence, such as artificial general intelligence (AGI), that
may have the most significant impacts on national and global security. Experts
in both countries expressed concern about risks from AGI, risks from
intelligence explosions, and risks from AI systems that escape human control.
Both countries have also launched initial efforts designed to promote
international cooperation around safety standards and risk management
practices. Notably, our findings only reflect information from publicly
available sources. Nonetheless, our findings can inform policymakers and
researchers about the state of AI discourse in the US and China. We hope such
work can contribute to policy discussions around advanced AI, its global
security threats, and potential international dialogues or agreements to
mitigate such threats.

##  1 Introduction

Artificial intelligence is a transformative technology with major implications
for national and global security. Many AI experts have expressed concerns
about AI-related national and global security threats. Examples include risks
from AI-enabled biological weapons, risks from autonomous AI systems that
escape human control, and risks from AI applied in military operations (Bengio
et al., 2024; Hendrycks et al., 2023).

The United States and China are the world’s leaders in AI development. In
2022, the size of the AI market in the US amounted to USD 103.7B (Statista-1,
2024), while the size of the Chinese AI market was approximately USD 40B
(Statista-2, 2024). As of 2022, 26 percent of the world’s top AI researchers
came from China, while 28 percent came from the United States (Yang 2024). US
companies dominate the top of the large language model (LLM) rankings:
examples include Open AI’s Chat-GPT, Anthropic’s Claude, and Google’s Gemini
(Guinness, 2024). While most Chinese LLMs lag quite a bit behind, there has
been progress in recent years. Specifically, Moonshot AI’s Kimi model, under
certain conditions, can achieve performance comparable to Chat GPT-4 (Zhang,
2023).

We aimed to acquire a better understanding of AI policy discourse in China and
the United States. Discourse about US-China relations is often partisan,
rather than grounded in objective data. Discussions about US-China AI policy
and potential international agreements should be rooted in a concrete
understanding of attitudes toward AI, safety and security, and international
cooperation in both countries. To improve our understanding of these topics,
we compiled publicly available statements from Chinese and US officials
regarding artificial intelligence, safety and security risks, and
international cooperation and global governance. We focus our analysis on
statements regarding the most powerful forms of artificial intelligence (e.g.,
artificial general intelligence or“AGI”) and the most extreme risks posed by
AI (e.g., developing biological weapons, developing novel kinds of weapons of
mass destruction, or escaping human control.)

##  2 Methodology

We included publicly available material about extreme AI risks, advanced AI,
safety and security challenges, and international governance. Our approach was
semi-structured and not meant to be systematic or comprehensive. For both
searches, we focused on statements that acknowledged extreme risks from AI or
AI safety and security threats– notably, our aim was not to characterize
debates about the likelihood or plausibility of such risks. Below, we offer
more details about our approach for both the China search and the US search.

When searching for statements by Chinese leaders, we first identified commonly
used terms for extreme AI risks. We started with 通用人工智能, the most commonly
used word in Chinese for General Artificial Intelligence. Using searches with
this keyword, we identified additional keywords such as 生存风险 (existential
risk) and 对齐 (alignment). We prioritized the inclusion of statements from
government officials and officials from high-ranking academic institutions
that regularly advise government officials. When we identified a relevant
statement with an English translation, we used the English translation.
Otherwise, the 2nd author (who is fluent in Chinese) translated the
statements. Since our analysis was focused on Mainland China, we searched for
the keyword terms in Simplified Chinese. When searching for statements by US
leaders, we followed a similar procedure. We focused on statements from
policymakers or statements given to government bodies (e.g., testimony to
Congressional hearings).

When determining which statements to include, we prioritized statements by
senior faculty at top universities, experts working at frontier AI companies,
policymakers and government officials, and officials who would likely advise
policymakers. We were also most interested in characterizing statements
relating to extreme risks related to advanced AI, safety and security, and the
potential for international cooperation or global governance.

##  3 Results

###  3.1 China

####  3.1.1 Safety and security

When examining the discourse among Chinese scholars and policymakers, there is
a clear acknowledgment of long-term security risks from advanced AI
development. The discourse includes descriptions of specific safety and
security concerns– such as intelligence explosions, the potential for self-
replication and deception, and the importance of AGI security. Chinese
government officials are concerned about AI’s potential to generate
“unhealthy” or “illegal” information. Finally, some influential minority of
Chinese scholars have expressed concerns about potential existential risks
created by advanced AI.

Example quotes (bolding added):

  * •

“If there is an intelligence explosion once AI has evolved to a certain level,
the default result will inevitably be catastrophic.” – Paper by Scientists
from Peking University’s Dept. of Computer Science222Peking University is one
of the top two universities in China.

  * •

“[Referring to AI] Every technology is like nuclear technology– if humankind
had a choice, perhaps it would be best not to discover dangerous materials
like uranium and radium.” – Ya-Qin Zhang, director of the Tsinghua Institute
for AI Industry Research (Zhang, Ya-Qin, 2023)

  * •

“Service providers must pay close attention to the generative AI’s potential
long-term risks. They must treat with caution any artificial intelligence
(models) with the ability to deceive humans, self-replicate, or self-modify.”
– Gen. AI Security Requirements instituted by the National Information
Security Standardization Technical Committee of China (TC260)333The National
Information Security Standardization Technical Committee of China (often
abbreviated as TC260) is China’s leading organization for writing national
standards related to cybersecurity. (TC260, 2024)

  * •

“(AI systems) must use keywords, classification models, and other methods to
monitor user input. If the user makes 3 inputs in a row that contain illegal,
unhealthy informatio and/or are obviously designed to induce the AI system to
generate unhealthy information, or if the user makes a total of 5 such inputs
in one day, approved, legal methods shall be used to suspend said user’s (AI)
services.”– TC260 Gen. AI Security Requirements (TC260, 2024)

  * •

“Main security Risks of AI Corpus and AI Generated Content: Content that
violates core Socialist values, including content which: a) incites the
subversion of state power and/or the overthrow of the Socialist System; b)
harms national security, the national interest, and/or the nation’s image; c)
incites the dissolution of the state, damages national unity and/or social
stability; d) promotes terrorism and/or other extremist ideologies.” – TC260
Gen. AI Security Requirements (TC260, 2024)

  * •

“If AIA (AI Actants) evolve to the point where they need to defend their
“right to survive” and do not hesitate to harm human beings in order to
protect themselves, they will become autonomous systems that can fight against
human beings, which will lead to disasters for humanity.” – Bingxing Fang,
Director of Guangzhou University’s Network Security Technology National
Engineering Laboratory (Fang, 2020)

  * •

”The superintelligence of the future may see humans as humans see ants today,
and if humans can’t treat other types of life with kindness, why should the
superintelligence of the future treat humans with kindness? The biggest
bottleneck to whether humans and AI can coexist in the future lies in humans,
not AI.” – Yi Zeng, Director of the International Research Center for AI
Ethics and Governance at the Chinese Academy of Sciences (Zeng 2023).

####  3.1.2 International cooperation and global governance

When examining discourse on international cooperation and global governance in
China, we observed several expressions of interest in global cooperation on AI
and risk management. More specifically, the Chinese government has expressed a
desire to promote tiered testing systems with different requirements based on
AI risk levels. Additionally, China is very concerned about ensuring equal
access to AI technology across global borders.

Example quotes:

  * •

“We should promote the establishment of a testing and assessment system based
on AI risk levels, implement agile governance, and carry out tiered and
category-based management for rapid and effective response.” – China’s AI
Global Governance Initiative (Embassy of the People’s Republic of China, 2023)

  * •

“We support discussions within the United Nations framework to establish an
international institution to govern AI.” – China’s AI Global Governance
Initiative (Embassy of the People’s Republic of China, 2023)

  * •

“We call for global collaboration to foster the sound development of AI, share
AI knowledge, and make AI technologies available to the public under open-
source terms.” – China’s AI Global Governance Initiative (Embassy of the
People’s Republic of China, 2023)

  * •

“AI must be fair. All countries should be able to participate in AI governance
and share in its benefits; no country should be left behind.” – China’s AI
Global Governance Initiative (Embassy of the People’s Republic of China, 2023)

  * •

“AI research and governance need the participation and cooperation of people
in different fields all over the world. In addition to those engaged in AI R&D
and use, they also need the participation of people in different fields such
as law, morality and ethics. We need to clarify the standards of ethics and
morality and what it means to be ‘moral’ and ‘ethical’.” – Bo Zhang Professor,
Tsinghua University’s Department of Computer Science444Tsinghua University is
one of the top 2 universities in China. (Zhang, 2022)

###  3.2 United States

####  3.2.1 Safety and Security

Discourse among American scholars and policymakers also includes clear
acknowledgment of long-term security risks from advanced AI development. AI
experts in the US have expressed strong concern about potential catastrophic
outcomes from advanced AI. More recently, policymakers have begun to discuss
artificial general intelligence and its risks. Many US policymakers and AI
experts recognize the potential for extreme risks, both from malicious use of
advanced AI and from potential loss of control scenarios. US policymakers have
also expressed an interest in learning more about artificial general
intelligence, standardizing the definition of AGI, understanding the
likelihood and magnitude of AGI risks, and implementing policies to prepare
for AGI risks.

Example quotes:

  * •

“There is a spectrum of problems we could face related to this, at the extreme
end of which [are] concerns about whether a sufficiently powerful AI, without
appropriate safeguards, could be a threat to humanity as a whole–referred to
as existential risk. Left unchecked, highly autonomous, intelligent systems
could also be misused or simply make catastrophic mistakes.” – Dario Amodei,
testifying before the Senate Judiciary Committee (Amodei, 2023).

  * •

“The third possibility, which could emerge in as little as a few years, is
that of loss of control, when an AI is given a goal that includes or implies
maintenance of its own agency, which is equivalent to a survival objective… If
the AI is misspecified, powerful enough, and exploits a loophole in its goals,
the consequences could be unforeseen and severe. Therefore, a reactive
approach to mitigating misspecified goals could be extremely costly for
society, and we may only have a few chances of getting the alignment right for
superhuman AI.” – Yoshua Bengio, testifying before the Senate Judiciary
Committee (Bengio, 2023).

  * •

“As long as there are really thoughtful people, like Dr. Hinton or others, who
worry about the existential risks of artificial intelligence—the end of
humanity—I don’t think we can afford to ignore that… Even if there’s just a
one in a 1000 chance, one in a 1000 happens. We see it with hurricanes and
storms all the time.” – Representative Beyer (Henshall, 2024).

  * •

“The AI Working Group recognizes that there is not widespread agreement on the
definition of AGI or threshold by which it will officially be achieved.
Therefore, we encourage the relevant committees to better define AGI in
consultation with experts, characterize both the likelihood of AGI development
and the magnitude of the risks that AGI development would pose, and develop an
appropriate policy framework based on that analysis.” – Senators Schumer,
Rounds, Heinrich, and Young (Bipartisan Senate AI Working Group, 2024).

  * •

“Artificial intelligence (AI) has the potential to dramatically improve and
transform our way of life, but also presents a spectrum of risks that could be
harmful to the American public, some of which could have catastrophic effects.
Extremely powerful frontier AI could be misused by foreign adversaries,
terrorists, and less sophisticated bad actors to cause widespread harm and
threaten U.S. national security. Experts from across the U.S. government,
industry, and academia believe that advanced AI could one day enable or assist
in the development of biological, chemical, cyber, or nuclear weapons.” –
Senators Romney, Reed, Moran, and King in the Framework for Mitigating Extreme
AI Risks (Romney et al., 2024).

####  3.2.2 International cooperation and global governance

When examining discourse in the United States around international cooperation
and global governance, we observed an interest in international cooperation on
AI standards and risk management practices. For example, a recent bill has
outlined specific ways that the US can cooperate with international partners
on safety standards, and the recently established US AI Safety Institute has
highlighted international cooperation in its strategic vision.

Example quotes:

  * •

“[This bill] directs the heads of Commerce, State and the Office of Science
and Technology Policy (“OSTP”) to jointly seek to form alliances or coalition
with like-minded countries to cooperate on approaches to innovation in AI and
to coordinate and promote the development and adoption of common AI standards…
[the bill] ensures that participating countries maintain adequate research
security measures, intellectual property protections, safety standards, and
risk management approaches.” – Bill summary of the Future of AI Innovation Act
(Senate Committee on Commerce, Science, and Transportation, 2024).

  * •

“[The US AI Safety Institute aims to] lead an inclusive, international network
on the science of AI safety. AI safety practices must be globally adopted to
the greatest extent possible. We intend to serve as a partner for other AI
Safety Institutes, national research organizations, and multilateral entities
like the OECD and G7. We intend to work with our partners to foster commonly
accepted scientific methodologies with the intention of developing a shared
and interoperable suite of AI safety evaluations and agreed-upon risk
mitigations. In doing so, we hope to help develop the science and practices
that underpin future arrangements for international AI governance.” – US AI
Safety Institute (NIST, 2024)

###  3.3 Joint statements

####  3.3.1 Safety and security

Joint statements including representatives from the United States and China
have acknowledged safety and security threats from AI. Several of these
statements were made in the context of international summits or track II
dialogues.

Example quotes:

  * •

“Particular safety risks arise at the frontier of AI. Substantial risks may
arise from potential intentional misuse or unintended issues of control
relating to alignment with human intent.” (Bletchley Declaration, 2023)555The
Bletchley Declaration was signed jointly by the governments of the United
States, China, and 27 other nations.

  * •

“Unsafe development, deployment, or use of AI systems may pose catastrophic or
even existential risks to humanity within our lifetimes. These risks from
misuse and loss of control could increase greatly as digital intelligence
approaches or even surpasses human intelligence.” (International Dialogues on
AI Safety, 2024)666Major signatories to the IDAIS statement include: Andrew
Yao (Dean of Institute for Interdisciplinary Information Sciences, Tsinghua
University), Ya-Qin Zhang (Chair Professor of AI Science at Tsinghua
University), HongJiang Zhang (Chairman of the Beijing Academy of AI), Stuart
Russell (Electrical Engineering and Computer Science, UC Berkeley), and Yoshua
Bengio (Professor, Department of CS and Operations Research, Université de
Montréal).

  * •

“Coordinated global action on AI safety research and governance is critical to
prevent uncontrolled frontier AI development from posing unacceptable risks to
humanity.” (International Dialogues on AI Safety, 2023)

  * •

“We call on leading AI developers to make a minimum spending commitment of
one-third of their AI R&D on AI safety and for government agencies to fund
academic and non-profit AI safety and governance research in at least the same
proportion.” (International Dialogues on AI Safety, 2023).

  * •

“Safe, secure and trustworthy artificial intelligence systems… are such that
they are human-centric, reliable, explainable, ethical, inclusive, in full
respect, promotion and protection of human rights and international law,
privacy-preserving, sustainable development-oriented, and responsible.”
(United Nations General Assembly, 2024).

####  3.3.2 International cooperation and global governance

Joint statements including representatives from the United States and China
have also recognized the importance of global cooperation to avoid some of the
most catastrophic outcomes of advanced AI development. Such statements have
emphasized that risks arising from AI are “inherently international in nature”
and suggested that governments should work together to define “clear red lines
that, if crossed, mandate immediate termination of an AI system.”

Example quotes:

  * •

“In the depths of the Cold War, international scientific and governmental
coordination helped avert thermonuclear catastrophe. Humanity again needs to
coordinate to avert a catastrophe that could arise from unprecedented
technology.” (International Dialogues on AI Safety, 2024).

  * •

“Many risks arising from AI are inherently international in nature, and so are
best addressed through international cooperation.” (Bletchley Declaration,
2023).

  * •

“We also recommend defining clear red lines that, if crossed, mandate
immediate termination of an AI system — including all copies — through rapid
and safe shut-down procedures. Governments should cooperate to instantiate and
preserve this capacity. Moreover, prior to deployment as well as during
training for the most advanced models, developers should demonstrate to
regulators’ satisfaction that their system(s) will not cross these red lines.”
(International Dialogues on AI Safety, 2023)

##  4 Discussion

We compiled statements about advanced AI discourse in China and the United
States, with a focus on statements about safety and security risks and
international cooperation. Overall, major technical officials and policymakers
in both countries have acknowledged extreme risks from advanced AI
development. More specifically, experts in both China and the United States
seem to have a shared understanding of some of the concrete safety and
security challenges. For example, top Chinese scientists discuss concepts like
“intelligence explosions”, risks from systems that can “self-replicate or
self-modify”, and risks from systems that “do not hesitate to harm human
beings” in order to survive. American scientists and policymakers have also
acknowledged these risks, noting that AI could produce “loss of control”
scenarios and calling for more work on “the risks that AGI development would
pose.”

There were also some noteworthy differences. For example, the Chinese
discourse includes discussions about how AI may produce content that does not
conform to the Chinese Communist Party’s restrictions on speech. Experts have
discussed how AI might help users generate “content that violates Socialist
values”, such as content that can “incite the subversion of state power” or
“overthrow the Socialist System.” Chinese standards for AI security also note
that users should be suspended if they try to generate “illegal, unhealthy
information.” To highlight another difference, the United States discourse
included details about the role of the US AI Safety Institute in advancing the
science of AI safety and setting international standards. Several countries–
including the United States and United Kingdom– recently agreed to establish
an international network of AI safety institutes to “forge a common
understanding of AI safety” and share information about model capabilities and
risks (UK Government, 2024). So far, China is not part of this agreement and
has not yet announced plans for a formal AI safety institute.

Our work should be interpreted in light of a few important limitations. First,
our work did not aim to be a comprehensive summary of AI discourse. We focused
on a particular subset of the discourse (discourse related to safety and
security concerns around advanced AI development) among a particular group of
stakeholders (policymakers and technical experts who inform policymakers).
Future work could examine discourse around other topics (e.g., bias and
fairness, US-China competition, military applications of AI) among a wider
array of stakeholders (e.g., members of the general public, investors, civil
society groups). Second, our methodology was subjective– we did not aim to
perform a comprehensive literature review or statistically examine trends in
the entire corpus of discourse. Rather, we aimed to provide brief and
illustrative findings that can offer readers a general understanding of the
discourse. Future work could aim to be more comprehensive or quantitative.
Third, our work only included publicly available statements– we do not provide
insights into private discussions among policymakers. Finally, it should be
noted that AI discourse can change rapidly in response to new events. As AI
progress continues and policymakers’ understanding of AI increases, we should
suspect important changes in how policymakers in each country view the safety
and security risks.

Overall, we hope our work provides some useful insights into the state of
advanced AI discourse in China and the United States. Such work could
contribute to a broader body of literature that helps policymakers better
prepare for the global security implications of advanced AI and inform
discussions about global governance strategies.

## References

  * (1)

Amodei, Dario. (2023). Written Testimony for a hearing on Oversight of A.I.:
Principles for Regulation.
https://www.judiciary.senate.gov/imo/media/doc/2023-07-26_-_testimony_-
_amodei.pdf

Bengio, Yoshua. (2023). Written Testimony for a hearing on Oversight of A.I.:
Principles for Regulation.
https://www.judiciary.senate.gov/imo/media/doc/2023-07-26_-_testimony_-
_bengio.pdf

Bipartisan Senate AI Working Group. (2024). A Roadmap for Artificial
Intelligence Policy in the United States Senate.
https://www.schumer.senate.gov/imo/media/doc/Roadmap_Electronic1.32pm.pdf

Bletchley Declaration. (2023). The Bletchley Declaration by Countries
Attending the AI Safety Summit, 1-2 November 2023.
https://www.gov.uk/government/publications/ai-safety-summit-2023-the-
bletchley-declaration/the-bletchley-declaration-by-countries-attending-the-ai-
safety-summit-1-2-november-2023

Department for Science, Innovation and Technology. (2024). Global leaders
agree to launch first international network of AI Safety Institutes to boost
cooperation of AI [Press release]. https://www.gov.uk/government/news/global-
leaders-agree-to-launch-first-international-network-of-ai-safety-institutes-
to-boost-understanding-of-ai

Embassy of the People’s Republic of China. (2023). Global AI Governance
Initiative. http://gd.china-
embassy.gov.cn/eng/zxhd_1/202310/t20231024_11167412.htm

Fang, Bingxing. (2020). Artificial Intelligence Safety and Security.
https://chineseperspectives.ai/Binxing-FANG

Guinness, Harry (2024). The Best Large Language Models of 2024.
https://zapier.com/blog/best-llm/

International Dialogues on AI Safety. (2023). IADS Ditchley Park 2023.
https://idais.ai/

Henshall, W. (2024, February 21). How a New Bipartisan Task Force is Thinking
About Artificial Intelligence. TIME. https://time.com/6727264/house-
artificial-intelligence-task-force/

International Dialogues on AI Safety. (2024). IADS Beijing 2024.
https://idais.ai/

Ministry of Foreign Affairs of the People’s Republic of China. (2024). Wang Yi
on Global AI Governance: Ensure that AI is a Force for Good, Ensure Safety and
Ensure Fairness.
https://www.fmprc.gov.cn/eng/zxxx_662805/202403/t20240308_11256430.html#:~:text=Wang%20Yi%20said%20artificial%20intelligence,be%20checked%20before%20setting%20off

Statista (2024). Artificial Intelligence - United States.
https://www.statista.com/outlook/tmo/artificial-intelligence/united-states

National Institute of Standards and Technology. (2024). The United States
Artificial Intelligence Safety Institute: Vision, Mission, and Strategic
Goals. https://www.nist.gov/system/files/documents/2024/05/21/AISI-
vision-21May2024.pdf

Romney et. al. (2024). Framework for Mitigating Extreme AI Risks.
https://www.romney.senate.gov/wp-content/uploads/2024/04/AI-
Framework_2pager.pdf

Senate Committee on Commerce, Science, and Transportation. (2024). The Future
of AI Innovation Act of 2024: Section-by-Section Summary.
https://www.commerce.senate.gov/services/files/DE2C57C7-7F3C-4D0F-9DA3-FB66E15E9A8A

Statista (2024). Artificial Intelligence - China.
https://www.statista.com/statistics/1262377/china-ai-market-size/ TC260.
(2024). Basic Security Requirements for Generative Artificial Intelligence
Services. https://www.tc260.org.cn/upload/2024-03-01/1709282398070082466.pdf

United Nations General Assembly. (2024). A/78/L.49.
https://documents.un.org/doc/undoc/ltd/n24/065/92/pdf/n2406592.pdf?token=PWMWxny2akiFYYDipm&fe=true

Yang, Zeyi. (2024). Four things you need to know about China’s AI talent pool.
MIT Technology Review. https://www.technologyreview.com/2024/03/27/1090182/ai-
talent-global-china-
us/#:~:text=1.,industry%20to%20absorb%20that%20talent.%E2%80%9D

Yuqing, et. al. (2021), Technical Countermeasures for Security Risks of
Artificial Intelligence. Strategic Study of CAE 2021, Volume 23, Issue 3.
https://www.engineering.org.cn/en/article/29491/detail

Zeng, Yi. (2023). Chinese Perspectives on AI Safety.
https://chineseperspectives.ai/yi-zeng

Zhang, Bo. (2022). Academician Zhang Bo: Make Responsible AI.
https://chineseperspectives.ai/Bo-ZHANG

Zhang, Irene. (2023). Putting China’s Top LLMs to the Test. Chinatalk.
https://www.chinatalk.media/p/putting-chinas-top-llms-to-the-test

Zhang, Ya-Qin. (2023). Embracing AI by Prioritizing Values over Technology.
Internet Weekly, Issue 15, 2023. https://chineseperspectives.ai/Ya-qin-Zhang