# Beat the AI: Investigating Adversarial Human Annotation
for Reading Comprehension

Max Bartolo Alastair Roberts Johannes Welbl Sebastian Riedel Pontus Stenetorp
Department of Computer Science
University College London
<EMAIL_ADDRESS>
###### Abstract

Innovations in annotation methodology have been a catalyst for Reading
Comprehension (RC) datasets and models. One recent trend to challenge current
RC models is to involve a model in the annotation process: humans create
questions adversarially, such that the model fails to answer them correctly.
In this work we investigate this annotation methodology and apply it in three
different settings, collecting a total of 36,000 samples with progressively
stronger models in the annotation loop. This allows us to explore questions
such as the reproducibility of the adversarial effect, transfer from data
collected with varying model-in-the-loop strengths, and generalisation to data
collected without a model. We find that training on adversarially collected
samples leads to strong generalisation to non-adversarially collected
datasets, yet with progressive performance deterioration with increasingly
stronger models-in-the-loop. Furthermore, we find that stronger models can
still learn from datasets collected with substantially weaker models-in-the-
loop. When trained on data collected with a BiDAF model in the loop, RoBERTa
achieves 39.9F${}_{\text{1}}$ on questions that it cannot answer when trained
on SQuAD – only marginally lower than when trained on data collected using
RoBERTa itself (41.0F${}_{\text{1}}$).

Figure 1: Human annotation with a model in the loop, showing: i) the “Beat the
AI” annotation setting where only questions that the model does not answer
correctly are accepted, and ii) questions generated this way, with a
progressively stronger model in the annotation loop.

##  1 Introduction

Data collection is a fundamental prerequisite for Machine Learning-based
approaches to Natural Language Processing (NLP). Innovations in data
acquisition methodology, such as crowdsourcing, have led to major
breakthroughs in scalability and preceded the “deep learning revolution”, for
which they can arguably be seen as co-responsible Deng et al. (2009); Bowman
et al. (2015); Rajpurkar et al. (2016). Annotation approaches include expert
annotation, for example, relying on trained linguists Marcus et al. (1993),
crowd-sourcing by non-experts Snow et al. (2008), distant supervision Mintz et
al. (2009); Joshi et al. (2017), and leveraging document structure Hermann et
al. (2015). The concrete data collection paradigm chosen dictates the degree
of scalability, annotation cost, precise task structure (often arising as a
compromise of the above) and difficulty, domain coverage, as well as resulting
dataset biases and model blind spots Jia and Liang (2017); Schwartz et al.
(2017); Gururangan et al. (2018).

A recently emerging trend in NLP dataset creation is the use of a _model-in-
the-loop_ when composing samples: A contemporary model is used either as a
filter or directly during annotation, to identify samples wrongly predicted by
the model. Examples of this method are realised in _Build It Break It, The
Language Edition_ Ettinger et al. (2017), HotpotQA Yang et al. (2018a), SWAG
Zellers et al. (2018), Mechanical Turker Descent Yang et al. (2018b), DROP Dua
et al. (2019), CODAH Chen et al. (2019), Quoref Dasigi et al. (2019), and
AdversarialNLI Nie et al. (2019).111 The idea was alluded to at least as early
as Richardson et al. (2013), but it has only recently seen wider adoption.
This approach probes model robustness and ensures that the resulting datasets
pose a challenge to current models, which drives research to tackle new sets
of problems.

We study this approach in the context of RC, and investigate its robustness in
the face of continuously progressing models – do adversarially constructed
datasets quickly become outdated in their usefulness as models grow stronger?

Based on models trained on the widely used SQuAD dataset, and following the
same annotation protocol, we investigate the annotation setup where an
annotator has to compose questions for which the model predicts the wrong
answer. As a result, only samples that the model fails to predict correctly
are retained in the dataset – see Figure 1 for an example.

We apply this annotation strategy with three distinct models in the loop,
resulting in datasets with 12,000 samples each. We then study the
reproducibility of the adversarial effect when retraining the models with the
same data, as well as the generalisation ability of models trained using
datasets produced with and without a model adversary. Models can, to a
considerable degree, learn to generalise to more challenging questions, based
on training sets collected with both stronger and also weaker models in the
loop. Compared to training on SQuAD, training on adversarially composed
questions leads to a similar degree of generalisation to non-adversarially
written questions, both for SQuAD and NaturalQuestions Kwiatkowski et al.
(2019). It furthermore leads to general improvements across the model-in-the-
loop datasets we collect, as well as improvements of more than
20.0F${}_{\text{1}}$ for both BERT and RoBERTa on an extractive subset of DROP
Dua et al. (2019), another adversarially composed dataset. When conducting a
systematic analysis of the concrete questions different models fail to answer
correctly, as well as non-adversarially composed questions, we see that the
nature of the resulting questions changes: Questions composed with a model in
the loop are overall more diverse, use more paraphrasing, multi-hop inference,
comparisons, and background knowledge, and are generally less easily answered
by matching an explicit statement that states the required information
literally. Given our observations, we believe a model-in-the-loop approach to
annotation shows promise and should be considered when creating future RC
datasets.

To summarise, our contributions are as follows: First, an investigation into
the model-in-the-loop approach to RC data collection based on three
progressively stronger models, together with an empirical performance
comparison when trained on datasets constructed with adversaries of different
strength. Second, a comparative investigation into the nature of questions
composed to be unsolvable by a sequence of progressively stronger models.
Third, a study of the reproducibility of the adversarial effect and the
generalisation ability of models trained in various settings.

##  2 Related Work

#### Constructing Challenging Datasets

Recent efforts in dataset construction have driven considerable progress in
RC, yet datasets are structurally diverse and annotation methodologies vary.
With its large size and combination of free-form questions with answers as
extracted spans, SQuAD1.1 Rajpurkar et al. (2016) has become an established
benchmark that has inspired the construction of a series of similarly
structured datasets. However, mounting evidence suggests that models can
achieve strong generalisation performance merely by relying on superficial
cues – such as lexical overlap, term frequencies, or entity type matching Chen
et al. (2016); Weissenborn et al. (2017); Sugawara et al. (2018). It has thus
become an increasingly important consideration to construct datasets that RC
models find challenging, and for which natural language understanding is a
requisite for generalisation. Attempts to achieve this non-trivial aim have
typically revolved around extensions to the SQuAD dataset annotation
methodology. They include unanswerable questions Trischler et al. (2017);
Rajpurkar et al. (2018); Reddy et al. (2019); Choi et al. (2018), adding the
option of “Yes” or “No” answers Dua et al. (2019); Kwiatkowski et al. (2019),
questions requiring reasoning over multiple sentences or documents Welbl et
al. (2018); Yang et al. (2018a), questions requiring rule interpretation or
context awareness Saeidi et al. (2018); Choi et al. (2018); Reddy et al.
(2019), limiting annotator passage exposure by sourcing questions first
Kwiatkowski et al. (2019), controlling answer types by including options for
dates, numbers, or spans from the question Dua et al. (2019), as well as
questions with free form answers Nguyen et al. (2016); Kočiský et al. (2018);
Reddy et al. (2019).

#### Adversarial Annotation

One recently adopted approach to constructing challenging datasets involves
the use of an adversarial model to select examples that it does not perform
well on, an approach which superficially is akin to active learning Lewis and
Gale (1994). Here, we make a distinction between two sub-categories of
adversarial annotation: i) _adversarial filtering_ , where the adversarial
model is applied offline in a separate stage of the process, usually after
data generation; examples include SWAG Zellers et al. (2018), ReCoRD Zhang et
al. (2018), HotpotQA Yang et al. (2018a), and HellaSWAG Zellers et al. (2019);
ii)  _model-in-the-loop adversarial annotation_ , where the annotator can
directly interact with the adversary during the annotation process and uses
the feedback to further inform the generation process; examples include CODAH
Chen et al. (2019), Quoref Dasigi et al. (2019), DROP Dua et al. (2019),
FEVER2.0 Thorne et al. (2019), AdversarialNLI Nie et al. (2019), as well as
work by Dinan et al. (2019), Kaushik et al. (2020), and Wallace et al. (2019)
for the Quizbowl task.

We are primarily interested in the latter category, as this feedback loop
creates an environment where the annotator can probe the model directly to
explore its weaknesses and formulate targeted adversarial attacks. Although
Dua et al. (2019) and Dasigi et al. (2019) make use of adversarial annotations
for RC, both annotation setups limit the reach of the model-in-the-loop: In
DROP, primarily due to the imposition of specific answer types, and in Quoref
by focusing on co-reference, which is already a known RC model weakness.

In contrast, we investigate a scenario where annotators interact with a model
in its original task setting – annotators must thus explore a range of natural
adversarial attacks, as opposed to filtering out “easy” samples during the
annotation process.

##  3 Annotation Methodology

###  3.1 Annotation Protocol

The data annotation protocol is based on SQuAD1.1, with a model in the loop,
and the additional instruction that questions should only have one answer in
the passage, which directly mirrors the setting in which these models were
trained.

Formally, provided with a passage $p$, a human annotator generates a question
$q$ and selects a (human) answer $a_{h}$ by highlighting the corresponding
span in the passage. The input $(p,q)$ is then given to the model, which
returns a predicted (model) answer $a_{m}$. To compare the two, a word-overlap
F${}_{\text{1}}$ score between $a_{h}$ and $a_{m}$ is computed; a score above
a threshold of $40\%$ is considered a “win” for the model.222 This threshold
is set after initial experiments to not be overly restrictive given acceptable
answer spans, e.g., a human answer of “New York” vs. model answer “New York
City” would still lead to a model “win”.  This process is repeated until the
human “wins”; Figure 2 gives a schematic overview of the process. All
successful $(p,q,a_{h})$ triples, that is, those which the model is unable to
answer correctly, are then retained for further validation.

Figure 2: Overview of the annotation process to collect adversarially written
questions from humans using a model in the loop.  Figure 3: “Beat the AI”
question generation interface. Human annotators are tasked with asking
questions about a provided passage which the model in the loop fails to answer
correctly.

###  3.2 Annotation Details

#### Models in the Annotation Loop

We begin by training three different models, which are used as adversaries
during data annotation. As a seed dataset for training the models we select
the widely used SQuAD1.1 Rajpurkar et al. (2016) dataset, a large-scale
resource for which a variety of mature and well-performing models are readily
available. Furthermore, unlike cloze-based datasets, SQuAD is robust to
passage/question-only adversarial attacks Kaushik and Lipton (2018). We will
compare dataset annotation with a series of three progressively stronger
models as adversary in the loop, namely BiDAF Seo et al. (2017),
BERT${}_{\text{LARGE}}$ Devlin et al. (2019), and RoBERTa${}_{\text{LARGE}}$
Liu et al. (2019b). Each of these will serve as a model adversary in a
separate annotation experiment and result in three distinct datasets; we will
refer to these as $\mathcal{D_{\mathrm{BiDAF}}}$,
$\mathcal{D_{\mathrm{BERT}}}$, and $\mathcal{D_{\mathrm{RoBERTa}}}$
respectively. Examples from the validation set of each are shown in Table 1.
We rely on the _AllenNLP_ Gardner et al. (2018) and _Transformers_ Wolf et al.
(2019) model implementations, and our models achieve EM/F${}_{\text{1}}$
scores of 65.5%/77.5%, 82.7%/90.3% and 86.9%/93.6% for BiDAF, BERT, and
RoBERTa respectively on the SQuAD1.1 validation set, consistent with results
reported in other work.

BiDAF |  Passage: […] the United Methodist Church has placed great emphasis on the importance of education. As such, the United Methodist Church established and is affiliated with around one hundred colleges […] of Methodist-related Schools, Colleges, and Universities. The church operates three hundred sixty schools and institutions overseas.
---|---
Question: The United Methodist Church has how many schools internationally?
BiDAF |  Passage: In a purely capitalist mode of production (i.e. where professional and labor organizations cannot limit the number of workers) the workers wages will not be controlled by these organizations, or by the employer, but rather by the market. Wages work in the same way as prices for any other good. Thus, wages can be considered as a […]
Question: What determines worker wages?
BiDAF |  Passage: […] released to the atmosphere, and a separate source of water feeding the boiler is supplied. Normally water is the fluid of choice due to its favourable properties, such as non-toxic and unreactive chemistry, abundance, low cost, and its thermodynamic properties. Mercury is the working fluid in the mercury vapor turbine […]
Question: What is the most popular type of fluid?
BERT |  Passage: […] Jochi was secretly poisoned by an order from Genghis Khan. Rashid al-Din reports that the great Khan sent for his sons in the spring of 1223, and while his brothers heeded the order, Jochi remained in Khorasan. Juzjani suggests that the disagreement arose from a quarrel between Jochi and his brothers in the siege of Urgench […]
Question: Who went to Khan after his order in 1223?
BERT |  Passage: In the Sandgate area, to the east of the city and beside the river, resided the close-knit community of keelmen and their families. They were so called because […] transfer coal from the river banks to the waiting colliers, for export to London and elsewhere. In the 1630s about 7,000 out of 20,000 inhabitants of Newcastle died of plague […]
Question: Where did almost half the people die?
BERT |  Passage: […] was important to reduce the weight of coal carried. Steam engines remained the dominant source of power until the early 20th century, when advances in the design of electric motors and internal combustion engines gradually resulted in the replacement of reciprocating (piston) steam engines, with shipping in the 20th-century […]
Question: Why did steam engines become obsolete?
RoBERTa |  Passage: […] and seven other hymns were published in the Achtliederbuch, the first Lutheran hymnal. In 1524 Luther developed his original four-stanza psalm paraphrase into a five-stanza Reformation hymn that developed the theme of "grace alone" more fully. Because it expressed essential Reformation doctrine, this expanded version of "Aus […]
Question: Luther’s reformed hymn did not feature stanzas of what quantity?
RoBERTa |  Passage: […] tight end Greg Olsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and 10 touchdowns; […] receivers included veteran Jerricho Cotchery (39 receptions for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and […]
Question: Who caught the second most passes?
RoBERTa |  Passage: Other prominent alumni include anthropologists David Graeber and Donald Johanson, who is best known for discovering the fossil of a female hominid australopithecine known as "Lucy" in the Afar Triangle region, psychologist John B. Watson, American psychologist who established the psychological school of behaviorism, communication theorist Harold Innis, chess grandmaster Samuel Reshevsky, and conservative international relations scholar and White House Coordinator of Security Planning for the National Security Council Samuel P. Huntington.
Question: Who thinks three moves ahead?
Table 1: Validation set examples of questions collected using different RC
models (BiDAF, BERT, and RoBERTa) in the annotation loop. The answer to the
question is highlighted in the passage.

Our choice of models reflects both the transition from LSTM-based to pre-
trained transformer-based models, as well as a graduation among the latter; we
investigate how this is reflected in datasets collected with each of these
different models in the annotation loop. For each of the models we collect
10,000 training, 1,000 validation, and 1,000 test examples. Dataset sizes are
motivated by the data efficiency of transformer-based pretrained models Devlin
et al. (2019); Liu et al. (2019b), which has improved the viability of
smaller-scale data collection efforts for investigative and analysis purposes.

To ensure the experimental integrity provided by reporting all results on a
held-out test set, we split the existing SQuAD1.1 validation set in half
(stratified by document title) as the official test set is not publicly
available. We maintain passage consistency across the training, validation and
test sets of all datasets to enable like-for-like comparisons. Lastly, we use
the majority vote answer as ground truth for SQuAD1.1 to ensure that all our
datasets have one valid answer per question, enabling us to fairly draw direct
comparisons. For clarity, we will hereafter refer to this modified version of
SQuAD1.1 as $\mathcal{D_{\mathrm{SQuAD}}}$.

#### Crowdsourcing

We use custom-designed Human Intelligence Tasks (HITs) served through Amazon
Mechanical Turk (AMT) for all annotation efforts. Workers are required to be
based in Canada, the UK, or the US, have a HIT Approval Rate greater than
$98\%$, and have previously completed at least 1,000 HITs successfully. We
experiment with and without the AMT _Master_ requirement and find no
substantial difference in quality, but observe a throughput reduction of
nearly 90%. We pay USD 2.00 for every question generation HIT, during which
workers are required to compose up to five questions that “beat” the model in
the loop (cf. Figure 3). The mean HIT completion times for BiDAF, BERT, and
RoBERTa are 551.8s, 722.4s, and 686.4s. Furthermore we find that human workers
are able to generate questions that successfully “beat” the model in the loop
$59.4\%$ of the time for BiDAF, $47.1\%$ for BERT, and $44.0\%$ for RoBERTa.
These metrics broadly reflect the relative strength of the models.

###  3.3 Quality Control

#### Training and Qualification

We provide a two-part worker training interface in order to i) familiarise
workers with the process, and ii) conduct a first screening based on worker
outputs. The interface familiarises workers with formulating questions, and
answering them through span selection. Workers are asked to generate questions
for two given answers, to highlight answers for two given questions, to
generate one full question-answer pair, and finally to complete a question
generation HIT with BiDAF as the model in the loop. Each worker’s output is
then reviewed manually (by the authors); those who pass the screening are
added to the pool of qualified annotators.

#### Manual Worker Validation

In the second annotation stage, qualified workers produce data for the “Beat
the AI” question generation task. A sample of every worker’s HITs is manually
reviewed based on their total number of completed tasks $n$, determined by
$\lfloor 5\cdot\log_{10}(n)+1\rfloor$, chosen for convenience. This is done
after every annotation batch; if workers fall below an $80\%$ success
threshold at any point, their qualification is revoked and their work is
discarded in its entirety.

#### Question Answerability

As the models used in the annotation task become stronger, the resulting
questions tend to become more complex. However, this also means that it
becomes more challenging to disentangle measures of dataset quality from
inherent question difficulty. As such, we use the condition of human
answerability for an annotated question-answer pair as follows: It is
answerable if at least one of three additional non-expert human validators can
provide an answer matching the original. We conduct answerability checks on
both the validation and test sets, and achieve answerability scores of
$87.95\%$, $85.41\%$, and $82.63\%$ for $\mathcal{D_{\mathrm{BiDAF}}}$,
$\mathcal{D_{\mathrm{BERT}}}$, and $\mathcal{D_{\mathrm{RoBERTa}}}$. We
discard all questions deemed unanswerable from the validation and test sets,
and further discard all data from any workers with less than half of their
questions considered answerable. It should be emphasised that the main purpose
of this process is to create a level playing field for comparison across
datasets constructed for different model adversaries, and can inevitably
result in valid questions being discarded. The total cost for training and
qualification, dataset construction, and validation is approximately USD
27,000.

#### Human Performance

We select a randomly chosen validator’s answer to each question and compute
Exact Match (EM) and word overlap F${}_{\text{1}}$ scores with the original to
calculate non-expert human performance; Table 2 shows the result. We observe a
clear trend: the stronger the model in the loop used to construct the dataset,
the harder the resulting questions become for humans.

| _Dev_ | _Test_
---|---|---
Resource | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_
$\mathcal{D_{\mathrm{BiDAF}}}$ | 63.0 | 76.9 | 62.6 | 78.5
$\mathcal{D_{\mathrm{BERT}}}$ | 59.2 | 74.3 | 63.9 | 76.9
$\mathcal{D_{\mathrm{RoBERTa}}}$ | 58.1 | 72.0 | 58.7 | 73.7
Table 2: Non-expert human performance results for a randomly-selected
validator per question.

###  3.4 Dataset Statistics

Table 3 provides general details on the number of passages and question-answer
pairs used in the different dataset splits. The average number of words in
questions and answers, as well as the average longest n-gram overlap between
passage and question are given in Table 4.

| _#Passages_ | _#QAs_
---|---|---
Resource | _Train_ | _Dev_ | _Test_ | _Train_ | _Dev_ | _Test_
$\mathcal{D_{\mathrm{SQuAD}}}$ | 18,891 | 971 | 1,096 | 87,599 | 5,278 | 5,292
$\mathcal{D_{\mathrm{BiDAF}}}$ | 2,523 | 278 | 277 | 10,000 | 1,000 | 1,000
$\mathcal{D_{\mathrm{BERT}}}$ | 2,444 | 283 | 292 | 10,000 | 1,000 | 1,000
$\mathcal{D_{\mathrm{RoBERTa}}}$ | 2,552 | 341 | 333 | 10,000 | 1,000 | 1,000
Table 3: Number of passages and question-answer pairs for each data resource.

We can again observe two clear trends: From weaker towards stronger models
used in the annotation loop, the average length of answers increases, and the
largest n-gram overlap drops from 3 to 2 tokens. That is, on average there is
a trigram overlap between the passage and question for
$\mathcal{D_{\mathrm{SQuAD}}}$, but only a bigram overlap for
$\mathcal{D_{\mathrm{RoBERTa}}}$ (Figure 4).333Note that the original SQuAD1.1
dataset can be considered a limit case of the adversarial annotation
framework, in which the model in the loop always predicts the wrong answer,
thus every question is accepted. This is in line with prior observations on
lexical overlap as a predictive cue in SQuAD Weissenborn et al. (2017); Min et
al. (2018); questions with less overlap are harder to answer for any of the
three models.

Figure 4: Distribution of longest n-gram overlap between passage and question for different datasets. $\mu$: mean; $\sigma$: standard deviation. | $\mathcal{D_{\mathrm{SQuAD}}}$ | $\mathcal{D_{\mathrm{BiDAF}}}$ | $\mathcal{D_{\mathrm{BERT}}}$ | $\mathcal{D_{\mathrm{RoBERTa}}}$
---|---|---|---|---
Question length | 10.3 | 9.8 | 9.8 | 10.0
Answer length | 2.6 | 2.9 | 3.0 | 3.2
N-Gram overlap | 3.0 | 2.2 | 2.1 | 2.0
Table 4: Average number of words per question and answer, and average longest
n-gram overlap between passage and question.

We furthermore analyse question types based on the question wh-word. We find
that – in contrast to $\mathcal{D_{\mathrm{SQuAD}}}$ – the datasets collected
with a model in the annotation loop have fewer when, how and in questions, and
more which, where and why questions, as well as questions in the other
category, which indicates increased question diversity. In terms of answer
types, we observe more common noun and verb phrase clauses than in
$\mathcal{D_{\mathrm{SQuAD}}}$, as well as fewer dates, names, and numeric
answers. This reflects on the strong answer-type matching capabilities of
contemporary RC models. The training and validation sets used in this analysis
($\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$ and
$\mathcal{D_{\mathrm{RoBERTa}}}$) will be publicly released.

|  | _Original_ | _Re-init._
---|---|---|---
Model | Resource | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_
BiDAF |  $\mathcal{D_{\mathrm{BiDAF}}}$dev | 0.0 | 5.3 | 10.7${}_{\text{\thinspace{0.8}}}$ |  20.4${}_{\text{\thinspace{1.0}}}$
BERT |  $\mathcal{D_{\mathrm{BERT}}}$dev | 0.0 | 4.9 | 19.7${}_{\text{\thinspace{1.0}}}$ |  30.1${}_{\text{\thinspace{1.2}}}$
RoBERTa |  $\mathcal{D_{\mathrm{RoBERTa}}}$dev | 0.0 | 6.1 | 15.7${}_{\text{\thinspace{0.9}}}$ |  25.8${}_{\text{\thinspace{1.2}}}$
BiDAF |  $\mathcal{D_{\mathrm{BiDAF}}}$test | 0.0 | 5.5 | 11.6${}_{\text{\thinspace{1.0}}}$ |  21.3${}_{\text{\thinspace{1.2}}}$
BERT |  $\mathcal{D_{\mathrm{BERT}}}$test | 0.0 | 5.3 | 18.9${}_{\text{\thinspace{1.2}}}$ |  29.4${}_{\text{\thinspace{1.1}}}$
RoBERTa |  $\mathcal{D_{\mathrm{RoBERTa}}}$test | 0.0 | 5.9 | 16.1${}_{\text{\thinspace{0.8}}}$ |  26.7${}_{\text{\thinspace{0.9}}}$
Table 5: Consistency of the adversarial effect (or lack thereof) when
retraining the models in the loop on the same data again, but with different
random seeds. We report the mean and standard deviation (subscript) over 10
re-initialisation runs.

##  4 Experiments

###  4.1 Consistency of the Model in the Loop

We begin with an experiment regarding the consistency of the adversarial
nature of the models in the annotation loop. Our annotation pipeline is
designed to reject all samples where the model correctly predicts the answer.
How reproducible is this when retraining the model with the same training
data? To measure this, we evaluate the performance of instances of BiDAF,
BERT, and RoBERTa, which only differ from the model used during annotation in
their random initialisation and order of mini-batch samples during training.
These results are shown in Table 5.

First, we observe – as expected given our annotation constraints – that model
performance is 0.0EM on datasets created with the same respective model in the
annotation loop. We observe however that retrained models do not reliably
perform as poorly on those samples. For example, BERT reaches 19.7EM, whereas
the original model used during annotation provides no correct answer with
0.0EM. This demonstrates that random model components can substantially affect
the adversarial annotation process. The evaluation furthermore serves as a
baseline for subsequent model evaluations: This much of the performance range
can be learnt merely by retraining the same model. A possible takeaway for
employing the model-in-the-loop annotation strategy in the future is to rely
on ensembles of adversaries and reduce the dependency on one particular model
instantiation, as investigated by Grefenstette et al. (2018).

|  | Evaluation (Test) Dataset
---|---|---
Model | Trained On | $\mathcal{D_{\mathrm{SQuAD}}}$ | $\mathcal{D_{\mathrm{BiDAF}}}$ | $\mathcal{D_{\mathrm{BERT}}}$ | $\mathcal{D_{\mathrm{RoBERTa}}}$ | $\mathcal{D_{\mathrm{DROP}}}$ | $\mathcal{D_{\mathrm{NQ}}}$
|  | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_
| _$\mathcal{D_{\mathrm{SQuAD(10K)}}}$_ |  40.9${}_{\text{\thinspace{0.6}}}$ |  54.3${}_{\text{\thinspace{0.6}}}$ | 7.1${}_{\text{\thinspace{0.6}}}$ |  15.7${}_{\text{\thinspace{0.6}}}$ | 5.6${}_{\text{\thinspace{0.3}}}$ |  13.5${}_{\text{\thinspace{0.4}}}$ | 5.7${}_{\text{\thinspace{0.4}}}$ |  13.5${}_{\text{\thinspace{0.4}}}$ | 3.8${}_{\text{\thinspace{0.4}}}$ |  8.6${}_{\text{\thinspace{0.6}}}$ |  25.1${}_{\text{\thinspace{1.1}}}$ |  38.7${}_{\text{\thinspace{0.7}}}$
_BiDAF_ | _$\mathcal{D_{\mathrm{BiDAF}}}$_ | 11.5${}_{\text{\thinspace{0.4}}}$ |  20.9${}_{\text{\thinspace{0.4}}}$ | 5.3${}_{\text{\thinspace{0.4}}}$ |  11.6${}_{\text{\thinspace{0.5}}}$ | 7.1${}_{\text{\thinspace{0.4}}}$ |  14.8${}_{\text{\thinspace{0.6}}}$ | 6.8${}_{\text{\thinspace{0.5}}}$ |  13.5${}_{\text{\thinspace{0.6}}}$ | 6.5${}_{\text{\thinspace{0.5}}}$ |  12.4${}_{\text{\thinspace{0.4}}}$ | 15.7${}_{\text{\thinspace{1.1}}}$ |  28.7${}_{\text{\thinspace{0.8}}}$
| _$\mathcal{D_{\mathrm{BERT}}}$_ | 10.8${}_{\text{\thinspace{0.3}}}$ |  19.8${}_{\text{\thinspace{0.4}}}$ |  7.2${}_{\text{\thinspace{0.5}}}$ |  14.4${}_{\text{\thinspace{0.6}}}$ | 6.9${}_{\text{\thinspace{0.3}}}$ |  14.5${}_{\text{\thinspace{0.4}}}$ | 8.1${}_{\text{\thinspace{0.4}}}$ |  15.0${}_{\text{\thinspace{0.6}}}$ | 7.8${}_{\text{\thinspace{0.9}}}$ |  14.5${}_{\text{\thinspace{0.9}}}$ | 16.5${}_{\text{\thinspace{0.6}}}$ |  28.3${}_{\text{\thinspace{0.9}}}$
| _$\mathcal{D_{\mathrm{RoBERTa}}}$_ | 10.7${}_{\text{\thinspace{0.2}}}$ |  20.2${}_{\text{\thinspace{0.3}}}$ | 6.3${}_{\text{\thinspace{0.7}}}$ |  13.5${}_{\text{\thinspace{0.8}}}$ |  9.4${}_{\text{\thinspace{0.6}}}$ |  17.0${}_{\text{\thinspace{0.6}}}$ |  8.9${}_{\text{\thinspace{0.9}}}$ |  16.0${}_{\text{\thinspace{0.8}}}$ |  15.3${}_{\text{\thinspace{0.8}}}$ |  22.9${}_{\text{\thinspace{0.8}}}$ | 13.4${}_{\text{\thinspace{0.9}}}$ |  27.1${}_{\text{\thinspace{1.2}}}$
| _$\mathcal{D_{\mathrm{SQuAD(10K)}}}$_ |  69.4${}_{\text{\thinspace{0.5}}}$ |  82.7${}_{\text{\thinspace{0.4}}}$ | 35.1${}_{\text{\thinspace{1.9}}}$ |  49.3${}_{\text{\thinspace{2.2}}}$ | 15.6${}_{\text{\thinspace{2.0}}}$ |  27.3${}_{\text{\thinspace{2.1}}}$ | 11.9${}_{\text{\thinspace{1.5}}}$ |  23.0${}_{\text{\thinspace{1.4}}}$ | 18.9${}_{\text{\thinspace{2.3}}}$ |  28.9${}_{\text{\thinspace{3.2}}}$ | 52.9${}_{\text{\thinspace{1.0}}}$ |  68.2${}_{\text{\thinspace{1.0}}}$
_BERT_ | _$\mathcal{D_{\mathrm{BiDAF}}}$_ | 66.5${}_{\text{\thinspace{0.7}}}$ |  80.6${}_{\text{\thinspace{0.6}}}$ |  46.2${}_{\text{\thinspace{1.2}}}$ |  61.1${}_{\text{\thinspace{1.2}}}$ |  37.8${}_{\text{\thinspace{1.4}}}$ |  48.8${}_{\text{\thinspace{1.5}}}$ |  30.6${}_{\text{\thinspace{0.8}}}$ |  42.5${}_{\text{\thinspace{0.6}}}$ |  41.1${}_{\text{\thinspace{2.3}}}$ |  50.6${}_{\text{\thinspace{2.0}}}$ |  54.2${}_{\text{\thinspace{1.2}}}$ |  69.8${}_{\text{\thinspace{0.9}}}$
| _$\mathcal{D_{\mathrm{BERT}}}$_ | 61.2${}_{\text{\thinspace{1.8}}}$ |  75.7${}_{\text{\thinspace{1.6}}}$ | 42.9${}_{\text{\thinspace{1.9}}}$ |  57.5${}_{\text{\thinspace{1.8}}}$ | 37.4${}_{\text{\thinspace{2.1}}}$ |  47.9${}_{\text{\thinspace{2.0}}}$ | 29.3${}_{\text{\thinspace{2.1}}}$ |  40.0${}_{\text{\thinspace{2.3}}}$ | 39.4${}_{\text{\thinspace{2.2}}}$ |  47.6${}_{\text{\thinspace{2.2}}}$ | 49.9${}_{\text{\thinspace{2.3}}}$ |  65.7${}_{\text{\thinspace{2.3}}}$
| _$\mathcal{D_{\mathrm{RoBERTa}}}$_ | 57.0${}_{\text{\thinspace{1.7}}}$ |  71.7${}_{\text{\thinspace{1.8}}}$ | 37.0${}_{\text{\thinspace{2.3}}}$ |  52.0${}_{\text{\thinspace{2.5}}}$ | 34.8${}_{\text{\thinspace{1.5}}}$ |  45.9${}_{\text{\thinspace{2.0}}}$ | 30.5${}_{\text{\thinspace{2.2}}}$ |  41.2${}_{\text{\thinspace{2.2}}}$ | 39.0${}_{\text{\thinspace{3.1}}}$ |  47.4${}_{\text{\thinspace{2.8}}}$ | 45.8${}_{\text{\thinspace{2.4}}}$ |  62.4${}_{\text{\thinspace{2.5}}}$
| _$\mathcal{D_{\mathrm{SQuAD(10K)}}}$_ |  68.6${}_{\text{\thinspace{0.5}}}$ |  82.8${}_{\text{\thinspace{0.3}}}$ | 37.7${}_{\text{\thinspace{1.1}}}$ |  53.8${}_{\text{\thinspace{1.1}}}$ | 20.8${}_{\text{\thinspace{1.2}}}$ |  34.0${}_{\text{\thinspace{1.0}}}$ | 11.0${}_{\text{\thinspace{0.8}}}$ |  22.1${}_{\text{\thinspace{0.9}}}$ | 25.0${}_{\text{\thinspace{2.2}}}$ |  39.4${}_{\text{\thinspace{2.4}}}$ | 43.9${}_{\text{\thinspace{3.8}}}$ |  62.8${}_{\text{\thinspace{3.1}}}$
_RoBERTa_ | _$\mathcal{D_{\mathrm{BiDAF}}}$_ | 64.8${}_{\text{\thinspace{0.7}}}$ |  80.0${}_{\text{\thinspace{0.4}}}$ |  48.0${}_{\text{\thinspace{1.2}}}$ |  64.3${}_{\text{\thinspace{1.1}}}$ |  40.0${}_{\text{\thinspace{1.5}}}$ |  51.5${}_{\text{\thinspace{1.3}}}$ | 29.0${}_{\text{\thinspace{1.9}}}$ |  39.9${}_{\text{\thinspace{1.8}}}$ |  44.5${}_{\text{\thinspace{2.1}}}$ |  55.4${}_{\text{\thinspace{1.9}}}$ |  48.4${}_{\text{\thinspace{1.1}}}$ |  66.9${}_{\text{\thinspace{0.8}}}$
| _$\mathcal{D_{\mathrm{BERT}}}$_ | 59.5${}_{\text{\thinspace{1.0}}}$ |  75.1${}_{\text{\thinspace{0.9}}}$ | 45.4${}_{\text{\thinspace{1.5}}}$ |  60.7${}_{\text{\thinspace{1.5}}}$ | 38.4${}_{\text{\thinspace{1.8}}}$ |  49.8${}_{\text{\thinspace{1.7}}}$ | 28.2${}_{\text{\thinspace{1.5}}}$ |  38.8${}_{\text{\thinspace{1.5}}}$ | 42.2${}_{\text{\thinspace{2.3}}}$ |  52.6${}_{\text{\thinspace{2.0}}}$ | 45.8${}_{\text{\thinspace{1.1}}}$ |  63.6${}_{\text{\thinspace{1.1}}}$
| _$\mathcal{D_{\mathrm{RoBERTa}}}$_ | 56.2${}_{\text{\thinspace{0.7}}}$ |  72.1${}_{\text{\thinspace{0.7}}}$ | 41.4${}_{\text{\thinspace{0.8}}}$ |  57.1${}_{\text{\thinspace{0.8}}}$ | 38.4${}_{\text{\thinspace{1.1}}}$ |  49.5${}_{\text{\thinspace{0.9}}}$ |  30.2${}_{\text{\thinspace{1.3}}}$ |  41.0${}_{\text{\thinspace{1.2}}}$ | 41.2${}_{\text{\thinspace{0.9}}}$ |  51.2${}_{\text{\thinspace{0.8}}}$ | 43.6${}_{\text{\thinspace{1.1}}}$ |  61.6${}_{\text{\thinspace{0.9}}}$
Table 6: Training models on various datasets, each with 10,000 samples, and
measuring their generalisation to different evaluation datasets. Results
underlined indicate the best result per model. We report the mean and standard
deviation (subscript) over 10 runs with different random seeds.

###  4.2 Adversarial Generalisation

A potential problem with the focus on challenging questions is that they might
be very distinct from one another, leading to difficulties in learning to
generalise to and from them. We conduct a series of experiments in which we
train on $\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$, and
$\mathcal{D_{\mathrm{RoBERTa}}}$, and observe how well models can learn to
generalise to the respective test portions of these datasets. Table 6 shows
the results, and there is a multitude of observations.

First, one clear trend we observe across all training data setups is a
negative performance progression when evaluated against datasets constructed
with a stronger model in the loop. This trend holds true for all but the BiDAF
model, in each of the training configurations, and for each of the evaluation
datasets. For example, RoBERTa trained on $\mathcal{D_{\mathrm{RoBERTa}}}$
achieves 72.1, 57.1, 49.5, and 41.0F${}_{\text{1}}$ when evaluated on
$\mathcal{D_{\mathrm{SQuAD}}}$, $\mathcal{D_{\mathrm{BiDAF}}}$,
$\mathcal{D_{\mathrm{BERT}}}$, and $\mathcal{D_{\mathrm{RoBERTa}}}$
respectively.

Second, we observe that the BiDAF model is not able to generalise well to
datasets constructed with a model in the loop, independent of its training
setup. In particular it is unable to learn from
$\mathcal{D_{\mathrm{BiDAF}}}$, thus failing to overcome some of its own blind
spots through adversarial training. Irrespective of the training dataset,
BiDAF consistently performs poorly on the adversarially collected evaluation
datasets, and we also note a substantial performance drop when trained on
$\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$, or
$\mathcal{D_{\mathrm{RoBERTa}}}$ and evaluated on
$\mathcal{D_{\mathrm{SQuAD}}}$.

In contrast, BERT and RoBERTa are able to partially overcome their blind spots
through training on data collected with a model in the loop, and to a degree
that far exceeds what would be expected from random retraining (cf. Table 5).
For example, BERT reaches 47.9F${}_{\text{1}}$ when trained and evaluated on
$\mathcal{D_{\mathrm{BERT}}}$, while RoBERTa trained on
$\mathcal{D_{\mathrm{RoBERTa}}}$ reaches 41.0F${}_{\text{1}}$ on
$\mathcal{D_{\mathrm{RoBERTa}}}$, both considerably better than random
retraining, or when training on the non-adversarially collected
$\mathcal{D_{\mathrm{SQuAD(10K)}}}$ showing gains of 20.6F${}_{\text{1}}$ for
BERT and 18.9F${}_{\text{1}}$ for RoBERTa. These observations suggest that
there exists learnable structure among harder questions that can be picked up
by some of the models, yet not all, as BiDAF fails to achieve this. The fact
that even BERT can learn to generalise to $\mathcal{D_{\mathrm{RoBERTa}}}$,
but not BiDAF to $\mathcal{D_{\mathrm{BERT}}}$ suggests the existence of an
inherent limitation to what BiDAF can learn from these new samples, compared
to BERT and RoBERTa.

More generally, we observe that training on $\mathcal{D_{\mathrm{S}}}$, where
$S$ is a stronger RC model, helps generalise to $\mathcal{D_{\mathrm{W}}}$,
where $W$ is a weaker model, for example, training on
$\mathcal{D_{\mathrm{RoBERTa}}}$ and testing on $\mathcal{D_{\mathrm{BERT}}}$.
On the other hand, training on $\mathcal{D_{\mathrm{W}}}$ also leads to
generalisation towards $\mathcal{D_{\mathrm{S}}}$. For example, RoBERTa
trained on 10,000 SQuAD samples reaches 22.1F${}_{\text{1}}$ on
$\mathcal{D_{\mathrm{RoBERTa}}}$ ($\mathcal{D_{\mathrm{S}}}$), whereas
training RoBERTa on $\mathcal{D_{\mathrm{BiDAF}}}$ and
$\mathcal{D_{\mathrm{BERT}}}$ ($\mathcal{D_{\mathrm{W}}}$) bumps this number
to 39.9F${}_{\text{1}}$ and 38.8F${}_{\text{1}}$, respectively.

Third, we observe similar performance degradation patterns for both BERT and
RoBERTa on $\mathcal{D_{\mathrm{SQuAD}}}$ when trained on data collected with
increasingly stronger models in the loop. For example, RoBERTa evaluated on
$\mathcal{D_{\mathrm{SQuAD}}}$ achieves 82.8, 80.0, 75.1, and
72.1F${}_{\text{1}}$ when trained on $\mathcal{D_{\mathrm{SQuAD(10K)}}}$,
$\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$, and
$\mathcal{D_{\mathrm{RoBERTa}}}$ respectively. This may indicate a gradual
shift in the distributions of composed questions as the model in the loop gets
stronger.

These observations suggest an encouraging takeaway for the model-in-the-loop
annotation paradigm: Even though a particular model might be chosen as an
adversary in the annotation loop, which at some point falls behind more recent
state-of-the-art models, these future models can still benefit from data
collected with the weaker model, and also generalise better to samples
composed with the stronger model in the loop.

|  | Evaluation (Test) Dataset
---|---|---
Model | Training Dataset | $\mathcal{D_{\mathrm{SQuAD}}}$ | $\mathcal{D_{\mathrm{BiDAF}}}$ | $\mathcal{D_{\mathrm{BERT}}}$ | $\mathcal{D_{\mathrm{RoBERTa}}}$
|  | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_
| _$\mathcal{D_{\mathrm{SQuAD}}}$_ |  56.7${}_{\text{\thinspace{0.5}}}$ |  70.1${}_{\text{\thinspace{0.3}}}$ | 11.6${}_{\text{\thinspace{1.0}}}$ | 21.3${}_{\text{\thinspace{1.1}}}$ | 8.6${}_{\text{\thinspace{0.6}}}$ | 17.3${}_{\text{\thinspace{0.8}}}$ | 8.3${}_{\text{\thinspace{0.7}}}$ | 16.8${}_{\text{\thinspace{0.5}}}$
_BiDAF_ | _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{BiDAF}}}$_ | 56.3${}_{\text{\thinspace{0.6}}}$ | 69.7${}_{\text{\thinspace{0.4}}}$ | 14.4${}_{\text{\thinspace{0.9}}}$ | 24.4${}_{\text{\thinspace{0.9}}}$ | 15.6${}_{\text{\thinspace{1.1}}}$ | 24.7${}_{\text{\thinspace{1.1}}}$ | 14.3${}_{\text{\thinspace{0.5}}}$ | 23.3${}_{\text{\thinspace{0.7}}}$
| _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{BERT}}}$_ | 56.2${}_{\text{\thinspace{0.6}}}$ | 69.4${}_{\text{\thinspace{0.6}}}$ | 14.4${}_{\text{\thinspace{0.7}}}$ | 24.2${}_{\text{\thinspace{0.8}}}$ | 15.7${}_{\text{\thinspace{0.6}}}$ | 25.1${}_{\text{\thinspace{0.6}}}$ | 13.9${}_{\text{\thinspace{0.8}}}$ | 22.7${}_{\text{\thinspace{0.8}}}$
| _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{RoBERTa}}}$_ | 56.2${}_{\text{\thinspace{0.7}}}$ | 69.6${}_{\text{\thinspace{0.6}}}$ |  14.7${}_{\text{\thinspace{0.9}}}$ |  24.8${}_{\text{\thinspace{0.8}}}$ |  17.9${}_{\text{\thinspace{0.5}}}$ |  26.7${}_{\text{\thinspace{0.6}}}$ |  16.7${}_{\text{\thinspace{1.1}}}$ |  25.0${}_{\text{\thinspace{0.8}}}$
| _$\mathcal{D_{\mathrm{SQuAD}}}$_ | 74.8${}_{\text{\thinspace{0.3}}}$ | 86.9${}_{\text{\thinspace{0.2}}}$ | 46.4${}_{\text{\thinspace{0.7}}}$ | 60.5${}_{\text{\thinspace{0.8}}}$ | 24.4${}_{\text{\thinspace{1.2}}}$ | 35.9${}_{\text{\thinspace{1.1}}}$ | 17.3${}_{\text{\thinspace{0.7}}}$ | 28.9${}_{\text{\thinspace{0.9}}}$
_BERT_ | _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{BiDAF}}}$_ | 75.2${}_{\text{\thinspace{0.4}}}$ |  87.2${}_{\text{\thinspace{0.2}}}$ | 52.4${}_{\text{\thinspace{0.9}}}$ | 66.5${}_{\text{\thinspace{0.9}}}$ | 40.9${}_{\text{\thinspace{1.3}}}$ | 51.2${}_{\text{\thinspace{1.5}}}$ | 32.9${}_{\text{\thinspace{0.9}}}$ | 44.1${}_{\text{\thinspace{0.8}}}$
| _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{BERT}}}$_ | 75.1${}_{\text{\thinspace{0.3}}}$ | 87.1${}_{\text{\thinspace{0.3}}}$ |  54.1${}_{\text{\thinspace{1.0}}}$ |  68.0${}_{\text{\thinspace{0.8}}}$ | 43.7${}_{\text{\thinspace{1.1}}}$ | 54.1${}_{\text{\thinspace{1.3}}}$ | 34.7${}_{\text{\thinspace{0.7}}}$ | 45.7${}_{\text{\thinspace{0.8}}}$
| _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{RoBERTa}}}$_ |  75.3${}_{\text{\thinspace{0.4}}}$ | 87.1${}_{\text{\thinspace{0.3}}}$ | 53.0${}_{\text{\thinspace{1.1}}}$ | 67.1${}_{\text{\thinspace{0.8}}}$ |  44.1${}_{\text{\thinspace{1.1}}}$ |  54.4${}_{\text{\thinspace{0.9}}}$ |  36.6${}_{\text{\thinspace{0.8}}}$ |  47.8${}_{\text{\thinspace{0.5}}}$
| _$\mathcal{D_{\mathrm{SQuAD}}}$_ | 73.2${}_{\text{\thinspace{0.4}}}$ | 86.3${}_{\text{\thinspace{0.2}}}$ | 48.9${}_{\text{\thinspace{1.1}}}$ | 64.3${}_{\text{\thinspace{1.1}}}$ | 31.3${}_{\text{\thinspace{1.1}}}$ | 43.5${}_{\text{\thinspace{1.2}}}$ | 16.1${}_{\text{\thinspace{0.8}}}$ | 26.7${}_{\text{\thinspace{0.9}}}$
_RoBERTa_ | _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{BiDAF}}}$_ |  73.9${}_{\text{\thinspace{0.4}}}$ |  86.7${}_{\text{\thinspace{0.2}}}$ | 55.0${}_{\text{\thinspace{1.4}}}$ | 69.7${}_{\text{\thinspace{0.9}}}$ | 46.5${}_{\text{\thinspace{1.1}}}$ | 57.3${}_{\text{\thinspace{1.1}}}$ | 31.9${}_{\text{\thinspace{0.8}}}$ | 42.4${}_{\text{\thinspace{1.0}}}$
| _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{BERT}}}$_ | 73.8${}_{\text{\thinspace{0.2}}}$ |  86.7${}_{\text{\thinspace{0.2}}}$ | 55.4${}_{\text{\thinspace{1.0}}}$ | 70.1${}_{\text{\thinspace{0.9}}}$ | 48.9${}_{\text{\thinspace{1.0}}}$ | 59.0${}_{\text{\thinspace{1.2}}}$ | 32.9${}_{\text{\thinspace{1.3}}}$ | 43.7${}_{\text{\thinspace{1.4}}}$
| _$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{RoBERTa}}}$_ | 73.5${}_{\text{\thinspace{0.3}}}$ | 86.5${}_{\text{\thinspace{0.2}}}$ |  55.9${}_{\text{\thinspace{0.7}}}$ |  70.6${}_{\text{\thinspace{0.7}}}$ |  49.1${}_{\text{\thinspace{1.2}}}$ |  59.5${}_{\text{\thinspace{1.2}}}$ |  34.7${}_{\text{\thinspace{1.0}}}$ |  45.9${}_{\text{\thinspace{1.2}}}$
Table 7: Training models on SQuAD, as well as SQuAD combined with different
adversarially created datasets. We report the mean and standard deviation
(subscript) over 10 runs with different random seeds.

We further show experimental results for the same models and training
datasets, but now including SQuAD as additional training data in Table 7. In
this training setup we generally see improved generalisation to
$\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$, and
$\mathcal{D_{\mathrm{RoBERTa}}}$. Interestingly, the relative differences
between $\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$, and
$\mathcal{D_{\mathrm{RoBERTa}}}$ as training sets used in conjunction with
SQuAD are much diminished, and especially $\mathcal{D_{\mathrm{RoBERTa}}}$ as
(part of) the training set now generalises substantially better. We see that
BERT and RoBERTa both show consistent performance gains with the addition of
the original SQuAD1.1 training data, but unlike in Table 6, this comes without
any noticeable decline in performance on $\mathcal{D_{\mathrm{SQuAD}}}$,
suggesting that the adversarially constructed datasets expose inherent model
weaknesses, as investigated by Liu et al. (2019a).

Furthermore, RoBERTa achieves the strongest results on the adversarially
collected evaluation sets, in particular when trained on
$\mathcal{D_{\mathrm{SQuAD}}}$ \+ $\mathcal{D_{\mathrm{RoBERTa}}}$. This
stands in contrast to the results in Table 6, where training on
$\mathcal{D_{\mathrm{BiDAF}}}$ in several cases led to better generalisation
than training on $\mathcal{D_{\mathrm{RoBERTa}}}$. A possible explanation is
that training on $\mathcal{D_{\mathrm{RoBERTa}}}$ leads to a larger degree of
overfitting to specific adversarial examples in
$\mathcal{D_{\mathrm{RoBERTa}}}$ than training on
$\mathcal{D_{\mathrm{BiDAF}}}$, and that the inclusion of a large number of
standard SQuAD training samples can mitigate this effect.

Results for the models trained on all the datasets combined
($\mathcal{D_{\mathrm{SQuAD}}}$, $\mathcal{D_{\mathrm{BiDAF}}}$,
$\mathcal{D_{\mathrm{BERT}}}$, and $\mathcal{D_{\mathrm{RoBERTa}}}$) are shown
in Table 8. These further support the previous observations and provide
additional performance gains where, for example, RoBERTa achieves
F${}_{\text{1}}$ scores of 86.9 on $\mathcal{D_{\mathrm{SQuAD}}}$, 74.1 on
$\mathcal{D_{\mathrm{BiDAF}}}$, 65.1 on $\mathcal{D_{\mathrm{BERT}}}$, and
52.7 on $\mathcal{D_{\mathrm{RoBERTa}}}$, surpassing the best previous
performance on all adversarial datasets.

Finally, we identify a risk of datasets constructed with weaker models in the
loop becoming outdated. For example, RoBERTa achieves
58.2EM/73.2F${}_{\text{1}}$ on $\mathcal{D_{\mathrm{BiDAF}}}$, in contrast to
0.0EM/5.5F${}_{\text{1}}$ for BiDAF – which is not far from the non-expert
human performance of 62.6EM/78.5F${}_{\text{1}}$ (cf. Table 2).

It is also interesting to note that, even when training on all the combined
data (cf. Table 8), BERT outperforms RoBERTa on
$\mathcal{D_{\mathrm{RoBERTa}}}$ and vice versa, suggesting that there may
exist weaknesses inherent to each model class.

| Evaluation (Test) Dataset
---|---
Model | $\mathcal{D_{\mathrm{SQuAD}}}$ | $\mathcal{D_{\mathrm{BiDAF}}}$ | $\mathcal{D_{\mathrm{BERT}}}$ | $\mathcal{D_{\mathrm{RoBERTa}}}$
| _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_ | _EM_ | _F ${}_{\text{1}}$_
_BiDAF_ | 57.1${}_{\text{\thinspace{0.4}}}$ | 70.4${}_{\text{\thinspace{0.3}}}$ | 17.1${}_{\text{\thinspace{0.8}}}$ | 27.0${}_{\text{\thinspace{0.9}}}$ | 20.0${}_{\text{\thinspace{1.0}}}$ | 29.2${}_{\text{\thinspace{0.8}}}$ | 18.3${}_{\text{\thinspace{0.6}}}$ | 27.4${}_{\text{\thinspace{0.7}}}$
_BERT_ |  75.5${}_{\text{\thinspace{0.2}}}$ |  87.2${}_{\text{\thinspace{0.2}}}$ | 57.7${}_{\text{\thinspace{1.0}}}$ | 71.0${}_{\text{\thinspace{1.1}}}$ | 52.1${}_{\text{\thinspace{0.7}}}$ | 62.2${}_{\text{\thinspace{0.7}}}$ |  43.0${}_{\text{\thinspace{1.1}}}$ |  54.2${}_{\text{\thinspace{1.0}}}$
_RoBERTa_ | 74.2${}_{\text{\thinspace{0.3}}}$ | 86.9${}_{\text{\thinspace{0.3}}}$ |  59.8${}_{\text{\thinspace{0.5}}}$ |  74.1${}_{\text{\thinspace{0.6}}}$ |  55.1${}_{\text{\thinspace{0.6}}}$ |  65.1${}_{\text{\thinspace{0.7}}}$ | 41.6${}_{\text{\thinspace{1.0}}}$ | 52.7${}_{\text{\thinspace{1.0}}}$
Table 8: Training models on SQuAD combined with all the adversarially created
datasets $\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$, and
$\mathcal{D_{\mathrm{RoBERTa}}}$. We report the mean and standard deviation
(subscript) over 10 runs with different random seeds.

###  4.3 Generalisation to Non-Adversarial Data

Compared to standard annotation, the model-in-the-loop approach generally
results in new question distributions. Consequently, models trained on
adversarially composed questions might not be able to generalise to standard
(“easy”) questions, thus limiting the practical usefulness of the resulting
data. To what extent do models trained on model-in-the-loop questions
generalise differently to standard (“easy”) questions, compared to models
trained on standard (“easy”) questions?

To measure this we further train each of our three models on either
$\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$, or
$\mathcal{D_{\mathrm{RoBERTa}}}$ and test on $\mathcal{D_{\mathrm{SQuAD}}}$,
with results in the $\mathcal{D_{\mathrm{SQuAD}}}$ columns of Table 6. For
comparison, the models are also trained on 10,000 SQuAD1.1 samples (referred
to as $\mathcal{D_{\mathrm{SQuAD(10K)}}}$) chosen from the same passages as
the adversarial datasets, thus eliminating size and paragraph choice as
potential confounding factors. The models are tuned for EM on the held-out
$\mathcal{D_{\mathrm{SQuAD}}}$ validation set. Note that, although performance
values on the majority vote $\mathcal{D_{\mathrm{SQuAD}}}$ dataset are lower
than on the original, for the reasons described earlier, this enables direct
comparisons across all datasets.

Remarkably, neither BERT nor RoBERTa show substantial drops when trained on
$\mathcal{D_{\mathrm{BiDAF}}}$ compared to training on SQuAD data
($-$2.1F${}_{\text{1}}$, and $-$2.8F${}_{\text{1}}$): Training these models on
a dataset with a weaker model in the loop still leads to strong generalisation
even to data from the original SQuAD distribution, which all models in the
loop are trained on. BiDAF, on the other hand, fails to learn such information
from the adversarially collected data, and drops >30F${}_{\text{1}}$ for each
of the new training sets, compared to training on SQuAD.

We also observe a gradual decrease in generalisation to SQuAD when training on
$\mathcal{D_{\mathrm{BiDAF}}}$ towards training on
$\mathcal{D_{\mathrm{RoBERTa}}}$. This suggests that the stronger the model,
the more dissimilar the resulting data distribution becomes from the original
SQuAD distribution. We later find further support for this explanation in a
qualitative analysis (Section 5). It may however also be due to a limitation
of BERT and RoBERTa – similar to BiDAF – in learning from a data distribution
designed to beat these models; an even stronger model might learn more from,
for example, $\mathcal{D_{\mathrm{RoBERTa}}}$.

###  4.4 Generalisation to DROP and NaturalQuestions

Finally, we investigate to what extent models can transfer skills learned on
the datasets created with a model in the loop to two recently introduced
datasets: DROP Dua et al. (2019), and NaturalQuestions Kwiatkowski et al.
(2019). In this experiment we select the subsets of DROP and NaturalQuestions
that align with the structural constraints of SQuAD to ensure a like-for-like
analysis. Specifically, we only consider questions in DROP where the answer is
a span in the passage and where there is only one candidate answer. For
NaturalQuestions, we consider all non-tabular long answers as passages, remove
HTML tags and use the short answer as the extracted span. We apply this
filtering on the validation sets for both datasets. Next we split them,
stratifying by document (as we did for $\mathcal{D_{\mathrm{SQuAD{}}}}$),
which results in $1409/1418$ validation and test set examples for DROP, and
$964/982$ for NaturalQuestions, respectively. We denote these datasets as
$\mathcal{D_{\mathrm{DROP}}}$ and $\mathcal{D_{\mathrm{NQ}}}$ for clarity and
distinction from their unfiltered versions. We consider the same models and
training datasets as before, but tune on the respective validation sets of
$\mathcal{D_{\mathrm{DROP}}}$ and $\mathcal{D_{\mathrm{NQ}}}$. Table 6 shows
the results of these experiments in the respective
$\mathcal{D_{\mathrm{DROP}}}$ and $\mathcal{D_{\mathrm{NQ}}}$ columns.

Figure 5: Comparison of comprehension types for the questions in different
datasets. The label types are neither mutually exclusive nor comprehensive.
Values above columns indicate excess of the axis range.

First, we observe clear generalisation improvements towards
$\mathcal{D_{\mathrm{DROP}}}$ across all models compared to training on
$\mathcal{D_{\mathrm{SQuAD(10K)}}}$ when training on any of
$\mathcal{D_{\mathrm{BiDAF}}}$, $\mathcal{D_{\mathrm{BERT}}}$, or
$\mathcal{D_{\mathrm{RoBERTa}}}$. That is, including a model in the loop for
the training dataset leads to improved transfer towards
$\mathcal{D_{\mathrm{DROP}}}$. Note that DROP also makes use of a BiDAF model
in the loop during annotation; these results are in line with our prior
observations when testing the same setups on $\mathcal{D_{\mathrm{BiDAF}}}$,
$\mathcal{D_{\mathrm{BERT}}}$ and $\mathcal{D_{\mathrm{RoBERTa}}}$, compared
to training on $\mathcal{D_{\mathrm{SQuAD(10K)}}}$.

Second, we observe overall strong transfer results towards
$\mathcal{D_{\mathrm{NQ}}}$, with up to 69.8F${}_{\text{1}}$ for a BERT model
trained on $\mathcal{D_{\mathrm{BiDAF}}}$. Note that this result is similar
to, and even slightly improves over model training with SQuAD data of the same
size. That is, relative to training on SQuAD data, training on adversarially
collected data $\mathcal{D_{\mathrm{BiDAF}}}$ does not impede generalisation
to the $\mathcal{D_{\mathrm{NQ}}}$ dataset, which was created without a model
in the annotation loop. We then however see a similar negative performance
progression as observed before when testing on $\mathcal{D_{\mathrm{SQuAD}}}$:
the stronger the model in the annotation loop of the training dataset, the
lower the test accuracy on test data from a data distribution composed without
a model in the loop.

##  5 Qualitative Analysis

Having applied the general model-in-the-loop methodology on models of varying
strength, we next perform a qualitative comparison of the nature of the
resulting questions. As reference points we also include the original SQuAD
questions, as well as DROP and NaturalQuestions in this comparison: These
datasets are both constructed to overcome limitations in SQuAD and have
subsets sufficiently similar to SQuAD to make an analysis possible.
Specifically, we seek to understand the qualitative differences in terms of
reading comprehension challenges posed by the questions in each of these
datasets.

###  5.1 Comprehension Requirements

There exists a variety of prior work that seeks to understand the types of
knowledge, comprehension skills or types of reasoning required to answer
questions based on text Rajpurkar et al. (2016); Clark et al. (2018); Sugawara
et al. (2019); Dua et al. (2019); Dasigi et al. (2019); we are however unaware
of any commonly accepted formalism. We take inspiration from these but develop
our own taxonomy of comprehension requirements which suits the datasets
analysed. Our taxonomy contains 13 labels, most of which are commonly used in
other work. However, the following three deserve additional clarification: i)
explicit – for which the answer is stated nearly word-for-word in the passage
as it is in the question, ii) filtering – a set of answers is narrowed down to
select one by some particular distinguishing feature, and iii) implicit – the
answer builds on information implied by the passage and does not otherwise
require any of the other types of reasoning.

We annotate questions with labels from this catalogue in a manner that is not
mutually exclusive, and neither fully comprehensive; the development of such a
catalogue is itself very challenging. Instead, we focus on capturing the most
salient characteristics of each given question, and assign it up to three of
the labels in our catalogue. In total, we analyse 100 samples from the
validation set of each of the datasets; Figure 5 shows the results.

###  5.2 Observations

An initial observation is that the majority (57%) of answers to SQuAD
questions are stated explicitly, without comprehension requirements beyond the
literal level. This number decreases substantially for any of the model-in-
the-loop datasets derived from SQuAD (e.g., 8% for
$\mathcal{D_{\mathrm{BiDAF}}}$) and also $\mathcal{D_{\mathrm{DROP}}}$, yet
42% of questions in $\mathcal{D_{\mathrm{NQ}}}$ share this property. In
contrast to SQuAD, the model-in-the-loop questions generally tend to involve
more paraphrasing. They also require more external knowledge, and multi-hop
inference (beyond co-reference resolution) with an increasing trend for
stronger models used in the annotation loop. Model-in-the-loop questions
further fan out into a variety of small, but non-negligible proportions of
more specific types of inference required for comprehension, for example,
spatial or temporal inference (both going beyond explicitly stated spatial or
temporal information) – SQuAD questions rarely require these at all. Some of
these more particular inference types are common features of the other two
datasets, in particular _comparative_ questions for DROP (60%) and to a small
extent also NaturalQuestions. Interestingly, $\mathcal{D_{\mathrm{BiDAF}}}$
possesses the largest number of comparison questions (11%) among our model-in-
the-loop datasets, whereas $\mathcal{D_{\mathrm{BERT}}}$ and
$\mathcal{D_{\mathrm{RoBERTa}}}$ only possess 1% and 3%, respectively. This
offers an explanation for our previous observation in Table 6, where BERT and
RoBERTa perform better on $\mathcal{D_{\mathrm{DROP}}}$ when trained on
$\mathcal{D_{\mathrm{BiDAF}}}$ rather than on $\mathcal{D_{\mathrm{BERT}}}$ or
$\mathcal{D_{\mathrm{RoBERTa}}}$. It is likely that BiDAF as a model in the
loop is worse than BERT and RoBERTa at _comparative_ questions, as evidenced
by the results in Table 6 with BiDAF reaching 8.6F${}_{\text{1}}$, BERT
reaching 28.9F${}_{\text{1}}$, and RoBERTa reaching 39.4F${}_{\text{1}}$ on
$\mathcal{D_{\mathrm{DROP}}}$ (when trained on
$\mathcal{D_{\mathrm{SQuAD(10K)}}}$).

The distribution of NaturalQuestions contains elements of both the SQuAD and
$\mathcal{D_{\mathrm{BiDAF}}}$ distributions, which offers a potential
explanation for the strong performance on $\mathcal{D_{\mathrm{NQ}}}$ of
models trained on $\mathcal{D_{\mathrm{SQuAD(10K)}}}$ and
$\mathcal{D_{\mathrm{BiDAF}}}$. Finally, the gradually shifting distribution
away from both SQuAD and NaturalQuestions as the model-in-the-loop strength
increases reflects our prior observations on the decreasing performance on
SQuAD and NaturalQuestions of models trained on datasets with progressively
stronger models in the loop.

##  6 Discussion and Conclusions

We have investigated an RC annotation paradigm that requires a model in the
loop to be “beaten” by an annotator. Applying this approach with progressively
stronger models in the loop (BiDAF, BERT, and RoBERTa), we produced three
separate datasets. Using these datasets, we investigated several questions
regarding the annotation paradigm, in particular whether such datasets grow
outdated as stronger models emerge, and their generalisation to standard (non-
adversarially collected) questions. We found that stronger models can still
learn from data collected with a weak adversary in the loop, and their
generalisation improves even on datasets collected with a stronger adversary.
Models trained on data collected with a model in the loop further generalise
well to non-adversarially collected data, both on SQuAD and on
NaturalQuestions, yet we observe a gradual deterioration in performance with
progressively stronger adversaries.

We see our work as a contribution towards the emerging paradigm of model-in-
the-loop annotation. While this paper has focused on RC, with SQuAD as the
original dataset used to train model adversaries, we see no reason in
principle why findings would not be similar for other tasks using the same
annotation paradigm, when crowdsourcing challenging samples with a model in
the loop. We would expect the insights and benefits conveyed by model-in-the-
loop annotation to be the greatest on mature datasets where models exceed
human performance: Here the resulting data provides a magnifying glass on
model performance, focused in particular on samples which models struggle on.
On the other hand, applying the method to datasets where performance has not
yet plateaued would likely result in a more similar distribution to the
original data, which is challenging to models a priori. We hope that the
series of experiments on replicability, observations on transfer between
datasets collected using models of different strength, as well as our findings
regarding generalisation to non-adversarially collected data, can support and
inform future research and annotation efforts using this paradigm.

## Acknowledgements

The authors would like to thank Christopher Potts for his detailed and
constructive feedback, and our reviewers. This work was supported by the
European Union’s Horizon 2020 research and innovation programme under grant
agreement No. 875160 and the UK Defence Science and Technology Laboratory
(Dstl) and Engineering and Physical Research Council (EPSRC) under grant
EP/R018693/1 as a part of the collaboration between US DOD, UK MOD, and UK
EPSRC under the Multidisciplinary University Research Initiative (MURI).

## References

  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Chen et al. (2016) Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016.  A thorough examination of the CNN/Daily Mail reading comprehension task.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2358–2367, Berlin, Germany. Association for Computational Linguistics.
  * Chen et al. (2019) Michael Chen, Mike D’Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019.  CODAH: An adversarially-authored question answering dataset for common sense.  In _Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP_ , pages 63–69, Minneapolis, USA. Association for Computational Linguistics.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  QuAC: Question answering in context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? Try ARC, the AI2 reasoning challenge.  _CoRR_ , abs/1803.05457.
  * Dasigi et al. (2019) Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019.  Quoref: A reading comprehension dataset with questions requiring coreferential reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5925–5932, Hong Kong, China. Association for Computational Linguistics.
  * Deng et al. (2009) Jia Deng, R. Socher, Li Fei-Fei, Wei Dong, Kai Li, and Li-Jia Li. 2009.  ImageNet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , volume 00, pages 248–255.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dinan et al. (2019) Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019.  Build it break it fix it for dialogue safety: Robustness from adversarial human attack.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4537–4546, Hong Kong, China. Association for Computational Linguistics.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ettinger et al. (2017) Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender. 2017.  Towards linguistically generalizable NLP systems: A workshop and shared task.  _CoRR_ , abs/1711.01505.
  * Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018.  AllenNLP: A deep semantic natural language processing platform.  In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_ , pages 1–6, Melbourne, Australia. Association for Computational Linguistics.
  * Grefenstette et al. (2018) Edward Grefenstette, Robert Stanforth, Brendan O’Donoghue, Jonathan Uesato, Grzegorz Swirszcz, and Pushmeet Kohli. 2018.  Strength in numbers: Trading-off robustness and computation via adversarially-trained ensembles.  _CoRR_ , abs/1811.09300.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 1693–1701. Curran Associates, Inc.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Kaushik et al. (2020) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020.  Learning the difference that makes a difference with counterfactually-augmented data.  In _International Conference on Learning Representations_.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C. Lipton. 2018.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5010–5015, Brussels, Belgium. Association for Computational Linguistics.
  * Kočiský et al. (2018) Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The NarrativeQA reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6:317–328.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lewis and Gale (1994) David D. Lewis and William A. Gale. 1994.  A sequential algorithm for training text classifiers. In _SIGIR_ , pages 3–12. ACM/Springer.
  * Liu et al. (2019a) Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019a.  Inoculation by fine-tuning: A method for analyzing challenge datasets.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2171–2179, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  RoBERTa: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330.
  * Min et al. (2018) Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018.  Efficient and robust question answering from minimal context over documents.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1725–1735, Melbourne, Australia. Association for Computational Linguistics.
  * Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009.  Distant supervision for relation extraction without labeled data.  In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_ , pages 1003–1011, Suntec, Singapore. Association for Computational Linguistics.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated MAchine Reading COmprehension dataset.  _arXiv preprint arXiv:1611.09268_.
  * Nie et al. (2019) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019.  Adversarial NLI: A new benchmark for natural language understanding.  _arXiv preprint arXiv: 1910.14599_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Richardson et al. (2013) Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013.  MCTest: A challenge dataset for the open-domain machine comprehension of text.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 193–203, Seattle, Washington, USA. Association for Computational Linguistics.
  * Saeidi et al. (2018) Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018.  Interpretation of natural language rules in conversational machine reading.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2087–2097, Brussels, Belgium. Association for Computational Linguistics.
  * Schwartz et al. (2017) Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017.  The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 15–25, Vancouver, Canada. Association for Computational Linguistics.
  * Seo et al. (2017) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Bidirectional attention flow for machine comprehension.  In _The International Conference on Learning Representations (ICLR)_.
  * Snow et al. (2008) Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008.  Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 254–263, Honolulu, Hawaii. Association for Computational Linguistics.
  * Sugawara et al. (2018) Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018.  What makes reading comprehension questions easier? In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4208–4219, Brussels, Belgium. Association for Computational Linguistics.
  * Sugawara et al. (2019) Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2019.  Assessing the benchmarking capacity of machine reading comprehension datasets.  _CoRR_ , abs/1911.09241.
  * Thorne et al. (2019) James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019.  The FEVER2.0 shared task.  In _Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)_ , pages 1–6, Hong Kong, China. Association for Computational Linguistics.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
  * Wallace et al. (2019) Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019\.  Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering.  _Transactions of the Association for Computational Linguistics_ , 7:387–401.
  * Weissenborn et al. (2017) Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017.  Making neural QA as simple as possible but not simpler.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 271–280, Vancouver, Canada. Association for Computational Linguistics.
  * Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018.  Constructing datasets for multi-hop reading comprehension across documents.  _Transactions of the Association for Computational Linguistics_ , 6:287–302.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  _ArXiv_ , abs/1910.03771.
  * Yang et al. (2018a) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018a.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yang et al. (2018b) Zhilin Yang, Saizheng Zhang, Jack Urbanek, Will Feng, Alexander Miller, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018b.  Mastering the dungeon: Grounded language learning by mechanical turker descent.  In _International Conference on Learning Representations_.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  SWAG: A large-scale adversarial dataset for grounded commonsense inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 93–104, Brussels, Belgium. Association for Computational Linguistics.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018.  ReCoRD: Bridging the gap between human and machine commonsense reading comprehension.  _arXiv preprint arXiv:1810.12885_.

##  Appendix A Additional Dataset Statistics

#### Question statistics

In Figure 7 we analyse question lengths across SQuAD1.1 and compare them to
questions constructed with different models in the annotation loop. While the
mean of the distributions is similar, there is more question length
variability when using a model in the loop. We also perform analysis of
question types by wh- word as described earlier (see Figure 6). This is in
further detail displayed using sunburst plots of the first three question
tokens for $\mathcal{D_{\mathrm{SQuAD}}}$ (cf. Figure 11),
$\mathcal{D_{\mathrm{BiDAF}}}$ (cf. Figure 13), $\mathcal{D_{\mathrm{BERT}}}$
(cf. Figure 12) and $\mathcal{D_{\mathrm{RoBERTa}}}$ (cf. Figure 14). We
observe a general trend towards more diverse questions with increasing model-
in-the-loop strength.

Figure 6: Analysis of question types across datasets. Figure 7: Question
length distribution across datasets. Figure 8: Analysis of answer types across
datasets. Figure 9: Answer length distribution across datasets.

#### Answer statistics

Figure 9 allows for further analysis of answer lengths across datasets. We
observe that answers for all datasets constructed with a model in the loop
tend to be longer than in SQuAD. There is furthermore a trend of increasing
answer length and variability with increasing model-in-the-loop strength. We
show an analysis of answer types in Figure 8.

Figure 10: Worker distribution, together with the number of manually validated
QA pairs per worker.

##  Appendix B Annotation Interface Details

We have three key steps in the dataset construction process: i) training and
qualification, ii) “Beat the AI” annotation and iii) answer validation.

#### Training and Qualification

This is a combined training and qualification task; a screenshot of the
interface is shown in Figure 15. The first step involves a set of five
assignments requiring the worker to demonstrate an ability to generate
questions and indicate answers by highlighting the corresponding spans in the
passage. Once complete, the worker is shown a sample “Beat the AI” HIT for a
pre-determined passage which helps facilitate manual validation. In earlier
experiments, these two steps were presented as separate interfaces, however,
this created a bottleneck between the two layers of qualification and slowed
down annotation considerably. In total, 1,386 workers completed this task with
752 being assigned the qualification.

#### “Beat the AI Annotation”

The “Beat the AI” question generation HIT presents workers with a randomly
selected passage from SQuAD1.1, about which workers are expected to generate
questions and provide answers. This data is sent to the corresponding model-
in-the-loop API running on AWS infrastructure and primarily consisting of a
load balancer and a t2.xlarge EC2 instance with the T2/T3 Unlimited setting
enabled to allow high sustained CPU performance during annotation runs. The
model API returns a prediction which is scored against the worker’s answer to
determine whether the worker has successfully managed to “beat” the model.
Only questions which the model fails to answer are considered valid; a
screenshot for this interface is shown in Figure 16. Workers are tasked to
ideally submit at least three valid questions, however fewer are also accepted
– in particular for very short passages. A sample of each worker’s HITs is
manually validated; those who do not satisfy the question quality requirements
have their qualification revoked and all their annotated data discarded. This
was the case for 99 workers. Worker validation distributions are shown in
Figure 10.

#### Answer Validation

The answer validation interface (cf. Figure 17) is used to validate the
answerability of the validation and test sets for each different model used in
the annotation loop. Every previously collected question generation HIT from
these dataset parts, which had not been discarded during manual validation, is
submitted to at least 3 distinct annotators. Workers are shown the passage and
previously generated questions and are asked to highlight the answer in the
passage. In a post-processing step, only questions with at least 1 valid
matching answer out of 3 are finally retained.

##  Appendix C Catalogue of Comprehension Requirements

We give a description for each of the items in our catalogue of comprehension
requirements in Table 9, accompanied with an example for illustration. These
are the labels used for the qualitative analysis performed in Section 5.

Figure 11: Question sunburst plot for $\mathcal{D_{\mathrm{SQuAD}}}$. Figure
12: Question sunburst plot for $\mathcal{D_{\mathrm{BERT}}}$. Figure 13:
Question sunburst plot for $\mathcal{D_{\mathrm{BiDAF}}}$. Figure 14: Question
sunburst plot for $\mathcal{D_{\mathrm{RoBERTa}}}$.

Figure 15: Training and qualification interface. Workers are first expected to
familiarise themselves with the interface and them complete a sample “Beat the
AI” task for validation. Figure 16: “Beat the AI” question generation
interface. Human annotators are tasked with asking questions about a provided
passage which the model in the loop fails to answer correctly. Figure 17:
Answer validation interface. Workers are expected to provide answers to
questions generated in the “Beat the AI” task. The additional answers are used
to determine question answerability and non-expert human performance.

Type |  Description |  Passage |  Question
---|---|---|---
Explicit |  Answer stated nearly word-for-word in the passage as it is in the question. |  Sayyid Abul Ala Maududi was an important early twentieth-century figure in the Islamic revival in India […] |  Who was an important early figure in the Islamic revival in India?
Paraphrasing |  Question paraphrases parts of the passage, generally relying on context-specific synonyms. |  Seamans’ establishment of an ad-hoc committee […] |  Who created the ad-hoc committee?
External Knowledge |  The question cannot be answered without access to sources of knowledge beyond the passage. |  […] the 1988 film noir thriller Stormy Monday, directed by Mike Figgis and starring Tommy Lee Jones, Melanie Griffith, Sting and Sean Bean. |  Which musician was featured in the film Stormy Monday?
Co-reference |  Requires resolution of a relationship between two distinct words referring to the same entity. |  Tamara de Lempicka was a famous artist born in Warsaw. […] Better than anyone else she represented the Art Deco style in painting and art […] |  Through what creations did Lempicka express a kind of art popular after WWI?
Multi-Hop |  Requires more than one step of inference, often across multiple sentences. |  […] and in 1916 married a Polish lawyer Tadeusz Lempicki. Better than anyone else she represented the Art Deco style in painting and art […] |  Into what family did the artist who represented the Art Deco style marry?
Comparative |  Requires a comparison between two or more attributes (e.g.,  _smaller than_ , _last_) |  The previous chairs were Rajendra K. Pachauri, elected in May 2002; Robert Watson in 1997; and Bert Bolin in 1988. |  Who was elected earlier, Robert Watson or Bert Bolin?
Numeric |  Any numeric reasoning (e.g., some form of calculation is required to arrive at the correct answer). |  […] it has been estimated that Africans will make up at least 30% of the delegates at the 2012 General Conference, and it is also possible that 40% of the delegates will be from outside […] |  From which continent is it estimated that members will make up nearly a third of participants in 2012?
Negation |  Requires interpreting a single or multiple negations. |  Subordinate to the General Conference are the jurisdictional and central conferences which also meet every four years. |  What is not in charge?
Filtering |  Narrowing down a set of answers to select one by some particular distinguishing feature. |  […] was engaged with Johannes Bugenhagen, Justus Jonas, Johannes Apel, Philipp Melanchthon and Lucas Cranach the Elder and his wife as witnesses […] |  Whose partner could testify to the couple’s agreement to marry?
Temporal |  Requires an understanding of time and change, and related aspects. Goes beyond directly stated answers to _When_ questions or external knowledge. |  In 2010 the Amazon rainforest experienced another severe drought, in some ways more extreme than the 2005 drought. |  What occurred in 2005 and then again five years later?
Spatial |  Requires an understanding of the concept of space, location, or proximity. Goes beyond finding directly stated answers to _Where_ questions. |  Warsaw lies in east-central Poland about 300 km (190 mi) from the Carpathian Mountains and about 260 km (160 mi) from the Baltic Sea, 523 km (325 mi) east of Berlin, Germany. |  Is Warsaw closer to the Baltic Sea or Berlin, Germany?
Inductive |  A particular case is addressed in the passage but inferring the answer requires generalisation to a broader category. |  […] frequently evoked by particular events in his life and the unfolding Reformation. This behavior started with his learning of the execution of Johann Esch and Heinrich Voes, the first individuals to be martyred by the Roman Catholic Church for Lutheran views […] |  How did the Roman Catholic Church deal with non-believers?
Implicit |  Builds on information implied in the passage and does not otherwise require any of the above types of reasoning. |  Despite the disagreements on the Eucharist, the Marburg Colloquy paved the way for the signing in 1530 of the Augsburg Confession, and for the […] |  What could not keep the Augsburg confession from being signed?
Table 9:  Comprehension requirement definitions and examples from adversarial
model-in-the-loop annotated RC datasets. Note that these types are not
mutually exclusive. The annotated answer is highlighted in yellow.