where $G$ is the set of gold entity mentions. $\textnormal{IoU}(b,g)$ is a
function measuring how many tokens are shared between two spans:

$\textnormal{IoU}(b,g)=\frac{\textnormal{length}(b\cap
g)}{\textnormal{length}(b\cup g)}$ (2.12)

and $\Gamma$ is a hyperparameter tuned on different datasets.

###### Reducing search space

Instead of exhaustive classifying over all possible spans, a two-stage
paradigm is investigated to reduce the size of candidate mentions. Zheng et
al. (2019) propose a boundary-aware model, where first sequence labelling
models are used to detect possible span boundaries, and then the span based
models are used to predict entity categories of a small number of candidate
spans. Similarly, Xia et al. (2019a) separate the task into two stages:
deciding whether the candidate span is an entity mention or not via a
detector, and then classifying detected candidates into predefined entity
categories via a classifier.

Lin et al. (2019) detect entity mentions by using what they call _head-driven
phrase structures_. They first identify possible head words of entity
mentions, and then recognise the mention boundaries by exploiting phrase
structures. They argue that although entity mentions might nest with each
other, they cannot share the same head words, and the head words are
informative to decide the entity category. They also propose an objective
function—bag loss—which does not require gold head word annotations. This is
done by exploiting the association between words and entity categories.

###### Modelling surrounding mentions

To take the surrounding mentions of a given span into consideration, Xu et al.
(2017) introduce a _2nd-pass_ mechanism. They train two models: one standard
model, and the other model using outputs from the first model, where the
predicted entity categories are used to replace the entity mentions. During
inference, the span score is the linear interpolation between scores from
these two models.

Luan et al. (2018) propose a multi-task learning framework where entity
recognition, relation extraction, and coreference resolution are treated as
classification problems with shared span representations. By sharing low-level
LSTM encoder, information about relation types with surrounding mentions and
coreferences can be used to create input span representations to entity
classifier. Instead of sharing only LSTM encoder, Luan et al. (2019) further
extend the multi-task model using dynamically constructed span (node) graphs.
At each training step, the most confident entity spans are treated as nodes in
a graph structure, and arcs are confidence-weighted relation types and
coreferences. Then, the span representations are refined using updates which
are propagated from neighbouring relation types and co-refeerred entities.

####  2.4.4. Sentence-level Approach

Instead of predicting whether a token belongs to an entity mention and its
role in the mention (token-level approach) or whether a consecutive sequence
of tokens form an entity mention (span-level approach), some methods predict
directly a combination of entity mentions within a sentence. We call these
methods sentence-level approach.

McDonald et al. (2005) consider NER as a structured multi-label
classification. Instead of starting and ending indices, they represent each
entity mention using the set of token positions that belong to the mention. An
example of this representation, with each token tagged using an I/O schema is
shown in Figure 2.12. The advantage of this method is that the representation
is very flexible as it allows entity mentions consisting of discontinuous
tokens and does not require mentions to exclude each other. Using this
representation, the NER problem is converted into the multi-label
classification problem of finding up to $k$ correct labels among all possible
$(T+1)^{n}$ labels, where $k$ is a hyper-parameter of the model, $T$ is the
number of entity categories, and $n$ is the length of the sentence. Note that
labels do not come from a pre-defined category but depend on the sentence
being processed. McDonald et al. use large-margin online learning algorithms
to train the model, so that the scores of the correct labels (entity mentions)
are higher than those of all other possible incorrect mentions. Another
advantage of this method is that the outputs of the model are unambiguous for
all kinds of complex entity mentions and easy to be decoded. However, the
method suffers from a $O(n^{3}T)$ inference complexity.

Figure 2.12.  An example of a sentence with three entity mentions: ‘Bill
Clinton’ and ‘Hilary Clinton’ are Person mentions, and ‘Canada’ is a Location
mention. P and L refer to the entity categories: Person and Location,
respectively.

Finkel and Manning (2009) use a discriminative constituency parser to
recognise nested entity mentions. They represent each sentence as a
constituency tree, where each mention corresponds to a phrase in the tree. In
addition, each node needs to be annotated with its parent and grandparent
labels, so that the parser can learn how entity mentions nest. Ringland (2016)
also employ a joint model using the Berkeley parser (Petrov et al., 2006), and
show that it performs well even without specialised NER features. However, one
disadvantage of these parsing based models, as in (McDonald et al., 2005), is
that their time complexity is cubic in the number of tokens in the sentence.
Furthermore, the high quality parse training data, which is not always
available, plays a crucial role in the success of the joint model (Li et al.,
2017).

Lu and Roth (2015) propose a novel hypergraph to represent exponentially many
possible nested mentions in one sentence, and one sub-hypergraph of the
complete hypergraph can therefore be used to represent a combination of
mentions in the sentence. The mention hypergraph consist of five types of
nodes:

$A^{k}$ nodes:

represent all mentions whose left boundaries are exactly at or after the
$k$-th token;

$E^{k}$ nodes:

represent all mentions whose left boundaries are exactly at the $k$-th token;

$T^{k}_{j}$ nodes:

represent all mentions whose left boundaries are exactly at the $k$-th token
and have the mention type $j$;

$I^{k}_{j}$ nodes:

represent all mentions which contain the $k$-th token and have the mention
type $j$; and

$X$ nodes:

indicate the completion of a path.

Hyper-edges, each of which consists of a parent node and an ordered list of
child nodes, are used to connect nodes. Figure 2.13 is an example of such a
sub-hypergraph, which represents two nested entity mentions.

Figure 2.13.  An example sub-hypergraph with two nested entity mentions:
‘HIV-1’ (Virus) and ‘HIV-1 enhancer’ (DNA). Here, one mention corresponds to a
path consisting of (AETI+X) nodes. Specifically, the path
(A4E4T${}_{4}^{1}$I${}_{4}^{1}$X) corresponds to the mention ‘HIV-1’, and the
path (A4E4T${}_{4}^{2}$I${}_{4}^{2}$I${}_{5}^{2}$X) corresponds to the mention
‘HIV-1 enhancer’.

The training objective of this hypergraph-based model is to maximise the log-
likelihood of training instances consisting of the sentence and mention-
encoded hypergraph. During inference, the model first predicts a sub-
hypergraph among all possible sub-hypergraphs of the complete hypergraph, and
predicted mentions can be decoded from the output sub-hypergraph. Different to
Lu and Roth (2015) who build hand-crafted features defined over the input
sentence and the output hypergraph structure, Katiyar and Cardie (2018) learn
the hypergraph representation using features extracted from a recurrent
network.

Although this hypergraph-based model enjoys a time complexity that is linear
in the number of tokens in the input sentence, it suffers from some degree of
ambiguity during decoding stage. For example, when one mention is contained by
another mention with the same entity category and their boundaries are all
different, the hypergraph can be decoded in different ways. This ambiguity
comes from the fact that, if one node has multiple parent nodes and multiple
child nodes, there is no mechanism to decide which of the parent node is
paired with which child node. Therefore, Wang and Lu (2018) propose an
extension of the I node where they use $I^{k}_{i,n}$ nodes to represent all
mentions of type $k$ which contain the $j$-th token and start with the $i$-th
token.

To represent discontinuous mentions, Muis and Lu (2016) expand the node types
in the hypergraph representation to capture discontinuous mentions. That is,
they add two new node types: B for tokens within the mention, and O for tokens
belonging to part of the gap. Figure 2.14 is an example of the sub-hypergraph,
which encodes two mentions: ‘muscle pain’ and ‘muscle fatigue’. Wang and Lu
(2019) propose a two-stage approach that all spans are first identified using
the hypergraph representation and then a classifier is used to predict whether
two spans form a discontinuous mention.

Figure 2.14.  An example sub-hypergraph with two entity mentions: ‘muscle
pain’ and ‘muscle fatigue’. Muis and Lu (2016) extend the hypergraph
representation proposed by Lu and Roth (2015) to capture discontinuous
mentions through two new node types: B${}_{k}^{i}$ representing the $k$-th
token is part of the $i$-th component of an entity mention, and O${}_{k}^{i}$
representing the $k$-th token appears in between $(i-1)$-th and $i$-th
components of an entity mention. In this example, the path
(A3E3TB${}_{3}^{0}$B${}_{4}^{1}$X) corresponds to the mention ‘muscle pain’
and the path (A3E3TB${}_{3}^{0}$O${}_{4}^{1}$O${}_{5}^{1}$B${}_{6}^{1}$X)
corresponds to the discontinuous mention ‘muscle fatigue’.

####  2.4.5. Summary

Despite the potential applications of complex NER recognition, there is
comparatively few studies on recognising discontinuous and overlapping
mentions. The span-based approach focuses on building effective span
representations which are used to predict whether the span is an entity
mention and its entity category. However, it cannot be directed used for
discontinuous NER, because discontinuous mentions consist of multiple spans.

We observe that there is usually a trade-of between the expressiveness power
and the modelling difficulty. In other words, the more flexible (less
constraints) the representation is, the more interactions are ignored,
therefore the model might be more difficult to train. For example, the multi-
label representation proposed by McDonald et al. (2005) is the most flexible
representation (Figure 2.12); however, it does not take any interactions
between different mentions into consideration. Tang et al. (2018) empirically
show that a less flexible representation—BIO variant tagging based model—can
outperform the multi-label representation when the training data are limited.

Methods belonging to token-level and sentence-level approaches first predict
intermediate representations, and then the intermediate representations are
decoded into entity mentions. That is, token level representations use a
sequence of tags (Tang et al., 2013a; Metke-Jimenez and Karimi, 2016) as the
intermediate representation, and sentence-level approach uses a graph
structure (Lu and Roth, 2015; Katiyar and Cardie, 2018). Inspired by these
approaches, in Chapter 5, we propose a transition-based model for
discontinuous NER. Similar to token-level and sentence-level approaches, our
transition-based model first predicts the intermediate representation, i.e., a
sequence of actions, but greatly reduce the ambiguity problem in these two
approaches. Our method also employs effective methods to build span
representations, which are inspired by span-based approach.

##  Chapter 3 Data Augmentation for NER

Data augmentation, expanding the training set by transforming training
instances without changing their labels, is heavily studied in the field of
computer vision and sentence level NLP tasks. Inspired by these efforts, we
design several easy to use data augmentation methods for NER. Through
experiments on two English datasets from the biomedical domain, we demonstrate
that our proposed augmentation methods can boost performance over a strong
baseline where large scale pre-trained models are used, especially when the
original labelled training set is small.

###  3.1. Overview

Modern deep learning techniques typically require a large amount of labelled
data for training (Bowman et al., 2015; Conneau et al., 2017). However, in
real world applications, such large labelled data sets are not always
available. This is especially true in some specific domains, such as the
biomedical domain, where annotating data requires expert knowledge and is
usually time-consuming (Karimi et al., 2015a; Nye et al., 2018).

Different approaches have been investigated to solve this _low-resource_
problem (Hedderich et al., 2020). For example, transfer learning pre-trains
language representations on self-supervised or rich-resource _source_ tasks
and then adapts these representations to the _target_ task (Ruder, 2019;
Gururangan et al., 2020). Data augmentation expands the training set by
applying transformations to training instances without changing their original
labels (Shorten and Khoshgoftaar, 2019).

Recently, there has been an increased interest on applying data augmentation
techniques on sentence-level NLP tasks, such as text classification (Wei and
Zou, 2019; Xie et al., 2019), natural language inference (Min et al., 2020),
and machine translation (Wang et al., 2018a). Augmentation methods explored
for these tasks include creating augmented instances by manipulating a few
words in the original instance, such as word replacement (Zhang et al., 2015b;
Wang and Yang, 2015; Cai et al., 2020), random deletion (Wei and Zou, 2019),
and word position swap (Şahin and Steedman, 2019; Min et al., 2020); or
creating entirely artificial instances via generative models, such as
variational autoencoders (Yoo et al., 2019; Mesbah et al., 2019) and back-
translation models (Yu et al., 2018; Iyyer et al., 2018).

Different from these sentence-level NLP tasks, NER is usually regarded as a
token-level NLP task. That is, for each token in the sentence, an NER model
predicts a label indicating whether the token belongs to an entity mention and
which entity category the mention belongs to. Therefore, applying
transformations to individual tokens may also change their labels. Due to such
a difficulty, data augmentation for NER is relatively less studied. In this
chapter, we describe our efforts to fill this gap by exploring data
augmentation techniques for NER, solved as a sequence tagging problem.

###  3.2. Proposed Data Augmentation Methods

We surveyed the existing data augmentation techniques for sentence-level NLP
tasks in Section 2.2. Inspired by these efforts, we design several easy to use
data augmentation methods for NER. Note that our proposed methods do not rely
on any external trained models, such as machine translation models (Yu et al.,
2018; Iyyer et al., 2018) or syntactic parsing models (Şahin and Steedman,
2019), which are by themselves difficult to train in low-resource domain
specific scenarios.

Given an original training instance, consisting of a sequence of tokens and
the corresponding sequence of labels, we use the following transformations to
create augmented instances.

Method | Instance |
---|---|---
None | She | did | not | complain | of | headache | or
O | O | O | O | O | B-problem | O
any | other | neurological | symptoms | . |  |
B-problem | I-problem | I-problem | I-problem | O |  |
LwTR | L. | One | not | complain | of | headache | he
O | O | O | O | O | B-problem | O
any | interatrial | neurological | current | . |  |
B-problem | I-problem | I-problem | I-problem | O |  |
SR | She | did | non | complain | of | headache | or
O | O | O | O | O | B-problem | O
whatsoever | former | neurologic | symptom | . |  |
B-problem | I-problem | I-problem | I-problem | O |  |
MR | She | did | not | complain | of | neuropathic | pain
O | O | O | O | O | B-problem | I-problem
syndrome | or | acute | pulmonary | disease | . |
I-problem | O | B-problem | I-problem | I-problem | O |
SiS | not | complain | She | did | of | headache | or
O | O | O | O | O | B-problem | O
neurological | any | symptoms | other | . |  |
B-problem | I-problem | I-problem | I-problem | O |  |
Table 3.1. An original training instance and different types of augmented
instances. We highlight changes using italics.

###### Label-wise Token Replacement (LwTR)

For each token which is not a stop word, we use a binomial distribution to
randomly decide whether it should be replaced. If yes, we then use a label-
wise token distribution, built from the original training set, to randomly
select another token with the same label. Thus, we keep the original label
sequence unchanged. Taking the instance in Table 3.1 as an example, there are
five tokens replaced by other tokens which share the same label as the
original tokens.

###### Synonym Replacement (SR)

Our second approach is similar to LwTR, except that we replace the token with
one of its synonyms retrieved from WordNet (Miller et al., 1990). Note that
the retrieved synonym may consist of more than one token. Its BIO labels can
be derived using a straightforward rule: If the replaced token is the first
token within a mention (i.e., the corresponding label is ‘B-Entity’), we
assign the same label to the first token of the retrieved multi-word synonym,
and ‘I-Entity’ to the other tokens.

###### Mention Replacement (MR)

For each mention in the instance, we use a binomial distribution to randomly
decide whether it should be replaced. If yes, we randomly select another
mention from the original training set which has the same entity category as
the replacement. The corresponding sequence of BIO labels can be changed
accordingly. For example, in Table 3.1, the mention ‘headache [B-problem]’ is
replaced by another problem mention ‘neuropathic pain syndrome [B-problem
I-problem I-problem]’.

###### Shuffle within Segments (SiS)

We first split the token sequence into segments of the same entity category.
Thus, each segment corresponds to either an entity mention or a sequence of
tokens that does not belong to any mention. For example, the original instance
in Table 3.1 is split into five segments: ‘She did not complain of [Out-of-
Mention]’, ‘headache [Problem]’, ‘or [Out-of-Mention]’, ‘any other
neurological symptoms [Problem]’, ‘. [Out-of-Mention]’. Then for each segment,
we use a binomial distribution to randomly decide whether it should be
shuffled. If yes, the order of the tokens within the segment is shuffled,
while the label order is kept unchanged.

###### All

We also explore the augmentation of the training set using all aforementioned
augmentation methods. That is, for each training instance, we create multiple
augmented instances, one per transformation.

###  3.3. Evaluation

We present an empirical analysis of the data augmentation methods described in
Section 3.2 on two English datasets from the biomedical domain 111In (Dai and
Adel, 2020), we also evaluate these methods on a dataset from the materials
science domain.: i2b2-2010 (Uzuner et al., 2011) and NCBI-Disease (Doğan et
al., 2014).

We use a BERT-CRF model (Beltagy et al., 2019; Baevski et al., 2019) as the
backbone model, and we investigate the impact of applying data augmentation on
training data of different sizes.

####  3.3.1. Datasets

| i2b2-2010 |  NCBI-Disease
---|---|---
| Train | Dev | Test | Train | Dev | Test
# Sentences | 13,868 | 2,447 | 27,625 | 5,424 | 923 | 940
# Tokens | 129,087 | 20,454 | 267,249 | 135,701 | 23,969 | 24,497
# Mentions | 14,376 | 2,143 | 31,161 | 5,134 | 787 | 960
Table 3.2. The descriptive statistics of the two English datasets from the
biomedical domain: i2b2-2010 (Uzuner et al., 2011) and NCBI-Disease (Doğan et
al., 2014).

i2b2-2010 focuses on the identification of three entity types of problem,
treatment and test from patient reports. We use the train-test split from its
corresponding shared task and randomly select $15$% of sentences from the
training set as the development set. NCBI-Disease contains scholarly articles
annotated with disease names. We use the train-dev-test split provided by the
authors. Descriptive statistics of these two datasets is listed in Table 3.2.

To simulate a low-resource setting, we select the first $50$, $150$, $500$
sentences which contain at least one mention from the complete training set to
create the corresponding small, medium, and large subsets (denoted as S, M, L
in Table 3.3, whereas the complete training set is denoted as F). Note that we
apply data augmentation only on the training set, without changing the
development and test sets.

####  3.3.2. Backbone model

We regard the NER task as a token-level sequence tagging problem, where each
token in the sentence is assigned a tag. The tag can be used to infer whether
the token is the first token within a mention, inside a mention or does not
belong to any mention.

The backbone model, illustrated in Figure 3.1, is a BERT-CRF model (Beltagy et
al., 2019; Baevski et al., 2019). It takes advantage of large scale pre-
trained language models—using BERT-based encoder to create contextual
representations for each token, and a probabilistic graphical model—using
conditional random fields (Sutton and McCallum, 2007) to capture dependencies
between neighbouring tags.

Figure 3.1. High level overview of the BERT-CRF model.

###### BERT-based encoder

Given a sentence, the tokenizer, coupled with the pre-trained BERT-based
model, first converts each token in the sentence into word pieces. That is, if
the original token does not exist in the vocabulary, it will be segmented into
several pieces from the vocabulary (Sennrich et al., 2015). Then the word
pieces are mapped to dense vectors—token embeddings—via a lookup table.
Finally, the sum of token embeddings and positional embeddings, which indicate
the position of each token in the sequence, are fed into a stack of multi-head
self-attention and fully-connected feed-forward layers (Vaswani et al., 2017).
Following the study in (Devlin et al., 2019), we use the final outputs
corresponding to the first word piece within each token as the token
representation.

Recent studies on domain-specific BERT models show that effectiveness on
downstream tasks can be improved when the BERT models are further pre-trained
on in-domain data (Gururangan et al., 2020). We thus choose SciBERT (Beltagy
et al., 2019), which is pre-trained on full text of scholarly articles about
biology and computer science, and fine-tune it on the target NER task. In our
preliminary experiments, we observe that SciBERT achieves significant better
results than vanilla BERT (Devlin et al., 2019) and slightly better results
than BioBERT (Lee et al., 2020).

###### Conditional random fields (CRF)

Instead of assigning a tag to each token independently, we model them jointly
using a conditional random fields. That is, given a sequence of token
representations
$\mathbf{R}=\left(\mathbf{r}_{1},\mathbf{r}_{2},\ldots,\mathbf{r}_{n}\right)$,
we aim to predict a sequence of tags
$\mathbf{y}=\left(y_{1},y_{2},\ldots,y_{n}\right)$ which has the maximum
probability over all possible tag sequences. This conditional probability can
be calculated using:

$p(\mathbf{y}\mid\mathbf{R})=\frac{e^{s(\mathbf{R},\mathbf{y})}}{\sum_{\widetilde{\mathbf{y}}\in\mathbf{Y}_{\mathbf{R}}}e^{s(\mathbf{R},\widetilde{\mathbf{y}})}}$

and

$s(\mathbf{R},\mathbf{y})=\sum_{i=0}^{n}A_{y_{i},y_{i+1}}+\sum_{i=1}^{n}P_{i,y_{i}},$

where $A_{i,j}$ is the compatibility score of a transition from the tag i to
tag j, and $P_{i,j}$ is the score of the tag $j$ given token representation
$r_{i}$.

The parameters, of both the SciBERT encoder and CRFs, are trained jointly to
maximise the conditional probability of gold tag sequence given the training
sentences.

####  3.3.3. Experimental results

The evaluation results on the effectiveness of data augmentation methods are
shown in Table 3.3. We use the Micro-average string match $F_{1}$ score to
evaluate the effectiveness of the models. The model which is most effective on
the development set, measured using the $F_{1}$ score, is finally evaluated on
the test set. All experiments are repeated five times with different random
seeds. Mean values and standard deviations are reported. The $\Delta$ row
shows the averaged improvement due to data augmentation, comparing against the
baseline without using data augmentation. In general, we find that all data
augmentation methods improve over the baseline, and synonym replacement
outperforms other augmentation on average.

Corpus | Size | Baseline | LwTR | SR | MR | Sis | All
---|---|---|---|---|---|---|---
i2b2-2010 | S | 34.6 $\pm$ 1.4 | 39.9 $\pm$ 0.7 | 43.8 $\pm$ 2.0 | 39.6 $\pm$ 1.3 | 39.2 $\pm$ 1.8 | 42.6 $\pm$ 1.4
M | 62.9 $\pm$ 1.0 | 64.0 $\pm$ 1.3 | 64.5 $\pm$ 0.5 | 63.5 $\pm$ 0.8 | 63.1 $\pm$ 1.4 | 63.9 $\pm$ 1.8
L | 69.6 $\pm$ 0.3 | 70.5 $\pm$ 1.6 | 70.7 $\pm$ 1.1 | 70.6 $\pm$ 0.9 | 71.0 $\pm$ 1.2 | 70.5 $\pm$ 1.0
F | 87.6 $\pm$ 0.3 | 87.3 $\pm$ 0.2 | 87.7 $\pm$ 0.2 | 87.6 $\pm$ 0.1 | 87.1 $\pm$ 0.2 | 86.8 $\pm$ 0.3
NCBI-disease | S | 59.9 $\pm$ 2.6 | 62.9 $\pm$ 1.4 | 63.6 $\pm$ 2.3 | 65.1 $\pm$ 1.3 | 63.0 $\pm$ 1.1 | 63.8 $\pm$ 1.3
M | 71.6 $\pm$ 1.4 | 73.2 $\pm$ 1.5 | 74.7 $\pm$ 1.0 | 73.6 $\pm$ 1.1 | 73.4 $\pm$ 1.1 | 73.3 $\pm$ 1.2
L | 81.0 $\pm$ 0.4 | 80.4 $\pm$ 1.1 | 82.2 $\pm$ 1.0 | 80.5 $\pm$ 1.3 | 80.6 $\pm$ 1.0 | 81.3 $\pm$ 0.5
F | 87.6 $\pm$ 0.3 | 85.7 $\pm$ 1.1 | 87.9 $\pm$ 0.7 | 88.1 $\pm$ 0.7 | 87.4 $\pm$ 0.4 | 86.0 $\pm$ 1.0
$\Delta$ | 1.1 | 2.5 | 1.7 | 1.2 | 1.6
Table 3.3. Evaluation results in terms of span-level $F_{1}$ score. Small set
contains 50 training instances; Medium contains 150 instances; Large contains
500 instances; Full uses the complete training set. Results that are better
than the baseline model without using data augmentation are highlighted in
bold. underline: the result is significantly better than the baseline model
without data augmentation (paired student’s t-test, p: $0.05$)

Another observation is that the data augmentation methods are more effective
when the original training sets are small. For example, all data augmentation
methods achieve improvements when the training set contains only $50$ training
instances. In contrast, when the complete training sets are used, only synonym
replacement and mention replacement achieve improvements. This has also been
observed in previous work on applying data augmentation on other NLP tasks
(Fadaee et al., 2017; Şahin and Steedman, 2019; Xia et al., 2019b).

###  3.4. Analysis

After demonstrating the effectiveness of proposed data augmentation methods,
we present an analysis of the best two performing transformations: synonym
replacement and mention replacement. We aim to provide practical suggestions
on hyperparameter settings as well as understandings about how they improve
the performance.

####  3.4.1. The impact of hyperparameters

For each augmentation method, we tune the number of augmented instances per
original training instance from a list of numbers: {1, 3, 6, 10}. We also tune
the $p$ value of the binomial distribution which is used to decide whether a
token or a mention should be replaced. It is searched over the range from 0.1
to 0.7, with an incremental step of 0.2. We perform grid search to find the
best combination of these two hyperparameters on the development set.

The main question we aim to answer is how much augmentation is enough? More
specifically, how the number of augmented instances per original training
instance affects performance, and how the ratio a token, or a mention, is
replaced affects performance?

(a) i2b2-2010, SR

(b) NCBI-Disease, SR

(c) i2b2-2010, MR

(d) NCBI-Disease, MR

Figure 3.2. Impact of the number of augmented instances per original training
instance on the effectiveness of data augmentation. SR: synonym replacement.
MR: mention replacement.

Figure 3.2 shows the impact of the number of augmented instances per original
training instance on the performance gain of synonym replacement and mention
replacement. We use the improvement of absolute $F_{1}$ score over the
baseline without using data augmentation as the performance gain.

In general, we find that _larger number of augmented instances can bring
larger performance gain, especially when the training sets are small (i.e.,
$50$ training instances)_. However, the performance gain becomes relatively
small when the number of augmented instances per original training instance is
greater than $6$. The second observation is that when the training sets are
medium (i.e., $150$ training instances) or large (i.e., $500$ training
instances), the benefits of more augmented instances become small. On
i2b2-2010, creating more augmented instances using synonym replacement on
large training set even decreases the performance.

(a) i2b2-2010, SR

(b) NCBI-Disease, SR

(c) i2b2-2010, MR

(d) NCBI-Disease, MR

Figure 3.3. The impact of the ratio a token or a mention is replaced on the
effectiveness of data augmentation. SR: synonym replacement. MR: mention
replacement.

Figure 3.3 A and B shows the impact of the ratio a token is replaced with one
of its synonyms on the performance gain. We note a moderate ratio (e.g., $0.3$
or $0.6$) performs well across different setups. If the ratio is too small,
the augmented instances may be very similar to the original one. Training on
these augmented instances may have a similar effect as training on the
original training instances for more epochs. In contrast, a large ratio is
more likely to create syntactically invalid instance. These syntactically
invalid instances are noisy, and training on such a combination of small
amount of clean data and large amount of noisy data may underperform training
on clean data only (more discussions in Section 2.2.1).

The pattern with mention replacement is different from the one with synonyms
replacement (Figure 3.3 C and D). When the training sets are small, increasing
the ratio a mention is replaced with another mention of the same entity
category always enlarges the performance gain. We find that most of these
entity mentions have similar part of speech patterns. That is, most of them
are either nouns or noun phrases. Because of this feature, the risk of
creating syntactically invalid instance is much lower using mention
replacement than using synonyms replacement. It can be the reason why _a
moderate ratio for synonyms replacement is best, whereas mention replacement
can benefit from a large replace ratio_.

####  3.4.2. A closer look at errors

The next question we aim to answer is how data augmentation improves the
performance. Put another way, are data augmentation methods guaranteed to fix
some particular errors predicted by the baseline model without using data
augmentation, and if yes, which types of errors are more likely to get
rectified.

| i2b2-2010 |  NCBI-Disease
---|---|---
FPs | FNs | FPs | FNs
Baseline $\cap$ SR $\cap$ MR | 12574 (26.9) | 13500 (68.8) | 170 (20.5) | 197 (45.8)
Baseline $\cap$ SR $\cap$ $\neg$ MR |  02424 (05.2) |  02001 (10.2) |  053 (06.4) |  050 (11.6)
Baseline $\cap$ $\neg$ SR $\cap$ MR |  04261 (09.1) |  00658 (03.4) | 105 (12.7) |  018 (04.2)
Baseline $\cap$ $\neg$ SR $\cap$ $\neg$ MR |  06674 (14.3) |  01613 (08.2) | 122 (14.7) |  058 (13.5)
$\neg$ Baseline $\cap$ SR $\cap$ MR |  02949 (06.3) |  00335 (01.7) |  072 (08.7) |  042 (09.8)
$\neg$ Baseline $\cap$ SR $\cap$ $\neg$ MR |  05034 (10.8) |  01107 (05.6) |  084 (10.1) |  046 (10.7)
$\neg$ Baseline $\cap$ $\neg$ SR $\cap$ MR | 12767 (27.3) |  00421 (02.1) | 223 (26.9) |  019 (04.4)
Table 3.4. The comparison of different types of errors—FPs (false positives)
and FNs (false negatives)—made by the baseline model without using data
augmentation and models using Synonym Replacement (SR) and Mention Replacement
(MR) data augmentation methods. $\cap$ indicates the intersection of two sets,
and $\neg$ indicates the negative set. For example, the ‘FPs’ column
corresponding to the ‘Baseline $\cap$ SR $\cap$ $\neg$ MR’ row shows the
number of false positives predicted by both the baseline model and the model
using SR data augmentation, but not by the one using MR data augmentation.

To answer this question, we train three models—one baseline model without
using data augmentation, two models using synonym replacement and mention
replacement, respectively—and then compare the error predictions by these
three models.

From Table 3.4, we find _data augmentations are more likely to reduce false
positives than false negatives_. In other words, if the baseline model fails
to recall some entity mentions, the model trained using data augmentation
usually fails to recall them as well. However, data augmentation can fix those
mistakenly predicted entity mentions. On one hand, we believe this improvement
can be linked to the _over-fitting_ problem. That is, the model trained
without using data augmentation may overfit some patterns observed in the
small training set, and data augmentation can relieve this problem, by
creating a new combination of mention and context. On the other hand, training
model using data augmentation provides very little improvement on fixing those
false negatives. Note that data augmentation may also make large amount of new
false positives. In other words, there is no guarantee data augmentation can
fix some particular errors predicted by models without using data
augmentation, since they may provide a mechanism to prevent the training from
over-fitting, but not help the learning algorithm to discover new
regularities.

###  3.5. Summary

We design several easy to use data augmentation methods for NER: label-wise
token replacement, synonym replacement, mention replacement, and shuffle
within segments. Through experiments on two datasets from the biomedical
domain, we find that all proposed data augmentation methods can improve over
the strong baseline, where large scale pre-trained models are used, and
synonym replacement outperforms other augmentation on average.

##  Chapter 4 Cost-effective Selection of Pre-training Data

Pre-training language representation models on unlabelled data and then
adapting them to downstream supervised tasks has become a standard practice in
NLP. However, the selection of pre-training data usually resorts to intuition,
which varies across NLP practitioners. We make use of similarity measures to
nominate in-domain pre-training data. Experimental results suggest that simple
similarity measures are good predictors of the usefulness of pre-trained
language representation models on downstream NER tasks.

###  4.1. Overview

Sequential transfer learning—which pre-trains a model from a _source_ task and
then adapts it to a different _target_ task—has demonstrated its effectiveness
on a range of NLP tasks (Pan and Yang, 2009; Weiss et al., 2016; Ruder, 2019).
There are two stages in this procedure: pre-training, and adaptation.
Researchers who work on low-resource NLP usually spend a considerable amount
of efforts and resources on choosing useful external data sources and
investigating how to transfer knowledge to their target tasks.

Mikolov et al. (2013b); Peters et al. (2018); Devlin et al. (2019) make the
most of limited labelled data by incorporating language representation models
which are pre-trained on a large amount of unlabelled data. This benefits a
range of NLP tasks where appropriate unlabelled data is available, and has
become a standard practice in NLP.

However, there is still a lack of systematic study on how to select
appropriate data to pre-train language representation models. We observe two
heuristic strategies in the literature:

  1. (1)

collecting as large as possible generic data, such as news (Mikolov et al.,
2013b; Peters et al., 2018; Liu et al., 2019), web crawl (Pennington et al.,
2014; Mikolov et al., 2018), and Wikipedia (Bojanowski et al., 2017; Devlin et
al., 2019); and,

  2. (2)

selecting moderate size data focusing on a specific domain. The resulting pre-
trained models are called _domain-specific models_ (Chiu et al., 2016; Karimi
et al., 2017; Chronopoulou et al., 2019; Nguyen et al., 2020; Lee et al.,
2020).

The advantage of the first strategy is that the pre-trained generic models can
be re-used in various domains, however, the corresponding training cost is
high and unbearable to many academic labs. For example, Liu et al. (2019) pre-
train the RoBERTa model using 1024 V100 GPUs, which are only accessible by
large companies. Therefore, we focus on studying the second strategy, and we
aim to pre-train domain-specific models, optimising the performance on
downstream biomedical NER datasets.

Studies on domain-specific language representation models empirically show
that target task performance can be improved, when in-domain data is used for
pre-training (Alsentzer et al., 2019; Lee et al., 2020; Beltagy et al., 2019).
These publicly available domain-specific models are valuable to the NLP
community. However, the selection of in-domain data usually resorts to
intuition, which varies across NLP practitioners (Section 4.2). According to
Halliday and Hasan (1989), the context specific usage of language is affected
by three factors: _field_ (the subject matter being discussed), _tenor_ (the
relationship between the participants in the discourse and their purpose) and
_mode_ (communication medium, such as ‘spoken’ or ‘written’). Generally, the
selection of pre-training data in existing domain-specific models is mainly
based on the field rather than the tenor. For example, BioBERT (Lee et al.,
2020) and SciBERT (Beltagy et al., 2019) are both pre-trained on scholar
articles, but on different fields (biology and computer science).

We first show in Section 4.2 that human intuition regarding selecting pre-
training data varies across practitioners, motivating our work on employing
quantitative measures to nominate in-domain pre-training data. We then
describe several measures which can quantify similarity between two datasets
in Section 4.3. We pre-train several domain-specific language representation
models on different sources, and investigate their effectiveness on various
downstream NER datasets, respectively (Section 4.4). Finally, through
correlation analysis, we show that simple similarity measures can be used to
nominate in-domain pre-training data (Section 4.5.1).

###  4.2. What Human Intuition Indicates

We surveyed $30$ NLP or machine learning practitioners to learn the human
intuition regarding selection of pre-training data. Participants were provided
short descriptions of the target data T, and two possible source data S1 and
S2 as

  * •

T: Online forum posts about medications;

  * •

S1: Research papers about biology and health;

  * •

S2: Online reviews about restaurants, hotels, barbers, mechanics, etc.

A screenshot is shown in Figure 4.1.

Figure 4.1. Survey questions regarding selection of pre-training data.

We constructed each of the descriptions as ‘$t$ about $f$’ where $t$ is
intended to indicate the tenor and $f$ the field. Each participant rated both
sources on a five-point Likert, indicating agreement with the statement
“Unsupervised pre-training on S would be useful for supervised named entity
recognition learning on T”.

Figure 4.2. Likert scale ratings from NLP and ML practitioners ($N=30$) for
the statement ‘Unsupervised pre-training on S would be useful for supervised
named entity recognition learning on T.’ Target data T is described as ‘Online
forum posts about medications,’ source data S1 as ‘Research papers about
biology and health,’ and source data S2 as ‘Online reviews about restaurants,
hotels, barbers, mechanics, etc.’

Survey results show that $73$% of the participants agreed or strongly agreed
that S1—sharing similar _biomedical_ field with the target—would be useful.
Only $27$% agreed that S2—sharing similar _social media_ tenor with the
target—would be useful (Figure 4.2). On the one hand, a Wilcoxon signed-rank
test indicates that scores are significantly higher for S1 than for S2
($Z=43.0,p<0.001$). On the other hand, these results show the variety across
practitioners, motivating our work on employing quantitative measures to
nominate pre-training data. Our empirical investigations (detailed in Section
4.4) also suggest that human intuition maybe unreliable regarding selecting
pre-training data. That is, practitioners favour field over tenor when
selecting pre-training data, and this would be detrimental to accuracy of the
target NER tasks.

###  4.3. Similarity Measures

Recall that the context specific usage of language is affected by three
factors: field, tenor and mode (Halliday and Hasan, 1989). Researchers who
select pre-training data from a similar field believe that, if the source data
has a similar field to the target data, they tend to share similar topical
vocabulary. Conversely, vocabularies are different from each other if source
and target are from different fields. Imagine datasets about medications and
restaurants. Those who select pre-training data from a similar tenor believe
that tenor may impact the writing style of text. Imagine the participants in
online reviews and scientific papers, their relationships to each other, their
purposes and how these affect text style, including punctuation, lexical
normalisation, politeness, emotiveness and so on (Lee, 2001; Solano-Flores,
2006). We do not explicitly consider mode, because all of the datasets studied
in this thesis are written text.

Below, we detail different measures based on these intuitions to quantify
different aspects of similarity between two datasets.

####  4.3.1. Target vocabulary covered

The first measure is simply the percentage of the target vocabulary that is
also present in the source data. An extremely dissimilar example is that of
different languages. They have a totally different vocabulary and are
considered dissimilar, even if they are written in a similar style and talking
about the same subject. Note that our focus is on transferring through pre-
trained models using one single source and we do not consider multilingual
similarity. We propose _Target Vocabulary Covered (TVC)_ as a measure of
field, calculated as

$TVC(D_{S},D_{T})=\frac{|V_{D_{S}}\cap V_{D_{T}}|}{|V_{D_{T}}|},$

where $V_{D_{S}}$ and $V_{D_{T}}$ are sets of unique content words (nouns,
verbs, adjectives) in source and target datasets respectively.

####  4.3.2. Jaccard similarity of vocabularies

Jaccard similarity coefficient (Agresti, 2003), is a statistic used for
estimating the similarity and diversity of two sets. By calculating

$JSC(D_{S},D_{T})=\frac{|V_{D_{S}}\cap V_{D_{T}}|}{|V_{D_{S}}\cup
V_{D_{T}}|},$

_Jaccard Similarity of Vocabularies (JSV)_ can be used to measure the
similarity between source and target vocabularies, meanwhile, factoring out
the source vocabulary size.

####  4.3.3. Language model perplexity

A language model (Schütze et al., 2008) assigns a probability to any sequence
of words $[w_{1},\cdots,w_{N}]$ using chain rule of probability:

$p(w_{1},w_{2},\cdots,w_{N})=\prod_{i=1}^{N}p(w_{i}|w_{1}^{w-1}),$

where $N$ is the length of the sequence and $w_{1}^{i-1}$ are all words before
word $w_{i}$. In practice, this equation can be simplified by n-gram models
based on Markov Assumption:

$p(w_{1},w_{2},\cdots,w_{N})=\prod_{i=1}^{N}p(w_{i}|w_{i-n+1}^{i-1}),$

where $w_{i-n+1}^{i-1}$ represents only $n$ preceding words of $w_{i}$. To
make the model generalise better, smoothing techniques can be used to assign
non-zero probabilities to unseen events. We use Kneser-Ney smoothed 3-gram
models (Heafield, 2011) to measure the similarity between two datasets.
Specifically, we first train the language model on the source data, then
evaluate it on the target data using perplexity to represent the degree of
similarity. The intuition is that, if the model finds a sentence very unlikely
(dissimilar from the data where this language model is trained on), it will
assign a low probability and therefore high perplexity. The summed up
_perplexity (PPL)_ is then:

$PPL(D_{S},D_{T})=\sum_{i=1}^{m}P(D_{T}^{i})^{-\frac{1}{N_{i}}},$

where $m$ is the number of sentences in the target data set, and
$P(D_{T}^{i})$ is the probability assigned by the language model trained on
the source data to the $i$-th sentence from the target data set, whose
sentence length is $N_{i}$.

Similar to TVC, PPL is token-based but also captures surface structure. We
therefore propose PPL as a proxy to measure tenor as well as field.

####  4.3.4. Jensen-Shannon divergence

Jensen-Shannon divergence (JSD), based on term distributions, has been
successfully used for domain adaptation (Ruder and Plank, 2017). We first
measure the probability of each term (up to 3-gram) in source $S$ and target
data $T$, separately. Then, we use the Jensen-Shannon divergence (Fuglede and
Topsoe, 2004) between these two probability distributions

$JSD(S||T)=\frac{1}{2}KL(S||M)+\frac{1}{2}KL(T||M),$

where

$KL(P||Q)=\sum_{x\in X}P(x)\log(\frac{P(x)}{Q(x)})$

and

$M=\frac{1}{2}(S+T)$

as a proxy to measure tenor as well as field.

###  4.4. Datasets

We use six datasets as source data, covering a range of fields and tenors:

News:

The original one billion word language model benchmark data (Chelba et al.,
2013), produced from News Crawl data. _Popular reporting_ usually involves one
or several writers, and a large number of readers. The text is usually edited
several times for easy understanding.

Books:

A corpus of 11,038 books in 16 different genres, e.g., Romance, Fantasy,
Science fiction, etc., collected by Zhu et al. (2015). There books are all
free books written by yet unpublished authors. _Fiction books_ usually
involves one writer, and a moderate size of readers. The text is usually
edited many times, reflecting writer personality.

MIMIC:

A clinical database comprising over 58,000 hospital admissions for intensive
care unit (ICU) patients (Johnson et al., 2016). _Clinical notes_ are usually
written by doctors and nurses under time pressure, and read by their
colleagues. The text is seldom carefully edited, so it is syntactically noisy
and usually contains a lot of jargon for efficient communication.

PubMed:

Titles and abstracts of biomedical scholar articles. _Scholar articles_ are
usually written by a small groups of writers, and the readers usually have
similar knowledge background with authors. The text is usually edited many
times for more comprehensible and less ambiguous.

Yelp:

Crowd-sourced reviews about local businesses, including restaurants, hotels,
barbers, mechanics, etc. _Online review_ is usually written by a single
customer, and read by a small group of people who are interested in the
business. The text is usually edited once, and writer tends to use descriptive
language to share their experiences.

Wikipedia:

A free online encyclopedia. _Online encyclopedia_ is created and edited by
volunteers around the world, and it has around 250 million page views every
day 111Siteviews Analysis: shorturl.at/ayQR5. Accessed data: 2021-Jan-31..
Another important feature of Wikipedia as an open-collaborative website is
that articles are edited all the time by human users and bots, reflecting the
newest development of the world knowledge.

To isolate the impact of source data size, we randomly sample all source data
to approximately 700 million tokens. The only exception is on MIMIC. Although
all text from MIMIC data set has been used, it is still relatively small,
comparing to other sources. The data statistics of source data is listed in
Table 4.1. Based on the number of unique tokens and average sentence length,
we can see these sources are roughly split into two categorise: formal
text—PubMed, Wikipedia and News—with large vocabulary and long sentences, and
informal text—Books, MIMIC, and Yelp—with small vocabulary and short
sentences.

Source | # sentences | # tokens | # unique tokens | Avg. sentence length
---|---|---|---|---
Books | 53.9M | 0.69B | 0.5M | 12.8
MIMIC | 61.9M | 0.61B | 0.5M | 9.8
News | 27.5M | 0.70B | 2.1M | 25.4
PubMed | 29.3M | 0.69B | 4.2M | 23.5
Wikipedia | 31.1M | 0.69B | 3.3M | 22.1
Yelp | 50.7M | 0.69B | 1.0M | 13.5
Table 4.1. Descriptive statistics of the source datasets.

Ten NER datasets are used as target data: BC2GM (BioCreative II Gene Mention
Recognition) (Smith et al., 2008), BTC (Broad Twitter Corpus) (Derczynski et
al., 2016), CADEC (CSIRO Adverse Drug Event Corpus) (Karimi et al., 2015a)),
CoNLL 2003 (Sang and Meulder, 2003), EBM (Evidence Based Medicine) (Nye et
al., 2018), i2b2 2010 (Uzuner et al., 2011), JNLPBA (Kim et al., 2004), NCBI-
Disease (Doğan et al., 2014), SciERC (Luan et al., 2018), WetLab (Kulkarni et
al., 2018), and W-NUT 2016 (Strauss et al., 2016). Details of these target
data are listed in Table 4.2.

Target |  Entity Categories |  Description
---|---|---
BC2GM |  Gene |  Biomedical scholar articles
BTC |  Person, Organisation, Location |  Tweets sampled across different regions, temporal periods, and types of Twitter users
CADEC |  Adverse Drug Event, Disease, Drug, Finding, Symptom |  Posts taken from AskaPatient, which is a forum where consumers can discuss their experiences with medications.
EBM |  Intervention, Outcome and Comparator |  Scholar articles about clinical trials
i2b2 2010 |  Problem, Treatment and Test |  Clinical notes about health
JNLPBA |  Protein, DNA, RNA, Cell line and Cell type |  Abstract of journal articles about biology.
NCBI-disease |  Disease |  Abstract of journal articles about health.
SciERC |  Generic, Material, Method, Metric, Other-Scientific-Term, Task |  Journal articles about Computer Science, Material Sciences and Physics
Wetlab |  Action, 9 object-based (Amount, Concentration, Device, Location, Method, Reagent, Speed, Temperature, Time) entity types, 5 measure-based (Numerical, Generic-Measure, Size, pH, Measure-Type) and 3 other (Mention, Modifier, Seal) types |  Protocols written by researchers about conducting biology and chemistry experiments.
Table 4.2. List of the target NER datasets and their specifications.

###  4.5. Experimental Results

###### Similarity Between Source and Target Datasets

The results shown in Table 4.3 and 4.4 indicate that PubMed is the most
similar source to most of these target datasets from the Biomedical domain. It
achieves lower language model perplexity, higher target vocabulary covered,
and Jensen-Shannon Divergence when evaluated against BC2GM, EBM, JNLPBA, NCBI-
disease, SciERC and Wetlab compared to other sources. On one hand, it is
expected that PubMed is similar to BC2GM, EBM, JNLPBA, NCBI-disease and
Wetlab, since they are all scientific writing about biology and health, thus
being similar in terms of both field and tenor. On the other hand, although
SciERC does not have the same field as PubMed (computer science, material and
physics versus biology and health), they are similar because they share a
similar tenor (scholarly publications). On i2b2-2010 (clinical notes), only
the target vocabulary covered measure indicates PubMed is the most similar
source, whereas other three metrics indicate MIMIC as the most similar source.

|  | Similarity
---|---|---
Target | Source | TVC (%) | JSV (%) | PPL | JSD
BC2GM | Books | 37.39 | 9.54 | 109.65 | 36.29
MIMIC | 41.95 | 13.69 | 101.39 | 38.17
News | 48.12 | 6.15 | 101.34 | 38.02
PubMed | 81.20 | 7.24 | 75.43 | 48.32
Wikipedia | 60.19 | 5.05 | 93.10 | 39.28
Yelp | 36.26 | 8.70 | 109.11 | 37.01
BTC | Books | 47.96 | 10.09 | 61.63 | 38.44
MIMIC | 26.80 | 6.83 | 68.54 | 33.92
News | 54.92 | 5.58 | 59.49 | 36.61
PubMed | 41.97 | 2.84 | 71.44 | 33.48
Wikipedia | 54.46 | 3.57 | 61.19 | 35.00
Yelp | 47.58 | 9.40 | 60.58 | 39.22
CADEC | Books | 80.16 | 5.08 | 47.37 | 42.72
MIMIC | 78.21 | 6.54 | 47.92 | 38.21
News | 85.59 | 2.47 | 45.90 | 39.67
PubMed | 82.03 | 1.57 | 52.38 | 37.69
Wikipedia | 84.41 | 1.55 | 49.41 | 38.18
Yelp | 81.89 | 4.85 | 45.52 | 44.82
EBM | Books | 29.68 | 10.29 | 146.01 | 36.03
MIMIC | 32.61 | 14.06 | 130.42 | 37.70
News | 44.30 | 8.21 | 125.28 | 38.47
PubMed | 70.66 | 9.30 | 91.87 | 51.85
Wikipedia | 47.20 | 5.79 | 125.61 | 39.00
Yelp | 29.94 | 9.85 | 142.15 | 36.82
Table 4.3. Similarity values measured between source and target datasets. TVC: Target Vocabulary Covered. JSC: Jaccarrd similarity of Vocabularies. PPL: language model perplexity. JSD: Jensen-Shannon Divergence based on term distributions. |  | Similarity
---|---|---
Target | Source | TVC (%) | JSV (%) | PPL | JSD
i2b2 2010 | Books | 44.65 | 5.74 | 45.55 | 37.75
MIMIC | 58.36 | 9.95 | 29.29 | 48.99
News | 56.28 | 3.42 | 43.32 | 37.30
PubMed | 64.86 | 2.64 | 38.92 | 38.23
Wikipedia | 59.59 | 2.32 | 42.24 | 37.94
Yelp | 45.76 | 5.52 | 44.75 | 37.92
JNLPBA | Books | 31.25 | 5.87 | 105.66 | 35.84
MIMIC | 32.30 | 7.74 | 101.82 | 36.25
News | 40.46 | 3.72 | 97.64 | 37.31
PubMed | 71.49 | 4.52 | 65.00 | 47.80
Wikipedia | 46.46 | 2.78 | 91.20 | 38.29
Yelp | 30.46 | 5.38 | 107.22 | 36.55
NCBI-disease | Books | 54.78 | 5.05 | 95.03 | 36.07
MIMIC | 57.02 | 6.90 | 90.02 | 37.24
News | 65.84 | 2.82 | 86.97 | 37.43
PubMed | 87.15 | 2.50 | 64.48 | 44.46
Wikipedia | 76.17 | 2.08 | 80.70 | 38.39
Yelp | 52.78 | 4.55 | 95.63 | 36.82
SciERC | Books | 70.19 | 4.34 | 88.42 | 36.11
MIMIC | 57.06 | 4.60 | 93.64 | 35.31
News | 77.67 | 2.19 | 80.88 | 37.03
PubMed | 84.14 | 1.58 | 71.03 | 39.72
Wikipedia | 81.23 | 1.46 | 77.72 | 37.56
Yelp | 67.37 | 3.88 | 89.39 | 36.89
Wetlab | Books | 48.82 | 3.97 | 60.62 | 36.43
MIMIC | 44.69 | 4.74 | 58.02 | 36.52
News | 58.06 | 2.19 | 57.47 | 36.11
PubMed | 68.54 | 1.72 | 52.11 | 37.15
Wikipedia | 61.24 | 1.47 | 55.99 | 36.41
Yelp | 50.29 | 3.83 | 57.84 | 37.12
Table 4.4. Similarity values measured between source and target datasets
(continued). TVC: Target Vocabulary Covered. JSC: Jaccarrd similarity of
Vocabularies. PPL: language model perplexity. JSD: Jensen-Shannon Divergence
based on term distributions.

The second observation is that tenor might be reflected more than field by
these measures. Source data Yelp is more similar to CADEC than PubMed and
MIMIC from both language model perplexity and Jensen-Shannon Divergence
perspectives. CADEC is a data set focusing on recognising drugs, diseases and
adverse drug events. The field of CADEC is therefore more similar to PubMed
which includes journal articles in health discipline and MIMIC which contains
clinical notes. However, CADEC is written by patients, and can be considered
as ‘drug reviews’. The tenor is therefore closer to the one in Yelp, where
customers use informal language to describe their experiences. Target
vocabulary covered nominates News as the most similar source. Note that News
has a moderate size vocabulary, 2.1 millions unique tokens, whereas the
vocabulary size of PubMed is 4.2 million.

The last observation is that using different measures can lead to almost the
same answer regarding which source is the most similar one to a given target,
except for the Jaccard similarity of vocabularies. Using Jaccard similarity of
vocabularies measure, MIMIC source is nominated as the most similar source
against target sets, except for BTC. This might be explained by the fact that
the vocabulary size of MIMIC is the smallest one in all sources, and Jaccard
similarity of vocabularies measure favours sources with small vocabulary size
than the ones with large vocabulary size.

###### Effectiveness of domain-specific models on downstream NER tasks

After we quantify the similarity between source and target datasets, the next
step is to investigate the impact of source data on pre-trained language
representation models. We pre-train ELECTRA (Clark et al., 2020)—a sample-
efficient variant of BERT—on different sources separately, then observe how
the effectiveness of these pre-trained models varies in different downstream
NER datasets.

The most common approach of training domain-specific models is _continue pre-
training_ , which is used by BioBERT (Lee et al., 2020), ClinicalBERT
(Alsentzer et al., 2019), and BERTweet (Nguyen et al., 2020). Continue pre-
training approach starts from an existing pre-trained model—usually pre-
trained on large size generic data set—and continues training on a domain-
specific corpus. The main advantage of continue pre-training is that they can
inherit knowledge from language representation models pre-trained on generic
data, and thus be considered as capturing both generic domain and domain-
specific knowledge. However, we aim to investigate the impact of pre-training
data, therefore, we use the _learning from scratch_ approach, eschewing the
potential impact of generic data. We follow the hyper-parameter setting in
(Clark et al., 2020), shown in Table 4.5 to train the domain-specific models.
Training of each model took four days using 1 Nvidia Tesla v100 GPU.

Hyper-parameter | Value
---|---
Number of layers | 12
Hidden size | 256
Intermediate size | 1024
Attention heads | 4
Attention head size | 64
Embedding size | 128
Learning rate | 5e-4
Train steps | 800K
Vocab size | 30,994
Table 4.5. Pre-train hyper-parameters, which follow the practice of training
ELECTRA-SMALL in (Clark et al., 2020).

Evaluation results using these domain-specific models on downstream NER tasks
show that the effectiveness varies in different target datasets (Table 4.6).
In other words, no single source is suitable for all target NER datasets. It
is worthy note that most of these results (for example on BC2GM, BTC, CADEC,
SciERC) are lower than state-of-the-art results on these datasets with large
margin. This is mainly because our pre-trained models are smaller—smaller
hidden size, less number of attention heads, smaller embedding size—than the
ones in other studies.

The best performing models on each target data are all pre-trained on the most
similar source which is nominated by at least one similarity measure, except
for SciERC (Table 4.3 and 4.4).

| Book | MIMIC | News | PubMed | Wiki | Yelp
---|---|---|---|---|---|---
BC2GM | 72.4 (0.2) | 72.7 (0.5) | 74.4 (0.4) | 80.7 (0.2) | 75.1 (0.4) | 72.4 (0.3)
BTC | 70.4 (0.2) | 63.1 (0.3) | 75.2 (1.3) | 67.1 (0.7) | 74.9 (0.4) | 70.8 (0.5)
CADEC | 64.3 (0.6) | 67.5 (0.3) | 65.1 (0.3) | 66.1 (0.5) | 65.5 (0.5) | 66.1 (0.6)
EBM | 39.8 (0.4) | 41.0 (0.4) | 41.1 (0.6) | 43.5 (0.3) | 40.7 (0.4) | 40.8 (0.5)
i2b2-2010 | 78.7 (0.3) | 87.7 (0.2) | 79.8 (0.9) | 85.1 (0.3) | 79.6 (0.4) | 79.4 (0.3)
JNLPBA | 69.7 (0.2) | 70.1 (0.3) | 70.1 (0.7) | 73.1 (0.2) | 70.7 (0.2) | 70.0 (0.2)
NCBI-Disease | 77.7 (0.5) | 80.3 (0.7) | 77.9 (4.8) | 85.8 (0.5) | 80.8 (0.3) | 79.2 (0.8)
SciERC | 37.2 (4.1) | 23.8 (1.3) | 25.9 (2.2) | 41.0 (20.6) | 47.3 (1.4) | 38.7 (3.1)
WetLab | 78.2 (0.1) | 78.1 (0.2) | 78.1 (0.3) | 78.7 (0.2) | 78.3 (0.1) | 78.0 (0.1)
Table 4.6. The effectiveness of domain-specific pre-trained models on
downstream NER tasks. We report the mention level $F_{1}$ scores.

####  4.5.1. Predictiveness of similarity measures

To analyse how proposed similarity measures can be used to nominate the best
pre-training data option, we investigate the correlation between these
similarity values and the effectiveness of pre-trained models on target tasks.
Specifically, we employ Spearman rank-order correlation coefficient to measure
the relationship between the ranking of similarity values and NER results. For
example, given the target data set NCBI-disease, the rank of sources is
PubMed, Wikipedia, MIMIC, Yelp, News, and Book, if they are sorted based on
the effectiveness of different domain-specific models. Similarity, if they are
sorted based on the target vocabulary covered measure, the rank of sources is
PubMed, Wikipedia, News, MIMIC, Books, and Yelp. The Spearman rank-order
correlation coefficient between these two rankings is 0.71.

The results in Figure 4.3 show that these proposed similarity measures are
predictive of the effectiveness of the pre-training data, except for Jaccard
similarity of vocabularies.

Figure 4.3. Correlation between different similarity measures and the
effectiveness of domain-specific pre-trained models.

####  4.5.2. Comparison to publicly available pre-trained models

Literature shows substantial improvements are sometimes possible when pre-
training on large generic corpora (Liu et al., 2019; Baevski et al., 2019).
Given that pre-trained models are freely available, is it even necessary to
pre-train on similar data as proposed above? We compare to publicly available
ELECTRA models trained on 3.3 Billion tokens generic data. Note that the
publicly available model we choose has the same model size. It is also pre-
trained using the same hyper-parameters as ours, except that it is pre-trained
longer than ours (1.45M vs.0.8M).

| Domain-specific model | Generic domain model
---|---|---
BC2GM | 80.7 (0.2) | 63.6 (0.7)
BTC | 75.2 (1.3) | 75.3 (0.2)
CADEC | 67.5 (0.3) | 57.9 (0.3)
EBM | 43.5 (0.3) | 41.6 (0.2)
i2b2-2010 | 87.7 (0.2) | 82.5 (0.1)
JNLPBA | 73.1 (0.2) | 71.2 (0.4)
NCBI-Disease | 85.8 (0.5) | 81.8 (0.6)
SciERC | 47.3 (1.4) | 47.3 (0.5)
WetLab | 78.7 (0.2) | 78.8 (0.1)
Table 4.7. Comparison between our best performing domain-specific models and
the publicly available generic domain model.

These results, shown in Table 4.7 indicate that a small similar source reduces
the computational cost without sacrificing the performance. This is especially
important in practice, because collecting data and pre-training models are
expensive, in terms of both computational and environmental cost (Schwartz et
al., 2019).

###  4.6. Summary

This chapter focuses on whether there are cost-effective methods to nominate
datasets to pretrain language representation models that are building blocks
of NER models. We propose using different measures to measure different
aspects of similarity between source and target data. We investigate how these
measures correlate with the effectiveness of pre-trained models for NER tasks.
While different NLP tasks may rely on different aspects of language, our study
is a step towards systematically guiding researchers on their choice of data
for pre-training, and models pre-trained on small size domain-specific corpus
can outperform the one pre-trained on large size generic domain data.

##  Chapter 5 Transition-based Model for Discontinuous NER

Discontinuous mentions represent compositional concepts, for example disorders
or symptoms, that differ from concepts represented by individual components,
for example body locations or general feelings. In downstream applications
such as pharmacovigilance and summarization, recognising these discontinuous
mentions is more useful than recognising separate components. We propose a
transition-based model that can effectively recognise discontinuous mentions
without sacrificing the accuracy on continuous mentions.

###  5.1. Overview

NER is a critical component of biomedical text mining applications. In
pharmacovigilance, it can be used to identify adverse drug events in consumer
reviews in online medication forums, alerting medication developers,
regulators, and clinicians (Leaman et al., 2010; Sarker et al., 2015; Karimi
et al., 2015b). In clinical settings, NER can be used to extract and summarise
key information from electronic medical records such as conditions hidden in
unstructured doctors’ notes (Feblowitz et al., 2011; Wang et al., 2018b).
These applications require identification of complex entity mentions,
discontinuous and overlapping mentions, not seen in generic domains.

Widely used sequence tagging techniques encode two assumptions that do not
always hold: (1) mentions do not overlap, therefore each token can belong to
at most one mention; and, (2) mentions comprise continuous sequences of
tokens. Nested entity recognition addresses violations of the first assumption
(more discussions in Section 2.4). However, the violation of the second
assumption is comparatively less studied and requires handling discontinuous
mentions (see examples in Figure 5.1).

Figure 5.1. Examples involving discontinuous mentions, taken from the
ShARe/CLEF 13 (Pradhan et al., 2013) and CADEC (Karimi et al., 2015a)
datasets, respectively. The first example contains a discontinuous mention
‘left atrium dilated’, the second example contains two mentions that overlap:
‘muscle pain’ and ‘muscle fatigue’ (discontinuous).

In contrast to continuous mentions which are often short spans of text,
discontinuous mentions consist of _components_ that are separated by
_intervals_. Recognising discontinuous mentions is particularly challenging as
exhaustive enumeration of possible mentions, including discontinuous and
overlapping spans, is exponential to sentence length. Existing approaches for
discontinuous NER either suffer from high time complexity (McDonald et al.,
2005) or ambiguity in translating intermediate representations into mentions
(Tang et al., 2013a; Metke-Jimenez and Karimi, 2016; Muis and Lu, 2016). In
addition, current arts use traditional approaches that rely on manually
designed features, which are tailored to recognise specific entity categories.
Also, these features usually do not generalise well in different types of text
(Leaman et al., 2015).

###### Motivations

The main motivation for recognising discontinuous mentions is that they
usually represent _compositional concepts_ that differ from concepts
represented by individual components. For example, the mention ‘left atrium
dilated’ in the first example of Figure 5.1 describes a disorder which has its
own CUI (Concept Unique Identifier) in UMLS (Unified Medical Language System),
whereas both ‘left atrium’ and ‘dilated’ also have their own CUIs. In
downstream applications such as pharmacovigilance and summarization,
recognising these discontinuous mentions that refer to disorders or symptoms
is more useful than recognising separate components which may refer to body
locations or general feelings.

Another motivation for discontinuous NER is that discontinuous mentions
usually overlap, and separating these overlapping mentions rather than
identifying them as a single mention is important for downstream tasks, such
as entity linking where the assumption is that the input mention refers to one
entity (Shen et al., 2014).

In this chapter, we first characterise three datasets, from the biomedical
domain, with a substantial number of discontinuous mentions (Section 5.2).
Then, we introduce a transition-based model that can recognise discontinuous
mentions (Section 5.3). Through experiments, we show that our model can
effectively recognise discontinuous mentions without sacrificing the accuracy
on continuous mentions (Section 5.4). Analysis also suggests that our model is
better than existing discontinuous NER models at handling long mentions and
mentions that do not overlap or overlap at left, resulting in higher recall
(Section 5.5).

###  5.2. Datasets

Although some text annotation tools, such as BRAT (Stenetorp et al., 2012),
allow discontinuous annotations, corpora annotated with large number of
discontinuous mentions are still rare because they are hard to annotate. We
describe three datasets from the biomedical domain that include a substantial
number of discontinuous mentions: CADEC (Karimi et al., 2015a), ShARe/CLEF
2013 (Pradhan et al., 2013) and ShARe/CLEF 2014 (Mowery et al., 2014). We then
motivate the discontinuous NER task.

CADEC corpus is sourced from posts from
AskaPatient111https://www.askapatient.com/. Accessed data: 3rd September 2024,
a forum where patients can discuss their experiences with medications. The
entity categories annotated in CADEC include drug, Adverse Drug Event (ADE),
disease and symptom. In this work, we only consider the ADE annotations
because only the ADEs involve discontinuous mentions. Note that ADEs in CADEC
are defined as span of text that are clearly associated with a drug and should
have the corresponding MedDRA (Medical Dictionary for Regulatory Activities)
term. ShARe/CLEF 2013, specifically Task 1 of the ShARe/CLEF eHealth
evaluation lab 2013, focuses on identification of disorder mentions in
clinical reports. The corpus is sourced from de-identified clinical reports,
including discharge summaries, electrocardiogram, echocardiogram, and
radiology reports (Johnson et al., 2016). A disorder mention is defined as any
span of text which can be mapped onto a concept in the Disorder semantic group
of SNOMED-CT (Systematised Nomenclature of Medicine – Clinical Terms).
ShARe/CLEF 2014, an extension of the ShARe/CLEF 2013 task, focuses on template
filling of disorder attributes. That is, given a disorder mention and its
surrounding words, recognise the attributes of the disorder mention from its
context, including subject class, severity indicator, uncertainty indicator.
In this work, we frame ShARe/CLEF 2014 as a disorder-NER dataset. That is, we
consider only disorder annotations, without taking their attributes into
consideration.

| Dataset
---|---
| CADEC |  ShARe/CLEF 13 |  ShARe/CLEF 14
Text type | online posts | clinical notes | clinical notes
Entity type | ADE | Disorder | Disorder
# Documents | 1,250 | 298 | 433
# Tokens | 121K | 264K | 494K
# Sentences | 7,597 | 18,767 | 34,618
# Mentions | 6,318 | 11,161 | 19,131
# Disc.M | 675 (10.6) | 1,090 (9.7) | 1,710 (8.9)
Avg mention L. | 2.7 | 1.8 | 1.7
Avg Disc.M L. | 3.5 | 2.6 | 2.5
Avg interval L. | 3.3 | 3.0 | 3.2
Discontinuous Mentions
2 components | 650 (95.7) | 1,026 (94.3) | 1,574 (95.3)
3 components |  027 (03.9) |  0062 (05.6) |  0076 (04.6)
4 components |  002 (00.2) |  0000 (00.0) |  0000 (00.0)
No overlap |  082 (12.0) |  0582 (53.4) |  0820 (49.6)
Overlap at left | 351 (51.6) |  0376 (34.5) |  0616 (37.3)
Overlap at right | 152 (22.3) |  0102 (09.3) |  0170 (10.3)
Multiple overlaps |  094 (13.8) |  0028 (02.5) |  0044 (02.6)
Continuous Mentions
Overlap | 326 (05.7) |  0157 (01.5) |  0228 (01.3)
Table 5.1. The descriptive statistics of the datasets. ADE: adverse drug
events; Disc.M: discontinuous mentions; Disc.M L.: discontinuous mention
length, where intervals are not counted. Numbers in parentheses are the
percentage of each category. Note that due to sentence segmentation issue,
there are 13 and 64 mentions crossing multiple sentences in ShARe/CLEF 2013
and ShARe/CLEF 2014, respectively. We remove these mentions, as we frame the
task as a sentence-level NER problem.

Descriptive statistics of these three datasets is listed in Table 5.1. On
average, discontinuous mentions are longer than continuous mentions, because
they consist of several components, and the intervals between different
components make the total length of span even longer. Another important
characteristic of discontinuous mentions is that they usually _overlap_. That
is, several mentions may share components that refer to the same body location
(e.g., ‘muscle’ in ‘muscle pain and fatigue’), or the same feeling (e.g.,
‘Pain’ in ‘Pain in knee and foot’). From this perspective, we also categorise
discontinuous mentions into the following groups:

  * •

No overlap: in such cases, the discontinuous mention can be intervened by
severity indicators (e.g., ‘is mildly’ in sentence ‘left atrium is mildly
dilated’), preposition (e.g., ‘on my’ in sentence ‘…rough on my stomach…’) and
so on. This category accounts for half of discontinuous mentions in the
ShARe/CLEF datasets but only 12% in CADEC.

  * •

Left overlap: the discontinuous mention shares one component with other
mentions, and the shared component is at the beginning of the discontinuous
mention. This is usually accompanied with coordination structure (e.g., the
shared component ‘muscle’ in ‘muscle pain and fatigue’). Conjunctions (e.g.,
‘and’, ‘or’) are clear indicators of the coordination structure. However,
clinical notes (ShARe/CLEF datasets) are usually written by practitioners
under time pressure. They often use commas or slashes rather than
conjunctions. This category accounts for more than half of discontinuous
mentions in CADEC and one third in ShARe/CLEF.

  * •

Right overlap: similar to left overlap, although the shared component is at
the end. For example, ‘hip/leg/foot pain’ contains three mentions that share
the token ‘pain’.

  * •

Multi-overlap: the discontinuous mention shares multiple components with the
others, which usually forms _crossing compositions_. For example, the sentence
‘Joint and Muscle Pain / Stiffness’ contains four mentions: ‘Joint Pain’,
‘Joint Stiffness’, ‘Muscle Stiffness’ and ‘Muscle Pain’, where each
discontinuous mention share two components with the others.

Although these three datasets – CADEC, ShARe/CLEF 2013 and ShARe/CLEF 2014 –
share similar field (the subject matter of the content being discussed), the
tenor (the participants in the discourse, their relationships to each other,
and their purposes) of CADEC is very different from the ShARe/CLEF datasets.
Specially, laymen authors (CADEC) tend to use idioms or ungrammatical phrases
to describe their feelings, whereas professional practitioners (ShARe/CLEF)
tend to use compact terms for efficient communications. This difference of
tenor results in different features of mentions between these datasets. That
is, the mentions in CADEC are overall longer than those in ShARe/CLEF
datasets, and larger ratio of discontinuous mentions in CADEC are involved in
overlapping structure (Table 5.1).

###  5.3. Proposed Model

We propose a transition-based model based on the shift-reduce parser (Watanabe
and Sumita, 2015; Lample et al., 2016) that employs a _stack_ to store
partially processed spans and a _buffer_ to store unprocessed tokens. The
learning problem is then framed as: given the state of the parser, predict an
action which is applied to change the state of the parser. This process is
repeated until the parser reaches the end state, which is the stack and buffer
are both empty.

Similar to prior work (Metke-Jimenez and Karimi, 2016; Muis and Lu, 2016) that
first predict an intermediate representation of mentions, which are then
decoded into the final mentions, our proposed transition-based model uses a
sequence of actions as the intermediate representation (refer to Section 2.4).

The main difference between our model and the ones in (Watanabe and Sumita,
2015; Lample et al., 2016) is the set of transition actions. Watanabe and
Sumita (2015) use SHIFT, REDUCE, UNARY, FINISH, and IDEA for the constituent
parsing system. Lample et al. (2016) use SHIFT, REDUCE, OUT for the flat NER
system. Inspired by these models, we design a set of actions specifically for
recognising discontinuous and overlapping structure. There is a total of six
actions in our model:

  * •

SHIFT moves the first token from the buffer to the stack; it implies this
token is part of an entity mention.

  * •

OUT pops the first token of the buffer, indicating it does not belong to any
mention.

  * •

COMPLETE pops the top span of the stack, outputting it as an entity mention.
If we are interested in multiple entity categories, we can extend this action
to COMPLETE-$y$ which labels the mention with entity category $y$.

  * •

REDUCE pops the top two spans $s_{0}$ and $s_{1}$ from the stack and
concatenates them as a new span which is then pushed back to the stack.

  * •

LEFT-REDUCE is similar to the REDUCE action, except that the span $s_{1}$ is
kept in the stack. This action indicates the span $s_{1}$ is involved in
multiple mentions. In other words, several mentions share $s_{1}$ which could
be a single token or several tokens.

  * •

RIGHT-REDUCE is the same as LEFT-REDUCE, except that $s_{0}$ is kept in the
stack.

Figure 5.2 shows an example of how the proposed parser recognises entity
mentions from a sentence.

Figure 5.2. An example sequence of transitions. Given the states of stack and
buffer (blue highlighted), as well as the previous actions, predict the next
action (i.e., LEFT-REDUCE) which is then applied to change the states of stack
and buffer.

####  5.3.1. Representation of the parser state

Given a sequence of $N$ tokens, we first run a bi-directional LSTM (Graves et
al., 2013) to derive the contextual representation of each token.
Specifically, for the $i$-th token in the sequence, its representation can be
denoted as:

$\bm{\tilde{c_{i}}}=\left[\overrightarrow{\textsc{LSTM}}(\bm{t_{0}},\dots,\bm{t_{i}});\overleftarrow{\textsc{LSTM}}(\bm{t_{i}},\dots,\bm{t_{N-1}})\right],$

where $\mathbf{t_{i}}$ is the concatenation of the embeddings for the $i$-th
token, its character level representation learned using a CNN network (Ma and
Hovy, 2016). Pretrained contextual word representations have shown its
usefulness on improving various NLP tasks. Here, we can also concatenate
pretrained contextual word representations using ELMo (Peters et al., 2018)
with $\mathbf{\tilde{c_{i}}}$, resulting in:

$\bm{c_{i}}=\big{[}\bm{\tilde{c_{i}}};\textsc{ELMo}_{i}\big{]},$ (5.1)

where $\textsc{ELMo}_{i}$ is the output representation of pretrained ELMo
models (frozen) for the $i$-th token. These token representations $\mathbf{c}$
are directly used to represent tokens in the buffer.

Following the work in (Dyer et al., 2015), we use StackLSTM to represent spans
in the stack. That is, if a token is moved from the buffer to the stack, its
representation is learned using:

$\bm{s_{0}}=\textsc{StackLSTM}(\bm{s_{D}}\dots\bm{s_{1}};\bm{c}_{\text{SHIFT}}),$

where $D$ is the number of spans in the stack. Once REDUCE related actions are
applied, we use a multi-layer perceptron to learn the representation of the
concatenated span. For example, the REDUCE action takes the representation of
the top two spans in the stack: $\bm{s_{0}}$ and $\bm{s_{1}}$, and produces a
new span representation:

$\bm{\tilde{s}}=\bm{W}^{T}\big{[}\bm{s_{0}};\bm{s_{1}}\big{]}+\bm{b},$ (5.2)

where $\bm{W}$ and $\bm{b}$ denote the parameters for the composition
function. The new span representation $\bm{\tilde{s}}$ is pushed back to the
stack to replace the original two spans: $\bm{s_{0}}$ and $\bm{s_{1}}$.

####  5.3.2. Capturing discontinuous dependencies

We hypothesise that the interactions between spans in the stack and tokens in
the buffer are important factors in recognising discontinuous mentions.
Considering the example in Figure 5.2, a span in the stack (e.g., ‘muscle’)
may need to combine with a future token in the buffer (e.g., ‘fatigue’). To
capture this interaction, we use multiplicative attention (Luong et al., 2015)
to let the span in the stack $\bm{s_{i}}$ learn which token in the buffer to
attend, and thus a weighted sum of the representation of tokens in the buffer
$\bm{B}$:

$\begin{split}\bm{s^{a}_{i}}&=\textsc{Attention}(\bm{s_{i}},\bm{B},\bm{B})\\\
&=\textsc{softmax}(\bm{s_{i}^{T}W^{a}_{i}B})\bm{B}.\end{split}$ (5.3)

We use distinct $\bm{W}_{i}^{a}$ for spans in different positions $\bm{s_{i}}$
separately.

####  5.3.3. Selecting an action

Finally, we build the parser representation as the concatenation of the
representation of top three spans from the stack
($\bm{s_{0}},\bm{s_{1}},\bm{s_{2}}$) and its attended representation
($\bm{s^{a}_{0}}$, $\bm{s^{a}_{1}}$, $\bm{s^{a}_{2}}$), as well as the
representation of the previous action $\bm{a}$, which is learned using a
simple unidirectional LSTM. If there are less than 3 spans in the stack or no
previous action, we use randomly initialised vectors $\bm{s_{empty}}$ or
$\bm{a_{empty}}$ to replace the corresponding vector. This parser
representation is used as input for the final softmax prediction layer to
select the next action.

Note that, given one parser state, not all types of actions are valid. For
example, if the stack does not contain any span, only SHIFT and OUT actions
are valid because all other actions involve popping spans from the stack. We
employ hard constraints that we only select the most likely action from valid
actions.

###  5.4. Experimental Results

To evaluate the effectiveness of our proposed model, we run experiments on
previous described three datasets: CADEC, ShARe/CLEF 2013 and ShARe/CLEF 2014,
and compare the effectiveness of our model against several baselines.

####  5.4.1. Baseline models

We choose one flat NER model which is strong at recognising continuous
mentions, and two discontinuous NER models as our baseline models:

###### Flat model

To train the flat model on our datasets, we use an off-the-shelf framework:
Flair (Akbik et al., 2018), which achieves the state-of-the-art performance on
CoNLL 2003 dataset. Recall that the flat model cannot be directly applied to
datasets containing discontinuous mentions. Following the practice in
(Stanovsky et al., 2017), we replace the discontinuous mention with the
shortest span that fully covers it, and merge overlapping mentions into a
single mention that covers both. Different from (Stanovsky et al., 2017), we
apply these changes only on the training set, and not on the development and
the test sets.

###### BIO extension model

The original implementation in (Metke-Jimenez and Karimi, 2016) used a CRF
model with manually designed features. We report their results on CADEC in
Table 5.2 and re-implement a BiLSTM-CRF-ELMo model using their tag schema
(denoted as ‘BIO extension’ in Table 5.2).

###### Graph-based model

The original paper of (Muis and Lu, 2016) only reported the evaluation results
on sentences which contain at least one discontinuous mention. We use their
implementation to train the model and report evaluation results on the whole
test set (denoted as ‘Graph’ in Table 5.2). We argue that it is important to
see how a discontinuous NER model works not only on the discontinuous mentions
but also on all the mentions, especially since, in real datasets, the ratio of
discontinuous mentions cannot be made a priori.

####  5.4.2. Experimental setup

As CADEC does not have an official train-test split, we follow (Metke-Jimenez
and Karimi, 2016) and randomly assign 70% of the posts as the training set,
15% as the development set, and the remaining posts as the test set. The
train-test splits of ShARe/CLEF 13 and 14 are both from their corresponding
shared task settings, except that we randomly select 10% of documents from
each training set as the development set. The original ShARe/CLEF 14 task
focuses on template filling of disorder attributes: that is, given a disorder
mention, recognise the attribute from its context. In this work, we use its
mention annotations and frame the task as a discontinuous NER task. Micro
average strict match $F_{1}$ score is used to evaluate the effectiveness of
the model. The trained model which is most effective on the development set,
measured using the $F_{1}$ score, is used to evaluate the test set. All
experiments are repeated five times using different random seeds and averaged
results are reported.

####  5.4.3. Results

When evaluated on the whole test set, our model outperforms three baseline
models, as well as over previous reported results in the literature, in terms
of recall and $F_{1}$ scores (Table 5.2).

|  | CADEC |  | ShARe 2013 |  | ShARe 2014
---|---|---|---|---|---|---
Model |  | P | R | F |  | P | R | F |  | P | R | F
Metke-Jimenez and Karimi (2016) |  | 64.4 | 56.5 | 60.2 |  | – | – | – |  | – | – | –
Tang et al. (2018) |  | 67.8 | 64.9 | 66.3 |  | – | – | – |  | – | – | –
Tang et al. (2013b) |  | – | – | – |  | 80.0 | 70.6 | 75.0 |  | – | – | –
Flat |  | 65.3 | 58.5 | 61.8 |  | 78.5 | 66.6 | 72.0 |  | 76.2 | 76.7 | 76.5
BIO extension |  | 68.7 | 66.1 | 67.4 |  | 77.0 | 72.9 | 74.9 |  | 74.9 | 78.5 | 76.6
Graph |  | 72.1 | 48.4 | 58.0 |  | 83.9 | 60.4 | 70.3 |  | 79.1 | 70.7 | 74.7
Ours |  | 68.9 | 69.0 | 69.0 |  | 80.5 | 75.0 | 77.7 |  | 78.1 | 81.2 | 79.6
Table 5.2. Evaluation results in terms of precision (P), recall (R) and
$F_{1}$ score (F).

The graph-based model achieves highest precision, but with substantially lower
recall, therefore obtaining lowest $F_{1}$ scores. In contrast, our model
improves recall over flat and BIO extension models as well as previously
reported results, without sacrificing precision. This results in more balanced
precision and recall. Improved recall is especially encouraging for our
motivating pharmacovigilance and medical record summarization applications,
where recall is at least as important as precision. Note that most of these
previous models are tailored for specific entity categories, and utilise
domain-specific resources. For example, (Tang et al., 2013b), the best-
performing system participated in the ShARe/CLEF 2013 shared task, utilise
several external domain-specific resources, such as MetaMap, cTAKES and UMLS.
We avoid these tailored resources in our model. We argue that this makes our
model more generic and robust, especially since we apply the hyper-parameters
tuned on CADEC directly to ShARe/CLEF datasets and obtain similar improvements
with respect to the benchmarks.

###### Effectiveness on recognising discontinuous mentions

Recall that only 10% of mentions in these three datasets are discontinuous. To
evaluate the effectiveness of our proposed model on recognising discontinuous
mentions, we follow the evaluation approach in (Muis and Lu, 2016) where we
construct a subset of test set where only sentences with at least one
discontinuous mention are included (Table 5.3). We also report the evaluation
results when only discontinuous mentions are considered (Table 5.4). Note that
sentences in the former setting usually contain continuous mentions as well,
including those involved in overlapping structure (e.g., ‘muscle pain’ in the
sentence ‘muscle pain and fatigue’). Therefore, the flat model, which cannot
predict any discontinuous mentions, still achieves $38\%$ $F_{1}$ on average
when evaluated on these sentences with at least one discontinuous mention, but
fails to recognise discontinuous mentions.

|  | CADEC |  |  ShARe/CLEF 2013 |  |  ShARe/CLEF 2014
---|---|---|---|---|---|---
Model |  | P | R | F |  | P | R | F |  | P | R | F
Flat |  | 50.2 | 36.7 | 42.4 |  | 43.5 | 28.1 | 34.2 |  | 41.5 | 31.9 | 36.0
BIO extension |  | 63.8 | 52.0 | 57.3 |  | 51.8 | 39.5 | 44.8 |  | 37.5 | 38.4 | 37.9
Graph |  | 69.5 | 43.2 | 53.3 |  | 82.3 | 47.4 | 60.2 |  | 60.0 | 52.8 | 56.2
Ours |  | 66.5 | 64.3 | 65.4 |  | 70.5 | 56.8 | 62.9 |  | 61.9 | 64.5 | 63.1
Table 5.3. Evaluation results on sentences that contain at least one discontinuous mention.  |  | CADEC |  |  ShARe/CLEF 2013 |  |  ShARe/CLEF 2014
---|---|---|---|---|---|---
Model |  | P | R | F |  | P | R | F |  | P | R | F
Flat |  | 0 | 0 | 0 |  | 0 | 0 | 0 |  | 0 | 0 | 0
BIO extension |  | 5.8 | 1.0 | 1.8 |  | 39.7 | 12.3 | 18.8 |  | 8.8 | 4.5 | 6.0
Graph |  | 60.8 | 14.8 | 23.9 |  | 78.4 | 36.6 | 50.0 |  | 42.7 | 39.5 | 41.1
Ours |  | 41.2 | 35.1 | 37.9 |  | 78.5 | 39.4 | 52.5 |  | 56.1 | 43.8 | 49.2
Table 5.4. Evaluation results on discontinuous mentions only.

Our model again achieves the highest $F_{1}$ and recall in all three datasets
under both settings. The comparison between these two evaluation results also
shows the necessity of comprehensive evaluation settings. The BIO extension
model outperforms the graph-based model in terms of $F_{1}$ score on CADEC,
when evaluated on sentences with discontinuous mentions. However, it achieves
only $1.8$ $F_{1}$ when evaluated on discontinuous mentions only. The main
reason is that most of discontinuous mentions in CADEC are involved in
overlapping structure ($88\%$, cf. Table 5.1), and the BIO extension model is
better than the graph-based model at recognising these continuous mentions. On
ShARe/CLEF 2013 and 2014, where the portion of discontinuous mentions involved
in overlapping is much less than on CADEC, the graph-based model clearly
outperforms BIO extension model in both evaluation settings.

Graph based model again achieves highest precision on all three datasets. It
also outperforms BIO extension model on ShARe/CLEF 2013 and 2014 in terms of
$F_{1}$ score, but not on CADEC. Graph based model employs lots of handcrafted
features for clinical notes (e.g., note type, section name, word-level
semantic category extracted from UMLS). These handcrafted features usually
lead to high precision but are not general enough to recall unseen mentions.
In addition, they usually do not generalise well in different types of text
(i.e., online posts in CADEC). In contrast, we avoid these handcrafted
features in our model.

###  5.5. Analysis

####  5.5.1. Impact of mention and interval length

Discontinuous mentions usually represent compositional concepts that consist
of multiple components. Therefore, these mentions are usually longer than
continuous mentions (Table 5.1). In addition, intervals between components
make the total length of span involved even longer. Previous work shows that
flat NER performance degrades when applied on long mentions (Augenstein et
al., 2017; Xu et al., 2017; Lange et al., 2020).

(a) CADEC

(b) CADEC

(c) ShARe/CLEF 2013

(d) ShARe/CLEF 2013

(e) ShARe/CLEF 2014

(f) ShARe/CLEF 2014

Figure 5.3. The impact of mention length and interval length on recall.
Mentions with interval length of zero are continuous mentions. Numbers in
parentheses are the number of gold mentions.

We experiment to measure the ability of different models on recalling mentions
of different lengths, and to observe the impact of interval lengths. We find
that the recall of all models decreases with the increase of mention length in
general (Figure 5.3 (a – c)), which is similar to previous observations in the
literature on flat mentions Lange et al. (2020). However, the impact of
interval length is not straightforward. Mentions with very short interval
lengths are as difficult as those with very long interval lengths to be
recognised (Figure 5.3 (d – f)). On CADEC, discontinuous mentions with
interval length of two are easiest to be recognised (Figure 5.3 (d)), whereas
those with interval length of three are easiest on ShARe/CLEF 2013 and 2014.
We hypothesise this also relates to annotation inconsistency, because very
short intervals may be overlooked by annotators.

Our method achieves highest recall among all models in most settings. This
demonstrates our model is effective to recognise both continuous and
discontinuous mentions with various lengths. In contrast, the BIO extension
model is only strong at recalling continuous mentions (outperforming the
graph-based model), but fails on discontinuous mentions (interval lengths
larger than zero).

####  5.5.2. Impact of overlapping structure

Another characteristic of discontinuous mentions is that they usually overlap
(Section 5.2). Previous study shows that the intervals between components can
be problematic for coordination boundary detection (Ficler and Goldberg
(2016)). Conversely, we want to observe whether the overlapping structure may
help or hinder discontinuous entity recognition. We categorise discontinuous
mentions into different subsets, described in Section 5.2, and measure the
effectiveness of different discontinuous NER models on each category.

|  |  | CADEC |  | ShARe 2013 |  | ShARe 2014
---|---|---|---|---|---|---|---
| Model |  | # | F |  | # | F |  | # | F
No overlap | BIO extension |  | 9 | 0.0 |  | 41 | 7.5 |  | 39 | 0.0
Graph |  | 0.0 |  | 32.1 |  | 45.2
Ours |  | 0.0 |  | 36.1 |  | 57.1
Overlap at left | BIO extension |  | 54 | 6.0 |  | 11 | 25.0 |  | 30 | 15.7
Graph |  | 9.2 |  | 45.5 |  | 37.7
Ours |  | 28.6 |  | 33.3 |  | 49.2
Overlap at right | BIO extension |  | 16 | 0.0 |  | 19 | 0.0 |  | 5 | 0.0
Graph |  | 45.2 |  | 21.4 |  | 0.0
Ours |  | 29.3 |  | 13.3 |  | 0.0
Multiple overlaps | BIO extension |  | 15 | 0.0 |  | 0 | – |  | 6 | 0.0
Graph |  | 0.0 |  | – |  | 0.0
Ours |  | 0.0 |  | – |  | 0.0
Table 5.5. Evaluation results on different categories of discontinuous
mentions. ‘#’ columns show the number of gold discontinuous mentions in
development set of each category.

From Table 5.5, we find that our model achieves better results on
discontinuous mentions belonging to _No overlap_ category on ShARe/CLEF 2013
and 2014, and _Overlap at left_ category on CADEC and ShARe/CLEF 2014\. Note
that _No overlap_ category accounts for half of discontinuous mentions in
ShARe/CLEF 2013 and 2014, whereas _Overlap at left_ accounts for half in CADEC
(Table 5.1). Graph-based model achieves better results on _Overlap at right_
category. On the _Multiple overlaps_ category, no models is effective 222Our
model cannot recognise all mentions belonging to this category in theory. For
example, if two mentions overlap at both the left and the right, our model can
predict only one of them., which emphasises the challenges of dealing with
this syntactic phenomena (examples can be found in Section 5.5.3). We note,
however, the portion of discontinuous mentions belonging to this category is
very small in all three datasets.

Although our model achieves better results on _No overlap_ category on
ShARe/CLEF 2013 and 2014, it does not predict correctly any discontinuous
mention belonging to this category on CADEC. The ineffectiveness of our model,
as well as other discontinuous NER models, on CADEC _No overlap_ category can
be attributed to two reasons: 1) the number of discontinuous mentions
belonging to this category in CADEC is small (around 12%), rending the
learning process more difficult. 2) the gold annotations belonging to this
category are inconsistent from a linguistic perspective. For example, severity
indicators are annotated as the interval of the discontinuous mention
sometimes, but not often. Note that this may be reasonable from a medical
perspective, as some symptoms are roughly grouped together no matter their
severity, whereas some symptoms are linked to different concepts based on
their severity and severe adverse drug reactions are especially on the radar.

####  5.5.3. Example predictions

We find that previous models often fail to identify discontinuous mentions
that involve long and overlapping spans. For example, the sentence ‘Severe
joint pain in the shoulders and knees.’ contains two mentions: ‘Severe joint
pain in the shoulders’ and ‘Severe joint pain in the knees’. Graph-based model
does not identify any mention from this sentence, resulting in a low recall.
The BIO extension model predicts most of these tags (8 out of 9) correctly,
but fails to decode into correct mentions (predict ‘Severe joint pain in the’,
resulting in a false positive, while it misses ‘Severe joint pain in the
shoulders’). In contrast, our model correctly identifies both of these two
mentions.

Sentence |  Walked like that for about six months with increasing pain , especially in right thigh which felt .
---|---
Gold mentions |  1\. pain in right thigh
Predictions |  1\. pain [BIO extension]
|  No prediction [Graph]
|  1\. increasing pain [Ours]
Sentence |  Stated with joint and pain and muscle weakness , depression , fatigue and cramps .
Gold mentions |  1\. joint pain; 2. muscle weakness; 3. depression; 4. fatigue; 5. cramps
Predictions |  1\. joint and pain; 2. muscle weakness; 3. depression; 4. fatigue; 5. cramps [BIO extension]
|  No prediction [Graph]
|  1\. muscle weakness; 2. pain weakness; 3. joint weakness; 4. depression; 5.
fatigue; 6. cramps [Ours]
Sentence |  stopped taking them 4 years ago and still suffer terrible muscle pain and wasting .
Gold mentions |  1\. muscle pain; 2. muscle wasting
Predictions |  1\. terrible muscle pain; 2. wasting [BIO extension]
|  No prediction [Graph]
|  1\. muscle pain; 2. wasting [Ours]
Sentence |  Then I sated having hip / leg / foot pain and numbness .
Gold mentions |  1\. hip pain; 2. leg pain; 3. foot pain; 4. numbness
Predictions |  1\. foot pain; 2. pain; 3. numbness [BIO extension]
|  No prediction [Graph]
|  1\. stated; 2. hip pain; 3. leg pain; 4. foot pain; 5. numbness [Ours]
Sentence |  Severe joint pain in the shoulders and knees .
Gold mentions |  1\. Severe joint pain in the shoulders; 2. Severe joint pain in the knees
Predictions |  1\. Severe joint pain in the; 2. Severe joint pain in the knees [BIO extension]
|  None [Graph]
|  1\. Severe joint pain in the shoulders; 2. Severe joint pain in the knees
[Ours]
Sentence |  Joint and Muscle Pain / Stiffness .
Gold mentions |  1\. Joint pain; 2. Muscle Stiffness; 3. Muscle Pain; 4. Joint Stiffness
Predictions |  1\. Joint; 2. Muscle Pain; 3. Stiffness [BIO extension]
|  1\. Joint pain; 2. Muscle Pain / Stiffness; 3. Stiffness [Graph]
|  1\. Joint pain; 2. Muscle Pain; 3. Stiffness [Ours]
Table 5.6. Example sentences involving discontinuous entity mentions and
predictions using different methods. These examples are taken from CADEC. Gold
discontinuous mentions are highlighted in bold. We cross out the incorrect
predictions (false positives) for easy understanding.

Another observation is that no model can fully recognise mentions which form
crossing compositions. For example, the sentence ‘Joint and Muscle Pain /
Stiffness’ contains four mentions: ‘Joint Pain’, ‘Joint Stiffness’, ‘Muscle
Stiffness’ and ‘Muscle Pain’, all of which share multiple components with the
others. Our model correctly predicts ‘Joint Pain’ and ‘Muscle Pain’, but it
mistakenly predicts ‘Stiffness’ itself as a mention (Table 5.6).

####  5.5.4. Ablation studies

To empirically evaluate the importance of attention and ELMo components, we
test the performance of model variants where attention and ELMo are removed
separately on CADEC and ShARe/CLEF 2013 datasets.

|  | CADEC |  |  ShARe/CLEF 2013
---|---|---|---|---
Model |  | All | Subset w. Disc. |  | All | Subset w. Disc.
Full |  | 68.4 | 65.4 |  | 77.2 | 64.3
-Attention |  | 68.4 | 63.3 |  | 76.8 | 62.3
\- ELMo |  | 66.7 | 62.2 |  | 75.2 | 60.9
Table 5.7. Ablation study to estimate the contribution of attention and ELMo
components.

The results in Table 5.7 show that removing attention hurts the performance
when evaluated on sentences with discontinuous mentions (w. Disc. columns),
but have little impact on the complete test set where continuous mentions are
prevalent. Since we use BiLSTM to derive contextual representation for each
token, we believe these contextual representations are effective at
recognising continuous mentions, but have trouble identifying intervals within
discontinuous mentions. Attention mechanism, via allowing tokens interacting
with distant tokens, can capture additional discontinuous dependencies which
are not captured by BiLSTM. In terms of the ELMo component, we find that it
contributes approximately 2 $F_{1}$ score when evaluated on the complete test
set and around 4 $F_{1}$ when evaluated on sentences with discontinuous
mentions, demonstrating the usefulness of pretrained word representations.

###  5.6. Summary

Recognising discontinuous mentions that represent compositional concepts is
important for downstream applications such as pharmacovigilance. We propose an
end-to-end transition-based model for discontinuous NER. It makes use of
specialised actions and attention mechanism to determine whether a span is the
component of a discontinuous mention or not. We evaluate our model on three
biomedical datasets with a substantial number of discontinuous mentions and
demonstrate that our model can effectively recognise discontinuous mentions
without sacrificing the accuracy on continuous mentions. Analysis also
suggests that our model is better than existing discontinuous NER models at
handling long mentions, resulting in higher recall.

##  Chapter 6 Conclusions

Recognising biomedical names from scholarly articles, clinical notes, and
social media data is a fundamental NLP task that can benefit many downstream
biomedical NLP and information retrieval applications. However, due to the
unique characteristics of biomedical names and the stylistic variation in
biomedical language—used by biomedical researchers, practitioners, patients
and other participants—biomedical NER needs to solve challenges comparatively
less studied in the generic domain NER applications.

In this thesis, we first identified challenges of applying standard sequence
tagger to recognise biomedical names. Although sequence tagging techniques
have demonstrated their effectiveness in generic domain NER, achieving state-
of-the-art performance in many benchmarks, they suffer from three problems
when being applied in the biomedical domain:

  * •

Biomedical names may consist of non-consecutive spans and they may overlap
with each other. The main reason of this complex structure in biomedical names
is that many biomedical concepts are compositional. For example, a symptom
description may consist of several components: body location, severity
indicator, and general feeling, and these components may locate far away from
each other.

  * •

Training of neural based sequence taggers usually requires large training set,
which is difficult to obtain in the biomedical domain. Annotating biomedical
NER datasets usually requires domain-knowledge, and sometimes even unlabelled
data are unavailable due to legal reasons.

  * •

State-of-the-art sequence taggers are usually enhanced by language
representation models pre-trained on large set of generic domain unlabelled
data. Domain shift between these out-of-domain pre-training data and the
target biomedical data usually results in a performance drop.

Targeting these three problems, we explored the corresponding research
directions.

We proposed a transition-based model for discontinuous NER. The proposed model
is an end-to-end model with generic neural encoding that allows us to leverage
specialised actions and attention mechanism to determine whether a span is the
component of a discontinuous mention or not. We evaluate our model on three
biomedical datasets with a substantial number of discontinuous mentions and
demonstrate that our model can effectively recognise discontinuous mentions
without sacrificing the accuracy on continuous mentions.

We designed several easy to use data augmentation methods for the NER task:
Label-wise token replacement, Synonym replacement, Mention replacement and
Shuffle within segments. These augmentations do not rely on any externally
trained models, such as machine translation models or syntactic parsing
models, which are by themselves difficult to train in a low-resource domain-
specific scenario. Through experiments on two biomedical datasets, we show
that simple data augmentation can improve performance even over strong
baselines, where large scale pre-trained language representation models are
used. We leave the exploration of combining these data augmentation methods
with other NER models, such as the transition-based model we proposed, for
future work.

We analysed different aspects of similarity between domains, and employed
cost-effective measures to quantify domain similarity. We demonstrated that
these measures are good predictors of the usefulness of pre-trained language
representation models on downstream NER task. We find that human intuition
favour field (the subject matter being discussed) over tenor (the participants
of the discourse and their purpose) when they select in-domain pre-training
data. Results suggest that this intuition may be unreliable when the target
data set locates in the intersection of several domains.

Based on the discoveries presented in this thesis, we see two future
directions worth exploring.

  * •

The first one is on incorporating existing biomedical knowledge base. In
Chapter 3, we explored the data augmentation methods that make use of the
original training set and a generic lexical database of English. Similar
augmentation methods can also be applied to biomedical knowledge base. For
example, a biomedical concept—defined using CUI in UMLS—can have several
aliases from various vocabularies. These aliases can be used to create
augmented sentences in the data augmentation settings. In Chapter 4, we
investigated the impact of pre-training data on the effectiveness of pre-
trained language representation models, where we pre-train models using only
the unlabelled text. We believe more sophisticated pre-training tasks based on
biomedical knowledge base can create pre-trained models that capture both
language and biomedical knowledge.

  * •

The second direction is on investigating the impact of NER performance on
downstream tasks, such as entity linking, relation extraction, or biomedical
literature search. In Chapter 5, we showed that our proposed model can
effectively recognise discontinuous biomedical names without sacrificing the
performance of continuous ones. It worth investing how this improvement can
benefit downstream tasks whose results are directly presented to end users.

## References

  * Agresti (2003) Alan Agresti. 2003.  _Categorical data analysis_.  John Wiley & Sons.
  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1638–1649, Santa Fe, New Mexico.
  * Alex et al. (2007) Beatrice Alex, Barry Haddow, and Claire Grover. 2007.  Recognising nested named entities in biomedical text.  In _Biological, translational, and clinical language processing_ , pages 65–72, Prague, Czech Republic.
  * Alsentzer et al. (2019) Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota.
  * Anderson and Borlak (2011) Nora Anderson and Juergen Borlak. 2011.  Correlation versus causation? pharmacovigilance of the analgesic flupirtine exemplifies the need for refined spontaneous ADR reporting.  _PloS one_ , page e25221.
  * Argamon-Engelson and Dagan (1999) Shlomo Argamon-Engelson and Ido Dagan. 1999.  Committee-based sample selection for probabilistic classifiers.  _Journal of Artificial Intelligence Research_ , pages 335–360.
  * Aronson (2001) Alan R Aronson. 2001.  Effective mapping of biomedical text to the UMLS metathesaurus: the MetaMap program.  In _Proceedings of the AMIA Symposium_ , page 17, Washington, DC.
  * Aronson and Lang (2010) Alan R Aronson and François-Michel Lang. 2010.  An overview of MetaMap: historical perspective and recent advances.  _Journal of the American Medical Informatics Association_ , pages 229–236.
  * Asch and Daelemans (2010) Vincent Van Asch and Walter Daelemans. 2010.  Using domain similarity for performance estimation.  In _Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing_ , pages 31–36, Uppsala, Sweden.
  * Augenstein et al. (2017) Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017.  SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 546–555, Vancouver, Canada.
  * Baevski et al. (2019) Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019\.  Cloze-driven pretraining of self-attention networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5360–5369, Hong Kong, China.
  * Baldwin and Kim (2010) Timothy Baldwin and Su Nam Kim. 2010.  Multiword expressions.  _Handbook of natural language processing_ , pages 267–292.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards nlu: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of Machine Learning Research_ , pages 993–1022.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , pages 135–146.
  * Bonn (1998) Dorothy Bonn. 1998.  Adverse drug reactions remain a major cause of death.  _The Lancet_ , page 1183.
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal.
  * Brownstein et al. (2010) John S Brownstein, Shawn N Murphy, Allison B Goldfine, Richard W Grant, Margarita Sordo, Vivian Gainer, Judith A Colecchi, Anil Dubey, David M Nathan, and John P Glaser. 2010.  Rapid identification of myocardial infarction risk associated with diabetes medications using electronic medical records.  _Diabetes care_ , pages 526–531.
  * Cai et al. (2020) Hengyi Cai, Hongshen Chen, Yonghao Song, Cheng Zhang, Xiaofang Zhao, and Dawei Yin. 2020.  Data manipulation: Towards effective instance learning for neural dialogue generation via learning to augment and reweight.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6334–6343, Online.
  * Campos et al. (2012) David Campos, Sérgio Matos, and José Luís Oliveira. 2012.  _Biomedical named entity recognition: a survey of machine-learning tools_.  IntechOpen.
  * Chelba et al. (2013) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013.  One billion word benchmark for measuring progress in statistical language modeling.  _arXiv preprint arXiv:1312.3005_.
  * Chiu et al. (2016) Billy Chiu, Gamal Crichton, Anna Korhonen, and Sampo Pyysalo. 2016.  How to train good word embeddings for biomedical NLP.  In _Proceedings of the 15th workshop on biomedical natural language processing_ , pages 166–174, Berlin, Germany.
  * Chronopoulou et al. (2019) Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. 2019.  An embarrassingly simple approach for transfer learning from pretrained language models.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2089–2095, Minneapolis, Minnesota.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_ , Online.
  * Coavoux and Cohen (2019) Maximin Coavoux and Shay B Cohen. 2019.  Discontinuous constituency parsing with a stack-free transition system and a dynamic oracle.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 204–217, Minneapolis, Minnesota.
  * Coavoux et al. (2019) Maximin Coavoux, Benoît Crabbé, and Shay B Cohen. 2019.  Unlexicalized transition-based discontinuous constituency parsing.  _Transactions of the Association for Computational Linguistics_ , pages 73–89.
  * Collier et al. (1999) Nigel Collier, Hyun Seok Park, Norihiro Ogata, Yuka Tateisi, Chikashi Nobata, Tomoko Ohta, Tateshi Sekimizu, Hisao Imai, Katsutoshi Ibushi, and Jun’ichi Tsujii. 1999.  The genia project: corpus-based knowledge acquisition and information extraction from genome research papers.  In _Ninth Conference of the European Chapter of the Association for Computational Linguistics_ , pages 271–272, Bergen, Norway.
  * Collins and Singer (1999) Michael Collins and Yoram Singer. 1999.  Unsupervised models for named entity classification.  In _1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora_ , pages 100–110, College Park, MD.
  * Collobert et al. (2011) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011.  Natural language processing (almost) from scratch.  _Journal of machine learning research_ , pages 2493–2537.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online.
  * Conneau et al. (2017) Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. 2017.  Very deep convolutional networks for text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 1107–1116, Valencia, Spain.
  * Culotta and McCallum (2005) Aron Culotta and Andrew McCallum. 2005.  Reducing labeling effort for structured prediction tasks.  In _Proceedings of the Twentieth National Conference on Artificial Intelligence_ , pages 746–751, Pittsburgh, Pennsylvania.
  * Dai and Le (2015) Andrew M Dai and Quoc V Le. 2015.  Semi-supervised sequence learning.  In _Advances in neural information processing systems_ , pages 3079–3087, Montreal, Canada.
  * Dai (2018) Xiang Dai. 2018.  Recognizing complex entity mentions: A review and future directions.  In _Proceedings of ACL 2018, Student Research Workshop_ , pages 37–44, Melbourne, Australia.
  * Dai and Adel (2020) Xiang Dai and Heike Adel. 2020.  An analysis of simple data augmentation for named entity recognition.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3861–3867, Barcelona, Spain (Online).
  * Dai et al. (2019) Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2019.  Using similarity measures to select pretraining data for NER.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1460–1470, Minneapolis, Minnesota.
  * Dai et al. (2020a) Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2020a.  Cost-effective selection of pretraining data: A case study of pretraining BERT on social media.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1675–1681, Online.
  * Dai et al. (2020b) Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2020b.  An effective transition-based model for discontinuous NER.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5860–5870, Online.
  * Dai et al. (2017) Xiang Dai, Sarvnaz Karimi, and Cecile Paris. 2017.  Medication and adverse event extraction from noisy text.  In _Proceedings of the Australasian Language Technology Association Workshop 2017_ , pages 79–87, Brisbane, Australia.
  * Derczynski et al. (2016) Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016.  Broad twitter corpus: A diverse named entity recognition resource.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 1169–1179, Osaka, Japan.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota.
  * Doğan et al. (2014) Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014.  NCBI disease corpus: a resource for disease name recognition and concept normalization.  _Journal of biomedical informatics_ , pages 1–10.
  * Dozat and Manning (2016) Timothy Dozat and Christopher D Manning. 2016.  Deep biaffine attention for neural dependency parsing.  In _International Conference on Learning Representations_ , Toulon, France.
  * Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. 2015\.  Transition-based dependency parsing with stack long short-term memory.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 334–343, Beijing, China.
  * Etzioni et al. (2005) Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S Weld, and Alexander Yates. 2005.  Unsupervised named-entity extraction from the web: An experimental study.  _Artificial intelligence_ , pages 91–134.
  * Fadaee et al. (2017) Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017.  Data augmentation for low-resource neural machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 567–573, Vancouver, Canada.
  * Fang and Cohn (2016) Meng Fang and Trevor Cohn. 2016.  Learning when to trust distant supervision: An application to low-resource pos tagging using cross-lingual projection.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 178–186, Berlin, Germany.
  * Feblowitz et al. (2011) Joshua C Feblowitz, Adam Wright, Hardeep Singh, Lipika Samal, and Dean F Sittig. 2011.  Summarization of clinical information: a conceptual model.  _Journal of biomedical informatics_ , pages 688–699.
  * Ficler and Goldberg (2016) Jessica Ficler and Yoav Goldberg. 2016.  A neural network for coordination boundary prediction.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 23–32, Austin, Texas.
  * Finkel and Manning (2009) Jenny Rose Finkel and Christopher D Manning. 2009.  Nested named entity recognition.  In _Proceedings of the 2009 conference on empirical methods in natural language processing_ , pages 141–150, Singapore.
  * Florian et al. (2004) Radu Florian, Hany Hassan, Abraham Ittycheriah, Hongyan Jing, Nanda Kambhatla, Xiaoqiang Luo, H Nicolov, and Salim Roukos. 2004.  A statistical model for multilingual entity detection and tracking.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 1–8, Boston, Massachusetts.
  * Fuglede and Topsoe (2004) Bent Fuglede and Flemming Topsoe. 2004.  Jensen-shannon divergence and hilbert space embedding.  In _International Symposium onInformation Theory, 2004. ISIT 2004\. Proceedings._ , page 31.
  * Gao et al. (2019) Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019.  Soft contextual data augmentation for neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5539–5544, Florence, Italy.
  * Graves et al. (2013) Alex Graves, Abdel rahman Mohamed, and Geoffrey Hinton. 2013.  Speech recognition with deep recurrent neural networks.  In _2013 IEEE international conference on acoustics, speech and signal processing_ , pages 6645–6649, Vancouver, Canada.
  * Grishman and Sundheim (1996) Ralph Grishman and Beth M Sundheim. 1996.  Message understanding conference-6: A brief history.  In _COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics_.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online.
  * Hadi et al. (2017) Muhammad Abdul Hadi, Chin Fen Neoh, Rosdi M Zin, Mahmoud E Elrggal, and Ejaz Cheema. 2017.  Pharmacovigilance: pharmacists’ perspective on spontaneous adverse drug reaction reporting.  _Integrated pharmacy research and practice_ , page 91.
  * Halliday and Hasan (1989) M. A. K. Halliday and Ruqaiya Hasan. 1989.  _Language, Context and Text: Aspects of Language in a Social-Semiotic Perspective_.  Deakin University Press.
  * Han et al. (2018) Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _Advances in neural information processing systems_ , pages 8527–8537, Montréal, Canada.
  * Han and Eisenstein (2019) Xiaochuang Han and Jacob Eisenstein. 2019.  Unsupervised domain adaptation of contextualized embeddings for sequence labeling.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4238–4248, Hong Kong, China.
  * Hazell and Shakir (2006) Lorna Hazell and Saad AW Shakir. 2006.  Under-reporting of adverse drug reactions.  _Drug safety_ , pages 385–396.
  * Heafield (2011) Kenneth Heafield. 2011.  KenLM: Faster and smaller language model queries.  In _Proceedings of the sixth workshop on statistical machine translation_ , pages 187–197, Edinburgh, Scotland.
  * Hedderich and Klakow (2018) Michael A Hedderich and Dietrich Klakow. 2018.  Training a neural network in a low-resource setting on automatically annotated noisy data.  In _Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP_ , pages 12–18, Melbourne, Australia.
  * Hedderich et al. (2020) Michael A Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2020.  A survey on recent approaches for natural language processing in low-resource scenarios.  _arXiv:2010.12309_.
  * Hirschman et al. (2005) Lynette Hirschman, Alexander Yeh, Christian Blaschke, and Alfonso Valencia. 2005\.  Overview of biocreative: critical assessment of information extraction for biology.  _BMC Bioinformatics_.
  * Hobbs (2002) Jerry R Hobbs. 2002.  Information extraction from biomedical text.  _Journal of biomedical informatics_ , pages 260–264.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , pages 1735–1780.
  * Hoffmann et al. (2011) Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011\.  Knowledge-based weak supervision for information extraction of overlapping relations.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , pages 541–550, Portland, Oregon.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Iyyer et al. (2018) Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018.  Adversarial example generation with syntactically controlled paraphrase networks.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1875–1885, New Orleans, Louisiana.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  MIMIC-III, a freely accessible critical care database.  _Scientific data_ , pages 1–9.
  * Johnson et al. (2018) Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. 2018.  Predicting accuracy on large datasets from smaller pilot data.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 450–455, Melbourne, Australia.
  * Joshi et al. (2019) Aditya Joshi, Sarvnaz Karimi, Ross Sparks, Cécile Paris, and C Raina MacIntyre. 2019.  Survey of text-based epidemic intelligence: A computational linguistics perspective.  _ACM Computing Surveys_ , pages 1–19.
  * Ju et al. (2018) Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018.  A neural layered model for nested named entity recognition.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1446–1459, New Orleans, Louisiana.
  * Karimi et al. (2017) Sarvnaz Karimi, Xiang Dai, Hamed Hassanzadeh, and Anthony Nguyen. 2017.  Automatic diagnosis coding of radiology reports: a comparison of deep learning and conventional classification methods.  In _Proceedings of the 16th BioNLP Workshop_ , pages 328–332, Vancouver, Canada.
  * Karimi et al. (2015a) Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chen Wang. 2015a.  CADEC: A corpus of adverse drug event annotations.  _Journal of biomedical informatics_ , pages 73–81.
  * Karimi et al. (2015b) Sarvnaz Karimi, Chen Wang, Alejandro Metke-Jimenez, Raj Gaire, and Cecile Paris. 2015b.  Text and data mining techniques in adverse drug reaction detection.  _ACM Computing Surveys (CSUR)_ , pages 1–39.
  * Katiyar and Cardie (2018) Arzoo Katiyar and Claire Cardie. 2018.  Nested named entity recognition revisited.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 861–871, New Orleans, Louisiana.
  * Khalil and Huang (2020) H Khalil and C Huang. 2020.  Adverse drug reactions in primary care: a scoping review.  _BMC Health Services Research_ , page 5.
  * Kim et al. (2003) J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003.  Genia corpus—a semantically annotated corpus for bio-textmining.  _Bioinformatics_ , pages i180–i182.
  * Kim et al. (2004) Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. 2004\.  Introduction to the bio-entity recognition task at JNLPBA.  In _Proceedings of the international joint workshop on natural language processing in biomedicine and its applications (NLPBA/BioNLP)_ , pages 70–75, Geneva, Switzerland.