# Should We Rely on Entity Mentions for Relation Extraction?
Debiasing Relation Extraction with Counterfactual Analysis

Yiwei Wang1 Muhao Chen2 Wenxuan Zhou 2 Yujun Cai3 Yuxuan Liang1
Dayiheng Liu4 Baosong Yang4 Juncheng Liu1 Bryan Hooi1
1 National University of Singapore 2 University of Southern California
3 Nanyang Technological University 4Alibaba Group
<EMAIL_ADDRESS>
###### Abstract

Recent literature focuses on utilizing the entity information in the sentence-
level relation extraction (RE), but this risks leaking superficial and
spurious clues of relations. As a result, RE still suffers from unintended
entity bias, i.e., the spurious correlation between entity mentions (names)
and relations. Entity bias can mislead the RE models to extract the relations
that do not exist in the text. To combat this issue, some previous work masks
the entity mentions to prevent the RE models from over-fitting entity
mentions. However, this strategy degrades the RE performance because it loses
the semantic information of entities. In this paper, we propose the CoRE
(Counterfactual Analysis based Relation Extraction) debiasing method that
guides the RE models to focus on the main effects of textual context without
losing the entity information. We first construct a causal graph for RE, which
models the dependencies between variables in RE models. Then, we propose to
conduct counterfactual analysis on our causal graph to distill and mitigate
the entity bias, that captures the causal effects of specific entity mentions
in each instance. Note that our CoRE method is model-agnostic to debias
existing RE systems during inference without changing their training
processes. Extensive experimental results demonstrate that our CoRE yields
significant gains on both effectiveness and generalization for RE. The source
code is provided at: https://github.com/vanoracai/CoRE.

##  1 Introduction

Sentence-level relation extraction (RE) is an important step to obtain a
structural perception of unstructured text Distiawan et al. (2019) by
extracting relations between entity mentions (names) from the textual context.
From human oracle, textual context should be the main source of information
that determines the ground-truth relations between entities. Consider a
sentence “Mary gave birth to Jerry.”111We use underline and wavy line to
denote subject and object respectively by default.. Even if we change the
entity mentions from ‘Jerry’ and ‘Mary’ to other people’s names, the relation
‘parents’ still holds between the subject and object as described by the
textual context “gave birth to”.

Recently, some work aims to utilize entity mentions for RE Yamada et al.
(2020); Zhou and Chen (2021), which, however, leak superficial and spurious
clues about the relations Zhang et al. (2018). In our work, we observe that
entity information can lead to biased relation prediction by misleading RE
models to extract relations that do not exist in the text. Figure 1 visualizes
a relation prediction from a state-of-the-art RE model Alt et al. (2020) (see
more examples in Table 7). Although the context describes no relation between
the highlighted entity pair, the model extracts the relation as
“countries_of_residence”. Such an erroneous result can come from the spurious
correlation between entity mentions and relations, or the entity bias in
short. For example, if the model sees the relation “countries_of_residence”
many more times than other relations when the object entity is Switzerland
during training, the model can associate this relation with Switzerland during
inference even though the relation does not exist in the text.

To combat this issue, some work Zhang et al. (2017, 2018) proposes masking
entities to prevent the RE models from over-fitting entity mentions. On the
other hand, some other work Peng et al. (2020); Zhou and Chen (2021) finds
that this strategy degrades the performance of RE because it loses the
semantic information of entities.

Figure 1:  (left) An example of RE produced by LUKE Yamada et al. (2020). In
the input sentence, the subject is in blue and the object is in yellow. The
ground-truth relation between the subject and object is “no_relation”, since
there is not any relation reflected by the textual context. (right) Our
proposed counterfactual analysis for RE, which compares the original
prediction (upper) with the counterfactual one (lower) to mitigate the entity
bias.

For both machines and humans, RE requires a combined understanding of textual
context and entity mentions Peng et al. (2020). Humans can avoid the entity
bias and make unbiased decisions by correctly referring to the textual context
that describes the relation. The underlying mechanism is causality-based Van
Hoeck et al. (2015): humans identify the relations by pursuing the main causal
effect of the textual context instead of the side-effect of entity mentions.
In contrast, RE models are usually likelihood-based: the prediction is
analogous to looking up the entity mentions and textual context in a huge
likelihood table, interpolated by training Tang et al. (2020). In this paper,
our idea is to teach RE models to distinguish between the effects from the
textual context and entity mentions through counterfactual analysis Pearl
(2018):

Counterfactual analysis: If I had not seen the textual context, would I still
extract the same relation?

The counterfactual analysis essentially gifts humans the hypothesizing
abilities to make decisions collectively based on the textual context and
entity mentions, as well as to introspect whether the decision is deceived
(see Figure 1). Specifically, we are essentially comparing the original
instance with a counterfactual instance, where only the textual context is
wiped out, while keeping the entity mentions untouched. By doing so, we can
focus on the main effects of the textual context without losing the entity
information.

In our work, we propose a novel model-agnostic paradigm for debiasing RE,
namely CoRE (Counterfactual analysis based Relation Extraction), which adopts
the counterfactual analysis to mitigate the spurious influence of the entity
mentions. Specifically, CoRE does not touch the training of RE models, i.e.,
it allows a model to be exposed to biases on the original training set. Then,
we construct a causal graph for RE to analyze the dependencies between
variables in RE models, which acts as a “roadmap” for capturing the causal
effects of textual context and entity mentions. To rectify the test instances
from the potentially biased prediction, in inference, CoRE “imagines” the
counterfactual counterparts on our causal graph to distill the biases. Last
but not least, CoRE performs a bias mitigation operation with adaptive weights
to produce a debiased decision for RE.

We highlight that CoRE is a flexible debiasing method that is applicable to
popular RE models without changing their training processes. To evaluate the
effectiveness of CoRE, we perform extensive experiments on public benchmark
datasets. The results demonstrate that our proposed method can significantly
improve the effectiveness and generalization of the popular RE models by
mitigating the biases in an entity-aware manner.

##  2 Related Work

Sentence-level relation extraction. Early research efforts Nguyen and Grishman
(2015); Wang et al. (2016); Zhang et al. (2017) train RE models from scratch
based on lexicon-level features. The recent RE work fine-tunes pretrained
language models (PLMs; Devlin et al. 2019; Liu et al. 2019). For example,
K-Adapter Wang et al. (2020) fixes the parameters of the PLM and uses feature
adapters to infuse factual and linguistic knowledge. Recent work focuses on
utilizing the entity information for RE Zhou and Chen (2021); Yamada et al.
(2020), but this leaks superficial and spurious clues about the relations
Zhang et al. (2018). Despite the biases in existing RE models, scarce work has
discussed the spurious correlation between entity mentions and relations that
causes such biases. Our work investigates this issue and proposes CoRE to
debias RE models for higher effectiveness.

Debiasing for Natural Language Processing. Debiasing is a fundamental problem
in machine learning Torralba and Efros (2011). For natural language processing
(NLP), some work performs data re-sampling to prevent models from capturing
the unintended bias in training Dixon et al. (2018); Geng et al. (2007); Kang
et al. (2016); Rayhan et al. (2017); Nguyen et al. (2011). Alternatively, Wei
and Zou (2019) and Qian et al. (2020) develop data augmentation for debiasing.
Some recent work debiases the NLP models based on causal inference Qian et al.
(2021); Nan et al. (2021). In RE, how to deal with the entity bias is also an
important problem. For example, PA-LSTM Zhang et al. (2017) masks the entity
mentions with special tokens to prevent RE models from over-fitting entity
names, which was also adopted by C-GCN Zhang et al. (2018) and SpanBERT Joshi
et al. (2020). However, masking entities loses the semantic information of
entities and leads to performance degradation. Different from it, our CoRE
model tackles entity biases based on structured causal models. In this way, we
debias the RE models to focus on the textual context without losing the entity
information.

Figure 2:  The causual graph of RE models.

##  3 Methodology

Sentence-level relation extraction (RE) aims to extract the relation between a
pair of entities mentioned from a sentence. We propose CoRE (counterfactual
analysis based Relation Extraction) as a model-agnostic technique to endow
existing RE models with unbiased decisions during inference. CoRE follows the
regular training process of existing work regardless of the bias from the
entity mentions. During inference, CoRE post-adjusts the biased prediction
according to the effects of the bias. CoRE can be flexibly incorporated into
popular RE models to improve their effectiveness and generalization based on
the counterfactual analysis without re-training the model.

In this section, we first formulate the existing RE models in the form of a
causal graph. Then, we introduce our proposed bias distillation method to
distill the entity bias with our designed counterfactual analysis. We conduct
an empirical analysis to analyze how heavily the existing RE models rely on
the entity mentions to make decisions. Finally, we mitigate the distilled bias
from the predictions of RE models to improve their effectiveness.

###  3.1 Causality of Relation Extraction

In order to perform causal intervention, we first formulate the causal graph
Pearl et al. (2016); Pearl and Mackenzie (2018), a.k.a., structural causal
model, for the RE models as Figure 2, which sheds light on how the textual
context and entity mentions affect the RE predictions. The causal graph is a
directed acyclic graph $\mathcal{G}=\\{\mathcal{V},\mathcal{E}\\}$, indicating
how a set of variables $\mathcal{V}$ interact with each other through the
causal relations behind the data and how variables obtain their values, e.g.,
$(E,X)\rightarrow Y$ in Figure 2. Before we conduct counterfactual analysis
that deliberately manipulates the values of nodes and prunes the causal graph,
we first revisit the conventional RE systems in the graphical view.

The causal graph in Figure 2 is applicable to a variety of RE models and
imposes no constraints on the detailed implementations. Node $X$ is the input
text. On the edge $X\rightarrow E$, we obtain the spans of subject and object
entities as node $E$ through NER or human annotations Zhang et al. (2017). For
example, in the aforementioned sentence $X=$ “Mary gave birth to Jerry.”, the
entities are $E=$ [’Mary’, ’Jerry’].

On the edges $(X,E)\rightarrow Y$, existing RE models take different designs.
For example, C-GCN Zhang et al. (2018) obtains the relation prediction $Y$ by
encoding entity mentions $E$ on the pruned dependency tree of $X$ using a
graph convolutional network. IRE Zhou and Chen (2021) uses PLMs as the encoder
for $X$, and marks the entity information of $E$ with special tokens to
utilize the entity information.

###  3.2 Bias Distillation

Based on our causal graph in Figure 2, we diagnose how the entity bias affects
inference. After training, the causal dependencies among the variables are
learned in terms of the model parameters. The entity bias can mislead the
models to make wrong predictions while ignoring the actual relation-describing
textual context in $X$, i.e., biased towards the causal dependency:
$E\rightarrow Y$.

Figure 3:  The original causal graph of RE models (left) together with its two
counterfactual alternates for the entity bias (middle) and label bias (right).
The shading indicates the mask of corresponding variables.

The conventional biased prediction can only see the output $Y$ of the entire
graph given a sentence $X$, ignoring how specific entity mentions affect the
relation prediction. However, causal inference encourages us to think out of
the black box. From the graphical point of view, we are no longer required to
execute the entire causal graph as a whole. In contrast, we can directly
manipulate the nodes and observe the output. The above operation is termed
intervention in causal inference, which we denote as $do(\cdot)$. It wipes out
all the incoming links of a node and demands it to take a certain value.

We distill the entity bias by intervention and its induced counterfactual. The
counterfactual means “counter to the facts”, and takes one step that further
assigns the hypothetical combination of values to variables. For example, we
can remove the input textual context by masking $X$, but maintain $E$ as the
original entity mentions, as if $X$ still exists.

We will use the input text $X$ as our control variable where the intervention
is conducted, aiming to assess its effects, due to the fact that there would
not be any valid relation between entities in $E$ if the input text $X$ is
empty. We denote the output logits $Y$ after the intervention $X=\bar{x}$ as
follows:

$Y_{\bar{x}}=Y(do(X=\bar{x})).$ (1)

Following the above notation, the original prediction $Y$, i.e., can be re-
written as $Y_{x}$.

To distill the entity bias, we conduct the intervention $do(X=\bar{x})$ on
$X$, while keeping the variable $E$ as the original $e$, as if the original
input text $x$ had existed. Specifically, we mask the tokens in $x$ to produce
$\bar{x}$ but keep the entity mentions $e$ as original, so that the textual
context is removed and the entity information is maintained. Accordingly, the
counterfactual prediction is denoted as $Y_{\bar{x},e}$ (see Figure 3). In
this case, since the model cannot see any textual context in the factual input
$x$ after the intervention $\bar{x}$, but still has access to the original
entity mentions $e$ as the inputs, the prediction $Y_{\bar{x},e}$ purely
reflects the influence from $e$. In other words, $Y_{\bar{x},e}$ refers to the
output, i.e., a probability distribution or a logit vector, where only the
entity mentions are given as the input without textual context.

Figure 4:  Hit$@k$ (y-axis) is the fraction of the test instances, that have
the original relation prediction $\arg\max_{c}Y_{x}[c]$ ranked in the top $k$
most confident relations of the counterfactual prediction $Y_{\bar{x},e}$. We
report Hit$@k$ of the model IRERoBERTa on the test instances when the original
relation prediction is title, employee_of, or origin.

To investigate how heavily the state-of-the-art models rely on the entity
mentions for RE, we conduct an empirical study to compare the original
prediction $Y_{x}$ and the counterfactual one $Y_{\bar{x},e}$. Specifically,
we calculate the fraction of the test instances (y-axis) that have the
original relation prediction $\arg\max_{c}Y_{x}[c]$ ranked in the top $k$ most
confident relations of the counterfactual prediction $Y_{\bar{x},e}$. This
fraction is termed as Hit$@k$.

We present Hit$@k$ for IRERoBERTa Zhou and Chen (2021), a state-of-the-art RE
model, in Figure 4 on the test instances when the original relation prediction
is title, employee_of, or origin. Higher Hit$@1$ means that for more
instances, the model infers the same relation given only the entity mentions
no matter whether the textual context is given, which imply stronger causal
effects from the entity mentions $Y_{\bar{x},e}$, i.e., the models rely more
heavily on the entity mentions for RE.

We observe that when $k=1$, the Hit$@1$ is more than 50%, which implies that
the model typically extracts the same relations even without textual context
on more than a half of the instances. For a larger $k$, the Hit$@k$ increases
significantly and reaches more than 80% for $k\geq 2$. These observations
imply a promising but embarrassing result: the state-of-the-art model relies
on the entity bias for RE on many instances. The entity bias reflected by
$Y_{\bar{x},e}$ can lead to the wrong extraction if the relation implied by
the entity mentions does not exist in the input text. This poses a challenge
to the generalization of RE models, as validated by our experimental results
(Section 4.3).

In addition to $Y_{\bar{x},e}$ that reflects the causal effects of entity
mentions, there is another kind of bias not conditioned on the entity mentions
$e$, but reflecting the general bias in the whole dataset, which is
$Y_{\bar{x}}$. $Y_{\bar{x}}$ corresponds to the counterfactual inputs where
both textual context and entity mentions are removed. In this case, since the
model cannot access any information from the input after this removal,
$Y_{\bar{x}}$ naturally reflects the label bias that exists in the model from
the biased training. The causal graphic views of the original prediction
$Y_{x}$, the counterfactual $Y_{\bar{x},e}$ for the entity bias, and
$Y_{\bar{x}}$ for the label bias are visualized in Figure 3.

###  3.3 Bias Mitigation

As we have discussed in Section 1, instead of the static likelihood that tends
to be biased, the unbiased relation prediction lies in the difference between
the observed outcome $Y_{x}$ and its counterfactual predictions
$Y_{\bar{x},e},Y_{\bar{x}}$. The latter two are the biases that we want to
mitigate from the relation prediction.

Intuitively, the unbiased prediction that we seek is the linguistic stimuli
from blank to the observed textual context with specific relation
descriptions, but not merely from the entity bias. The context-specific clues
of the relations are key to the informative unbiased predictions, because even
if the overall prediction is biased towards the relation “schools_attended”
due to the object entity like “Duke University”, the textual context “work at”
indicates the relation as “employee_of” rather than “schools_attended”.

Our final goal is to use the direct effect of the textual context from $X$ to
$Y$ for debiased prediction, mitigating (denoted as \\) the label bias and the
entity bias from the prediction: $Y_{x}$ \$Y_{\bar{x},e}$ \$Y_{\bar{x}}$, so
as to block the spread of the biases from training to inference. The debiased
prediction via bias mitigation can be formulated via the conceptually simple
but empirically effective element-wise subtraction operation:

$Y_{\mathrm{final}}=Y_{x}-\lambda_{1}Y_{\bar{x},e}-\lambda_{2}Y_{\bar{x}},$
(2)

where $\lambda_{1}$ and $\lambda_{2}$ are two independent hyper-parameters
balancing the terms for mitigating entity and label biases respectively. Note
that the bias mitigation in Equation 2 for the entity and label biases
correspond to Total Direct Effect (TDE) and Total Effect (TE) in causal
inference Tang et al. (2020); VanderWeele (2015); Pearl (2009) respectively.
We adaptively set the values of $\lambda_{1}$ and $\lambda_{2}$ for different
datasets based on the grid beam search Hokamp and Liu (2017) in a scoped two
dimensional space:

$\lambda_{1}^{\star},\lambda_{2}^{\star}=\arg\max_{\lambda_{1},\lambda_{2}}\psi(\lambda_{1},\lambda_{2})\
\ \lambda_{1},\lambda_{2}\in[a,b],$ (3)

where $\psi$ is a metric function (e.g., F1 scores) for evaluation, $a,b$ are
the boundaries of the search range. We search the values of
$\lambda_{1},\lambda_{2}$ once on the validation set, and use the fixed values
for inference on all testing instances. Since the entity types can restrict
the candidate relations Lyu and Chen (2021), we use the entity type
information, if available, to restrict the candidate relations for inference,
which strengthens the effects of entity types for relation extraction.

Overall, the proposed CoRE replaces the conventional one-time prediction with
$Y_{\mathrm{final}}$ to produce the debiased relation predictions, which
essentially “thinks” twice: one for the original observation $Y_{x}$, the
other for hypothesized $Y_{\bar{x}},Y_{\bar{x},e}$.

Dataset $\\#$Train $\\#$Dev $\\#$Test $\\#$Classes TACRED 68,124 22,631 15509
42 SemEval 6,507 1,493 2,717 19 Re-TACRED 58,465 19,584 13418 40 TACRED-
Revisit 68,124 22,631 15509 42

Table 1: Statistics of datasets.

##  4 Experiments

In this section, we evaluate the performance of our CoRE methods when applied
to RE models. We compare our methods against a variety of strong baselines on
the task of sentence-level RE. Our experimental settings closely follow those
of the previous work Zhang et al. (2017); Zhou and Chen (2021); Nan et al.
(2021) to ensure a fair comparison.

###  4.1 Experimental Settings

Datasets. We use four widely-used RE benchmarks: TACRED Zhang et al. (2017),
SemEval Hendrickx et al. (2019), TACRED-Revisit Alt et al. (2020), and Re-
TACRED Stoica et al. (2021) for evaluation. TACRED contains over 106k mention
pairs drawn from the yearly TAC KBP challenge. Alt et al. (2020) relabeled the
development and test sets of TACRED. Re-TACRED is a further relabeled version
of TACRED after refining its label definitions. The statistics of these
datasets are shown in Table 1.

Method | TACRED | TACRED-Revisit | Re-TACRED | SemEval
---|---|---|---|---
C-SGC Wu et al. (2019) | 52.1 | 62.8 | 69.8 | 71.3
SpanBERT Joshi et al. (2020) | 55.7 | 65.1 | 74.1 | 74.9
CP Peng et al. (2020) | 56.8 | 67.1 | 78.1 | 79.6
RECENT Lyu and Chen (2021) | 63.3 | 70.5 | 81.1 | 74.6
KnowPrompt Chen et al. (2021) | 57.6 | 68.7 | 79.0 | 81.8
IREBERT Zhou and Chen (2021) | 59.2 | 68.4 | 78.6 | 79.1
LUKE Yamada et al. (2020) | 58.8 | 67.5 | 80.2 | 82.1
LUKE + Resample Burnaev et al. (2015) | 59.3 | 68.2 | 80.5 | 82.5
LUKE + Focal Lin et al. (2017) | 59.1 | 67.7 | 80.3 | 82.4
LUKE + CFIE Nan et al. (2021) | 59.8 | 68.0 | 80.4 | 82.2
LUKE + Entity Mask Zhang et al. (2017) | 57.9 | 67.0 | 79.5 | 82.0
LUKE + CoRE | 61.7 | 70.2 | 81.6 | 83.6
IRERoBERTa Zhou and Chen (2021) | 63.1 | 70.6 | 81.5 | 81.4
IRERoBERTa \+ Resample Burnaev et al. (2015) | 63.3 | 71.0 | 81.9 | 81.6
IRERoBERTa \+ Focal Lin et al. (2017) | 62.9 | 70.7 | 81.2 | 81.1
IRERoBERTa \+ CFIE Nan et al. (2021) | 63.3 | 70.9 | 81.6 | 81.7
IRERoBERTa \+ Entity Mask Zhang et al. (2017) | 61.4 | 69.3 | 79.6 | 81.2
IRERoBERTa \+ CoRE | 64.4 | 71.8 | 82.8 | 82.3
Table 2: F1-macro scores (%) of RE on the test sets of TACRED, TACRED-Revisit,
Re-TACRED, and SemEval. The best results in each column are highlighted in
bold font.

We use the widely-used F1-macro score as the main evaluation metric Nan et al.
(2021), which is the balanced harmonic mean of precision and recall, as well
as F1-micro for a more comprehensive evaluation. F1-macro is more suitable
than F1-micro to reflect the extent of biases, especially for the highly-
skewed cases, since F1-macro is evenly influenced by the performance in each
category, i.e. category-sensitive, but F1-micro simply gives equal weights to
all instances Kim et al. (2019).

Method | TACRED | TACRED-Revisit | Re-TACRED | SemEval
---|---|---|---|---
LUKE Yamada et al. (2020) | 72.7 | 80.6 | 90.3 | 87.8
LUKE + Resample Burnaev et al. (2015) | 73.1 | 80.9 | 90.5 | 87.9
LUKE + Focal Lin et al. (2017) | 72.9 | 80.7 | 90.4 | 87.6
LUKE + CFIE Nan et al. (2021) | 73.3 | 80.8 | 90.5 | 88.0
LUKE + Entity Mask Zhang et al. (2017) | 72.3 | 80.4 | 90.1 | 87.5
LUKE + CoRE | 74.6 | 81.4 | 90.9 | 88.7
Table 3: F1-micro scores (%) of RE on the test sets of TACRED, TACRED-Revisit,
Re-TACRED, and SemEval. The best results in each column are highlighted in
bold font.

Method TACRED Re-TACRED LUKE Yamada et al. (2020) 51.9 65.3 w/ Resample
Burnaev et al. (2015) 53.2 66.7 w/ Focal Lin et al. (2017) 52.4 65.9 w/ CFIE
Nan et al. (2021) 52.1 65.6 w/ Entity Mask Zhang et al. (2017) 54.5 67.1 w/
CoRE (ours) 69.3 83.1 IRERoBERTa Zhou and Chen (2021) 56.4 68.1 w/ Resample
Burnaev et al. (2015) 58.1 70.3 w/ Focal Lin et al. (2017) 56.8 68.7 w/ CFIE
Nan et al. (2021) 57.1 68.4 w/ Entity Mask Zhang et al. (2017) 57.3 68.9 w/
CoRE (ours) 73.6 85.4

Table 4: F1-macro scores (%) of RE on the challenging test sets of TACRED and Re-TACRED, in which the relations implied by the entity mentions do not exist in the textual context. ‘w’ denotes ‘with’. The best results in each column are highlighted in bold font. Method | TACRED | TACRED-Revisit | Re-TACRED | SemEval
---|---|---|---|---
IRERoBERTa Zhou and Chen (2021) | 61.2 | 59.3 | 57.5 | 54.1
IRERoBERTa \+ Resample Burnaev et al. (2015) | 60.5 | 58.4 | 56.8 | 53.5
IRERoBERTa \+ Focal Lin et al. (2017) | 60.9 | 58.9 | 57.1 | 53.7
IRERoBERTa \+ CFIE Nan et al. (2021) | 60.1 | 57.8 | 56.2 | 52.9
IRERoBERTa \+ Entity Mask Zhang et al. (2017) | 61.5 | 60.1 | 57.3 | 54.2
IRERoBERTa \+ CoRE | 57.3 | 55.6 | 54.3 | 50.8
Table 5: Experimental results (unfairness; %) of Relation Extraction on the
test sets of TACRED, TACRED-Revisit, Re-TACRED, and SemEval (lower is better).
The best results in each column are highlighted in bold font.

Compared methods. We take the following RE models into comparison. (1) C-SGC
Wu et al. (2019) simplifies GCN, and combines it with LSTM, leading to
improved performance over each method alone. (2) SpanBERT Joshi et al. (2020)
extends BERT by introducing a new pretraining objective of continuous span
prediction. (3) CP Peng et al. (2020) is an entity-masked contrastive pre-
training framework for RE. (4) RECENT Lyu and Chen (2021) restricts the
candidate relations based on the entity types. (5) KnowPrompt Chen et al.
(2021) is Knowledge-aware Prompt-tuning approach. (6) LUKE Yamada et al.
(2020) pretrains the language model on both large text corpora and knowledge
graphs and further proposes an entity-aware self-attention mechanism. (7) IRE
Zhou and Chen (2021) proposes an improved entity representation technique in
the data preprocessing.

Among the above RE models, we apply our CoRE on LUKE and IRE. To demonstrate
the effectiveness of debiased inference, we also compare with the following
debiasing techniques that are applied to the same two RE models. (1) Focal Lin
et al. (2017) adaptively reweights the losses of different instances so as to
focus on the hard ones. (2) Resample Burnaev et al. (2015) up-samples rare
categories by the inversed sample fraction during training. (3) Entity Mask
Zhang et al. (2017): masks the entity mentions with special tokens to reduce
the over-fitting on entities. (4) CFIE Nan et al. (2021) is also a causal
inference method. In contrast to our method, CFIE strengthens the causal
effects of entities by masking entity-centric information in the
counterfactual predictions.

Model configuration. For the hyper-parameters of the considered baseline
methods, e.g., the batch size, the number of hidden units, the optimizer, and
the learning rate, we set them as suggested by their authors. For the hyper-
parameters of our CoRE method, we set the search range of the hypermeters in
Equation 3 as $[-2,2]$ and the search step $0.1$. For all experiments, we
report the median $F1$ scores of five runs of training using different random
seeds.

###  4.2 Overall Performance

We implement our CoRE with LUKE and IRERoBERTa. Table 3 reports the RE results
on the TACRED, TACRED-Revisit, Re-TACRED, and SemEval datasets. Our CoRE
method improves the F1-macro scores of LUKE by 4.9% on TACRED, 4.0% on TACRED-
Revisit, 1.7% on Re-TACRED, and 1.7 on SemEval, and improves IRERoBERTa by
1.2% on TACRED, 1.4% on TACRED-Revisit, 0.9% on Re-TACRED, and 1.8% on
SemEval. As a result, our CoRE achieves substantial improvements for LUKE and
IRERoBERTa, and enables them to outperform the baseline methods. Additionally,
we report the experimental results in terms of F1-micro scores in Table 3,
showing the improvement from CoRE on LUKE by 2.6% on TACRED, 1.0% on TACRED-
Revisit, 0.7% on Re-TACRED, and 1.0% on SemEval. Overall, our CoRE method
improves the effectiveness of RE significantly in terms of both F1-macro and
F1-micro scores. The above experimental results validate the effectiveness and
generalization of our proposed method.

Among the baseline debiasing methods, Resample, Focal, CFIE cannot distill the
entity bias in an entity-aware manner like ours. Entity Mask leads to the loss
of information, while our CoRE enables RE models to focus on the main effects
of textual context without losing the entity information. The superiority of
CoRE highlights the importance of the causal inference based entity bias
analysis for debiasing RE, which compares traditional likelihood-based
predictions and hypothesized counterfactual ones to produce debiased
predictions. Besides, the proposed CoRE works in inference and thus can be
employed on the previous already-trained models. In this way, CoRE serves as a
model-agnostic approach to enhance RE models without changing their training
process.

LUKE + CoRE 61.7 $\Delta$ IRE + CoRE 64.4 $\Delta$ w/o CoRE 58.8
2.9$\downarrow$ w/o CoRE 63.1 1.3$\downarrow$ w/o EBM 59.5 2.2$\downarrow$ w/o
EBM 63.4 1.0$\downarrow$ w/o LBM 60.8 0.9$\downarrow$ w/o LBM 63.9
0.5$\downarrow$ w/o BSH 60.1 1.6 $\downarrow$ w/o BSH 63.8 0.6$\downarrow$

Table 6: Ablation study based on the TACRED dataset. The analyzed model
components include entity bias mitigation operation (EBM), the label bias
mitigation operation (LBM) and the beam search for hyper-parameters (BSH).
‘w/o’ denotes ‘without’. $\downarrow$ denotes performance drop in terms of
F1-macro scores.

###  4.3 Analysis on Entity Bias

Some work argues that RE models may rely on the entity mentions to make
relation predictions instead of the textual context Zhang et al. (2018); Joshi
et al. (2020). The empirical results in Figure 3 validates this argument.
Regardless of whether the textual context exists or not, the baseline RE model
makes the same predictions given only entity mentions on many instances. The
entity bias can mislead the RE models to make wrong predictions when the
relation implied by the entity mentions does not exist in the textual context.

Input sentence Original Debiased Counterfactual More than 1,100 miles (1,770
kilometers) away, Alan Gross passes his days in a Cuban military hospital,
watching baseball on a small television or jamming with his jailers on a
stringed instrument they gave him. origin ✗ countries_of_residence ✓ origin He
said that according to his investigation, Bibi drew the ire of fellow
farmhands after a dispute in June 2009, when they refused to drink water she
collected and she refused their demands that she convert to Islam. religion ✗
no_relation ✓ religion ShopperTrak also estimates foot traffic in the U.S. was
11.2 percent below what it would have been Sunday if the blizzard had not
occurred and 13.9 percent below what it could have been Monday.
country_of_headquarters ✗ no_relation ✓ country_of_headquarters

Table 7: A case study for IRERoBERTa and our CoRE on the relation extraction
dataset TACRED. Underlines and wavy lines highlight the subject and object
entities respectively. We report the original prediction, the corresponding
counterfactual prediction and the debiased prediction.

To evaluate whether RE models can generalize well to particularly challenging
instances where relations implied by the entity mentions do not exist in the
textual context, we propose a filtered evaluation setting, where we keep the
test instances having the entity bias different from their ground-truth
relations. In this setting, RE models cannot overly rely on the entity
mentions for RE, since the entity mentions no longer provide the superficial
and spurious clues for the ground-truth relations.

We present the evaluation results on the filtered test set in Table 4. Our
CoRE method consistently and substantially improves the effectiveness of LUKE
and IRE on the filtered test set and outperforms the baseline methods by a
significant margin, which validates the effectiveness and generalization of
our method to mitigate the entity bias in the challenging cases.

###  4.4 Evaluation on Fairness

According to Sweeney and Najafian (2019), the more imbalanced/skewed a
prediction produced by a trained model is, the more unfair opportunities it
gives over predefined categories, and the more unfairly discriminative the
trained model is. We thus follow previous work Xiang et al. (2020); Sweeney
and Najafian (2019); Qian et al. (2021) to use the metric – imbalance
divergence – to evaluate how imbalanced/skewed/unfair a prediction $P$ is :

$D(P,U)=JS(P\|U),$ (4)

where $D(\cdot)$ is defined as the distance between $P$ and the uniform
distribution $U$. Specifically, we use the JS divergence as the distance
metric since it is symmetric (i.e., $JS(P\|U)=JS(U\|P)$) and strictly scoped
Fuglede and Topsoe (2004). Based on this, to evaluate the entity bias of a
trained RE model, we average the following relative entity mention imbalance
(REI) measure over all the testing instances containing whichever entity
mentions:

$\mathrm{REI}=\frac{1}{\mathcal{E}}\sum_{e\in\mathcal{E}}D(P(\\{x|e\in x\land
x\in\mathcal{D}\\}),U),$ (5)

where $x$ is an input instance, $\mathcal{D}$ is the testing set, $P(x)$ is
the prediction output, $e$ is an entity mention, and $\mathcal{E}$ is the
corpus of entity mentions. This metric captures the distance between all
predictions and the fair uniform distribution $U$.

We follow the experimental settings in Section 4.2 and report the fairness
test in Table 5. The results show that our CoRE method reduces the imbalance
metrics (lower is better) when employed on IRERoBERTa significantly and
consistently, indicating that it is helpful to mitigate the entity bias.

###  4.5 Ablation and Case Study

We conduct ablation studies on CoRE to empirically examine the contribution of
its main technical components. including the entity bias mitigation operation
(EBM), the label bias mitigation operation (LBM) and the beam search for
hyper-parameters (BSH).

We report the experimental results of the ablation study in Table 6. We
observe that removing our CoRE causes serious performance degradation. This
provides evidence that using our counterfactual framework for RE can
explicitly mitigate biases to generalize better on unseen examples. Moreover,
we observe that mitigating the two types of biases is consistently helpful for
RE. The key reason is that the distilled label bias provides an instance-
agnostic offset and the distilled entity bias provides an entity-aware one in
the prediction space, which makes the RE models focus on extracting relations
on the textual context without losing the entity information. Meanwhile, the
beam search for hyper-parameters effectively finds two dynamic scaling factors
to amplify or shrink two biases, making the biases be mitigated properly and
adaptively.

Table 7 gives a qualitative comparison example between CoRE and IRERoBERTa on
TACRED. The results show that the state-of-the-art RE model IRERoBERTa returns
the relations that do not exist in the textual context between the considered
entities. For example, given “Bibi drew the ire of fellow farmhands after a
dispute in June 2009, when they refused to drink water she collected and she
refused their demands that she convert to Islam.”, there is no relation
between Bibi and Islam exists in the text but the baseline model believes that
the relation between them is “religion”. The counterfactual prediction can
account for this disappointing result, where given only the entity mentions
Bibi and Islam, the RE model returns the relation “religion” without any
textual context. This implies that the model makes the prediction for the
original input relying on the entity mentions, which leads to the wrong RE
prediction. Our CoRE method distills the biases through counterfactual
predictions and mitigates the biases to distinguish the main effects from the
textual context, which leads to the correct predictions as shown in Table 7.

Last but not least, we conduct experiments on the fairness of different
models, and present respective results in the appendix.

##  5 Conclusion

We have designed a counterfactual analysis based method named CoRE to debias
RE. We distill the entity bias and mitigate the distilled biases with the help
of our causal graph for RE, which is a road map for analyzing the RE models.
Based on the counterfactual analysis, we can analyze the side-effects of
entity mentions in the RE and debias the models in an entity-aware manner.
Extensive experiments demonstrate that our methods can improve the
effectiveness and generalization of RE. Future work includes analyzing the
effects of other factors that can cause bias in natural language processing.

## Acknowledgement

The authors would like to thank the anonymous reviewers for their discussion
and feedback.

Muhao Chen and Wenxuan Zhou are supported by the National Science Foundation
of United States Grant IIS 2105329, and by the DARPA MCS program under
Contract No. N660011924033 with the United States Office Of Naval Research.
Except for Muhao Chen and Wenxuan Zhou, this paper is supported by NUS ODPRT
Grant R252-000-A81-133 and Singapore Ministry of Education Academic Research
Fund Tier 3 under MOEs official grant number MOE2017-T3-1-007.

## References

  * Alt et al. (2020) Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020.  TACRED revisited: A thorough evaluation of the TACRED relation extraction task.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1558–1569, Online. Association for Computational Linguistics.
  * Burnaev et al. (2015) Evgeny Burnaev, Pavel Erofeev, and Artem Papanov. 2015.  Influence of resampling on accuracy of imbalanced classification.  In _Eighth international conference on machine vision (ICMV 2015)_ , volume 9875, page 987521. International Society for Optics and Photonics.
  * Chen et al. (2021) Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2021.  Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Distiawan et al. (2019) Bayu Distiawan, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019.  Neural relation extraction for knowledge base enrichment.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 229–240.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and mitigating unintended bias in text classification.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 67–73.
  * Fuglede and Topsoe (2004) Bent Fuglede and Flemming Topsoe. 2004.  Jensen-shannon divergence and hilbert space embedding.  In _International Symposium onInformation Theory, 2004. ISIT 2004\. Proceedings._ , page 31. IEEE.
  * Geng et al. (2007) Guang-Gang Geng, Chun-Heng Wang, Qiu-Dan Li, Lei Xu, and Xiao-Bo Jin. 2007.  Boosting the performance of web spam detection with ensemble under-sampling classification.  In _Fourth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD 2007)_ , volume 4, pages 583–587. IEEE.
  * Hendrickx et al. (2019) Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2019.  Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.  _arXiv preprint arXiv:1911.10422_.
  * Hokamp and Liu (2017) Chris Hokamp and Qun Liu. 2017.  Lexically constrained decoding for sequence generation using grid beam search.  _arXiv preprint arXiv:1704.07138_.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Kang et al. (2016) Qi Kang, XiaoShuang Chen, SiSi Li, and MengChu Zhou. 2016.  A noise-filtered under-sampling scheme for imbalanced classification.  _IEEE transactions on cybernetics_ , 47(12):4263–4274.
  * Kim et al. (2019) Kang-Min Kim, Yeachan Kim, Jungho Lee, Ji-Min Lee, and SangKeun Lee. 2019.  From small-scale to large-scale text classification.  In _The World Wide Web Conference_ , pages 853–862.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017\.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lyu and Chen (2021) Shengfei Lyu and Huanhuan Chen. 2021.  Relation classification with entity type restriction.  _arXiv preprint arXiv:2105.08393_.
  * Nan et al. (2021) Guoshun Nan, Jiaqi Zeng, Rui Qiao, Zhijiang Guo, and Wei Lu. 2021.  Uncovering main causalities for long-tailed information extraction.  _arXiv preprint arXiv:2109.05213_.
  * Nguyen et al. (2011) Hien M Nguyen, Eric W Cooper, and Katsuari Kamei. 2011.  Borderline over-sampling for imbalanced data classification.  _International Journal of Knowledge Engineering and Soft Data Paradigms_ , 3(1):4–21.
  * Nguyen and Grishman (2015) Thien Huu Nguyen and Ralph Grishman. 2015.  Relation extraction: Perspective from convolutional neural networks.  In _Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing_ , pages 39–48, Denver, Colorado. Association for Computational Linguistics.
  * Pearl (2009) Judea Pearl. 2009.  _Causality_.  Cambridge university press.
  * Pearl (2018) Judea Pearl. 2018.  Causal and counterfactual inference.  _The Handbook of Rationality_ , pages 1–41.
  * Pearl et al. (2016) Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. 2016.  _Causal inference in statistics: A primer_.  John Wiley & Sons.
  * Pearl and Mackenzie (2018) Judea Pearl and Dana Mackenzie. 2018.  _The book of why: the new science of cause and effect_.  Basic books.
  * Peng et al. (2020) Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020.  Learning from context or names? an empirical study on neural relation extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3661–3672.
  * Qian et al. (2020) Chen Qian, Fuli Feng, Lijie Wen, Li Lin, and Tat-Seng Chua. 2020.  Enhancing text classification via discovering additional semantic clues from logograms.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1201–1210.
  * Qian et al. (2021) Chen Qian, Fuli Feng, Lijie Wen, Chunping Ma, and Pengjun Xie. 2021.  Counterfactual inference for text classification debiasing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5434–5445.
  * Rayhan et al. (2017) Farshid Rayhan, Sajid Ahmed, Asif Mahbub, Rafsan Jani, Swakkhar Shatabda, and Dewan Md Farid. 2017.  Cusboost: Cluster-based under-sampling with boosting for imbalanced classification.  In _2017 2nd International Conference on Computational Systems and Information Technology for Sustainable Solution (CSITSS)_ , pages 1–5. IEEE.
  * Stoica et al. (2021) George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021.  Re-tacred: Addressing shortcomings of the tacred dataset.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 13843–13850.
  * Sweeney and Najafian (2019) Chris Sweeney and Maryam Najafian. 2019.  A transparent framework for evaluating unintended demographic bias in word embeddings.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1662–1667.
  * Tang et al. (2020) Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020.  Unbiased scene graph generation from biased training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3716–3725.
  * Torralba and Efros (2011) Antonio Torralba and Alexei A Efros. 2011.  Unbiased look at dataset bias.  In _CVPR 2011_ , pages 1521–1528. IEEE.
  * Van Hoeck et al. (2015) Nicole Van Hoeck, Patrick D Watson, and Aron K Barbey. 2015.  Cognitive neuroscience of human counterfactual reasoning.  _Frontiers in human neuroscience_ , 9:420.
  * VanderWeele (2015) Tyler VanderWeele. 2015.  _Explanation in causal inference: methods for mediation and interaction_.  Oxford University Press.
  * Wang et al. (2016) Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016.  Relation classification via multi-level attention CNNs.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1298–1307, Berlin, Germany. Association for Computational Linguistics.
  * Wang et al. (2020) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. 2020.  K-adapter: Infusing knowledge into pre-trained models with adapters.  _arXiv preprint arXiv:2002.01808_.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  _arXiv preprint arXiv:1901.11196_.
  * Wu et al. (2019) Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019.  Simplifying graph convolutional networks.  In _International conference on machine learning_ , pages 6861–6871. PMLR.
  * Xiang et al. (2020) Liuyu Xiang, Guiguang Ding, and Jungong Han. 2020.  Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification.  In _European Conference on Computer Vision_ , pages 247–263. Springer.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020\.  LUKE: Deep contextualized entity representations with entity-aware self-attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6442–6454, Online. Association for Computational Linguistics.
  * Zhang et al. (2018) Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018.  Graph convolution over pruned dependency trees improves relation extraction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2205–2215, Brussels, Belgium. Association for Computational Linguistics.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017\.  Position-aware attention and supervised data improve slot filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 35–45.
  * Zhou and Chen (2021) Wenxuan Zhou and Muhao Chen. 2021.  An improved baseline for sentence-level relation extraction.  _arXiv preprint arXiv:2102.01373_.


utf8

# ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English
Translation

Mohammed Khalil
Independent Researcher
<EMAIL_ADDRESS>
Mohammed Sabry
ADAPT/DCU, Dublin, Ireland
<EMAIL_ADDRESS>

###### Abstract

Classical Arabic represents a significant era, encompassing the golden age of
Arab culture, philosophy, and scientific literature. With a broad consensus on
the importance of translating these literatures to enrich knowledge
dissemination across communities, the advent of large language models (LLMs)
and translation systems offers promising tools to facilitate this goal.
However, we have identified a scarcity of translation datasets in Classical
Arabic, which are often limited in scope and topics, hindering the development
of high-quality translation systems. In response, we present the ATHAR
dataset, comprising 66,000 high-quality Classical Arabic to English
translation samples that cover a wide array of subjects including science,
culture, and philosophy. Furthermore, we assess the performance of current
state-of-the-art LLMs under various settings, concluding that there is a need
for such datasets in current systems. Our findings highlight how models can
benefit from fine-tuning or incorporating this dataset into their pretraining
pipelines. The dataset is publicly available on the HuggingFace Data Hub at
https://huggingface.co/datasets/mohamed-khalil/ATHAR.

##  1 Introduction

Classical Arabic is the foundation of Arabic linguistic theory and is well
comprehended by educated Arabic readers. It significantly differs from Modern
Standard Arabic (MSA) it is also called (Arangiyya), which is more simplified
in terms of its vocabulary, syntax, morphology, phraseology, and semantics.

Classical Arabic poses unique challenges for accurate translation into
English. Unlike MSA, which dominates formal speeches, news channels, and
modern literary works, and urban dialects prevalent on social media platforms,
Classical Arabic is less commonly used today. Yet, it remains vital, present
in many historical documents, books, and literary texts rich with knowledge
from the Arab and Muslim golden ages, all awaiting translation and broader
exposure.

Current translation systems, including Google Translate and large language
models like ChatGPT and Llama, struggle with Classical Arabic, often
neglecting it in favour of MSA and urban dialects during dataset creation for
machine translation.

This work introduces the ATHAR dataset, a translation resource from Classical
Arabic to English. “ATHAR” “أثر>” means "legacy" or "ancient work." It
represents the literary and cultural heritage and underscores the dataset’s
role in illuminating classical Arabic texts, emphasizing their importance in
preserving and conveying this heritage. The ATHAR dataset aims to address the
representativeness and quality limitations of previous datasets.

This work is organised as follows: Section 2 explores the challenges faced by
previous researchers in translating Classical Arabic and details how the ATHAR
dataset addresses these challenges. Section 3 elaborates on the methodologies
used to create the ATHAR dataset, including steps for data collection,
cleaning, and preprocessing to ensure the quality and reliability of the data.
In Section 4 we conduct experiments to assess the performance of state-of-the-
art LLMs on the ATHAR dataset across various settings such as zero-shot, few-
shot, and fine-tuning scenarios. The paper concludes with Section 5,
highlighting the importance of the ATHAR dataset in developing culturally and
linguistically authentic Arabic language models and advancing Arabic natural
language processing.

##  2 Related Work

The notable gap in datasets for Classical Arabic has led to several efforts to
gather more resources for Arabic Natural Language Processing (NLP). Prominent
among these are the Tanzil and Authentic Hadith datasets, which draw from
religious texts. The Tanzil dataset offers translations of the Quran in over
40 languages, including Arabic to English, and is hosted on Tanzil.net and the
OPUS database Tiedemann (2012). The Authentic Hadith dataset provides
translations of the sayings and practices of the Prophet Muhammad, known for
its authenticity and rigorous translation process Altammami et al. (2020).
While these datasets are rich, they mainly focus on religious content and
don’t fully represent the diverse genres of classical Arabic literature.
Additionally, the Poem Comprehensive Dataset (PCD) Yousef et al. (2019)
provides a dataset focused on Classical Arabic poetry. While this dataset is a
valuable resource, it encompasses a limited range of thematic areas.

In contrast, there are numerous datasets for Modern Arabic that include a rich
and diverse context, such as the OPUS-100 dataset Zhang et al. (2020), the
MultiUN dataset Eisele and Chen (2010), and the IWSLT2017 dataset Cettolo et
al. (2017). However, Modern Arabic differs significantly from Classical Arabic
in its vocabulary, syntax, and stylistic features, which are not well-
represented in these contemporary datasets.

Additionally, significant efforts like those by Alrabiah et al. (2014) have
focused on Arabic historical linguistics, producing datasets that explore the
evolution and contexts of the Arabic language. Although these datasets are not
directly applicable in practical translation tasks due to their lack of
translations into other languages, they offer invaluable resources for
pretraining LLMs with the knowledge necessary to distinguish between Classical
and Modern Arabic. Moreover, the initiative by Aloui et al. (2024) introduced
a corpus of 101 billion Arabic words, crucial for developing LLMs targeted at
the Semitic Arabic language. This extensive corpus, predominantly in Modern
Arabic with some Classical content, could help LLMs understand Classical
Arabic, particularly when combined with smaller, specialized downstream
translation datasets.

ATHAR dataset aims to address the representativeness issues in previous
classical Arabic datasets by compiling sentences from various contexts and
historical periods on topics like science, medicine, philosophy, and culture.
This dataset will help fill the gaps in classical Arabic resources and provide
a more comprehensive foundation for developing effective translation models.

##  3 ATHAR Dataset

This section outlines the development of the ATHAR dataset. We start by
identifying the sources from which the data was collected. Subsequently, we
detail the processing steps implemented to ensure the dataset’s high quality.
Additionally, we compare ATHAR to previous classical Arabic datasets and well-
known modern Arabic datasets.

###  3.1 Data Collection

The ATHAR dataset comprises 66K sentences extracted from seminal Classical
Arabic texts, expertly translated into English. These texts are cornerstone
works spanning diverse genres and historical periods, providing profound
insights into Islamic and world history, philosophy, science, medicine, and
culture. This rich literary heritage is showcased in our dataset, sourced from
the Rasaif website at https://rasaif.com/111At the time of data collection and
publication of this work, there were no restrictions on scraping and using
resources from this website.. The specific books included in our dataset are
detailed below:

  * •

تاريخ الطبري> (History of al-Tabari): Authored by the Persian historian
Muhammad ibn Jarir al-Tabari, this comprehensive history covers the world from
creation to 915 AD and is a pivotal source for early Islamic history.

  * •

تُحفة النُّظار - الرحلة> (The Travels of Ibn Battuta): This travelogue by Ibn
Battuta documents his 14th-century travels across the Islamic world and
beyond, offering insights into the era’s geography, cultures, and politics.

  * •

مقدمة ابن خلدون> (The Muqaddimah of Ibn Khaldun): A pioneering work in
historiography and sociology by Ibn Khaldun that analyzes the rise and fall of
empires and the dynamics of society, completed in 1377 AD.

  * •

الاموال> (The Book of Revenue): Abu Ubayd al-Qasim ibn Sallam’s exploration of
Islamic economic principles, taxation, and public finance, written around 837
AD.

  * •

العقد الفريد> (The Unique Necklace): A collection by Ibn Abd Rabbih
encompassing Arabic prose and poetry on literature, history, and moral
teachings, finalized in the early 10th century.

  * •

المناظر> (The Optics): Ibn al-Haytham’s foundational text in optics that
discusses the nature of light and vision, pivotal to the development of the
scientific method, completed around 1021 AD.

  * •

النوادر السلطانية و المحاسن اليوسفية> (The Sultan’s Anecdotes and Yusuf’s
Merits): A biographical account of Sultan Salah al-Din by Ibn Shaddad,
emphasizing his military and administrative achievements, written in the late
12th century.

  * •

التصريف لمن عجز عن التأليف> (The Method of Healing): Al-Zahrawi’s extensive
medical encyclopedia that details surgical procedures and treatments,
influential in medieval European medicine, completed around 1000 AD.

  * •

نشوار المحاضرة و أخبار المذاكرة> (Anecdotes of the Session and Stories of
Recollection): Al-Tanukhi’s compilation of stories and historical accounts
that offer a window into the social and cultural life of the Islamic world,
from the 10th century.

  * •

القانون في الطب> (The Canon of Medicine): Ibn Sina’s comprehensive medical
guide that became a foundational text in Islamic and European medicine,
covering various health aspects, finalized around 1025 AD.

  * •

الاعتبار> (The Book of Reflection): Usama ibn Munqidh’s autobiographical
reflections on the Crusades and life in the medieval Islamic world, completed
in the mid-12th century.

  * •

الرسالة> (The Epistle): Imam al-Shafi’i’s seminal work on Islamic
jurisprudence, outlining legal theory and methodology, completed around 820
AD.

  * •

البخلاء> (The Book of Misers): Al-Jahiz’s satirical critique of miserliness,
offering insights into the social and economic conditions of his time,
completed in the 9th century.

  * •

نَهْجُ البَلاغَةِ> (The Path of Eloquence): A compilation of sermons, letters,
and sayings attributed to Ali ibn Abi Talib, noted for its literary quality
and philosophical depth, assembled in the 10th century.

  * •

فتوح الشام> (Fattouh Al-Sham): Al-Waqidi’s detailed historical account of the
Islamic conquests in Syria, documenting battles and strategies of the era,
completed in the 9th century.

  * •

الأخلاق والسير> (Ethics and Conduct): Ibn Miskawayh’s treatise on ethics and
morality from an Islamic philosophical perspective, penned in the 11th
century.

  * •

حي بن يقظان> (Hayy ibn Yaqdhan): Ibn Tufail’s philosophical novel about a
boy’s journey to intellectual and spiritual enlightenment on a deserted
island, completed in the 12th century.

  * •

الاصنام> (The Book of Idols): Hisham ibn al-Kalbi’s detailed account of pre-
Islamic Arabian deities and their worship rituals, written in the 9th century.

###  3.2 Preprocessing

To prepare the dataset for use in machine translation models, several
preprocessing steps were undertaken:

#### Cleaning the Data:

During the initial stages of the ATHAR dataset collection process, the primary
challenge we encountered involved entries where Arabic and English texts were
flipped within HTML class labels. For further details on this issue, see
Appendix A. To address this, we implemented a simple rule-based technique that
identifies the language of the text based on the predominance of characters
from the respective language’s alphabet. After collecting the data, we found
the texts contained various types of noise such as empty entries, incorrect
sentences, duplicate entries, entries consisting solely of numbers, and other
unwanted characters. These issues were systematically identified and removed
to enhance the dataset’s quality. Additionally, unnecessary columns like
"book" and "author" were deleted to focus exclusively on the translation
pairs. We also removed religious Quranic verses from the dataset, as they were
few in number and not dealt with correctly.

#### Alignment Verification:

As in the Rasaif websites—where we collected the translations from—the
translations are created by human volunteers. Given the lack of detailed
insights into their methods, and to ensure that each Arabic sentence was
correctly aligned with its English translation, thereby maintaining the
context and intended meaning, the authors manually verified the collected
datasets. This verification process was crucial to confirm that the Arabic-
English pairs were properly aligned and accurately conveyed the content of
each other.

###  3.3 Comparative Analysis of ATHAR and Other Arabic Datasets

In this subsection, we analyze our dataset in comparison to existing classical
and modern Arabic datasets, focusing on several linguistic measures: lexical
diversity, stopword ratio, and the distribution of short versus long
sentences, in addition to unique words count and dataset sizes.

For lexical diversity, we utilized the Measure of Textual Lexical Diversity
(MTLD) index McCarthy (2005), which calculates the average length of text
segments where the vocabulary diversity remains above a specified
threshold—0.75 in our case—indicating how consistently a text maintains its
lexical richness. The stopword ratio was calculated by determining the
occurrence of stopwords relative to the total word count in the datasets.
Short sentences were defined as any sentence containing 10 or fewer words,
while long sentences are those with 30 or more words.

Before conducting the analysis, all datasets were standardized by removing
redundant diacritics and letters. As detailed in Table 1, the ATHAR dataset
boasts one of the highest MTLD scores, suggesting that the text can sustain a
high level of lexical diversity over a large number of words. This implies
that the vocabulary is varied and the text does not quickly repeat words.
Furthermore, our dataset maintains a balanced representation of both short and
long sentences, providing a stark contrast to the variable sentence lengths
found in other datasets.

Dataset Attributes | ATHAR | Tanzil | Arabic PCD | KSUCCA | OPUS-100-ar-en | iwslt2017-ar-en | multiun-ar-en
---|---|---|---|---|---|---|---
Dataset size | 66K | 187K | 1.8M | 1.9M | 1M | 241K | 9.67M
Unique words count | 138944 | 48104 | 720167 | 908771 | 370601 | 185390 | 841732
Lexical diversity (MTLD) | 55.63 | 101.31 | 11.86 | 40.87 | 17.46 | 34.12 | 70.10
Ratio of stopwords (%) | 26.04 | 30.35 | 24.62 | 24.71 | 27.59 | 29.67 | 21.31
Average length of sentences | 20.78 | 34.35 | 9.26 | 25.33 | 8.39 | 13.86 | 22.89
Proportion of very short sentences (%) | 24.06 | 11.18 | 76.57 | 41.28 | 79.81 | 45.98 | 23.07
Proportion of very long sentences (%) | 23.11 | 47.53 | 0.00 | 24.44 | 4.71 | 7.04 | 26.61

Table 1: Overview of Linguistic Characteristics in Arabic Language Datasets:
Size, Diversity, and Sentence Metrics

##  4 Evaluating State-of-the-Art LLMs on the ATHAR Dataset

In this section, we aim to evaluate the performance of state-of-the-art
language models on classical Arabic translations using the ATHAR dataset. We
selected four leading models for this analysis: GPT-4o, Llama-3 70B, Llama-3
8B, and Llama-2 7B.

Initially, we assessed the zero-shot capabilities of these models.
Subsequently, we evaluated the Llama-3 8B and Llama-2 7B models under few-shot
conditions. Finally, we focused on fine-tuning the Llama-3 8B model using two
distinct methods: full fine-tuning, where all parameters of the model were
adjusted, and LoRA Hu et al. (2021) fine-tuning, which only involved
adjustments to a subset of newly added parameters.

The objective of these comprehensive experiments is to maximize the potential
of these models, understand performance variations under different settings,
and explore how the ATHAR dataset can bridge existing performance gaps.

In the following subsections, we will detail the hyperparameters and metrics
used in our experiments and analyze the results.

###  4.1 Hyperparameters and Evaluation Metrics

#### Hyperparameters:

During inference, the generation decoding strategy involved setting the
maximum number of new tokens to $2048$. Sampling strategies included Top-K and
Top-P settings at $100$ and $0.95$, respectively, with a temperature parameter
set at $0.3$.

For the fine-tuned models, specifically Llama-3 8B with both full and LoRA
tuning, training was implemented in an instruction-input/response format. The
input consisted of Arabic text, and the models were trained to generate the
corresponding English translation as the response. The training dataset
included $65k$ samples. Models were trained using $FP16$ precision, with a
learning rate of $5e-6$, adjusted via a linear scheduler across three epochs.
The batch size was set at $16k$ tokens, which was achieved by accumulating
gradients of four samples twice, each sample having a maximum length of $2048$
tokens. An AdamW optimizer was utilized, with beta values of $0.90$ and
$0.999$ for the first and second moment estimates, respectively.

Regarding the prompt structures used in our experiments, Table 3 details the
specific prompt structures we utilized across zero-shot, few-shot, and fine-
tuning settings.

#### Evaluation Metrics:

In assessing our models, we employed well-established metrics commonly used in
translation evaluations: METEOR Banerjee and Lavie (2005), ROUGE-L Lin (2004),
and SacreBLEU Post (2018). These metrics are all scored on a scale where
higher values indicate better performance, though each has a different range.
METEOR focuses on the alignment between the translation output and reference
translations, considering synonymy and stemming. ROUGE-L measures the longest
common subsequence, which is useful for evaluating the fluency of the text.
SacreBLEU provides a consistent and comparable score across studies by
standardizing the BLEU score calculation. Together, these metrics provide a
comprehensive view of translation quality, covering aspects from accuracy to
fluency.

###  4.2 Results and Discussion

#### Results:

The evaluation results, presented in Table 2, highlight significant variances
in model performance across different settings. The GPT-4o model excelled in a
zero-shot (ZS) setting, outperforming all other models with scores of $0.357$
on METEOR, $0.441$ on ROUGE-L, and $14.705$ on SacreBLEU. In contrast, the
Llama-3 70B Instruct model, also evaluated in a zero-shot setting, registered
slightly lower scores of $0.342$ on METEOR, $0.413$ on ROUGE-L, and $13.011$
on SacreBLEU. This disparity might reflect differences in training regimes or
underlying model architectures.

In the same zero-shot context, both the Llama-3 8B Instruct and Llama-2 7B
models displayed considerably lower performance, especially noted by sharp
declines in their SacreBLEU scores to $0.339$ and $0.310$, respectively. These
findings suggest inherent limitations in the zero-shot capabilities of these
models for translation tasks.

Remarkable improvements were evident when the Llama-3 8B model was tested in a
few-shot (FS) setting using just three samples. This adjustment led to marked
score increases to $0.174$ on METEOR, $0.167$ on ROUGE-L, and $0.971$ on
SacreBLEU, underscoring the potential of in-context learning to significantly
boost model performance. In contrast, the few-shot enhancement for the Llama-2
7B model was minimal, which may indicate model-specific sensitivity to
training data increments.

The Llama-3 8B model demonstrated further enhancements upon undergoing full
fine-tuning, achieving a METEOR score of $0.275$, a ROUGE-L score of $0.336$,
and a SacreBLEU score of $6.061$. Additionally, the LoRA tuning method, which
involves less extensive modifications, also yielded improved results, with
scores reaching $0.279$ on METEOR, $0.339$ on ROUGE-L, and $8.792$ on
SacreBLEU.

#### Discussion:

The results presented in Table 2 underscore the challenges faced by state-of-
the-art LLMs when tasked with translating Classical Arabic to English. By
providing state-of-the-art models with targeted training opportunities, the
ATHAR dataset not only boosts model performance but also contributes
significantly to the broader NLP community’s understanding of and engagement
with Classical Arabic. This dataset, therefore, holds substantial value, as it
aids in developing more nuanced and capable translation systems.

Model |  METEOR $\uparrow$ |  ROUGE-L $\uparrow$ |  SacreBLEU $\uparrow$
---|---|---|---
GPT-4o + ZS | 0.357 | 0.441 | 14.705
Llama-3 70B Instruct + ZS | 0.342 | 0.413 | 13.011
Llama-3 8B Instruct + ZS | 0.115 | 0.068 | 0.339
Llama-2 7B + ZS | 0.116 | 0.099 | 0.310
Llama-3 8B Instruct + FS3 | 0.174 | 0.167 | 0.971
Llama-2 7B + FS3 | 0.036 | 0.039 | 0.408
Llama-3 8B + Full-Tuning | 0.275 | 0.336 | 6.061
Llama-3 8B + LoRA | 0.279 | 0.339 | 8.792

Table 2: Performance of State-of-the-Art LLMs on the Classical Arabic to English Translation Task. The table displays METEOR, ROUGE-L, and SacreBLEU scores for various models under different settings: zero-shot (ZS), few-shot with three samples (FS3), and fine-tuning (Full-Tuning & LoRA) on a 1k test set. Model |  Prompt
---|---
GPT-4o + ZS Llama-3 70B Instruct + ZS Llama-3 8B Instruct + ZS Llama-2 7B + ZS |  Translate the following text from Classical Arabic to English\nPlease return only the translated text without any introductions or additions: {Arabic text}
Llama-3 8B Instruct + FS3 Llama-2 7B + FS3 |  Translate the following Classical Arabic text into English. Follow the provided examples for consistency and accuracy. Examples: Arabic: باجر قَالَ ابْن دُرَيْد وَهُوَ صَنَمٌ كَانَ لِلأَزْدِ فِي الْجَاهِلِيَّةِ وَمَنْ جَاوَرَهُمْ من طَيء وقضاعة. كَانُوا يَعْبُدُونَهُ. بِفَتْحِ الْجِيمِ, وَرُبَّمَا قَالُوا بَاجِرُ بِكَسْرِ الْجِيمِ English: According to Ibn Durayd, Bajar was an idol worshipped by the Azd tribe and neighboring tribes such as Tayyi’ and Quda’ah during the Jahiliyyah period. It is also pronounced Bajir. Arabic: (ذبل فِي آخر النُّسْخَة الَّتِي اعتمدتها فِي الطَّبْع) الْيَعْبُوبُ صَنَمٌ لِجَدِيلَةِ طَيْءٍ. وَكَانَ لَهُمْ صَنَمٌ أَخَذَتْهُ مِنْهُمْ بَنُو أَسَدٍ فَتَبَدَّلُوا الْيَعْبُوبَ بَعْدَهُ. قَالَ عبيد فَتَبَدَّلُوا الْيَعْبُوبَ بَعْدَ إِلهِهِمْ … صَنَمًا فَقَرُّوا يَا جديل واعذبوا (أَي لَا تَأْكُلُوا على ذَلِك وَلَا تشْربُوا) English: Al-Ya’bub was the idol of the Jadilah tribe of Tayyi’. They initially had a different idol, but after it was taken by Banu Asad, they adopted Al-Ya’bub in its place. ‘Abid said they replaced their former god with Al-Ya’bub, and as a result, O Jadilah, abstain from food and drink. Arabic: فَلم يزل الْفلس يعبد حَتَّى ظهرت دَعْوَة النَّبِي عَلَيْهِ السَّلَام فَبعث إِلَيْهِ على ابْن أَبِي طَالِبٍ فَهَدَمَهُ وَأَخَذَ سَيْفَيْنِ كَانَ الْحَارِثُ بن أبي شمرٍ الغساني, ملك غَسَّان قَلَّدَهُ إِيَّاهُمَا, يُقَالُ لَهُمَا مِخْذَمٌ وَرَسُوبٌ(وَهُمَا السَّيْفَانِ اللَّذَانِ ذَكَرَهُمَا عَلْقَمَةُ بْنُ عَبْدَةَ فِي شِعْرِهِ). فَقَدِمَ بِهِمَا عَلِيُّ بْنُ أَبِي طَالِبٍ عَلَى النَّبِيِّ صَلَّى اللَّهُ عَلَيْهِ وَسَلَّمَ فَتَقَلَّدَ أَحَدَهُمَا ثُمَّ دَفَعَهُ إِلَى عَلِيِّ بْنِ أَبِي طَالِبٍ, فَهُوَ سَيْفُهُ الَّذِي كَانَ يَتَقَلَّدُهُ English: Al-Fals continued to be worshipped until the advent of the Prophet. When ‘Ali ibn Abi Talib was sent to destroy it, he did so and took two swords, Mikhdham and Rasub, which had been given to Al-Fals by Al-Harith ibn Abi Shamir, the king of Ghassan. ‘Ali presented them to the Prophet, who kept one and returned the other to ‘Ali, who continued to wear it. Translate the following: Arabic: {Arabic text} English:
Llama-3 8B + Full-Tuning Llama-3 8B + LoRA |  Translate the following input text from Classical Arabic to English, please return only the translated text without any introductions or additions. ### Input: {Arabic text} ### Response:
Table 3: Prompt Structures Used and Their Corresponding Models in Zero-Shot
(ZS), Few-Shot with Three Samples (FS3), and Full-Tuning Evaluation
Experiments.

##  5 Conclusion

To conclude, we introduce the ATHAR dataset, which enhances the existing
corpus of Classical Arabic datasets by incorporating a broader range of
topics. Our evaluation of the current status of LLMs underscores the critical
need for the ATHAR dataset within the fine-tuning and training pipelines. More
broadly, this need highlights the necessity for more comprehensive Classical
Arabic datasets to improve the quality of translation systems in this domain.
Future work will aim to expand the ATHAR dataset to include an even wider
array of texts and topics, thereby further enhancing translation quality.

## References

  * Aloui et al. (2024) Manel Aloui, Hasna Chouikhi, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 2024.  101 billion arabic words dataset.
  * Alrabiah et al. (2014) M Alrabiah, A Al-Salman, ES Atwell, and N Alhelewh. 2014.  Ksucca: a key to exploring arabic historical linguistics.  _International Journal of Computational Linguistics (IJCL)_ , 5(2):27 – 36.  (c) 2014, Alrabiah, M, Al-Salman, A, Atwell, ES and Alhelewh, N. This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial ShareAlike (CC BY-NC-SA 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, provided the original work is properly cited, the use is non-commercial and any derivative works are licensed under the same terms.
  * Altammami et al. (2020) Shatha Altammami, Eric Atwell, and Ammar Alsalka. 2020.  The arabic–english parallel corpus of authentic hadith.  _International Journal on Islamic Applications in Computer Science And Technology_ , 8(2).
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Cettolo et al. (2017) Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017.  Overview of the IWSLT 2017 evaluation campaign.  In _Proceedings of the 14th International Conference on Spoken Language Translation_ , pages 2–14, Tokyo, Japan. International Workshop on Spoken Language Translation.
  * Eisele and Chen (2010) Andreas Eisele and Yu Chen. 2010.  MultiUN: A multilingual corpus from united nation documents.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_ , Valletta, Malta. European Language Resources Association (ELRA).
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * McCarthy (2005) Philip M McCarthy. 2005.  _An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD)_.  Ph.D. thesis, The University of Memphis.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Yousef et al. (2019) Waleed A. Yousef, Omar M. Ibrahime, Taha M. Madbouly, and Moustafa A. Mahmoud. 2019.  Learning meters of arabic and english poems with recurrent neural networks: a step forward for language understanding and synthesis.  _arXiv preprint arXiv:1905.05700_.
  * Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020.  Improving massively multilingual neural machine translation and zero-shot translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1628–1639, Online. Association for Computational Linguistics.

##  Appendix A Preprocessing: Flipped Cells in Data Collection

During the scraping process, we encountered difficulties in extracting the
English and Arabic texts from the containers (cells) because the Arabic texts
were sometimes labeled as “flex-right” and English texts as “flex-left” in
many instances, with the positions reversed in other cases. To address this,
we counted the number of Arabic and English characters in each label and
assigned the language based on the predominance of characters from either
alphabet. Examples of such inconsistencies are provided below, where the
labels for “flex-right” and “flex-left” are swapped, complicating the
identification process:

Example.1 Arabic text on left and English text on right:

<div class="flex">
<div class="flex-right">
<span>Farewell my brother, whom it was my duty to help. The blessings and the
mercy of God upon you</span>
</div>
<div class="flex-left">
والسلام عليك أيها الأخ المفترض إسعافه ورحمة الله وبركاته
</div>
</div>

Example.2 Arabic text on right and English text on left:

<div class="flex">
<div class="flex-right">
أنشدني أبو القاسم الصرويّ لنفسه" ويوم كيوم البين حرّا قطعته … على سابح طاوي
الأياطل سابق، أخوض عليه جمرة القيظ حاسرا … كأنّي على الهجران في قلب عاشق
</div>
<div class="flex-left">
<span>Abu’l-Qasim Suri recited to me the following lines of his own: "O day as
hot as day of lovers’ parting I spent upon a courser lean of flank! On him in
summer’s wave like heart of lover burning with separation’s pain I
sank."</span>
</div>
</div>


# Learning the progression and clinical subtypes of Alzheimer’s disease from
longitudinal clinical data

Vipul Satone1
&Rachneet Kaur1
&Faraz Faghri1,2
&Mike A Nalls2
&Andrew B Singleton2
&Roy H Campbell1
&for the Alzheimer’s Disease Neuroimaging Initiative

Data used in preparation of this article were obtained from the Alzheimer’s
Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such,
the investigators within the ADNI contributed to the design and implementation
of ADNI and/or provided data but did not participate in analysis or writing of
this report. A complete listing of ADNI investigators can be found at:
http://adni.loni.usc.edu/wp-
content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf 1 University of
Illinois at Urbana-Champaign, USA
2 National Institutes of Health, USA

###### Abstract

Alzheimer’s disease (AD) is a degenerative brain disease impairing a person’s
ability to perform day to day activities. The clinical manifestations of
Alzheimer’s disease are characterized by heterogeneity in age, disease span,
progression rate, impairment of memory and cognitive abilities. Due to these
variabilities, personalized care and treatment planning, as well as patient
counseling about their individual progression is limited. Recent developments
in machine learning to detect hidden patterns in complex, multi-dimensional
datasets provides significant opportunities to address this critical need. In
this work, we use unsupervised and supervised machine learning approaches for
subtype identification and prediction. We apply machine learning methods to
the extensive clinical observations available at the Alzheimer’s Disease
Neuroimaging Initiative (ADNI) data set to identify patient subtypes and to
predict disease progression. Our analysis depicts the progression space for
the Alzheimer’s disease into low, moderate and high disease progression zones.
The proposed work will enable early detection and characterization of distinct
disease subtypes based on clinical heterogeneity. We anticipate that our
models will enable patient counseling, clinical trial design, and ultimately
individualized clinical care.

##  1 Introduction

Alzheimer’s disease (AD) is an irreversible and age-associated
neurodegenerative disease. Alzheimer’s disease is the most common form of
dementia which progressively affects memory, intellectual skills, and other
mental functions. Researchers have shown that AD initiates 10-15 years before
the actual disease symptoms are manifested [2]. In the absence of any cure,
early detection of AD onset and progression rate is significant as it helps in
designing disease modifying treatment strategies. A major challenge for AD
prediction and individualized clinical care is the phenotypic heterogeneity
that exists within the AD population.

Attempts thus far at the characterization of disease subtypes have followed a
path of clinical observation based cognitive abilities. The disease is often
separated into dementia and mild cognitive impairment (MCI), a stage between
expected cognitive decline of normal aging and a more serious decline of
dementia. MCI patients are at a higher risk of progressing to dementia, but
not all the patients end up developing AD [11]. Previous studies attempted to
detect incipient AD in patients with MCI or predict early stage of the AD
using Cerebrospinal Fluid (CSF) [10], [11]. Psychometric and imaging data has
also been used for predicting the progression of dementia in patients with
amnestic mild cognitive impairment [13]. However, less work has been done on
using just clinical data and predicting the Alzheimer’s disease progression
rate.

We have previously used multimodal clinical data to produce progression space
for Parkinson’s Disease [8]. Here we describe our work on clustering and early
prediction of the clinical progression of the AD. We have used dimension
reduction techniques to identify progression space, Gaussian Mixture Model
(GMM) for subtyping, and Random Forest to predict an individual’s progression.
This work has identified Alzheimer’s Disease progression space with three
clinically distinct disease subtypes corresponding to memory decline and
cognitive decline. Following the successful creation of disease subtypes
within a progression space, we created a baseline predictor that accurately
predicts an individual patient’s clinical group membership two years later.

##  2 Methods

###  2.1 Study design and participants

This study included data from the Alzheimer’s Disease Neuroimaging Initiative
(ADNI) database (adni.loni.usc.edu). The ADNI was launched in 2003 as a
public-private partnership, led by Principal Investigator Michael W. Weiner,
MD. For up-to-date information, see www.adni-info.org. Machine learning models
are developed using baseline and 12 months of data, for each patients’
position after 24 and 48 months. The study consist of 248 cases (including 123
(49.59%) female, average age for all participants is 71.56 $\pm$ 6.78 years,
94.75% of them are of European ancestry) for prediction at $48^{th}$ month and
453 cases (including 208 (45.91%) female, average age for all participants is
72.34 $\pm$ 7.12 years, 93.59% of them are of European ancestry) for
prediction at $24^{th}$ month. Cases in $24^{th}$ month have a mean age of
72.89 $\pm$ 6.06, 71.61 $\pm$ 7.49 and 72.92 $\pm$ 8.11 corresponding to
controls, MCI, and dementia patients. Cases in $48^{th}$ month have a mean age
of 72.18 $\pm$ 6.63, 71.36 $\pm$ 6.67 and 70.34 $\pm$ 7.42 corresponding to
control, MCI and dementia patients. We used comprehensive and longitudinally
collected measurements including: Montreal Cognitive Assessment [14], Clinical
Dementia Rating [3], Neuropsychiatric Inventory Questionnaire [6],
Neuropsychological Battery [7], Mini Mental State Exam [5], Geriatric
depression scale [1], Everyday cognition - study partner [16], Everyday
cognition - participant [2] and Functional Assessment Questionnaire [9].

###  2.2 Procedures and statistical analysis

Cases with data recorded for all the measurements were included. After
imputations using domain knowledge, data cleansing and preprocessing, 218
clinical features were retained. If required, one hot encoding was used for
all the categorical variables. Min-max normalization was used to retain
progression trend in multimodal measurements. To reduce the dimensionality of
the data set, Non-negative Matrix Factorization (NMF) [19] (with a rank of 2)
was used. NMF reduced high dimensional data into a two-dimensional space. The
transformed data were used to project the patient’s disease progression stage
at the $24^{th}$ and $48^{th}$ month. Coefficient matrix obtained from the NMF
was used as a progression indicator to interpret the representation of the
reduced space dimensions.

Unsupervised clustering method Gaussian Mixture Model (GMM) [12] was used to
identify clusters. Bayesian Information Criterion (BIC) [17] was used to
select the optimum number of clusters in GMM. After obtaining the progression
space and classifying MCI and dementia patients into different progression
subtypes, various supervised learning classifiers were compared to predict an
individual patient’s progression. We validate the effectiveness of our
predictive models using a five-fold cross-validation and measuring the AUC of
the ROC curve.

Figure 1: Left: Total 453 cases are projected in the Alzheimer’s Disease
Progression Space at $24^{th}$ month. Controls are represented in red, MCI in
blue and dementia in green. Right: Three different progression rates are
identified in MCI and dementia patients at $24^{th}$ month. In this figure low
progression rate zone is represented in red, moderate progression rate in blue
and high progression rate in green.

Figure 2: Left: Area under ROC curve for each individual class. The predictions for disease stage at $24^{th}$ month were made using Random Forest Algorithm. Right: Shows positions of the reversions (only MCI to control and dementia to MCI) at $24^{th}$ month. Table 1: Accuracy of the two models | Accuracy | AUC -control | AUC -low | AUC -moderate | AUC -high
---|---|---|---|---|---
M24 | 84.54 $\pm$ 3.35 | 0.95 $\pm$ 0.03 | 0.90 $\pm$ 0.06 | 0.92 $\pm$ 0.03 | 0.97 $\pm$ 0.01
M48 | 80.24 $\pm$ 4.33 | 0.90 $\pm$ 0.04 | 0.84 $\pm$ 0.03 | 0.88 $\pm$ 0.03 | 0.96 $\pm$ 0.03

##  3 Results & Discussion

Figure 1 demonstrates Alzheimer’s disease progression space for $24^{th}$
month, generated by the dimension reduction step. Using the values from the
coefficient matrix, the latent features for the x and y-axis were interpreted
and identified as cognition decline and memory decline representations. Along
the positive direction of y-axis, the memory decline increases and along the
negative direction of x-axis the cognitive decline increases. Therefore, the
progression rate increases in the direction of the positive y-axis and
negative x-axis. Using GMM, three optimum clusters corresponding to low,
moderate and high progression rate were identified. Figure 1 shows the
distribution of the three clusters in MCI and dementia patients. Random forest
(RF) classifier [4] provided the most accurate model. RF parameters were fine-
tuned using grid search and evaluated by 5 fold cross-validation accuracy. Two
models were developed for predicting progression after 24 and 48 months
(accuracy results are presented in Table 1). Figure 2 shows the area under ROC
curve of the progression prediction model.

ADNI is a longitudinal study in which the disease state of patients is
assessed every 12 months. Generally, most patients’ condition either
deteriorated or stayed the same. However, in rare cases, the patient’s
condition has reversed to a better state. Essentially, some patients were
observed moving from dementia to MCI or MCI to control stage. Interestingly,
this behavior is been captured in our progression space. Figure 2 shows these
reversion cases in the progression space. Patients moving from dementia to MCI
fall in the intermediate region between dementia and MCI (moderate progression
rate region). Similarly, patients moving from MCI to control lie in
intermediate progression region between MCI and control. This observation
signifies the robustness of our proposed progression space model.

In another observation, we see the effect of Apolipoprotein E (APOE) genetic
mutation. APOE comes in three variants namely, 2, 3, and 4, out of which APOE4
gene variant is closely associated with higher risk of Alzheimer’s disease
[18]. Figure 3 shows the cases with 0 and 2 counts of APOE4 variants in our AD
progression space. As evident from the figure, observations with 0 count of
APOE4 variant are concentrated towards the low progression rate zone, whereas
observations with 2 counts of APOE4 variant are concentrated towards moderate
and high disease progression rate. This observation is validated by the
existing literature which has identified a significant correlation between
APOE4 genetic variants and cognitive performance [15]. Noting that APOE4 data
was not considered during the construction of the projection space as only the
time-variant measurements are used to obtain progression space.

Using our progression space model we can also observe the aging pattern in
controls. Figure 3 shows the normal cognition and memory decline in controls
attributed to increase of age. Since this decline is not severe, control cases
do not lie in moderate or high progression rate zones. A simple clustering of
observations into two clusters shows the stark difference in the mean age of
the clusters. It is interesting to note that the mean age for the cluster
which is relatively close to the moderate progression rate zone is 75.25 years
and mean age of the cluster away from the moderate progression rate zone is
70.75 years.

Figure 3: Left: Projection of AD cases with the different number of APOE4
variants to the projection space at $24^{th}$ month. Right: Shows the mean age
of cases in the two clusters of controls. Clusters represent aging pattern in
controls at $24^{th}$ month. The cluster near moderate progression rate zone
has high mean age than one which is away from it.

##  4 Conclusions

Prediction of disease and disease course is a critical challenge in the care,
treatment, and research of complex heterogeneous diseases such as Alzheimer’s.
In this work, we used machine learning techniques to identify Alzheimer’s
disease subtypes based on distinct progression stages. Furthermore, we
developed predictive models of the progression stage after 24 and 48 months
from baseline. The developed machine learning solutions potentially offer
substantial clinical impact by augmenting clinical decision-making for
physicians and healthcare specialists.

##  5 Acknowledgement

This work was supported in part by the Intramural Research Programs of the
National Institute on Aging (NIA)Z01-AG000949-02. Data collection and sharing
for this project was funded by the Alzheimer’s Disease Neuroimaging Initiative
(ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI
(Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the
National Institute on Aging, the National Institute of Biomedical Imaging and
Bioengineering, and through generous contributions from the following: AbbVie,
Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon
Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir,
Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company;
EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech,
Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy
Research and Development, LLC.; Johnson and Johnson Pharmaceutical Research
and Development LLC.; Lumosity; Lundbeck; Merck and Co., Inc.; Meso Scale
Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis
Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda
Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes
of Health Research is providing funds to support ADNI clinical sites in
Canada. Private sector contributions are facilitated by the Foundation for the
National Institutes of Health (www.fnih.org). The grantee organization is the
Northern California Institute for Research and Education, and the study is
coordinated by the Alzheimer’s Therapeutic Research Institute at the
University of Southern California. ADNI data are disseminated by the
Laboratory for Neuro Imaging at the University of Southern California.

## References

  * gd [1986] Yesavage ja, sheikh ji. 9/geriatric depression scale (gds) recent evidence and development of a shorter version.  _International Journal of Geriatric Psychiatry_ , 5:165––173, 1986.
  * Allaire and Marsiske [1999] Jason C Allaire and Michael Marsiske.  Everyday cognition: age and intellectual ability correlates.  _Psychology and aging_ , 14(4):627, 1999.
  * Berg [1984] Leonard Berg.  Clinical dementia rating.  _The British Journal of Psychiatry_ , 145(3):339–339, 1984.
  * Breiman [2001] Leo Breiman.  Random forests.  _Machine learning_ , 45(1):5–32, 2001.
  * Burns et al. [1998] A Burns, C Brayne, and M Folstein.  Mini-mental state: A practical method for grading the cognitive state of patients for the clinician. m. folstein, s. folstein and p. mchugh, journal of psychiatric research (1975) 12, 189-198.  _International Journal of Geriatric Psychiatry_ , 13(5):285–294, 1998.
  * De Jonghe et al. [2003] JF De Jonghe, Martin G Kat, CJ Kalisvaart, and Leo Boelaarts.  Neuropsychiatric inventory questionnaire (npi-q): A validity study of the dutch form.  _Tijdschr Gerontol Geriatr_ , 34(2):74–77, 2003\.
  * Dodrill [1978] Carl B Dodrill.  A neuropsychological battery for epilepsy.  _Epilepsia_ , 19(6):611–623, 1978.
  * Faghri et al. [2018] Faraz Faghri, Sayed Hadi Hashemi, Hampton Leonard, Sonja W Scholz, Roy H Campbell, Mike A Nalls, and Andrew B Singleton.  Predicting onset, progression, and clinical subtypes of parkinson disease using machine learning.  _bioRxiv_ , page 338913, 2018.
  * Fillenbaum and Smyer [1981] Gerda G Fillenbaum and Michael A Smyer.  The development, validity, and reliability of the oars multidimensional functional assessment questionnaire.  _Journal of gerontology_ , 36(4):428–434, 1981\.
  * Hansson et al. [2006] Oskar Hansson, Henrik Zetterberg, Peder Buchhave, Elisabet Londos, Kaj Blennow, and Lennart Minthon.  Association between csf biomarkers and incipient alzheimer’s disease in patients with mild cognitive impairment: a follow-up study.  _The Lancet Neurology_ , 5(3):228–234, 2006.
  * Hassan and Khan [2017] Syed Asif Hassan and Tabrej Khan.  A machine learning model to predict the onset of alzheimer disease using potential cerebrospinal fluid (csf) biomarkers.  _INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS_ , 8(12):124–131, 2017.
  * McLachlan and Basford [1988] Geoffrey J McLachlan and Kaye E Basford.  _Mixture models: Inference and applications to clustering_ , volume 84.  Marcel Dekker, 1988.
  * Moreland et al. [2018] Jamie Moreland, Timo Urhemaa, Mark van Gils, Jyrki Lötjönen, Jan Wolber, and Christopher J. Buckley.  Validation of prognostic biomarker scores for predicting progression of dementia in patients with amnestic mild cognitive impairment.  _Nuclear Medicine Communications_ , 39(4):297–303, 2018.
  * Nasreddine et al. [2005] Ziad S Nasreddine, Natalie A Phillips, Valérie Bédirian, Simon Charbonneau, Victor Whitehead, Isabelle Collin, Jeffrey L Cummings, and Howard Chertkow.  The montreal cognitive assessment, moca: a brief screening tool for mild cognitive impairment.  _Journal of the American Geriatrics Society_ , 53(4):695–699, 2005.
  * Raber et al. [2000] Jacob Raber, Derek Wong, Gui-Qui Yu, Manuel Buttini, Robert Mahley, Robert Pitas, and Lennart Mucke.  Apolipoprotein e and cognitive performance.  _Nature International Journal of Science_ , 404:352–354, 2000.
  * Salomon et al. [1991] Gavriel Salomon, David N Perkins, and Tamar Globerson.  Partners in cognition: Extending human intelligence with intelligent technologies.  _Educational researcher_ , 20(3):2–9, 1991.
  * Schwarz et al. [1978] Gideon Schwarz et al.  Estimating the dimension of a model.  _The annals of statistics_ , 6(2):461–464, 1978\.
  * Strittmatter et al. [1993] Warren J Strittmatter, Karl H Weisgraber, David Y Huang, Li-Ming Dong, Guy S Salvesen, Margaret Pericak-Vance, Donald Schmechel, Ann M Saunders, Dmitry Goldgaber, and Allen D Roses.  Binding of human apolipoprotein e to synthetic amyloid beta peptide: isoform-specific effects and implications for late-onset alzheimer disease.  _Proceedings of the National Academy of Sciences_ , 90(17):8098–8102, 1993.
  * Wang and Zhang [2013] Yu-Xiong Wang and Yu-Jin Zhang.  Nonnegative matrix factorization: A comprehensive review.  _IEEE Transactions on Knowledge and Data Engineering_ , 25(6):1336–1353, 2013.

##  6 Supplementary Material

Figure 4: Our machine learning analysis workflow

Figure 5: Left: Total 248 cases are projected in the Alzheimer’s Disease
Progression Space at $48^{th}$ month. Controls are represented in red, MCI in
blue and dementia in green. Right: Three different progression rates are
identified in MCI and dementia patients at $48^{th}$ month. In this figure low
progression rate zone is represented in red, moderate progression rate in blue
and high progression rate in green.  Figure 6: Comparison of different
algorithms for the prediction of progression after 24 months from the
baseline. Five-fold cross-validation accuracy was used to evaluate each model.
Random Forest algorithm provides the highest accuracy.  Figure 7:  Comparison
of different algorithms for the prediction of progression after 48 months from
the baseline. Five-fold cross-validation accuracy was used to evaluate each
model. Random Forest algorithm provides the highest accuracy.

Figure 8: Left: Area under ROC curve for each individual class. The
predictions for disease stage at $48^{th}$ month were made using Random Forest
Algorithm. Right: Shows positions of the reversions (only MCI to control and
dementia to MCI) at $48^{th}$ month..

Figure 9: Left: Projection of AD cases with the different number of APOE4
variants to the projection space at $48^{th}$ month. Right: Shows the mean age
of cases in the two clusters of controls at $48^{th}$ month. Clusters
represent aging pattern in controls. The cluster near moderate progression
rate zone has high mean age than one which is away from it.