# PQuAD: A Persian Question Answering Dataset

Kasra Darvishi, Newsha Shahbodagh, Zahra Abbasiantaeb, Saeedeh Momtazi*
Department of Computer Engineering
Amirkabir University of Technology (Tehran Polytechnic)

###### Abstract

We present Persian Question Answering Dataset (PQuAD), a crowdsourced reading
comprehension dataset on Persian Wikipedia articles. It includes 80,000
questions along with their answers, with 25% of the questions being
adversarially unanswerable. We examine various properties of the dataset to
show the diversity and the level of its difficulty as a MRC benchmark. By
releasing this dataset, we aim to ease research on Persian reading
comprehension and development of persian question answering systems. Our
experiments on different state-of-the-art pre-trained contextualized language
models shows 74.8% Exact Match (EM) and 87.6% F1-score that can be used as the
baseline results for further research on Persian QA.

Keywords: Machine Reading Comprehension - Natural Language Processing -
Persian Dataset - Question Answering

##  1 Introduction

Machine Reading Comprehension (MRC) is one of the central tasks in natural
language understanding which requires a system to read a passage and then
answer the given questions from the passage. Developing a machine that has the
comprehension ability helps Artificial Intelligence (AI) to reach the goal of
competing with human. Developing such a machine that can comprehend the
available data in the text format would be highly beneficial, because in many
jobs human workers are required to get information from text and having a high
performance MRC model will help to reduce manual tasks significantly.

MRC has undergone significant advancements in recent years due to the
availability of many large-scale annotated datasets, including MCTest
(Richardson et al., 2013), BookTest (Bajgar et al., 2016), SQuAD (Rajpurkar et
al., 2016), SearchQA (Dunn et al., 2017), NewsQA (Trischler et al., 2017),
ReCoRD (Zhang et al., 2018) and ReCO (Wang et al., 2020). These datasets
provide the opportunity to train data-intensive deep learning models and
achieve performance comparable to humans.

The main MRC datasets, however, are particularly in English, while low-
resource languages such as Persian require further efforts in this regard.
Recently, some progress has been made towards this goal, namely PersianQA
(Ayoubi, 2021) and ParSQuAD (Abadani et al., 2021), which are gathered
automatically by different methodologies. However, they either lack the
sufficient number of examples to be used in a supervised learning framework
for today’s large models or are not as high quality as manual ones. ParSQuAD
(Abadani et al., 2021) is collected by translating the SQuAD (Rajpurkar et
al., 2016) dataset into Persian by google-translate and PersianQA (Ayoubi,
2021) is a small size MRC dataset consisting of 10,000 questions, which is not
suitable for training the large models.

In this paper, we introduce PQuAD, a large-scale and high quality Persian
span-extraction MRC dataset consisting of 80,000 human-annotated questions.
PQuAD’s questions are based on Persian Wikipedia articles and cover a wide
variety of subjects. The dataset has been used for training the state-of-the-
art models in the field.

The rest of the paper is organized as follows: in Section 2, we briefly
overview related MRC datasets. Section 3 describes the data collection process
of PQuAD and methodologies used to ensure a high quality dataset. In Section 4
we analyze various properties of the dataset to show the level of its
difficulty as a MRC benchmark. Section 5 is devoted to introduction of
baseline models and their results on PQuAD. Finally, Section 6 concludes the
paper.

##  2 Related Works

The increasing attention of the researchers toward MRC has enhanced the
importance of providing the MRC datasets. Although a large number of MRC
datasets are available (Richardson et al., 2013; Bajgar et al., 2016;
Rajpurkar et al., 2016; Dunn et al., 2017; Trischler et al., 2017; Zhang et
al., 2018; Wang et al., 2020), it is still important to collect a MRC with new
features.

Recently, there has been a great motivation toward collecting the MRC datasets
for low-resource languages (Mozannar et al., 2019a; Carrino et al., 2020a;
d’Hoffschmidt et al., 2020; Lim et al., 2019). The availability of the multi-
lingual pre-trained language models, such as Multilingual BERT (Devlin et al.,
2019), XLM-RoBERTa (Conneau et al., 2020), can be considered as one of the
motivations behind this motivation. Most of these datasets, are collected from
wikipedia pages similar to SQuAD or are curated by translating the SQuAD
dataset.

MCTest (Richardson et al., 2013) is one of the earliest MRC datasets which
includes fictional stories and multiple choice questions collected for each
story. This dataset is very small and it is not sufficient for training large
models.

SQuAD 1.1 (Rajpurkar et al., 2016) is a large-scale MRC dataset which is
collected from wikipedia pages. Answer of each question is represented as a
span of text within the corresponding passage. In SQuAD 2.0, more than 50,000
unanswerable questions are added. The unanswerable questions are very similar
to the answerable questions and this feature makes the SQuAD 2.0 dataset a
more challenging dataset. Nevertheless, the available studies on MRC using
SQuAD 2.0 (Yamada et al., 2020; Yang et al., 2019; Joshi et al., 2020) have
achieved a great performance over human evaluation.

NewsQA (Trischler et al., 2017) is a large-scale dataset which includes
119,633 questions gathered by human workers from 12,744 CNN’s news articles.
Among the available studies on NewsQA dataset (Joshi et al., 2020; Tay et al.,
2018; Kundu and Ng, 2018), the SpanBERT model (Joshi et al., 2020) has
achieved a higher performance compared to the human judgment.

SearchQA (Dunn et al., 2017) is collected according to the procedure of
retrieving the correct answer in a real question answering system. Different
from other datasets which a question is posed given the passage, the question
is posed and then the relevant passages are retrieved for the given question.
SearchQA includes more than 140000 question-answer pairs and each question-
answer pair is linked to about 50 text snippets.

ReCoRD (Zhang et al., 2018) is collected automatically from news articles. The
distinctive feature of this dataset is that despite the other MRC datasets,
such as SQuAD and NewsQA, it needs commonsense reasoning over several
sentences for comprehending the answer. A small portion of the questions of
this dataset can be answered by paraphrasing.

There are several MRC datasets for other languages than English, including
ReCo (Wang et al., 2020) in Japanese, ParSQuAD (Abadani et al., 2021) in
Persian, FQuAD (d’Hoffschmidt et al., 2020) in French, ARCD (Mozannar et al.,
2019b) in Arabic, KorQuAD1.0 (Lim et al., 2019) in Korea, and Spanish
translation of SQuAD (Carrino et al., 2020b).

ARCD (Mozannar et al., 2019b) consists of Arabic translation of SQuAD and
questions posed by crowd workers. ParSQuAD (Abadani et al., 2021) in Persian
is also gathered by translating the SQuAD into Persian. For Spanish SQuAD
(Carrino et al., 2020b), a trained neural machine translation and a trained
unsupervised word alignment model are developed for automatically translating
the SQuAD dataset to Spanish. FQuAD (d’Hoffschmidt et al., 2020) is collected
from French Wikipedia pages. ReCO (Wang et al., 2020) is a very large opinion-
based MRC dataset including 300,000 Japanese questions which includes both
factoid and non-factoid questions.

Another type of the datasets is cloze datasets. In the cloze datasets, a word
is omitted from the text and goal is to detect the omitted word from the given
text. Children’s Book test (CBT) (Hill et al., 2015) dataset is a cloze
dataset. Each sample in this dataset is 21 consecutive sentences and one word
in the last sentence is omitted. The first 20 sentences are given as context
and the missing word in the next sentence must be predicted. BookTest (Bajgar
et al., 2016) dataset is very similar to the CBT dataset but it is designed
for training large models due to its 60 times larger size than CBT.

##  3 Dataset Collection

The data collection process of PQuAD consists of three stages: passage
curation, question-answer pair annotation, and additional answer collection.

###  3.1 Passage curation

The most important articles in the Persian Wikipedia are selected based on two
criteria: (1) having an info-box, and (2) being among the top pages in the
Persian Wikipedia based on the PageRank algorithm.

We believe that important Wikipedia pages, that have enough content for
question collection, usually contain an info-box and we use it as the primary
condition to retrieve 600,000 articles.

In the next step, we build a graph for these articles based on their links and
rank the collected articles using the NetworkX
library’s111https://networkx.org/ PageRank method with damping parameter set
to 0.4. Having the obtained set of ordered articles, for each article, we
choose the introduction section and the first 20 paragraphs, whose length is
500-1100 characters. The total number of paragraphs is 11,000, which belong to
1,125 articles spanning a wide range of topics.

###  3.2 Question-answer annotation

Crowdworkers were instructed to spend 15 minutes on each paragraph and collect
about five to ten questions as well as their answers, if they are answerable
using the paragraph’s content. At the same time, they discarded the paragraphs
that were hard to understand or contained too many non-Persian words. To
further regulate the process, we developed a toolkit for the annotation
process. In this toolkit, crowdworkers typed each question in a text box and
then either specified an answer by marking a span or labeled the question as
unanswerable. Workers were also encouraged to use paraphrased sentences in
their questions and avoid choosing the answers comprising non-Persian words.
Unanswerable questions roughly form 25% of all questions and were collected in
such a way that are in the same content of the paragraph, but they could not
be answered just by reading the paragraph.

###  3.3 Additional answer collection

To have a better evaluation of models, we further check the question-answer
pairs in the test and validation sets while additional answers are being
collected. A new group of crowdworkers answer the questions in these two sets.
The crowdworkers were encouraged to include the other correct answer spans
with minor differences to the original answers. Moreover, if they find that
the original answer is incorrect, they resolved the incorrect answer either by
choosing the more appropriate answer or removing the corresponding question
from the passage. The distribution of the number of answers per question is
represented in Figure 1.

Figure 1: Distribution of number of answers available for each answerable
question in test and validation
sets.1-answer2-answer3-answer-&more$1{,}000$$2{,}000$$3{,}000$$4{,}000$$4{,}319$$1{,}227$$542$$3{,}941$$1{,}722$$332$CountTest
setValidation set

##  4 Dataset Analysis

We investigate PQuAD in order to demonstrate the level of its challenge and
its various properties. Types of answers and some statistics about the
different domains available in the dataset, as well as general information,
are included in this analysis.

###  4.1 Dataset Structure

PQuAD is stored in the JSON format and consists of passages where each passage
is linked to a set of questions. The unanswerable questions are marked as
unanswerable and answer of the answerable questions is specified with answer’s
span. An example of a passage with corresponding question and answers is shown
in Figure 2.

Figure 2: An example of a passage with corresponding questions. The left part
is the original Persian data and the right part is the English translation of
that. The third question is an example of a question with no answer. Answers
are represented by the text of answer and the start token of the answer.

PQuAD includes 11,000 passages and 80,000 questions where each passage
includes about 7 questions. PQuAD is divided into three categories including
train, validation, and test sets. A detailed statistics of PQuAD is presented
in Table 1.

Table 1: Statistics of PQuAD dataset. | Train | Validation | Test | Total
---|---|---|---|---
Total Questions | 63994 | 7976 | 8002 | 79972
Unanswerable Questions | 15721 | 1981 | 1914 | 19616
Mean # of paragraph tokens | 125 | 121 | 124 | 125
Mean # of question tokens | 10 | 11 | 11 | 10
Mean # of answer tokens | 5 | 6 | 5 | 5

As mentioned before, PQuAD features a diverse range of topics. Table 2
categorizes topics based on the domains associated with the articles present
in the whole dataset.

Table 2: Diversity of article topics present in PQuAD and sample titles of source Wikipedia pages. Domain | Example | Percentage(%)
---|---|---
Person | Pablo Picasso | 22.3
Geographical Locations | Caspian Sea | 20.1
Science & Tech | Database | 6.5
Organization | United Nations | 6.0
Sports | Olympic Games | 5.8
Fields of Specialty | Psychology | 5.0
Plants & Animals | Starfish | 4.0
Art | Pop music | 3.3
Historical Eras | Precambrian | 3.2
Books & Movies | Star Wars | 2.9
Religious | God | 2.7
Events | Nowruz | 2.6
Groups | Vikings | 2.5
Languages | Middle Persian | 2.4
Chemistry & Biology | Hydrogen | 2.4
Astronomy | Solar System | 2.2
Diseases & Medicines |  COVID-19 pandemic | 2.0
Objects | Carpet | 1.6
Others | Immigration | 2.4

###  4.2 Answer types

We categorize answer spans of answerable questions of the validation set based
on their POS and NER tags, similarly to (Rajpurkar et al., 2016). The results
of categorization are shown in Table 3. As can be seen, the most popular types
of the answers are numerical, proper noun phrase, and common noun phrase
accounting for 24%, 37%, and 27% of all answers, respectively. The remaining
8.5% is distributed across adjective phrases, verb phrases, and other types.

Table 3: Categorization of validation set’s answers based on POS and NER tags. POS | NER | Percentage(%)
---|---|---
Numeric | Date | 10.2
Other Numeric | 13.4
Proper Noun Phrase | Person | 14.9
Location | 12.4
Group | 2.8
Organization | 2.3
Field | 1.1
Language | 1.3
Other Entity | 3.7
Common Noun Phrase | - | 27.0
Adjective Phrase | - | 1.9
Verb Phrase | - | 4.9
Other | - | 4.1

##  5 Experiments

###  5.1 Human performance

To estimate the human performance on our PQuAD dataset, we asked a new group
of crowdworkers to answer 1000 questions in the test set. These questions are
from 15 randomly selected articles and all questions of each article are
tasked to one of the workers. The percentage of the unanswerable questions in
the samples is the same as in the dataset. Workers were informed about the
existence of unanswerable questions and were asked to either mark the
questions for which they found no answer or specify the answer span in the
paragraph. Performance measures used in this experiment are F1 and Exact Match
(EM) metrics, the same as measures used by (Rajpurkar et al., 2016). The
estimated human performance on the test set is 88.3% for F1 and 80.3% for EM.

###  5.2 Models

We have evaluated PQuAD using two pre-trained transformer-based language
models, namely ParsBERT (Farahani et al., 2021) and XLM-RoBERTa (Conneau et
al., 2020), as well as BiDAF (Levy et al., 2017) which is an attention-based
model proposed for MRC.

  * •

ParsBERT: ParsBERT is a transform-based language model which uses the
architecture of BERTbase model. ParsBERT is specially developed for the
Persian language and is trained on Persian texts gathered from diverse
sources.

  * •

XLM-RoBERTa: XLM-RoBERTa is a transformer-based language model. Similar to the
mono-lingual RoBERTa language model, XLM-RoBERTa does not use the Next
Sentence Prediction (NSP) task for training and it is only trained using the
multilingual Masked Language Model (MLM). Size of the training data is very
large and it is capable of training by 100 different languages. XLM-RoBERTa is
a strong multi-lingual pre-trained language model, which has outperformed the
multilingual BERT (mBEERT) on several cross-lingual tasks.

  * •

BiDAF: BiDAF is a neural architecture which utilizes character-level, word-
level, and contextualized embedding for text representation. Attention
mechanism is used in the model for creating a query-aware representation of
the input context. The model detects the span of answer from context by
calculating the probability of being the begin and end token of the span for
each word within the context.

Table 4: Scores of baseline models and human performance. HasAns and NoAns columns show the scores over positive and negative questions respectively. Model | EM | F1 | HasAns_EM | HasAns_F1 | NoAns_EM/F1
---|---|---|---|---|---
BNA | 54.4 | 71.4 | 43.9 | 66.4 | 87.6
ParsBERT | 68.1 | 82.0 | 61.5 | 79.8 | 89.0
XLM-RoBERTa | 74.8 | 87.6 | 69.1 | 86.0 | 92.7
Human | 80.3 | 88.3 | 74.9 | 85.6 | 96.8

###  5.3 Results

Table 4 shows the performance of baseline models on the test set. According to
our experiments, mono-lingual BERT underperforms multi-lingual XLM-RoBERTa
which might be the result of a less effective pre-training process. Despite
the close F1 scores of XLM-RoBERTa and humans, there is a 5.5% gap between
their EM scores. Based on the detailed scores of answerable and unanswerable
questions, this lower performance is mostly due to the model’s inability to
select the shortest span containing the answer for positive questions. Using
the same model, and to have a better understanding of the model’s weak points,
we perform an analysis on the error rates based on answer types. As reported
in table5, the model has comparatively high performance on noun phrases and
numbers. However, differences between model and human performance are more
obvious in adjective and verb phrases. These types of answer spans are more
variable in their structures, especially in the Persian language, and
specifying the exact start and end position of them is a challenging task for
the model, suggesting a direction for future works.

Table 5: XLM-RoBERTa and human performance on different answer categories. Answer Type | XLM-RoBERTa | Human
---|---|---
EM | F1 | EM | F1
Numeric | 83.51 | 92.64 | 86.9 | 92.7
Proper Noun Phrase | 78.4 | 88.7 | 78.6 | 86.2
Common Noun Phrase | 70.5 | 88.0 | 71.7 | 84.9
Adjective Phrase | 57.1 | 79.5 | 85.7 | 85.7
Verb Phrase | 53.3 | 83.4 | 62.0 | 84.7
Other | 62.3 | 76.5 | 68.7 | 76.4

##  6 Conclusion

In this paper, we present PQuAD, a Persian reading comprehension dataset
consisting of 80,000 questions. This dataset is based on Wikipedia articles
and its question-answer pairs are annotated by humans. Since unanswerable
questions are also included in PQuAD, models are required to abstain from
answering whenever no proper answer span is present in the given context. By
releasing this dataset, we aim to ease research on Persian reading
comprehension and the development of Persian question answering systems.

## References

  * Abadani et al. (2021) Abadani N, Mozafari J, Fatemi A, Nematbakhsh MA, Kazemi A (2021) Parsquad: Machine translated squad dataset for persian question answering. In: 2021 7th International Conference on Web Research (ICWR), IEEE, pp 163–168
  * Ayoubi (2021) Ayoubi MY Sajjad & Davoodeh (2021) Persianqa: a dataset for persian question answering. https://github.com/SajjjadAyobi/PersianQA
  * Bajgar et al. (2016) Bajgar O, Kadlec R, Kleindienst J (2016) Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:161000956
  * Carrino et al. (2020a) Carrino CP, Costa-jussà MR, Fonollosa JAR (2020a) Automatic Spanish translation of SQuAD dataset for multi-lingual question answering. In: Proceedings of the 12th Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, pp 5515–5523, URL https://aclanthology.org/2020.lrec-1.677
  * Carrino et al. (2020b) Carrino CP, Costa-jussà MR, Fonollosa JAR (2020b) Automatic Spanish translation of SQuAD dataset for multi-lingual question answering. In: Proceedings of the 12th Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, pp 5515–5523, URL https://aclanthology.org/2020.lrec-1.677
  * Conneau et al. (2020) Conneau A, Khandelwal K, Goyal N, Chaudhary V, Wenzek G, Guzmán F, Grave E, Ott M, Zettlemoyer L, Stoyanov V (2020) Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, pp 8440–8451, DOI 10.18653/v1/2020.acl-main.747, URL https://aclanthology.org/2020.acl-main.747
  * Devlin et al. (2019) Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, pp 4171–4186, DOI 10.18653/v1/N19-1423, URL https://aclanthology.org/N19-1423
  * d’Hoffschmidt et al. (2020) d’Hoffschmidt M, Belblidia W, Brendlé T, Heinrich Q, Vidal M (2020) Fquad: French question answering dataset. arXiv preprint arXiv:200206071
  * d’Hoffschmidt et al. (2020) d’Hoffschmidt M, Belblidia W, Heinrich Q, Brendlé T, Vidal M (2020) FQuAD: French question answering dataset. In: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, pp 1193–1208, DOI 10.18653/v1/2020.findings-emnlp.107, URL https://aclanthology.org/2020.findings-emnlp.107
  * Dunn et al. (2017) Dunn M, Sagun L, Higgins M, Guney VU, Cirik V, Cho K (2017) Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:170405179
  * Farahani et al. (2021) Farahani M, Gharachorloo M, Farahani M, Manthouri M (2021) Parsbert: Transformer-based model for persian language understanding. Neural Processing Letters 53(6):3831–3847
  * Hill et al. (2015) Hill F, Bordes A, Chopra S, Weston J (2015) The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:151102301
  * Joshi et al. (2020) Joshi M, Chen D, Liu Y, Weld DS, Zettlemoyer L, Levy O (2020) Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics 8:64–77
  * Kundu and Ng (2018) Kundu S, Ng HT (2018) A question-focused multi-factor attention network for question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 32
  * Levy et al. (2017) Levy O, Seo M, Choi E, Zettlemoyer L (2017) Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:170604115
  * Lim et al. (2019) Lim S, Kim M, Lee J (2019) Korquad1. 0: Korean qa dataset for machine reading comprehension. arXiv preprint arXiv:190907005
  * Mozannar et al. (2019a) Mozannar H, Hajal KE, Maamary E, Hajj H (2019a) Neural arabic question answering. arXiv preprint arXiv:190605394
  * Mozannar et al. (2019b) Mozannar H, Hajal KE, Maamary E, Hajj H (2019b) Neural arabic question answering. arXiv preprint arXiv:190605394
  * Rajpurkar et al. (2016) Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, pp 2383–2392, DOI 10.18653/v1/D16-1264, URL https://aclanthology.org/D16-1264
  * Richardson et al. (2013) Richardson M, Burges CJ, Renshaw E (2013) MCTest: A challenge dataset for the open-domain machine comprehension of text. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Seattle, Washington, USA, pp 193–203, URL https://aclanthology.org/D13-1020
  * Tay et al. (2018) Tay Y, Tuan LA, Hui SC, Su J (2018) Densely connected attention propagation for reading comprehension. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp 4911–4922
  * Trischler et al. (2017) Trischler A, Wang T, Yuan X, Harris J, Sordoni A, Bachman P, Suleman K (2017) NewsQA: A machine comprehension dataset. In: Proceedings of the 2nd Workshop on Representation Learning for NLP, Association for Computational Linguistics, Vancouver, Canada, pp 191–200, DOI 10.18653/v1/W17-2623, URL https://aclanthology.org/W17-2623
  * Wang et al. (2020) Wang B, Yao T, Zhang Q, Xu J, Wang X (2020) Reco: A large scale chinese reading comprehension dataset on opinion. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, pp 9146–9153
  * Yamada et al. (2020) Yamada I, Asai A, Shindo H, Takeda H, Matsumoto Y (2020) LUKE: Deep contextualized entity representations with entity-aware self-attention. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, pp 6442–6454, DOI 10.18653/v1/2020.emnlp-main.523, URL https://aclanthology.org/2020.emnlp-main.523
  * Yang et al. (2019) Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV (2019) Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32
  * Zhang et al. (2018) Zhang S, Liu X, Liu J, Gao J, Duh K, Van Durme B (2018) Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:181012885