# Cloning Ideology and Style using Deep Learning

Dr. Omer Beg, Muhammad Nasir Zafar
NUCES,Pakistan
Islamabad
{omer.beg<EMAIL_ADDRESS>
Waleed Anjum
NUCES,Pakistan
Islamabad
<EMAIL_ADDRESS>

###### Abstract

Text generation tasks have gotten the attention of researchers in the last few
years because of their applications on a large scale. In the past, many
researchers focused on task-based text generations. Our research focuses on
text generation based on the ideology and style of a specific author, and text
generation on a topic that was not written by the same author in the past. Our
trained model requires an input prompt containing initial few words of text to
produce a few paragraphs of text based on the ideology and style of the author
on which the model is trained. Our methodology to accomplish this task is
based on Bi-LSTM. The Bi-LSTM model is used to make predictions at the
character level, during the training corpus of a specific author is used along
with the ground truth corpus. A pre-trained model is used to identify the
sentences of ground truth having contradiction with the author’s corpus to
make our langugae model inclined. During training, we have achieved a
perplexity score of 2.23 at the character level. The experiments show a
perplexity score of around 3 over the test dataset.

_K_ eywords Bi-LSTM $\cdot$ Perplexity $\cdot$ Pre-Trained

##  1 Introduction

In this age, content becomes a highly demanding thing because it plays a very
important role to attract users in different cases. In this research, text
content is focused. Many articles are written in different magazines,
newspapers, and websites on daily basis. Every article is written by an author
which is based on his ideology and style of the author. Each author writes
according to his unique different point of view from other authors. On other
hand, every reader has his own taste and likes to read some article written by
his favourite author due to his style and ideology. And each reader wants to
read more and more material from his favourite author and want to know his
point of view on different more topics. Moreover, now this world becomes the
world of devices because everyone has computational devices in the
pocket[1].That’s why day to day it needs to be much and more content to keep
users active. But quality text generation is a slow and costly process. Text
generation using machines becomes very important to generate text content
fastly with low expenses[2]. To generate text which is task-based requires
labelled data about the current task and this is done using a knowledge-based
graph.This graph-based on ground reality which is used to develop a common
sense in the model .This task seems to be very difficult because the goal is
to make a model which is generalized in a way that can generate the text on a
new task on which the model is not trained[3]. Some models are available which
are capable to generate the report or text from structured data e.g generate
weather reports from structured data about weather.Fig1 shows the text
generation mechanism when initial text is given[4].
This research’s objective is to develop the computational intelligence which
can understand the written text of the author to capture ideology and style
and be capable to produce the text on a new topic with the same ideology and
style. A few words of the sentence is input to the model and the model’s
output would be some paragraphs of text continued to these input words.
Understanding the already written text and producing new meaningful text
requires a generic type of intelligence [5].

Figure 1: Example Scenario

In literature, many techniques are proposed, but they got stuck on several
points e.g how to deal with abstraction in text and use of limited words in
the training corpora.The following Fig1 shows an example scenario. Language
models assign a greater probability to the sequences which have a high
frequency of occurring in the training corpus but this thing leads to stuck in
the circle of specific words, as shown in Fig1.To evaluate the quality of
generated text from in creativity’s perspective human evolution is used
because no model or evaluation metric is here to check creativity in the
text[6].
It needs much research work and struggles to capture the ideology of a
specific author and style.The main goal of our research includes the use of
new words which are not used in the training corpus but are required to
explain the new topic on which model have to generate text.The following Fig2
shows the working of language model.

Figure 2: Working of Language Model

The only way we have to vector embeddings for this type of words take a ground
truth text’s corpus and generate the vector embedding from here. As our
purpose is to get the semantics of the word to use it in newly generated text
correctly. So, it does not contribute to decreasing the performance of the
model[7]. This research also focuses on how unknown words can be handled using
the nearest neighbour technique. To generate the candidate vocabulary for
specific author k-nearest neighbour words are taken of a word that is part of
the training corpus[8]. After taking a cluster of these words, conditional
probability is calculated against each word and words having lower conditional
probabilities are excluded from the candidate vocabulary list.
In this digital world, a huge amount of text data exists and is produced on
daily basis, to manipulate this text data requires human power which becomes
very slow and costly[9].

###  1.1 Contributions

This section contains the major contributions of our proposed approach. The
state-of-the-art language models are very successful to generate meaningful
texts but they are based on the ground reality or knowledge embedded in the
given training corpus. As, the ground reality is the same for everyone, but
everyone has their thoughts and way of thinking. And everyone speaks and
writes based on their perception and ideology. Our research focused on this
type of language model based on ground reality and authors perceptions and
thoughts as well to generate text based on the style and ideology of specific
authors. The two major contributions of our proposed approach are as follows.

  * •

Generating a language model based on ground truth reality and inclined towards
given training corpus of the author to generate specific ideology and style
based text.

  * •

Overriding the weights(probability) values with the weight obtained against a
sequence in the author’s corpus if there is a clash, in case of entailment
weights are just refined.

  * •

Extraction of most probable vocabulary words which are not included in
author’s training corpus, but need to explain new aspects during the text
generation process. And refining the appropriate weights against sequences in
which they occur.

##  2 Related Work

In previous work many authors shows how different techniques can work to
generate the text with track of current context.Here, some of them are
discussed sequentially.Neural text generation is a seq2seq model[10].These
models generate the text by sampling words, generating next word based on
previous word to maintain the track of context.These models are validated
using perplexity measure by missing a word from a sentence[11].Generative
Adversarial Networks are used to improve quality of the samples.Actor-critic
conditional GAN are introduced by filling the missing text based on
surrounding context[12]. Likelihood generation of text leads to dull and
repeti- tive text because model assigns the very high probability to sequences
which repeats themselves in the training corporus [13].In this paper author
shows how unlikelihood method is used to generate the better text which
compar- atively less dull and contains non repeating sentences.

In next paper, its shown that how BLEURT, a learned evaluation metrics based
on BERT that can model human judgements by providing the few thousands
possibly biased examples. Generative Adversarial Nets are used to generate
meaningful text by using discriminative models to guide the training of
generative model as reinforcement learning[14].In this paper a new framework
named as LeakGAN to 1 address problem in generation of long
text[15].Discriminative model is allowed to leak its own high level extracted
features to generative models.It also improves performance in short text
scenarios.The following papers puts effort to develop common sense using
knowledge based common sense graph.

In-text generation tasks a lot of researchers focus on task-based text
generation but becomes costly if have to generate the text on the new
task[16]. It is difficult to rely on a significant amount of data for each
task because it is costly to acquire[17]. Text generation can be done in many
different ways and each way has its applications[18]. If we talk about task-
based text generation it is done for the specific domain and it can be used as
an application in different domains until we have a significant amount of
data, which is very costly to acquire[19]. Web crawling can be used for
knowledge grounded pre-training to generate text from data. Sometimes, it is
required to generate text, including some specific words this technique is
named as mention flags[20]. The mentioned flags models are trained to generate
the tokens of text, and the goal is to satisfy the all constraints[21]. This
model takes words required to satisfy the constraint, as input and these words
are passed to the encoder and these encodings passed to the decoder, decoder’s
task is to generate outputs against it and this decoder’s output passed to the
next iteration as input to generate next output and so on[22][23].

Transformers are the latest language models used in text generation[24].
Before the invention of transformers LSTM and RNN language modelling was
considered the best choice for text generation[25]. But introducing the
mechanism of attention for the sequence to sequence modelling invents the
language model name as Transformer[26]. It contains an encoder and decoder
mechanism along with an attention mechanism. Attention mechanism forces
decoder to pay attention to encoder where needed[27].

Here, we summarize the literature into tabular form, by, dataset used in
research. In the following table1 we summarize the literature review of 7
papers, each paper is cited as well. In the above paragraphs, we have
explained these all papers in the detail.

Paper | Dataset
---|---
Wang et al. 2021[28] | Wikipedia Web
Yermakov, Drago, and Ziletti 2021[13] | BioLeaflets
Topal, Bas, and van Heerden 2021[11] | Web Crawling
Chen et al. 2020[5] | Web NLG
Ji et al. 2020[9] | Web NLG
Raffel et al. 2019[7] | RealNews Dataset
Zeng, Shoeybi, and Liu 2020[3] | Wikiperson
Table 1: Summary of Papers along with dataset

The following table shows the summary of 7 different papers along with the
techniques used.

Paper | Technique
---|---
[28] | Mention Flags
[13] | Fine-Tune Trans
[11] | Attention Mechanism
[5] | KGPT
[9] | CNN+GNN+Trans
[7] | Unified Framework
[3] | Encoder+Decoder
Table 2: Summary of Papers along with Techniques

The previous research on this topic shows how structured data can be used to
generate the text or report of a specific domain and task[29][30]. Some of the
researchers show how a sentence can be generated when previous sentences are
given and the goal is to generate the sentence in the same context[31]. In
literature, a lot of work is done on constraints based text generation e.g
generated text must consist of some given words [32]. This work is extended to
text generation based on content matching and style based text generation
where the model generates the text which is the same in style as reference
text[33][34].

Our research focuses on how creative text can be generated by giving a few
starting words of the sentence as input to the model, by training it on a
ground knowledge-based graph and corpora consisting of articles already
written by the author to capture the ideology and style of a specific author.
In other words, our goal is to capture ideology and style from already written
topics from an author and generate new text based on the same ideology and
style. Text generated by the model will be the same in context with a given
input, specific to a topic and continued to some lines or paragraphs. The
second thing is how unknown words(not present in the training corpus of the
author) can be used to explain the new topic i.e finding the correct
dimensional values of new words and their place in generated sentences. It
includes the extraction of the probable vocabulary of the author and refining
it to get the list of guessed vocabulary of the author.
It is expensive and time-consuming to write articles based on specific
ideologies and styles. It becomes difficult to maintain balance in demand and
supply, as it requires human intelligence. Our research includes exploration
of techniques, which can be used to extract the ideology and style of a
specific author and write text in some different contexts with the same
ideology and style.

##  3 Text Generation using LSTM-based Language Model

Pre-trained language models acquired the probabilities values over a sequence
of words based on ground truth or training corpora on which they have been
trained. As, language models can capture the preferences and realities, so
language models got inclined toward these realities[35]. In our proposed
methodology, we will be using LSTM based, neural language model. Then training
this language model on the training corpora of the author, training of the
language model is based on the predicting the next character when a sequence
of characters having specific lenth is given[36]. The training of the model on
the training corpus of the auther will make model to extract the idelogy and
the style of the author and then use it in text generation.The chracter based
training may lead to generating some non-dictionary words some time but it
plays very important role in extracting the ideology and style of the
author[37]. After training on the author’s corpus, second step is to train the
language model on the ground truth corpus to embbed ground realities in it.
But if some ground truth have contradiction with the author’s ideology, it
will be excluded during training. If there is entailment in both, then text
chunk of ground truth will go for training or even neutral[38]. To check
contradiction we will be using a pre-trained model named as ’roberta-large-
mnli’ taken from www.huggingface.co. After contradiction check, text chunks of
the ground truth goes for training same as text chunks of author’s corpus
went. Its is done to generate the languge model inclined towards the idelogy
and style of the author.

As, there are huge chances that appropriate words to explain the new topic may
be missing in the training corpora of the author. Out next step is to find the
appropriate words to explain the new topic which are probably part of the
author’s vocabulary. To extract the most probable vocabulary words from the
author, the first step is to take the dictionary from NLTK. So, using the NLTK
dictionary words which are missing in the author’s corpus can be extracted
easily. But problem is that NLTK dictionary is much lengthy, there will be
very large list of the missing words. Then, most appropiate words are
extracted from them. During extraction of the appropiate words, it assumed
that all stop words can be vocablory of the author.

So, all stop words can be taken. Now, we have to find the appropate text
chunks including these stop words one by one, but taking these chunks from
ground truth or any web page may lead to cause contradiction in ideology. So,
we take NLTK book to get text chunks for training beacuse these chunks are
neural with author’s corpus, although does not have entailment but not
contradiction as well.

Figure 3: System Architecture

All stope words are taken, because we are just considering that all stop words
need to explain any topic or to generate any new text. There may be chances
that author’s corpus already includes the all stop words . The above Fig3
shows the system architecture.The following Algorithm1 shows how contradiction
is checked between author and ground facts.

Algorithm 1 Algorithm for checking Contradiction

Input: Text Chunk of Ground Truth Corpus
Output: Contradicted or Not

1: Let t is a Threshold

2: if Text Chunk in Bin then

3: Return

4: end if

5: while Auther’s Text(Used) Not Ends do

6: Get Prediction from Pre-Trained Model

7: if Contradiction Value>=t then

8: Append Text Chunk to Bin

9: Return True

10: else

11: Continue

12: end if

13: end while

14: return False

The Fig3 shows system architecture, training corpus of the specific author
crawled from the web is passed to three different pipelines. One pipeline is
used to train lstm, second is connected with pipe line of ground truth corpus
at pretrained model taken from ’huggingface.co’. This pre-trained model
enables to check contradition between both corpuses. The third one is used to
extract missing words. All three pipelines, are going into lstm mode. After
training lstm model, it is able to generate new meaningful text.

###  3.1 LSTM Unit

In this section working of the LSTM unit is explained. There are three vector
inputs to the LSTM unit, the first represents an input character at a
timestamp (t) and the second is a hidden state obtained at a timestamp
(t-1)[39]. The input vector size is 100 because we are handling 100 characters
in this problem during the training of model and text generation.The size of
hidden state vector is also 100 in case of simple LSTM and becomes 200 in case
of Bi-LSTM[40]. The third vector represents the memory obtained from all
previous iteration, and the size of this vector is same as hidden state
vector.The concatenated vector used as input, and sigmoid as activation
function in forget gate, input gate and output gate[41].The following Eq1,Eq2
and Eq3 shows how output vector of forget gate, input gate and output gate
calculated respectively. Here w represents matrice of weights for each gate.

$f_{t}=\sigma(w_{f}[h_{t-1},x_{t}]+b_{f})$ (1)

$i_{t}=\sigma(w_{i}[h_{t-1},x_{t}]+b_{i})$ (2)
$o_{t}=\sigma(w_{o}[h_{t-1},x_{t}]+b_{o})$ (3)

To obtained candidate cell state or memory tanh function is used.The following
Eq4 shows how candidate cell state is obtained using concatenated input
vector, Eq5 shows how cell state is obtained for timestamp(t).

$c^{-}_{t}=tanh(w_{c}[h_{t-1},x_{t}]+b_{c})$ (4)
$c_{t}=f_{t}*c_{t-1}+i_{t}*c^{-}_{t}$ (5)

The following Eq6 is used to calculate the hidden state for timestamp(t).

$h_{t}=o_{t}*tanh(c_{t})$ (6)

After the obtaining hidden state, this hidden state is mapped on output layer
of size 100.This output vector represent a single character, predicted by the
model.A loss function is explained in the next section, used to refine the
weights of model.

###  3.2 Loss Function

Our model is based on loss function of cross entropy loss.As, our model is
based on making predictions at character level so loss is computers between
predicted character and actual character which model ideally should
predict.There is one hot encoded vector against all character.It makes working
of cross entropy loss perfectly. The following Eq7 shows the formula for
cross-entropy loss.

$Loss=-\sum_{c=1}^{N}y_{o,c}\log(p_{o,c})$ (7)

In the above equation N is the number of total classes. In our case, each
character represents a uniques class and has a unique hot encoded vector
against it. So, we can say that N is equal to the number of characters. Each
index-value of a hot encoded vector having a length of N is put into equation
then summing up all give total loss against predicted character and actual
output character.

##  4 Results and Disscussion

To evaluate the model there are different evaluation metrics, as our model is
working on character-based predictions so we will evaluate accordingly.

###  4.1 Overview of Evaluation Metrics

To evaluate how much a model is good in extracting and writing with a specific
ideology and style is not an easy task. Because it requires a very general
type of intelligence. But there are several evaluation metrics e.g. loss and
perplexity scores, that can be used for evaluation we are going to discuss one
by one as follows.

The following equation 8 shows how loss is computed.

$Loss=-\sum_{c=1}^{N}y_{o,c}\log(p_{o,c})$ (8)

We would like to explain how it will be used during the evaluation. We already
used it during training, during evaluation we will generate the output against
input, and the model will generate the output against it. The output generated
will be compared with the actual output character by character. The loss will
be computed character by character and summing up all will be given total
loss.

The following equation 9 shows how perplexity is computed over a sentence.

$PP(S)=P(w1,w2,...,wn)^{-1/n}$ (9)

So, it’s very easily understandable that high perplexity score is not in the
favour of the model. The model’s parameters should be adjusted in such a way
that its perplexity score is reduced. We will visualize the perplexity scores
during training and then during testing by conducting different experiments.

###  4.2 Training Results

The major part of model training consists of training the model on a corpus of
author and corpus of ground truth. The following section describes the results
during training.

####  4.2.1  Perplexity vs Loss

The training of the model on the author’s and ground truth corpus consists of
around 24900 epochs. After a specific number of epochs, we compute the average
of the specific epochs’ loss and perplexity scores. In this section, we plot
the graph between perplexity score and loss to visualize the relationship
between them. As we discussed earlier the loss and perplexity score shows the
model’s performance in the same manner. Decreasing the loss of a model is in
favor of the model in the same way decreasing the perplexity also indicates
the better performance of the model.

The following Fig4 shows the graph between perplexity score and loss. The
following fig4 shows that there is a strong linear relationship between
perplexity score and the loss value and the nature of the relationship is
positive.

Figure 4: Training Results

Almost, 90 percent of the total number of epochs there is a very straight and
linear line. But during the remaining 10 percent of the epochs decrement in
perplexity score becomes slower but not in loss value that why the slope
during these epochs seems to be greater.

###  4.3 Results over Test Dataset

To evaluate the inclination of the model toward the ideology and style of a
specific author requires a general type of intelligence, here we used
perplexity score over a test set of the same author and different authors to
evaluate.

####  4.3.1 Inclination of the Model towards Ideology and Style

In this section, we are going to evaluate the model from the perspective that
how much the model is inclined towards the ideology and style of an author
whose corpus is used in training. It is done by comparing perplexity scores
obtained over the test dataset of the same author and different authors. As,
the model is not able to manipulate the words through semantics, but can keep
the track of probability of a character being next after a sequence of
characters through perplexity score. So, the same ideology and style will try
to reduce perplexity scores and vice versa.The following Fig5 shows diffferent
experiements, each experiment is performed on a text chunk of same auhtor’s
test set and a text chunk from different author.

Figure 5: Same author vs Different author(Test Set)

In fig5 it can be seen that most of experiment shows higher perplexity against
different author becasue model is inclined towards ideology and style of a
specific author when input text from corpus of different author is given to
model, model tries to predict the characters according to idelogy and style of
same author but perplexity is calculated according to different auhtor’s
corpus, due to contradiction in idelogy and style of both authors perplexity
scores goes higheir.In case of same author, model predictions and acutal text
have much similer ideology and style due to which perplexity scores are goes
down.

####  4.3.2 Loss and Perplexity over Testset

In this section, we are going to visualize and explain the loss and perplexity
scores over the test dataset with help of different experiments. The following
Fig6 shows the comparison of loss and perplexity scores against each of the
five experiments.

Figure 6: Comparison of Loss and Perplexity

The above graph shows the loss and perplexity score over the test dataset
against 5 different experiments. It can be seen that in each experiment both
values seem to be very close to each other which shows that model is
consistent in predicting the text in different experiments. In each
experiment, different text chunks are taken from the test dataset, and it
means that selecting the text chunk from the test dataset as input to the
model does not matter for perplexity scores and loss value. In experiment no1
value of the loss is just above the 1.5 and the perplexity score is just above
the 3. It means that having the loss of 1.5 defines the model as having 3
options averagely at each point of prediction. As the ideal case of
perplexity, the score is having a value of 1, but it is very hard to achieve.
That means the loss value should be reduced to around 0.2 to have a perplexity
score of 1. Our model is based on character level predictions, it causes the
non-dictionary words generation whenever the perplexity score is greater than
1. In case of having a perplexity score value of 1, it will finish the chances
of predicting non-dictionary words. In experiment no3 loss value is the
maximum which is 1.6 and the perplexity score for this experiment is 3.3, but
it’s not much greater than the others.

####  4.3.3 Non-Dictionary Words as Predictions

In this section, we going to discuss the percentage of non-dictionary words
predicted by our model. In the following Fig7, the graph shows the percentage
of non-dictionary words predicted by our model.

Figure 7: Percentage of Non-Dictionary Words

The above graph shows that in experiment no1 percentage of non-dictionary word
prediction is just above the 7. In other experiments, this percentage is
between 6 and 7. In the previous section, we briefly explain the cause of non-
dictionary word prediction and explain how much work is required to reduce
this percentage to zero.

####  4.3.4 Analysis and Discussion

In this section, we will analyze and discuss the results of our model on the
test dataset. The text generated by the model against the sequence of words
given to the model as the input seems to be based on the ideology and style of
the author. To analyze the generated output requires a comprehensive study of
the training corpus of the author.However, the inclination of the model
towards ideology and the style of the specific author can be analyzed by
comparing the perplexity scores on a test set of the same author and different
authors. In section 4.3.1,graph in fig5 shows that the perplexity score on a
test set of the same author is less than a different author in most of the
experiments,it happened because when the model is tested on a different
author’s test set there is a difference between style and ideology, model is
trying to predict and actual style and ideology(model should predict). That is
the reason the perplexity score goes up.On the other hand, perplexity scores
for test set of same auhtor are less becasue style and ideology, model is
trying to predict and model should predict is same. That’s why perplexity
scores goes down in this case.So, our trained model is good at capturing style
and ideology of specific auhtor.

###  4.4 Comparison with other Models

In this section, we are going to explain the results generated by other models
on our test dataset by comparing each other. The models we used for comparison
are based on RNN, LSTM(unidirectional), RNN+LSTM, and Transformers. The
following Table 3 shows the perplexity scores generated by the mentioned
models over the test dataset.

Model | Perplexity
---|---
RNN | 3.80
LSTM(Uni) | 3.20
RNN+LSTM | 2.73
Transformer | 1.67
Table 3: Comparison with other Models

The above table shows the performance of each model as perplexity scores.In
the table, the loss is not mentioned but is explained in this paragraph. Now,
we will discuss them one by one. First of all, there is a Recurrent Neural
Network(RNN) based neural language model. It is trained on the same dataset
and tested against the same test dataset. This model achieved a minimum of
1.95 as a loss value against which the perplexity score is 3.80. The second
model is Long Short Term Memory(LSTM-Uni), which achieves a minimum of 1.79
loss value and perplexity score of 3.20. The third neural language model is
based on the RNN and LSTM. It gives a minimum loss value of 1.65 and 2.73 as
perplexity values. The last one is the Transformer based language model which
achieves a loss value of 1.29 and a perplexity score of 1.67. In the
comparison table, it can be seen that Transformer outperforms, but if we
compare this model with our model although the perplexity score achieved by
this model is better in the extraction of ideology and style this model does
not seems to be good because of word-level predictions. So, we can say that
Bidirectional LSTM outperforms overall in the context of ideology and style.

##  5 Limitations

This research work does not focus on generating only meaningful words, due to
character-level predictions sometimes the model generates non-dictionary
words.It happened due to perplexity score is higher than 1. It is hard to
reduce the perplexity scores of a model to 1, but it grantee the generation of
only dictionary words. The second limitation of this research is a model only
can digest a limited amount of ground facts dataset in a reasonable amount of
time during training.The main reason of model to digest a limited amount of
ground facts dataset is contradiction check between ground facts and author’s
corpus.Because before sending each text chunk of ground facts datset to the
model for training, it sent to pre-trained model for contradiciton check with
chunks of author’s datset.It is done for every chunk of ground facts datset
that’s why model is limited to digest small volume of ground facts datset.

##  6 Conclusion

Our thinking about any individual is based on their ideology, even we can
predict the response of anyone we used to talk. This is because of training of
our mind on the previous response of the same individual against different
actions. In the same way, ideology and responses can be extracted from text
using a neural language model, after extraction this ideology and style new
text can be generated against a sequence of input words. Our model ables us to
extract the ideology and style of a specific author by training the model on a
corpus of the same author. To generate text on ground truth, a model needs to
train on ground truth corpus as well but only those chunks are used which have
no contradiction with the author’s corpus because the goal is to generate a
model inclined towards the ideology and style of the author. To stop
generating non-dictionary words loss should be reduced to around 0.23 to have
a perplexity score of 1. This will make the model ables to predict only
dictionary words.

##  7 Future Work

This works gives two different ways for future research. The first one is how
the model can be trained on the maximum ground truth corpus without the
inclination of the model towards it. It can be done with a logical
contradiction check with feasible requirements of computational power. The
second one is reducing the model’s loss to 0.23 to make perplexity 1. As it
will make a model generate only dictionary words.

## Acknowledgement.

This work was supported by AIM Lab, National University of Computer and
Emerging Sciences.

## References

  * [1] William Fedus, Ian J. Goodfellow, and Andrew M. Dai.  Maskgan: Better text generation via filling in the _______.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  * [2] Raza Ali, Umar Farooq, Umair Arshad, Waseem Shahzad, and Mirza Omer Beg.  Hate speech detection on twitter using transfer learning.  Computer Speech & Language, 74:101365, 2022.
  * [3] Kuo-Hao Zeng, Mohammad Shoeybi, and Ming-Yu Liu.  Style example-guided text generation using generative adversarial transformers.  arXiv preprint arXiv:2003.00674, 2020.
  * [4] Muzammil Hussain Shahid, Muhammad Arshad Islam, and Mirza Beg.  Exploiting time series based story plot popularity for movie success prediction.  Multimedia Tools and Applications, pages 1–26, 2022.
  * [5] Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang.  Kgpt: Knowledge-grounded pre-training for data-to-text generation.  arXiv preprint arXiv:2010.02307, 2020.
  * [6] Adil Majeed, Mirza Omer Beg, Umair Arshad, and Hasan Mujtaba.  Deep-emoru: mining emotions from roman urdu text using deep learning ensemble.  Multimedia Tools and Applications, pages 1–26, 2022.
  * [7] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  arXiv preprint arXiv:1910.10683, 2019.
  * [8] Sibgha Anwar, Mirza Omer Beg, Kiran Saleem, Zeeshan Ahmed, Abdul Rehman Javed, and Usman Tariq.  Social relationship analysis using state-of-the-art embeddings.  Transactions on Asian and Low-Resource Language Information Processing, 2022.
  * [9] Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, and Minlie Huang.  Language generation with multi-hop reasoning on commonsense knowledge graph.  arXiv preprint arXiv:2009.11692, 2020.
  * [10] Muhammad Farrukh Bashir, Abdul Rehman Javed, Muhammad Umair Arshad, Thippa Reddy Gadekallu, Waseem Shahzad, and Mirza Omer Beg.  Context aware emotion detection from low resource urdu language using deep neural network.  Transactions on Asian and Low-Resource Language Information Processing, 2022.
  * [11] M Onat Topal, Anil Bas, and Imke van Heerden.  Exploring transformers in natural language generation: Gpt, bert, and xlnet.  arXiv preprint arXiv:2102.08036, 2021.
  * [12] Sadia Ismail, Hasan Mujtaba, and Mirza Omer Beg.  Spems: A sustainable parasitic energy management system for smart homes.  Energy and Buildings, 252:111429, 2021.
  * [13] Ruslan Yermakov, Nicholas Drago, and Angelo Ziletti.  Biomedical data-to-text generation via fine-tuning transformers.  In Proceedings of the 14th International Conference on Natural Language Generation, pages 364–370, Aberdeen, Scotland, UK, August 2021. Association for Computational Linguistics.
  * [14] Hamza Mustafa Alvi, Hammad Majeed, Hasan Mujtaba, and Mirza Omer Beg.  Mlee: Method level energy estimation—a machine learning approach.  Sustainable Computing: Informatics and Systems, 32:100594, 2021\.
  * [15] Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen.  Towards faithful neural table-to-text generation with content-matching constraints.  arXiv preprint arXiv:2005.00969, 2020.
  * [16] Muhammad Saad Javed, Hammad Majeed, Hasan Mujtaba, and Mirza Omer Beg.  Fake reviews classification using deep learning ensemble of shallow convolutions.  Journal of Computational Social Science, 4(2):883–902, 2021.
  * [17] Hammad Majeed, Abdul Wali, and Mirza Beg.  Optimizing genetic programming by exploiting semantic impact of sub trees.  Swarm and Evolutionary Computation, 65:100923, 2021.
  * [18] Mubashar Nazar Awan and Mirza Omer Beg.  Top-rank: a topicalpostionrank for extraction and classification of keyphrases in text.  Computer Speech & Language, 65:101–116, 2021.
  * [19] Saira Qamar, Hasan Mujtaba, Hammad Majeed, and Mirza Omer Beg.  Relationship identification between conversational agents using emotion analysis.  Cognitive Computation, pages 1–15, 2021.
  * [20] Sara Afzal, Muhammad Asim, Abdul Rehman Javed, Mirza Omer Beg, and Thar Baker.  Urldeepdetect: A deep learning approach for detecting malicious urls using semantic vector models.  Journal of Network and Systems Management, 29(3):1–27, 2021.
  * [21] Abdul Rehman Javed, Raza Faheem, Muhammad Asim, Thar Baker, and Mirza Omer Beg.  A smartphone sensors-based personalized human activity recognition system for sustainable smart cities.  Sustainable Cities and Society, page 102970, 2021.
  * [22] Talha Anwar and Omer Baig.  Tac at semeval-2020 task 12: Ensembling approach for multilingual offensive language identification in social media.  In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2177–2182, 2020.
  * [23] Adeel Zafar, Hasan Mujtaba, Sohrab Ashiq, and Mirza Omer Beg.  A constructive approach for general video game level generation.  In 2019 11th Computer Science and Electronic Engineering (CEEC), pages 102–107. IEEE, 2019.
  * [24] Saad Naeem, Majid Iqbal, Muhammad Saqib, Muhammad Saad, Muhammad Soban Raza, Zaid Ali, Naveed Akhtar, Mirza Omer Beg, Waseem Shahzad, and Muhhamad Umair Arshad.  Subspace gaussian mixture model for continuous urdu speech recognition using kaldi.  In 2020 14th International Conference on Open Source Systems and Technologies (ICOSST), pages 1–7. IEEE, 2020.
  * [25] Abdul Rehman Javed, Muhammad Usman Sarwar, Mirza Omer Beg, Muhammad Asim, Thar Baker, and Hissam Tawfik.  A collaborative healthcare framework for shared healthcare plan with ambient intelligence.  Human-centric Computing and Information Sciences, 10(1):1–21, 2020\.
  * [26] Adil Majeed, Hasan Mujtaba, and Mirza Omer Beg.  Emotion detection in roman urdu text using machine learning.  In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering Workshops, pages 125–130, 2020.
  * [27] Muhammad Asad, Muhammad Asim, Talha Javed, Mirza O Beg, Hasan Mujtaba, and Sohail Abbas.  Deepdetect: detection of distributed denial of service attacks using deep learning.  The Computer Journal, 63(7):983–994, 2020.
  * [28] Yufei Wang, Ian D. Wood, Stephen Wan, Mark Dras, and Mark Johnson.  Mention flags (MF): constraining transformer-based text generators.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 103–113. Association for Computational Linguistics, 2021.
  * [29] Abdul Rehman Javed, Mirza Omer Beg, Muhammad Asim, Thar Baker, and Ali Hilal Al-Bayatti.  Alphalogger: Detecting motion-based side-channel attack using smartphone keystrokes.  Journal of Ambient Intelligence and Humanized Computing, pages 1–14, 2020.
  * [30] Muhammad Umair Arshad, Muhammad Farrukh Bashir, Adil Majeed, Waseem Shahzad, and Mirza Omer Beg.  Corpus for emotion detection on roman urdu.  In 2019 22nd International Multitopic Conference (INMIC), pages 1–6. IEEE, 2019.
  * [31] Sobha Lalitha Devi et al.  An alternate approach towards meaningful lyric generation in tamil.  In Proceedings of the NAACL HLT 2010 Second Workshop on Computational Approaches to Linguistic Creativity, pages 31–39, 2010.
  * [32] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In Advances in neural information processing systems, pages 3104–3112, 2014.
  * [33] Adeel Zafar, Hasan Mujtaba, and Mirza Omer Beg.  Search-based procedural content generation for gvg-lg.  Applied Soft Computing, 86:105909, 2020.
  * [34] Hussain S Khawaja, Mirza O Beg, and Saira Qamar.  Domain specific emotion lexicon expansion.  In 2018 14th International Conference on Emerging Technologies (ICET), pages 1–5. IEEE, 2018.
  * [35] Noman Dilawar, Hammad Majeed, Mirza Omer Beg, Naveed Ejaz, Khan Muhammad, Irfan Mehmood, and Yunyoung Nam.  Understanding citizen issues through reviews: A step towards data informed planning in smart cities.  Applied Sciences, 8(9):1589, 2018.
  * [36] Adeel Zafar, Hasan Mujtaba, Mirza Omer Beg, and Sajid Ali.  Deceptive level generator.  In AIIDE Workshops, 2018.
  * [37] Muhammad Umer Farooq, Mirza Omer Beg, et al.  Bigdata analysis of stack overflow for energy consumption of android framework.  In 2019 International Conference on Innovative Computing (ICIC), pages 1–9. IEEE, 2019.
  * [38] Martin Karsten, Srinivasan Keshav, Sanjiva Prasad, and Mirza Beg.  An axiomatic basis for communication.  ACM SIGCOMM Computer Communication Review, 37(4):217–228, 2007\.
  * [39] Muhammad Umer Farooq, Saif Ur Rehman Khan, and Mirza Omer Beg.  Melta: A method level energy estimation technique for android development.  In 2019 International Conference on Innovative Computing (ICIC), pages 1–10. IEEE, 2019.
  * [40] Hamza M Alvi, Hareem Sahar, Abdul A Bangash, and Mirza O Beg.  Ensights: A tool for energy aware software development.  In 2017 13th International Conference on Emerging Technologies (ICET), pages 1–6. IEEE, 2017.
  * [41] Mirza Beg and Peter van Beek.  A constraint programming approach for integrated spatial and temporal scheduling for clustered architectures.  ACM Transactions on Embedded Computing Systems (TECS), 13(1):1–23, 2013.


# COVIDRead: A Large-scale Question Answering Dataset on COVID-19

Tanik Saikh<EMAIL_ADDRESS>Sovan Kumar Sahoo  Asif Ekbal  Pushpak
Bhattacharyya  Department of Computer Science and Engineering
Indian Institute of Technology Patna
Bihta, Patna, India  Department of Computer Science and Engineering
Indian Institute of Technology Bombay
Mumbai, Maharashtra, India

###### Abstract

During this pandemic situation, extracting any relevant information related to
COVID-19 will be immensely beneficial to the community at large. In this
paper, we present a very important resource, COVIDRead, a Stanford Question
Answering Dataset (SQuAD) like dataset over more than 100k question-answer
pairs. The dataset consists of Context-Answer-Question triples. Primarily the
questions from the context are constructed in an automated way. After that,
the system-generated questions are manually checked by humans annotators. This
is a precious resource that could serve many purposes, ranging from common
people queries regarding this very uncommon disease to managing articles by
editors/associate editors of a journal. We establish several end-to-end neural
network based baseline models that attain the lowest F1 of 32.03% and the
highest F1 of 37.19%. To the best of our knowledge, we are the first to
provide this kind of QA dataset in such a large volume on COVID-19. This
dataset creates a new avenue of carrying out research on COVID-19 by providing
a benchmark dataset and a baseline model.

###### keywords:

Machine Reading Comprehension Dataset , Covid-19 Scholarly Articles , BERT ,
BioBERT , Question Answering , Automatic Article Peer Review System

††journal: Knowledge-Based Systems

##  1 Introduction

The Coronavirus diseases have affected human beings all over the world. This
is a very new but infectious disease that are very unknown to the common men.
So getting any kinds of information on this COVID-19 is of utmost need to all
of us. Extracting such information is a very challenging problem. There is a
pressing need that will serve the purpose of extracting information to any
kinds of queries. A Question Answering (QA) system that would give answer to a
query would serve this purpose. Recent advances in Artificial Intelligence
(AI) have matured the task of field specific Question-Answering (QA). Lately,
AI based deep learning techniques have shown its potential in many downstream
Natural Language Processing (NLP) tasks. To build any end-to-end deep learning
based QA system, dataset with a large volume is needed. This paper tries to
bridge this particular gap. It offers a large volume of QA dataset to the
research community to carry forward the research on COVID. In response to the
COVID-19 pandemic, the White House and a coalition of leading research groups
have prepared the COVID-19 Open Research Dataset (CORD-19) [1]. CORD-19 is a
resource of over 500k scholarly articles, including over 200k with full text,
about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available
dataset is a valuable resource to the research community that propels in
applying recent advances NLP and other AI based techniques (specially deep
learning based approaches) to generate new insights in support of the ongoing
fight against this infectious disease. There is an emerging importance for
these approaches because of the rapid growth in the number of coronavirus
related literature, making it difficult for the medical research community to
keep up.
The ongoing pandemic of coronavirus disease in 2019 (COVID-19) was first
reported in Wuhan, China in December 2019. The coronavirus disease is caused
by severe acute respiratory syndrome coronavirus 2 (SARS CoV 2) and is
primarily spread among people in proximity (within about 6 feet) most often
via droplets produced by sneezing, coughing, talking. As the reports of the
World Health Organization (WHO), no genuine licensed vaccines are yet
available. Hence, the key public health strategies, such as surveillance,
contract tracing, isolation and quarantine (wherever necessary) become the
core methods to combat the deadly disease.
This kind of QA would help researchers, developers, editors/associate editors
of a Journal and also to answer common men queries. The key attributes of the
current work are summarized as below:

  * 1.

This paper offers a dataset of QA. The dataset is having more than 100k
instances of Context-Answer-Question triples. Out of which 40k questions are
being human-annotated. This dataset is suitable for training data sensitive
various end-to-end deep neural network models.

  * 2.

A set of 1000 question-answer pairs are prepared manually by human experts.
Human made questions after reading the articles and put his/her answers for
that particular question. We provide answers for each question and its’
supporting text. So we prepare 1k such example pairs having abstract-question-
answer triples. This can be used as a gold standard test for future research.

  * 3.

We propose several baseline models based on BERT, BioBERT, and ClinicalBERT.

##  2 Related Work

The paper by [2] first presented dataset for QA on COVID named as COVID-QA.
They made use of the CORD-19 scientific articles. They offered biomedical
experts annotated 2019 question-answer pairs related to COVID-19 by
considering 147 scientific articles. They provided RoBERT based baseline,
fine-tuned on SQuAD along with model fine-tuned on the created COVID-QA
dataset. The model obtained the performance in terms of Exact Match and F1 as
25.9% and 59.53%, respectively.
The work defined in [3] offered CAiRE-COVID system. The system consists of
three modules, viz. Document Retriever, Relevant Snippet Selector, and Multi-
Document Summarizer. The first two modules perform an open-domain QA, whereas
the last one is responsible for generating concise summary (abstractive and/or
extractive) of the top-ranked retrieved relevant paragraphs. To better
generalise the system to unseen questions, the system is trained in a multi-
task learning scheme on six datasets, namely SQuAD, NewsQA, TriviaQA,
SearchQA, HotpotQA, Natural Question. The system is evaluated on Covid-QA
dataset. The paper [4] proposed a system that combines an Information
Retrieval module and a reading comprehension module that finds the answers in
the retrieved passages.

The paper described in [5] provided a real-time QA system, namely covidASK.
This also offered a small amount of dataset on Covid-19. The covidASK system
has borrowed many techniques from Open-domain QA.
The task defined in [6] presents CovidQA that comprises 124 question-article
pairs. This is a very small in size. This dataset is not suitable for running
any deep neural network models. But it can be helpful for evaluating the zero-
shot or transfer capabilities of the existing models on COVID-19 domain.
The paper of [7] shows an effective end-to-end method for zero-shot adaptation
for an open domain QA system to a target domain, COVID-19 in this case. They
also proposed a novel example generation model that can produce synthetic
training examples for both information retrieval and machine reading
comprehension.
[8] proposed CO-Search, that is a semantic, multi-stage search engine designed
to handle complex queries over the COVID-19 literature. CO-Search comprises a
hybrid semantic-keyword retriever and a re-ranker. The retriever (Siamese-BERT
based model) outputs a list of relevant documents given a query. The re-ranker
orders them by relevance that also assign a relevance score to each document.
However, they test the efficacy of the system on the combination of CORD-19
corpus and the TREC-COVID competition’s evaluation dataset.
The task that we tackle here is based on QA over COVID-19. The dataset is
human-annotated and large in size. We provide a rigorous experimentation and
detailed error analysis. We believe that our dataset is the largest one among
the datasets published so far. This kind of dataset is suitable for developing
domain-specific end-to-end automated deep neural models.

##  3 Dataset Creation

We create COVIDRead, a dataset for supporting text grounded extractive QA
system on COVID-19. This section describes our dataset creation procedure
including data collection, data pre-processing etc. In light of COVID-19
pandemic, the white house and coalition of leading research groups have
prepared the COVID-19 Open Research Dataset (CORD-19) [9]. CORD-19 is a
collection of various scholarly articles, about COVID-19, SARS-CoV-2, and
related corona viruses. These articles were collected from the sources like
PMC, Elsevier, WHO, Medrxiv, Biorxiv etc. We collect the articles available in
the Kaggle Portal 111https://www.kaggle.com/allen-institute-for-
ai/CORD-19-research-challenge. We collect 51,078 (as these many were available
at the time of starting our work) such articles. Among them, 8726 such
articles were there whose abstracts are missing. We consider the remaining
(i.e., 42,359) articles for the experiment purpose. These articles are in
Comma-Separated Values (.csv) format each column having components like
abstract, introduction etc. We extract abstracts for each article. Now we are
having a collection of 42,359 abstracts. We split each abstract into sentences
using NLTK sentence splitter 222https://www.nltk.org/api/nltk.tokenize.html.
We pass each sentence of a particular abstract through a constituency parser.
We foster Stanford CoreNLP 333https://stanfordnlp.github.io/CoreNLP/ for this
purpose. Constituency parser essentially breaks each sentence contained in the
abstract into noun and verb phrases. We take noun phrases into account. These
noun phrases become the plausible answers for that particular abstract 444We
use the terms Abstract, Document and Paragraph interchangeably throughout this
paper..

The studies like [10, 11] have suggested that noun phrases of a particular
document are the plausible answer of that particular document. We borrow the
concept from there and utilized in our task. These answers are paired with
its’ corresponding document. Now, we are having document and answer pairs.
These pairs are further passed through a Question Generator (QG) model. The QG
model produces questions conditioned on the given abstract and answer pairs.
Now we are having Context-Answer-Question triples. In this way, we generate
over 100k such pairs. Keeping the quality of the generated questions in mind,
we employ 3 annotators having expertise in medical domain with very good
knowledge in the English language. The annotators manually checked the
generated questions and also edited. This way we have created over 100k data-
points out of which 18k are human annotated. We coin this newly created
dataset as CovidRead 555We will make the dataset publicly available after
acceptance of this paper..

We split the whole dataset into training, development, and testing sets, as
shown in Table 1.

| Training | Development | Testing
---|---|---|---
CovidRead | 82415 | 10000 | 10000
Table 1: The standard distribution of CovidRead into Training, Development,
and Testing sets Figure 1: The Proposed System for Synthetic data generation
from COVID related Research Articles

###  3.1 Evaluation of Generated Questions

Keeping the quality of the generated questions in mind we perform various
evaluation schemes i.e. automatic as well as manual. Evaluating the outcomes
of any Natural Language Generation (NLG) systems is much more complex compared
to that of conventional NLP tasks (e.g., sentiment analysis, semantic parsing)
where only label matching is required. For NLG, there are no such scopes. We
apply the metrics that are widely used for Machine Translation and
Summarization (i.e., BLEU [12], METEOR [13], and ROUGE [14] etc.) tasks. We
apply these metrics on a sample of 2000 output examples for evaluation. These
evaluations metrics’ outputs are shown in the Table 2. The BLEU, METEOR, and
ROUGH are the n-gram based metrics, where n-gram lexical matching is
performed. We also apply Consensus-based image description evaluation (CIDEr)
metric, which is a popular one for evaluating various tasks in computer
vision. Apart from these metrics, we also perform a human evaluation. For this
we employ human annotators. Additionally, we propose a novel metric that could
be a potential metric for any kinds of NLG tasks. The approach is based on
logic (i.e., Entailment [15, 16]). Textual Entailment (TE) refers to the task
of determining whether sentence A is the logical consequence of another
sentence B or not. We can argue that this would the most appropriate metric to
judge whether two pieces of sentences are similar or not. With this motivation
we determine the entailment relation between machine-generated questions and
reference questions by applying one of the state-of-the-art entailment models
that is equipped with external knowledge [17]. The model is trained with the
combination of SNLI and Multi-NLI [18]. We evaluate the trained NLI system by
giving the inputs (i.e., system generated and human-generated questions as
premise and hypothesis respectively) to predict the entailment relation
between the system generated and human-generated questions. We asked the
annotators to provide inference labels (i.e., Entailment, Contradiction, and
Neutral) to each reference question given the corresponding system-generated
questions. We compare these labels with the system’s predicted labels.
Further, we compute the accuracy and got around 72%. Entailment relation
between these two questions indicates that generated questions are logically
accurate and correct, i.e., much closer to the question as humans generally
used to ask. This Entailment based metric could be a promising evaluation
metric for any NLG tasks.

BLEU | METEOR | ROUGE | CIDEr
---|---|---|---
0.18 | 0.12 | 0.12 | 0.345
Table 2: Evaluation results on NLG metrics

###  3.2 Annotation Guidelines

We employ three annotators to check the quality of the system generated
questions for a period of more than one and half year i.e. from the inception
of COVID-19 pandemic. The annotators are of post-graduate level in Bio Science
with an age group of 35-50 and having good expertise in English. We generally
do hypothesis testing while doing any kind of evaluation. We consider the
following hypotheses for the generated questions i.) Are the generated
questions appreciable by a human? and ii.) Are the generated questions
readable and comprehensible easily and also grammatically correct? Keeping
these parameters in mind, we instructed the annotators as follows:

  * 1.

The generated questions should be grammatically correct, and that should be
spelled correctly; no conjunction, proper punctuation should be there;
questions and proper nouns should begin with a capital letter.

  * 2.

The question should be understood by anyone, even to someone who is totally
unaware of the context.

  * 3.

For all the factoid questions, the answer should be unique and factual, and
for others types of questions the answers should be descriptive as COVIDRead
is a mixture of factoid and non-factoid questions.

We randomly chose 500 samples. To judge the naturalness (i.e., verify how the
generated questions are grammatically correct and how much fluent), we employ
two annotators. They were asked to give scores (between 0-4) to each such
question based on the above two parameters. We compute the inter-annotator
agreement ratio in terms of the kappa coefficient [19]. It was found to be
0.83, which is considered to be good agreement as per [20].

###  3.3 Dataset Analysis

The CovidRead contains over 100k synthetic data points, i.e., triples of
Context-Answer-Question. We manually check 40K such instances. We make a
rigorous analysis of these data points. The following Table 3 shows the
average length of Context, Answer, and Question. All the lengths are larger
compared other datasets in this line.

| Context | Question | Answer
---|---|---|---
Average Length | 1616.84 | 52.31 | 28.46
Table 3: Average length of Context, Question, and Answer in the CovidRead
dataset

To understand the properties of the CovidRead, we analyse the Questions and
Answers in the Training/Development set. We also categories our dataset into
factoid and non-factoid questions.
Diversity in Answer: From the Table 3, it is shown that the average length of
the answers is pretty high. So these should not be the answers of only factoid
questions. These could be the answer of non-factoid questions too. We assumed
NPs of a passage are the plausible answers for that passage. So sometimes
constituency parser produces incomplete noun phrases, that too become our
answers. In this way we obtain incomplete answers sometimes, such as : ”CT
equipment installation and environment” turned into ”installation and
environment”. This could be fault of the constituency parser used.

As we assumed that the NPs are the answers for a particular document, our
answers’ types are noun phrases.
Diversity in Questions: Over the time, QA systems have matured enough that can
answer factoid questions quite comfortably but they are far from giving answer
to non-factoid queries. For an accurate QA system that would produce accurate
answer to queries, understanding of questions is important that requires to
understand the type of questions. We compute percentage of each question for
each type of Questions. Quantitatively, the percentage of questions for
different types is shown in Table 4. Among them, the maximum number of
questions are of What type. As the Table 4 shows that the other types are
categorized into ”When”, ”Where”, ”Who” types etc. in descending order. To
better understand the questions contained in our dataset we compute the
distribution of question length and percentage of questions that are confined
within a particular range of length that is shown in Figure 2.

Figure 2: Percentage of questions with different word length

We see from the Figure 2 that most questions range from seven to ten words
that is quite logical. We discard the questions along with its context and
answer from the dataset, having number of words less than 3. These types of
questions could be occurred in community question answering scenario or QA on
ChatBot. As we are not in such scenarios, we have omitted those questions. To
get better view of our questions classification, we plot another chart, namely
a multi-level donate chart. Figure 3 shows such chart. The distribution of
questions is based on the first two words of the questions. There exists a
surprising variety, like what followed by be verb (is, am, are, was, were etc)
forming the maximum number of questions. Then What followed by kind/type are
the second most questions. In How categories How many type is the most number
of questions. In Misc category other types of questions including those that
are not starting with any wh words, it may contain at the middle and end of
the question. Understanding questions is one of the key topics in any QA
problem.

Figure 3: Distribution of questions by their first two words for a random
sample of 40K questions. The arc length is proportional to the number of
questions containing the word.

We also perform a rigorous analysis of 1000 sample questions and categories
them into Class-I, Class-II, and Class-III. Class-I defines the set of
questions that are easily understandable, grammatically correct, contextual
and also understandable even to those who are unaware of the context at all.
It is observed that 35% of the questions lay in this category. Questions that
are understandable to some extent with some minor mistakes in grammar fall
into Class-II category. There are 40% such questions. Remaining 25% of the
questions could be categorized into Class-III category, i.e. there are
incomplete questions, not understandable and the grammar is also poor. The
Figures 4, 6 and 5 show the examples of Class-I, Class-II and Class-III
respectively.

Question Type | % of Questions
---|---
What | 78.76
When | 3.14
Where | 2
Who | 37.44
Whom | 0.11
Which | 0.16
Whose | 0.03
Why | 0.72
How | 9.73
How many | 7.7
Is/Are | 0.48
Table 4: Percentage of questions for each type of question

Question Entailment and Answer Entailment: Question Entailment and Answer
Entailment drive its definition from the basic definition of Text Entailment
[21]. [22] defines Question Entailment between two Questions ($Q_{1}$ and
$Q_{2}$) if every proposition giving an answer to $Q_{1}$ is also giving an
answer to $Q_{2}$. [23] considered the first question $Q_{1}$ as a
superquestion $Q_{2}$ as a subquestion, if we answers to subquestions then we
have the answer to superquestions. From these two definitions we can say,
Question Entailment is defined as question A entails question B if every
answer to B is also a complete or partial answer to A [24]. Conversely, we can
introduce the concept of Answer Entailment as, if two answers say: Answer A
and Answer B are the answers of a same question and/or its paraphrased
question, then it could be termed as Answer Entailment. In the Figure 4, we
show the example of these two types of entailment. The aim of recognizing
question entailment (RQE) is to find answer to an incoming question by mining
entailed questions with its associated answer. This can therefore be defined
as:
a question A entails a question B if every answer to B is also a complete or
partial answer to A. The figure 4 shows the three questions Q1, Q2 and Q3
pointing to the same answer. So this is the example of Question Entailment. On
the contrary, Answer Entailment could be termed as follows:
If one question has two/multiple answers (Answer A and answer B), then this
scenario could be termed as the answer entailment. Refer to the Figure 4,
where the question 4 and question 5 have the same answers. This is therefore
an example of Answer Entailment. This Answer Entailment could help in Question
Answering, specially in handling contradictory questions. We identify such
kinds of Question Entailment and Answer Entailment examples in our CovidRead
dataset.

Figure 4: Example of data set having Class - I questions. Table also shows the
examples of Question Entailment and Answer Entailment.

As said, we have extracted these two type of examples from our dataset. This
provides a new direction of improving QA system. One of the studies like [25]
has showed the potential of Question Entailment in improvement of QA system.
There are no such studies that have shown Answer Entailment helping QA.
Reasoning required to answer questions: To get a better understanding of the
reasoning required to answer the questions, we picked up examples randomly. We
analyse those examples, i.e., what are required to give the answer to the
questions, and like SQuAD we manually label the examples with the categories
shown in Table 5. We have categorised them into a few categories.

In the Table 5, in the first row to answer the question system has to take
care of co-reference resolution. To answer the question contained in the
second row, QA system has to efficiently handle the negations. Handling
negation in any context is one of the toughest problem in NLP. The contexts
are full of Named Entities (NEs) (specially contain medical NEs), so maximum
questions are formed whose answers are NEs. To properly answer the questions
contained in the third row, system has to efficiently disambiguate the
appropriate NE among multiple NEs contained in the supporting text.

Figure 5: Example of Data set having Class - III questions. Figure 6: Example of Data set having Class - II questions. Reasoning | Context | Question
---|---|---
| Over multiple
---
sentences
| On December 31, 2019, several cases
---
of pneumonia of unknown etiology
have been reported in Wuhan, Hubei
province, China [1,2,3]. On January 7,
2020, Chinese health authorities confirmed
that these cases were associated with a novel
coronavirus, which was subsequently named
2019nCoV by WHO [4].
| Which cases were
---
associated with a novel
coronavirus that was
confirmed by Chinese
health authorities on
January 7, 2020?
| Negation
---
Handling
| Our integrative study not only presents
---
a systematic examination of
SARS-CoV-2-induced perturbation of
host targets and cellular networks to reflect
disease etiology, but also reveals insights into
the mechanisms by which SARS-CoV-2
triggers cytokine storms and represents a
powerful resource in the quest for therapeutic
intervention.
| What does this
---
investigation
present?
| Named Entity
---
Recognition
| A novel coronavirus (severe acute respiratory
---
syndrome coronavirus 2,SARS-CoV-2)
causing a cluster of respiratory infections
(coronavirus disease 2019, COVID-19) in
Wuhan, China, was identified on 7 January 2020.
| Q1: What is SARS-CoV-2?
---
Q2: What is novel
coronavirus?
|  |
Table 5: We manually labeled $\sim$200 examples into one or more of the above
categories.

From Table 5 it is obvious that some percentage of questions in CovidRead are
not straightforward to answer, whereas many are so easy to answer.

##  4 Methods

We propose different transformer based methods for QA. Transformer based
models have shown tremendous progress in various downstream tasks in NLP.
Actually, we make use of bi-directional version of this network. The following
subsections are showing the different models used in this paper.
BERT: Any QA system requires understanding of the underlying language.
Recently transformer architecture based models have shown this kind of
capability of capturing language understandability, specifically the bi-
directional version of this. Keeping this point in mind we apply this
transformer based model into our COVID QA task. This is our simple baseline
model. The diagram is shown in the Figure 7.
BioBERT: Our second model is BioBERT [26] based. This is pre-trained
biomedical language representation model. It is initialized with BERT’s pre-
trained parameters. After pre-training, the model is further fine-tuned on
PubMed abstract and PubMed Central full-text articles. We apply the best
version of this model. The motivation is as we are dealing with the
vocabularies that contain bio-medical texts, we make use of this model. The
architecture is same as the previous one with slide modification in the
training data.
ClinicalBERT: Our third model is another version of this language model, i.e.
ClinicalBERT [27]. This model is initialised from BioBERT v1.0, and then pre-
trained on more than 2 million clinical notes contained in the MIMIC-III v1.4
database of patient notes [28]. We make use of the best variant of this of
Clinical BERT that has 108M parameters.

Figure 7: A BERT/BioBERT based QA Model. Image courtesy: ([29];
https://blog.scaleway.com/2019/understanding-text-with-bert/) with minimal
modifications

##  5 Experiments, Results and Discussions

We run the baseline and the proposed models on our created dataset. We perform
exhaustive experiments with these models by different combinations of the
datasets. We perform six experiments.
BERT: There is a recent style of knowledge learning and utilizing vibrates the
NLP field with huge success authorized by the pre-trained language model
(i.e., BERT) and its variants. This model is based on vanilla BERT. We follow
the architecture provided in the Figure 7. We have 100,000 examples
(synthetic) of Abstract-Question-Answer triples. Out of which nearly 40K
questions are manually checked. We train the model on 39178 such instances,
and tested with 1000 instances. The result obtained is shown in the Figure.
From the Figure 7, it is clear that the system takes concatenation of Context
and Question (context and question separated by [SEP] token that starts with
[CLS] token) as input. The system produces span of texts as output, that is
being the predicted answer. There are two classifiers on top of this model.
One classifier (with a start vector $S$) predicts start index and the second
one (with end vector $H$) yields end index of the answer. The probability of
token being start of an answer is computed by the dot product $T_{i}$ and a
start vector $S$. Then there is a soft-max over all the tokens in the
abstract.

$P_{i}=\frac{e^{ST_{i}}}{\sum_{j}e^{ST_{i}}}$ (1)

In the similar way, the end index of the answer span is calculated. Candidate
span score from i to j is defined as $S.T_{i}+E.T_{j}$. The span having
maximum score (where $j>=i$); is considered as the predicted answer. We take
the implements of the models available in huggingface.
BioBERT The second experiment is performed using BioBERT based model. This is
basically BERT model enriched with biomedical knowledge via pre-training over
biomedical corpora like PubMed. Keeping the same data distribution we perform
this experiment too. Evaluation results of this model are shown in the Table
6.
ClinicalBERT Our third experiment is performed using the clinical BERT model.
In this experiment the data distribution is also the same. Results obtained by
this model are shown in Table 6.
BioBERT-I This experiment is based on the bioBERT model with some
modifications. We named this as BioBERT-I. We apply a SQuAD pre-trained QA
model and that further find-tuned on the Gold Standard dataset. We test this
model with our test set of 1000 instances. The results produced by this model
are shown in Table 6.
BioBERT-II This model is trained on the whole dataset i.e. mixture of human
annotated and synthetic data. We called this model as BioBERT-II. The size of
such dataset is over 100k instances. We further test this trained model with
the gold standard test dataset having 1000 instances.
BioBERT-III Synthetic-Train-fineTuned: This experiment is same the previous
one but slightly different from the previous one. Here, we perform the
training of this model with the same dataset as the previous one, then fine-
tune and then trained. We further text the trained model with the gold
standard dataset. The results obtained are shown in Table 6.

We rely on two standard evaluation metrics (i.e., Exact Match and F1) that are
widely used for evaluating Span-of-Texts based extractive QA systems [10, 30].
The metrics avoid punctuation and articles (e.g., a, an, the etc.). We have
only one reference answer for testing. Using this we compute Exact Match and
F1 scores. These two are the standard metrics for evaluating Span-of-Texts
based QA Systems, and widely used in various notable QA Systems [31, 11, 32,
30].
Exact Match:  This metric computes the number of matching predicted answers
with the ground truth answers character by character.
F1 score:  This is a macro-averaged F1 score. It converts the predicted and
ground-truth answers as bag-of-words. Then calculate the average overlap
between the predicted and ground-truth answers. Further, computes their F1,
and then take the average over all the instances.
The results obtained are shown in Table 6. From the Table, we can say that all
the proposed models perform better compared to the baseline in terms of exact
match. The vanilla BERT model is fine-tuned on a large dataset that performs
better compared to the baseline. In these experiments, we deal with biomedical
scientific texts, so our next model utilises the BERT model that is trained on
biomedical scientific texts (i.e., BioBERT). This model performs better
compared to the vanilla BERT model. Our third proposed model is based on BERT
that is trained on Clinical Bio-medical texts. It perform better compared to
BERT and also Bio-BERT based model. Our next model is based on Bio-BERT that
is pre-trained on SQuAD and further trained and tested on the gold data. This
model is performed inferior compared to the last three models. As SQuAD is
full of factoid questions, and our dataset is a mixture of factoid and non-
factoid questions, that may be the cause of poor performance of this model.The
next model is also based on Bio-BERT but train on whole 100k examples and
tested on the gold test data. The model yields superior results compared to
the previous models. As training and testing is perform on the same domain
dataset, this is one of the reasons of good performance. The last model is
same as the previous one but fine tuned on the gold dataset. This one performs
the best among all the proposed models in terms of both the evaluation
schemes.

Models | Results (%)
---|---
| Exact Match | F1
Proposed
BioBERT-III | 28.65 | 37.19
BioBERT-II | 25.2 | 33.95
BioBERT-I | 22.25 | 29.93
ClinicalBERT | 25 | 34.09
BioBERT | 24.95 | 34.37
BERT | 23.2 | 32.03
State-of-the-art
COVIDQA Model | 25.90 | 59.53
Baseline
COVIDQA Model | 21.84 | 49.43
Table 6: Proposed methods’ results and comparison with baseline model.

###  5.1 Error Analysis

We perform qualitative and quantitative error analysis. We rigorously compare
the actual and predicted output. The following are our observations:

  * 1.

Most of BERT-like models have limitations of max input of 512 tokens, but in
our case, some contexts are having more than 2000 tokens. BERT models trim the
remaining tokens, resulting information lost in the context. It would be one
of the reasons for low performance.

  * 2.

The system is able to predict the answer of the questions, where there is a
token overlap between question and answer containing the sentence.

  * 3.

Sometimes the system talking the whole phrase before the generating question
sentence.

  * 4.

Sometimes the system taking the answer containing full phrase instead of the
exact answer, as a result the system yields better F1 compared to exact match.

  * 5.

System unable to predict numerical type answer.

  * 6.

Abstracts are confined with 260-300 words, which is considered a long document
compared to the document in SQuAD. Such long documents make the QA task
difficult.

###  5.2 Observation and main message

We offer QA dataset having Context-Question-Answer examples triples over 100K
from research articles to the research community on COVID-19. Out of 100K
examples, 40K questions are human-annotated, the remaining are synthetic. We
hope this novel dataset can serve as a precious resource for research and
evaluation of machine reading comprehension models (specifically, QA and QG
models) on research articles. Our test dataset is paraphrased. Testing a QA
model with this parapharsed enriched data would test the robustness of a QA
systems. So our dataset would serve this purpose also.

##  6 Robustness of QA model

A question could be asked in many ways. We human being generally follow this
trend and used to answer accordingly. To make an automatic QA system to act as
human, machine has to be trained in this manner. For this we paraphrased our
test set’s each question upto 3 questions. We make use of a pre-trained
paraphrased model [33] for this purpose. Then we augment our test set with
these paraphrased questions. We test our best performing system again on this
augmented test set, and observe that the results get decreased as shown in
Table 7. So to test the robustness of a QA system, our paraphrased enriched
corpus will be useful.

Data | Model | Exact Match | F1
---|---|---|---
| Trained and tested on
---
paraphrased enriched corpus
BioBERT | 20.77 | 28.86
| Trained and tested on
---
without paraphrased enriched corpus
BioBERT | 29.91 | 36.88
Table 7: BioBERT based model tested on i.) paraphrased enriched corpus and
ii.) without paraphrased enriched corpus

##  7 Conclusion and Future Work

In this paper, we have presented CovidRead, a new dataset for benchmark
evaluation of methods for QA/MRC on Covid-19. The dataset is created in a
semi-automatic way. The dataset consists of Context-Question-Answer triples
over 100k, out of which 40K are human-annotated. Researchers and
editors/associate editors would be benefited out of this kind of models and
the dataset. We have proposed three approaches that are based on viz. (i.)
Vanilla BERT (ii.) Bio-medical BERT (i.e., BioBERT) and (iii.) Clinical BERT.
Our proposed models are competitive compared to the existing state-of-the-art
models. Our future works would include:

  1. 1.

Considering the full-text of manuscripts instead of only abstracts. As users’
all questions could not be found in the abstract only.

  2. 2.

Implementation of the Bi-directional Attention flow (Bi-DAF) model by
incorporating multi-hop attention.

  3. 3.

Enrichment of this dataset up to 500K in size.

  4. 4.

Multi-hop version of this CovidRead (like HotpotQA) dataset.

  5. 5.

Building a Visual Question Answering (VQA) datasets and models on the images
available in the covid-19 literature.

  6. 6.

Model based on Generative Pre-trained Transformer (GPT)-3.

##  8 Acknowledgement

On behalf of all the co-authors Mr. Tanik Saikh would like to acknowledge
”Elsevier Centre of Excellence for Natural Language Processing” at Indian
Institute of Technology Patna for supporting research work carried out in this
paper.

## References

  * [1] L. L. Wang, K. Lo, Y. Chandrasekhar, R. Reas, J. Yang, D. Burdick, D. Eide, K. Funk, Y. Katsis, R. M. Kinney, Y. Li, Z. Liu, W. Merrill, P. Mooney, D. A. Murdick, D. Rishi, J. Sheehan, Z. Shen, B. Stilson, A. D. Wade, K. Wang, N. X. R. Wang, C. Wilhelm, B. Xie, D. M. Raymond, D. S. Weld, O. Etzioni, S. Kohlmeier, CORD-19: The COVID-19 open research dataset, in: Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Association for Computational Linguistics, Online, 2020.
URL https://www.aclweb.org/anthology/2020.nlpcovid19-acl.1

  * [2] T. Möller, A. Reina, R. Jayakumar, M. Pietsch, COVID-QA: A question answering dataset for COVID-19, in: Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Association for Computational Linguistics, Online, 2020.
URL https://www.aclweb.org/anthology/2020.nlpcovid19-acl.18

  * [3] D. Su, Y. Xu, T. Yu, F. B. Siddique, E. Barezi, P. Fung, CAiRE-COVID: A question answering and query-focused multi-document summarization system for COVID-19 scholarly information management, in: Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Association for Computational Linguistics, Online, 2020.  doi:10.18653/v1/2020.nlpcovid19-2.14.
URL https://www.aclweb.org/anthology/2020.nlpcovid19-2.14

  * [4] A. Otegi, J. A. Campos, G. Azkune, A. Soroa, E. Agirre, Automatic evaluation vs. user preference in neural textual QuestionAnswering over COVID-19 scientific literature, in: Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Association for Computational Linguistics, Online, 2020.  doi:10.18653/v1/2020.nlpcovid19-2.15.
URL https://www.aclweb.org/anthology/2020.nlpcovid19-2.15

  * [5] J. Lee, S. S. Yi, M. Jeong, M. Sung, W. Yoon, Y. Choi, M. Ko, J. Kang, Answering questions on COVID-19 in real-time, CoRR abs/2006.15830.
  * [6] R. Tang, R. Nogueira, E. Zhang, N. Gupta, P. Cam, K. Cho, J. Lin, Rapidly bootstrapping a question answering dataset for covid-19 (2020).  arXiv:2004.11339.
  * [7] R. G. Reddy, B. Iyer, M. A. Sultan, R. Zhang, A. Sil, V. Castelli, R. Florian, S. Roukos, End-to-end qa on covid-19: Domain adaptation with synthetic training, arXiv preprint arXiv:2012.01414.
  * [8] A. Esteva, A. Kale, R. Paulus, K. Hashimoto, W. Yin, D. R. Radev, R. Socher, Co-search: COVID-19 information retrieval with semantic search, question answering, and abstractive summarization, CoRR abs/2006.09595.
URL https://arxiv.org/abs/2006.09595

  * [9] L. L. Wang, K. Lo, Y. Chandrasekhar, R. Reas, J. Yang, D. Eide, K. Funk, R. Kinney, Z. Liu, W. Merrill, P. Mooney, D. A. Murdick, D. Rishi, J. Sheehan, Z. Shen, B. Stilson, A. D. Wade, K. Wang, C. Wilhelm, B. Xie, D. Raymond, D. S. Weld, O. Etzioni, S. Kohlmeier, CORD-19: the covid-19 open research dataset, CoRR abs/2004.10706.
URL https://arxiv.org/abs/2004.10706

  * [10] P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, SQuAD: 100,000+ questions for machine comprehension of text, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, 2016, pp. 2383–2392.  doi:10.18653/v1/D16-1264.
URL https://www.aclweb.org/anthology/D16-1264

  * [11] A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, K. Suleman, NewsQA: A Machine Comprehension Dataset, in: Proceedings of the 2nd Workshop on Representation Learning for NLP, Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 191–200.  doi:10.18653/v1/W17-2623.
URL https://www.aclweb.org/anthology/W17-2623

  * [12] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, BLEU: A Method for Automatic Evaluation of Machine Translation, in: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, 2002, pp. 311–318.
  * [13] S. Banerjee, A. Lavie, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, in: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Association for Computational Linguistics, Ann Arbor, Michigan, 2005, pp. 65–72.
URL https://www.aclweb.org/anthology/W05-0909

  * [14] C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81.
URL https://www.aclweb.org/anthology/W04-1013

  * [15] I. Dagan, O. Glickman, B. Magnini, The PASCAL recognising textual entailment challenge, in: J. Q. Candela, I. Dagan, B. Magnini, F. d’Alché-Buc (Eds.), Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, Vol. 3944 of Lecture Notes in Computer Science, Springer, 2005, pp. 177–190.  doi:10.1007/11736790\\_9.
URL https://doi.org/10.1007/11736790_9

  * [16] B. MacCartney, C. D. Manning, Natural language inference, Citeseer, 2009.
  * [17] Q. Chen, X. Zhu, Z.-H. Ling, D. Inkpen, S. Wei, Neural natural language inference models enhanced with external knowledge, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Melbourne, Australia, 2018, pp. 2406–2417.  doi:10.18653/v1/P18-1224.
URL https://www.aclweb.org/anthology/P18-1224

  * [18] A. Williams, N. Nangia, S. Bowman, A broad-coverage challenge corpus for sentence understanding through inference, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, 2018, pp. 1112–1122.
URL http://aclweb.org/anthology/N18-1101

  * [19] J. L. Fleiss, Measuring Nominal Scale Agreement Among Many Raters., Psychological bulletin 76 (5) (1971) 378.
  * [20] J. R. Landis, G. G. Koch, The Measurement of Observer Agreement for Categorical Data, biometrics (1977) 159–174.
  * [21] I. Dagan, O. Glickman, B. Magnini, The pascal recognising textual entailment challenge, in: Machine Learning Challenges Workshop, Springer, 2005, pp. 177–190.
  * [22] J. A. G. Groenendijk, M. J. B. Stokhof, Studies on the semantics of questions and the pragmatics of answers, Ph.D. thesis, Univ. Amsterdam (1984).
  * [23] C. Roberts, Information structure in discourse: towards an integrated formal theory of pragmatics.’ms. the ohio state university.
  * [24] A. B. Abacha, D. Demner-Fushman, Recognizing question entailment for medical question answering, in: AMIA Annual Symposium Proceedings, Vol. 2016, American Medical Informatics Association, 2016, p. 310.
  * [25] A. Ben Abacha, D. Demner-Fushman, A question-entailment approach to question answering, CoRR abs/1901.08079.  arXiv:1901.08079.
URL http://arxiv.org/abs/1901.08079

  * [26] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang, BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformaticsdoi:10.1093/bioinformatics/btz682.
URL https://doi.org/10.1093/bioinformatics/btz682

  * [27] E. Alsentzer, J. R. Murphy, W. Boag, W.-H. Weng, D. Jin, T. Naumann, M. McDermott, Publicly available clinical bert embeddings, arXiv preprint arXiv:1904.03323.
  * [28] A. E. Johnson, T. J. Pollard, L. Shen, H. L. Li-Wei, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, R. G. Mark, Mimic-iii, a freely accessible critical care database, Scientific data 3 (1) (2016) 1–9.
  * [29] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186.  doi:10.18653/v1/N19-1423.
URL https://www.aclweb.org/anthology/N19-1423

  * [30] G. Lai, Q. Xie, H. Liu, Y. Yang, E. Hovy, RACE: Large-scale ReAding comprehension dataset from examinations, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 785–794.  doi:10.18653/v1/D17-1082.
URL https://www.aclweb.org/anthology/D17-1082

  * [31] P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, Squad: 100,000+ questions for machine comprehension of text (2016).  arXiv:1606.05250.
  * [32] M. Joshi, E. Choi, D. S. Weld, L. Zettlemoyer, Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Vancouver, Canada, 2017\.
  * [33] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, CoRR abs/1910.10683.  arXiv:1910.10683.
URL http://arxiv.org/abs/1910.10683