# Self-supervised Contrastive Learning of Multi-view Facial Expressions

Shuvendu Roy, Ali Etemad  Department of Electrical and Computer Engineering &
Ingenuity Labs Research Institute
Queen’s UniversityKingstonCanada shuvendu.roy<EMAIL_ADDRESS>
(2021)

###### Abstract.

Facial expression recognition (FER) has emerged as an important component of
human-computer interaction systems. Despite recent advancements in FER,
performance often drops significantly for non-frontal facial images. We
propose Contrastive Learning of Multi-view facial Expressions (CL-MEx) to
exploit facial images captured simultaneously from different angles towards
FER. CL-MEx is a two-step training framework. In the first step, an encoder
network is pre-trained with the proposed self-supervised contrastive loss,
where it learns to generate view-invariant embeddings for different views of a
subject. The model is then fine-tuned with labeled data in a supervised
setting. We demonstrate the performance of the proposed method on two multi-
view FER datasets, KDEF and DDCF, where state-of-the-art performances are
achieved. Further experiments show the robustness of our method in dealing
with challenging angles and reduced amounts of labeled data.

Contrastive Learning, Self-Supervised Learning, Multi-View Learning, Affective
Computing

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
2021 International Conference on Multimodal Interaction; October 18–22, 2021;
Montréal, QC, Canada††booktitle: Proceedings of the 2021 International
Conference on Multimodal Interaction (ICMI ’21), October 18–22, 2021,
Montréal, QC, Canada††price: 15.00††doi: 10.1145/3462244.3479955††isbn:
978-1-4503-8481-0/21/10††ccs: Computer systems organization Embedded
systems††ccs: Computer systems organization Redundancy††ccs: Computer systems
organization Robotics††ccs: Networks Network reliability

##  1\. Introduction

Facial expression recognition (FER) is an important attribute of non-verbal
communication. FER systems play an important role in human-machine systems by
enabling customization and adaptation of such systems to user reactions.
Examples of such devices include emotion-aware multimedia and smart devices
(Cho et al., 2019), personal mood management systems (Thrasher et al., 2011;
Sanchez-Cortes et al., 2013), driving assistants (Leng et al., 2007), health-
care assistants (Tokuno et al., 2011), and others. However, despite the
potential for a wide variety of applications, FER remains challenging due to
various factors like illuminations, scene backgrounds, occlusions, and
challenging viewing angles among others.

Despite the recent progress in FER systems with deep learning (Liu et al.,
2018, 2019; Pourmirzaei et al., 2021), most prior work in the area focus on
only utilizing frontal facial views (Pourmirzaei et al., 2021; Hasani et al.,
2020). As a result, developed models may not be able to recognize expressions
in faces captured from sharp side angles. Another problem in deep learning-
based FER systems is the requirement for large amounts of training data given
the millions of trainable parameters.

Figure 1. High-level visualization of CL-MEx. The method aims to minimize the
distance between all positive examples (i.e. the radius of the black ellipses)
which include similar expressions obtained from different angles, and maximize
the distance between positive and negative pairs. Subject ID of the depicted
face: AF01.

In this work, to deal with the problems of view-point sensitivity and labeled
training set size, we propose Contrastive Learning for Multi-view facial
Expressions (CL-MEx). Our proposed framework learns facial expression
representations through a two-step training process. The first step is self-
supervised pre-training, whereby our novel loss function forces the latent
representations of ‘different’ views with the same expression from the same
subject to be grouped together. This will allow for challenging viewpoints
(e.g. side view) to be augmented by and benefit from the more effective
representations of frontal views, allowing better performance (see Figure 1).
After pre-training, we fine-tune the model with the labeled data in a
supervised setting. Our experiments show that the CL-MEx training protocol
obtains better results than fully supervised training and outperforms the
state-of-the-art solutions on the two datasets used (KDEF (Lundqvist et al.,
1998) and DDCF (Dalrymple et al., 2013)). Moreover, we demonstrate that our
model does not require all the available labels to reach the performance of a
fully supervised model. Lastly, we demonstrate that as a result of our
proposed loss, our model is better equipped to deal with challenging views.

In summary, our contributions are as follows.

  * •

We propose a novel contrastive learning framework, CL-MEx, for multi-view
data. Our model utilizes a novel loss function designed specifically for
multi-view images.

  * •

Experiments demonstrate the strong performance of our approach by setting a
new state-of-the-art, and reduced sensitivity to challenging views. Our model
also requires less labeled data to obtain competitive results with respect to
fully supervised training.

##  2\. Related Works

In this section, we review recent works in two key areas related to this
paper: self-supervised and contrastive learning, and multi-view facial
expression recognition.

###  2.1. Self-Supervised and Contrastive Learning

To reduce the influence of output labels and human-annotations, self-
supervised learning has been proposed to use specific augmentations (Chen et
al., 2020a) or transformations (Noroozi et al., 2017; Dosovitskiy et al.,
2015) applied to input data, and produce pseudo-labels to train or pre-train
the model. Successive to training or pre-training the model with the
augmented/transformed data and associated pseudo-labels (‘pretext’ tasks), the
main building blocks of the model, for instance the convolutional blocks in a
CNN, can then be used for ‘downstream’ tasks, e.g. classification (Dosovitskiy
et al., 2015). In this phase, the convolutional blocks are typically frozen
and new fully connected layers are added and trained from scratch using the
original output labels (Chen et al., 2020a). The pre-trained components can
also be fine-tuned to help achieve better performance (Noroozi and Favaro,
2016).

Self-supervised learning has been utilized in a variety of different problem
domains including image analysis (Noroozi and Favaro, 2016; Chen et al.,
2019), wearable-based activity recognition (Saeed et al., 2019; Rahimi
Taghanaki and Etemad, 2020), and affective computing with bio-signals (Sarkar
and Etemad, 2020b; Sarkar et al., 2020; Sarkar and Etemad, 2020a).
Specifically for image-based recognition, a variety of different self-
supervised solutions have been proposed in recent years (Doersch et al., 2015;
Noroozi and Favaro, 2016; Zhang et al., 2016). A key differentiator in many of
these methods is the design of novel and interesting pretext tasks. Examples
include jigsaw puzzle solving (where the images is divided into pieces and
shuffled with the pretext aiming to solve the puzzle) (Doersch et al., 2015;
Noroozi and Favaro, 2016), inpainting (where patches are removed and the
pretext aims to complete the image) (Pathak et al., 2016), transformation
classification (where the pretext aims to classify the type of transform
applied to the image, e.g., rotation, mirroring, etc.) (Gidaris et al., 2018),
and colorization (where the pre-text aims to color the image converted to
gray-scale) (Zhang et al., 2016).

Contrastive learning is a variation of self-supervised learning that has
recently shown great success, achieving state-of-the-art performance in a
variety of tasks such as object detection (Yang et al., 2021), medical image
analysis (Chaitanya et al., 2020), image classification (Hadsell et al., 2006;
Dosovitskiy et al., 2014; Oord et al., 2018; Bachman et al., 2019), and video-
based affect classification (Roy and Etemad, 2021). A recent method called
SimCLR (Chen et al., 2020b) proposed the use of transformations, referred to
as ‘augmentations’, to create different variations of input images and
maximize the agreement between positive pairs (augmented versions of the same
image) via contrastive loss. A number of different variations of SimCLR have
been proposed to exploit this framework and provide robust results (Chen and
He, 2020; Khosla et al., 2020; Chen et al., 2020c). Later, a variation of
contrastive learning was proposed to also make use of the output labels,
effectively creating a ‘supervised’ contrastive learning framework (Khosla et
al., 2020). In this approach, the positive examples were generated not only
from different augmentations of the same input, but also the augmented
versions of other examples in the ‘same class’.

In the context of FER, which is the focus of this paper, self-supervised or
contrastive learning have rarely been explored. In a recent work, FER was
performed with a combination of contrastive learning and rotation prediction
as pretext tasks (Pourmirzaei et al., 2021).

###  2.2. Multi-view FER

Multi-view FER exploits facial images obtained from different angles to learn
a mapping between images of the face performing specific types of expressions
and the expression class. This approach allows for the methods to capture
information that might otherwise be hidden from the camera in certain view-
points, resulting in more robust performance. Deep learning solutions have
recently been proposed to jointly learn the different view-points and perform
FER in a variety of different settings (Mahesh et al., 2021; Liu et al., 2018,
2019).

A multi-channel pose-aware CNN (MPCNN) was proposed in (Liu et al., 2018), and
showed strong performance for multi-view FER. In PhaNet (Liu et al., 2019), a
pose-adaptive hierarchical attention network was proposed that could jointly
recognize the facial expressions and poses. At any angle of view, the
attention module helped the model identify the most relevant regions for
detecting the expression. In (Sepas-Moghaddam et al., 2021), a new LSTM cell
architecture was proposed to jointly learn different perspectives, which was
then used for a number of different tasks, including multi-view FER.

##  3\. Method

We propose a framework called CL-MEx for exploiting images acquired from
multiple views to perform FER with self-supervised contrastive learning. Like
SimCLR (Chen et al., 2020b), our proposed method uses positive and negative
examples to carry out contrastive learning. However, we do not limit the
definition of positive pairs as augmentations of a particular input, and
expand this notion to include the augmentations of all the views of the input
image. CL-MEx then uses our proposed loss function to bring the
representations of different views of a subject with the same expression
closer in the embedding space (as shown earlier in Figure 1). Successive to
the pre-training step, we fine-tune the model by supervised learning using the
labels from the dataset. The proposed method uses the multiple views of an
image in the self-supervised pre-training step only, while at inference time,
it predicts the expression using only a single-view image. Following we
present the details of our proposed method including the encoder architecture,
proposed loss, and implementation details.

###  3.1. CL-MEx Components

The main components of our proposed framework are as follows.

Data Augmentation Module, $Aug(.)$. This module randomly applies combinations
of the following augmentations: random cropping, random resizing, random
horizontal flip, random color distortion, and random gray-scaling, as
$x^{\prime}=Aug(x)$, where $x$ is the input image. The module applies random
flip and gray-scaling with a probability of 0.5. For resize followed by random
crop, first the image is randomly resized by a factor of 0.2 to 1, and then
randomly cropped to obtain an augmented image of the original size. For colour
distortion, the brightness, hue, saturation, and contrast are randomly changed
with a coefficient of 0.4 to 1. For each input image we generate two augmented
versions of the same image.

Encoder Network, $Enc(.)$, which maps $x^{\prime}$ to a representation
embedding vector, $r=Enc(x^{\prime})$. We pass both of the augmented images to
the same encoder, resulting in a pair of representation vectors. We use the
embedding dimension of 512 for all our experiments. In this work, we use the
ResNet-18 (He et al., 2016) as our encoder network.

Projection Head, $Proj(.)$, which maps the vector $r$ onto a projection
embedding vector, $z=Proj(r)$. Following the intuition of (Chen et al., 2020b;
Khosla et al., 2020), we use a 2-layer dense network for the projection head
which gives an output dimension of 128. We normalize the output of this layer
to enable using a linear product to measure distances in the projection space.
This projection has proven to be important for learning effective
representations by self-supervised pre-training (Chen et al., 2020b). After
the pre-training, we discard the projection head and only fine-tune the
encoder for the FER task.

Figure 2. Training pipeline of the proposed method.

###  3.2. Multi-view Contrastive Loss

For the purpose of pre-training the model in a self-supervised setting, we
propose a contrastive loss function which is inspired by SupCon (Khosla et
al., 2020). However, unlike SupCon, we only need the multi-view information to
train the model and do not use the output labels in the pre-training stage.

In a batch of size $2N$, the images are obtained by applying two random
augmentations on $N$ random input samples. These images are passed through the
encoder and the projection head to generate the embedding vector $Z$, on which
the contrastive loss is calculated. The multi-view contrastive loss is defined
as:

(1) $\small\begin{split}\mathcal{L}^{\text{CL-
MEx}}=\sum_{i=1}^{2N}\bigg{(}\frac{-1}{2N_{v_{i}}-1}\sum_{j=1}^{2N}\mathds{1}_{i\neq
j}\cdot\mathds{1}_{v_{i}=v_{j}}\cdot\\\ log\frac{exp(z_{i}\cdot
z_{j}/\tau)}{\sum_{k=1}^{2N}\mathds{1}_{[k\neq i]}exp(z_{i}\cdot
z_{k}/\tau)}\bigg{)}~{},\end{split}$

where $v_{i}$ is the view-invariant id of image $i$, which indicates that for
$v_{i}=v_{j}$, $i$ and $j$ are different views of the same subject.
$\mathds{1}_{[a=b]}\in\\{0,1\\}$ is a function of $a$ and $b$, which outputs 1
when $a=b$, and 0 otherwise. $N_{v_{i}}$ is the number of instances with id
$v_{i}$. $\tau$ represents a temperature parameter which scales the dot
product similarity. Figure 2 depicts a visual illustration of the proposed CL-
MEx method, where the data sampler generates two augmented images along with
their corresponding view-invariant ids, which generate the embedding by means
of the proposed loss function.

###  3.3. Downstream FER

While the projection head is an important component for learning
representations in the contrastive pre-training step, it is no longer required
for the final downstream FER classification task. During the second step for
FER, we discard the projection head and add a linear layer for predicting the
output class probability. We then freeze the encoders and train the new linear
layer with class labels using categorical cross-entropy loss. This is followed
by fine-tuning the entire model including the encoders to maximize
performance.

###  3.4. Implementation Details

During the pre-training step, the model is trained with the proposed multi-
view contrastive loss function for 500 epochs with an Adam optimizer, a
learning rate of 0.0001 with cosine learning rate decay, and weight decay of
1e-4. The downstream FER step consists of a total of 60 training epochs.
First, we train the newly added classification layer for 10 iterations, and
then fine-tune the full model for 50 additional iterations. During this step,
an Adam optimizer was again used, this time with a plateau learning rate decay
with an initial learning rate of 1e-4, decay factor of 0.5, and patience of 3.
The method is implemented with PyTorch and trained on 4 NVIDIA V100 GPUs. All
image resolutions are $224\times 224$.

##  4\. Experiments and Results

This section describes the experimental setup and results for CL-MEx on two
benchmark datasets that contain multi-view facial expressions, Karolinska
Directed Emotional Faces (KDEF) (Lundqvist et al., 1998) and Dartmouth
Database of Children’s Faces (DDCF) (Dalrymple et al., 2013).

###  4.1. Datasets

KDEF (Lundqvist et al., 1998) is a multi-view emotion recognition dataset with
7 expressions (afraid, angry, happy, sad, surprised, disgust, and neutral),
where each subject is imaged from 5 different views (-90∘: full left (FL),
-45∘: half left (HL), 0∘: straight (S), +45∘: half right (HR), and +90∘: full
right (FR)). The dataset was collected from 140 subjects. The DDCF (Dalrymple
et al., 2013) is also a multi-view dataset collected from 80 subjects. It
contains 8 expression classes (afraid, angry, happy, sad, surprised, disgust,
pleased, and neutral), from 5 different views (-60∘: full left (FL), -30∘:
half left (HL), 0∘: straight (S), +30∘: half right (HR), and +60∘: full right
(FR)).

Table 1. Results and comparison with other methods.

(a) KDEF Method | Acc. $\pm$ SD
---|---
SVM (Moore and Bowden, 2011) | 70.5$\pm$1.2
SURF (Rao et al., 2015) | 74.05$\pm$0.9
TLCNN (Zhou and Shi, 2017) | 86.43$\pm$1.0
PhaNet (Liu et al., 2019) | 86.5
MPCNN (Liu et al., 2018) | 86.9$\pm$0.6
RBFNN (Mahesh et al., 2021) | 88.87
CL-MEx | 94.64$\pm$0.92

(b) DDCF Method | Acc. $\pm$ SD
---|---
LBP (Khan et al., 2019) | 82.3
SVM (Albu et al., 2015) | 91.27
RBFNN (Albu et al., 2015) | 91.80
CL-MEx | 95.26$\pm$0.84

(a) KDEF

(b) DDCF

Figure 3. Confusion matrices for different expressions.

###  4.2. Results

Accuracy is measured for the downstream network which predicts the expression
labels. The input to this network is a single image (a single view of a
subject). The accuracy is therefore measured as the rate of correct
classifications over all the test images in 10-fold cross-validation.Table
1(b) compares the results of our proposed method with previous methods on KDEF
and DDCF datasets. We observe that CL-MEx outperforms other methods on these
two datasets when a single-view image is used during inference. It should be
noted that some of the prior work mentioned in the table utilize all existing
views to obtain the prediction. Figure 3(a) shows the confusion matrix for
KDEF, where we observe that our method performs best on classifying ‘Happy’
and ‘Neutral’, while the least effective performance is obtained on ‘Afraid’
and ‘Sad’. ‘Afraid’ is often confused with ‘Sad’ and ‘Surprised’, whereas
‘Sad’ is mostly confused with ‘Disgust’. In the case of the DDCF dataset, in
Figure 3(b) shows that strong performance is obtained for ‘Happy’, ‘Neutral’,
‘Pleased’, and ‘Sad’, while the method exhibits the least effectiveness on
‘Afraid’ and ‘Angry’, which are confused with ‘Surprised’ and ‘Disgust’
respectively.

One of the main contributions of our proposed method is that it is robust to
changes in viewing angles by forcing the representations obtained from
different views to come together in the embedding space. To objectively
validate this notion, we compare the deterioration of the accuracy of the
side-views with respect to the frontal view for CL-MEx in comparison to a
fully supervised version of the same network (the model is trained with all
the data and associated labels without the contrastive learning step). The
results of this experiment are presented in Figure 4, where we observe that
across all side-view angles, our method shows a smaller drop in performance
compared to supervised learning. It can be seen that as the angles become more
severe, the supervised approach exhibits a larger drop, while our method shows
less sensitivity. For example, in FL, the supervised method on KDEF has 7%
lower accuracy compared to its frontal view, whereas our method reduces that
difference to only 2.8%. In case of the less challenging views such as HL and
HR, our method shows less than 1% deviation for both of the datasets.

Figure 4. Drop in performance for different viewing angles with respect to the
frontal view, for self-supervised CL-MEx and its fully supervised counterpart.
Figure 5. Performance of self-supervised CL-MEx and its fully supervised
counterpart when different amounts of labeled data are used.

Next, we perform a sensitivity study on the proposed method for the amount of
labeled data by comparing the performance of CL-MEx to its fully supervised
counterpart when 75%, 50%, 25%, 10%, and 5% of the labels are used. The
results are presented in Figure 5, where when all the data are used, our
method shows 4% and 3% improvements over the fully supervised model for KDEF
and DDCF datasets respectively. It is then seen that as the amount of output
training labels are reduced, our method experiences smaller drops in
performance versus the fully supervised model, showing a clear advantage by
reduced reliance on the output labels.

##  5\. Conclusion and Future Work

This paper presents a self-supervised contrastive framework specialized in
learning facial expressions from multiple views. Our method uses the multi-
view images of the subjects during the pre-training and learns view-invariant
embedding representations. The model is then fine-tuned in a supervised
setting. We test our solution on KDEF and DDCF dataset and perform rigorous
experiments. The model outperforms existing methods and shows robustness
towards challenging viewing angle. Experiments also show the robustness of our
model towards reduced amounts of labeled data.For future work, the proposed
method can be used for other multi-view or multi-modal image-based
applications in self-supervised and contrastive learning settings, for example
multi-view face/activity recognition or even multi-view object recognition.

## Acknowledgements

We would like to thank BMO Bank of Montreal and Mitacs for funding this
research.

## References

  * (1)
  * Albu et al. (2015) Felix Albu, Daniela Hagiescu, Liviu Vladutu, and Mihaela-Alexandra Puica. 2015.  Neural network approaches for children’s emotion recognition in intelligent learning applications. In _7th Annu Int Conf Educ New Learn Technol Barcelona_.
  * Bachman et al. (2019) Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019\.  Learning representations by maximizing mutual information across views.  _arXiv preprint arXiv:1906.00910_ (2019).
  * Chaitanya et al. (2020) Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. 2020.  Contrastive learning of global and local features for medical image segmentation with limited annotations.  _arXiv preprint arXiv:2006.10511_ (2020).
  * Chen et al. (2019) Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert. 2019.  Self-supervised learning for medical image analysis using image context restoration.  _Medical Image Analysis_ 58 (2019), 101539.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a.  A simple framework for contrastive learning of visual representations. In _International Conference on Machine Learning_. 1597–1607.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b.  A simple framework for contrastive learning of visual representations. In _International Conference on Machine Learning_. 1597–1607.
  * Chen et al. (2020c) Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. 2020c.  Big self-supervised models are strong semi-supervised learners.  _arXiv preprint arXiv:2006.10029_ (2020).
  * Chen and He (2020) Xinlei Chen and Kaiming He. 2020.  Exploring Simple Siamese Representation Learning.  _arXiv preprint arXiv:2011.10566_ (2020).
  * Cho et al. (2019) Youngjun Cho, Simon J Julier, and Nadia Bianchi-Berthouze. 2019\.  Instant stress: detection of perceived mental stress through smartphone photoplethysmography and thermal imaging.  _JMIR Mental Health_ 6, 4 (2019).
  * Dalrymple et al. (2013) Kirsten A Dalrymple, Jesse Gomez, and Brad Duchaine. 2013\.  The Dartmouth Database of Children’s Faces: Acquisition and validation of a new face stimulus set.  _PloS One_ 8, 11 (2013), e79131.
  * Doersch et al. (2015) Carl Doersch, Abhinav Gupta, and Alexei A Efros. 2015\.  Unsupervised visual representation learning by context prediction. In _Proceedings of the IEEE International Conference on Computer Vision_. 1422–1430.
  * Dosovitskiy et al. (2015) Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. 2015.  Discriminative unsupervised feature learning with exemplar convolutional neural networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 38, 9 (2015), 1734–1747.
  * Dosovitskiy et al. (2014) Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. 2014\.  Discriminative unsupervised feature learning with convolutional neural networks. In _Proceedings of the 27th International Conference on Neural Information Processing Systems_. 766–774.
  * Gidaris et al. (2018) Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018\.  Unsupervised representation learning by predicting image rotations.  _arXiv preprint arXiv:1803.07728_ (2018).
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006\.  Dimensionality reduction by learning an invariant mapping. In _IEEE Conference on Computer Vision and Pattern Recognition_ , Vol. 2. 1735–1742.
  * Hasani et al. (2020) Behzad Hasani, Pooran Singh Negi, and Mohammad Mahoor. 2020.  BReG-NeXt: Facial affect computing using adaptive residual networks with bounded gradient.  _IEEE Transactions on Affective Computing_ (2020).
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _IEEE Conference on Computer Vision and Pattern Recognition_. 770–778.
  * Khan et al. (2019) Rizwan Ahmed Khan, Alexandre Meyer, Hubert Konik, and Saida Bouakaz. 2019.  Saliency-based framework for facial expression recognition.  _Frontiers of Computer Science_ 13, 1 (2019), 183–198.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  _arXiv preprint arXiv:2004.11362_ (2020).
  * Leng et al. (2007) H Leng, Y Lin, and LA Zanzi. 2007.  An experimental study on physiological parameters toward driver emotion recognition. In _International Conference on Ergonomics and Health Aspects of Work with Computers_. 237–246.
  * Liu et al. (2019) Yuanyuan Liu, Jiyao Peng, Jiabei Zeng, and Shiguang Shan. 2019\.  Pose-adaptive Hierarchical Attention Network for Facial Expression Recognition.  _arXiv preprint arXiv:1905.10059_ (2019).
  * Liu et al. (2018) Yuanyuan Liu, Jiabei Zeng, Shiguang Shan, and Zhuo Zheng. 2018\.  Multi-channel pose-aware convolution neural networks for multi-view facial expression recognition. In _13th IEEE International Conference on Automatic Face & Gesture Recognition_. 458–465.
  * Lundqvist et al. (1998) Daniel Lundqvist, Anders Flykt, and Arne Öhman. 1998\.  The Karolinska Directed Emotional Faces (KDEF).  _CD ROM from Department of Clinical Neuroscience, Psychology Section, Karolinska Institutet_ 91, 630 (1998), 2–2.
  * Mahesh et al. (2021) Vijayalakshmi GV Mahesh, Chengji Chen, Vijayarajan Rajangam, Alex Noel Joseph Raj, and Palani Thanaraj Krishnan. 2021.  Shape and Texture Aware Facial Expression Recognition Using Spatial Pyramid Zernike Moments and Law’s Textures Feature Set.  _IEEE Access_ 9 (2021), 52509–52522.
  * Moore and Bowden (2011) Stephen Moore and Richard Bowden. 2011.  Local binary patterns for multi-view facial expression recognition.  _Computer Vision and Image Understanding_ 115, 4 (2011), 541–558.
  * Noroozi and Favaro (2016) Mehdi Noroozi and Paolo Favaro. 2016.  Unsupervised learning of visual representations by solving jigsaw puzzles. In _European Conference on Computer Vision_. 69–84.
  * Noroozi et al. (2017) Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. 2017\.  Representation learning by learning to count. In _Proceedings of the IEEE International Conference on Computer Vision_. 5898–5906.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Pathak et al. (2016) Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016\.  Context encoders: Feature learning by inpainting. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2536–2544.
  * Pourmirzaei et al. (2021) Mahdi Pourmirzaei, Farzaneh Esmaili, and Gholam Ali Montazer. 2021.  Using Self-Supervised Co-Training to Improve Facial Representation.  _arXiv preprint arXiv:2105.06421_ (2021).
  * Rahimi Taghanaki and Etemad (2020) Setareh Rahimi Taghanaki and Ali Etemad. 2020.  Self-supervised Wearable-based Activity Recognition by Learning to Forecast Motion.  _arXiv e-prints_ (2020), arXiv–2010.
  * Rao et al. (2015) Qiyu Rao, Xing Qu, Qirong Mao, and Yongzhao Zhan. 2015\.  Multi-pose facial expression recognition based on SURF boosting. In _IEEE International Conference on Affective Computing and Intelligent Interaction_. 630–635.
  * Roy and Etemad (2021) Shuvendu Roy and Ali Etemad. 2021.  Spatiotemporal Contrastive Learning of Facial Expressions in Videos.  _arXiv preprint arXiv:2108.03064_ (2021).
  * Saeed et al. (2019) Aaqib Saeed, Tanir Özçelebi, and Johan J Lukkien. 2019\.  Multi-Task Self-Supervised Learning for Human Activity Detection.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 3, 2 (2019), 61.
  * Sanchez-Cortes et al. (2013) Dairazalia Sanchez-Cortes, Joan-Isaac Biel, Shiro Kumano, Junji Yamato, Kazuhiro Otsuka, and Daniel Gatica-Perez. 2013.  Inferring mood in ubiquitous conversational video. In _12th International Conference on Mobile and Ubiquitous Multimedia_. 1–9.
  * Sarkar and Etemad (2020a) Pritam Sarkar and Ali Etemad. 2020a.  Self-supervised ecg representation learning for emotion recognition.  _IEEE Transactions on Affective Computing_ (2020).
  * Sarkar and Etemad (2020b) Pritam Sarkar and Ali Etemad. 2020b.  Self-supervised learning for ecg-based emotion recognition. In _IEEE International Conference on Acoustics, Speech and Signal Processing_. 3217–3221.
  * Sarkar et al. (2020) Pritam Sarkar, Silvia Lobmaier, Bibiana Fabre, Gabriela Berg, Alexander Mueller, Martin G Frasch, Marta C Antonelli, and Ali Etemad. 2020\.  Detection of Maternal and Fetal Stress from ECG with Self-supervised Representation Learning.  _arXiv preprint arXiv:2011.02000_ (2020).
  * Sepas-Moghaddam et al. (2021) Alireza Sepas-Moghaddam, Fernando Pereira, Paulo Lobato Correia, and Ali Etemad. 2021.  Multi-Perspective LSTM for Joint Visual Representation Learning.  _arXiv preprint arXiv:2105.02802_ (2021).
  * Thrasher et al. (2011) Michelle Thrasher, Marjolein D Van der Zwaag, Nadia Bianchi-Berthouze, and Joyce HDM Westerink. 2011\.  Mood recognition based on upper body posture and movement features. In _International Conference on Affective Computing and Intelligent Interaction_. 377–386.
  * Tokuno et al. (2011) Shinichi Tokuno, Gentaro Tsumatori, Satoshi Shono, Eriko Takei, Taisuke Yamamoto, Go Suzuki, Shunnji Mituyoshi, and Makoto Shimura. 2011\.  Usage of emotion recognition in military health care. In _Defense Science Research Conference and Expo_. 1–5.
  * Yang et al. (2021) Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. 2021\.  Instance Localization for Self-supervised Detection Pretraining.  _arXiv preprint arXiv:2102.08318_ (2021).
  * Zhang et al. (2016) Richard Zhang, Phillip Isola, and Alexei A Efros. 2016\.  Colorful image colorization. In _European Conference on Computer Vision_. 649–666.
  * Zhou and Shi (2017) Yuqian Zhou and Bertram E Shi. 2017.  Action unit selective feature maps in deep networks for facial expression recognition. In _IEEE International Joint Conference on Neural Networks_. 2031–2038.


# RoR: Read-over-Read
for Long Document Machine Reading Comprehension

Jing Zhao1, Junwei Bao1, Yifan Wang1, Yongwei Zhou2,
Youzheng Wu1, Xiaodong He1, Bowen Zhou1
1JD AI Research, Beijing, China
2Harbin Institute of Technology, Harbin, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Corresponding Author<EMAIL_ADDRESS>
###### Abstract

Transformer-based pre-trained models, such as BERT, have achieved remarkable
results on machine reading comprehension. However, due to the constraint of
encoding length (e.g., 512 WordPiece tokens), a long document is usually split
into multiple chunks that are independently read. It results in the reading
field being limited to individual chunks without information collaboration for
long document machine reading comprehension. To address this problem, we
propose RoR, a read-over-read method, which expands the reading field from
chunk to document. Specifically, RoR includes a chunk reader and a document
reader. The former first predicts a set of regional answers for each chunk,
which are then compacted into a highly-condensed version of the original
document, guaranteeing to be encoded once. The latter further predicts the
global answers from this condensed document. Eventually, a voting strategy is
utilized to aggregate and rerank the regional and global answers for final
prediction. Extensive experiments on two benchmarks QuAC and TriviaQA
demonstrate the effectiveness of RoR for long document reading. Notably, RoR
ranks 1st place on the QuAC leaderboard 111https://quac.ai/ at the time of
submission (May 17th, 2021)222Our code is available at https://github.com/JD-
AI-Research-NLP/RoR.

##  1 Introduction

The task of machine reading comprehension (MRC), which requires machines to
answer questions through reading and understanding a given document, has been
a growing research field in natural language understanding Hermann et al.
(2015); Trischler et al. (2017); Rajpurkar et al. (2016, 2018); Joshi et al.
(2017); Choi et al. (2018).

Transformer-based pre-trained models have been widely proven to be effective
in a range of natural language processing tasks, including the MRC task Devlin
et al. (2019); Liu et al. (2019); Yang et al. (2019); Clark et al. (2020).
Typically, these models consist of a stack of transformer blocks that only
encode a length-limited sequence (e.g., 512). However, the input sequences in
some MRC tasks may exceed the length constraint. For example, each instance in
open-domain MRC usually consists of a collection of passages, such as TriviaQA
Joshi et al. (2017), one of the most popular open-domain MRC datasets,
containing 6,589 tokens on average. In addition, for conversational MRC task,
such as QuAC Choi et al. (2018), existing methods incorporate conversation
history by prepending the previous utterances to the current question, which
is packed with the document into a length input (707 tokens on average).

To handle a long document that exceeds the length constraint, a commonly used
approach is to split a document into multiple individual chunks and then
predict answers from each chunk separately. The highest scoring span in these
answers is selected as the final answer. This approach is straightforward but
results in two problems: (1) the reading field is limited to the regional
chunk instead of the complete document; and (2) the scores of the answers are
not comparable as they are not globally normalized over chunks.

To address these problems, we propose RoR, a read-over-read pipeline, which is
able to expand the reading field from chunk-level to document-level. RoR
contains a chunk reader and a document reader, both of which are based on the
pre-trained model. Specifically, the chunk reader first predicts the regional
answers from each chunk. These answers are then compacted into a new document
through a minimum span coverage algorithm guaranteeing that its sequence
length is shorter than the limitation (i.e., 512). By this means, all regional
answers can be normalized in one document. This document serves as the highly-
condensed version of the original document, which is further read by the
document reader to predict a set of global answers. As the chunk reader and
global reader provide high confidence answers from different views, we fully
leverage both of them for final answer prediction. Specifically, after
predicting the regional and global answer spans, a voting strategy is proposed
and utilized to rerank them. This voting strategy is based on the idea that a
candidate regional or global answer span overlapped more with the others is
more likely to be correct.

The contributions are summarized as follows:

  * •

We propose a read-over-read pipeline containing an enhanced chunk reader and a
document reader, which is able to solve the problem of long document reading
limitation in existing models.

  * •

We propose a voting strategy to rerank the answers from regional chunks and a
condensed document, overcoming the major drawback in aggregating the answers
from different sources.

  * •

Extensive experiments on long document benchmarks are conducted to verify the
effectiveness of our model. Especially on the QuAC dataset, our model achieves
state-of-the-art results over all evaluation metrics on the leaderboard.

##  2 Related Work

MRC is a fundamental task in natural language understanding that aims to
determine the correct answers to questions after reading a given passage
Hermann et al. (2015); Trischler et al. (2017); Rajpurkar et al. (2016, 2018).
The best performing models in various MRC tasks are commonly based on the pre-
trained language models (PLMs) within the typical encoding limit of 512
tokens. However, the input sequence in some MRC tasks usually exceeds the
length limit, such as conversational MRC and open-domain MRC.

Conversational MRC, which extends the traditional single-turn MRC, requires
the models to additionally understand the conversation history Reddy et al.
(2019); Choi et al. (2018); Gao et al. (2018); Huang et al. (2019); Gupta et
al. (2020) as dialog and conversational recommendation systems Lu et al.
(2021). A straightforward but effective approach of modeling the history is to
prepend the previous dialogs to the current question, which will compose a
lengthy input sequence with the relatively long document Gong et al. (2020).

Open-domain MRC is a task of answering questions using a large collection of
passages Joshi et al. (2017); Dunn et al. (2017); Kwiatkowski et al. (2019).
The main challenge of this task is that the sequence length of multiple
passages relevant to each question far exceeds the length limit of 512 tokens.
For example, documents in TriviaQA Joshi et al. (2017) contain 6,589 tokens on
average.

To enable the PLMs to encode long documents, a common approach is to chunk the
document into overlapping chunks of length 512, then process each chunk
separately, which inevitably causes the two problems aforementioned. Another
intuitive approach is to increase the encoding length of the PLMs. For
example, the recently proposed PLMs Longformer Beltagy et al. (2020) and Big
bird Zaheer et al. (2021), specifically for long document modeling, have
extended the encoding length from 512 to 4,096. However, their encoding length
is fixed. The two problems caused by chunking still exist when encoding the
sequences longer than 4,096. In contrast, our proposed model RoR is flexible
which is able to encode sequences of arbitrary length. Moreover, RoR is
assembleable and its encoder can be replaced with any PLMs, such as BERT and
Longformer.

Theoretically, hierarchical models can be adapted to long document MRC Yang et
al. (2016); Wang et al. (2018); Yang et al. (2020). However, deploying the
large transformer-based PLMs as the encoders of hierarchical models can be
prohibitively costly. Typically, hierarchical models parallelly encode the
splitted chunks of a long document with multiple transformers, which requires
extremely large GPU support. In contrast, RoR only needs to read a chunk at
each encoding process, then gradually predict all answers from chunk to
document. Therefore, RoR is able to deal with a long document without
consuming too much computing resources and can be more widely used than
hierarchical models.

Figure 1: The architecture of the proposed Read-over-Read pipeline.

##  3 Approach

###  3.1 Task Formulation

Given a document $P$, a question $q$, the task of MRC is to predict an answer
span $y$ from $P$ based on the comprehension of $P$ and $q$. If $q$ is an
unanswerable question, the QuAC dataset requires the model to give an
unanswerable tag as the final answer. To model the dialog history in QuAC, we
prepend previous pairs of (question, answer) to the current question to form
the question $q$. Formally, $q=[H_{k};\mathtt{[SEP]};q_{k}]$, where $q_{k}$ is
$k$-th question and $H_{k}$ is dialog history.

Additionally, a special task in the QuAC dataset is dialog act prediction.
QuAC provides two dialog acts, namely, continuation (Follow up) and
affirmation (Yes/No). The continuation dialog act consists of three possible
labels (follow up, maybe follow up or don’t follow up). The affirmation dialog
act also consists of three labels (yes, no or neither). Both two dialog act
predictions are three-label classification tasks.

###  3.2 Framework Overview

The architecture of our proposed read-over-read pipeline is illustrated in
Figure 1. RoR includes a chunk reader and a document reader, both of which
employ the PLM as the text encoder. Given a data sample ($P$, $q$), we split
it into multiple chunk-based sample {$(P_{1},q),...,(P_{N},q)$} with slide-
window, where $N$ is the number of split chunks. The chunk reader first
predicts a set of regional answers
$\\{\\{r_{i,j}\\}^{T}_{j\mbox{=}1}\\}^{N}_{i\mbox{=}1}$ from all chunks, where
$T$ is the max number of the predicted answers for one chunk. The regional
answers are then compacted to a new document $P^{q}$ by a minimum span
coverage algorithm (MSC). Notably, most of the answers in TriviaQA dataset are
named entities that cannot reflect enough contextual information. Therefore,
for the TriviaQA dataset, we use the sentences where the regional answers are
located to compact to $P^{q}$. $P^{q}$ is the condensed version of original
document $P$ to the question $q$, which is packed into the text encoder at
once. $P^{q}$ is further read by the document reader to predict the global
answers $\\{g_{i}\\}^{T}_{i\mbox{=}1}$. The answers from all chunks and the
condensed document are aggregated together and reranked by a voting strategy
to choose the final answer.

###  3.3 Chunk Reader

The chunk reader predicts the regional answers for each chunk based on the
contextualized representations of a given document which are obtained by the
pre-trained encoder. This section introduces the components of the chunk
reader in detail.

####  3.3.1 Text Encoder

The goal of a text encoder is to convert the input sequence into a series of
contextualized feature representations $\\{\mathbf{h}_{i}\\}^{L}_{i=1}$, where
$L$ is the length of input sequence. The input sequence of the encoder
contains a chunk $P$, the question $q$ which are concatenated to one token
sequence with a special splitter $\mathtt{[SEP]}$, represented as
$\mathtt{X}=[\mathtt{[CLS]};q;\mathtt{[SEP]};P]$, where $\mathtt{X}$ is the
input token sequence. the representation of $\mathtt{[CLS]}$ is treated as the
sentence-level feature for the sentence classification tasks, i.e.
answerability and dialog acts prediction.

####  3.3.2 Answer Prediction

The answers prediction of conversational MRC requires two levels of feature,
token level feature for predicting answer span and sentence level feature for
predicting dialog acts and answerability. Open-domain MRC only requires token
level feature.

Token level answer. The encoder representations
$\\{\mathbf{h}_{i}\\}^{L}_{i=1}$ serve as the token level features, which are
used to compute the probability of each token being the begin or the end of an
answer span. Concretely, $\\{\mathbf{h}_{i}\\}^{L}_{i=1}$ are projected onto
start logit and end logit through multi-layer perceptrons separately, which
are then sent to a softmax function to compute the start and end probability
distributions along all tokens in this sequence. The probabilities of each
token being the begin and the end of predicted span are calculated as follows:

$\displaystyle r^{s}_{i}$
$\displaystyle=\mathbf{W}^{s}_{2}\mathtt{tanh}(\mathbf{W}^{s}_{1}\mathbf{h}_{i})$
(1) $\displaystyle r^{e}_{i}$
$\displaystyle=\mathbf{W}^{e}_{2}\mathtt{tanh}(\mathbf{W}^{e}_{1}[\mathbf{h}_{i};\mathbf{h}_{s}])$
(2) $\displaystyle p^{s}$ $\displaystyle=\mathtt{softmax}(r^{s})$ (3)
$\displaystyle p^{e}$ $\displaystyle=\mathtt{softmax}(r^{e})$ (4)

where
$\mathbf{W}^{s}_{1},\mathbf{W}^{s}_{2},\mathbf{W}^{e}_{1},\mathbf{W}^{e}_{2}$
are trainable parameters of the projection function. $\mathbf{h}_{s}$ is the
token representation of the start label. $p^{s},p^{e}$ are the start and the
end probability distributions over all tokens respectively, where
$p^{s}\subseteq\mathbb{R}^{L},p^{e}\subseteq\mathbb{R}^{L}$. Different from
predicting start and end independently, we explicitly model the relation
between them. As shown in Equation 2, the calculation of end distribution
depends on start position. The training objective of token level prediction is
defined as the cross entropy loss of start and end predictions:

$\displaystyle\mathcal{L}_{t}=-\frac{1}{M}\sum_{j=1}^{M}[log(p^{s}_{y_{j}^{s}})+log(p^{e}_{y_{j}^{e}})]$
(5)

where $y_{j}^{s}$ and $y_{j}^{e}$ are the ground-truth of start and end
positions of $j\mbox{-}$th example respectively. $M$ is the number of
examples.

Sentence level answer. Encoder representation of $\mathtt{[CLS]}$ token
$\mathbf{h}_{[\mathtt{CLS}]}$ is viewed as the sentence level feature, which
is used to predict dialog acts and answerability by:

$\displaystyle p^{f}$
$\displaystyle=\delta(\mathbf{W}^{f}_{2}\mathtt{tanh}(\mathbf{W}^{f}_{1}\mathbf{h}_{[\mathtt{CLS}]})$
(6) $\displaystyle p^{y}$
$\displaystyle=\delta(\mathbf{W}^{y}_{2}\mathtt{tanh}(\mathbf{W}^{y}_{1}\mathbf{h}_{[\mathtt{CLS}]})$
(7) $\displaystyle p^{u}$
$\displaystyle=\sigma(\mathbf{W}^{u}_{2}\mathtt{tanh}(\mathbf{W}^{u}_{1}\mathbf{h}_{[\mathtt{CLS}]})$
(8)

where
$\mathbf{W}^{f}_{1},\mathbf{W}^{f}_{2},\mathbf{W}^{y}_{1},\mathbf{W}^{y}_{2},\mathbf{W}^{u}_{1},\mathbf{W}^{u}_{2}$
are trainable parameters. $\delta$ is a softmax function. $\sigma$ is a
sigmoid function. $p^{f},p^{y}$ are prediction distributions of continuation
and affirmation respectively, where
$p^{f}\subseteq\mathbb{R}^{3},p^{y}\subseteq\mathbb{R}^{3}$. $p^{u}$ is the
prediction score of answerability. Their corresponding cross entropy losses
are defined as:

$\displaystyle\mathcal{L}_{ct}$
$\displaystyle=-\frac{1}{M}\sum_{j=1}^{M}[logp^{f}_{y_{j}^{ct}}]$ (9)
$\displaystyle\mathcal{L}_{af}$
$\displaystyle=-\frac{1}{M}\sum_{j=1}^{M}[logp^{y}_{y_{j}^{af}}]$ (10)
$\displaystyle\mathcal{L}_{na}$
$\displaystyle=-\frac{1}{M}\sum_{j=1}^{M}[y^{na}_{j}logp^{u}+(1-y^{na}_{j})log(1-p^{u})]$
(11)

where $y_{j}^{ct},y_{j}^{af},y_{j}^{na}$ are the ground-truths of
continuation, affirmation and answerability respectively. The training
objective of sentence level prediction is defined as:

$\displaystyle\mathcal{L}_{s}=\mathcal{L}_{ct}+\mathcal{L}_{af}+\mathcal{L}_{na}$
(12)

####  3.3.3 Answer Calibration

The highest scoring span among the regional answers is sometimes not the span
with the highest F1 score. Motivated by this issue, we introduce an answer
calibration mechanism, with the goal of predicting more accurate regional
answers. Particularly, given the answer candidates, we first compute their
span representation, which is a weighted self-aligned vector:

$\displaystyle\alpha^{t}$
$\displaystyle=\mathtt{softmax}(\mathbf{W}_{r}\mathbf{h}_{s_{t}:e_{t}})$ (13)
$\displaystyle\mathbf{c_{t}}$
$\displaystyle=\sum_{j=s_{t}}^{e_{t}}\alpha^{t}_{j}\mathbf{h}_{j}$ (14)

where $\mathbf{W}_{r}$ is a trainable parameter. $\mathbf{h}_{s_{t}:e_{t}}$ is
a shorthand for stacking a list of vectors $\mathbf{h}_{j}$ $(s_{t}\leqslant
j\leqslant e_{t})$. $s_{t},e_{t}$ are the start and end of the $t$-th answer
candidate. $\mathbf{c}_{t}$ is the span representation of the $t$-th answer
candidate. Then, all candidate representations are transferred to a multi-head
self-attention layer ($\mathtt{Multi\mbox{-}SelfAtt}$) to capture the
similarities and the differences among them, and the detailed calculation is
shown as:

$\displaystyle\mathbf{c}_{t}$ $\displaystyle=\mathbf{c}_{t}+\mathtt{Emb}(t)$
(15) $\displaystyle\mathbf{c^{{}^{\prime}}}$
$\displaystyle=\mathtt{Multi\mbox{-}SelfAtt}(\mathbf{c})$ (16)
$\displaystyle\beta$
$\displaystyle=\mathtt{softmax}(\mathtt{tanh}(\mathbf{W}_{o}\mathbf{c^{{}^{\prime}}}_{0:T}))$
(17)

where $\mathtt{Emb}$ is the position embedding of $t$ and a smaller value $t$
means a higher original prediction score, which is a valuable feature for the
model to identify the different importance of candidates. $\mathbf{c}$ is list
of candidate representations, formally
$\mathbf{c}=[\mathbf{c}_{1},...,\mathbf{c}_{T}]$. $\mathbf{W}_{o}$ is
trainable parameter. $\beta$ is the distribution of calibration score over the
answer candidates, $\beta\subseteq\mathbb{R}^{T}$. The cross entropy loss for
answer calibration is given as:

$\mathcal{L}_{ac}=-\frac{1}{T}\sum_{j=1}^{T}[log\beta_{y_{j}^{ac}}]$ (18)

where $y_{j}^{ac}$ is a manually constructed label. We define $y_{j}^{ac}$ is
the candidate which obtains the highest F1 score with the gold span among all
candidates. If the highest F1 score is zero and the corresponding question is
answerable, we randomly replace a candidate with the gold span.

###  3.4 Document Reader

The predicted spans from different chunks are compacted to a new text $P^{q}$
with a designed Minimum Span Coverage (MSC) algorithm, as shown in Algorithm
1. MSC guarantees that $P^{q}$ covers all regional spans and is sufficiently
condensed to be encoded once.

Algorithm 1 Minimum Span Coverage

Input:  A = $\\{a_{i}\\}^{NT}_{i=1}$
A is the set of regional spans from all chunks
$N$ is the number of chunks, $T$ is the number of regional spans for one chunk

1:for $a_{i}$ in A do

2: for $a_{j,j\neq i}$ in A do

3: if $\mathtt{overlap}$($a_{i}$, $a_{j}$) then

4: A += $\mathtt{coverage}$($a_{i}$, $a_{j}$)

5: A -= $a_{i}$, A -= $a_{j}$

6:$\mathtt{coverage}$($a_{i}$,$a_{j}$) is the span corresponding to
(start,end) = $(\mathtt{min}(s_{i},s_{j}),\mathtt{max}(e_{i},e_{j}))$, where
$(s_{i},e_{i})$ and $(s_{j},e_{j})$ are (start,end) of $a_{i}$ and $a_{j}$.

7:Recursively execute the above steps until no condition of
$\mathtt{overlap}$($a_{i}$,$a_{j}$)

8:Concatenate the elements in A together as $P^{q}$

9:return $P^{q}$

The input sequence of the text encoder is
$\mathtt{X}=[\mathtt{[CLS]};q;\mathtt{[SEP]};P^{q}]$, which is further read by
the document reader to predict the answers as the global answers. The span
label of the document reader is the longest common substring between $P^{q}$
and the original gold span.

The global answers and the regional answers are aggregated as final
predictions. For answerability prediction, the document reader predicts a
global no answer score $U_{g}$ and the chunk reader predicts a series of no
answer score $U_{r}=\\{u_{k}\\}_{k=1}^{N}$, where $u_{k}$ is the predicted no
answer score of $k$-th chunk. The final no answer score $\mathcal{S}_{na}$ is
defined as:

$\displaystyle\mathcal{S}_{na}$ $\displaystyle=\lambda
U_{g}+(1-\lambda)\mathtt{min}(U_{r})$ (19)

where $\lambda$ is a hyperparameter to tune the weights of the global answers
and the regional answers.

###  3.5 Voting Strategy

After aggregating the answer spans, we re-score them with a voting strategy
that is based on a hypothesis: the spans predicted by both the chunk reader
and the document reader are more likely to be correct. This strategy allows
all spans to vote with each other to choose the most common span. Concretely,
the voting score of each span is obtained by:

$\displaystyle\mathtt{Voting}(x_{i})$
$\displaystyle=\frac{1}{T\mbox{-}1}\sum_{j=1}^{T}(\mathtt{F1}(x_{i},x_{j,j\neq
i}))$ (20) $\displaystyle\mathtt{F1}(x_{i},x_{j})$
$\displaystyle=\frac{2R(i,j)P(i,j)}{R(i,j)+P(i,j)}$ (21) $\displaystyle
R(i,j)$ $\displaystyle=\frac{|x_{i}\cap x_{j}|}{|x_{j}|}$ (22) $\displaystyle
P(i,j)$ $\displaystyle=\frac{|x_{i}\cap x_{j}|}{|x_{i}|}$ (23)

where $|\ |$ denotes the number of words, $|x_{i}\cap x_{j}|$ denotes the
number of the common words between $x_{i}$ and $x_{j}$. The function of
$\mathtt{F1}(x_{i},x_{j})$ represents the sequence similarity between $x_{i}$
and $x_{j}$. A larger voting score means that the corresponding span is
similar to more candidates than the others. Finally, the voting strategy
reranks answer spans according to the original prediction score
$\mathcal{S}(x)$ and the voting score:

$\displaystyle\mathtt{score}(x)=\gamma\mathcal{S}(x)+(1-\gamma)\mathtt{Voting}(x)$
(24)

$\gamma$ is the weight of two scores.

###  3.6 Training and Inference

We adopt the multi-task learning idea to jointly learn the predictions of
answer span, answerability and dialog acts. All parameters are trained with an
end-to-end manner. The training loss of the chunk reader $L_{c}$ and the
document reader $L_{d}$ are:

$\displaystyle\mathcal{L}_{c}$
$\displaystyle=\mathcal{L}_{t}+\mathcal{L}_{s}+\mathcal{L}_{ac}$ (25)
$\displaystyle\mathcal{L}_{d}$ $\displaystyle=\mathcal{L}_{t}+\mathcal{L}_{s}$
(26)

The detailed training and inference processes are given in Algorithm 2.

Algorithm 2 Training and Inference Process

Input:  D = training set, d = test set
Initialize: $\Theta_{1},\Theta_{2}$ $\leftarrow$ pre-trained parameters

1:Training

2:Train $\Theta_{1}$ on D with $\mathcal{L}_{c}$, then predict answers on D to
construct a new dataset D${}^{{}^{\prime}}$ through Algorithm 1

3:Train $\Theta_{2}$ on D${}^{{}^{\prime}}$ with $\mathcal{L}_{d}$

4:Inference

5:$\Theta_{1}$ predict the regional answers set $A$ on d and construct a new
dataset d${}^{{}^{\prime}}$ through Algorithm 1

6:$\Theta_{2}$ predict the global answers set $A^{{}^{\prime}}$ on
d${}^{{}^{\prime}}$

7:Aggregate and rerank the final answers set $\widetilde{A}=A\cup
A^{{}^{\prime}}$ with Equation 24

###  3.7 Experiment Setup

####  3.7.1 Dataset

Our experiments are mainly conducted on two long document datasets QuAC
(Question Answering in Context) Choi et al. (2018) and TriviaQA Joshi et al.
(2017). QuAC is a large-scale dataset created for simulating information-
seeking conversations. Its questions are often more open-ended, unanswerable,
or only meaningful within the dialog context. TriviaQA is a large-scale open-
domain MRC dataset, which requires cross sentence reasoning to find answers.
It contains data from Wikipedia and Web domains, where Wikipedia subset is
used in our work. The statistic information of these two dataset is summarized
in Table 1.

| Train | Dev | Test
---|---|---|---
TriviaQA |  |  |
# questions | 61,888 | 7,993 | 7,701
# tokens / input | 11,222 | 11,382 | -
QuAC |  |  |
# questions | 83,568 | 7,354 | 7,353
# tokens / input | 641 | 707 | -
# dialogs | 11,567 | 1000 | 1002
# question / dialog | 7.2 | 7.4 | 7.4
% unanswerable | 20.2 | 20.2 | 20.1
Table 1: Statistics of two datasets. # denote the number of each item. %
denote a percentage value.

####  3.7.2 Evaluation Metrics

For answer span prediction, the QuAC challenge provides two evaluation
metrics, the word-level F1 and the human equivalence score (HEQ). The word-
level F1 measures the overlap of the prediction and the gold span after
removing stopwords. HEQ measures the percentage of examples for which model F1
score is higher than the average human F1 score. HEQ contains two variants
HEQ-Q and HEQ-D. HEQ-Q is 1 if model performance exceeds the human performance
for each question. HEQ-D is 1 if model performance of all the questions in the
dialog exceeds human. For dialog act prediction, the accuracy is adopted as
evaluation metric. For the TriviaQA dataset, word-level F1 score and exact
match (EM) are used as evaluation metrics.

####  3.7.3 Implementation Details

We tried three different PLMs, BERT-large 333https://github.com/google-
research/BERT, ELECTRA-large 444https://github.com/google-research/electra and
Longformer-large 555https://github.com/allenai/Longformer as initialization
parameters of text encoder to verify the effectiveness of RoR comprehensively.
The max sequence length of questions is set to 128 and the answer length is
set to 64. The stride of the sliding window for splitting documents is set to
128. The batch size is set to 12. The model is optimized using Adam Kingma and
Ba (2015) with learning rate = 2e-5, maximal gradient clipping = 1.0. The
hyperparameter $\lambda$ is set to 0.9, $\gamma$ is set to 0.5. In the
inference process, we use beam search to predict end position based on start
position and the beam size is 5. The decision of answerability depends on the
numerical comparison between the no answer score $\mathcal{S}_{na}$ in
equation 19 and a threshold $\zeta$, which is set to 0.3. If
$\mathcal{S}_{na}$ is higher than $\zeta$, the corresponding question is
unanswerable.

Model | F1 | HEQ-Q | HEQ-D | Answerability | Continuation | Affirmation
---|---|---|---|---|---|---
Longformer | 74.3 | 71.5 | 14.5 | 76.5 | 64.4 | 89.8
BERT | 67.4 | 63.7 | 7.9 | 68.3 | 63.3 | 88.4
BERT-RoR | 69.6 | 65.8 | 9.8 | 74.7 | 63.5 | 88.6
ELECTRA | 73.8 | 71.2 | 14.3 | 76.1 | 64.9 | 89.7
ELECTRA-RoR | 75.7 | 73.4 | 17.8 | 78.2 | 65.0 | 90.0
Table 2: Results on the development set of QuAC.

MSC algorithm is important in RoR which guarantees the condensed form of an
arbitrarily long input sequence shorter than 512 tokens through limiting the
total number and length of the regional answers. In practice, we perform the
following four operations on TriviaQA and QuAC:

  * •

The max number of answer candidates for each chunk $T$ is set to 5.

  * •

A document is split up to 7 chunks in QuAC and 15 chunks in TriviaQA.

  * •

The regional answer is truncated if longer than 15 tokens.

  * •

Many regional answers overlap or even differ by a few words. MSC algorithm
removes the duplicate words to ensure the condensed document shorter.

After the operations above, the longest condensed document contains 471 tokens
in TriviaQA and 184 tokens in QuAC. When RoR is adapted to other datasets, the
length of the condensed documents can be guaranteed to be shorter than 512 as
long as the parameters in the above four operations are adjusted
correspondingly.

In order to improve the model performance, some data augmentations are applied
to better train the model. Specifically, ELECTRA is fine-tuned on other MRC
datasets before fine-tuned on QuAC, such as SQuAD Rajpurkar et al. (2018) and
CoQA Reddy et al. (2019), hoping to transfer the knowledge in other datasets
to our model. Experimental results show that CoQA has a much higher lifting
effect than SQuAD. This is because both CoQA and QuAC are conversational MRC
datasets, while SQuAD is a single-turn MRC dataset. The answers in CoQA are
free-form and generally short (average answer length = 2.7), which is quite
different from QuAC (average answer length = 15.1). As a result, we choose the
rationale sentence of the gold span in CoQA as the prediction target.

###  3.8 Main Results

Results on QuAC. Table 2 displays the experimental results on the development
set of QuAC. In view of the fact that the sequence length in QuAC dataset does
not exceed the encoding length limit of Longformer (i.e., 4096), we did not
apply RoR to the Longformer. The results show that RoR significantly improves
the performance of PLMs on all evaluation metrics, illustrating the
effectiveness of RoR on long document modeling. Among the three PLMs,
Longformer, which can encode longer sequences, performs best, followed by
ELECTRA. Nevertheless, with the enhancement of RoR, ELECTRA-RoR outperforms
Longformer and achieves state-of-the-art results over all metrics on the dev
set of QuAC.

Model | F1 | HEQ-Q | HEQ-D
---|---|---|---
Human | 81.1 | 100 | 100
ELECTRA-RoR | 74.9 | 72.2 | 16.4
EL-QA | 74.6 | 71.6 | 16.3
History QA | 74.2 | 71.5 | 13.9
TR-MT | 74.4 | 71.3 | 13.6
GraphFlow Chen et al. (2020) | 64.9 | 60.3 | 5.1
HAM Qu et al. (2019b) | 65.4 | 61.8 | 6.7
FlowDelta Yeh and Chen (2019) | 65.5 | 61.0 | 6.9
HAE Qu et al. (2019a) | 62.4 | 57.8 | 5.1
FlowQA Huang et al. (2019) | 64.1 | 59.6 | 5.8
BiDAF++ w/ 2-Context | 60.1 | 54.8 | 4.0
BiDAF++ Peters et al. (2018) | 50.2 | 43.3 | 2.2
Table 3: Test results on QuAC with sample methods on the leaderboard https://quac.ai/. Model | F1 | HEQ-Q | HEQ-D | Answerability | Continuation | Affirmation
---|---|---|---|---|---|---
chunk reader | 73.8 | 71.2 | 14.3 | 76.1 | 64.9 | 89.7
w/ calibration | 74.1 | 71.4 | 15.3 | 76.9 | 64.6 | 89.7
w/ document reader | 74.8 | 72.1 | 15.7 | 77.4 | 64.7 | 89.7
w/ voting strategy | 75.4 | 72.9 | 16.9 | 77.4 | 64.7 | 89.7
w/ knowledge transfer | 75.7 | 73.4 | 17.8 | 78.2 | 65.0 | 90.0
Table 4: Ablation study of ELECTRA-RoR on the development set of QuAC.

Official leaderboard results on QuAC. QuAC challenge provides a hidden test
set, where the dialog acts prediction is not the main task of the leaderboard
and their evaluation scores are not considered in the final model ranking.
Table 3 displays the span prediction results of all baselines and our model,
from which we can see that our model ELECTRA-RoR outperforms the previous best
performing model EL-QA and achieves new state-of-the-art on all three metrics.
From the results of the leaderboard, we observe that the top ranking models
almost all use the advanced pre-trained models, such as ELECTRA, RoBERTa and
BERT. Although some models have a well-designed model structure, they still
lag behind the models that uses the pre-trained models as encoder, showing the
powerful modeling capabilities of the pre-trained model. For example,
FlowDelta boosts the F1 score of FlowQA from 64.1 to 65.5 with the help of
BERT. Compared to BiDAF++, BiDAF++ w/ 2-Context incorporates two turns of
previous dialog history and significantly improves the performance of BiDAF++,
verifying the importance of historical information in the conversational MRC.

Results on TriviaQA. Table 5 displays the experimental results on the TriviaQA
dataset. The experimental results show that RoR is able to comprehensively
improve the score of the PLMs and the average F1 gain is 2.1, proving that RoR
is also highly effective in the task of open-domain MRC.

Similar to the performance on the QuAC dataset, among the three PLMs,
Longformer performs best on the TriviaQA dataset, ELECTRA followed, and the
worst is BERT. Although Longformer has such excellent performance which used
to be state-of-the-art model on the TriviaQA dataset, RoR still further
improve its performance (F1 score from 77.8 to 80.0). The test results show
that all improvements of RoR in Table 3 and Table 5 are statistically
significant (paired t-test, p-value < 0.01).

Model | F1 | EM
---|---|---
BERT | 68.4 | 60.7
BERT-RoR | 70.3 | 62.1
ELECTRA | 70.6 | 65.3
ELECTRA-RoR | 72.9 | 67.8
Longformer | 77.8 | 73.0
Longformer-RoR | 80.0 | 75.0
Table 5: Results on the TriviaQA dataset.

###  3.9 Ablation Test

We conduct an ablation analysis on development set of QuAC to investigate the
contributions of each module of the best model ELECTRA-RoR.

Table 4 displays the results of ablated systems, where we gradually add the
proposed modules to the model structure. It can be observed that answer
calibration mechanism boosts the performance of the chunk reader in all three
span evaluation metrics. An interesting phenomenon is the calibration
mechanism has the ability to improve the prediction accuracy of unanswerable
questions. This may be because we mask some training instances during
calculating the calibration loss $\mathcal{L}_{ac}$ if their questions are
unanswerable, since all span candidates are not correct. This may provide some
supervision signals for model training to identify unanswerable questions.
Next we analyze the contributions of the document reader. It can be seen that
the evaluation scores of span prediction are further improved when adding the
document reader. Especially the F1 score is improved by 0.7. Moreover, the
accuracy of answerability prediction is also improved. This is because the
document reader predicts a global no answer score which contributes to the
final decision of answerability through equation 19. Afterwards, the ablation
results show that the voting strategy yields substantial improvement over the
RoR model. Finally, we can see that the transfer module could also
comprehensively enhance the model performance, proving the efficiency of
knowledge transfer in MRC task.

In terms of continuity and affirmation, the accuracy scores do not change much
in the ablation experiments. Nevertheless, we cannot completely ignore them
when training our model, as we find that the performance of RoR will drop if
remove the losses of $\mathcal{L}_{ct}$ and $\mathcal{L}_{af}$. This reflects
that the dialog acts can indeed encourage the encoder to produce more generic
representations that benefit the answer span prediction task.

###  3.10 Discussion on Answer Calibration

As shown in Table 4, the answer calibration mechanism improves the performance
of the chunk reader. This section further discusses its other promoting
effects on RoR. The predicted regional answers determine the quality of global
answers that will be predicted by the document reader. An ablation test on the
condensed document $P^{q}$ is conducted on the dev set. We find that the
average F1 score of $P^{q}$ drops from 30.3 to 29.7 if without the answer
calibration. This test validates that the calibration mechanism is able
improve the input accuracy of the document reader. Meanwhile, the quality of
global answers are further improved.

We also attempt to integrate the answer calibration mechanism into the
document reader, but the results show that the improvement is not significant.
We speculate the reason is that the predictions of the document reader is
sufficiently accurate.

Figure 2: The influence of weight.

###  3.11 Influence of Weight Parameter

In the voting strategy, the parameter $\gamma$ weights the prediction score
and the voting score. In the process of answers aggregation, $\lambda$ weights
the score of the regional answers and the global answers. This section
explores the influence of these two parameters on the model.

Figure 4 displays the F1 and mean average precision (MAP) curves of ELECTRA-
RoR on QuAC with respect to different $\lambda$ and $\gamma$. The results
suggest that the model performance are sensitive to the weights. As the
weights increases, both curves show a trend of increasing at first and then
decreasing, reaching peaks at 0.5 and 0.9 respectively. For the changing
degree of the curves, $\gamma$ actually has a greater influence on RoR than
$\lambda$. We notice that MAP score dramatically declines when reducing the
weight of the voting score (i.e., $\gamma$ from 0.5 to 1.0), indicating that
the voting score is a reliable basis to rerank the final prediction results.

##  4 Conclusion

In this work, a read-over-read (RoR) pipeline is proposed for long document
MRC, which contains an enhanced chunk reader to predict the regional answers
and a document reader to predict the global answers. Moreover, a voting
strategy is designed to optimize the process of answer aggregation in RoR.
Comprehensive empirical studies on QuAC and TriviaQA demonstrate the
effectiveness of RoR, which comprehensively improves the performances of the
PLMs on long document reading. Meanwhile, ELECTRA-RoR achieves state-of-the-
art over all evaluation metrics on QuAC leaderboard.

##  5 Acknowledge

We would like to thank three anonymous reviewers for their useful feedback. We
also sincerely appreciate He He and Mark Yatskar for their help in evaluating
our models on QuAC dataset. This work is supported by the National Key
Research and Development Program of China under Grant No. 2020AAA0108600.

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.
  * Chen et al. (2020) Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020.  Graphflow: Exploiting conversation flow with graph neural networks for conversational machine comprehension.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pages 1230–1236. ijcai.org.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  QuAC: Question answering in context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dunn et al. (2017) Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017.  Searchqa: A new q&a dataset augmented with context from a search engine.
  * Gao et al. (2018) Jianfeng Gao, Michel Galley, and Lihong Li. 2018.  Neural approaches to conversational AI.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, pages 1371–1374. ACM.
  * Gong et al. (2020) Hongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen, and Dong Yu. 2020.  Recurrent chunking mechanisms for long-text machine reading comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6751–6761, Online. Association for Computational Linguistics.
  * Gupta et al. (2020) Somil Gupta, Bhanu Pratap Singh Rawat, and Hong Yu. 2020.  Conversational machine comprehension: a literature review.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2739–2753, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Huang et al. (2019) Hsin-Yuan Huang, Eunsol Choi, and Wen-tau Yih. 2019.  Flowqa: Grasping flow in history for conversational machine comprehension.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Lu et al. (2021) Yu Lu, Junwei Bao, Yan Song, Zichen Ma, Shuguang Cui, Youzheng Wu, and Xiaodong He. 2021.  RevCore: Review-augmented conversational recommendation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1161–1173, Online. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Qu et al. (2019a) Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019a.  BERT with history answer embedding for conversational question answering.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_ , pages 1133–1136. ACM.
  * Qu et al. (2019b) Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W. Bruce Croft, and Mohit Iyyer. 2019b.  Attentive history selection for conversational question answering.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019_ , pages 1391–1400. ACM.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  NewsQA: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
  * Wang et al. (2018) Yizhong Wang, Kai Liu, Jing Liu, Wei He, Yajuan Lyu, Hua Wu, Sujian Li, and Haifeng Wang. 2018.  Multi-passage machine reading comprehension with cross-passage answer verification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1918–1927, Melbourne, Australia. Association for Computational Linguistics.
  * Yang et al. (2020) Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. 2020.  Beyond 512 tokens: Siamese multi-depth transformer-based hierarchical encoder for long-form document matching.  In _CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020_ , pages 1725–1734. ACM.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1480–1489, San Diego, California. Association for Computational Linguistics.
  * Yeh and Chen (2019) Yi-Ting Yeh and Yun-Nung Chen. 2019.  FlowDelta: Modeling flow information gain in reasoning for conversational machine comprehension.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 86–90, Hong Kong, China. Association for Computational Linguistics.
  * Zaheer et al. (2021) Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2021.  Big bird: Transformers for longer sequences.