# A Dataset for Discourse Structure in Peer Review Discussions

Neha Nayak Kennard Tim O’Gorman Akshay Sharma
Chhandak Bagchi Matthew Clinton Pranay Kumar Yelugam
Rajarshi Das Hamed Zamani Andrew McCallum
{kennard, togorman, akshaysharma, cbagchi, mfclinton
pyelugam, rajarshi, zamani<EMAIL_ADDRESS>
###### Abstract

At the foundation of scientific evaluation is the labor-intensive process of
peer review. This critical task requires participants to consume and interpret
vast amounts of highly technical text. We show that discourse cues from
rebuttals can shed light on the quality and interpretation of reviews.
Further, an understanding of the argumentative strategies employed by the
reviewers and authors provides useful signal for area chairs and other
decision makers.

This paper presents a new labeled dataset of 20k sentences contained in 506
review-rebuttal pairs in English, annotated by experts. While existing
datasets annotate a subset of review sentences using various schemes, ours
synthesizes existing label sets and extends them to include fine-grained
annotation of the rebuttal sentences, characterizing the authors’ stance
towards the reviewers’ criticisms and their commitment to addressing them.
Further, we annotate every sentence in both the review and the rebuttal,
including a description of the context for each rebuttal sentence.

##  1 Introduction

Peer review lies at the foundation of scientific evaluation, performing the
essential role of quality control in the dissemination of scientific
knowledge. The rapid increase in academic output places an immense burden on
conference area chairs, journal editors, and other decision makers. Decisions
must take into account not only extensive manuscripts, but enormous additional
amounts of text including reviews, rebuttals, and other discussions. Many of
the solutions to peer review’s various shortcomings require systemic change
and institutional buy-in across international communities (Church, 2020;
Nobarany, 2014; Ko, 2019). However, any such effort could be aided by a deeper
understanding of the interaction between the participants in the discussion.
This paper describes a dataset that enables the study of the discourse
structure of these interactions.

Figure 1: A depiction of our annotation scheme on a minimal, fictional review-
rebuttal pair. Sentences in the review are annotated with a review action;
they may also have additional labels that indicate the qualities of the
manuscript being commented upon (aspect) as well as whether these comments are
positive or negative in nature (polarity). Each sentence in the rebuttal is
mapped to zero or more sentences in the review, which constitute its context.
This is a many-to-many relation. The sentences in the rebuttal are labeled
with domain-specific discourse acts; each discourse act may be categorized
according to whether it concurs with (✓) or disputes ($\times$) the premise of
the sentence(s) it is responding to.

Most peer review formats allow authors to respond to the reviewers’ comments
on their manuscript. In this paper, we use the term rebuttal to refer to this
response. We argue that the discourse structure of the rebuttal can be
particularly instructive to observers of the peer review process, providing
additional context on the manuscript itself, as well as some indication of the
quality of the peer reviews. Although there has been extensive prior work on
peer review, including labeling argument structure (Hua et al., 2019; Fromm et
al., 2020; Cheng et al., 2020), highlighting different aspects (Yuan et al.,
2021), and examining the effect of rebuttals on paper acceptance (Gao et al.,
2019), there has been relatively little inquiry into the discourse structure
of rebuttals.

We argue that in order to interpret a portion of the rebuttal, it is essential
to know its context in the review, as well as the reviewer’s argumentative
intent in supplying this context. Thus, our dataset provides three types of
annotations on review-rebuttal pairs: rich characterizations of the
argumentative function of all review sentences and all rebuttal sentences, and
a mapping from each rebuttal sentence to its context in the review. Our
rebuttal labels focus on characterizing the authors’ stance towards the
reviewer’s claims, and their commitment to addressing their concerns.

The dataset we present is intended as a rich, comprehensive and high-quality
test collection, along with training data to fine-tune models. Our annotations
are carried out by graduate students in computer science who have undergone
training and calibration, amounting to over 850 person-hours of annotation
work. Much of the test data is double-annotated, and we report IAA on all
aspects of the annotation. This dataset is the first in this domain to
characterize the function of every sentence in a review and a rebuttal,
allowing our analysis to extend beyond the purview of conventional argument
types.

We believe that this dataset can support varied research questions in the area
of peer review, while also enabling the construction of practical tools for
peer reviewers and decision makers. We present one example in which
information from our annotations can directly benefit area chairs of a
conference. Introducing the concept of agreeability, we show that measuring
the extent to which authors agree with the premises of reviewers’ criticisms
can be used to locate problematic forums that require area chairs’
intervention.

The contributions of this paper are as follows: (1) a new labeled training
dataset of 506 review-rebuttal pairs over 20,394 sentences of ICLR peer review
discussion text in English (2) a taxonomy of discourse labels synthesizing
prior work on discourse in the peer review domain and extending it to add
useful sub-categories. (3) a summary of the performance of baseline models on
this dataset, elucidating error categories that may be addressed by future
work and (4) analyses illustrative of trends that could be studied with this
data. (5) extensive annotation guidelines along with annotation software to
support future labeling efforts in this area.

##  2 Dataset

Category | Label | Description | Percentage
---|---|---|---
Review action | Evaluative | A subjective judgement of an Aspect of the paper | 35.14%
Structuring | Text used to organize an argument | 29.65%
Request | A request for information or change in regards to the paper | 21.20%
Fact | An objective truth, typically used to support a claim | 9.15%
Social | Non-substantive text typically governed by social conventions | 1.51%
Aspect | Substance | Are there substantial experiments and/or detailed analyses? | 18.30%
Clarity | Is the paper clear, well-written and well-structured? | 11.86%
Soundness/Correctness | Is the approach sound? Are the claims supported? | 10.25%
Originality | Are there new topics, technique, methodology, or insights? | 4.12%
Motivation/Impact | Does the paper address an important problem? | 3.95%
Meaningful Comparison | Are the comparisons to prior work sufficient and fair? | 3.37%
|  | Replicability | Is it easy to reproduce and verify the correctness of the results? | 3.06%
Polarity | Negative | Negatively describes an aspect of the paper (reason to reject) | 31.49%
Neutral | Does not commit to being positive or negative | 12.68%
Positive | Positively describes an aspect of the paper (reason to accept) | 11.95%
Fine review action |  Struct. | Summary | Reviewer’s summary of the manuscript | 19.45%
Heading | Text used to organize sections of the review | 9.14%
Quote | A quote from the manuscript text | 1.07%
Request | Explanation | Request to explain of scientific choices (question) | 5.89%
Experiment | Request for additional experiments or results | 5.11%
Edit | Request to edit the text in the manuscript | 4.43%
Clarification | Request to clarify of the meaning of some text (question) | 2.99%
Typo | Request to fix a typo in the manuscript | 2.11%

Table 1: A list of the review sentence labels, their descriptions, and the
percentage of review sentences they apply to. Note that the percentages should
not add up to 100%, as a single sentence may have labels from multiple
categories.

Category | Label | Description | Reply to | Percentage
---|---|---|---|---
Argumentative |  Concur | Answer | Answer a question | Request | 33.14%
Task has been done | Claim that a requested task has been completed | Request | 8.68%
Concede criticism | Concede the validity of a negative eval. statement | Evaluative | 2.73%
Task will be done | Change promised by camera ready deadline | Request | 2.03%
Accept for future work | Express approval for a suggestion, but for future work | Request | 1.31%
Accept praise |  | Evaluative | 0.36%
Dispute | Reject criticism | Reject the validity of a negative eval. statement | Evaluative | 10.49%
Mitigate criticism | Mitigate the importance of a negative eval. statement | Evaluative | 2.46%
Refute question | Reject the validity of a question | Request | 0.96%
Contradict assertion | Contradict a statement presented as a fact | Fact | 0.87%
Non-arg |  | Structuring | Text used to organize sections of the review | - | 18.02%
| Summary | Summary of the rebuttal text | - | 8.04%
| Social | Non-substantive social text | - | 6.79%
| Followup question | Clarification question addressed to the reviewer | - | 0.32%

Table 2: A list of the rebuttal sentence labels, their descriptions, and the
percentage of rebuttal sentences they apply to. The “Reply to” column shows
the review action types that a particular rebuttal type would canonically
reply to.

###  2.1 Structure

Each example in our dataset consists of a pair of texts: a review and a
rebuttal. Our discourse labels are applied at the sentence level. Various
properties of each sentence are labeled as described below; rebuttal sentences
are further annotated by associating them with the set of review sentences
that form their context.

###  2.2 Review sentence labels

Each sentence in a review can have up to four labels: a review-action label
characterizing the intended function of the sentence, a fine-review-action
label further subdividing these intents, an aspect label characterizing the
qualities of the manuscript being evaluated, and a polarity label indicating
whether the quality is evaluated as positive or negative. Descriptions of each
label are provided in Table 1, and the rationale for this taxonomy is
described in Section 2.4.

####  review-action

review-action annotations characterize a sentence’s intended function in the
review. Annotators label each sentence with one of six coarse-grained sentence
types including evaluative and fact sentences, request sentences (including
questions, which are requests for information), as well as non-argument types:
social , and structuring for organization of the text. We also extend two of
these types with subtypes: structuring sentences are distinguished between
headers, quotations, or summarization sentences, and request sentences are
subdivided by the nature of the request, distinguishing between clarification
of factual information, requests for new experiments, requests for an
explanation (e.g. of motivations or claims), requests for edits, and
identification of minor typos.

####  aspect

aspect annotations follow the list of aspects used in Chakraborty et al.
(2020) and Yuan et al. (2021), which in turn were based on the ACL
rubric111https://acl2018.org/downloads/acl_2018_review_form.html. These
distinguish clarity, originality, soundness/correctness, replicability,
substance, impact/motivation, and meaningful comparison. Following Yuan et al.
(2021), arguments with an aspect are also annotated for polarity .

###  2.3 Rebuttal sentence labels

We annotate two properties of the each rebuttal sentence: a label
characterizing the type of rebuttal response, and a span indicating which
parts of a given review are being responded to.

Rebuttal sentences are annotated with labels at two levels of a taxonomy. The
taxonomy consists of fourteen fine-grained review intents, which fall into
three categories according to the author’s stance. These are elaborated on in
Table 2.

#### Concur

The author concurs with the premise of the context. This includes answering a
question or discussing a request change that has been made to the manuscript,
conceding a criticism in an evaluative sentence.

#### Dispute

The author disputes the premise of the context. The rebuttal sentence may
reject a criticism or request, disagree with an underlying fact or assertion,
or mitigate criticism (accepting a criticism while, e.g., arguing it to be
offset by other properties).

#### Non-argument

Encompassing rebuttals such as social actions (such as thanking reviewers),
structuring labels, for sentences that organize the review, and summary when
the authors summarize the contributions of their manuscript or the contents of
the rebuttal

Every sentence in a given rebuttal is annotated with a sentence-level label
and a subset of the set of review sentences. The set of sentences may be the
entire review (global context) or the empty set (no context). By not mandating
a fixed discourse chunking (of the sort as used in Cheng et al. (2020)), these
labels may handle complex mappings between hierarchies, where some rebuttals
refer to larger (possibly non-contiguous) chunks of text and others refer to
specific individual sentences.

###  2.4 Rationale

Our label sets leverage ideas from and commonalities between existing work in
this domain, including AMPERE (Hua et al., 2019), AMSR (Fromm et al., 2020),
and ASAP-Review (Yuan et al., 2021):

  * •

ASAP-Review’s polarity labels approximately correspond to arg-pos and arg-neg
labels in AMSR

  * •

AMSR and AMPERE each label non-argumentative sentences in a similar manner

  * •

aspect labels from ASAP-Review apply only to certain types of sentences;
namely request and evaluative sentences from AMPERE’s taxonomy.

  * •

summary is an exception among ASAP-Review’s aspects, behaving similarly to
AMPERE’s quote. We thus include both of these under a structuring category.

  * •

Further, in order to gauge the extent to which authors acquiesced to
reviewers’ requests, we introduce a fine-grained categorization of the types
of requests.

  * •

Gao et al. (2019) enumerates some features of rebuttals, including expressing
gratitude, promising revisions, and disagreeing with criticisms. We formalize
these observations into our rebuttal label taxonomy.

###  2.5 Data Sampling and Annotation

This dataset uses English text from scientific discussions from OpenReview
Soergel et al. (2013). We draw data from the International Conference on
Learning Representations (ICLR) in 2019 and 2020, in which anonymized
discussion between the authors and reviewers on OpenReview was encouraged.
Review-rebuttal pairs are split into train, development and test sets in a
3:1:2 ratio such that all texts associated with any manuscript occur in the
same subset. Overall statistics for the dataset are summarized in Table 3.

| Train | Dev | Test
---|---|---|---
Num. review-rebuttal pairs | 251 | 88 | 167
Num. forums | 94 | 37 | 57
Num. adjudicated pairs | 0 | 0 | 65
Num. review sentences | 4846 | 1358 | 3087
Num. rebuttal sentences | 5805 | 2015 | 3283
Avg. sents per review | 19.31 | 15.43 | 18.49
Avg. sents per rebuttal | 23.13 | 22.9 | 19.66

Table 3:  Statistics for the dataset

Each ICLR manuscript is reviewed by three or more reviewers. Authors are able
to respond to each review by adding a comment. Although rebuttals are not
formally named, we consider direct replies by the author to the initial review
comment to constitute a rebuttal. Reviewers and authors may engage in multi-
turn interactions beyond this initial response, but we focus on the set of
reviews and initial responses, and leave study of extended discussion for
future work. The text is separated into sentences using the SpaCy (Honnibal et
al., 2020) sentence separator.

Annotation was accomplished with a custom annotation tool designed for this
task, which will be released upon publication. Review labels are annotated
first on a sentence-by-sentence basis, followed by a rebuttal annotation pass
wherein, for each rebuttal sentence, the annotator selects a set of review
sentences aligned to that sentence. Annotators were instructed to select large
discourse spans of text when they were relevant to a particular rebuttal, and
permitted to directly propagate context selections to subsequent rebuttal
sentences when necessary.

Note that while the current annotation focused upon links between sentences,
this is not intended to eschew discourse structure. Indeed, in contrast to
approaches in which discourse chunks are annotated and then aligned (as in
Cheng et al. (2020)), this annotation can handle situations where some
rebuttal sentences refer to single sentences, and other rebuttal sentences
refer to larger chunks of discourse. However, we admit that any representation
of how review sentences cluster together is a somewhat latent structure
implied by whether sentences are referred to as a unit during rebuttal
alignment, rather than an explicitly annotated discourse chunking or discourse
tree.

###  2.6 Agreement

We report chance-corrected measures of agreement on sentence-level labels of
review and rebuttal sentences.

Table 4 shows chance-corrected inter-annotator agreement measures for these
annotations over double-annotated data. Cohen’s kappa (Cohen, 1960) for each
label ranges from moderate to substantial agreement.

Label | $\kappa$
---|---
review-action | 0.694
fine-review-action | 0.675
aspect | 0.557
polarity | 0.657
rebuttal-stance | 0.602
rebuttal-action | 0.553
Table 4:  IAA for review labels (top) and rebuttals (bottom), scored on
double-annotated examples.

##  3 Analysis

###  3.1 Context types

Each rebuttal sentence is annotated with its context in terms of a subset of
review sentences. We separate the different types of contexts in terms of the
number and relative position of selected sentences in Table 5, along with the
four cases in which a sentence’s context cannot be described in terms of a
subset of review sentences.

| Context type |  Num. reb. sents |  % reb. sents
---|---|---|---
Sents. selected | Multiple contiguous | 12375 | 60.31%
Single sentence | 4313 | 21.02%
Mult. non-contiguous | 2144 | 10.45%
No sents. selected | Global context | 816 | 3.98%
Context in rebuttal | 647 | 3.15%
No context | 112 | 0.55%
| Context error | 101 | 0.49%
| Cannot be determined | 11 | 0.05%

Table 5: Different types of rebuttal sentence contexts. Top: Over 90% of
sentences are linked to a subset of sentences in the review. The remaining
sentences are subdivided into four categories. Bottom: sentences not linked to
anyparticular subset of review sentences.

###  3.2 Alignment

Since authors often respond to reviewers’ points in order, one would expect
alignment to be trivial, yet this is not necessarily the case. In Figure 2, we
calculate Spearman’s $\rho$ between rebuttal sentence indices and their
aligned review sentence indices. Rebuttals responding to each point in order
would achieve a $\rho$ of $1.0$. However, this is rare – instead, we find that
$\rho$ is even negative in some cases. This indicates that while the alignment
task may benefit from linear inductive bias, the task is not trivial.

Figure 2: Spearman’s $\rho$ between rebuttal sentence indices and aligned
review sentence indices.

###  3.3 Author interpretations of criticism

The rebuttal sentence labels were designed with the review actions in mind,
and thus tend to explicitly refer to different responses to criticism (accept,
reject, mitigate) or to requests (answer, refute, etc.) However, annotations
revealed that authors often interpret a review sentence in ways that support
their argumentative goals, such as treating a negative evaluative statement as
a request for an improvement. Figure 3 shows the distribution of contexts for
three different rebuttal actions. This also relates to the issue of
“declaratives of trouble” as requests (Fox and Heinemann, 2021), and indicates
a possible avenue for further research .

Figure 3: Distribution over context review actions for three rebuttal actions.
The canonical review action is marked by cross hatching. Note that authors
sometimes interpret requests as criticisms (“Concede criticism”); often
respond to evaluative sentences as if they are questions (“Answer”), and
sometimes treat criticisms in the form of evaluative sentences as requests
which they then carry out. (“Task has been done”)

##  4 Application: Agreeability

Gao et al. (2019) showed that the most important factor influencing a
reviewer’s decision to change their score is conformity bias with respect to
other reviewers’ scores, rather than any properties of the rebuttal. Although
area chairs are supposed to take both reviews and rebuttals into account when
making decisions, it is often necessary to resort to heuristics for practical
reasons. For example, area chairs often detect manuscripts that need closer
attention by examining the variance in reviewer scores. However, it is
frequently the case that a review with a particularly high or low score may be
based on incorrect assumptions, and as such should be disregarded entirely. We
propose that our dataset can be used to provide heuristics that are data-
driven, yet interpretable, and leverage information from the content of
reviews rather than just numerical scores.

To this end, we define agreeability as the ratio of concur sentences to
argumentative sentences, i.e.:
$\mathit{agreeability}=\frac{n_{\mathit{concur}}}{n_{\mathit{concur}}+n_{\mathit{dispute}}}$,
and argue that low agreeability can indicate problematic reviews even in cases
where the variance in scores does not reveal an issue, as illustrated in Fig
4. Agreeability is only weakly correlated with rating, with $r=0.347$. in
Figure 4, 18% (28/159) of manuscripts would not meet the bar for high variance
scores, although their low agreeability indicates that area chairs may want to
pay closer attention.

Figure 4: Mean agreeability for a manuscript’s reviews v/s reviewer variance.
Manuscripts above the dotted line are in the top quartile of rating variance,
and are more likely to be reviewed by area chairs. Manuscripts to the left of
the dashed line are in the bottom quartile of mean agreeability, in which
authors take issue with the premises of reviewers’ comments.

##  5 Experiments

###  5.1 Sentence classification

Cohan et al. (2019) introduces the task of sequential sentence classification
(SSC), in which the goal is to classify each sentence in a sequence of $n$
sentences. Since reviews and rebuttals can be viewed as sequences of
sentences, we can apply state-of-the-art methods for SSC for the sentence
classification task. We define a SSC task for each of the label categories in
Table 1 and Table 2 for reviews and rebuttals respectively.

Cohan et al. (2019) found that when using a representation in which the
sentences were concatenated using special separator tokens, a thin task-
specific layer over a BERT(Devlin et al., 2019) model was able to decode
sentence labels from the output representations of the separator tokens.

The results of this model are shown in Table 6. In general, F1 is lower for
tasks with larger label spaces. While the performance is reasonable in most
cases, there is still room for improvement. aspect achieves a particularly low
F1 score, but its $\kappa$ is within the bounds of moderate agreement, this
must be accounted for by the inherent difficulty of the task rather than a
deficit in the quality of the training set.

Label |  Avg F1 (test) | $\kappa$ |  Num. labels
---|---|---|---
review-action | 58.28% | 0.694 | 6
fine-review-action | 41.61% | 0.675 | 10
aspect | 18.10% | 0.557 | 9
polarity | 62.44% | 0.657 | 4
rebuttal-stance | 42.41% | 0.602 | 4
rebuttal-action | 25.92% | 0.553 | 17
Table 6: Results of sequential sentence classification. Top: review labels;
Bottom: rebuttal labels.

###  5.2 Rebuttal context alignment

The extensive annotation of rebuttal sentences to one or more sentences in
reviews provides opportunity to train models that map a rebuttal sentence to
its review context. We treat this task as a sentence pair alignment problem.
Ideally, a model should rank all relevant review sentences higher than non-
relevant review sentences. Our evaluation metric is mean reciprocal rank (MRR)
which is a widely used metric for evaluating ranked lists.

We train two models for context alignment — (a) an information retrieval (IR)
model based on BM25 that, given a rebuttal sentence ranks all the
corresponding review sentences and (b) a neural sentence alignment model based
on a two-tower Siamese-BERT (S-BERT) model Reimers and Gurevych (2019). Each
review and rebuttal sentence are encoded independently using a S-BERT encoder
and the similarity between two sentences is computed using cosine similarity.
We initialize with a model222We initialize from a sentence-transformers/all-
MiniLM-L6-v2 model pre-trained on various sentence-pair datasets.

Table 7 lists the MRR of both the IR and the neural model. The neural model
outperforms the IR baseline indicating that simple lexical matching models are
not enough to achieve good performance on our dataset. The neural model
performs better than the IR baseline, however the MRR is still quite low
suggesting that there is a significant scope for improvement.

BM-25 | S-BERT
---|---
0.3544 | 0.4171
Table 7: Rebuttal Context-alignment results.

##  6 Related work

|  | AMPERE | AMSR | ASAP-Review | APE | Ours
---|---|---|---|---|---|---
Overall | Total number of examples | 14.2k reviews | 12.1k reviews | 28k reviews | 4764 pairs | 448 pairs
Number of human-annotated examples | 400 reviews | 77 reviews | 1000 reviews | 4764 pairs | 448 pairs
Review | Argument segmentation | ✓ | ✓ | ✓ | ✓ | $\times$
Argument type labeled? | ✓ |  |  |  | ✓
Aspect labeled? |  |  | ✓ |  | ✓
Polarity labeled? |  | ✓ | ✓ |  | ✓
Non-argument subtypes labeled? |  |  |  |  | ✓
Rebuttal | Rebuttal included? | $\times$ | $\times$ | $\times$ | ✓ | ✓
Argument segmentation | - | - | - | ✓ | $\times$
Argument type labeled? | - | - | - |  | ✓
Non-argument subtypes labeled? | - | - | - |  | ✓

Table 8: Comparison between our dataset and prior work: AMPERE (Hua et al.,
2019), AMSR (Fromm et al., 2020), ASAP-Review (Yuan et al., 2021), APE (Cheng
et al., 2020). Our dataset is the only work to annotate every sentence in the
review and rebuttal, and the only work that applies discourse labels to the
author’s actions in the rebuttal.

The design of this dataset draws upon extensive, but disparate prior work on
this topic. Many works, some addressed below, have taken advantage of the
availability of review text hosted on OpenReview, and have developed label
sets that address different phenomena. In our dataset, we attempted to unify
these schemas to form a single hierarchical schema for review discourse
structure. We then expanded this hierarchical schema to introduce fine-grained
classes for implicit and explicit requests made by the reviewers.

Hua et al. (2019) introduced the idea of studying discourse structure in peer
review. In the AMPERE dataset described in this paper, 400 reviews from a ICLR
2018 are annotated to indicate argumentative propositions. A new scheme for
labeling these propositions was introduced, specifically tailored to the peer
review domain and including evaluation, request, fact, reference, and quote.
The relative prevalence of argument types is compared across conferences, as
well as across review ratings within a conference. The paper identifies major
challenges in the peer review domain including the prevalence of multiple
arguments within a sentence. Our work detects similar phenomena, but labels
these at a sentence level rather than an argument level. This enables more
thorough coverage of the text while avoiding the propagation of errors from
machine learning models earlier in the annotation pipeline.

Similarly, Fromm et al. (2020) describes the AMSR dataset, which also takes an
argument mining approach to peer review. Rather than focusing on argument
types, this paper frames the problem as an argumentation process, in which the
stance of each argument towards the paper’s acceptance or rejection is of
paramount importance. In contrast to Hua et al. (2019), they find that
transfer learning works relatively well for labeling polarities alone. This
paper also proposes a system to present information to decision makers such as
area chairs, however, this system’s outputs are tested as a replacement for
the full review text rather than a supplement to it. To the best of our
knowledge, the system is not made public.

While its goal is not to examine discourse structure per se, Yuan et al.
(2021) also uses polarity labels to indicate each argument’s support or attack
of the authors’ bid for acceptance. This paper is accompanied by a dataset of
1,000 annotated rebuttals extracted from OpenReview. Besides polarity, these
examples follow Chakraborty et al. (2020) by annotating each argument with an
aspect. Yuan et al. (2021) state that the goal of their work is to examine the
feasibility of automatically generating peer reviews. To this end, they define
metrics for review quality. In our work, we do not recommend generating peer
review text, but do adopt the metrics used in Yuan et al. (2021) as possible
useful signals for area chairs.

We also expand on work by Cheng et al. (2020), who first annotated discourse
relations between sentences in reviews and rebuttals, by developing a taxonomy
over rebuttal argument types. Again, we extend our analysis to include every
sentence in the rebuttal separately, rather than relying on the predictions of
a argument detection model. While Cheng et al. (2020) introduce deep learning
architectures, in this paper we focus on less specialized models and emphasize
the characteristics of the dataset.

There has also been significant additional inquiry into artificial
intelligence approaches to modeling the process of peer review. Gao et al.
(2019) in particular focuses on a similar part of the peer review process,
determining that the author response has a relatively small effect on
reviewers’ tendency to change their score; instead, they are motivated by the
‘peer pressure’ caused by the difference in their scores from other reviewers.
However, Gao et al. (2019) do show that more specific, convincing and explicit
responses are more likely to elicit a score change. We hope that information
from tools trained on our dataset will provide information that more
specifically correlates with the fulfilment of reviewers’ expectations,
allowing area chairs to nudge reviewers to pay more attention to rebuttals
that address their concerns.

## References

  * Chakraborty et al. (2020) Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020.  Aspect-based sentiment analysis of scientific reviews.  _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020_.
  * Cheng et al. (2020) Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020.  APE: Argument pair extraction from peer review and rebuttal via multi-task learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7000–7011, Online. Association for Computational Linguistics.
  * Church (2020) Kenneth Ward Church. 2020.  Emerging trends: Reviewing the reviewers (again). _NATURAL LANGUAGE ENGINEERING_ , 26(2):245 – 257.
  * Cohan et al. (2019) Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, and Dan Weld. 2019.  Pretrained language models for sequential sentence classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3693–3699, Hong Kong, China. Association for Computational Linguistics.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ , 20(1):37–46.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fox and Heinemann (2021) Barbara A. Fox and Trine Heinemann. 2021.  Are they requests? an exploration of declaratives of trouble in service encounters.  _Research on Language and Social Interaction_ , 54(1):20–38.
  * Fromm et al. (2020) Michael Fromm, Evgeniy Faerman, Max Berrendorf, Siddharth Bhargava, Ruoxia Qi, Yao Zhang, Lukas Dennert, Sophia Selle, Yang Mao, and Thomas Seidl. 2020.  Argument mining driven analysis of peer-reviews.  _arXiv preprint arXiv:2012.07743_.
  * Gao et al. (2019) Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019.  Does my rebuttal matter? insights from a major NLP conference.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1274–1290, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Hua et al. (2019) Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019.  Argument mining for understanding peer reviews.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2131–2137, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ko (2019) Amy J. Ko. 2019.  A modern vision for peer review.
  * Nobarany (2014) Syavash Nobarany. 2014.  Rethinking the peer review process.  In _Proceedings of the companion publication of the 17th ACM conference on Computer supported cooperative work & social computing_, CSCW Companion ’14, pages 77–80, Baltimore, Maryland, USA. Association for Computing Machinery.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Soergel et al. (2013) David Soergel, Adam Saunders, and Andrew McCallum. 2013.  Open scholarship and peer review: a time for experimentation.
  * Yuan et al. (2021) Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2021.  Can we automate scientific reviewing?

##  Appendix A Annotation tool

Figures 5-7 show screen captures from the annotation tool used to collect this
dataset.

Figure 5: Review annotation interface. Annotators select label values from
dropdown menus for each review sentence in turn. Figure 6: Rebuttal annotation
interface. Annotators select review sentences that form the context of the
rebuttal sentence being annotated. Figure 7: Review annotation interface.
Annotators select a label for the rebuttal sentence.


# N-Best Hypotheses Reranking for Text-To-SQL Systems

###### Abstract

Text-to-SQL task maps natural language utterances to structured queries that
can be issued to a database. State-of-the-art (SOTA) systems rely on
finetuning large, pre-trained language models in conjunction with constrained
decoding applying a SQL parser. On the well established Spider dataset, we
begin with Oracle studies: specifically, choosing an Oracle hypothesis from a
SOTA model’s 10-best list, yields a $7.7\%$ absolute improvement in both exact
match (EM) and execution (EX) accuracy, showing significant potential
improvements with reranking. Identifying coherence and correctness as
reranking approaches, we design a model generating a query plan and propose a
heuristic schema linking algorithm. Combining both approaches, with T5-Large,
we obtain a consistent $1\%$ improvement in EM accuracy, and a $~{}2.5\%$
improvement in EX, establishing a new SOTA for this task. Our comprehensive
error studies on DEV data show the underlying difficulty in making progress on
this task.

Index Terms—  Text-To-SQL, Semantic parsing

##  1 Introduction

Large language models (LM) are widely used for natural language generation [1,
2, 3]. Recently, the use of large LMs has extended to semantic parsing tasks
such as code generation. For general-purpose code generation, large LMs such
as Codex [4] are trained on massive, paired codebases (code, NL). For domain-
specific code generation tasks, such as Text-to-SQL that aims to convert
natural language instructions to SQL queries, multiple public domain datasets
are available (with associated leaderboards), including Spider [5], CoSQL [6]
and SParC [7]. However, the amount of training data in these datasets is much
smaller than the natural language and code pairs mined from the internet. In
such cases, instead of training a new model from scratch, the
pretrain/finetune strategy with publicly available LMs has been shown to be
more accurate. For example, in UnifiedSKG [8], the T5 model [1] is finetuned
for various semantic parsing tasks (including Text-to-SQL), achieving SOTA
performance. Besides the amount of the training data, another challenge with
SQL generation is that the generated code is underspecified without the
corresponding schema 111For Text-to-SQL, in addition to SQL grammar
constraints, there are schema constraints.. To handle this challenge, implicit
or explicit schema linking becomes an important sub-task for SQL code
generation [9, 10, 11].

In this paper, we focus on complex, cross-domain, SQL generation using Spider
[5] a large-scale, Text-to-SQL dataset consisting of 200 complex databases
covering 138 domains. Spider is a well established dataset with nearly 70
entries on the leaderboard, demonstrating the difficulty of the task. The data
is split into training, development (DEV), and test sets without overlaps in
databases across these sets, as the aim of this task is to learn models that
can issue queries from natural language text to previously unseen databases in
the training set. Furthermore, SQL queries in this dataset contain nested sub-
queries, requiring the model to understand compositional structures. The model
performance is based on two metrics, exact-set-match accuracy (EM) and
execution accuracy (EX); the former compares individual query components
between the predicted and groundtruth SQL queries, while the latter compares
their execution output.

Similar to general-purpose code generation, the output of Text-to-SQL models
is constrained to follow SQL grammar. Previous solutions have employed
encoder/decoder models, with the decoder being constrained to produce well-
formed outputs [12, 13]. An approach that is more compatible with large
pretrained LMs is to remove the constraints on decoder: [14, 15] prune the
finalized hypotheses from beam search to those that are syntactically correct.
Meanwhile, PICARD [16], a top entry on the Spider leaderboard 222https://yale-
lily.github.io/spider, in addition to finetuning the pretrained T5 model, also
imposes SQL syntax constraints during beam search (via constrained decoding).
It achieves an EM of 74.8% and an EX of 79.2% on the Spider DEV set. Natural
ways to potentially improve these metrics include increasing the model size or
collecting and/or synthesizing additional training data [17].

In this paper, we take a different approach: we first perform an Oracle
analysis on n-best lists obtained from PICARD and observe significant
improvements (7.7% absolute improvement in EM and EX) even at small beam sizes
(such as 10). The gap in performance between 1-best and Oracle motivates
reranking approaches. We propose 2 reranking approaches that are motivated by
the issues with the current, large pretrained LMs: coherence [18] and
correctness. To improve coherence, we explore n-best reranking using a query
plan produced by an independent model. Next, to improve correctness, we
propose a heuristic algorithm performing schema linking on n-best lists –
imposing constraints that are missing in PICARD. The combined reranking
approaches consistently yield improvements across all T5 model sizes,
obtaining a 1.0% absolute improvement in EM and a 2.5% absolute improvement on
EX. Lastly, to analyze persistent gap with Oracle results, we performed a
detailed analysis of errors and metrics. Our studies show that annotation and
metrics issues can be significant factors to improving models and evaluation
on Spider.

Contributions. The contributions of this paper are: a) Using Oracle analysis
and rerankers on n-best lists, we outperform SOTA performance on a competitive
Spider task; b) Analysis of errors on Spider DEV data, showing much work
remains to be done on metrics and annotation.

##  2 Related Work

We review related work in 4 categories: a) reranking approaches; b) coherence
in LMs; c) schema linking for text-to-SQL; d) noise in Spider.

Rerankers. N-best list reranking has a long history in speech recognition,
machine translation, and semantic parsing communities [19, 20, 21, 22, 23, 24,
25]. These are performed with a much larger $2^{nd}$-stage model [26, 27] or
using additional knowledge sources not available during the first pass [28,
29]. On Spider, a larger $2^{nd}$-stage reranker was successfully used in
[30]. Some reranking methods combine outputs from multiple systems [31] or use
improved output confidence [32]. Imposing task specific constraints [14, 15]
to remove invalid options can be a helpful strategy to improve reranking. In
our work, in addition to the output target being structured queries (instead
of natural language), the $1^{st}$-stage model is large (T5 family), and the
baseline model imposes additional knowledge during beam search in the form of
SQL parsing. Our proposed methods are designed for this setting.

Coherence with query plan. Coherence issues with LMs [33] are associated with
hallucination phenomenon [34, 35]. While these are well-known problems in
unstructured text generation with LMs [36, 37, 38, 39], coherence issues with
structured text generation is somewhat less unexplored: [40] uses semantic
coherence for data augmentation, while [41] uses a reconstruction model for
reranking semantic parses, and [14, 15, 16] use parsers to improve coherence
with SQL grammar. Our reranker is designed to improve semantic coherence, and
it is distinct from [41] in that, our query plan model predicts the structure
of a query (from natural language), and orders the n-best list for
consistency.

Correctness with schema linking. Historically, schema linking (SL) has been
explored for Text-to-SQL. Schema has been encoded in model input for small
encoder/decoder models [15, 42, 43] and large pretrained LMs [8, 16]. More
work has been done on modeling schema linking [9], where the role of SL in
Text-to-SQL is recognized. RAT-SQL [12], a system that had been influential in
text-to-SQL, proposed a framework to encode relational structure in the
database schema and simultaneously address schema linking. In contrast, we use
schema linking as a post-processor and a reranker on the output of large LMs.

Noise in Spider. Spider dataset [5] has been well-explored in Text-to-SQL, and
it’s corpus noise are documented in related work [9, 12, 44]. One type of
corpus noise comes from annotation errors in groundtruth SQL queries and typos
in natural language queries [9]. Another source of the noise is in the pairing
of only one groundtruth SQL query to each natural language query, which
results in a large portion of predicted queries being incorrectly marked as
“mispredicted” [12, 44].

Fig. 1: PICARD explained by an example. The prediction pattern is “⟨Database_name⟩ | ⟨pred_SQL⟩”.

##  3 Text-to-SQL Using Pre-trained LMs

We use PICARD as our baseline system, and use it to produce the n-best
hypotheses in this paper. It uses finetuned T5 models, and implements
incremental SQL parsing to constrain decoding during beam search. Input to the
T5 model includes natural language query, database name, and database schema
(table_name : col1 , ... , coln). PICARD employs a post-processor during beam
search, and to save computing time it only checks the next top-k tokens for
validity. Figure 1 shows how predictions are generated when PICARD is enabled,
using $k=2$ and beam size of 3. Shadowed box means that the result is not
expanded or explored for the next time stamp; blue (solid) boxes show valid
SQL query prefixes, while orange (dashed) boxes show sequences that violate
SQL syntax.

The incremental parser not only filters out hypotheses that violate the SQL
grammar, but also imposes rudimentary schema checks. Fig. 1 gives two examples
of filtering hypotheses, the first one is due to a syntax error (e.g.,
count()) and the other one is due to schema error (e.g., table name vocal does
not exist in the database).

##  4 Reranking Approaches

In this section, we discuss our two reranking approaches over n-best
hypotheses.

###  4.1 Improving Coherence with Query Plan

Text generation is a challenging topic, and large LMs tend to lose coherence
[18, 39]. This is compounded in text-to-SQL with such models: apart from
conforming to syntactic structures, models have to perform semantic mapping
from natural language to SQL over long spans in complex questions involving
compositionality, grouping/ordering/counting. Furthermore, nested queries are
likely to appear with WHERE, EXCEPT, UNION, and INTERSECT clauses, while
grouping/ordering/counting tend to occur with GROUP BY, HAVING, ORDER BY,
LIMIT.

We build a model that focuses specifically on this aspect: a multi-label
classification model that generates a query plan predicting whether a SQL
query contains the aforementioned 8 clauses. The output of this model is then
used as a coherence check reranker against the n-best hypotheses from the
baseline model. We finetune a RoBERTa-Large using the classification head with
binary cross entropy (BCE) loss. The model is trained using the training
partition of Spider dataset. Input of the model consists of a natural language
query and a database schema, while the labels are obtained from groundtruth
SQL queries.

###  4.2 Improving Correctness with Schema Linking

The motivation for this approach comes from the fact that certain aspects of
schema linking (values, columns, and tables) from natural language query (on
the input side of the model) to a SQL query (on the output side of the model)
can be hard for the model to learn to map: this is because these mappings tend
to be irregular, and a relatively small training dataset may not cover a large
fraction of the irregularities. For example, a primary key field on “student
identifcation” could be named in several ways as a column such as “stud_id”,
“sid”, “s_id” etc. Similarly, values that are strings can be arbitrarily
represented, such as “True” vs “T” vs “Yes”.

We implement an algorithm to perform value linking in WHERE clauses. For each
predicted SQL query in the n-best list, we follow three steps:

  1. 1.

Extract slot names and their respective values from the conditions in the
WHERE clause; then check if the slot value exists in any of the referenced
tables in the FROM clause of the query;

  2. 2.

Obtain a list of candidate slot names and values, which are exact or partial
occurrences of the column/table names and string values in the question with
name-based and value-based linking described in RAT-SQL [12];

  3. 3.

Perform prefix and abbreviation matches on slot values with categorical types
in a table schema definition (such as matching “left” to “L”).

##  5 Experiments

This section presents the experimental setup, baselines, Oracle analysis, and
the main results.

###  5.1 Dataset and Metrics

We briefly describe the dataset and metrics used in this paper. More details
can be found in [5].

####  5.1.1 Dataset

Spider is a large-scale, cross-domain paired text-to-SQL dataset: it contains
10,181 questions and 5,693 unique complex SQL queries on 200 databases
(covering 138 domains), each with multiple tables. The standard protocol
splits this into 8,659 examples on 146 databases for training, and 1034
examples on 20 databases are used for development (DEV); the test set that
includes 2,147 examples from 34 databases are held back. The split is done
without overlaps of databases across these sets.

Task difficulty levels. Spider categorizes the SQL query complexity into 4
difficulty levels: easy, medium, hard, and extra hard. Queries that contain
more SQL keywords (GROUP BY, ORDER BY, INTERSECT), nested subqueries, column
selections and aggregators are considered harder.

####  5.1.2 Metrics

On Spider, Text-to-SQL model performance is evaluated based on two metrics:
exact-set-match accuracy (EM) and execution accuracy (EX). Note that all our
results are on the Spider DEV set.

Exact-set-match accuracy (EM). EM compares each clause333Note that EM
implemented by Spider ignores table join conditions. between a prediction and
its corresponding groundtruth SQL query. The predicted SQL query is correct
only if all of the components match. This metric does not take values into
account. EM can lead to false positives and false negatives.

Execution Accuracy (EX). EX compares the execution output of the predicted SQL
query and its corresponding groundtruth SQL queries. However, it is important
to note that EX can also lead to false positives and false negatives.

###  5.2 Models

We use PICARD [16] as our baseline model in this paper. We finetune on three
T5 model sizes, T5-Base444Note that the models are LM adapted, initialized
from T5 model and trained for additional 100K steps on the LM objective
discussed in [1], T5-Large44footnotemark: 4, and T5-3B on p3dn.24xlarge
instances (8 NVIDIA Tesla V100 GPUs) and employ DeepSpeed to save
memory555https://github.com/microsoft/DeepSpeed. The input to the models
contain several segments separated by |, including natural language query,
database name, serialized database schema. We take the serialization scheme
mentioned in [45] and enable database content by appending database values to
the column names [15]. All the T5 models are finetuned with teacher forcing
and cross-entropy (CE) loss for up to 3000 epochs using a batch size of 2000
and a learning rate of $1e^{-4}$. During inference, an incremental SQL parser
is integrated into beam search, with a beam size of 4.

To improve coherence, we finetune a RoBERTa-Large model with a sequence
classification head on p3.2xlarge instances (1 NVIDIA Tesla V100 GPU) for
query plan prediction. We reuse the input from the baseline model. The output
is one-hot encoding label extracted from groundtruth queries based on
existence of WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, EXCEPT, UNION, and
INTERSECT clauses. The RoBERTa model is finetuned with binary cross entropy
(BCE) loss for up to 100 epochs using a batch size of 5 and a learning rate of
$1e^{-5}$.

###  5.3 Oracle Analysis

We open the beam up and perform Oracle analysis on n-best hypotheses.
Specifically, we conducted these experiments over a set of beam sizes (and
correspondingly n-best hypotheses), 10, 15, 20, and 25; also a few selected T5
model sizes, T5-Base44footnotemark: 4, T5-Large44footnotemark: 4, and T5-3B.

Table 1: 1-best accuracies as beam size is changed. Beam | T5-Base | T5-Large | T5-3B
---|---|---|---
size | EM% | EX% | EM% | EX% | EM% | EX%
4 | 66.6 | 68.3 | 74.8 | 79.2 | 75.5 | 79.3
10 | 67.1 | 68.4 | 75.1 | 79.4 | 75.6 | 79.3
15 | 67.1 | 68.5 | 74.7 | 79.2 | 75.5 | 79.2
20 | 67.3 | 68.7 | 74.8 | 79.2 | 75.5 | 79.2
25 | 67.3 | 68.7 | 74.7 | 79.1 | 75.6 | 79.2

Search errors. Table 1 presents the analysis by increasing the beam size (and
choosing the 1-best hypothesis): opening up the beam, improves the accuracy by
a small amount initially (going from 4 to 10) for all models. However, the
performance on EM and EX saturate after that, suggesting that search errors
contribute a small proportion of overall errors.

Table 2: Oracle accuracies as beam size is changed.

Beam | T5-Base | T5-Large | T5-3B | Note
---|---|---|---|---
size | EM% | EX% | EM% | EX% | EM% | EX% |
10 | 67.1 | 68.3 | 75.1 | 79.4 | 75.6 | 79.3 | 1-best
10 | 74.7 | 75.5 | 82.8 | 87.1 | 85.2 | 87.3 | Oracle
15 | 75.4 | 76.6 | 82.9 | 87.6 | 86.2 | 87.9
20 | 76.1 | 77.6 | 83.5 | 88.0 | 86.3 | 88.0
25 | 76.5 | 78.3 | 83.8 | 88.0 | 86.8 | 88.8

Model errors. In Table 2, the first row is the 1-best obtained from baseline,
while the other rows show Oracle accuracies for the three models sizes over
different beams. As can be observed, both EM and EX improve significantly: for
example, the T5-Large model achieves 7.7%, 7.8%, 8.4% and 8.7% absolute
improvements for EM; similarly 7.7%, 8.2%, 8.6% and 8.6% absolute improvements
for EX.

Table 3: T5-large: 1-best and Oracle over difficulty levels. Difficulty | count | 1-best | Oracle
---|---|---|---
|  | EM% | EX% | EM% | EX%
Easy | 248 | 89.1 | 91.9 | 93.1 | 96.0
Medium | 446 | 80.9 | 84.8 | 88.6 | 92.6
Hard | 174 | 65.5 | 71.3 | 74.7 | 79.3
Extra | 166 | 48.8 | 54.8 | 60.2 | 67.5
Total | 1034 | 75.1 | 79.4 | 82.8 | 87.1

Difficulty levels. Table 3 compares the 1-best with Oracle using baseline
(with T5-Large) over different difficulty levels. It can be observed that the
Oracle results are consistently higher over both metrics (EM and EX) at all
difficulty levels. The biggest improvement occurs on extra hard queries, with
absolute improvements of more than 10% for both metrics. Overall, these
results suggest that significant improvements can be obtained using reranking
approaches over n-best lists.

###  5.4 Results

####  5.4.1 Improving Coherence with Query Plan Modeling

Table 4 lists reranking results using query plan (QP) coherence on the 10-best
hypotheses obtained from the baseline using a T5-Large model. It can be seen
that using structural consistency with QP can help both EM and EX, obtaining
0.7% and 0.5% absolute improvements respectively. We present an example
improved by QP. A query plan that combines GROUPBY and HAVING clauses is
presented below.

Table 4: Reranking with query plan (QP) coherence checks on 10-best hypotheses from baseline using T5-Large. Method | T5-Large
---|---
| EM% | EX%
Baseline | 75.1 | 79.4
QP | 75.8 | 79.9

To understand the QP model better, we present its performance on each of the
clauses in Table 5 in terms of F-1 score, precision and recall. While the
model performs well on WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, INTERSECT, it
does not perform as well on EXCEPT and UNION. We conjecture low training data
counts for these clauses as a potential reason. The macro F-1 score for this
model is 0.89.

Table 5: QP model performance per clause. Clause |  F1 |  R |  P
---|---|---|---
where | 0.97 | 0.97 | 0.98
groupBy | 0.96 | 0.96 | 0.96
having | 0.97 | 0.95 | 0.99
orderBy | 0.96 | 0.97 | 0.95
limit | 0.95 | 0.96 | 0.94
except | 0.73 | 0.76 | 0.71
union | 0.62 | 0.44 | 0.99
intersect | 0.96 | 0.97 | 0.95

####  5.4.2 Improving Correctness with Schema Linking

Table 6 shows the reranking performance with schema linking over 10-best
hypotheses obtained from a baseline model with T5-Large. With this approach,
we see an absolute improvement in EX of 1.9%.

Table 6: Reranking with schema linking (SL) on 10-best hypotheses obtained from baseline using T5-Large. Method | T5-Large
---|---
| EM% | EX%
Baseline | 75.1 | 79.4
SL | 75.4 | 81.3

Below, we present 2 examples of schema linking errors (one of column name; and
another of cell value) which are fixed by the proposed method.

In Ex 1, the question asks for the abbreviation of Airline “JetBlue Airways”,
where “JetBlue Airways” is a slot value for a slot name “airline”. The model
gets misled by an available column name that matches the query word
“abbreviation”. In Ex 2, we need a SQL query for the total population of
“Gelderland” district. The expected slot value is “Gelderland”, the predicted
SQL obtained a cell value of “Geeland”.

Syntax Checks with a Full Parser. Since incremental parsing can still allow
syntactically incorrect full hypotheses, we investigated the errors from the
baseline model. Around 2% of the hypotheses are indeed syntactically
incorrect. For example, intermediate SQL queries from Fig. 1 such as select
count(*) from, which are invalid queries but valid prefixes, are allowed by
the incremental parser as outputs. We run a full SQL parser, filtering invalid
hypotheses, and selecting the hypothesis with the highest remaining score.

Analysis. Results from our experiments showed that the full parser does not
improve performance. We investigated the errors for the 23 examples with
syntactically incorrect 1-best predictions: for 16 of them all other
hypotheses are incomplete SQL queries, while the other 7 do not have any
hypotheses that improves either EM or EX. These are errors that cannot be
improved even with an Oracle selection, meaning that these are residual search
errors.

####  5.4.3 Combined Results

We examine if the gains from SL and QP are additive, and if they carryover to
different model sizes. We then combine SL and QP in order, presenting the
results in Table 7.

Table 7: Performance of rerankers (QP, SL, SL + QP) on 10-best lists obtained
from different T5 model sizes.

Method | T5-Base | T5-Large | T5-3B
---|---|---|---
| EM% | EX% | EM% | EX% | EM% | EX%
Baseline | 67.1 | 68.3 | 75.1 | 79.4 | 75.6 | 79.3
QP | 67.8 | 68.9 | 75.8 | 79.9 | 75.9 | 79.6
SL | 67.2 | 70.0 | 75.4 | 81.3 | 76.1 | 80.4
SL + QP | 67.9 | 70.4 | 76.1 | 81.9 | 76.4 | 80.6

From the table, it can be seen that the performance of the different rerankers
are consistent across model sizes: the SL approach yields a consistent gain of
between 1.1% to 1.9% in EX. Similarly, QP yields a consistent gain as well.
Overall the gains combining the approaches are additive on both EM and EX
across model sizes. Based on the one-sided t-tests, improvements on T5-base
and T5-large are significant at $\alpha=0.1$, while the gain on T5-3B is
significant at $\alpha=0.15$.

####  5.4.4 Analysis of gains across difficulty levels

Table 8 shows that the gains from the proposed approach (combining SL and QP)
consistently carry over all the pre-defined difficulty levels. Furthermore,
the gains are bigger on harder examples, considering both EM and EX.

Table 8: T5-large model: Performance of rerankers (combining SL and QP) on 10-best lists Diff | count | Baseline | SL + QP
---|---|---|---
|  | EM% | EX% | EM% | EX%
Easy | 248 | 89.1 | 91.9 | 89.5 | 94.8
Med | 446 | 80.9 | 84.8 | 81.6 | 87.9
Hard | 174 | 65.5 | 71.3 | 68.4 | 73.6
Extra | 166 | 48.8 | 54.8 | 49.4 | 55.4
Total | 1034 | 75.1 | 79.4 | 76.1 | 81.9

##  6 Discussion on Errors

Given that proposed reranker still has a gap with Oracle, we analyze errors
with metrics computation and annotation.

###  6.1 Metrics Errors

PK/FK Swaps. EM implementation by Spider is done by replacing foreign keys
with primary keys. This is problematic and may lead to false positives. The
following example contains a prediction that is marked as exact match
incorrectly.

We fixed the EM implementation and obtained the revised $EM^{*}$ in Table 9.
Generally there is a 3-4% absolute gap between the original and revised EM
implementation.

Table 9: Revised EM implementation. Method | T5-large LM | T5-3b
---|---|---
|  $EM$% |  $EM^{*}$% |  $EM$% |  $EM^{*}$%
PICARD | 75.1 | 72.1 | 75.6 | 71.9
SL + QP | 76.1 | 72.9 | 76.4 | 72.5

Query rewrite errors. Since SQL queries can sometimes be rewritten in multiple
ways, EM can yield false negatives. We conducted error analysis over
hypotheses selected after our combined reranking system (SL + QP) using
finetuned T5-large model with PICARD. We found that 7.9% of predictions are
correct using EX, but incorrect with EM; also, 4.8% of hypotheses are correct
SQL but different to the corresponding groundtruth SQL. In such examples with
SQL rewrites, EX serves as a complementary metric to EM.

Content-only match errors. As described in [5], it is possible to obtain a
predicted SQL that returns the same result as the groundtruth SQL but
semantically different. We found that 4.4% of the hypotheses on DEV data
generated from T5-Large model using PICARD are false positives of this type.

Redundant column errors. We found that some predicted or groundtruth SQL
queries contain redundant columns. The columns repeat information or do not
need to be included, however retaining such column won’t affect the semantics
of the query, but can result in EX errors.

###  6.2 Annotation Errors

We found 5 unique groundtruth SQL queries with annotation errors in DEV set.
In Spider, some natural language queries are presented multiple times;
consequently, the 5 unique groundtruth SQL queries are assigned to 7 DEV
examples. We show an example with annotation error whose natural language
query has been written in two different ways below. The annotation error is at
least 0.7%.

###  6.3 Summary and Outlook on Errors

As we have seen, 0.7% of the DEV examples have incorrect groundtruth SQL
queries. We also identified issues with EM implementation in Spider – foreign
keys are automatically replaced with the corresponding primary keys, resulting
in 3-4% of the predictions on DEV examples being incorrectly marked as exact
matches (false positives). Also, each example in Spider is annotated with only
one groundtruth SQL – leading to about 4.8% of the predictions on DEV examples
being false negatives. Using EX, results in 4.4% of the predictions on DEV
being false positive, while another 0.8% of the predictions are false
negative. Our studies show that metrics and annotation need further research
attention on Spider.

##  7 Conclusions

We began with Oracle studies, showing large gains can be obtained with n-best
hypotheses reranking for a SOTA Text-to-SQL system. Motivated by correctness
and coherence issues in text generation using large PLMs, we proposed two
reranking approaches: a) schema linking; b) query plan modeling. On a
competitive Spider dataset, applying our proposed reranking methods to 10-best
hypotheses obtained from a SOTA system (PICARD), we achieve improvements of
$1\%$ in EM and $~{}2.5\%$ in EX. We analyzed errors in metrics computation
and annotation – showing these can be significant factors to improving models
and evaluation on Spider.

## References

  * [1] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu,  “Exploring the limits of transfer learning with a unified text-to-text transformer,”  Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020.
  * [2] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer,  “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,”  arXiv preprint arXiv:1910.13461, 2019.
  * [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.,  “Language models are few-shot learners,”  Proc. of NeurIPS, vol. 33, pp. 1877–1901, 2020.
  * [4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.,  “Evaluating large language models trained on code,”  arXiv preprint arXiv:2107.03374, 2021.
  * [5] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al.,  “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,”  arXiv preprint arXiv:1809.08887, 2018.
  * [6] Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al.,  “Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases,”  arXiv preprint arXiv:1909.05378, 2019.
  * [7] Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, et al.,  “Sparc: cross-domain semantic parsing in context,”  arXiv preprint arXiv:1906.02285, 2019.
  * [8] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al.,  “Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models,”  arXiv preprint arXiv:2201.05966, 2022.
  * [9] Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua,  “Re-examining the role of schema linking in text-to-sql,”  in Proc. of EMNLP, 2020, pp. 6943–6954.
  * [10] Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang,  “Towards complex text-to-sql in cross-domain database with intermediate representation,”  arXiv preprint arXiv:1905.08205, 2019.
  * [11] Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev,  “Typesql: Knowledge-based type-aware neural text-to-sql generation,”  arXiv preprint arXiv:1804.09769, 2018.
  * [12] Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson,  “Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers,”  arXiv preprint arXiv:1911.04942, 2019.
  * [13] Pengcheng Yin and Graham Neubig,  “A syntactic neural model for general-purpose code generation,”  arXiv preprint arXiv:1704.01696, 2017.
  * [14] Alane Laughlin Suhr, Kenton Lee, Ming-Wei Chang, and Pete Shaw,  “Exploring unexplored generalization challenges for cross-database semantic parsing,”  2020\.
  * [15] Xi Victoria Lin, Richard Socher, and Caiming Xiong,  “Bridging textual and tabular data for cross-domain text-to-sql semantic parsing,”  arXiv preprint arXiv:2012.12627, 2020.
  * [16] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau,  “Picard: Parsing incrementally for constrained auto-regressive decoding from language models,”  arXiv preprint arXiv:2109.05093, 2021.
  * [17] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei,  “Scaling laws for neural language models,”  arXiv preprint arXiv:2001.08361, 2020.
  * [18] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.,  “Webgpt: Browser-assisted question-answering with human feedback,”  arXiv preprint arXiv:2112.09332, 2021.
  * [19] Richard Schwartz, Long Nguyen, and John Makhoul,  “Multiple-pass search strategies,”  in Automatic Speech and Speaker Recognition, pp. 429–456. Springer, 1996.
  * [20] Long Nguyen and Richard M Schwartz,  “Efficient 2-pass n-best decoder.,”  in EuroSpeech. Citeseer, 1997.
  * [21] Richard Zens, Oliver Bender, Saša Hasan, Shahram Khadivi, Evgeny Matusov, Jia Xu, Yuqi Zhang, and Hermann Ney,  “The rwth phrase-based statistical machine translation system,”  in Proc. of IWSLT, 2005.
  * [22] Ngoc-Quang Luong, Laurent Besacier, and Benjamin Lecouteux,  “Word confidence estimation for smt n-best list re-ranking,”  in Proc. of the Workshop on HaCaT during EACL, 2014.
  * [23] Michael Collins and Terry Koo,  “Discriminative reranking for natural language parsing,”  Computational Linguistics, vol. 31, no. 1, pp. 25–70, 2005.
  * [24] Ruifang Ge and Raymond Mooney,  “Discriminative reranking for semantic parsing,”  in Proc. of COLING/ACL, 2006, pp. 263–270.
  * [25] Binyuan Hui, Ruiying Geng, Qiyu Ren, Binhua Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, Pengfei Zhu, and Xiaodan Zhu,  “Dynamic hybrid relation exploration network for cross-domain context-dependent semantic parsing,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, vol. 35, pp. 13116–13124.
  * [26] Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin,  “Document ranking with a pretrained sequence-to-sequence model,”  arXiv preprint arXiv:2003.06713, 2020.
  * [27] Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang,  “Retrieval, re-ranking and multi-task learning for knowledge-base question answering,”  in Proc. of the European Chapter of ACL: Main Volume, 2021, pp. 347–357.
  * [28] Manny Rayner, David Carter, Vassilios Digalakis, and Patti Price,  “Combining knowledge sources to reorder n-best speech hypothesis lists,”  arXiv preprint cmp-lg/9407010, 1994.
  * [29] Rebecca Jonson,  “Dialogue context-based re-ranking of asr hypotheses,”  in SLT. IEEE, 2006, pp. 174–177.
  * [30] Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, Chandra Khatri, and Peter Relan,  “Bertrand-dr: Improving text-to-sql using a discriminative re-ranker,”  arXiv preprint arXiv:2002.00557, 2020.
  * [31] Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz, and Bonnie Dorr,  “Combining outputs from multiple machine translation systems,”  in HLT 2007: The Conference of NAACL; Proc. of the Main Conference, 2007, pp. 228–235.
  * [32] Nguyen Bach, Fei Huang, and Yaser Al-Onaizan,  “Goodness: A method for measuring machine translation confidence,”  in Proc. of the 49th Annual Meeting of ACL: HLT, 2011, pp. 211–219.
  * [33] Robert Porzel and Iryna Gurevych,  “Contextual coherence in natural language processing,”  in International and Interdisciplinary Conference on Modeling and Using Context. Springer, 2003, pp. 272–285.
  * [34] Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt,  “The curious case of hallucinations in neural machine translation,”  arXiv preprint arXiv:2104.06683, 2021.
  * [35] Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo,  “Hallucinations in neural machine translation,”  2018\.
  * [36] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald,  “On faithfulness and factuality in abstractive summarization,”  arXiv preprint arXiv:2005.00661, 2020.
  * [37] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi,  “The curious case of neural text degeneration,”  arXiv preprint arXiv:1904.09751, 2019.
  * [38] Clara Meister and Ryan Cotterell,  “Language model evaluation beyond perplexity,”  arXiv preprint arXiv:2106.00085, 2021.
  * [39] Nikolay Malkin, Zhen Wang, and Nebojsa Jojic,  “Coherence boosting: When your pretrained language model is not paying enough attention,”  in Proc. of the 60th Annual Meeting of ACL (Volume 1: Long Papers), 2022, pp. 8214–8236.
  * [40] Ning Li, Bethany Keller, Mark Butler, and Daniel Cer,  “Seqgensql–a robust sequence generation model for structured query language,”  arXiv preprint arXiv:2011.03836, 2020.
  * [41] Pengcheng Yin and Graham Neubig,  “Reranking for neural semantic parsing,”  in Proc. of the 57th Annual Meeting of ACL, 2019.
  * [42] Ben Bogin, Matt Gardner, and Jonathan Berant,  “Representing schema structure with graph neural networks for text-to-sql parsing,”  arXiv preprint arXiv:1905.06241, 2019.
  * [43] Zhi Chen, Lu Chen, Yanbin Zhao, Ruisheng Cao, Zihan Xu, Su Zhu, and Kai Yu,  “Shadowgnn: Graph projection neural network for text-to-sql parser,”  arXiv preprint arXiv:2104.04689, 2021.
  * [44] DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin,  “Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases,”  Computational Linguistics, vol. 47, no. 2, pp. 309–332, 2021.
  * [45] Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova,  “Compositional generalization and natural language variation: Can a semantic parsing approach handle both?,”  arXiv preprint arXiv:2010.12725, 2020.


* Gorfine et al. (2021) Gorfine, M., N. Keret, A. Ben Arie, D. Zucker, and L. Hsu (2021).  Marginalized frailty-based illness-death model: application to the uk-biobank survival data.  Journal of the American Statistical Association 116(535), 1155–1167.
  * Hua et al. (2016) Hua, Q., X. Lv, X. Gu, Y. Chen, H. Chu, M. Du, W. Gong, M. Wang, and Z. Zhang (2016).  Genetic variants in lncrna h19 are associated with the risk of bladder cancer in a chinese population.  Mutagenesis 31(5), 531–538.
  * Huang and Qin (2013) Huang, C.-Y. and J. Qin (2013).  Semiparametric estimation for the additive hazards model with left-truncated and right-censored data.  Biometrika 100(4), 877–888.
  * Huyghe et al. (2017) Huyghe, J. R., S. Chen, H. M. Kang, T. Harrison, S. I. Berndt, S. Bézieau, H. Brenner, G. Casey, A. T. Chan, J. Chang-Claude, et al. (2017).  The contribution of rare and low-frequency variants to colorectal cancer heritability.  Cancer Research 77(13_Supplement), 1304–1304.
  * Janson (1984) Janson, S. (1984).  The asymptotic distributions of incomplete u-statistics.  Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete 66(4), 495–505.
  * Janssen (1994) Janssen, P. (1994).  Weighted bootstrapping of u-statistics.  Journal of Statistical Planning and Inference 38(1), 31–41.
  * Jeon et al. (2018) Jeon, J., M. Du, R. E. Schoen, M. Hoffmeister, P. A. Newcomb, S. I. Berndt, B. Caan, P. T. Campbell, A. T. Chan, J. Chang-Claude, et al. (2018).  Determining risk of colorectal cancer and starting age of screening based on lifestyle, environmental, and genetic factors.  Gastroenterology 154(8), 2152–2164.
  * Kalbfleisch (1978) Kalbfleisch, J. D. (1978).  Likelihood methods and nonparametric tests.  Journal of the American Statistical Association 73(361), 167–170.
  * Kaplan and Meier (1958) Kaplan, E. L. and P. Meier (1958).  Nonparametric estimation from incomplete observations.  Journal of the American Statistical Association 53(282), 457–481.
  * Karahalil et al. (2006) Karahalil, B., N. A. Kocabas, and T. ÖZÇELIK (2006).  Dna repair gene polymorphisms and bladder cancer susceptibility in a turkish population.  Anticancer Research 26(6C), 4955–4958.
  * Keret and Gorfine (2023) Keret, N. and M. Gorfine (2023).  Analyzing big ehr data—optimal cox regression subsampling procedure with rare events.  Journal of the American Statistical Association, 1–14.
  * Kiemeney et al. (2010) Kiemeney, L. A., P. Sulem, S. Besenbacher, S. H. Vermeulen, A. Sigurdsson, G. Thorleifsson, D. F. Gudbjartsson, S. N. Stacey, J. Gudmundsson, C. Zanon, et al. (2010).  A sequence variant at 4p16. 3 confers susceptibility to urinary bladder cancer.  Nature Genetics 42(5), 415–419.
  * Kiemeney et al. (2008) Kiemeney, L. A., S. Thorlacius, P. Sulem, F. Geller, K. K. Aben, S. N. Stacey, J. Gudmundsson, M. Jakobsdottir, J. T. Bergthorsson, A. Sigurdsson, et al. (2008).  Sequence variant on 8q24 confers susceptibility to urinary bladder cancer.  Nature Genetics 40(11), 1307–1312.
  * Kim et al. (2005) Kim, E.-J., P. Jeong, C. Quan, J. Kim, S.-C. Bae, S. J. Yoon, J.-W. Kang, S.-C. Lee, J. J. Wee, and W.-J. Kim (2005).  Genotypes of tnf-$\alpha$, vegf, hogg1, gstm1, and gstt1: useful determinants for clinical outcome of bladder cancer.  Urology 65(1), 70–75.
  * Klein and Moeschberger (2003) Klein, J. P. and M. L. Moeschberger (2003).  Survival analysis: techniques for censored and truncated data, Volume 1230.  Springer.
  * Kraft et al. (2009) Kraft, P., E. Zeggini, and J. P. Ioannidis (2009).  Replication in genome-wide association studies.  Statistical Science 24(4), 561.
  * Lehmann et al. (2010) Lehmann, M.-L., S. Selinski, M. Blaszkewicz, M. Orlich, D. Ovsiannikov, O. Moormann, C. Guballa, A. Kress, M. C. Truss, H. Gerullis, et al. (2010).  Rs710521 [a] on chromosome 3q28 close to tp63 is associated with increased urinary bladder cancer risk.  Archives of Toxicology 84, 967–978.
  * Li and Peng (2011) Li, R. and L. Peng (2011).  Quantile regression for left-truncated semicompeting risks data.  Biometrics 67(3), 701–710.
  * Li and Peng (2014) Li, R. and L. Peng (2014).  Varying coefficient subdistribution regression for left-truncated semi-competing risks data.  Journal of Multivariate Analysis 131, 65–78.
  * Liang and Qin (2000) Liang, K.-Y. and J. Qin (2000).  Regression analysis under non-standard situations: a pairwise pseudolikelihood approach.  Journal of the Royal Statistical Society: Series B (Statistical Methodology) 62(4), 773–786.
  * Lipunova et al. (2019) Lipunova, N., A. Wesselius, K. K. Cheng, F.-J. van Schooten, R. T. Bryan, J.-B. Cazier, T. E. Galesloot, L. A. Kiemeney, and M. P. Zeegers (2019).  Genome-wide association study for tumour stage, grade, size, and age at diagnosis of non–muscle-invasive bladder cancer.  European Urology Oncology 2(4), 381–389.
  * Luo et al. (2021) Luo, F., Y. Wu, Q. Ding, Y. Yuan, and W. Jia (2021).  Rs884225 polymorphism is associated with primary hypertension by compromising interaction between epithelial growth factor receptor (egfr) and mir-214.  Journal of Cellular and Molecular Medicine 25(8), 3714–3723.
  * Ma et al. (2012) Ma, L., H. Chu, M. Wang, D. Shi, D. Zhong, P. Li, N. Tong, C. Yin, and Z. Zhang (2012).  hogg1 s er326 c ys polymorphism is associated with risk of bladder cancer in a c hinese population: A case-control study.  Cancer Science 103(7), 1215–1220.
  * Ma et al. (2013) Ma, Z., Q. Hu, Z. Chen, S. Tao, L. Macnamara, S.-T. Kim, L. Tian, K. Xu, Q. Ding, S. L. Zheng, et al. (2013).  Systematic evaluation of bladder cancer risk-associated single-nucleotide polymorphisms in a chinese population.  Molecular Carcinogenesis 52(11), 916–921.
  * Mamdouh et al. (2022) Mamdouh, S., F. Khorshed, G. Hammad, K. Elesaily, G. Safwat, O. Hammam, and T. Aboushousha (2022).  Molecular detection of genetic susceptibility to bladder cancer in egyptian patients.  Asian Pacific Journal of Cancer Prevention: APJCP 23(1), 221.
  * Meng et al. (2017) Meng, X.-Y., M.-J. Shi, J.-F. Chen, Y. Liao, B.-W. Hu, A. Hireche, et al. (2017).  Association between the tacc3 rs798766 polymorphism and risk of urinary bladder cancer: a synthesis based on current evidence.  Disease Markers 2017.
  * Nelson (1972) Nelson, W. (1972).  Theory and applications of hazard plotting for censored failure data.  Technometrics 14(4), 945–966.
  * Newey (1991) Newey, W. K. (1991).  Uniform convergence in probability and stochastic equicontinuity.  Econometrica: Journal of the Econometric Society, 1161–1167.
  * Pang et al. (2018) Pang, Y., C. Kartsonaki, Y. Guo, Y. Chen, L. Yang, Z. Bian, F. Bragg, I. Y. Millwood, E. Mao, Y. Li, et al. (2018).  Adiposity and risks of colorectal and small intestine cancer in chinese adults: a prospective study of 0.5 million people.  British Journal of Cancer 119(2), 248–250.
  * Polat et al. (2019) Polat, F., M. Yilmaz, and S. B. Diler (2019).  The association of mynn and terc gene polymorphisms and bladder cancer in a turkish population.  Urology Journal 16(1), 50.
  * Qian and Betensky (2014) Qian, J. and R. A. Betensky (2014).  Assumptions regarding right censoring in the presence of left truncation.  Statistics & Probability Letters 87, 12–17.
  * Rafnar et al. (2009) Rafnar, T., P. Sulem, S. N. Stacey, F. Geller, J. Gudmundsson, A. Sigurdsson, M. Jakobsdottir, H. Helgadottir, S. Thorlacius, K. K. Aben, et al. (2009).  Sequence variants at the tert-clptm1l locus associate with many cancer types.  Nature Genetics 41(2), 221–227.
  * Rafnar et al. (2014) Rafnar, T., P. Sulem, G. Thorleifsson, S. H. Vermeulen, H. Helgason, J. Saemundsdottir, S. A. Gudjonsson, A. Sigurdsson, S. N. Stacey, J. Gudmundsson, et al. (2014).  Genome-wide association study yields variants at 20p12. 2 that associate with urinary bladder cancer.  Human Molecular Genetics 23(20), 5545–5557.
  * Rafnar et al. (2011) Rafnar, T., S. H. Vermeulen, P. Sulem, G. Thorleifsson, K. K. Aben, J. A. Witjes, A. J. Grotenhuis, G. W. Verhaegh, C. A. Hulsbergen-van de Kaa, S. Besenbacher, et al. (2011).  European genome-wide association study identifies slc14a1 as a new urinary bladder cancer susceptibility gene.  Human Molecular Genetics 20(21), 4268–4281.
  * Reid and Crépeau (1985) Reid, N. and H. Crépeau (1985).  Influence functions for proportional hazards regression.  Biometrika 72(1), 1–9.
  * Rothman et al. (2010) Rothman, N., M. Garcia-Closas, N. Chatterjee, N. Malats, X. Wu, J. D. Figueroa, F. X. Real, D. Van Den Berg, G. Matullo, D. Baris, et al. (2010).  A multi-stage genome-wide association study of bladder cancer identifies multiple susceptibility loci.  Nature Genetics 42(11), 978–984.
  * Saarela et al. (2009) Saarela, O., S. Kulathinal, and J. Karvanen (2009).  Joint analysis of prevalence and incidence data using conditional likelihood.  Biostatistics 10(3), 575–587.
  * Selinski et al. (2012) Selinski, S., M.-L. Lehmann, M. Blaszkewicz, D. Ovsiannikov, O. Moormann, C. Guballa, A. Kress, M. C. Truß, H. Gerullis, T. Otto, et al. (2012).  Rs11892031 [a] on chromosome 2q37 in an intronic region of the ugt1a locus is associated with urinary bladder cancer risk.  Archives of Roxicology 86, 1369–1378.
  * Stern et al. (2009) Stern, M. C., D. Van Den Berg, J.-M. Yuan, D. V. Conti, M. Gago-Dominguez, M. C. Pike, Y.-B. Xiang, Y.-T. Gao, and V. K. Cortessis (2009).  Sequence variant on 3q28 and urinary bladder cancer risk: findings from the los angeles-shanghai bladder case-control study.  Cancer Epidemiology, Biomarkers & Prevention 18(11), 3057–3061.
  * Tsai (1990) Tsai, W.-Y. (1990).  Testing the assumption of independence of truncation time and failure time.  Biometrika 77(1), 169–177.
  * Tsiatis (2006) Tsiatis, A. A. (2006).  Semiparametric theory and missing data.
  * Vakulenko-Lagun and Mandel (2016) Vakulenko-Lagun, B. and M. Mandel (2016).  Comparing estimation approaches for the illness–death model under left truncation and right censoring.  Statistics in Medicine 35(9), 1533–1548.
  * Vakulenko-Lagun et al. (2017) Vakulenko-Lagun, B., M. Mandel, and Y. Goldberg (2017).  Nonparametric estimation in the illness-death model using prevalent data.  Lifetime Data Analysis 23(1), 25–56.
  * Van der Vaart (2000) Van der Vaart, A. W. (2000).  Asymptotic statistics, Volume 3.  Cambridge university press.
  * Wang et al. (2014) Wang, M., H. Chu, Q. Lv, L. Wang, L. Yuan, G. Fu, N. Tong, C. Qin, C. Yin, Z. Zhang, et al. (2014).  Cumulative effect of genome-wide association study-identified genetic variants for bladder cancer.  International Journal of Cancer 135(11), 2653–2660.
  * Wang et al. (2009) Wang, M., M. Wang, W. Zhang, L. Yuan, G. Fu, Q. Wei, and Z. Zhang (2009).  Common genetic variants on 8q24 contribute to susceptibility to bladder cancer in a chinese population.  Carcinogenesis 30(6), 991–996.
  * Wang et al. (2010) Wang, S., J. Tang, M. Wang, L. Yuan, and Z. Zhang (2010).  Genetic variation in psca and bladder cancer susceptibility in a chinese population.  Carcinogenesis 31(4), 621–624.
  * Wu et al. (2018) Wu, F., S. Kim, J. Qin, R. Saran, and Y. Li (2018).  A pairwise likelihood augmented cox estimator for left-truncated data.  Biometrics 74(1), 100–108.
  * Wu et al. (2009) Wu, X., Y. Ye, L. A. Kiemeney, P. Sulem, T. Rafnar, G. Matullo, D. Seminara, T. Yoshida, N. Saeki, A. S. Andrew, et al. (2009).  Genetic variation in the prostate stem cell antigen gene psca confers susceptibility to urinary bladder cancer.  Nature Genetics 41(9), 991–995.
  * Xiao et al. (2015) Xiao, X., G. Ma, S. Li, M. Wang, N. Liu, L. Ma, Z. Zhang, H. Chu, Z. Zhang, and S.-L. Wang (2015).  Functional por a503v is associated with the risk of bladder cancer in a chinese population.  Scientific Reports 5(1), 11751.
  * Xu et al. (2010) Xu, J., J. D. Kalbfleisch, and B. Tai (2010).  Statistical analysis of illness–death processes and semicompeting risks data.  Biometrics 66(3), 716–725.
  * Yuan et al. (2010) Yuan, L., X. Gu, J. Shao, M. Wang, M. Wang, Q. Zhu, and Z. Zhang (2010).  Cyclin d1 g870a polymorphism is associated with risk and clinicopathologic characteristics of bladder cancer.  DNA and Cell Biology 29(10), 611–617.
  * Zhang et al. (2014) Zhang, B., W.-H. Jia, K. Matsuda, S.-S. Kweon, K. Matsuo, Y.-B. Xiang, A. Shin, S. H. Jee, D.-H. Kim, Q. Cai, et al. (2014).  Large-scale genetic study in east asians identifies six new loci associated with colorectal cancer risk.  Nature Genetics 46(6), 533–542.
  * Zhang et al. (2014) Zhang, Y., Y. Sun, T. Chen, H. Hu, W. Xie, Z. Qiao, N. Ding, L. Xie, S. Li, W. Wang, et al. (2014).  Genetic variations rs11892031 and rs401681 are associated with bladder cancer risk in a chinese population.  International Journal of Molecular Sciences 15(11), 19330–19341.
  * Zhu and Kosorok (2012) Zhu, R. and M. R. Kosorok (2012).  Recursively imputed survival trees.  Journal of the American Statistical Association 107(497), 331–340.