# Mirror Matching: Document Matching Approach in Seed-driven Document Ranking
for Medical Systematic Reviews

Grace E. Lee  School of Computer Science and Engineering, Nanyang
Technological University, Singapore<EMAIL_ADDRESS>and  Aixin Sun
School of Computer Science and Engineering, Nanyang Technological University,
Singapore<EMAIL_ADDRESS>
###### Abstract.

When medical researchers conduct a systematic review (SR), screening studies
is the most time-consuming process: researchers read several thousands of
medical literature and manually label them relevant or irrelevant. Screening
prioritization (_i.e.,_ document ranking) is an approach for assisting
researchers by providing document rankings where relevant documents are ranked
higher than irrelevant ones. Seed-driven document ranking (SDR) uses a known
relevant document (_i.e.,_ seed) as a query and generates such rankings.
Previous work on SDR seeks ways to identify different term weights in a query
document and utilizes them in a retrieval model to compute ranking scores.
Alternatively, we formulate the SDR task as finding similar documents to a
query document and produce rankings based on similarity scores. We propose a
document matching measure named Mirror Matching, which calculates matching
scores between medical abstract texts by incorporating common writing
patterns, such as background, method, result, and conclusion in order. We
conduct experiments on CLEF 2019 eHealth Task 2 TAR dataset, and the empirical
results show this simple approach achieves the higher performance than
traditional and neural retrieval models on Average Precision and Precision-
focused metrics.

Document ranking, Medical systematic reviews, Document matching

##  1\. Introduction

In evidence-based medicine, a systematic review (SR) plays an important role
in informing healthcare professionals and policy makers. It provides a
conclusive answer for a clinical question by synthesizing all existing
studies. When researchers write a SR, finding all relevant studies is a
critical goal to derive a reliable answer. Researchers strive for the goal by
carefully following sequential steps: (i) define a clinical question and
relevance conditions, (ii) retrieve documents (candidate documents) from
digital libraries, (iii) screen the candidate documents to identify relevant
documents, (iv) analyze and synthesize the relevant documents and derive an
overall conclusion (O’Mara-Eves et al., 2015).

Screening is the most time-consuming and expensive step above all because of
exhaustive manual labeling. SR researchers read abstracts of all candidate
documents (typically around 6,000 documents) and identify intermediate
relevant documents based on the content presented in abstracts (_i.e.,_
abstract stage screening). They then prepare and read full documents of the
intermediate relevant documents, and determine final relevant documents
(_i.e.,_ full document stage screening). This manual and multi-stage screening
step takes weeks to months to complete.

Screening prioritization (_i.e.,_ document ranking) is an approach for
assisting researchers by providing document rankings where relevant documents
are ranked higher than irrelevant documents. This approach makes the time-
consuming screening step more efficient by allowing the abstract and full
stage screenings to be conducted in parallel: in the abstract stage screening
researchers identify intermediate relevant documents as early as possible from
ranked documents, and they can start the full document screening on the
intermediate documents, while the rest documents are still in the abstract
screening. And also they are likely to be irrelevant. In this work, we focus
on improving the abstract stage screening in screening prioritization.

Seed-driven document ranking (SDR) approach uses a known relevant document
(seed) as a query in screening prioritization. Existing work in SDR seeks ways
to measure different terms weights in a query document and combine them in a
retrieval model to compute ranking scores (Lee and Sun, 2018). In this work,
we study an alternative way in SDR: ranking by similarities to a query
document. We propose a simple document matching measure named Mirror Matching
(M-Match, for short), which leverages a common writing pattern in abstracts in
medical literature. Abstract text is comprised of background, method, result
and conclusion in order. The proposed measure matches terms in two documents
with respect to their positions, so that it enables to match terms that are
used in the same section.

The proposed measure is motivated by observations on positional patterns of
PICO elements. PICO elements indicate four elements, Patient, Intervention,
Comparator, and Outcome, and they are the main components of relevance
conditions in systematic reviews. Hence, the relevance of documents is mainly
determined by PICO elements in documents. A document must have the same PICO
elements as a query document does in order to be a relevant document. For
example, if a term ‘insulin’ is in O element in a document and it is in I
element in a query, the document can’t be a relevant document, even though
both contain the same term. In other words, terms must be compared with
respect to each of PICO elements in order to identify relevant documents. We
observed that each PICO element appears with different patterns in abstract
texts. For example, P element frequently appears in the beginning of document,
and most O element terms appear at the end of document. Using positional
information is a relaxed and cost-effective way to match terms in regard to
PICO elements, compared to specifically identifying PICO element terms, which
is not a simple task (Brockmeier et al., 2019; Lee and Sun, 2019).

M-Match is a document similarity measure, inspired by the asymmetric scoring
in retrieval models (_i.e.,_ BM25). We discuss the connections and differences
between M-Match and existing document similarity measures and retrieval
models. We conduct experiments on CLEF 2019 eHealth Task 2 TAR dataset and
compare M-Match to them as well. In recent years, neural retrieval models gain
significant attention for matching texts (Onal et al., 2018; Craswell et al.,
2018; Guo et al., 2019b). We also evaluate M-Match in comparison to several
neural IR models. The evaluation results show that the proposed model
outperforms all baseline models on Average Precision and Precision-focused
metrics.

The main contributions of this work are as follows. First, we study the SDR
task in the context of the document matching for the first time and propose
the effective document matching measure for medical abstracts. It allows SR
researchers to find multiple relevant documents just after screening a couple
of top-ranked documents. Second, the proposed approach is extensively
evaluated compared to a wide range of baseline models from document similarity
measures to classical to neural IR models. Lastly, we discuss the strength and
the weakness of M-Match and suggest ways to utilize M-Match at most in
practice while minimizing its limitation.

##  2\. Related Work

Improving the screening process. Several approaches have been proposed to
reduce the costs of the expensive screening step in SRs (O’Mara-Eves et al.,
2015). They are mainly divided into two categories based on its end goals. One
category is to directly help the screening step to be more effective, either
by decreasing the number of candidate documents to be screened or by reducing
the number of SR experts needed. This direct goal is pursued by training a
classification model. However, adopting a trained classifier is prone to fail
to find all relevant documents, because it is hard to ensure the nearly
perfect level of accuracy and also different SRs are meant to answer different
clinical questions. That is, each SR has its own topic and it is hard to learn
a general classification model for all SRs. Moreover, the small number of
relevant documents in each SR makes training a model more challenging. With
this regards, several studies suggest active learning setting through
relevance feedback (Cormack and Grossman, 2014; Wallace et al., 2010).

The other category is a relatively indirect way to improve the screening
process. While the number of documents to be screened remains unchanged, it
assists SR experts during the manual screening process. An advantage of this
approach is not to imperil finding all relevant documents. Screening
prioritization is an approach in this category. As aforementioned, it makes
the screening process efficient by allowing abstract screening and full
document screening to be carried out simultaneously. Another approach is to
visualize groups of similar candidate documents and provide their topics. In
this way, SR experts are able to screen similar documents together, instead of
documents in a random order. We refer to (O’Mara-Eves et al., 2015) for more
details.

Ranking documents using a query document. Using an example document as a query
is an important research problem in various domains, such as patent search and
literature search (Yang et al., 2009; Weng et al., 2011; Kim and Croft, 2015,
2014; Yang et al., 2018; Lee and Sun, 2018). Most current solutions first
transform a long query to a short query, and then apply a retrieval function.
As most retrieval functions are optimized for short queries, verbose queries
often become a challenge in computation and query understanding (Gupta et al.,
2015).

In screening prioritization, formulating a short and meaningful query from a
known relevant document is non-trivial because of the detailed relevance
conditions. There are a few possible approaches: keyword extraction and PICO
element extraction. Keyword extraction techniques can be adopted, but they are
prone to lose detailed yet important relevance conditions such as age or
gender of patients in P-element. Extracting medical concepts can be an
alternative (Lee and Sun, 2018), but this approach has to rely on the accuracy
of toolkits for medical concept extraction. PICO element extraction can be
employed to extract relevance conditions appeared in a query document (Lee and
Sun, 2019; Boudin et al., 2010c; Brockmeier et al., 2019). Extracting PICO
elements without loss of relevance information therefore becomes a challenge.

Document matching approaches have been less explored in ranking by a query
document. Computing pairwise document matching scores in large-scale corpora
online is costly. Several studies propose ways to improve the efficiency in
pairwise similarity search problems (Bayardo et al., 2007; Lin, 2009; Elsayed
et al., 2008). However, in screening prioritization, candidate documents are
typically in the scale of several thousands documents. In the CLEF TAR 2019
dataset, the average number of candidate documents is about 6,000. The
computation on a few thousand documents is small compared to the costs of
manual screening which takes place on prioritized documents for weeks. The
effectiveness of prioritizing documents is more crucial than the efficiency in
improving the manual screening process.

##  3\. Mirror Matching

We focus on screening prioritization using a known relevant document in
systematic reviews. The task is to rank candidate documents where the unknown
relevant documents are aimed to be at top positions. We view this task in the
context of document matching. Given a document pair (_i.e.,_ the known
relevant document and a candidate document), we compute a matching score
between the two documents and rank candidate documents according to their
matching scores with the known relevant document. Existing pairwise document
similarity measures can also be used to compute the matching scores. For easy
presentation, in the following, we refer the known relevant document as a
query document.

###  3.1. Overview

Mirror Matching is a simple two-way semantic matching for a pair of documents.
It examines (i) whether a candidate document is the best reflection of a query
document, and (ii) whether a query document is the best reflection of a
candidate document. Specifically, given a query document $Q$ and a candidate
document $D$, it computes two scores: $sc_{Q\shortrightarrow D}$ and
$sc_{D\shortrightarrow Q}$

(1) $\text{M-Match}(Q,D)=sc_{Q\shortrightarrow D}+sc_{D\shortrightarrow Q}$

where $sc_{Q\shortrightarrow D}$ is a candidate document score computed by
matching terms in $Q$ to terms in $D$; $sc_{D\shortrightarrow Q}$ is a query
document score computed by matching terms in $D$ to terms in $Q$.

As shown above, in M-Match a matching score is computed in two directions
(from $Q$ to $D$ and from $D$ to $Q$). Hence, we call it two-way matching. We
design the two scores to affect the final matching score $\text{M-Match}(Q,D)$
equally.111We experiment with alternative combinations, multiplication of two
scores, and observe similar performance.

Before explaining further details of M-Match, we present an analysis on
positions of PICO elements. Our analysis shows that PICO elements appear with
positional patterns in medical literature writing. This analysis serves as the
base to design our M-Match to be position-based when computing
$sc_{Q\shortrightarrow D}$ and $sc_{D\shortrightarrow Q}$.

###  3.2. Analysis: Distributions of PICO Elements

Medical scientific abstracts are carefully written by authors to provide
comprehensive overview of a full scientific literature. They summarize the
major aspects of the entire literature in a prescribed sequence that includes
background/purpose, method, results, and conclusion. For example, American
National Standard Institute advises authors to follow the rhetorical sequence
when they write abstracts ((US) and Institute, 1997). Furthermore, the ordered
elements are explicitly indicated with headings in some abstracts.

Our hypothesis is that PICO words appear in patterns in medical scientific
abstracts. We study the positional distributions of PICO elements using EBM-
NLP dataset (Nye et al., 2018). EBM-NLP dataset provides manually annotated
PICO spans and every token in documents is labeled. The interests on positions
of PICO elements are not new (Boudin et al., 2010b, a). Existing studies
detect PICO elements using string matching with PICO keywords and report
coarse-grained (_e.g.,_ chunk-level or sentence-level) distributions of PICO
elements. However, a string matching occurs common vocabulary mismatch between
words describing PICO elements in medical literature and the PICO keywords.
Here, we cover various surface forms of PICO elements such as paraphrasing and
synonyms using the manual PICO annotations available in EBM-NLP dataset.
Besides, this analysis shows distributions of PICO elements in a fine-grained
level (_e.g.,_ words).

More specifically, EBM-NLP dataset contains about 5,000 PubMed documents
(title and abstract only) with PICO span annotations. The dataset combines
C-element and I-element as one label, I-element. Hence, every single-word term
has one of the four labels: Patient, Intervention, Outcome, or None. When PIO
elements are described in a long phrase, consecutive terms are labeled with
the same labels. EBM-NLP consists of the labels by multiple annotators and the
combined label that aggregates the labels by multiple annotators. We use the
combined version of labels.

Figure 1. Distributions of Patient, Intervention, and Outcome elements in
medical literature (title and abstract). Each row indicates a document. Within
a row dark blocks are terms labeled with a given element (best viewed in
color).

Figure 1 shows distributions of P-, I-, and O-elements. Documents are sorted
by their lengths in number of words, in descending order. About 250 documents
in EBM-NLP dataset have only a title, so that their lengths are much shorter
than the rest and hard to show any patterns (see the bottom of the figure). In
Figure 1 each row is a document, consisting of a sequence of terms. Terms in
different colors indicate with or without a corresponding element label. Since
documents have different lengths, the positions of terms within a row
represent their relative positions in the document.

As shown in Figure 1, P-element often appears at the beginning of documents
where a title, background and/or purpose are likely to be stated. In the
second half of documents it seldom appears. Next, I-element tends to appear in
the beginning of documents, similar to P, but its occurrence in the first half
of documents is not as apparent as that of P-element. Instead, I-element
constantly appears in the second half of documents. Intervention is the main
topic in medical articles, so that all parts (background/purpose, method,
results, and conclusion) in documents are likely to involve I-element. Another
distinct characteristic of I-element is that it is mostly short keywords,
while P-element tends to be phrases. Observe that the difference in lengths of
labeled words in P- and I-elements. Lastly, O-element often appears at the end
of documents where results and conclusion are. More generally, O-element is
mainly found in the second half of documents.

This analysis shows that each PICO element presents its own distributional
pattern in documents and it is similar to the findings reported in (Boudin et
al., 2010b). Hence, terms in same PICO elements are likely to be in similar
parts (_e.g.,_ beginning or end) of documents. We therefore design M-Match to
be position-aware by matching terms at similar positions. In that way, we
attempt to match terms that are likely to be used for same PICO elements. In
the next section, we explain details of M-Match.

###  3.3. Mirror Matching: Two-Way Matching

Figure 2. Mirror Matching uses two one-way interactions: from a query document
$Q$ to a candidate document $D$ and from a candidate document $D$ to a query
document $Q$. Term-term matchings are based on positional information and
among them a maximum matching value is taken.

Figure 2 visualizes Mirror Matching, consisting of two one-way matchings. Each
one-way matching is position-based matching between terms in a query document
and terms in a candidate document.

We detail Mirror Matching in Algorithm 1. Let $Q$ and $D$ denote a query
document and a candidate document, respectively. A query document $Q$ consists
of a sequence of query terms, $\langle q_{1},q_{2},\cdots,q_{|Q|}\rangle$, and
a candidate document $D$ consists of a sequence of document terms $\langle
t_{1},t_{2},\cdots,t_{|D|}\rangle$. The subscription of each term indicates
its position in a given document. In Mirror Matching, a matching score between
$Q$ and $D$, $\text{M-Match}(Q,D)$, is the sum of two one-way scores:
$sc_{Q\shortrightarrow D}$ and $sc_{D\shortrightarrow Q}$, as presented in
Equation 1. The concept behind the two one-way scores is identical. Hence, we
only explain the computation of $sc_{Q\shortrightarrow D}$ as a
representative.

Algorithm 1 Mirror Matching Overview

1:Query document $Q=\langle q_{1},q_{2},\cdots,q_{\left|Q\right|}\rangle$,
Candidate document $D=\langle t_{1},t_{2},\cdots,t_{\left|D\right|}\rangle$

2:M-Match$\left(Q,D\right)$ score between $Q$ and $D$,

3:

4:initialize $sc_{Q\shortrightarrow D}$ and $sc_{D\shortrightarrow Q}$ to zero

5:for $i\leftarrow 1$ to $\left|Q\right|$ do$\triangleright$ start one-way
matching from $Q$ to $D$

6: initialize $S_{temp}$ an empty list/array

7: for each $j\in$
matching_position$\left(\lambda,i,\left|Q\right|,\left|D\right|\right)$ do

8: $S_{temp}.append\left(sim\left(q_{i},t_{j}\right)\right)$

9: end for

10: $sc_{Q\shortrightarrow D}=sc_{Q\shortrightarrow
D}+max\left(S_{temp}\right)$

11:end for

12:$sc_{Q\shortrightarrow D}=sc_{Q\shortrightarrow D}/\left|Q\right|$

13:for $j\leftarrow 1$ to $\left|D\right|$ do$\triangleright$ start one-way
matching from $D$ to $Q$

14: initialize $S_{temp}$ an empty list/array

15: for each $i\in$
matching_position$\left(\lambda,j,\left|D\right|,\left|Q\right|\right)$ do

16: $S_{temp}.append\left(sim\left(q_{i},t_{j}\right)\right)$

17: end for

18: $sc_{D\shortrightarrow Q}=sc_{D\shortrightarrow
Q}+max\left(S_{temp}\right)$

19:end for

20:$sc_{D\shortrightarrow Q}=sc_{D\shortrightarrow Q}/\left|D\right|$

21:$M-Match(Q,D)=sc_{Q\shortrightarrow D}+sc_{D\shortrightarrow
Q}$$\triangleright$ final matching score

22:return $M-Match(Q,D)$

The score $sc_{Q\shortrightarrow D}$ is the average of scores computed on each
query term. For each term $q_{i}$, a score is computed using its positional
information by finding a term in $D$ which has a similar position in $D$ as
the position of $q_{i}$ in $Q$ and is also the best semantic matching to
$q_{i}$. Therefore, we first find terms in $D$ that are in similar relative
positions to given $q_{i}$, and then identify the best matching term among
them.

More specifically, we find terms in $D$ that are in similar positions to
$q_{i}$ using a window size. The position of $q_{i}$ is transformed into a
relative position in $D$, which is $|D|\cdot i/|Q|$. Note that
$i/|Q|\in(0,1]$. We then set a range of positions using a window size. As
document lengths vary, we define a window size proportional to the length of a
given document using a hyperparameter $\lambda$. For example, if
$\lambda=0.3$, then $30\%$ of the document length is set as a window size.
Hence, the range for similar positions in $D$ given $q_{i}$ is calculated as
$(|D|\cdot i/|Q|)\pm\lambda\cdot|D|$. Different signs denote preceding and
following directions from a relative position. The range includes the
beginning and ending numbers of the interval. In Algorithm 1,
matching_position denotes the procedure of finding matching terms that are in
similar positions. Next, each candidate term in the positional range is
matched to a query term $q_{i}$. A matching between two terms is a cosine
similarity of their word embeddings. Among the cosine similarity values, a
maximum value is chosen ($1$-max pooling) to exploit the most informative
matching signal. Word embeddings allow semantic matching, not limited to
surface forms of terms.

In summary, given $q_{i}\in Q$ we find the best matching term $t_{j}\in D$
within the matching position in $D$ and the term-term matching score between
$q_{i}$ and $t_{j}$ is computed by cosine similarity in word embedding space.
In this sense, our model is able to consider both PICO elements matching and
semantic matching. Finally $sc_{Q\shortrightarrow D}$ is the average of scores
obtained from each query term $q_{i}\in Q$.

The computational cost of $sc_{Q\shortrightarrow D}$ is governed by the number
of terms in $Q$, the number of terms in $D$, and $\lambda$ that determines a
window size. The estimation of $sc_{Q\shortrightarrow D}$ has the time
complexity of $O(nk)$ where $n$ is the number of terms in $Q$ (_i.e.,_ $|Q|$)
and $k$ is the number of terms within matching positions of $D$.

Likewise, $sc_{D\shortrightarrow Q}$ is the average of scores computed on each
term in $D$. It is worth noting that the two one-way scores,
$sc_{Q\shortrightarrow D}$ and $sc_{D\shortrightarrow Q}$, are asymmetric as
they are computed with respect to the terms in query and candidate documents,
respectively. In our evaluation, we set $\lambda$ to $0.35$.

###  3.4. Discussion

We discuss the connections and differences of M-Match with ad hoc retrieval
models and document similarity measures.

Relationship with ad hoc retrieval models. In ad hoc retrieval, typically a
query is short and a document is long. Since a given query is a primary
information source for relevance, retrieval models (_e.g.,_ BM25) estimate
document scores computed on each term in query (one-way scoring) (Zhai and
Massung, 2016). In our problem setting, a query is a (known relevant)
document, same as a candidate document. Besides, if a candidate document
happens to be a relevant document, it can be a query document as well. Hence,
our M-Match model employs a two-way scoring which demonstrates both of them
can be a query document. The connection to retrieval models is that our model
maintains the asymmetry in the one-way scorings, similar to that in retrieval
models. We discuss the purpose in comparison with document similarity
measures.

Relationship with document similarity measures. Common document similarity
measures (_e.g.,_ cosine similarity between two document representations) are
symmetric. Given two documents $d_{1}$ and $d_{2}$, $similarity(d_{1},d_{2})$
is the same as $similarity(d_{2},d_{1})$. Unlike symmetric document similarity
measures, our proposed M-Match employs two asymmetric scores. We design
asymmetry in the scores in order to find the most similar documents, from both
query and candidate documents’ perspective. We use an example to further
illustrate this point.

Let us assume that $d_{1}$ and $d_{2}$ are similar, but $d_{2}$ includes more
detailed content while fully covering all content of $d_{1}$. Given the two
documents, existing document similarity measures are likely to estimate a high
similarity value since they are similar as a whole. In the context of document
matching, when finding documents matching to $d_{1}$, $d_{2}$ can be a good
document for $d_{1}$ because all content of $d_{1}$ is fully covered in
$d_{2}$. However, when finding documents matching to $d_{2}$, $d_{1}$ may not
be a good document for $d_{2}$, as $d_{1}$ lacks the detailed content in
$d_{2}$. The asymmetric scores in M-Match give a lower score in this case,
especially in the matching score from $d_{2}$ to $d_{1}$.

##  4\. Evaluation Settings

###  4.1. SR Dataset

We conduct our evaluation on CLEF2019 dataset. This dataset contains 79
diagnostic test accuracy (DTA) SRs, 20 Intervention, 2 Qualitative, and 1
Prognosis SRs in CLEF eHealth 2019 Technologically Assisted Reviews (TAR)
(Kanoulas et al., 2019). CLEF2019 is currently the most comprehensive dataset
for screening prioritization. We use 79 DTA SRs in the evaluation. DTA SRs are
considered as a difficult type of SRs to automatically identify relevant
documents because of their broad range of topics. The other types of SRs in
CLEF2019 are much fewer than DTA SRs, and are not included222Note that for DTA
SRs, the original CLEF2019 consists of 80 SRs, which are 72 SRs from CLEF2018
and 8 new SRs. One new SR (id: CD011686) is already in CLEF2018 and it has
different relevance labels in CLEF2018 and in CLEF2019. We use the labels from
CLEF2018.. Training/testing data splits are defined in CLEF2019 but we use all
SRs since our model does not need training.

CLEF2019 provides document identifiers (PMIDs) of candidate documents with
their two relevance labels (from abstract screening and full document
screening). We use the label from full document screening in the evaluation
because these labels indicate the final relevant and non-relevant documents in
SRs. From a public MEDLINE 2019 corpus, we obtain document contents (title,
abstract, and MeSH keywords) using document identifiers. For the detailed
statistics of the dataset, we refer to (Kanoulas et al., 2019).

Any relevant document can be the known document in a SR. We use every relevant
document as a query and rank the rest candidate documents. In CLEF2019, a few
SRs have relevant documents fewer than 2. In this setting, as one relevant
document is used as a query, they are not included in the evaluation.

Pre-processing. In medical literature, short forms (_e.g.,_ abbreviations or
acronyms) of long medical terminologies are often used for effective
communication. However, there is no standard rule for constructing short
forms. Hence, a long medical term is written in different short forms across
different documents. To increase consistency within and between documents, we
identify short forms and their long forms using a simple regular expression,
and transform the short forms into the long form. We also detect three kinds
of numeric expressions and normalize them into INT, FLOAT, and PERCENT.
Lastly, stop words are removed.

Word Embeddings. We use word embeddings locally trained on candidate documents
per SRs. As candidate documents are retrieved by Boolean keyword queries
derived from relevance conditions, they are a collection of similar documents.
Multiple studies report that large corpora do not necessarily produce good
word embeddings in biomedical domain, compared to a smaller but topic-specific
collection (Chiu et al., 2016; Zhu et al., 2017). A similar finding has also
been reported in a general domain problem (Diaz et al., 2016). Our empirical
results also show that local word embeddings are more effective than global
word embeddings trained on a corpus (we report details in Section 5.1.4). We
use skip-gram with negative sampling implemented in gensim. The dimension of
word embeddings and a window size are set to 300 and 7, respectively and the
minimum word occurrence is set to 5. The default settings are used unless it
is specified.

###  4.2. Baselines

The evaluation is done with ad hoc retrieval models and document similarity
measures. Candidate documents are prioritized by retrieval scores or
similarity scores.

Retrieval models. Two classical retrieval models (BM25 and query likelihood
model) and a retrieval model specifically designed for screening
prioritization (SDR-BOC) are evaluated. BM25 is a simple and strong retrieval
model and it has continuously proved its effectiveness (Yang et al., 2019). We
use BM25 implementation in gensim. Query likelihood model (QL) is a language
model approach and we experiment QL with JM smoothing and a smoothing
parameter is empirically set to 0.2. SDR-BOC is a retrieval model proposed for
screening prioritization (Lee and Sun, 2018). It is QL combined with an
additional term weighting function. As suggested in the original paper, we
represent documents in bag-of-concepts (BOC) after extracting medical
concepts.

Document similarity measures. We examine document similarities using lexical
and semantic matching signals. The first measure is a cosine similarity with
different document representations: TF-IDF, AvgEm-W, AvgEm-C, and ParaVec. TF-
IDF is a bag-of-words document representation using the classic tf-idf
weighting scheme. AvgEm-W is the average of word embeddings of all words
contained in a document. Likewise, AvgEm-C is the average of concept
embeddings of all concepts contained in a document. A concept embedding is the
average of word embeddings of all words contained in a medical concept.
Paragraph vector (ParaVec) is a distributional document representation
proposed by (Le and Mikolov, 2014). We use PV-DBOW in gensim. In addition, we
tested different similarity functions such as Euclidean distance in pilot
experiments, but cosine similarity showed the highest performance.

We also evaluate document similarity measures proposed by existing studies.
TF-inner (Lin, 2009) computes an inner product of two bag-of-words document
representations using tf weighting. OK (Whissell and Clarke, 2013) is
motivated by BM25 and computes a document similarity using BM25 weights. Word
mover’s distance (WMD) (Kusner et al., 2015) estimates a distance between two
documents defined as a cost when all words in one document move to words in
the other document. WMD is a symmetric distance metric and its negative value
is considered as a similarity. (Guo et al., 2016) showed that WMD can be
applied to ad hoc retrieval.

AvgEm-W, AvgEm-C, and WMD use the same word embeddings as M-Match. In SDR-BOC
and AvgEm-C, QuickUMLS (Soldaini and Goharian, 2016) is used to extract
medical concepts.

###  4.3. Evaluation Metrics

We use evaluation metrics commonly adopted in screening prioritization task:
average precision, precision at $k$, recall at $k$, and work saved over
sampling at recall 100%. Average precision (AP) is a metric combining a recall
component to precision-focused metric by averaging all values of precision at
each relevant document. Precision at $k$ (Pr@$k$) is the ratio of relevant
documents out of top $k$ documents, and recall at $k$ (Re@$k$) is the ratio of
relevant documents in top $k$ documents out of the total number of relevant
documents $k$ is set to 10, 20, and 30. Work saved over sampling at recall
100% (WSS100) was introduced by (Cohen et al., 2006) and it measures the ratio
of documents that are ranked lower than a last relevant document out of the
total candidate documents. WSS100 estimates the workload to be reduced in the
manual screening with the assumption that after a last relevant document is
found the rest unlabeled documents are not screened. Finally, each SR is
evaluated by these evaluation metrics and then the mean over all SRs is
reported.

##  5\. Results and Discussion

In this section, we aim to answer three research questions.

RQ1. Is M-Match more effective than retrieval models and/or document
similarity measures for screening prioritization using a query document?

RQ2. To what extent does each component (_e.g.,_ two-way matching, position-
aware matching) in M-Match contribute to its performance?

RQ3. Regarding recent success of neural IR models in matching two texts, how
do neural models perform in this task?

Table 1. Comparison between Mirror Matching and the baselines. The best model is in bold and the second best is underlined in each metric. ∗ denotes statistically significant performance of the best model compared to a given model (p-value $<0.01$). Models | AP | Pr@10 | Pr@20 | Pr@30 | Re@10 | Re@20 | Re@30 | WSS100
---|---|---|---|---|---|---|---|---
BM25 | 0.169 | 0.209 | 0.173 | 0.151 | 0.178 | 0.246 | 0.299 | 0.437∗
QL | 0.155 | 0.189 | 0.158 | 0.139 | 0.170 | 0.235 | 0.279 | 0.466∗
SDR-BOC | 0.172 | 0.209 | 0.174 | 0.152 | 0.176 | 0.251 | 0.303 | 0.541∗
TF-IDF | 0.180 | 0.208 | 0.173 | 0.153 | 0.127 | 0.192 | 0.243 | 0.690
AvgEm-W | 0.176 | 0.201 | 0.167 | 0.147 | 0.177 | 0.242 | 0.290 | 0.530∗
AvgEm-C | 0.167 | 0.190 | 0.162 | 0.145 | 0.110∗ | 0.175 | 0.228 | 0.692
ParaVec | 0.139∗ | 0.164∗ | 0.141∗ | 0.126∗ | 0.093∗ | 0.153∗ | 0.198∗ | 0.598
TF-inner | 0.119∗ | 0.136∗ | 0.115∗ | 0.102∗ | 0.112 | 0.172 | 0.214 | 0.477∗
OK | 0.113∗ | 0.113∗ | 0.107∗ | 0.100∗ | 0.066∗ | 0.125∗ | 0.173∗ | 0.650
WMD | 0.203 | 0.232 | 0.194 | 0.170 | 0.141 | 0.225 | 0.272 | 0.592
M-Match | 0.220 | 0.247 | 0.207 | 0.183 | 0.149 | 0.234 | 0.293 | 0.687
Rank ($\Delta$) | 1 (+0.017) | 1 (+0.015) | 1 (+0.013) | 1 (+0.013) | 5 (-0.029) | 5 (-0.017) | 3 (-0.010) | 3 (-0.005)

###  5.1. Overall Results

First, to answer RQ1 we present the overall results of the proposed model and
the baselines in Table 1. We also report the rank of the proposed model among
all models and a performance gap between the proposed model and the best
baseline model in the last row. A positive/negative sign in the performance
gap indicates win/loss of the proposed model.

M-Match outperforms all baseline models in terms of AP and Pr@$k$. Two-way
matching scores effectively rank relevant documents higher than non-relevant
documents. WMD is the second best model in AP and Pr@$k$. While both M-Match
and WMD use semantic term matching, a key difference is that M-Match matches
terms using positional information to match terms with same PICO elements. On
the other hand, WMD matches terms by minimizing the cases of matching same
terms multiple times regardless positions and it also leads to higher
computation cost than M-Match.

Two retrieval models, BM25 and SDR-BOC, show the highest Re@$k$ values and
AvgEm-C has the highest WSS100 value. Indicated in the last row of Table 1,
M-Match follows behind these models with a small gap. The high performance of
SDR-BOC and AvgEm-C supports that bag-of-medical concepts (BOC) is an
effective representation for medical documents (Koopman et al., 2012; Lee and
Sun, 2018; Gehrmann et al., 2018). Bag-of-words (BOW) and bag-of-concepts
(BOC) represent documents with their own advantages and disadvantages. BOW
takes full advantage of all words in a document, despite sparsity. On the
other hand, BOC relies on essential medical terminologies with a risk of
losing contextual information in non-medical words.

We claim that the strong AP and Pr@$k$ of M-Match are particularly useful in
the beginning of manual screening process. The higher AP and Pr@$k$ values
represent the more relevant documents to be found when SR researchers start
screening a handful of top-ranked (_e.g.,_ top 10) documents. When more
relevant documents are identified early, they can further improve the rankings
of remaining unlabeled documents.

####  5.1.1. Ablation Study

Table 2. AP of degraded models without key components of Mirror Matching. Model | AP | loss
---|---|---
M-Match w/o position | $0.216$ | $-0.004$
M-Match w/o two-way | $0.204$ | $-0.016$
M-Match w/o {two-way, position} | $0.196$ | $-0.024$
M-Match | $0.220$ | -

We investigate the effect of major components in the proposed model: two-way
matching mechanism and position-aware matching. Three degraded models are
examined in an ablation test.

M-Match w/o position removes positional information in finding the maximum
term matching similarity. In other words, a max-pooling is applied to the
entire row/column in Figure 2. M-Match w/o two-way removes two-way matching
mechanism. A final matching score is $sc_{Q\shortrightarrow D}$, without
$sc_{D\shortrightarrow Q}$. M-Match w/o {two-way, position} removes both
components in Mirror Matching. A matching score is $sc_{Q\shortrightarrow D}$
and also it does not use positional information in max pooling.

Table 2 presents performances of the degraded models compared to M-Match. We
observe that: (1) M-Match w/o two-way performs worse than M-Match w/o
position, showing that incorporating two one-way scores has the bigger impact
than position-aware matching. (2) Performance of M-Match w/o {two-way,
position} has a significant drop from M-Match. AP of M-Match w/o {two-way,
position} is even lower than that of WMD reported in Table 1. These two
observations suggest that both components contribute to the proposed model. In
the next section, we further study the two-way matching mechanism and
investigate its major contribution to M-Match.

####  5.1.2. Analysis on Two-Way Matching

The key claim of two-way matching is to employ a score from candidate
document’s perspective ($sc_{D\shortrightarrow Q}$) as well as the commonly
used score from query’s perspective ($sc_{Q\shortrightarrow D}$). We study how
$sc_{D\shortrightarrow Q}$ make a change in matching scores of relevant
documents and eventually improves the rankings.

First we find relevant and non-relevant document pairs that
$sc_{Q\shortrightarrow D}$ of a non-relevant document has the smallest
difference to $sc_{Q\shortrightarrow D}$ of a relevant document. That is, the
two documents in pairs are ranked next to each other without two-way matching
(_i.e.,_ $sc_{Q\shortrightarrow D}$ alone is a matching score). We then
compare their $sc_{D\shortrightarrow Q}$. We calculate the portion of pairs
such that $sc_{D\shortrightarrow Q}$ of the relevant document is greater than
that of the non-relevant document within ranked candidate documents by a query
document. We then report the average portion over all queries from all SRs in
CLEF2019.333We also calculate the average portion via two levels: average over
a SR and then average over all SRs. The resulting portion is consistent. As a
result, 63.5% of the pairs have greater $sc_{D\shortrightarrow Q}$ of the
relevant document than $sc_{D\shortrightarrow Q}$ of the non-relevant
document. The result suggests that scores from candidate document’s
perspective $sc_{D\shortrightarrow Q}$ tend to promote relevant documents.

We note that the above analysis does not confirm the rise of relevant
documents in a ranked document list as the rest of candidate documents are
also involved in the total ranking. Nevertheless, this close analysis helps
better understand the two-way matching in finding relevant documents.

####  5.1.3. Impact of Window Size in Position-aware Matching

Figure 3. Impact of different ranges of matching positions. A range is twice
of a window size, which is determined by $\lambda$.

Position-aware matching in two one-way matchings helps match terms that are
used as the same PICO element. Positional information of terms is incorporated
with a window size determined by $\lambda$. Here we study the effect of
different window sizes. We experiment different values of $\lambda$ from
$0.05$ to $0.35$ with a $0.05$ increment. We plot AP of M-Match in Figure 3.
Recall that a window size indicates one side range (either preceding or
following) from a given term. When $\lambda=0.35$, for a term in the middle of
document, it has a matching area of 70% of a document length. Additionally, we
display AP with no positional information _i.e.,_ M-Match w/o position.

In Figure 3, AP gradually increases along the increase of window size. The
highest performance is observed when $\lambda$ is 0.35. It is worth noting
that when the model pinpoints terms with a very small window ($\lambda=0.05$),
the performance deteriorates significantly. We believe the large drop
originates from the different granularity of the rhetorical structure in
documents. For example, some abstracts have much longer results part than
others. Lastly, while using positional information of terms improves
performance, the performance gain is relatively small compared to the case
without using it. Referring to Figure 1, the positional patterns of PICO
elements can be seen in a collective manner among several thousands documents
and the pattern in a pair of two documents may not be strong enough. Besides,
using positional information is a soft way of incorporating PICO elements in
matching as the PICO spans are not explicitly pre-detected (Brockmeier et al.,
2019).

####  5.1.4. Impact of Word Embeddings

Figure 4. Impact of different word embeddings on cosine similarity with
AvgEm-W, WMD, and M-Match. (REDRAW A FIGURE WITH CHANGED MODEL NAMES (AvgEm-W)

The quality of word embeddings affects the performance of Mirror Matching,
WMD, as well as cosine similarity over AvgEmb-W and AvgEmb-C. We now examine
the impact of different word embeddings on AvgEmb-W, WMD, and the proposed
model. We evaluate two word embeddings pre-trained on large corpora: medical
domain word2vec444 http://bio.nlplab.org/ (Global word2vec) and
GloVe555https://nlp.stanford.edu/projects/glove/ and also present the earlier
evaluation with local word embeddings trained on candidate documents (Local
word2vec). As shown in Figure 4, in comparison of different word embeddings,
locally trained word embeddings consistently show the best performance in all
three models compared to the other two embeddings. Furthermore, M-Match
consistently outperforms the two baselines in all different kinds of word
embeddings.

We also experiment contextualized word embeddings using pre-trained BioBERT
(version 1.0 PubMed PMC) (Lee et al., 2019). However, while M-Match
outperforms the other models, the overall performance in all three models is
much lower than distributional word representations (_e.g.,_ word2vec, GloVe).
We presume that this is related to the vocabulary coverage of BioBERT with
respect to our dataset. Moreover, to the best of our knowledge, it is unknown
how to properly adopt BERT in a word-level intrinsic evaluation setting,
instead of extrinsic evaluation settings (_i.e.,_ downstream tasks). We leave
this as our future work.

###  5.2. Ranking Updates with Multiple Documents

Next, we simulate the real-world scenario when the proposed model is used in
the manual screening process. We assume the manual screening is conducted on
prioritized documents (_i.e.,_ evaluated in Table 1). When top 20 documents
are screened, the rankings of the remaining unlabeled documents are updated
with new relevant documents found in the top 20 documents. We repeat this
three times.

As the proposed model computes document-to-document matching scores, each of
newly found relevant documents produces a new ranked list. Multiple ranked
lists also have different ranges of ranking scores, so we combine them using
rankings. For example, for an unlabeled document, we get its rankings in
multiple ranked lists and then the averaged ranking is set to its updated
ranking. Unlabeled documents are ordered by updated rankings in ascending
order. The rank-wise combination allows multiple ranked lists carry equal
weights in the updated ranked list.

We evaluate an updated list by AP and WSS100 and they evaluate different
aspects of the list (details in Section 4.3. We compare the proposed model
against TF-IDF and AvgEm-C which showed the higher WSS100 values than the
proposed model in Table 1.

Table 3. Simulating the manual screening process. A ranked list of unlabeled documents is updated when every top 20 documents are labeled, which is defined as one round. | AP | WSS100
---|---|---
round | 1 | 2 | 3 | 1 | 2 | 3
TF-IDF | 0.213 | 0.241 | 0.252 | 0.745 | 0.767 | 0.777
AvgEm-C | 0.178 | 0.204 | 0.239 | 0.704 | 0.721 | 0.731
M-Match | 0.259 | 0.262 | 0.279 | 0.719 | 0.728 | 0.732

Table 3 presents the evaluation results of the simulation in three rounds. As
more relevant documents are used as a query document, all three models
continue to improve the rankings for the unlabeled documents. M-Match shows
the highest AP in all three rounds and it is consistent with the high AP in
the initial ranking results shown in Table 1. On the other hand, the proposed
model has the second best WSS100, following TF-IDF. The proposed model is not
the most effective one to find a last relevant document.

Limitations. We believe the weak WSS100 in M-Match is closely related to topic
specificity/broadness of SRs. When a SR has broad topic, its relevant
documents may share relatively lower similarities (_e.g.,_ relevant documents
cover different parts of broad relevance conditions). In this case, it is hard
for the proposed model to effectively find all relevant documents (_i.e.,_
including a last relevant document). To examine the relation with topic
specificity of SRs, we conduct a small analysis.

We assume that when the proposed model monotonically decreases WSS values over
the three rounds for a SR, the proposed model does not work well for that SR
to find all relevant documents. To estimate topic specificity, we use the
minimum value in pairwise similarities between relevant documents in a SR. If
a SR has specific topic, all relevant document pairs can have a high
similarity and the minimum value is high as well. Hence, the minimum value can
reflect topic specificity of SR.

The analysis shows that the estimated topic specificity for SRs with
increasing WSS values is 0.498. In contrast, the estimated topic specificity
for SRs with decreasing WSS values is only 0.143. This result describes that
SRs in which the proposed model is less effective tend to have broad topic.

Suggestions. We have discussed the strength and the weakness of M-Match: it is
effective to find relevant documents at the top, but less effective to find a
last relevant document (_i.e.,_ a document less similar to a query document).
Therefore, adopting M-Match in the beginning of the manual screening is
particularly useful. It allows to identify more relevant documents after the
small number of documents are screened. Then, we suggest to combine/switch
M-Match to another ranking model (_e.g.,_ TF-IDF) for the better ranking of a
last relevant document. A supervised approach can also be used since several
relevant and non-relevant documents are available.

###  5.3. Neural Models in Screening Prioritization

We have extensively evaluated M-Match with RQ1 and RQ2. To provide a complete
study, we test a few neural IR models and answer the last research question
RQ3. Similar to M-Match, neural models also find a pair of texts that match
each other, but we first need to train them.

####  5.3.1. Train and Test Settings.

Neural IR models, a supervised approach, typically require large amounts of
training data. It is difficult to train neural models specialized for
individual SRs because of too few relevant documents in a SR. We use train and
test splits defined in CLEF2019, consisting of 72 and 7 SRs, respectively666A
development dataset is not defined in CLEF2019. For a development (dev)
dataset, we randomly sample 7 SRs in the training 72 SRs. Because of the
random sampling, we examine two sets of train and dev datasets with different
SRs. Both evaluation results are reported.

We generate train/dev instances (document pairs) using relevant and under-
sampled non-relevant documents within each SR: {relevant - relevant} pairs for
positive instances and {relevant - non-relevant} for negative instances. In
total, the train and dev datasets contain 145,104 instances. In the test
dataset, a query document is paired with every candidate document.

We use MatchZoo (version 2.1.0) toolkit (Guo et al., 2019a) in this
experiment. MatchZoo toolkit provides a collection of representative neural IR
models and it has been gaining attention from research communities as a solid
experimental platform. We evaluate two simple neural baseline models (Dense,
BiLSTM) and another two neural IR models (Duet, MV-LSTM) which share similar
matching (learning) mechanisms with Mirror Matching.

  * •

Dense is a naive baseline. Word tokens are represented as one-hot vectors and
they are concatenated and encoded using two layers of feedforward neural
networks (FNNs).

  * •

BiLSTM is bidirectional LSTMs (BiLSTM) in a siamese architecture. After
embedding lookup and encoding by BiLSTM, the last hidden states from documents
in a pair are concatenated. Then, two layers of FNNs are added.

  * •

Duet (Mitra et al., 2017) uses semantic matching as well as syntactic matching
in character $n$-graph interactions between two documents.

  * •

MV-LSTM (Wan et al., 2016) is based on a learning goal similar to M-Match. It
uses term-term semantic interactions encoded by BiLSTM with positional
information.

MatchZoo provides the implementations of Dense, Duet, and MV-LSTM. We refer to
MatchZoo documentations for their implementation details and settings. We
implement BiLSTM using MatchZoo platform. We use a softmax activation function
and cross entropy as the loss function. All models truncate a document to its
first 300 tokens. We set epoch to 30 and select the best model on the dev
dataset in terms of AP and report its performance on the test dataset. When
training the neural models we use global word embeddings4 instead of local
embeddings. Note that our local embeddings trained for each SR do not share
embedding spaces.

Figure 5. Performance of neural models on test dataset in CLEF2019. Two sets
of AP are presented when the train and dev datasets are comprised of different
SRs.

Results and Discussion. Figure 5 presents two sets of AP when the train and
dev datasets are comprised of different SRs. We also show the performance of
M-Match on the test dataset for reference. Here, M-Match uses the same global
word embeddings as the neural models do. Among all four models, MV-LSTM
achieves the highest performance in both evaluation sets, while it is still
lower than that of Mirror Matching.

One concern we anticipate is that neural models in ad hoc retrieval are often
adopted as a re-ranker. First, documents are ranked by a simple model like
BM25 and a neural model then re-ranks the top 1,000 documents, for example. We
claim that the setting in screening prioritization in fact similar to the re-
ranking setup. Candidate documents in SRs are filtered documents from digital
libraries by Boolean retrieval.

It is worth noting that the randomly selected development dataset
significantly influences the performance on the test set. One development
dataset consistently leads to the higher performance than the other dataset in
all models. Moreover, the performance gap between the two development datasets
is not negligible.

We believe that different relevance conditions in SRs (or uniqueness of SRs)
are a major contributing factor to the performance difference when the train
and dev datasets have different splits of SRs. Relevant documents within one
SR can have higher (in narrow scope of relevance conditions) or lower (in
broad scope) inter-similarities than other SRs. The train/dev instances
represent different characteristics when they are comprised of different SRs.
Hence, the patterns learned and validated by one set of train and dev datasets
can differ from those by the other set of train and dev datasets. Furthermore,
they can be different from patterns required for correct predictions on test
SRs.

Another contributing factor can be the size of dataset. Due to the nature of
deep neural models, a large amount of data is prerequisite for high
performance. It is possible that CLEF2019 dataset might not be sufficient
enough to encode all patterns in SRs. More data might mitigate this issue, but
it requires further investigations.

Summary. We examine neural IR models for screening prioritization. The
empirical results show large performance changes by different data splits. We
discuss a few factors that might be related. To this end, it is hard to draw a
conclusion on performance of neural models and how to obtain reliable results
remains an open question.

##  6\. Conclusion

We investigate screening prioritization using a query document in the context
of document matching. Since the query document is a known relevant document
and used for finding unknown relevant documents, we introduce two-way document
matching named Mirror Matching. Our proposed model aims to find out how
similar two documents (_i.e.,_ a query document and a candidate document) are.
There are two important components in the model that make it different from
alternative approaches, and effective for this task: (1) Two one-way
(asymmetric) scores better model to find the most similar two documents.
Moreover, it demonstrates that a candidate document can be used as a query if
it is a relevant document. (2) Each of the two one-way scores is position-
based and it enables PICO element matching as well as semantic matching. This
design choice is motivated by our analysis on the distributional patterns of
PICO elements in medical literature. Extensive experimental results show the
effectiveness of our proposed model compared to unsupervised retrieval models,
document similarity measures, and neural models. For future work, we plan to
explore approaches for multiple known documents within the document matching
framework of screening prioritization.

## References

  * (1)
  * Bayardo et al. (2007) Roberto J. Bayardo, Yiming Ma, and Ramakrishnan Srikant. 2007\.  Scaling Up All Pairs Similarity Search. In _WWW_. 131–140.
  * Boudin et al. (2010a) Florian Boudin, Jian-Yun Nie, and Martin Dawes. 2010a.  Clinical information retrieval using document and PICO structure. In _NAACL_. 822–830.
  * Boudin et al. (2010b) Florian Boudin, Jian-Yun Nie, and Martin Dawes. 2010b.  Positional language models for clinical information retrieval. In _EMNLP_. 108–115.
  * Boudin et al. (2010c) Florian Boudin, Lixin Shi, and Jian-Yun Nie. 2010c.  Improving medical information retrieval with pico element detection. In _ECIR_. 50–61.
  * Brockmeier et al. (2019) Austin J Brockmeier, Meizhi Ju, Piotr Przybyła, and Sophia Ananiadou. 2019.  Improving reference prioritisation with PICO recognition.  _BMC Medical Informatics and Decision Making_ 19, 1 (2019), 256\.
  * Chiu et al. (2016) Billy Chiu, Gamal Crichton, Anna Korhonen, and Sampo Pyysalo. 2016.  How to train good word embeddings for biomedical NLP. In _Proceedings of the 15th Workshop on Biomedical Natural Language Processing_. 166–174.
  * Cohen et al. (2006) Aaron M Cohen, William R Hersh, K Peterson, and Po-Yin Yen. 2006\.  Reducing workload in systematic review preparation using automated citation classification.  _Journal of the American Medical Informatics Association_ 13, 2 (2006), 206–219.
  * Cormack and Grossman (2014) Gordon V Cormack and Maura R Grossman. 2014.  Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. In _SIGIR_. 153–162.
  * Craswell et al. (2018) Nick Craswell, W Bruce Croft, Maarten de Rijke, Jiafeng Guo, and Bhaskar Mitra. 2018.  Neural information retrieval: introduction to the special issue.  _Information Retrieval Journal_ 21, 2-3 (2018), 107–110.
  * Diaz et al. (2016) Fernando Diaz, Bhaskar Mitra, and Nick Craswell. 2016\.  Query Expansion with Locally-Trained Word Embeddings. In _ACL_ , Vol. 1. 367–377.
  * Elsayed et al. (2008) Tamer Elsayed, Jimmy J. Lin, and Douglas W. Oard. 2008\.  Pairwise Document Similarity in Large Collections with MapReduce. In _ACL_. 265–268.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Franck Dernoncourt, Yeran Li, Eric T Carlson, Joy T Wu, Jonathan Welt, John Foote Jr, Edward T Moseley, David W Grant, Patrick D Tyler, et al. 2018\.  Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives.  _PloS one_ 13, 2 (2018).
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016\.  Semantic matching by non-linear word transportation for information retrieval. In _CIKM_. 701–710.
  * Guo et al. (2019a) Jiafeng Guo, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2019a.  MatchZoo: A Learning, Practicing, and Developing System for Neural Text Matching. In _SIGIR_. 1297–1300.
  * Guo et al. (2019b) Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W Bruce Croft, and Xueqi Cheng. 2019b.  A deep look into neural ranking models for information retrieval.  _arXiv preprint arXiv:1903.06902_ (2019).
  * Gupta et al. (2015) Manish Gupta, Michael Bendersky, et al. 2015\.  Information retrieval with verbose queries.  _Foundations and Trends® in Information Retrieval_ 9, 3-4 (2015), 209–354.
  * Kanoulas et al. (2019) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019\.  CLEF 2019 Technology Assisted Reviews in Empirical Medicine Overview. In _Working Notes of CLEF 2019_.
  * Kim and Croft (2014) Youngho Kim and W. Bruce Croft. 2014.  Diversifying Query Suggestions Based on Query Documents. In _SIGIR_. 891–894.
  * Kim and Croft (2015) Youngho Kim and W. Bruce Croft. 2015.  Improving Patent Search by Search Result Diversification. In _ICTIR_. 201–210.
  * Koopman et al. (2012) Bevan Koopman, Guido Zuccon, Peter Bruza, Laurianne Sitbon, and Michael Lawley. 2012.  Graph-based Concept Weighting for Medical Information Retrieval. In _ADCS_. 80–87.
  * Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015.  From word embeddings to document distances. In _ICML_. 957–966.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _ICML_. 1188–1196.
  * Lee and Sun (2018) Grace E. Lee and Aixin Sun. 2018.  Seed-driven Document Ranking for Systematic Reviews in Evidence-Based Medicine. In _SIGIR_. 455–464.
  * Lee and Sun (2019) Grace E. Lee and Aixin Sun. 2019.  A Study on Agreement in PICO Span Annotations. In _SIGIR_. 1149–1152.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ (09 2019).
  * Lin (2009) Jimmy Lin. 2009\.  Brute Force and Indexed Approaches to Pairwise Document Similarity Comparisons with MapReduce. In _SIGIR_.
  * Mitra et al. (2017) Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017\.  Learning to match using local and distributed representations of text for web search. In _WWW_. 1291–1299.
  * Nye et al. (2018) Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron Wallace. 2018.  A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature. In _ACL_.
  * O’Mara-Eves et al. (2015) Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. 2015.  Using text mining for study identification in systematic reviews: a systematic review of current approaches.  _Systematic reviews_ 4, 1 (2015), 5.
  * Onal et al. (2018) Kezban Dilek Onal, Ye Zhang, Ismail Sengor Altingovde, Md Mustafizur Rahman, Pinar Karagoz, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quinten McNamara, et al. 2018\.  Neural information retrieval: At the end of the early years.  _Information Retrieval Journal_ 21, 2-3 (2018), 111–182.
  * Soldaini and Goharian (2016) Luca Soldaini and Nazli Goharian. 2016.  Quickumls: a fast, unsupervised approach for medical concept extraction. In _MedIR workshop, SIGIR_.
  * (US) and Institute (1997) National Information Standards Organization (US) and American National Standards Institute. 1997\.  _Guidelines for Abstracts: An American National Standard_. Vol. 14.  Niso Press.
  * Wallace et al. (2010) Byron C Wallace, Kevin Small, Carla E Brodley, and Thomas A Trikalinos. 2010.  Active learning for biomedical citation screening. In _KDD_. 173–182.
  * Wan et al. (2016) Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016\.  A deep architecture for semantic matching with multiple positional sentence representations. In _AAAI_.
  * Weng et al. (2011) Linkai Weng, Zhiwei Li, Rui Cai, Yaoxue Zhang, Yuezhi Zhou, Laurence T. Yang, and Lei Zhang. 2011.  Query by Document via a Decomposition-based Two-level Retrieval Approach. In _SIGIR_. 505–514.
  * Whissell and Clarke (2013) John S Whissell and Charles LA Clarke. 2013.  Effective measures for inter-document similarity. In _CIKM_. 1361–1370.
  * Yang et al. (2018) Eugene Yang, David D Lewis, Ophir Frieder, David A Grossman, and Roman Yurchak. 2018.  Retrieval and Richness when Querying by Document.. In _DESIRES_. 68–75.
  * Yang et al. (2019) Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019\.  Critically Examining the “Neural Hype”: Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models. In _SIGIR_. 1129–1132.
  * Yang et al. (2009) Yin Yang, Nilesh Bansal, Wisam Dakka, Panagiotis Ipeirotis, Nick Koudas, and Dimitris Papadias. 2009\.  Query by Document. In _WSDM_. 34–43.
  * Zhai and Massung (2016) ChengXiang Zhai and Sean Massung. 2016.  _Text data management and analysis: a practical introduction to information retrieval and text mining_.  Morgan & Claypool.
  * Zhu et al. (2017) Yongjun Zhu, Erjia Yan, and Fei Wang. 2017.  Semantic relatedness and similarity of biomedical terms: examining the effects of recency, size, and section of biomedical publications on the performance of word2vec.  _BMC medical informatics and decision making_ 17, 1 (2017), 95\.


Further author information:
S. A.: E-mail<EMAIL_ADDRESS>
# A cascaded deep network for automated tumor detection and segmentation in
clinical PET imaging of diffuse large B-cell lymphoma

Shadab Ahamed  Department of Physics & Astronomy, University of British
Columbia, Vancouver, BC, Canada  Department of Integrative Oncology, BC Cancer
Research Institute, Vancouver, BC, Canada  Natalia Dubljevic  Department of
Physics & Astronomy, University of British Columbia, Vancouver, BC, Canada
Ingrid Bloise  BC Cancer, Vancouver, BC, Canada  Claire Gowdy  BC Children’s
Hospital, Vancouver, BC, Canada  Patrick Martineau  BC Cancer, Vancouver, BC,
Canada  Don Wilson  BC Cancer, Vancouver, BC, Canada  Carlos F. Uribe
Department of Radiology, University of British Columbia, Vancouver, BC, Canada
BC Cancer, Vancouver, BC, Canada  Arman Rahmim  Department of Physics &
Astronomy, University of British Columbia, Vancouver, BC, Canada  Department
of Integrative Oncology, BC Cancer Research Institute, Vancouver, BC, Canada
Department of Radiology, University of British Columbia, Vancouver, BC, Canada
Fereshteh Yousefirizi  Department of Integrative Oncology, BC Cancer Research
Institute, Vancouver, BC, Canada

###### Abstract

Accurate detection and segmentation of diffuse large B-cell lymphoma (DLBCL)
from PET images has important implications for estimation of total metabolic
tumor volume, radiomics analysis, surgical intervention and radiotherapy.
Manual segmentation of tumors in whole-body PET images is time-consuming,
labor-intensive and operator-dependent. In this work, we develop and validate
a fast and efficient three-step cascaded deep learning model for automated
detection and segmentation of DLBCL tumors from PET images. As compared to a
single end-to-end network for segmentation of tumors in whole-body PET images,
our three-step model is more effective (improves 3D Dice score from 58.9% to
78.1%) since each of its specialized modules, namely the slice classifier, the
tumor detector and the tumor segmentor, can be trained independently to a high
degree of skill to carry out a specific task, rather than a single network
with suboptimal performance on overall segmentation.

###### keywords:

Diffuse large B-cell lymphoma, 18F-FDG PET, deep learning, classification,
detection, segmentation

##  1 INTRODUCTION

Diffuse large B-cell lymphoma (DLBCL) is the most common form of non-Hodgkin
lymphoma among adults [1]. Accurate detection and segmentation of DLBCL tumors
from 18F-FDG PET images has important implications for surgical intervention
and radiation therapy [2, 3]. Tumor segmentation is required to calculate the
total metabolic tumor volume (TMTV); a metric that has been shown to have
predictive value for patient outcome in lymphoma [4, 5, 6, 7, 8]. Manual
segmentation of tumors in whole-body PET images is time-consuming, labor
intensive and operator-dependent, with high intra- and inter-operator
variability, hence not performed routinely [9].

With the advent of more computational power and various breakthroughs in the
development of AI algorithms, supervised deep learning-based methods are being
increasingly used for localizing the regions of interest (ROIs) in PET images.
Unfortunately, these supervised methods require a large amount of accurately
annotated data to learn to perform significantly accurate predictions of
segmentation contours [10, 11]. Their performance can, in fact, be
particularly challenging, especially when the disease is generalized, or the
tumors are small and occur in close proximity to one another. Furthermore,
automated lesion detection/segmentation based on traditional machine learning
techniques have not found widespread use in clinics due to limitations in
performance, reproducibility, and explanability [12].

Our overarching aim is to enable routine detection and segmentation of all
tumors in PET images of DLBCL patients. In this study, we have implemented a
deep learning-based, fully-automated three-step network architecture that
takes a 3D PET image as input and outputs the segmentation contours for each
of the 2D axial slices containing tumors. This three-step model also improves
the Dice score between the ground truth and the predicted segmentation
contours on the DLBCL test set by about 19% as compared the Dice score
evaluated using a single end-to-end segmentation network (U-Net) applied on
the whole-body PET image.

Figure 1: The proposed cascaded 3-step deep network for PET tumor
segmentation. Module-1 (Slice classifier) is a ResNet152 based binary
classification network trained on all axial PET slices. Module-2 (Tumor
detector) is a Faster R-CNN based object detection network trained only on the
axial foreground PET slices. Module-3 (Tumor segmentor) is a 2D U-Net network
trained on ROIs extracted from the ground truths of Module-2.

##  2 METHODS

Object detection in medical images refers to identifying location of
tumors/organs (objects) via the prediction of bounding boxes around them and
classifying different objects. Lesion/tumor segmentation in medical images
involves the prediction of the boundaries of lesions/tumors, which is in an
essential step towards the calculation of TMTV. There are various approaches
to performing fully-automated supervised deep learning-based segmentation of
clinical PET images [13]. Some approaches employ a single-step method via a
single end-to-end segmentation network applied on whole-body images such as
U-Net [14]. Others employ a two-step approach, where the detection step is
performed along with the segmentation step within the detected bounding box,
either simultaneously [15, 16, 17, 18] or one after the other [19, 2, 20]. In
this section, we give a detailed description of our 3-step network
architecture and the lymphoma datasets used for training and validation of the
model.

###  2.1 Structure of the proposed cascaded deep network

Our proposed cascaded segmentation pipeline (Figure 1) consists of the
following three consecutively stacked modules:

  1. 1.

Module-1 (Slice classifier): This module classifies the axial slices of 3D PET
images into slices that contain tumors (foreground slices) and slices that do
not (background slices). This module is a binary classification network that
uses a ResNet152 [21] feature extractor (backbone) pre-trained on the ImageNet
dataset [22]. This backbone is followed by a fully connected layer ending with
a sigmoid activation function to facilitate binary classification. The axial
slices that contain tumors are given as input to Module-2.

  2. 2.

Module-2 (Tumor detector): This module localizes tumors on the axial slices
obtained from Module-1 by generating rectangular bounding boxes around the
suspicious regions. This module uses the object detection network Faster-RCNN
[23] with ResNet50 feature extractor and an associated feature pyramid network
[24], pre-trained on the ImageNet dataset. The coordinates of these bounding
boxes are given as input to Module-3.

  3. 3.

Module-3 (Tumor segmentor): This module segments the tumor inside the
rectangular bounding boxes obtained from Module-2. It uses a conventional 2D
U-Net architecture [14] modified for smaller images with a final sigmoid
activation layer for the pixel-wise classification into foreground and
background pixels.

In this work, we have employed a three-step model instead of directly
segmenting the tumors in 3D PET images. The motivation behind this are as
follows: (i) In the first step, we train the slice classifier network
(Module-1) to learn to reject background slices and accept only the foreground
slices, which are subsequently given as input to the tumor detector
(Module-2). Rejecting the background slices before the tumor detection step is
beneficial as it reduces the dependence of training the detection module on
the background slices that contain no tumors or suspicious ROIs. (ii)
Furthermore, we hypothesize that segmenting tumors inside the detected
bounding boxes improves the segmentation performance by centering the object
of interest (a DLBCL tumor) in the receptive field of view and cropping out
extraneous information such as normal active organs and other high uptake
regions that may degrade the performance.

###  2.2 Dataset

Our lymphoma PET images dataset consisted of 126 cases of primary mediastinal
large B-cell lymphoma (PMBCL) (for pre-training) and 50 cases of DLBCL. All
images were annotated by nuclear medicine physicians, following initial
discussions and development of consensus procedures [25]. We used a
60%:20%:20% splitting of these axial slices for training, validation and
testing, respectively. Class imbalance, i.e., proportion of slices with tumors
(foreground slices) and slices without tumors (background slices) for both
these datasets was about 10%:90%, across all training, validation and testing
sets. Across the three sets, we had a total of 31126 axial slices (2958
foreground and 28168 background) of PMBCL cases and 13185 axial slices (1217
foreground and 11968 background) of DLBCL cases. The axial slices of the PMBCL
cases were either 168$\times$168 pixels (pixel dimension: 4.06 mm$\times$4.06
mm) or 192$\times$192 pixels (pixel dimension: 3.64 mm$\times$3.64 mm) in
dimension, while the axial slices of the DLBCL cases were 192$\times$192
pixels (pixel dimension: 3.64 mm$\times$3.64 mm). All the axial slices were
resized to 224$\times$224 pixels and z-score normalized for training both the
classifier (Module-1) and the detector (Module-2) modules. The segmentator
module (Module-3) was pre-trained on ground truth PMBCL ROIs (bounding boxes)
and fine-tuned on DLBCL ROIs cropped from the foreground slices and resized to
128$\times$128 pixels. The detailed count for the images in the training,
validation and test set for these three tasks are given in Table 1.

Table 1: 3D PET axial slices splitting across training, validation and test sets for PMBCL and DLBCL cases for the three modules. Task | Dataset | Train | Valid | Test | Total
---|---|---|---|---|---
Classification | PMBCL (axial slices) | 18641 | 6384 | 6101 | 31126
DLBCL (axial slices) | 5786 | 4506 | 2893 | 13185
Detection | PMBCL (axial slices) | 1364 | 743 | 851 | 2958
DLBCL (axial slices) | 715 | 261 | 241 | 1217
Segmentation | PMBCL (ROIs) | 1701 | 1050 | 1306 | 4057
DLBCL (ROIs) | 1034 | 316 | 329 | 1679

###  2.3 Training the cascaded deep model

  1. 1.

Module-1 (Slice classifier): To train this module, we used the ImageNet-
trained ResNet152 backbone, and pre-trained and tested the fully-connected
layers using 31126 axial slices of 3D PET images from the PMBCL cases (for 150
epochs) and subsequently fine-tuned and tested the module using 13185 axial
slices from the DLBCL cases (for another 50 epochs). The minibatch size for
training was set to 64 axial images. The model weights were optimized using
Adam optimizer with learning rate $=10^{-5}$, $\beta_{1}=0.9$,
$\beta_{2}=0.999$, and $\epsilon=10^{-8}$. The classification loss function
$\mathcal{L}_{classification}$ used is given by,
$\mathcal{L}_{classification}=\mathcal{L}_{Focal}$, where
$\mathcal{L}_{Focal}$ is the focal loss [26] with parameters $\alpha=0.25$ and
$\gamma=2$. This loss function was chosen to increase the classification
accuracy in the presence of class imbalance between the foreground and the
background slices.

  2. 2.

Module-2 (Tumor detector): To train this module, we pre-trained and tested the
ImageNet-trained Faster R-CNN network using 2958 foreground axial slices of 3D
PET images from the PMBCL cases (for 100 epochs) and then fine-tuned and
tested the module using 1217 axial slices from the DLBCL cases (for another
100 epochs). The minibatch size for training was set to 16 axial images. The
Adam optimizer was used with the same hyperparameters as used for the
classifier module above (except the learning rate was set to $10^{-3}$). The
detection loss function $\mathcal{L}_{detection}$ used is given by,
$\mathcal{L}_{detection}=\mathcal{L}_{class}+\lambda\mathcal{L}_{box}$, where
$\mathcal{L}_{class}$ is the cross-entropy classification loss [27] for class
prediction (tumor), and $\mathcal{L}_{box}$ is the L1 regression loss [27] for
bounding box coordinates prediction. The weight hyperparameter $\lambda=10.0$
was used during training.

  3. 3.

Module-3 (Tumor segmentor): This module was pre-trained and tested on 4057
ground truth PMBCL ROIs (for 25 epochs) and fine-tuned and tested on 1679
DLBCL ROIs (for another 15 epochs) cropped from foreground slices. The
minibatch size for training was set to 32 axial images. The segmentation loss
function $\mathcal{L}_{segmentation}$ used is given by,
$\mathcal{L}_{segmentation}=\mathcal{L}_{generalizedDice}+\lambda\mathcal{L}_{Focal}$,
where $\mathcal{L}_{generalizedDice}$ is the generalized Dice loss [28] and
$\mathcal{L}_{Focal}$ is the focal loss (with $\alpha=0.25$, $\gamma=2$). The
weight hyperparameter $\lambda=10.0$ was used during training.

##  3 RESULTS

Figure 2: Performance of the slice classifier module on the DLBCL test set.
(a) Various classification metrics as a function of threshold. (b) The optimal
threshold = 0.16 was chosen using the maximum of the geometric mean of
specificity and sensitivity. (c) Receiver operating characteristic (ROC)
curve, with the area under curve (AUC) = 0.93. (d) Precision-recall (PR) with
average precision = 0.72.

  1. 1.

Module 1 (Slice classifier): After pre-training this module on the PMBCL
training dataset, a classification accuracy of 89% and ROC AUC score of 0.92
was obtained on the PMBCL test dataset. Upon fine-tuning the module on DLBCL
training set, the module obtained a classification accuracy of 90% and an ROC
AUC score of 0.93. For the DLBCL test dataset containing 2893 axial slices
(241 foreground and 2652 background slices), at the optimal threshold = 0.16
(the threshold at which the geometric mean of specificity and sensitivity is
maximum, see Figure 2 (b)), we obtained a specificity of 0.90, sensitivity of
0.83, precision of 0.43, and F1-score of 0.57 (Figure 2 (a)). The ROC curve
(AUC = 0.93) and the precision-recall curve (average precision = 0.72) for the
performance of the slice classifier on the DLBCL test dataset are shown in
Figure 2 (c)-(d).

  2. 2.

Module-2 (Tumor detector): After pre-training and testing this module on only
the 2958 foreground slices of the PMBCL test dataset, a tumor detection
accuracy of 80% and mean average precision (mAP) of 0.67 was obtained. Upon
subsequent fine-tuning and testing on the 1217 foreground slices of the DLBCL
dataset, the trained module obtained a tumor detection accuracy of 81% and mAP
of 0.69 (Table 2).

Table 2: Evaluation of the tumor detector module on the PMBCL and DLBCL test set. Test set | Detection accuracy (%) | Mean average precision (mAP)
---|---|---
PMBCL | 80% | 0.67
DLBCL | 81% | 0.69
  3. 3.

Module-3 (Tumor segmentor): After pre-training and testing on PMBCL ROIs
(cropped from foreground slices), the module achieved an average 2D Dice score
of 80.7% $\pm$ 16.0% on the detected axial slices in the PMBCL test set. Upon
aggregating the 2D predictions, a 3D prediction could be generated. An average
3D Dice score of 84.5% $\pm$ 6.7% was obtained on the PMBCL test set. After,
fine-tuning and testing on DLBCL ROIs, an average 2D Dice score of 77.9% $\pm$
13.2% was achieved on the DLBCL test dataset. Some examples of detection and
segmentation tasks on images from the DLBCL test set are shown in Figure 3. An
average 3D Dice score of 78.1% $\pm$ 8.6% was seen across patients in the
DLBCL test dataset. The 3D Dice score obtained using a standard end-to-end
(single-step) 3D U-Net architecture on the same DLBCL test set was 58.9% $\pm$
16.1%. This demonstrates that our three-step model performs better than a
single-step segmentation network by about 19% on the same DLBCL test set
(Table 3).

Table 3: Evaluation of segmentation performance by our proposed model and the 3D U-Net. Model | Test set | Dice score
---|---|---
Proposed model | PMBCL | 80.7% $\pm$ 16.0 % (2D)
84.5% $\pm$ 6.7% (3D)
DLBCL | 77.9% $\pm$ 13.2% (2D)
78.1% $\pm$ 8.6%(3D)
3D U-Net | DLBCL | 58.9% $\pm$ 16.1% (3D)

Figure 3: Performance of the tumor detector and segmentor modules: (left
column) (a)-(c) show 3 representative DLBCL PET images in the coronal view.
(middle column) (d)-(f) show the corresponding selected axial slices (shown as
white horizontal lines in (a)-(c)) with the predicted bounding boxes (ROIs)
around the tumors (shown in red). The average detection accuracy on the DLBCL
test set was 81% and average mAP was 0.69. (right column) (g)-(i) show the
corresponding ground truth (via physicians, shown in green) and predicted (via
our implemented 2D U-Net, shown in red) segmentation contours in 2D, with the
average 2D Dice score of 77.9% $\pm$ 13.2% across all the slices in the DLBCL
test set obtained as output of the detection module. The 2D segmentation
contours were aggregated to obtain an average 3D Dice score of 78.1% $\pm$
8.6% across all test patients.

##  4 CONCLUSION AND DISCUSSION

Commonly, segmentation of tumors, organs, and other features has been
performed using whole-body 3D PET images, either manually, semi-automatically,
or via automated deep-learning methods such as the U-Net [29]. Automated deep
learning methods working on whole-body PET images have so far not been
accurate enough [30, 31].

In this work, we have presented a fully automated three-step deep model for
fast and efficient segmentation of tumors from the DLBCL PET images. Our
supervised slice classifier and tumor detector modules help improve the
overall network segmentation performance since they localize the tumors for
the segmentor module to delineate. The slice classifier module also helps
eliminate the need to train the detector module on tumor-less background
slices. Moreover, as compared to direct deep learning-based segmentation of
tumors in whole-body PET images (U-Net 3D Dice score = 58.9% $\pm$ 16.1%), our
three-step model shows improved performance (3D Dice score = 78.1% $\pm$ 8.6%)
by about 19%. Another reason for the performance boost can also be attributed
to the fact that in our method each specialized module performing a specific
task can be trained independently to a high degree of skill, rather than a
single network with suboptimal performance on overall segmentation (unless a
very large dataset is provided).

As the performance of the overall network largely depended on the performance
of the slice classifier and the tumor detector modules (perfect tumor
detection leads to highly accurate segmentation prediction), in the future we
would like to explore various methods for improving the performance of our
detection module, such as by incorporating deformable convolutions units into
the detection network [32]. Overall, deep learning-based segmentation methods
for oncological PET images hold significant potential towards personalized
cancer therapy in the future.

###### Acknowledgements.

This work is supported by the Canadian Institutes of Health Research (CIHR)
Project Grant PJT-173231. S.A. would also like to acknowledge the support from
SNMMI Medical & Science Student Research grant 2021 and the Education and
Research Foundation for Nuclear Medicine and Molecular Imaging.

## References

  * [1] “A clinical evaluation of the International Lymphoma Study Group classification of non-Hodgkin’s lymphoma. The Non-Hodgkin’s Lymphoma Classification Project,” Blood 89, 3909–3918 (Jun 1997).
  * [2] Weisman, A. J., Kim, J., Lee, I., McCarten, K. M., Kessel, S., Schwartz, C. L., Kelly, K. M., Jeraj, R., Cho, S. Y., and Bradshaw, T. J., “Automated quantification of baseline imaging PET metrics on FDG PET/CT images of pediatric Hodgkin lymphoma patients,” EJNMMI Phys 7, 76 (Dec 2020).
  * [3] Slattery, A., “Validating an image segmentation program devised for staging lymphoma,” Australas Phys Eng Sci Med 40, 799–809 (Dec 2017).
  * [4] Capobianco, N., Meignan, M., Cottereau, A. S., Vercellino, L., Sibille, L., Spottiswoode, B., Zuehlsdorff, S., Casasnovas, O., Thieblemont, C., and Buvat, I., “F-FDG Uptake Classification Enables Total Metabolic Tumor Volume Estimation in Diffuse Large B-Cell Lymphoma,” J Nucl Med 62, 30–36 (01 2021).
  * [5] Guo, B., Tan, X., Ke, Q., and Cen, H., “Prognostic value of baseline metabolic tumor volume and total lesion glycolysis in patients with lymphoma: A meta-analysis,” PLoS One 14(1), e0210224 (2019).
  * [6] Vercellino, L., Cottereau, A. S., Casasnovas, O., Tilly, H., Feugier, P., Chartier, L., Fruchart, C., Roulin, L., Oberic, L., Pica, G. M., Ribrag, V., Abraham, J., Simon, M., Gonzalez, H., Bouabdallah, R., Fitoussi, O., Sebban, C., López-Guillermo, A., Sanhes, L., Morschhauser, F., Trotman, J., Corront, B., Choufi, B., Snauwaert, S., Godmer, P., Briere, J., Salles, G., Gaulard, P., Meignan, M., and Thieblemont, C., “High total metabolic tumor volume at baseline predicts survival independent of response to therapy,” Blood 135, 1396–1405 (04 2020).
  * [7] Martín-Saladich, Q., Reynés-Llompart, G., Sabaté-Llobera, A., Palomar-Muñoz, A., Domingo-Domènech, E., and Cortés-Romera, M., “Comparison of different automatic methods for the delineation of the total metabolic tumor volume in I-II stage Hodgkin Lymphoma,” Sci Rep 10, 12590 (07 2020).
  * [8] Casasnovas, R.-O., Kanoun, S., Tal, I., Cottereau, A.-S., Edeline, V., Brice, P., Bouabdallah, R., Salles, G. A., Stamatoullas, A., Dupuis, J., Reman, O., Gastinne, T., Joly, B., Bouabdallah, K., Nicolas-Virelizier, E., Andre, M., Mounier, N., Ferme, C., Meignan, M., and Berriolo-Riedinger, A., “Baseline total metabolic volume (tmtv) to predict the outcome of patients with advanced hodgkin lymphoma (hl) enrolled in the ahl2011 lysa trial.,” Journal of Clinical Oncology 34(15_suppl), 7509–7509 (2016).
  * [9] Meyer-Base, A., Schmid, V., and Books, E. A. A., [Pattern recognition and signal analysis in medical imaging ], Elsevier/Academic Press, Waltham, MA;Oxford, UK;, second ed. (2014).
  * [10] Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., and Lungren, M. P., “Preparing Medical Imaging Data for Machine Learning,” Radiology 295, 4–15 (04 2020).
  * [11] Lundervold, A. S. and Lundervold, A., “An overview of deep learning in medical imaging focusing on mri,” Zeitschrift fur Medizinische Physik 29(2), 102–127 (2019).  Special Issue: Deep Learning in Medical Physics.
  * [12] Buvat, I. and Orlhac, F., “The T.R.U.E. Checklist for Identifying Impactful Artificial Intelligence-Based Findings in Nuclear Medicine: Is It True? Is It Reproducible? Is It Useful? Is It Explainable?,” J Nucl Med 62, 752–754 (06 2021).
  * [13] Yousefirizi, F., Jha, A. K., Brosch-Lenz, J., Saboury, B., and Rahmim, A., “Toward High-Throughput Artificial Intelligence-Based Segmentation in Oncological PET Imaging,” PET Clin 16, 577–596 (Oct 2021).
  * [14] Ronneberger, O., Fischer, P., and Brox, T., [U-Net: Convolutional Networks for Biomedical Image Segmentation ], 234–241, Springer International Publishing, Cham (2015).
  * [15] He, K., Gkioxari, G., Dollar, P., and Girshick, R., “Mask r-cnn,” IEEE transactions on pattern analysis and machine intelligence 42(2), 386–397 (2020).
  * [16] Kumar, A., Fulham, M., Feng, D., and Kim, J., “Co-Learning Feature Fusion Maps from PET-CT Images of Lung Cancer,” IEEE Trans Med Imaging (Jun 2019).
  * [17] Blanc-Durand, P., Van Der Gucht, A., Schaefer, N., Itti, E., and Prior, J. O., “Automatic lesion detection and segmentation of 18F-FET PET in gliomas: A full 3D U-Net convolutional neural network study,” PLoS One 13(4), e0195798 (2018).
  * [18] Zhu, Z., Jin, D., Yan, K., Ho, T.-Y., Ye, X., Guo, D., Chao, C.-H., Xiao, J., Yuille, A., and Lu, L., [Lymph Node Gross Tumor Volume Detection and Segmentation via Distance-Based Gating Using 3D CT/PET Imaging in Radiotherapy ], 753–762, Springer International Publishing, Cham (2020).
  * [19] Lempitsky, V., Kohli, P., Rother, C., and Sharp, T., “Image segmentation with a bounding box prior,” in [2009 IEEE 12th International Conference on Computer Vision ], 277–284 (2009).
  * [20] Weisman, A. J., Kieler, M. W., Perlman, S. B., Hutchings, M., Jeraj, R., Kostakoglu, L., and Bradshaw, T. J., “Convolutional Neural Networks for Automated PET/CT Detection of Diseased Lymph Node Burden in Patients with Lymphoma,” Radiol Artif Intell 2, e200016 (Sep 2020).
  * [21] He, K., Zhang, X., Ren, S., and Sun, J., “Deep residual learning for image recognition,” in [2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ], 770–778 (2016).
  * [22] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L., “Imagenet: A large-scale hierarchical image database,” in [2009 IEEE Conference on Computer Vision and Pattern Recognition ], 248–255 (2009).
  * [23] Ren, S., He, K., Girshick, R., and Sun, J., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” IEEE Trans Pattern Anal Mach Intell 39, 1137–1149 (06 2017).
  * [24] Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S., “Feature pyramid networks for object detection,” (2016).
  * [25] Yousefirizi, F., Bloise, I., Martineau, P., Wilson, D., Benard, F., Bradshaw, T., Rahmim, A., and Uribe, C. F., “Reproducibility of a semi-automatic gradient-based segmentation approach for lymphoma pet,” European Journal of Nuclear Medicine and Molecular Imaging 48(1) (2021).
  * [26] Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P., “Focal loss for dense object detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2), 318–327 (2020).
  * [27] Murphy, K. P. and Central, E., [Machine learning: a probabilistic perspective ], MIT Press, Cambridge, MA (2012).
  * [28] Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S., and Jorge Cardoso, M., “Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations,” Deep learning in medical image analysis and multimodal learning for clinical decision support : Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, held in conjunction with MICCAI 2017 Quebec City, QC,… 2017, 240–248 (2017).  34104926[pmid].
  * [29] Weisman, A. J., Kieler, M. W., Perlman, S., Hutchings, M., Jeraj, R., Kostakoglu, L., and Bradshaw, T. J., “Comparison of 11 automated PET segmentation methods in lymphoma,” Phys Med Biol 65, 235019 (11 2020).
  * [30] Foster, B., Bagci, U., Mansoor, A., Xu, Z., and Mollura, D. J., “A review on segmentation of positron emission tomography images,” Comput Biol Med 50, 76–96 (Jul 2014).
  * [31] Rizwan I Haque, I. and Neubert, J., “Deep learning approaches to biomedical image segmentation,” Informatics in Medicine Unlocked 18, 100297 (2020).
  * [32] Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y., “Deformable convolutional networks,” 764–773, IEEE (2017).