# Query Refinement Prompts for Closed-Book Long-Form Question Answering

Reinald Kim Amplayo
Kellie Webster
Michael Collins
Dipanjan Das
Shashi Narayan
Google Research
{reinald, websterk, mjcollins, dipanjand<EMAIL_ADDRESS>Reinald
Kim Amplayo
Google Research
<EMAIL_ADDRESS>&Kellie Webster
Google Research
<EMAIL_ADDRESS>&Michael Collins
Google Research
<EMAIL_ADDRESS>&Dipanjan Das
Google Research
<EMAIL_ADDRESS>&Shashi Narayan
Google Research
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) have been shown to perform well in answering
questions and in producing long-form texts, both in few-shot closed-book
settings. While the former can be validated using well-known evaluation
metrics, the latter is difficult to evaluate. We resolve the difficulties to
evaluate long-form output by doing both tasks at once – to do question
answering that requires long-form answers. Such questions tend to be
multifaceted, i.e., they may have ambiguities and/or require information from
multiple sources. To this end, we define query refinement prompts that
encourage LLMs to explicitly express the multifacetedness in questions and
generate long-form answers covering multiple facets of the question. Our
experiments on two long-form question answering datasets, ASQA and AQuAMuSe,
show that using our prompts allows us to outperform fully finetuned models in
the closed book setting, as well as achieve results comparable to retrieve-
then-generate open-book models.

##  1 Introduction

Interest in large language models (LLMs) has exploded in recent years due to
their wide range of abilities improving the state-of-the-art with just a few
or no examples (Brown et al., 2020; Wei et al., 2022a; Chowdhery et al., 2022;
Wei et al., 2022b). One task that benefited greatly is closed-book question
answering (Roberts et al., 2020) in a few-shot setting, i.e., to produce
correct answers to questions without access to passages to read and find the
answer. Most of the impressive results, however, are limited to generating
short answers, and while previous work has utilized LLMs to generate long-form
texts (Elkins & Chun, 2020; Wei et al., 2022b), most of these outputs are
difficult to evaluate since they are subjective in nature (i.e., they may have
multiple correct and distinctive answers). In this paper, we attempt to
evaluate the long-form generation ability of LLMs through long-form question
answering (Fan et al., 2019) in the closed-book setting.

Unlike question answering with short answers, questions with long-form answers
are naturally multifacted, covering many aspects that are required to fully
answer the questions (see Table 1 for examples). This introduces several
challenges that LLMs would need to resolve:

  1. 1.

Ambiguity: Many questions in a real-world setting are ambiguous in some way;
Min et al. (2020) estimated that more than half of Natural Questions
(Kwiatkowski et al., 2019) have multiple plausible interpretations. In the
first example of Table 1, the question is ambiguous owing to there being
multiple types of cars. A system producing a long-form answer to this question
should present the multiple answers in a coherent natural language text.

  2. 2.

Information Consolidation: Some questions require consolidating information
from multiple sources to be able to fully answer them (Kulkarni et al., 2020).
In the second example of Table 1, information regarding the British and Dutch
origins of the term may be found in two different documents and LLMs should
therefore be able to synthesize knowledge from multiple sources in a coherent
manner.

  3. 3.

Correctness: Having a model to answer questions in a closed-book setting
requires it to leverage content it has learned during pretraining. While
previous work (Roberts et al., 2020; Brown et al., 2020) has shown that LLMs
can do so to produce factoid answers, generating long-form answers requires
more and complex information to be leveraged.

ASQA Question | when was the first car made in america?
---|---
Multiple Facets |  When was the first carriage-sized automobile made in America? |  1871
When was the first running, gasoline-powered car made in America? |  1893
|  … |  …
Long-form Answer |  The first carriage-sized automobile that could be driven on wagon roads in the United States was steam-powered and invented in 1871 by Dr. J.W. Carhart in Racine, Wisconsin. The first running, gasoline-powered car that was made in America, the Duryea Motor Wagon, was built in 1893. The Studebaker Automobile Company, which started building cars in 1897, sold electric vehicles in 1902, and gasoline vehicles in 1904.
AQuAMuSe Question | where did the term shooting brake come from?
Multiple Facets |  When did the term originate? |  early 19th century
As what term did the term originate? |  as a British term
|  … |  …
Long-form Answer |  “Shooting-brake” originated as an early 19th century British term for a vehicle used to carry shooting parties with their equipment and game. The etymology of the term brake is uncertain; initially a chassis used to break in horses, and subsequently used to describe a motorized vehicle. It is also possible, that the word ‘brake’ has its origins in the Dutch word ‘brik’ which means ‘cart’ or ‘carriage’.
Table 1: Example multifaceted questions from ASQA and AQuAMuSe and their
corresponding answers. A system for closed-book question answering needs to
understand that some questions have multiple valid answers and synthesise
these into a coherent natural language text output.

Our study is based on two long-form question answering benchmarks, ASQA
(Stelmakh et al., 2022) and AQuAMuSe (Kulkarni et al., 2020), which focus on
queries that may be ambiguous and/or require information from multiple
sources. Using these two datasets, our work is the first to show that LLMs are
capable of generating long-form answers to complex questions of various types
in a closed-book setting. We devise novel query refinement prompts that
encourage LLMs to express multiple facets of input question and generate
multifaceted answers discussing all identified interpretations coherently.
Specifically, we identify several different types of multifacetedness in
questions and produce a labeled set of query refinement prompts for question
answer pairs with a balanced coverage over different types. We then introduce
an intermediate query refinement step, a generation subtask akin to
explanations and reasoning chains (Wei et al., 2022b; Zhou et al., 2022),
where the goal is to identify multiple facets of a given question.

We evaluate our long-form answers using ROUGE (measuring stylistic similarity
to gold answers; Lin, 2004) and reading comprehension models (measuring
correctness; Stelmakh et al., 2022). Using our query refinement prompts in the
few-shot prompting and prompt tuning (Lester et al., 2021) settings, we are
able to achieve significantly better performance to fully finetuned closed-
book systems, as well as comparable performance to open-book retrieve-then-
generate systems, both finetuned on the full training dataset.

##  2 Related Work

###  2.1 Prompting in Large Language Models

Large language models have the revolutionary ability to generalize to tasks
presented with natural language prompts (Brown et al., 2020; Chowdhery et al.,
2022). This ability has been mostly attributed to the large scale of these
models and the learning objective of predicting the next token (Brown et al.,
2020). The prompting ability of LLMs has been used successfully in text
classification, reading comprehension and open-domain question answering
tasks. It has also been shown that LLMs improve on complex reasoning tasks by
generating intermediate long-form texts in the form of explanations or
reasoning steps (Wei et al., 2022b; Zhou et al., 2022). However, prompting is
still ineffective when tasked to generate long-form outputs as an end task,
e.g., generating long summaries for summarization (Brown et al., 2020;
Chowdhery et al., 2022). Our work is the first to show that LLMs can do long-
form text generation through question answering with the help of a refinement
step in the prompt.

Our work is related to and inspired by work on reasoning chains in LLMs (Wei
et al., 2022b; Zhou et al., 2022; Kojima et al., 2022; Snell et al., 2022),
where the goal is to explicitly generate a reasoning or an explanation before
producing an answer. Most of these papers focus on arithmetic and commonsense
reasoning questions, where reasoning and explanations are obvious. In this
paper, we show that such intermediate explanation generation can also be
helpful on tasks that implicitly involve multiple steps, such as long-form
question answering where question refinement is necessary. Moreover, we are
the first to explore structured explanations in the form of a list of answer
facets, which is shown in our experiments to be more effective than natural
language explanations.

Prompting is just one way of using the LLMs. There are several work (Sanh et
al., 2021; Wei et al., 2021; Chowdhery et al., 2022) that attempted to
finetune LLMs entirely for text generation tasks, which can be very expensive.
Prompt tuning (Lester et al., 2021) is a popular alternative where soft
prompts are prepended into the input and are finetuned. There are several
other alternatives to prompting that show promising results for generating
long-form texts, such as prefix tuning (Li & Liang, 2021), adapters (Bapna &
Firat, 2019), and several parameter-efficient finetuning techniques (Clive et
al., 2021; He et al., 2022; Liu et al., 2022) that introduce new parameters to
the model that is updated during training while leaving the LLM parameter
fixed. All of these, however, still require several forward passes to the
LLMs, which may not be available in many cases. Nevertheless, we show that
applying our query refinement step in prompt tuning improves the performance.

###  2.2 Long-form Question Answering

Question answering has emerged as a key way to discover and demonstrate
advances in large language models, which are showing their skill on
increasingly difficult formulations of the task. SQuAD (Rajpurkar et al.,
2016) proposed the first large-scale, human-created reading comprehension task
and was used to show the promise of neural architectures, which quickly
attained human-like performance on the dataset. Since, there has been a
proliferation of reading comprehension datasets developed which probe for
specific capabilities (Joshi et al., 2017; Choi et al., 2018; Reddy et al.,
2019; Rodriguez et al., 2019). The Natural Questions (Kwiatkowski et al.,
2019) effort provided a large reading comprehension dataset based on real
information-seeking queries to the Google search engine, and has served most
recently a basis for the exploration of questions where a simple short answer
is not sufficient to address the information need of a complex question.

One response strategy to such questions is a long-form answer, studied here.
Both ASQA (Stelmakh et al., 2022) and AQuaMuSe (Kulkarni et al., 2020) require
that systems consolidate information from multiple sources to generate
multifaceted long-form answers to questions from the Natural Questions. ASQA
focuses on the subset of questions labeled in AmbigQA (Min et al., 2020) for
which it is possible to enumerate a collection of refinements and factoid
answers that should be covered in a long-form answer. On the other hand,
AQuaMuSe focuses on questions without short factoid answers, that typically
have a looser relationship to one another. We study both so as to understand
what prompting strategies work for the different style of reasoning required
to do well on each. ELI5 (Fan et al., 2019) is another long-form question
answering dataset that was automatically gathered from Reddit threads, but
subsequent work (Krishna et al., 2021) has shown problems in its evaluation,
including training/validation overlap and gameable metrics.

Finally, our work is also related to query-focused summarization (Dang, 2005;
Zhong et al., 2021; Kulkarni et al., 2020), where a set of relevant passages
is assumed to be available. AQuaMuSe was developed for this task but, where
our experiments are in the closed-book setting, we discard the given passages.

##  3 Closed-book Long-form Question Answering

###  3.1 Few-Shot Prompting Formulation

Given a question $q$, the goal of closed-book long-form question answering is
to produce a passage-length text $a$ without access to external context or
knowledge (beyond what was seen in pretraining). For the related task of
closed-book factoid question answering (Rajpurkar et al., 2016; Joshi et al.,
2017), this can been achieved with large language models using a few-shot
prompting setup (Brown et al., 2020). That is, given $k$ in-context exemplars
of question-answer pairs
$[(q^{\prime}_{1},a^{\prime}_{1}),...,(q^{\prime}_{k},a^{\prime}_{k})]$,
usually preceded by an instruction, an LLM will output an answer $a$ for
question $q$ from knowledge stored in its parameters (Roberts et al., 2020).

When the answer is instead long-form (see examples in Table 1), there are
three subtasks that the model need to do to produce an answer: (1) Determining
multiple facets of the question (Facet Identification), (2) Retrieving
multiple answers to the multiple facets of the question (Multifaceted Question
Answering), and (3) Realizing a long-form text that includes the multiple
answers in a coherent manner (Surface Realization with Information
Consolidation).

In the next sections, we extend the standard few-shot prompt in three
different ways to help LLMs explicitly do these steps before arriving to an
answer. Firstly, we identify several types of multifacetedness in questions
and produce a labeled and balanced set of exemplars (Section 3.2). Next, we
introduce a query refinement step in few-shot prompting that instructs the
model to explicitly do the intermediate subtasks (Section 3.3). Finally, we
dynamically select exemplars to form a $k$-shot prompt based on similarity
(Section 3.4).

###  3.2 Types of Multifaceted Questions

There are multiple reasons why a long-form answer would be more felicitous
than a factoid answer to a question. Table 2 shows six common types of
multifaceted questions in the ASQA (Stelmakh et al., 2022) and AQuAMuSe
datasets that fall into this category. These are highly related to the
ambiguity categories in AmbigQA (Min et al., 2020), which we used as a seed
set for exploring the data in this work. To encourage further work in
multifaceted question categorization, we detail the criteria we used to
determine what type an example demonstrates. We produce a manually labeled set
that contains 20 exemplars of each type to form a pool of possible training
instances (see Table 2 for examples). When an example exhibited multiple types
simultaneously, we opted not to include it as an exemplar.

(a) Conditional: The original question needs to be refined by specifying
additional conditions that may be specifications or constraints.
---
Question: When did movies start being made in color?
Multifaceted QA Pairs:
Q: When was the first film made that utilized any type of color? A: September
1, 1902
Q: When did the first feature length film come out that was made entirely in
three-strip Technicolor? A: June 13, 1935
Related AmbigQA categories: Event references, Properties
(b) Set-Valued: The answer to the question is a unstructured collection of
size two or greater.
Question: What are the neighboring countries of South Korea?
Multifaceted QA Pairs:
Q: What are the neighboring countries to the North of South Korea? A: North
Korea
Q: What are the neighboring countries to the South of South Korea? A: Japan
Related AmbigQA category: $\emptyset$
(c) Time Dependent: The answer depends on the time at which the question was
asked, or changed over time in the past.
Question: Where was indian independence league formed in 1942?
Multifaceted QA Pairs:
Q: Where was indian independence league brought together in March 1942? A:
Tokyo
Q: Where was indian independence league brought together in June 1942? A:
Bangkok Conference
Related AmbigQA category: Time-dependency
(d) Underspecified Reference: There is a noun phrase in the question which may
be resolved in multiple ways.
Question: When did bat out of hell come out?
Multifaceted QA Pairs:
Q: When did the album bat out of hell come out? A: October 21, 1977
Q: When did the TV series bat out of hell come out? A: 26 November 1966
Related AmbigQA category: Entity references
(e) Underspecified Type: The entity type or sub-type is not specified in the
question.
Question: Who is the mayor in horton hears a who?
Multifaceted QA Pairs:
Q: Who plays the mayor in the 2008 film Horton Hears a Who? A: Steve Carell
Q: Who is the mayor in the 2008 film Horton Hears a Who? A: Mayor Ned McDodd
Related AmbigQA category: Answer types
(f) Needs Elaboration: The answer needs to be elaborated to fully answer the
question
Question: Where did “you can’t have your cake and eat it too” come from?
Multifaceted QA Pairs:
Q: Where was the early recording of the phrase found? A: in a letter on 14
March 1538
Q: Who sent the letter? A: Thomas, Duke of Norfolk
Q: To whom was the letter sent to? A: Thomas Cromwell
Q: How was it phrased in the letter: A: “a man cannot have his cake and eat
his cake”
Related AmbigQA category: $\emptyset$
Table 2: Six types of multifacetedness in questions. The first five types are
sampled from the ASQA dataset, while the last type is sampled from the
AQuAMuSe dataset. For each type, we also include the related AmbigQA (Min et
al., 2020) categories if there are any.

###  3.3 Query Refinement Step

Given that long-form question answering is essentially sequentially solving
three subtasks, we propose to use an intermediate step that splits the tasks
into two parts. This forces the LLM to explicitly resolve the intermediate
subtasks before producing a long-form answer. We experimented with three types
of refinements:

  * •

Natural Language Explanations (NL): A sentence that explains why the question
is ambiguous or needs elaboration. This refinement step essentially does Facet
Identification, i.e., it identifies the multiple facets of a question, which
can either be disambiguations of the question, or things that need to be
elaborated to fully answer the question. In previous work, LLMs have been used
to generate NL explanations, e.g. for commonsense reasoning (Ji et al., 2020)
and jokes (Chowdhery et al., 2022).

  * •

Lists of QA Pairs (QA): A list of question-answer pairs. We use the
multifaceted QA pairs as shown in Table 2. This refinement does both Facet
Identification and Multifaceted Question Answering, which means that when
producing the answer, the model just needs to consolidate information into a
coherent long-form answer. This is related to question answering blueprints
(Narayan et al., 2022), which are used as content plans for summarization
models, but their usage has not yet been explored in LLMs.

  * •

Lists of Answer Facets (AF): A list of answer facets, pairings of the multiple
facets of the question and their corresponding answer/s in the form “Facet:
Answer”. This is a concise version of the BP refinement, where we remove
uninformative tokens such as wh-words and those that are repeated in the
original question.

Table 3 shows all three refinement strategies for all identified ambiguity
types. The refinement is inserted between the question and the answer in the
exemplar. At inference time, the LLM needs to generate both the refinement and
the long-form answer. An example prompt with a query refinement step is
illustrated in Figure 1 of Appendix A.

Question: When did movies start being made in color?
---
QA Refinement:
Q: When was the first film made that utilized any type of color? A: September
1, 1902
Q: When did the first feature length film come out that was made entirely in
three-strip Technicolor? A: June 13, 1935
NL Refinement: The answer depends on what is meant by in color (any type of
color or three-strip Technicolor).
AF Refinement:
\- any type of color: September 1, 1902
\- three-strip Technicolor: June 13, 1935
Answer: The first film that utilized any type of color was made September 1,
1902. Edward Raymond Turner shot test footage to demonstrate his system, but
projecting it proved problematic and he died a year later without having
satisfactorily projected the footage. Later, in 2012, curators at the National
Media Museum in Bradford, UK, were able to take the original custom-format
nitrate film copied to black-and-white 35 mm film, which was then scanned into
a digital video format by telecine and digital image processing was used to
align and combine each group of three frames into one color image so that
these films from 1902 could become viewable in full color. The first feature
length film that was made entirely in three-strip Technicolor, which was an
early process where color filters were used to photograph the color components
as completely separate images, came out on June 13, 1935.
Table 3: An example exemplar with QA, NL, and AF refinements. One exemplar in
the $k$-shot prompt would include a question, one of the three refinements,
and the answer. At inference time, the model would need to generate both the
refinement and the answer.

###  3.4 Dynamic Prompting

Finally, we form a k-shot prompt by selecting from our pool of exemplars
created in Section 3.2. We do so using dynamic prompting (DP; Rubin et al.,
2022), i.e., ranking exemplars
$[(\hat{q}_{1},\hat{a}_{1}),...,(\hat{q}_{e},\hat{a}_{e})]$ using the
similarity between input question $q$ and candidate exemplar question
$\hat{q}$. Dynamic prompting helps the model generate refinements for $q$
similarly to how refinements are done for a similar question $\hat{q}$. We use
BERTScore (Zhang* et al., 2020) as our similarity metric. The $k$ most similar
exemplars are written to the prompt in reverse order, such that the most
similar exemplar is written closest to input question $q$. In our experiments,
we primarily used exemplars with questions labeled as ambiguous for ASQA (top
5 in Table 2, for a total of 100 exemplars), and those that are labeled as
Needs Elaboration for AQuAMuSe (the last type in Table 2, for a total of 20
exemplars). We also experimented combining both ASQA and AQuAMuSe exemplars in
Section 5.

##  4 Experiments on Ambiguous Question Answering

We conducted experiments on two question answering tasks that require long-
form answers: Ambiguous Question Answering (ASQA; Stelmakh et al., 2022) and
Query-focused Multi-document Summarization (AQuAMuSe; Kulkarni et al., 2020).
In this section, we present results on ASQA; the following section presents
results on AquaMuSe.

###  4.1 ASQA Dataset

The ASQA dataset (Stelmakh et al., 2022) is a long-form question answering
dataset built on top of the subset of ambiguous questions identified in the
AmbigQA dataset (Min et al., 2020), which itself is a subset of the NQ dataset
(Kwiatkowski et al., 2019). ASQA consists of 4,353, 948, and 1,015 training,
development, and test examples. Each question (e.g., Who directed Scarface?)
is paired with a list of QA pairs which signify disambiguated questions and
their corresponding answers (e.g., Q: Who directed the 1932 film Scarface? A:
Howard Hawks, Q: Who directed the 1983 film Scarface? A: Brian de Palma).
Finally, each example also has two human-written long-form answers based on
the given the disambiguated QA pairs. Note that the list of disambiguated
questions is not given at inference time.

###  4.2 Evaluation

Following Stelmakh et al. (2022), we compare systems using three metrics:

  * •

ROUGE-L (Lin, 2004): Measures the comprehensibility of the system-generated
answer with respect to the gold answers. Since there are two gold answers, we
take the maximum ROUGE-L. We lowercase system and gold answers, and report on
ROUGE-LSum (f-measure) with the default stemmer on.111Using pypi package
rouge-score.

  * •

Disambig-F1 (Disambiguated F1 Accuracy): Measures the correctness of the
system-generated answer. Given the gold-standard disambiguated QA pairs, we
run a reading comprehension model (RoBERTa Liu et al., 2019 trained on SQuAD
2.0 Rajpurkar et al., 2018) where the system-generated long-form answer is the
context. We then evaluate the number of disambiguated questions that can be
answered using the long-form answer as context by calculating the Q1-F1
accuracy.

  * •

DR (Disambiguation-ROUGE): The geometric mean of ROUGE-L and Disambig-F1 which
penalizes methods that maximize one metric over another.

Finally, we also report the average number of words of the system-generated
answers.

###  4.3 Results

We compared several systems which can be divided into three types: few-shot
prompting, prompt tuning, and finetuning.

For few-shot prompting, we used five exemplars in the prompt, and compared
several prompt configurations using the 540B-sized PaLM model (Chowdhery et
al., 2022):

  * •

Random exemplars (random): Select at random five QA pairs from the training
dataset and use as exemplars. We report the average results of five different
runs.

  * •

Query diversified exemplars (QD): Select five QA pairs with questions that
have different classes of ambiguity as classified in Section 3.2. We only
considered the 100 exemplars with questions labeled as one of five
multifacetedness types, which are originally sampled from the ASQA training
dataset (i.e., top 5 in Table 2).

  * •

NL/QA/AF refinement: Include a query refinement step and dynamically select
exemplars from a pool of exemplars.

The first block of Table 4 reports the results from few-shot prompting. As can
be seen, using exemplars with questions of different types of ambiguity
significantly improves over the random baseline. All query refinement prompts
improve the performance, where the AF refinement performs the best among them.

Model | #Words | ROUGE-L | Disambig-F1 | DR
---|---|---|---|---
Few-shot Prompting
PaLM 540B (random) | 75.5 | 31.1 | 18.6 | 24.0
PaLM 540B (QD) | 62.1 | 31.3 | 22.8 | 26.7
\+ NL refinement | 63.6 | 33.6 | 23.9 | 28.3
\+ QA refinement | 41.9 | 32.3 | 25.3 | 28.6
\+ AF refinement | 62.4 | 34.5 | 25.3 | 29.6
GPT 175B (random)∗ | 21.6 | 12.9 | 7.7 | 10.0
\+ AF refinement | 46.5 | 30.0 | 17.1 | 22.6
InstructGPT-3 175B (random) | 40.7 | 31.8 | 25.0 | 28.2
\+ AF refinement | 39.0 | 31.6 | 23.4 | 27.2
Few-shot Prompt Tuning
PaLM 540B 100-shot | 62.5 | 36.1 | 25.0 | 30.0
\+ AF refinement | 53.8 | 36.7 | 25.4 | 30.6
Using the full dataset
T5-Large Closed Book | 62.5 | 31.0 | 7.4 | 15.1
T5-Large Open Book 1 Passage | 63.0 | 36.5 | 21.2 | 27.9
T5-Large Open Book 3 Passages | 71.1 | 38.8 | 25.1 | 31.2
T5-Large Open Book 5 Passages | 71.6 | 39.2 | 26.4 | 32.1
PaLM 540B Prompt Tuning | 64.1 | 37.4 | 27.8 | 32.1
Table 4: Evaluation of several systems on the dev set of the ASQA dataset. The
best values for each setting are bold-faced. We mark systems with an asterisk
(∗) if they failed to generate answers for at least half the total number of
examples.

To check whether our prompts work with other models, we also test our prompts
with 175B-sized GPT-3 (Brown et al., 2020) and InstructGPT-3 (Ouyang et al.,
2022) models, named davinci and text-davinci-002, respectively, the latter
finetuned further with humans in the loop to better follow user
instructions.222 We note that the training data used for text-davinci-002 is
unknown, i.e., we do not know whether the model had access to supervision from
long-form question answering datasets during its training.  As shown in the
second block of Table 4, using standard prompts in GPT-3 fails to generate
answers for at least half the total number of examples. Just by adding our
proposed configurations, the performance of GPT-3 significantly increases.
Interestingly, we do not see the same increasing trend with the InstructGPT-3
model.

Our prompts can also be applied to prompt tuning (Lester et al., 2021) where a
set of learned embeddings called soft prompts is prepended to the prompt. In
particular, we follow the method in Rubin et al. (2022), i.e., we prepend one
soft prompt333 We tried several soft prompt lengths and found that increasing
the length beyond one prompt does not lead to any improvements.  to the input
and finetune it using the 100 exemplars we used for dynamic prompting. The
third block in Table 4 reports prompt tuning results, where we see a slight
improvement when applying our AF refinement prompts.

Finally, we compare our few-shot systems with systems that are trained with
the full dataset. In the final block of Table 4, we show results reported in
Stelmakh et al. (2022), which are T5-large models in both closed-book (no
retrieved passages) and open-book (1/3/5 retrieved passages using Joint
Passage Ranker; Min et al., 2021) settings. We also report PaLM 540B prompt
tuned using the full dataset. Our best few-shot systems are surprisingly
competitive compared to fully finetuned T5 systems, outperforming the closed
book system and the open book system with one retrieved passage. Moreover, the
correctness of our best systems as measured by the Disambig-F1 score is on par
with the open book T5 models. Finally, prompt tuning PaLM using the full
dataset performs the best among all systems in terms of correctness, despite
having a lower ROUGE-L score than the best T5 system. We believe that open-
book models have higher ROUGE-L scores due to the fact that they have access
to retrieved passages that they can directly copy, and which may follow the
format of the human-generated answers. We discuss this and other annotator
biases further in Section 4.5.

###  4.4 Ablation Studies

We conducted ablation studies on the best few-shot prompting configuration,
which is shown in Table 5. In terms of the number of exemplars, while
increasing it from 1 to 5 improves the performance, increasing it from 5 to 10
slightly decreases the overall performance. However we see an increase in
ROUGE-L and STR-EM, which shows that access to more data increases the ability
of the model to copy answer formats. Moreover, increasing the number of
dynamic exemplars leads to performance improvements, which is unsurprising.
Also, when removing one component (refinement or dynamic prompting), we see a
substantial decrease in performance, where the decrease is larger when
refinement is not used. Using a different similarity metric for prompt
selection does not have significant changes in the model performance, however
model-based metrics such as BERTScore (Zhang* et al., 2020) and BLEURT (Sellam
et al., 2020) are slightly better than string-based metrics such as BM25
(Robertson et al., 1995). Finally, the increase in performance of using AF
query refinement prompts can also be seen when using smaller versions of PaLM.

Model | #Words | ROUGE-L | Disambig-F1 | DR
---|---|---|---|---
Number of exemplars in the prompt
1-shot | 55.1 | 31.7 | 23.2 | 27.1
3-shot | 37.9 | 32.0 | 23.7 | 27.6
5-shot | 62.4 | 34.5 | 25.3 | 29.6
10-shot | 66.0 | 34.9 | 24.6 | 29.3
Number of dynamic prompt exemplars
5 | 69.0 | 35.0 | 22.1 | 27.8
25 | 68.0 | 33.9 | 23.2 | 28.0
50 | 63.8 | 34.6 | 23.6 | 28.6
100 | 62.4 | 34.5 | 25.3 | 29.6
Without one component
Without refinement | 66.3 | 33.2 | 22.9 | 27.6
Without dynamic prompting | 40.1 | 32.5 | 25.1 | 28.6
Similarity metric for prompt selection
BERTScore | 62.4 | 34.5 | 25.3 | 29.6
BLEURT | 65.3 | 34.6 | 25.1 | 29.5
BM25 | 64.4 | 34.6 | 24.4 | 29.0
Smaller models
PaLM 8B (random) | 84.1 | 21.1 | 9.2 | 14.0
\+ AF refinement | 63.9 | 29.6 | 10.0 | 17.2
PaLM 62B (random) | 66.6 | 28.7 | 14.5 | 20.4
\+ AF refinement | 65.3 | 32.4 | 18.0 | 24.1
Table 5: Performance of ablated versions of the best few-shot prompting
configuration.

###  4.5 Further Analyses

In the next paragraphs, we discuss annotator biases in ASQA that few-shot
systems would not be able to capture.

#### Question Disambiguation Analysis

The first bias is in the way questions are disambiguated. We observed cases
where the majority ambiguity class selected from dynamic prompting was a
plausible type of ambiguity for the given question, but the particular
disambiguation in ASQA is of another type. This leads to few-shot systems
generating an entirely different yet also correct answer to the question, an
example of which is shown in the first block of Table 6. The gold and the T5
answers disambiguated the question based on the habitat of the animal (fresh
water or saltwater), while the PaLM answer disambiguated it based on both the
habitat and the characteristics (having gills, having lungs, or having both).
In fact, by just providing how the ASQA dataset expects the question to be
disambiguated (prompt template shown in Section A), the performance of our
best configuration improves by up to 8.8 DR points (29.6 vs 38.4). Future work
on long-form question answering evaluation should explore methods to deal with
questions that can be disambiguated in multiple ways.

#### Summary Format Analysis

The second bias is the way long-form answers are annotated. The human-written
long-form answers from all the splits of ASQA are written by the same set of
annotators using the same annotation template. This creates a formatting bias
that models finetuned using thousands of examples can capture. We attribute
the low ROUGE-L scores of few-shot systems, in comparison to finetuned
systems, to this bias. The second example in Table 6 shows this bias, where
both answers are equally answer the question, but are written differently and
thus have a significant difference in ROUGE-L. While both systems correctly
answered Tony Hawk and Danny Way, the T5 long answer looks more similar to the
gold-standard answer, whereas the PaLM long answer looks more concise and
arguably more readable. This finding is related to that of Goyal et al.
(2022), where they showed that document summaries generated by GPT-3 are
always preferred by humans but are ranked the lowest by ROUGE. Future work on
evaluation should explore reference-free evaluation, where answers are instead
compared to trustworthy sources, similar to Rashkin et al. (2021).

Question: What do you call animals live in water?
---
Gold: Marine life, or sea life or ocean life, is the plants, animals and other
organisms that live in the salt water of the sea or ocean, or the brackish
water of coastal estuaries. An aquatic animal is an animal, either vertebrate
or invertebrate, which lives in the water for most or all of its lifetime. The
term aquatic can be applied to animals that live in either fresh water or salt
water. However, the adjective marine is most commonly used for animals that
live in saltwater, i.e. in oceans, seas, etc.
T5 best: aquatic animals are animals that live in water for most or all of
their lifetime. aquatic animals can be either vertebrate or invertebrate.
fresh water animals are aquatic animals. marine animals are animals that live
in saltwater.
ROUGE-L: 44.96
PaLM best: Animals that live in water are called aquatic animals. Aquatic
animals that have gills are called fish. Aquatic animals that have lungs are
called marine mammals. Aquatic animals that have both gills and lungs are
called amphibians.
ROUGE-L: 20.16
Question: Who landed the first 900 on a skateboard?
Gold: The 900 is a 2 1/2-revolution (900 degrees) aerial spin performed on a
skateboard ramp. A number of opinions exist about pre-1999 900s. The most
prominent of these is the argument that Danny Way landed the 900 in 1989, and
it appears in an early Santa Cruz film. Tony Hawk, one of the most successful
vertical pro skateboarders in the world, is generally said to have landed the
first 900 on a skateboard. He landed the 900 at X Games V in 1999 after ten
failed attempts.
T5 best: the 900 is a 2 1/2-revolution (900 degrees) aerial spin performed on
a skateboard ramp. a number of opinions exist about pre-1999 900s. the most
prominent of these is the argument that danny way landed the 900 in 1989, and
it appears in an early santa cruz film. in 1999 tony hawk, one of the most
successful vertical pro skateboarders in the world, landed ”the 900” at the
1999 x-games after ten failed attempts. in october 2011, schaar became the 8th
person in history to land a 900 on a skateboard. in september 2016, schaar
became the 1st person in history to land a stalefish 900 on a skateboard.
ROUGE-L: 80.00
PaLM best: The 900 is a skateboarding trick in which the skateboarder does two
and a half vertical revolutions. American professional skateboarder Tony Hawk
landed the first 900 in a competition at the X Games in 1999. Australian
professional skateboarder Danny Way landed the first 900 on video in 1990.
ROUGE-L: 45.59
Table 6: Example ASQA outputs from the T5 and PaLM best configurations.
Disambiguated answers are bold-faced.

##  5 Experiments on Query-focused Multi-document Summarization

###  5.1 Dataset

The AQuAMuSe dataset (Kulkarni et al., 2020) is a query-focused multi-document
summarization dataset, which was created to simulate how a search engine would
consolidate information from multiple documents of high relevance to a given
query. The dataset is also a subset of the NQ dataset (Kwiatkowski et al.,
2019), but is extended with web documents extracted from Common Crawl and
long-form answers from Wikipedia. The dataset consists of 6,599, 714, and 849
training, development, and test examples, where each example is given, on
average, 6.46 web documents (2,008 tokens per document). We note that in the
closed book setting, the web documents are not used.

###  5.2 Evaluation

We compare systems using an n-gram overlap-based metric ROUGE-1/2/L (Lin,
2004) and a QA-based metric QAEval${}_{\text{rheme}}$ (Narayan et al., 2022).
QAEval${}_{\text{rheme}}$ is similar to the original QAEval (Deutsch et al.,
2021), where a question generation model is used to generate questions from
the gold summary, and a question answering model attempts to answer these
questions using the system-generated summary as the context. In
QAEval${}_{\text{rheme}}$, the way questions are generated is modified such
that we only consider questions that are information-seeking (i.e., based on
the theme-rheme structure; Vallduví & Vilkuna, 1998; Kruijff-Korbayová &
Steedman, 2003).

###  5.3 Results

We compared the following systems. Few-shot prompting systems include PaLM
540B using random exemplars, and using NL/QA/AF refinements. For the
refinements, we used the 20 exemplars labeled with the Needs Elaboration type,
which are sampled from the AQuAMuSe training dataset (i.e., the sixth type in
Table 2). We also experimented including from ASQA prompt exemplars during
dynamic prompt selection to allow the model to more effectively differentiate
question types. Few-shot prompt tuning systems include PaLM 540B, with and
without AF refinement, and with ASQA prompt exemplars during dynamic prompt
selection. Finally, fully finetuned systems include T5-XL closed book and open
book, LongT5-XL (Guo et al., 2022) open book that allows longer contexts, and
PaLM 540B prompt tuning. For the open book systems, we filled in the context
with as many passages as possible within the limits of their input lengths.
This results to T5 having 8.3 passages and LongT5 having 30.0 passages on
average.

Table 7 reports the scores of these systems. Here, we see similar results as
in ASQA; in the few-shot prompting and prompt tuning systems, the addition of
AF refinements improves the most over the random baseline among the three
types of refinements. Moreover, the inclusion of ASQA prompt exemplars during
dynamic prompt selection also substantially improves the scores. It is
interesting to note that, in contrast to ASQA results, prompt tuning performs
worse than few-shot prompting, having worse QAEval${}_{\text{rheme}}$ scores
overall. When compared with finetuning systems, our prompting systems
significantly outperform the closed book variant of T5, but fall behind the
open book systems. We believe that this is due to the fact that AQuAMuSe
summaries are extractive – only 2%/13%/24%/31% of the
unigrams/bigrams/trigrams/4-grams in the summaries are novel. This allows
open-book systems to just copy directly from the source.

System | ROUGE-1 | ROUGE-2 | ROUGE-L | QAEval${}_{\text{rheme}}$
---|---|---|---|---
Few-shot Prompting
PaLM 540B (random) | 31.52 | 15.66 | 28.12 | 10.26
\+ NL refinement | 37.52 | 17.34 | 33.62 | 12.20
\+ QA refinement | 34.78 | 16.97 | 31.21 | 11.50
\+ AF refinement | 36.84 | 16.92 | 32.94 | 12.50
\+ ASQA exemplars | 37.72 | 18.11 | 33.73 | 13.24
Few-shot Prompt Tuning
PaLM 540B 20-shot | 34.12 | 11.89 | 29.63 | 7.96
\+ AF refinement | 34.33 | 12.43 | 30.03 | 9.13
\+ ASQA exemplars | 37.45 | 16.55 | 33.30 | 11.77
Using the full dataset
T5-XL Closed Book | 29.39 | 10.45 | 26.17 | 5.62
T5-XL Open Book 8.3 Passages | 44.93 | 27.10 | 41.53 | 19.77
LongT5-XL Open Book 30.0 Passages | 64.11 | 50.60 | 61.43 | 39.22
PaLM 540B Prompt Tuning | 40.53 | 19.66 | 36.56 | 14.62
Table 7: Evaluation of several systems on the test set of the AQuAMuSe
dataset. The best values for each block are bold-faced.

##  6 Conclusions

In this paper, we investigated the ability of large language models to answer
questions in a long-form manner through two question answering datasets: ASQA
and AQuAMuSe. We introduced query refinement prompting that improves over
standard few-shot prompting and prompt tuning methods by encouraging the model
to explicitly express the multifacetedness in questions. With the use of query
refinement prompts on both few-shot closed book prompting and prompt tuning
settings, we are able to outperform systems trained using the full training
data in both ASQA and AQuAMuSe. While we also achieve comparable results with
open book systems in ASQA, we acknowledge that these systems still perform
better when they have access to more retrieved passages. For future work, we
plan to explore ways to few-shot prompt large language models in the open book
setting and ways to augment a retrieval component into large language models.

## Acknowledgements

The authors would like to thank Tal Schuster and Mirella Lapata for their
detailed feedback, and Ji Ma and Priyanka Agrawal for their help in prompt
tuning experiments. Finally, we are thankful to the PaLM team in Google
Research for providing easy-to-use tools to experiment with large language
models.

## References

  * Bapna & Firat (2019) Ankur Bapna and Orhan Firat.  Simple, scalable adaptation for neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 1538–1548, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1165.  URL https://aclanthology.org/D19-1165.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer.  QuAC: Question answering in context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 2174–2184, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1241.  URL https://aclanthology.org/D18-1241.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Clive et al. (2021) Jordan Clive, Kris Cao, and Marek Rei.  Control prefixes for text generation.  _arXiv preprint arXiv:2110.08329_ , 2021.
  * Dang (2005) Hoa Trang Dang.  Overview of duc 2005.  In _Proceedings of the document understanding conference_ , volume 2005, pp. 1–12, 2005.
  * Deutsch et al. (2021) Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth.  Towards question-answering as an automatic metric for evaluating the content quality of a summary.  _Transactions of the Association for Computational Linguistics_ , 9:774–789, 2021.  doi: 10.1162/tacl˙a˙00397.  URL https://aclanthology.org/2021.tacl-1.47.
  * Elkins & Chun (2020) Katherine Elkins and Jon Chun.  Can gpt-3 pass a writer’s turing test?  _Journal of Cultural Analytics_ , 5(2):17212, 2020\.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli.  ELI5: Long form question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 3558–3567, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1346.  URL https://aclanthology.org/P19-1346.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett.  News summarization and evaluation in the era of gpt-3.  _arXiv preprint arXiv:2209.12356_ , 2022.
  * Guo et al. (2022) Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang.  LongT5: Efficient text-to-text transformer for long sequences.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pp. 724–736, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-naacl.55.  URL https://aclanthology.org/2022.findings-naacl.55.
  * He et al. (2022) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=0RDcd5Axok.
  * Ji et al. (2020) Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, and Minlie Huang.  Generating commonsense explanation by extracting bridge concepts from reasoning paths.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pp. 248–257, Suzhou, China, December 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.aacl-main.28.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1147.  URL https://aclanthology.org/P17-1147.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_ , 2022.
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer.  Hurdles to progress in long-form question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 4940–4957, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.393.  URL https://aclanthology.org/2021.naacl-main.393.
  * Kruijff-Korbayová & Steedman (2003) Ivana Kruijff-Korbayová and Mark Steedman.  Discourse and information structure.  _Journal of logic, language and information_ , 12(3):249–259, 2003.
  * Kulkarni et al. (2020) Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie.  Aquamuse: Automatically generating datasets for query-based multi-document summarization.  _arXiv preprint arXiv:2010.12694_ , 2020.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466, 2019.  doi: 10.1162/tacl˙a˙00276.  URL https://aclanthology.org/Q19-1026.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.243.  URL https://aclanthology.org/2021.emnlp-main.243.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 4582–4597, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.353.  URL https://aclanthology.org/2021.acl-long.353.
  * Lin (2004) Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://aclanthology.org/W04-1013.
  * Liu et al. (2022) Xiaochen Liu, Yu Bai, Jiawei Li, Yinan Hu, and Yang Gao.  Psp: Pre-trained soft prompts for few-shot abstractive summarization.  _arXiv preprint arXiv:2204.04413_ , 2022.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer.  AmbigQA: Answering ambiguous open-domain questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 5783–5797, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.466.  URL https://aclanthology.org/2020.emnlp-main.466.
  * Min et al. (2021) Sewon Min, Kenton Lee, Ming-Wei Chang, Kristina Toutanova, and Hannaneh Hajishirzi.  Joint passage ranking for diverse multi-answer retrieval.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 6997–7008, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.560.  URL https://aclanthology.org/2021.emnlp-main.560.
  * Narayan et al. (2022) Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Dipanjan Das, and Mirella Lapata.  Conditional generation with a question-answering blueprint.  _arXiv preprint arXiv:2207.00397_ , 2022.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pp. 2383–2392, Austin, Texas, November 2016. Association for Computational Linguistics.  doi: 10.18653/v1/D16-1264.  URL https://aclanthology.org/D16-1264.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 784–789, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-2124.  URL https://aclanthology.org/P18-2124.
  * Rashkin et al. (2021) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter.  Measuring attribution in natural language generation models.  _arXiv preprint arXiv:2112.12870_ , 2021.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266, 2019.  doi: 10.1162/tacl˙a˙00266.  URL https://aclanthology.org/Q19-1016.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer.  How much knowledge can you pack into the parameters of a language model?  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 5418–5426, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.437.  URL https://aclanthology.org/2020.emnlp-main.437.
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al.  Okapi at trec-3.  _Nist Special Publication Sp_ , 109:109, 1995.
  * Rodriguez et al. (2019) Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber.  Quizbowl: The case for incremental question answering, 2019.  URL https://arxiv.org/abs/1904.04792.
  * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant.  Learning to retrieve prompts for in-context learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2655–2671, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.191.  URL https://aclanthology.org/2022.naacl-main.191.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush.  Multitask prompted training enables zero-shot task generalization.  _CoRR_ , abs/2110.08207, 2021.  URL https://arxiv.org/abs/2110.08207.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 7881–7892, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.704.  URL https://aclanthology.org/2020.acl-main.704.
  * Snell et al. (2022) Charlie Snell, Dan Klein, and Ruiqi Zhong.  Learning by distilling context.  _arXiv preprint arXiv:2209.15189_ , 2022.
  * Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang.  Asqa: Factoid questions meet long-form answers.  _arXiv preprint arXiv:2204.06092_ , 2022.
  * Vallduví & Vilkuna (1998) Enric Vallduví and Maria Vilkuna.  On rheme and kontrast.  In _The limits of syntax_ , pp. 79–108. Brill, 1998.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners.  _CoRR_ , abs/2109.01652, 2021.  URL https://arxiv.org/abs/2109.01652.
  * Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2022a.  URL https://openreview.net/forum?id=gEZrGCozdqR.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022b.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SkeHuCVFDr.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev.  QMSum: A new benchmark for query-based multi-domain meeting summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 5905–5921, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.472.  URL https://aclanthology.org/2021.naacl-main.472.
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_ , 2022.

##  Appendix A Prompt Templates

This section presents the prompt templates we used in our experiments. Figures
1 2 show prompt templates with AF refinements for ASQA and AQuAMuSe,
respectively. When using no refinements, the Disambiguations part of the
template is removed. When using a different query refinement step, the same
part is replaced with a natural language explanation or a list of question
answers. Finally, Figure 3 shows the template used when the oracle
disambiguated questions are given.

I will provide ambiguous questions that have multiple answers about different
aspects of the question, and answer them in detail with at least two
sentences.
---
Question: Who sang it’s a long way to the top?
Disambiguations:
\- band: AC/DC
\- lead vocal: Bon Scott
Answer: "It’s a Long Way to the Top (If You Wanna Rock ’n’ Roll)" is a song by
Australian hard rock band AC/DC. This was a signature song for lead singer Bon
Scott. Brian Johnson, who replaced Scott as AC/DC’s lead vocalist after
Scott’s death in 1980, does not perform it, out of respect for his
predecessor.
Question:
Figure 1: Example ASQA Prompt with AF refinements and one exemplar. I will
provide questions that need to be elaborated to be answered fully, and will
answer them in detail with at least two sentences.
---
Question: where did the term shooting brake come from
Details:
\- how the term originated: as an early 19th century British term
\- what it was for: a vehicle used to carry shooting parties with their
equipment and game
\- etymology of the term brake: uncertain; initially a chassis used to break
in horses, used to describe a motorized vehicle
\- its possible origins: in the Dutch word ’brik’ which means ’cart’ or
’carriage’"
Answer: "Shooting-brake" originated as an early 19th century British term for
a vehicle used to carry shooting parties with their equipment and game. The
etymology of the term brake is uncertain; initially a chassis used to break in
horses, and subsequently used to describe a motorized vehicle. It is also
possible, that the word’ brake’ has its origins in the Dutch word’ brik’ which
means’ cart’ or’ carriage’.
Question:
Figure 2: Example AQuAMuSe Prompt with AF refinements and one exemplar.. I
will provide ambiguous questions that have multiple answers about different
aspects of the question, and answer them in detail with at least two
sentences.
---
Question: Who sang it’s a long way to the top?
Disambiguated Questions:
Q: Which band sang it’s a long way to the top?
Q: Who was the lead vocal of it’s a long way to the top?
Disambiguated Answers:
\- band: AC/DC
\- lead vocal: Bon Scott
Answer: "It’s a Long Way to the Top (If You Wanna Rock ’n’ Roll)" is a song by
Australian hard rock band AC/DC. This was a signature song for lead singer Bon
Scott. Brian Johnson, who replaced Scott as AC/DC’s lead vocalist after
Scott’s death in 1980, does not perform it, out of respect for his
predecessor.
Question:
Figure 3: Example ASQA Prompt with AF refinements, oracle disambiguated
questions, and one examplar.


11institutetext: Sanofi, Chilly Mazarin, France 22institutetext: Aix-Marseille
University, Marseille, France 33institutetext: Sanofi, Bridgewater, NJ, USA
44institutetext: Sanofi, Cambridge, MA, USA
44email<EMAIL_ADDRESS>
44email<EMAIL_ADDRESS>
44email<EMAIL_ADDRESS>
44email<EMAIL_ADDRESS>
# Image Translation Based Nuclei Segmentation for Immunohistochemistry Images

Roger Trullo 11 Quoc-Anh Bui 22 Qi Tang 33 Reza Olfati-Saber 44

###### Abstract

Numerous deep learning based methods have been developed for nuclei
segmentation for H&E images and have achieved close to human performance.
However, direct application of such methods to another modality of images,
such as Immunohistochemistry (IHC) images, may not achieve satisfactory
performance. Thus, we developed a Generative Adversarial Network (GAN) based
approach to translate an IHC image to an H&E image while preserving nuclei
location and morphology and then apply pre-trained nuclei segmentation models
to the virtual H&E image. We demonstrated that the proposed methods work
better than several baseline methods including direct application of state of
the art nuclei segmentation methods such as Cellpose and HoVer-Net, trained on
H&E and a generative method, DeepLIIF, using two public IHC image datasets.

###### Keywords:

GAN H&E IHC Nuclei Segmentation.

##  1 Introduction

H&E images are the most common modality of histopathology images since they
can be stained quickly, economically and significance amount of microscopic
anatomy is revealed. As a result, many nuclei segmentation methods have been
developed recently leveraging pathologist’s annotations and the advance in
computer vision for H&E images [19]. Besides H&E, there is another popular
type of image, the immunohistochemistry (IHC) image, which is commonly used to
identify specific protein biomarkers and is complementary to H&E images [16].
IHC images play a central role in companion diagnostic tools for development
of precision medicines. Several novel oncology therapies have been approved by
regulatory agencies with a companion diagnostic device based on IHC images. On
the other hand, the analysis of IHC images has been typically based on manual
semi-quantitative methods, such as the 20X rule, for clinical decision making,
which may suffer from subjectivity and inter and intra-rater variability.
Several case studies suggest that deep learning based digital pathology
algorithms may provide improved patient selection comparing to the current
clinical standard [9]. At the core of these deep learning algorithms, there is
a nuclei segmentation method, which relies on human expert annotations. To
avoid the labor intensive manual annotation on IHC images and also to improve
upon the performance of direct application of models trained on H&E images to
IHC images, we proposed a two-step label-free approach where an H&E image is
first translated into an IHC image utilizing unsupervised image-to-image
translation methods, then for the second step, we apply existing methods that
performed well on H&E images to virtually generated H&E images to obtain
nuclei segmentation masks (Fig. 1).

Figure 1: Proposed IHC Segmentation Pipeline.

##  2 Related Work

Numerous cell morphology operation methods, such as color deconvolution,
Otsu’s thresholding, watershed, have been used either separately or in
combination to segment nuclei in histology images [6]. With the advent of deep
learning, it has been shown that supervised methods can outperform traditional
cell morphology based methods [29] and traditional machine learning based
methods [13]. However, a drawback of these methods and the supervised learning
methods in general is that labor intensive manual annotation is required by
trained human experts [1]. To avoid manual annotations, unsupervised deep
learning methods have been developed [12, 4] for nuclei segmentation, however
these general methods were not tailored for IHC images. To develop nuclei
segmentation models for IHC images with few annotations, researchers have
leveraged the power of multiplex immunofluorescence (IF) staining to provide a
ground truth of nuclei masks for the IHC nuclei segmentation through co-
registration of IF images with IHC images [7] and then trained a conditional
generative adversarial network (cGAN) [15] to generate nuclei masks.

##  3 Methodology

###  3.1 Translation from IHC Images to H&E Images

Image-to-image translation have achieved significant advancements since the
advent of GAN. Application of GAN and its variations created a new field in
digital pathology named virtual staining, where one or more image modalities
can be virtually generated based on input of only one modality of image.
Feasibility of virtual staining has been demonstrated across multiple
histology image modalities [2, 22, 28, 30, 21] including H&E and IHC. When it
comes to translation between H&E and IHC, almost all literature focused on the
translation from H&E to IHC since H&E images were more commonly available for
patients. However, for our purpose of leveraging well trained nuclei
segmentation models for H&E images, we are interested in the reserve direction
of the translation, that is, from IHC to H&E. Due to the challenges of
registering IHC images with H&E images and high resolution of these types of
images, generative methods that do not require paired images, such as CycleGAN
[31] and U-GAT-IT [17], are methods of interest for the image translation
task. These methods work by training simultaneously two GANs (two generator
and two discriminator models). One generator translates IHC to H&E and other
generator translates from H&E to IHC. In addition, in order to have pixel to
pixel correspondence, a cycle consistency loss is introduced. This loss
enforces the idea that when using as input an IHC image, the first generator
will produce a virtual H&E image and if we use this virtual H&E image as input
to the second generator, the produced image should match the input IHC image.
On the other hand, U-GAT-IT extended the method by adding new normalization
layers and by using attention modules which was claimed to improve the quality
of the produced images. For both of these methods, we used the hyperparameters
recommended by their authors including loss weights and learning rates. For
CycleGAN we used a batch size of 10, whereas for U-GAT-IT we used a batch size
of 2. This is due to the fact that U-GAT-IT uses more memory since it has
fully connected layers. We trained CycleGAN for 30 epochs and U-GAT-IT for 20
epochs. The batch size was selected to use the full GPU memory available and
the number of epochs was selected empirically by looking qualitatively at the
generated virtual H&E tiles. We used an NVIDIA P100 GPU for training of the
image translation models, which took around 4 hours.

###  3.2 Nuclei Segmentation on Virtual H&E Images

Once virtual H&E images are generated, pretrained models that work well for
H&E images can be applied to the virtual H&E images to obtain the nuclei
segmentation masks. Since the image translation step kept the location and the
morphology of each nuclei from the IHC images, the virtual H&E images
perfectly matched with the IHC images at pixel level without the need of image
registration. Thus, the nuclei segmentation masks obtained from the virtual
H&E images can be directly used as the nuclei masks for the original IHC
images. Without loss of generality, popular H&E image nuclei segmentation
methods such as Cellpose [26], StarDist [24], and HoVer-Net [10], were
considered for this step. Cellpose was trained on a diverse set of datasets
including image sets BBBC038v1 and BBBC039v1 [3], image set MoNuSeg [18], and
image set ISBI 2009 [5]. The mixed dataset consists of microscopy images, H&E
images and fluorescence images, with about 1139 images for the task of nuclei
segmentation. StarDist was trained on two H&E image datasets, MoNuSeg 2018
training dataset, which has 30 H&E images and around 22,000 nuclear boundary
annotations from a diverse set of patients including breast cancer, kidney
cancer, lung cancer, prostate cancer, bladder cancer, colon cancer and stomach
cancer patients, and a TNBC dataset [20], which has 50 annotated H&E images
from patients with triple negative breast cancer. Regarding HoVer-Net, it was
trained on 41 H&E stained colorectal adenocarcinoma image tiles, containing
24,319 exhaustively annotated nuclei with associated class labels.

##  4 Experiments

We systematically evaluated both the image translation component and the
nuclei segmentation component of the proposed approach to examine the impact
of each component in comparison to two different types of baseline methods,
the direct application of the nuclei segmentation models trained on H&E images
to IHC images and an pre-trained nuclei segmentation model tailored for IHC
images, DeepLIIF [7]. For all the experiments we used a P100 Nvidia GPU.

###  4.1 Datasets

The image translation component was trained based on an in-house IHC/H&E
dataset. The nuclei segmentation component was tested and compared on two
different IHC image datasets, the DeepLIIF testing dataset [7] and the LYON19
dataset [27].

####  4.1.1 In-house IHC/H&E Dataset

To train the generative model that transforms IHC images onto virtual H&E
images we use an in-house dataset, that consists of 123 IHC whole slide images
and 121 H&E whole slide images. The IHC images were stained to highlight cells
expressing a protein target, which cannot be disclosed due to confidentiality
reasons, however, it is shown in Fig. 2 that the generative model trained on
this specific type of IHC dataset generalized well to other protein targets,
such as CD3/CD8 and Ki67. For each of these slides, we randomly sample patches
of size $256\times 256$ and ended up with 2510 patches from IHC images and
2793 patches from H&E images.

####  4.1.2 DeepLIIF Testing Dataset

A public test set [7], which can be downloaded from
https://zenodo.org/record/5553268, it includes 598 Ki67 IHC images of size
512×512 and 40x magnification from bladder carcinoma and non-small cell lung
carcinoma slides. The expression of Ki67 is strongly associated with tumor
cell proliferation and growth, and is widely used in routine pathological
investigation as a proliferation marker.

####  4.1.3 LYON19 Dataset

Public testing set of LYON19 [27], which can be downloaded from
https://zenodo.org/record/3385420, it contains 441 Regions of Interest (ROIs)
from whole slide images (WSIs) of CD3/CD8 stained IHC images of lymphocytes
from breast, colon, and prostate cancer patients’ biopsy specimens. The 441
ROIs were selected from IHC images with 277 regular area ROIs, 59 clustered
cell ROIs, and 105 artifact area ROIs. The ground truth of this dataset was
not disclosed by the Grand Challenge competition. However, performance on this
testing set can be obtained after submitting the predictions to the challenge
here https://lyon19.grand-challenge.org/Submission/. The objective of the
competition is to detect positive stained cells; in particular, the center of
each positive cell. To accommodate for this, we computed our regular pipeline
and then applied a simple thresholding procedure based on the HSI color space
to classify cells as positively stained. We used the open source
implementation provided by HistomicsTK [11]. Finally, we computed the centroid
of each of those cells.

###  4.2 Baseline Methods and Evaluation Metrics

We compared the proposed methods with four baseline models: the pre-trained
Cellpose model v2.0.5 [25], the pre-trained 2D_versatile_he StarDist model
v0.8.2 [23], the pre-trained HoVer-Net model v1.0 [14], and the pre-trained
DeepLIIF model v1.1.2 [8].

The performance of all the methods will be evaluated based on Dice score,
which measures pixel level segmentation performance. We also evaluated cell
instance level performance metrics including accuracy, precision, recall and
F1 score, conditional on a given Intersection over Union (IoU) threshold.
Since the concept of true negatives in instance cell level detection is not
valid, accuracy is computed from the number of true positives, TP, false
positives, FP and false negatives, FN, as
accuracy$=\mbox{TP}/(\mbox{TP}+\mbox{FP}+\mbox{FN})$ as it is commonly done in
object detection.

###  4.3 Results

First, the proposed methods were compared against the baseline methods on the
DeepLIIF testing dataset. CycleGAN plus Cellpose achieved the best results in
terms of Dice score and precision at IoU$=0.5$ as shown in Table 1. By
translating IHC to H&E images using GAN based methods, such as CycleGAN and
U-GAT-IT, the performance of Cellpose and HoVer-Net improved at least 0.1 in
Dice score comparing to directly application of these methods to IHC images.
However, the combination of GAN based methods with StarDist failed to achieve
improvement and even lead to worse performance comparing to StarDist alone. We
also observed that CycleGAN performed better than or similarly to U-GAT-IT
when combined with StarDist, Cellpose or HoVer-Net. By changing the thresholds
for IoU, the accuracy curves in Fig. 3 confirm that the same conclusion holds
against different thresholds.

Table 1: Performance of proposed methods against baseline methods on DeepLIIF testing dataset. Cell instance level segmentation accuracy, precision, recall and F1 score were evaluated at $\mbox{IoU}=0.5$. Method | Dice Score | Accuracy | Precision | Recall | F1 Score
---|---|---|---|---|---
DeepLIIF | 0.66 | 0.20 | 0.31 | 0.37 | 0.34
StarDist | 0.64 | 0.33 | 0.54 | 0.45 | 0.49
CycleGAN+StarDist | 0.59 | 0.27 | 0.49 | 0.38 | 0.42
U-GAT-IT+StarDist | 0.50 | 0.18 | 0.34 | 0.28 | 0.31
Cellpose | 0.57 | 0.27 | 0.58 | 0.33 | 0.42
CycleGAN+Cellpose | 0.72 | 0.38 | 0.63 | 0.49 | 0.55
U-GAT-IT+Cellpose | 0.67 | 0.28 | 0.47 | 0.42 | 0.44
HoVer-Net | 0.42 | 0.20 | 0.64 | 0.23 | 0.33
CycleGAN+HoVer-Net | 0.68 | 0.34 | 0.62 | 0.42 | 0.44
U-GAT-IT+HoVer-Net | 0.68 | 0.34 | 0.59 | 0.44 | 0.50

Fig. 2 visualizes cell nuclei segmentation results of four selected tiles.
From the top to the bottom are the input IHC, the virtually generated H&E
based on CycleGAN, the ground truth, which was only available for the
visualized tiles, and the cell masks of all the methods previously presented.
Pixel-wise, the true positive (TP), false positive (FP) and false negative
(FN) are represented by blue, red and green color respectively. The first row
presents a testing Ki67 IHC image from the DeepLIIF testing set whereas the
next 3 are the CD3/CD8 IHC images from the LYON19 dataset. For the Ki67 IHC
image, the proposed methods perform similarly to DeepLIIF since DeepLIIF was
trained specifically for this type of staining. However, for the CD3/CD8 IHC
images, the proposed methods outperformed DeepLIIF and the direct application
of Cellpose and HoVer-Net models pretrained on H&E images.

Figure 2: Presentation of the proposed and the baseline methods’ performance
in Ki67 IHC images from DeepLIIF testing dataset and CD3/CD8 IHC images from
LYON19 dataset.

Next, we examined the performance of these methods on the LYON19 dataset. By
submitting the model predictions to the Grand Challenge competition website,
the F1 score were obtained and were tabulated in Table 2. Only CycleGAN was
used for the image translation step since it has better performance than
U-GAT-IT when tested in the DeepLIIF testing dataset. The proposed method
performed similarly as the baseline methods except the combination of CycleGAN
with HoVer-Net method, which lead to 4% higher F1 score in all detections, 9%
higher F1 score in clusterd cells and 8% higher F1 score in artifact areas
comparing to the DeepLIIF method. When compared against the HoVer-Net itself,
the improvement in F1 score is much larger with at least 20% higher F1 scores
across all categories.

Figure 3: Segmentation accuracy curves for proposed methods and baseline methods when IoU changes from 0.5 to 1 with a step size of 0.05. Table 2: F1 scores for different types of segmentation tasks of proposed methods against baseline methods on LYON19 dataset. Method | All Detections | Regular Areas | Clustered Cells | Artifact Areas
---|---|---|---|---
DeepLIIF | 0.53 | 0.64 | 0.59 | 0.17
StarDist | 0.54 | 0.59 | 0.63 | 0.22
CycleGAN+StarDist | 0.54 | 0.59 | 0.63 | 0.22
Cellpose | 0.54 | 0.53 | 0.68 | 0.24
CycleGAN+Cellpose | 0.54 | 0.61 | 0.67 | 0.20
HoVer-Net | 0.04 | 0.06 | 0.01 | 0.03
CycleGAN+HoVer-Net | 0.58 | 0.63 | 0.68 | 0.25

##  5 Conclusions

We leveraged publicly available pre-trained nuclei segmentation models based
on H&E images to perform nuclei segmentation in IHC images without requiring
any manual anotation on them. This was achieved by virtually translating IHC
images into H&E images. To enable such translation, we trained image
translation models, such as CycleGAN and U-GAT-IT models, based on an in-house
dataset of IHC and H&E images. This approach achieved better performance than
several of the baseline methods, including direct application of pre-trained
models based on H&E images, such as Cellpose and HoVer-Net, to IHC images, and
a pre-trained model tailored for IHC image nuclei segmentation, DeepLIIF, when
tested on DeepLIIF testing dataset. Such improvement was less pronounced in
the LYON19 dataset except for CycleGAN combined with HoVer-Net. An interesting
finding is that HoVer-Net alone performed badly on LYON19 dataset however when
combined with CycleGAN, it became the best performing method. These
observations together with the findings that the combination of CycleGAN or
U-GAT-IT with the StarDist method failed to achieve improved performance in
both testing datasets suggest that if the pre-trained nuclei segmentation
method for H&E images has strong generalizability to IHC images, translating
IHC to H&E images may not lead to improved performance, such as the case for
StarDist, however, on the other hand, if the pre-trained method utilized
features specific to H&E images, image translation can lead to substantial
improvement. Thus, adoption of the proposed method depends on the
generalizability of the pre-trained models to the target image modality. Our
work has been based completely on label-free IHC images. An interesting
perspective is to combine our method with a few manual annotations in a semi-
supervised learning setting which we believe can potentially improve the
performance even further.

## References

  * [1] Abdolhoseini, M., Kluge, M.G., Walker, F.R., Johnson, S.J.: Segmentation of heavily clustered nuclei from histopathological images. Scientific reports 9(1), 1–13 (2019)
  * [2] Bayramoglu, N., Kaakinen, M., Eklund, L., Heikkila, J.: Towards virtual h&e staining of hyperspectral lung histology images using conditional generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 64–71 (2017)
  * [3] Caicedo, J.C., Goodman, A., Karhohs, K.W., Cimini, B.A., Ackerman, J., Haghighi, M., Heng, C., Becker, T., Doan, M., McQuin, C., et al.: Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nature methods 16(12), 1247–1253 (2019)
  * [4] Chen, M., Artières, T., Denoyer, L.: Unsupervised object segmentation by redrawing. Advances in neural information processing systems 32 (2019)
  * [5] Coelho, L.P., Shariff, A., Murphy, R.F.: Nuclear segmentation in microscope cell images: a hand-segmented dataset and comparison of algorithms. In: 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. pp. 518–521. IEEE (2009)
  * [6] Di Cataldo, S., Ficarra, E., Acquaviva, A., Macii, E.: Automated segmentation of tissue images for computerized ihc analysis. Computer methods and programs in biomedicine 100(1), 1–15 (2010)
  * [7] Ghahremani, P., Li, Y., Kaufman, A., Vanguri, R., Greenwald, N., Angelo, M., Hollmann, T.J., Nadeem, S.: Deep learning-inferred multiplex immunofluorescence for immunohistochemical image quantification. Nature Machine Intelligence (2022)
  * [8] Ghahremani, P., Nadeem, S., Marino, J., Wang, W., Dodds, R., McCalmont, D., Nieto, R.: Deepliif. https://github.com/nadeemlab/DeepLIIF (2022)
  * [9] Glass, B., Stanford-Moore, S.A., Meghwal, D., Agrawal, N., Lin, M., Hedvat, C., Lee, G., Ely, S., Montalto, M., Wapinski, I., et al.: 821 machine learning models can quantify cd8 positivity in lymphocytes in melanoma clinical trial samples. Journal for ImmunoTherapy of Cancer 9(Suppl 2), A859–A859 (2021)
  * [10] Graham, S., Vu, Q.D., Raza, S.E.A., Azam, A., Tsang, Y.W., Kwak, J.T., Rajpoot, N.: Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis 58, 101563 (2019)
  * [11] Gutman, D.A., Khalilia, M., Lee, S., Nalisnik, M., Mullen, Z., Beezley, J., Chittajallu, D.R., Manthey, D., Cooper, L.A.: The Digital Slide Archive: A Software Platform for Management, Integration, and Analysis of Histology for Cancer Research. Cancer Research 77(21), e75–e78 (10 2017). https://doi.org/10.1158/0008-5472.CAN-17-0629
  * [12] Han, L., Yin, Z.: Unsupervised network learning for cell segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 282–292. Springer (2021)
  * [13] Hatipoglu, N., Bilgin, G.: Cell segmentation in histopathological images with deep learning algorithms by utilizing spatial relationships. Medical & biological engineering & computing 55(10), 1829–1848 (2017)
  * [14] Ho, M.Y., Chapman, V., Ali, Z., Graham, S., Vu, Q.D.: Hover-net. https://github.com/vqdang/hover_net (2022)
  * [15] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)
  * [16] Kaplan, K.: Quantifying ihc data from whole slide images is paving the way toward personalized medicine. MLO Med Lab Obs 47, 20–21 (2015)
  * [17] Kim, J., Kim, M., Kang, H., Lee, K.H.: U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. In: International Conference on Learning Representations (2020), https://openreview.net/forum?id=BJlZ5ySKPH
  * [18] Kumar, N., Verma, R., Sharma, S., Bhargava, S., Vahadane, A., Sethi, A.: A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE transactions on medical imaging 36(7), 1550–1560 (2017)
  * [19] Mahbod, A., Schaefer, G., Bancher, B., Löw, C., Dorffner, G., Ecker, R., Ellinger, I.: Cryonuseg: A dataset for nuclei instance segmentation of cryosectioned h&e-stained histological images. Computers in Biology and Medicine 132, 104349 (2021)
  * [20] Naylor, P., Laé, M., Reyal, F., Walter, T.: Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE transactions on medical imaging 38(2), 448–459 (2018)
  * [21] Rivenson, Y., de Haan, K., Wallace, W.D., Ozcan, A.: Emerging advances to transform histopathology using virtual staining. BME Frontiers 2020 (2020)
  * [22] Rivenson, Y., Wang, H., Wei, Z., de Haan, K., Zhang, Y., Wu, Y., Günaydın, H., Zuckerman, J.E., Chong, T., Sisk, A.E., et al.: Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nature biomedical engineering 3(6), 466–477 (2019)
  * [23] Schmidt, U., Weigert, M.: stardist. https://github.com/stardist/stardist (2022)
  * [24] Schmidt, U., Weigert, M., Broaddus, C., Myers, G.: Cell detection with star-convex polygons. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II. pp. 265–273 (2018). https://doi.org/10.1007/978-3-030-00934-2_30
  * [25] Stringer, C.: cellpose. https://github.com/MouseLand/cellpose (2022)
  * [26] Stringer, C., Wang, T., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algorithm for cellular segmentation. Nature methods 18(1), 100–106 (2021)
  * [27] Swiderska-Chadaj, Z., Pinckaers, H., van Rijthoven, M., Balkenhol, M., Melnikova, M., Geessink, O., Manson, Q., Sherman, M., Polonia, A., Parry, J., et al.: Learning to detect lymphocytes in immunohistochemistry with deep learning. Medical image analysis 58, 101547 (2019)
  * [28] Xu, Z., Moro, C.F., Bozóky, B., Zhang, Q.: Gan-based virtual re-staining: a promising solution for whole slide image analysis. arXiv preprint arXiv:1901.04059 (2019)
  * [29] Yang, L., Ghosh, R.P., Franklin, J.M., Chen, S., You, C., Narayan, R.R., Melcher, M.L., Liphardt, J.T.: Nuset: A deep learning tool for reliably separating and analyzing crowded cells. PLoS computational biology 16(9), e1008193 (2020)
  * [30] Zhang, Y., de Haan, K., Rivenson, Y., Li, J., Delis, A., Ozcan, A.: Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light: Science & Applications 9(1), 1–13 (2020)
  * [31] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)