#  BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives

Xiaoyue Wang Jianyou Wang11footnotemark: 1 Weili Cao11footnotemark: 1 Kaicheng
Wang11footnotemark: 1
Ramamohan Paturi Leon Bergen22footnotemark: 2

Laboratory for Emerging Intelligence
University of California, San Diego
{xiw027, jiw101, w2cao, kaw036, rpaturi<EMAIL_ADDRESS>Equal
contributionEqual Senior Authors

###### Abstract

We present the Benchmark of Information Retrieval (IR) tasks with Complex
Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve
documents given multi-faceted user objectives. The benchmark’s complexity and
compact size make it suitable for evaluating large language model (LLM)-based
information retrieval systems. We present a modular framework for
investigating factors that may influence LLM performance on retrieval tasks,
and identify a simple baseline model which matches or outperforms existing
approaches and more complex alternatives. No approach achieves satisfactory
performance on all benchmark tasks, suggesting that stronger models and new
retrieval protocols are necessary to address complex user needs.
111https://github.com/BIRCO-benchmark/BIRCO

BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives

Xiaoyue Wang ††thanks: Equal contribution Jianyou Wang11footnotemark: 1 Weili
Cao11footnotemark: 1 Kaicheng Wang11footnotemark: 1 Ramamohan Paturi ††thanks:
Equal Senior Authors Leon Bergen22footnotemark: 2 Laboratory for Emerging
Intelligence University of California, San Diego {xiw027, jiw101, w2cao,
kaw036, rpaturi<EMAIL_ADDRESS>

##  1 Introduction

Information retrieval (IR) tasks have traditionally been centered around
matching queries with semantically similar passages. However, user objectives
may go significantly beyond retrieving based on similarity. As a motivating
example, consider a user who wants to find papers that refute a particular
scientific claim. This would not be well-captured by similarity-driven search,
which would also retrieve papers that support the claim. In addition, the user
may have multiple objectives in their search. They may be searching for papers
that measure the response of a drug in a specific population, using a certain
set of measurements.

We propose the BIRCO benchmark for evaluating the performance of IR systems on
tasks with _complex objectives_. We curate 5 open-source datasets (DORIS-MAE
Wang et al. (2023), ArguAna Wachsmuth et al. (2018), WhatsThatBook Lin et al.
(2023), Clinical-Trial Koopman and Zuccon (2016), and RELIC Thai et al.
(2022)), which contain paragraph-length queries with multi-faceted task
objectives. This represents a challenging test bed for methods that aim to
address complex user search needs.

IR systems fall into three primary categories: pre-trained embedding models,
language models that have been fine-tuned for IR tasks, and task-agnostic
models based on Large Language Models (LLMs) like GPT4.

BIRCO provides a benchmark for LLM-based retrieval systems. The scale and
capabilities of these models present a unique set of challenges for IR
benchmarks. Due to their massive pretraining, LLMs may be able to answer many
user queries without examining the corresponding documents. This decreases the
validity of these tasks for measuring LLM retrieval performance on novel
queries and document sets. In addition, IR tasks may contain thousands of
documents per query, making it cost-prohibitive to evaluate LLM systems.

BIRCO is designed to address both of these challenges. The complexity of the
search objectives, combined with aggressive filtering, makes it impossible for
LLMs such as GPT4 to answer BIRCO queries without examining the corpus
documents. BIRCO also remains challenging for LLMs despite having only 50-100
documents per query, making it low-cost to evaluate LLM performance.

##  2 Related Work

Figure 1: BIRCO contains 5 IR tasks with complex objectives

IR Benchmarks
IR benchmarks such as MS MARCO Nguyen et al. (2016), NQ Kwiatkowski et al.
(2019), LOTTE Santhanam et al. (2022), BEIR Thakur et al. (2021), MIRACL Zhang
et al. (2023), MTEB Muennighoff et al. (2022), and BERRI Asai et al. (2023)
consist mostly of sentence-level queries, and their task objectives, while
varying to some degree, focus on finding semantically similar passages, with
one exception: the ArguAna dataset Wachsmuth et al. (2018), which is a
counterargument retrieval task.
Complex Query IR Tasks
Several recent datasets (DORIS-MAE, WhatsThatBook, Clinical-Trial, RELIC) pose
more complex retrieval tasks Wang et al. (2023); Lin et al. (2023); Koopman
and Zuccon (2016); Thai et al. (2022). In these datasets, the queries are
paragraph-length, and passages should match the queries along multiple
dimensions. See Figure 1.
Specialized retrieval models
Pretrained Greene et al. (2022); Wang et al. (2022); Gao et al. (2021) and
fine-tuned Liu et al. (2023); Chuang et al. (2022); Dai et al. (2022); Gao et
al. (2023); Ferraretto et al. (2023); Pan et al. (2023) embedding models have
formed the core of most IR systems due to their speed and simplicity. More
recently, there have been methods for fine-tuning language models for ranking
and retrieval, including monoT5 Nogueira et al. (2020) and RankLLaMA Ma et al.
(2023a). TART Asai et al. (2023) and INSTRUCTOR Su et al. (2023) are trained
to follow task-specific instructions during retrieval.
LLM-based IR systems
Recent research has shown that LLMs can be effectively used for the re-ranking
stage of IR. Sachan et al. (2022); Liang et al. (2022) compute a relevance
score with output logits, Qin et al. (2023) use pairwise comparison among
passages with open-source LLMs, Sun et al. (2023); Ma et al. (2023b) use list-
wise comparisons, and Zhuang et al. (2023) have the LLM assign a 4-way label
to each query/passage pair. These methods have primarily been evaluated on
sentence-level queries. Benchmarks for LLMs
Recent LLMs are evaluated across various datasets for specific skills. Big-
Bench Hard Suzgun et al. (2022) assesses model’s abilities in handling
challenging tasks that require multi-step reasoning. DROP Dua et al. (2019)
evaluates capabilities in reading comprehension, while HellaSwag Zellers et
al. (2019) examines commonsense reasoning skills. Meanwhile, MMLU Hendrycks et
al. (2020) employs multiple-choice problems to analyze LLMs’ problem-solving
skills across a wide range of subjects. To the best of our knowledge, no IR
benchmark designed to evaluate LLM-based IR system is released or documented.

##  3 A Benchmark for LLM-based IR Systems

Dataset |  Domain |  Task Objective |  Relevancy | Dev | Test
---|---|---|---|---|---
Num. of Triplets |  Num. of Queries |  Num. of Docs |  Relevant Doc. per Query |  Query Length |  Doc Length |  Lexical Overlap
BIRCO
DORIS-MAE |  AI/ML |  Retrieve relevant abstracts for the research question. |  Multi-level |  3k |  60 |  100 |  18.2 |  166.7 |  199.5 |  0.147
ArguAna |  Debate |  Retrieve the most effective argument that counters the query. |  Binary |  19k |  100 |  100 |  1 |  213.0 |  212.3 |  0.139
WhatsThatBook |  Literature |  Find the book from user’s ambiguous descriptions. |  Binary |  3k |  100 |  100 |  1 |  173.6 |  219.5 |  0.049
Clinical-Trial |  BioMed |  Match patient records with suitable clinical trials. |  3-level |  1k |  50 |  100 |  20.9 |  89.7 |  197.9 |  0.048
RELIC |  Literature |  Recover the masked quotation in a literary analysis. |  Binary |  68k |  100 |  100 |  1 |  198.7 |  96.3 |  0.031
BIRCO Avg. |  - |  - |  - |  20k |  51.3 |  100 |  8.4 |  168.3 |  185.1 |  0.083
BEIR Avg. |  - |  - |  - |  - |  3710.9 |  2417k |  43.5 |  9.8 |  124.9 |  0.384
Table 1: BIRCO Dataset Statistics and Comparison with BEIR Statistics. Relevant Doc. per Query means the number of relevant passages per query. Lexical overlap is calculated as the number of overlapping monograms between query and its relevant passage(s) without counting stopwords. Num of Triplets is the number of triplets extracted for the dev set. See Appendix D for details of triplet extraction. | MS MARCO | BIRCO |  |  | NQ | BIRCO |  |  | BEIR | BIRCO
---|---|---|---|---|---|---|---|---|---|---
Models | MRR@10 | MRR@10 |  | Models | R@20 | R@20 |  | Models | nDCG@10 | nDCG@10
Max possible | 100 | 100 |  | Max possible | 100 | 94.3 |  | Max possible | 100 | 100
ANCEFirstP | 33.0 | 20.0 |  | ANCEFirstP | 81.9 | 49.6 |  | E5-L-v2 | 50.0 | 38.52
SimLM | 41.1 | 18.1 |  | SimLM | 85.2 | 44.6 |  | RankLLaMA | 56.6 | 47.42
SPLADE-v2 | 36.8 | 17.7 |  | BM25 | 59.1 | 33.5 |  | TART | 44.8 | 39.48
Table 2: Model performance on BIRCO and other IR datasets. Models and metrics
are chosen based on availability of data in the published literature. Max
possible means the maximum possible value for a metric on a dataset. Figure 2:
An example query from DORIS-MAE Wang et al. (2023) with multiple facets.

BIRCO is designed to evaluate LLM-based IR Systems. It has several features
that distinguish it from traditional IR benchmarks: longer query lengths,
multiple facets within each query, complex and varied task objectives (beyond
semantic similarity), minimal contamination with respect to external
knowledge, increased task difficulty, and a smaller candidate pool to reduce
the cost of LLM-based evaluations.

###  3.1 Dataset Descriptions

See Figure 1 for an overview of the datasets.
DORIS-MAE Wang et al. (2023)
60 queries that are complex research questions from computer scientists. The
query communicates specific requirements from research papers. Candidate pools
have approximately 110 documents.
ArguAna Wachsmuth et al. (2018)
100 queries, each with a candidate pool of around 50 passages. Queries and
passages are both complex one-paragraph arguments about current affairs. The
objective is to find matching counterarguments.
WhatsThatBook Lin et al. (2023)
100 queries, with each query describing a book in an ambiguous manner. Each
query has a pool of 50 passages, which are book descriptions.
Clinical-Trial Koopman and Zuccon (2016)
100 queries that are paragraph-length patient case reports. Each query has a
candidate pool comprising 30-110 passages that are paragraph-length
descriptions of clinical trials. The objective is to find the most suitable
clinical trial for a patient.
RELIC Thai et al. (2022)
100 queries which are excerpts from scholars analyzing classic English-
language literature. Passages are sentences from a novel that have been
extracted from the queries. The objective is to match a literary analysis with
its missing quotations. See Appendix K for examples of each dataset.

###  3.2 BIRCO Evaluation Methodology

BIRCO is constructed to allow for statistically valid evaluation of model
performance. Four of the five benchmark datasets do not have previously
defined development/test set splits. We, therefore, define splits for these
datasets, ensuring that there is no overlap between queries or passages across
the splits. Since we prioritize the construction of test set, and to ensure no
overlap between passages in test set and development set, we remove all
passages in development set candidate pools that appear in the test set.
Details of development sets can be found in Appendix D.

See Table 1 for general statistics comparing BIRCO with BEIR benchmark
datasets.

###  3.3 Data Decontamination

Due to the massive scale of LLM pretraining, there is a risk of data
contamination when evaluating LLM-based systems on IR benchmarks. This will
lead to inflated estimates of their performance on novel retrieval tasks. We
first measure the degree of contamination on existing tasks on BEIR, and then
describe the decontamination procedure used for BIRCO.

####  3.3.1 Measuring contamination

We measured contamination in the prior benchmark BEIR by evaluating whether an
LLM is able to correctly answer a query without access to the pool of
candidate documents. If this occurs, then this transforms the retrieval task,
and means that it no longer measures the model’s ability to compare queries
and documents.

GPT4 was prompted with a query from BEIR (see prompts in Appendix L.5), and
its response was then compared with the ground-truth answers provided in the
dataset.222In some cases, the dataset provides multiple ground truth answers.
When this occurred, the LLM-generated answer was compared with the ground
truth answer with the highest similarity, as measured by the ada-002 embedding
model. The responses from GPT4 were binned into three categories: GPT4 agreed
with the ground-truth answer (Category 0); GPT4 and the ground-truth answer
disagreed, but GPT4’s answer was also correct (Category 1); GPT4 and the
ground-truth answer disagreed, and the ground-truth answer was better than
GPT4’s (Category 2).

The experiment was performed on 10 datasets from BEIR: MS MARCO Nguyen et al.
(2016), NQ Kwiatkowski et al. (2019), HotpotQA Yang et al. (2018), FiQA Maia
et al. (2018), DBPedia Hasibi et al. (2017), TREC-COVID Voorhees et al.
(2021), FEVER Thorne et al. (2018), Climate-FEVER Diggelmann et al. (2020),
Touché-2020 Bondarenko et al. (2020), and SciFact Wadden et al. (2020). For
each dataset, 100 queries were randomly sampled333TREC-COVID and Touché only
have 50 queries, so all queries were used for this dataset., and two human
annotators performed the labeling.

In order to evaluate the reliability of the labeling process, we measured
inter-annotator agreement on a separate set of 100 queries from these 10
datasets. Labels from the annotators exactly matched 91% of the time, had a
Spearman correlation of 0.81, and had a macro F1-score of 0.76, indicating
strong agreement between the annotators.

Table 3 shows that GPT4 is able to answer a large fraction of queries
directly, without access to candidate documents. This suggests that data
contamination is a challenge for existing benchmarks aiming to evaluate LLM
performance on information retrieval tasks.

####  3.3.2 Reducing contamination

In order to address the potential for similar contamination in BIRCO, we
designed a procedure to remove all queries that the LLM could potentially
answer without access to candidate passages. GPT4 was prompted with a query as
well as task-specific instructions. For example, for Clinical-Trial, the model
was given a description of a patient, and was instructed to generate a
clinical trial that would be appropriate for the patient. A query was filtered
when the GPT-generated passage was similar to the ground-truth passage. See
Appendix I for the filtering criteria for each task.

The filtering process removed 7% of queries from Clinical-Trial; 8% from
RELIC; 56% from WhatsThatBook; and none from DORIS-MAE and ArguAna. The
WhatsThatBook task is retrieving literary novels, a topic that GPT4 has
extensive knowledge of, leading to a high filtering rate. In contrast, the
DORIS-MAE task is to retrieve specific computer science articles, which
appears to be too fine-grained for GPT4 to answer without a candidate pool.
Similarly, for ArguAna, the task is to retrieve counterarguments given an
argument. These counterarguments typically include detailed factual
information which GPT4 cannot generate given the argument alone.

Dataset/Level | C0 | C1 | C2
---|---|---|---
MS MARCO | 76 | 18 | 6
TREC-COVID | 47 | 2 | 1
NQ | 89 | 7 | 4
HotpotQA | 60 | 20 | 20
FiQA | 73 | 19 | 8

Dataset/Level | C0 | C1 | C2
---|---|---|---
DBPedia | 78 | 12 | 10
FEVER | 99 | 0 | 1
Climate-FEVER | 81 | 18 | 1
Touché-2020 | 47 | 1 | 0
SciFact | 71 | 25 | 4

Table 3: Data contamination in BEIR. GPT4 answers the query without access to
any candidate passages. C0,1,2 corresponds to Category 0,1,2. C0 means
contamination. Numbers are %.

###  3.4 Query Complexity

Dataset | Average # Facets | Max | Min
---|---|---|---
DORIS-MAE | 5.6 | 9 | 3
ArguAna | 5.8 | 11 | 2
WhatsThatBook | 11.3 | 25 | 5
Clinical-trial | 7.4 | 30 | 1
RELIC | 2.0 | 2 | 1
Table 4: BIRCO queries contain multiple facets. Facets represent semantic
components in DORIS-MAE, arguments in ArguAna, book details in WhatsThatBook,
patient symptoms in Clinical-Trial, and contrasts in the preceding and
subsequent contexts in RELIC.

Table 1 shows that queries in BIRCO are substantially longer than those in
BEIR. In addition to their length, the queries in BIRCO are multi-faceted,
meaning that relevant passages must satisfy multiple requirements given by the
query. We quantify the number of facets per query by prompting GPT4 to break
the queries into their distinct components. See Figure 2 for an illustration
of breaking a DORIS-MAE query into 4 facets. Appendix F provides more details
about the prompting procedure.

Table 4 shows the results of the facet analysis. The datasets in BIRCO have
between 2 and 11 facets per query on average. IR systems must, therefore,
assess documents along multiple dimensions when determining relevance.

###  3.5 Task Complexity

We use several approaches to quantify the difficulty of BIRCO. Table 1 shows
that there is low lexical overlap between queries and relevant documents in
BIRCO, making it difficult for lexical matching heuristics to succeed on the
benchmark.

IR systems also achieve lower performance on BIRCO compared to other IR
benchmarks, as shown in Table 2. It is difficult to directly compare metrics
across datasets, as metrics can be influenced by the number of relevant
passages per query as well as the relevance scale (e.g. binary, multi-level).
Table 2 uses metrics MRR@10, R@20 and nDCG@10, all of which can reach high
percentages on their respective datasets. See Appendix E.

###  3.6 Domain Diversity

BIRCO consists of datasets from diverse domains, including debate, computer
science, biomedicine, and literature. To evaluate this diversity in terms of
word overlap between each pair of datasets, we use pairwise weighted Jaccard
similarity Ioffe (2010). Figure 4 shows low word overlap between these
datasets. The most similar datasets are RELIC and WhatsThatBook, which focus
on classic English literature and contemporary novels, respectively.

###  3.7 Hard Negatives in BIRCO

Figure 3: Jaccard similarity between pairs of datasets from BIRCO

Figure 4: Passage embeddings for a randomly selected query from ArguAna.
Wachsmuth et al. (2018)

Due to their construction, two of the datasets in BIRCO contain hard
negatives, defined as non-relevant candidate documents which share many
factors in common with ground truth answers. In ArguAna, arguments and
counterarguments are paragraphs which have been sampled from longer debates.
Hard negatives for ArguAna are defined as candidate passages which have been
extracted from the same debate as the ground truth counterargument. In
Clinical-Trial, hard negatives are non-relevant clinical trials which also
match the patient’s disease. 22% of the passages from ArguAna are hard
negatives, and 24% percent from Clinical-Trial.

To illustrate the effect of hard negatives, we use UMAP McInnes et al. (2018)
to visualize the ada-002 embeddings of queries and their candidate pools.
Figure 4 shows a randomly sampled query from ArguAna, where its ground-truth
passage is surrounded by hard negative passages.

###  3.8 Candidate Pool Construction

It is prohibitively expensive to evaluate LLMs on the entire set of passages
for each query. We therefore define candidate pools for each query,
restricting LLM search to these smaller pools. This is standard for many other
IR tasks, where it is known as the passage re-ranking stage.

We construct candidate pools for each query using the lexicon-matching
algorithm BM25 Trotman et al. (2014) and state-of-the-art text embedding model
ada-002. Each query has a candidate pool of 50 passages. (The ground-truth
passage is inserted when necessary.) As shown in Appendix C and Table 9, the
difficulty is comparable to the original datasets for four of five tasks, and
remains challenging for the fifth.

##  4 A Framework for LLM-based Retrieval

This section presents a framework for using LLMs for information retrieval
tasks. The framework is modular, allowing us to investigate the effect of
different factors on model performance. See an illustration of the framework
in Appendix G.

###  4.1 Ranking vs. Scoring

LLMs can find relevant passages by either ranking them comparatively or by
directly scoring them. RankGPT4 Sun et al. (2023) puts a list of passages in
the LLM prompt, and iteratively filters for the most relevant passages using a
sliding window approach. LLMs can also score passages one at a time. For this
direct scoring approach, the LLM is prompted to generate a numerical score
from 0-10.

When passages are directly scored by the LLM, there can be ties in which
several passages receive the same score. By default, we use E5-Large-v2, an
embedding model, for tie-breaking.

###  4.2 Chain-of-Thought Reasoning

To investigate the role of natural language reasoning in complex query
retrieval, we compare two approaches to scoring a query/passage pair. As shown
in Figure 5, the first approach (shown in the middle of Figure 5, Reason+O) is
to generate a set of decision criteria for judging whether a query is relevant
to a passage. The LLM is instructed to follow its own decision criteria step-
by-step before producing a final score. An example set of decision criteria
can be seen in Figure 6. Another approach (top of Figure 5, Score+O) is to
directly produce a score given the query and passage, without any reasoning.
The detailed prompt structure for these two methods is recorded in Appendix
L.1 and L.2.

###  4.3 Task Decomposition

With task decomposition, LLM-generated decision criteria are used to define
substasks which are independently solved by the LLM. The final score is
computed by averaging scores from the subtasks. This strategy, denoted as
Subtask+O, aims to reduce the complexity of evaluating whether a passage is
relevant to a query.

###  4.4 Task Objective Awareness

BIRCO tasks vary in their objectives, and these objectives can be described in
LLM prompts. Any model that uses a prompt containing the task description is
suffixed with "O". Alternatively, as a simpler baseline, LLMs can be prompted
to retrieve semantically similar passages without knowing the task objective.

All prompts that describe task objectives are based on the description of
these datasets in their original papers, and tested on BIRCO’s development
sets. Prompts for task objectives in Appendix B.

##  5 LLM Benchmark Results

Model | DORIS-MAE | ArguAna | WhatsThatBook | Clinical-Trial | RELIC |
---|---|---|---|---|---|---
| nDCG@10 | Recall@5 | nDCG@10 | Recall@5 | nDCG@10 | Recall@5 | nDCG@10 | Recall@5 | nDCG@10 | Recall@5 |  Avg. nDCG@10
Maximum Possible | 100 | 45.6 | 100 | 100 | 100 | 100 | 100 | 37.91 | 100 | 100 |  100
Random | 49.9 | 4.5 | 9.1 | 10.0 | 9.1 | 10.0 | 22.3 | 7.6 | 9.0 | 9.9 |  19.9
Embedding Models
E5-L-v2 | 72.0 | 14.6 | 43.5 | 62.0 | 36.6 | 39.9 | 29.4 | 10.5 | 11.1 | 14.9 |  38.5
SIMCSE | 70.8 | 14.4 | 39.8 | 51.1 | 31.7 | 37.0 | 27.7 | 10.5 | 12.9 | 15.6 |  36.6
Promptagator | 76.0 | 15.6 | 63.0 | 77.8 | 59.1 | 62.9 | NA | NA | NA | NA |  NA
Finetuned Encoder-Decoder Models
TART | 64.9 | 10.2 | 47.6 | 63.7 | 42.7 | 46.1 | 32.8 | 10.8 | 9.4 | 7.9 |  39.5
TART+O | 49.6 | 4.9 | 23.1 | 28.0 | 17.9 | 20.7 | 23.6 | 7.7 | 14.3 | 18.3 |  25.7
MonoT5 | 66.9 | 12.4 | 45.9 | 52.0 | 50.8 | 59.0 | 33.2 | 14.2 | 13.9 | 11.9 |  42.1
Finetuned Decoder-Only Model
RankLLaMA | 75.1 | 18.4 | 53.2 | 71.9 | 63.7 | 69.8 | 29.7 | 9.8 | 15.4 | 19.0 |  47.4
Comparison-based LLM IR Systems
RankStripedHyena | 71.7 | 14.6 | 40.7 | 59.8 | 30.4 | 37.9 | 30.5 | 10.6 | 10.6 | 14.9 |  36.8
Rank+OStripedHyena | 70.6 | 15.5 | 41.4 | 48.9 | 34.2 | 40.9 | 28.7 | 10.1 | 11.3 | 14.9 |  37.2
RankLlama2-7b | 71.0 | 13.2 | 41.4 | 54.8 | 29.5 | 37.8 | 29.3 | 37.8 | 10.5 | 11.8 |  36.3
Rank+OLlama2-7b | 70.8 | 14.1 | 41.1 | 54.6 | 26.7 | 34.8 | 28.8 | 9.7 | 10.6 | 10.8 |  35.9
RankLlama2-13b | 71.2 | 13.0 | 40.1 | 53.9 | 26.5 | 36.8 | 29.6 | 10.6 | 10.7 | 14.9 |  35.6
Rank+OLlama2-13b | 70.5 | 14.2 | 39.1 | 51.7 | 28.5 | 35.8 | 28.4 | 10.0 | 8.9 | 12.8 |  35.0
RankLlama2-70b | 70.7 | 15.1 | 39.0 | 49.8 | 27.6 | 36.8 | 30.2 | 10.3 | 10.5 | 13.8 |  35.6
Rank+OLlama2-70b | 70.4 | 14.2 | 41.6 | 51.7 | 27.6 | 36.8 | 28.8 | 9.7 | 10.5 | 13.9 |  35.8
RankGPT4 | 76.2 | 17.5 | 26.7 | 17.0 | 79.5 | 85.9 | 39.9 | 14.8 | 40.1 | 48.0 |  52.5
Rank+OGPT4 | 77.4 | 17.6 | 54.4 | 70.9 | 82.1 | 88.0 | 38.6 | 14.6 | 62.3 | 70.1 |  63.0
Scoring-based LLM IR Systems
Score+OStripedHyena | 67.5 | 9.5 | 20.6 | 24.9 | 43.0 | 50.8 | 32.5 | 10.7 | 20.7 | 25.0 |  36.9
Reason+OStripedHyena | 65.9 | 10.9 | 26.3 | 32.0 | 22.2 | 22.1 | 27.5 | 11.0 | 12.8 | 11.8 |  30.9
Subtask+OStripedHyena | 74.8 | 15.5 | 14.1 | 20.9 | 44.4 | 49.0 | 32.4 | 11.0 | 24.7 | 29.1 |  38.1
Score+OLlama2-7b | 71.6 | 14.1 | 27.0 | 40.1 | 30.5 | 34.7 | 29.7 | 10.1 | 10.9 | 13.9 |  33.9
Reason+OLlama2-7b | 72.1 | 13.8 | 21.3 | 27.8 | 21.4 | 28.1 | 31.0 | 9.9 | 11.5 | 12.0 |  31.5
Subtask+OLlama2-7b | 69.2 | 11.8 | 16.4 | 18.8 | 29.2 | 32.9 | 25.1 | 8.6 | 10.5 | 11.8 |  30.1
Score+OLlama2-13b | 77.3 | 14.5 | 15.4 | 21.9 | 28.1 | 31.9 | 30.0 | 9.9 | 16.8 | 18.0 |  33.5
Reason+OLlama2-13b | 60.1 | 7.5 | 31.4 | 43.1 | 18.7 | 27.0 | 24.2 | 7.7 | 9.0 | 8.9 |  28.7
Subtask+OLlama2-13b | 71.3 | 14.1 | 12.4 | 14.9 | 48.7 | 60.9 | 29.4 | 10.7 | 17.6 | 21.0 |  35.9
Score+OLlama2-70b | 75.1 | 15.1 | 40.3 | 49.8 | 60.8 | 70.0 | 34.9 | 12.1 | 12.2 | 14.9 |  44.7
Reason+OLlama2-70b | 70.1 | 14.5 | 42.8 | 55.0 | 43.4 | 48.0 | 31.3 | 9.9 | 12.7 | 15.9 |  40.1
Subtask+OLlama2-70b | 76.7 | 16.3 | 19.3 | 27.0 | 49.1 | 60.1 | 32.8 | 10.5 | 19.6 | 21.0 |  39.5
Score+OGPT4 | 79.9 | 19.3 | 51.6 | 70.0 | 83.3 | 90.9 | 43.4 | 17.2 | 54.1 | 70.1 |  62.5
Reason+OGPT | 74.9 | 18.1 | 59.9 | 76.8 | 74.9 | 82.8 | 45.7 | 17.0 | 39.9 | 51.0 |  59.1
Subtask+OGPT4 | 78.5 | 18.5 | 69.5 | 85.9 | 79.6 | 85.9 | 43.5 | 16.5 | 53.2 | 62.9 |  64.9
Table 5: nDCG@10 and Recall@5 for the benchmark datasets. Bold indicates no
statistically significant difference from the highest numerical value, which
is indicated in red. Refer to Appendix Table 7 for standard errors. The
notation $+$O indicates task objective awareness. The consistently low recall
on DORIS-MAE and Clinical-Trial corresponds to their low maximum possible
recall.

We use BIRCO to evaluate the performance of the different LLM retrieval
strategies described in Section 4. The strategies are evaluated on GPT4 OpenAI
(2023), Llama-2 Touvron et al. (2023), and StripedHyena Poli et al. (2023),
all of which use the same prompt strategies described in Appendix B and
Appendix L.

Appendix A.2 provides details about LLM models, prompt configuration, compute
requirements, and additional experimental setups for (fine-tuned) embedding
models and fine-tuned decoder models.

Results are shown in Table 5. Overall, larger LLMs perform better than smaller
LLMs, with GPT4 achieving the strongest results across all retrieval
strategies. Detailed comparisons of the LLM approaches are provided below.

Table 5 also shows results for the baseline embedding models and language
models which have been fine-tuned for information retrieval tasks. Only
GPT4-based systems consistently outperform the strongest baselines, RankLLaMA
(a 7B model fine-tuned on MS MARCO) and MonoT5 (a 3B model fine-tuned on MS
MARCO). Llama 2 70B is the only other LLM that performs better than the
E5-L-v2 baseline, which is a 300M parameter embedding model.

Overall, the results show that small embedding models and fine-tuned language
models provide strong baselines relative to all LLMs except GPT4.

Ranking vs. Scoring We first evaluate the performance of systems that rank
documents by comparing them, and systems that directly score the relevance of
individual documents. We find no general effect of comparative ranking vs.
direct scoring, except for Llama 70B, which had stronger performance with
direct scoring.

Chain-of-Thought Reasoning We next evaluate whether chain-of-thought reasoning
improves retrieval performance for LLMs. While chain-of-thought improves
performance for certain task/model combinations, the effect is inconsistent,
and in aggregate chain-of-thought slightly decreases retrieval performance.

While chain-of-thought frequently improves performance on NLP and reasoning
tasks, other work has shown that improvements are not uniformly found across
all tasks (Suzgun et al., 2022), consistent with the current results.

Task Decomposition With task decomposition, the model first decomposes the
query into subtasks, and then evaluates each subtask separately. Task
decomposition does not lead to consistent improvement in performance. The
strongest average performance is achieved by GPT4 with task decomposition, but
this improvement is largely driven by a single task, ArguAna.

Task Objective Awareness Providing models with detailed descriptions of task
objectives does not improve performance on the smaller LLMs. For GPT4 with
comparative ranking, it substantially increases performance on two datasets,
ArguAna and RELIC. For GPT4 with direct scoring, we found poor performance on
the development sets when task objectives were not provided (results not shown
in Table 5). Qualitatively, this was due to the model not knowing how to
determine relevance between queries and passages.

ArguAna and RELIC have non-standard task objectives, likely accounting for the
importance of detailed prompting for these tasks.

###  5.1 Error Analysis

Two of the datasets contain hard negatives, non-relevant documents which are
constructed to be similar to the ground-truth answers, as described in Section
3.7. Table 6 shows the average reciprocal rank for hard negatives compared to
the other passages in the datasets, as ranked by GPT4-based systems. The
reciprocal rank of the hard negatives is much higher than the other passages,
indicating that GPT4 found these passages more relevant, and that these
passages increase the difficulty of retrieval. See more details of error
analysis in Appendix H.

Method | Clinical-Trial | ArguAna
---|---|---
| Hard | Other | Hard | Other
Reason+OGPT4 | .097 | .042 | .154 | .059
Score+OGPT4 | .097 | .040 | .170 | .056
Subtask+OGPT4 | .090 | .042 | .148 | .057
RankGPT+OGPT4 | .096 | .049 | .166 | .056
Avg. | .095 | .043 | .160 | .057
Table 6: Average reciprocal rank for hard negatives.

##  6 Conclusion

We have introduced BIRCO, a benchmark for IR tasks with complex objectives.
BIRCO includes 5 diverse retrieval tasks, and is significantly more
challenging than previous IR benchmarks.

We found that embedding methods and small language models have weak
performance on the BIRCO tasks. Methods that use LLMs for ranking have
stronger performance, though none achieve strong results across all tasks.

We evaluated several hypotheses regarding LLM performance. First, providing
clear instructions to the LLMs regarding task objectives improved performance
on certain tasks. Second, we did not find evidence that ranking by comparing
passages improved performance relative to directly scoring passages. Third, in
contrast to results in many other NLP domains Kojima et al. (2022); Huang et
al. (2022), we did not find evidence that chain-of-thought reasoning improves
retrieval performance. Finally, decomposing queries into subtasks improved
performance on only a single dataset.

The results underscore the need to develop IR methods that go beyond
similarity-based retrieval. Strong performance on BIRCO requires models that
can understand multi-faceted user intents. While GPT4-based methods had the
strongest performance, even they did not achieve adequate performance across
tasks. Furthermore, it is currently prohibitively expensive to perform
inference with LLMs for all but the smallest IR tasks. The challenge of
complex user objectives will require improvements in model abilities and
efficiency.

##  7 Limitations

There are many types of user retrieval needs. BIRCO only provides a benchmark
for a limited sample of these needs. We hope that future datasets and
benchmarks can provide more comprehensive coverage of these other cases.

Large language models are computationally expensive and can only be
effectively employed in the passage re-ranking stage of a multi-stage IR
process. In addition, depending on the use case, LLMs may be prohibitively
expensive even when the candidate pool size is limited.

While we optimized prompts to the best of our abilities on the development
sets, it is possible that better prompts would improve LLM performance on the
tasks.

##  8 Ethical Considerations

No ethical concerns for this work.

## References

  * Asai et al. (2023) Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023.  Task-aware retrieval with instructions.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 3650–3675, Toronto, Canada. Association for Computational Linguistics.
  * Bondarenko et al. (2020) Alexander Bondarenko, Maik Fröbe, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, et al. 2020.  Overview of touché 2020: argument retrieval.  In _Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings 11_ , pages 384–395. Springer.
  * Chuang et al. (2022) Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022.  DiffCSE: Difference-based contrastive learning for sentence embeddings.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4207–4218, Seattle, United States. Association for Computational Linguistics.
  * Dai et al. (2022) Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. 2022.  Promptagator: Few-shot dense retrieval from 8 examples.  In _The Eleventh International Conference on Learning Representations_.
  * Diggelmann et al. (2020) Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020.  Climate-fever: A dataset for verification of real-world climate claims.  _arXiv preprint arXiv:2012.00614_.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  _arXiv preprint arXiv:1903.00161_.
  * Ferraretto et al. (2023) Fernando Ferraretto, Thiago Laitz, Roberto Lotufo, and Rodrigo Nogueira. 2023.  Exaranker: Synthetic explanations improve neural rankers.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2409–2414.
  * Formal et al. (2021) Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021.  Splade v2: Sparse lexical and expansion model for information retrieval.  _arXiv preprint arXiv:2109.10086_.
  * Formal et al. (2022) Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2022.  From distillation to hard negative sampling: Making sparse neural ir models more effective.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2353–2359.
  * Gao et al. (2023) Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023.  Precise zero-shot dense retrieval without relevance labels.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1762–1777, Toronto, Canada. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Greene et al. (2022) Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan. 2022.  New and improved embedding model.  https://openai.com/blog/new-and-improved-embedding-model/.  Accessed: 2023-06-03.
  * Hasibi et al. (2017) Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017.  Dbpedia-entity v2: A test collection for entity search.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’17, page 1265–1268, New York, NY, USA. Association for Computing Machinery.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Huang et al. (2022) Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In _International Conference on Machine Learning_ , pages 9118–9147. PMLR.
  * Ioffe (2010) Sergey Ioffe. 2010.  Improved consistent sampling, weighted minhash and l1 sketching.  In _2010 IEEE international conference on data mining_ , pages 246–255. IEEE.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 22199–22213. Curran Associates, Inc.
  * Koopman and Zuccon (2016) Bevan Koopman and Guido Zuccon. 2016.  A test collection for matching patients to clinical trials.  In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_ , pages 669–672.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Lin et al. (2023) Kevin Lin, Kyle Lo, Joseph E Gonzalez, and Dan Klein. 2023.  Decomposing complex queries for tip-of-the-tongue retrieval.  _arXiv preprint arXiv:2305.15053_.
  * Liu et al. (2023) Jiduan Liu, Jiahao Liu, Qifan Wang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, Kai Chen, and Rui Yan. 2023.  RankCSE: Unsupervised sentence representations learning via learning to rank.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13785–13802, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ma et al. (2023a) Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023a.  Fine-tuning llama for multi-stage text retrieval.  _arXiv preprint arXiv:2310.08319_.
  * Ma et al. (2023b) Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023b.  Zero-shot listwise document reranking with a large language model.  _arXiv preprint arXiv:2305.02156_.
  * Maia et al. (2018) Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018.  Www’18 open challenge: Financial opinion mining and question answering.  In _Companion Proceedings of the The Web Conference 2018_ , WWW ’18, page 1941–1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022.  Mteb: Massive text embedding benchmark.  _arXiv preprint arXiv:2210.07316_.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _choice_ , 2640:660.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 708–718.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Pan et al. (2023) Kaihang Pan, Juncheng Li, Hongye Song, Hao Fei, Wei Ji, Shuo Zhang, Jun Lin, Xiaozhong Liu, and Siliang Tang. 2023.  Controlretriever: Harnessing the power of instructions for controllable retrieval.  _arXiv preprint arXiv:2308.10025_.
  * Poli et al. (2023) Michael Poli, Jue Wang, Stefano Massaroli, Jeffrey Quesnelle, Ryan Carlow, Eric Nguyen, and Armin Thomas. 2023.  StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models.
  * Prakash et al. (2024) Vipul Ved Prakash, Ce Zhang, Chris Ré, and Percy Liang. 2024.  Together ai.  https://www.together.ai/.  https://www.together.ai/.
  * Qin et al. (2023) Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. 2023.  Large language models are effective text rankers with pairwise ranking prompting.  _arXiv preprint arXiv:2306.17563_.
  * Sachan et al. (2022) Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022.  Improving passage retrieval with zero-shot question generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3781–3797, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Santhanam et al. (2022) Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022.  ColBERTv2: Effective and efficient retrieval via lightweight late interaction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3715–3734, Seattle, United States. Association for Computational Linguistics.
  * Singh et al. (2022) Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2022.  Scirepeval: A multi-format benchmark for scientific document representations.  _arXiv preprint arXiv:2211.13308_.
  * Su et al. (2023) Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023.  One embedder, any task: Instruction-finetuned text embeddings.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1102–1121, Toronto, Canada. Association for Computational Linguistics.
  * Sun et al. (2023) Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023.  Is chatgpt good at search? investigating large language models as re-ranking agent.  _arXiv preprint arXiv:2304.09542_.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _arXiv preprint arXiv:2210.09261_.
  * Thai et al. (2022) Katherine Thai, Yapei Chang, Kalpesh Krishna, and Mohit Iyyer. 2022.  RELiC: Retrieving evidence for literary claims.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7500–7518, Dublin, Ireland. Association for Computational Linguistics.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Trotman et al. (2014) Andrew Trotman, Antti Puurula, and Blake Burgess. 2014.  Improvements to bm25 and language models examined.  In _Proceedings of the 2014 Australasian Document Computing Symposium_ , pages 58–65.
  * Voorhees et al. (2021) Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021.  Trec-covid: constructing a pandemic information retrieval test collection.  _SIGIR Forum_ , 54(1).
  * Wachsmuth et al. (2018) Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018.  Retrieval of the best counterargument without prior topic knowledge.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 241–251, Melbourne, Australia. Association for Computational Linguistics.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, Online. Association for Computational Linguistics.
  * Wang et al. (2023) Jianyou Wang, Kaicheng Wang, Xiaoyue Wang, Prudhviraj Naidu, Leon Bergen, and Ramamohan Paturi. 2023.  Doris-mae: Scientific document retrieval using multi-level aspect-based queries.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Wang et al. (2022) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022.  Text embeddings by weakly-supervised contrastive pre-training.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Zhang et al. (2023) Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023.  MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages.  _Transactions of the Association for Computational Linguistics_ , 11:1114–1131.
  * Zhuang et al. (2023) Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Berdersky. 2023.  Beyond yes and no: Improving zero-shot llm rankers via scoring fine-grained relevance labels.  _arXiv preprint arXiv:2310.14122_.

##  Appendix A Experiment Details

Model | DORIS-MAE | ArguAna | WhatsThatBook | Clinical-Trial | RELIC |
---|---|---|---|---|---|---
| nDCG@10 | R@5 | nDCG@10 | R@5 | nDCG@10 | R@5 | nDCG@10 | R@5 | nDCG@10 | R@5 |  Avg. nDCG@10
Maximum | 100 | 45.6 | 100 | 100 | 100 | 100 | 100 | 37.91 | 100 | 100 |  100
Random | 49.9 | 4.5 | 9.1 | 10.0 | 9.1 | 10.0 | 22.3 | 7.6 | 9.0 | 9.9 |  19.9
Embedding Models
RoBERTa-L | 66.8 $\pm$ 1.3 | 12.0 $\pm$ 1.6 | 31.5 $\pm$ 3.6 | 40.2 $\pm$ 5.1 | 14.6 $\pm$ 2.7 | 16.0 $\pm$ 3.7 | 25.9 $\pm$ 2.2 | 8.9 $\pm$ 1.2 | 8.4 $\pm$ 2.2 | 8.9 $\pm$ 2.7 |  29.4
SPLADE++ | 66.8 $\pm$ 1.3 | 8.2 $\pm$ 1.2 | 34.0 $\pm$ 3.4 | 47.9 $\pm$ 5.0 | 9.5 $\pm$ 2.2 | 8.9 $\pm$ 2.9 | 27.5 $\pm$ 2.2 | 9.5 $\pm$ 1.2 | 11.4 $\pm$ 2.2 | 13.9 $\pm$ 3.3 |  29.8
SPLADE-v2 | 67.9 $\pm$ 1.4 | 10.6 $\pm$ 2.1 | 37.8 $\pm$ 3.7 | 40.7 $\pm$ 5.0 | 16.2 $\pm$ 3.0 | 20.2 $\pm$ 4.1 | 22.3 $\pm$ 2.3 | 7.2 $\pm$ 1.1 | 10.7 $\pm$ 2.1 | 11.0 $\pm$ 3.0 |  31.0
SPECTER-v2 | 71.4 $\pm$ 1.2 | 13.5 $\pm$ 2.3 | 37.7 $\pm$ 3.3 | 49.1 $\pm$ 5.1 | 9.8 $\pm$ 2.0 | 13.0 $\pm$ 3.4 | 30.4 $\pm$ 2.1 | 11.2 $\pm$ 1.2 | 7.7 $\pm$ 1.6 | 12.0 $\pm$ 3.1 |  31.4
E5-L-v2 | 72.0 $\pm$ 1.2 | 14.6 $\pm$ 1.9 | 43.5 $\pm$ 3.2 | 62.0 $\pm$ 4.8 | 36.6 $\pm$ 4.0 | 39.9 $\pm$ 4.8 | 29.4 $\pm$ 2.7 | 10.5 $\pm$ 1.7 | 11.1 $\pm$ 2.6 | 14.9 $\pm$ 3.4 |  38.5
SIMCSE | 70.8 $\pm$ 1.3 | 14.4 $\pm$ 1.8 | 39.8 $\pm$ 3.5 | 51.1 $\pm$ 5.1 | 31.7 $\pm$ 4.0 | 37.0 $\pm$ 5.0 | 27.7 $\pm$ 2.7 | 10.5 $\pm$ 1.6 | 12.9 $\pm$ 2.5 | 15.6 $\pm$ 3.7 |  36.6
Promptagator | 76.0 $\pm$ 1.4 | 15.6 $\pm$ 1.8 | 63.0 $\pm$ 3.3 |  77.8 $\pm$ 4.2 | 59.1 $\pm$ 4.2 | 62.9 $\pm$ 4.7 | NA | NA | NA | NA |  NA
Encoder-Decoder Models
TART | 64.9 $\pm$ 1.5 | 10.2 $\pm$ 1.7 | 47.6 $\pm$ 3.1 | 63.7 $\pm$ 4.8 | 42.7 $\pm$ 3.7 | 46.1 $\pm$ 4.9 | 32.8 $\pm$ 2.5 | 10.8 $\pm$ 1.3 | 9.4 $\pm$ 1.9 | 7.9 $\pm$ 2.7 |  39.5
TART+O | 49.6 $\pm$ 1.8 | 4.9 $\pm$ 1.0 | 23.1 $\pm$ 2.8 | 28.0 $\pm$ 4.4 | 17.9 $\pm$ 2.4 | 20.7 $\pm$ 4.1 | 23.6 $\pm$ 2.2 | 7.7 $\pm$ 1.0 | 14.3 $\pm$ 2.3 | 18.3 $\pm$ 3.8 |  25.7
MonoT5 | 66.9 $\pm$ 1.5 | 12.4 $\pm$ 1.8 | 45.9 $\pm$ 3.2 | 52.0 $\pm$ 4.9 | 50.8 $\pm$ 4.2 | 59.0 $\pm$ 4.9 | 33.2 $\pm$ 2.5 |  14.2 $\pm$ 2.3 | 13.9 $\pm$ 2.7 | 11.9 $\pm$ 3.2 |  42.1
Decoder-Only Model
RankLLaMA | 75.1 $\pm$ 1.3 |  18.4 $\pm$ 2.0 | 53.2 $\pm$ 2.9 | 71.9 $\pm$ 4.4 | 63.7 $\pm$ 3.9 | 69.8 $\pm$ 4.6 | 29.7 $\pm$ 2.2 | 9.8 $\pm$ 1.1 | 15.4 $\pm$ 2.4 | 19.0 $\pm$ 4.0 |  47.4
Comparison-based LLM IR Systems
RankStripedHyena | 71.7 $\pm$ 1.3 | 14.6 $\pm$ 1.9 | 40.7 $\pm$ 3.1 | 59.8 $\pm$ 5.0 | 30.4 $\pm$ 3.5 | 37.9 $\pm$ 4.8 | 30.5 $\pm$ 2.6 | 10.6 $\pm$ 1.7 | 10.6 $\pm$ 2.4 | 14.9 $\pm$ 3.4 |  36.8
Rank+OStripedHyena | 70.6 $\pm$ 1.3 | 15.5 $\pm$ 1.9 | 41.4 $\pm$ 3.4 | 48.9 $\pm$ 5.2 | 34.2 $\pm$ 3.9 | 40.9 $\pm$ 4.9 | 28.7 $\pm$ 2.4 | 10.1 $\pm$ 1.2 | 11.3 $\pm$ 2.5 | 14.9 $\pm$ 3.4 |  37.2
RankLlama2-7b | 71.0 $\pm$ 1.3 | 13.2 $\pm$ 1.6 | 41.4 $\pm$ 2.8 | 54.8 $\pm$ 5.1 | 29.5 $\pm$ 3.3 | 37.8 $\pm$ 4.8 | 29.3 $\pm$ 2.6 | 37.8 $\pm$ 1.8 | 10.5 $\pm$ 2.3 | 11.8 $\pm$ 3.1 |  36.3
Rank+OLlama2-7b | 70.8 $\pm$ 1.3 | 14.1 $\pm$ 1.7 | 41.1 $\pm$ 3.0 | 54.6 $\pm$ 4.9 | 26.7 $\pm$ 3.0 | 34.8 $\pm$ 4.6 | 28.8 $\pm$ 2.4 | 9.7 $\pm$ 1.3 | 10.6 $\pm$ 2.2 | 10.8 $\pm$ 3.0 |  35.6
RankLlama2-13b | 71.2 $\pm$ 1.4 | 13.0 $\pm$ 1.6 | 40.1 $\pm$ 2.9 | 53.9 $\pm$ 5.1 | 26.5 $\pm$ 3.1 | 36.8 $\pm$ 4.8 | 29.6 $\pm$ 2.7 | 10.6 $\pm$ 1.5 | 10.7 $\pm$ 2.3 | 14.9 $\pm$ 3.5 |  35.6
Rank+OLlama2-13b | 70.5 $\pm$ 1.3 | 14.2 $\pm$ 1.7 | 39.1 $\pm$ 2.9 | 51.7 $\pm$ 5.2 | 28.5 $\pm$ 3.3 | 35.8 $\pm$ 4.7 | 28.4 $\pm$ 2.7 | 10.0 $\pm$ 1.6 | 8.9 $\pm$ 2.0 | 12.8 $\pm$ 3.2 |  35.0
RankLlama2-70b | 70.7 $\pm$ 1.3 | 15.1 $\pm$ 1.9 | 39.0 $\pm$ 2.8 | 49.8 $\pm$ 5.0 | 27.6 $\pm$ 3.2 | 36.8 $\pm$ 4.7 | 30.2 $\pm$ 2.7 | 10.3 $\pm$ 1.4 | 10.5 $\pm$ 2.3 | 13.8 $\pm$ 3.4 |  35.6
Rank+OLlama2-70b | 70.4 $\pm$ 1.3 | 14.2$\pm$ 1.7 | 41.6$\pm$ 3.0 | 51.7 $\pm$ 5.0 | 27.6 $\pm$ 3.1 | 36.8$\pm$ 4.7 | 28.8 $\pm$ 2.4 | 9.7 $\pm$ 1.3 | 10.5 $\pm$ 2.4 | 13.9$\pm$ 3.3 |  35.8
RankGPT4 | 76.2 $\pm$ 1.3 |  17.5 $\pm$ 2.3 | 26.7 $\pm$ 2.2 | 17.0 $\pm$ 3.8 |  79.5 $\pm$ 3.5 |  85.9 $\pm$ 3.4 | 39.9 $\pm$ 2.6 |  14.8 $\pm$ 1.4 | 40.1 $\pm$ 4.0 | 48.0 $\pm$ 5.1 |  52.5
Rank+OGPT4 | 77.4 $\pm$ 1.3 |  17.6 $\pm$ 2.1 | 54.4 $\pm$ 3.2 | 70.9 $\pm$ 4.5 |  82.1 $\pm$ 3.3 |  88.0 $\pm$ 3.2 | 38.6 $\pm$ 2.8 | 14.6 $\pm$ 1.5 |  62.3 $\pm$ 3.9 |  70.1 $\pm$ 4.5 |  63.0
Scoring-based LLM IR Systems
Score+OSH-7b | 67.5 $\pm$ 1.4 | 9.5 $\pm$ 1.4 | 20.6 $\pm$ 2.5 | 24.9 $\pm$ 4.1 | 43.0 $\pm$ 4.0 | 50.8 $\pm$ 4.9 | 32.5 $\pm$ 2.3 | 10.7 $\pm$ 1.3 | 20.7 $\pm$ 3.0 | 25.0 $\pm$ 4.4 |  36.9
Reason+OSH-7b | 65.9 $\pm$ 1.1 | 10.9 $\pm$ 1.4 | 26.3 $\pm$ 3.4 | 32.0 $\pm$ 4.9 | 22.2 $\pm$ 3.6 | 22.1 $\pm$ 4.3 | 27.5 $\pm$ 2.6 | 11.0 $\pm$ 1.8 | 12.8 $\pm$ 2.5 | 11.8 $\pm$ 3.2 |  30.9
Subtask+OSH-7b | 74.8 $\pm$ 1.3 | 15.5 $\pm$ 2.3 | 14.1 $\pm$ 2.1 | 20.9 $\pm$ 4.0 | 44.4 $\pm$ 3.7 | 49.0 $\pm$ 4.9 | 32.4 $\pm$ 2.5 | 11.0 $\pm$ 1.3 | 24.7 $\pm$ 3.1 | 29.1 $\pm$ 4.5 |  38.1
Score+OLM2-7b | 71.6 $\pm$ 1.4 | 14.1 $\pm$ 1.9 | 27.0 $\pm$ 3.2 | 40.1 $\pm$ 4.9 | 30.5 $\pm$ 3.8 | 34.7 $\pm$ 4.8 | 29.7 $\pm$ 2.8 | 10.1 $\pm$ 1.5 | 10.9 $\pm$ 2.4 | 13.9 $\pm$ 3.3 |  33.9
Reason+OLM2-7b | 72.1 $\pm$ 1.2 | 13.8 $\pm$ 1.7 | 21.3 $\pm$ 3.3 | 27.8 $\pm$ 4.6 | 21.4 $\pm$ 3.1 | 28.1 $\pm$ 4.4 | 31.0 $\pm$ 2.5 | 9.9 $\pm$ 1.4 | 11.5 $\pm$ 2.3 | 12.0 $\pm$ 3.2 |  31.5
Subtask+OLM2-7b | 69.2 $\pm$ 1.4 | 11.8 $\pm$ 1.7 | 16.4 $\pm$ 2.4 | 18.8 $\pm$ 3.8 | 29.2 $\pm$ 3.4 | 32.9 $\pm$ 4.4 | 25.1 $\pm$ 2.2 | 8.6 $\pm$ 1.3 | 10.5 $\pm$ 2.6 | 11.8 $\pm$ 3.1 |  30.1
Score+OLM2-13b | 77.3 $\pm$ 0.9 | 14.5 $\pm$ 1.6 | 15.4 $\pm$ 2.4 | 21.9 $\pm$ 4.0 | 28.1 $\pm$ 3.3 | 31.9 $\pm$ 4.5 | 30.0 $\pm$ 2.4 | 9.9 $\pm$ 1.1 | 16.8 $\pm$ 2.5 | 18.0 $\pm$ 3.7 |  33.5
Reason+OLM2-13b | 60.1 $\pm$ 1.4 | 7.5 $\pm$ 1.4 | 31.4 $\pm$ 3.0 | 43.1 $\pm$ 4.9 | 18.7 $\pm$ 2.4 | 27.0 $\pm$ 4.4 | 24.2 $\pm$ 2.5 | 7.7 $\pm$ 1.1 | 9.0 $\pm$ 1.8 | 8.9 $\pm$ 2.8 |  28.7
Subtask+OLM2-13b | 71.3 $\pm$ 1.3 | 14.1 $\pm$ 1.7 | 12.4 $\pm$ 2.3 | 14.9 $\pm$ 3.5 | 48.7 $\pm$ 3.9 | 60.9 $\pm$ 5.0 | 29.4 $\pm$ 2.3 | 10.7 $\pm$ 1.3 | 17.6 $\pm$ 2.7 | 21.0 $\pm$ 4.0 |  35.9
Score+OLM2-70b | 75.1 $\pm$ 1.2 | 15.1 $\pm$ 2.0 | 40.3 $\pm$ 3.4 | 49.8 $\pm$ 5.0 | 60.8 $\pm$ 3.7 | 70.0 $\pm$ 4.7 | 34.9 $\pm$ 2.6 | 12.1 $\pm$ 1.6 | 12.2 $\pm$ 2.6 | 14.9 $\pm$ 3.4 |  44.7
Reason+OLM2-70b | 70.1 $\pm$ 1.4 | 14.5 $\pm$ 1.9 | 42.8 $\pm$ 3.2 | 55.0 $\pm$ 4.9 | 43.4 $\pm$ 4.1 | 48.0 $\pm$ 5.0 | 31.3 $\pm$ 2.2 | 9.9 $\pm$ 1.2 | 12.7 $\pm$ 2.6 | 15.9 $\pm$ 3.5 |  40.1
Subtask+OLM2-70b | 76.7 $\pm$ 1.2 | 16.3 $\pm$ 1.9 | 19.3 $\pm$ 2.7 | 27.0 $\pm$ 4.4 | 49.1 $\pm$ 3.8 | 60.1 $\pm$ 4.8 | 32.8 $\pm$ 2.9 | 10.5 $\pm$ 1.3 | 19.6 $\pm$ 3.2 | 21.0 $\pm$ 4.0 |  39.5
Score+OGPT4 |  79.9 $\pm$ 1.2 |  19.3 $\pm$ 2.0 | 51.6 $\pm$ 2.7 | 70.0 $\pm$ 4.6 |  83.3 $\pm$ 3.1 |  90.9 $\pm$ 2.8 |  43.4 $\pm$ 2.4 |  17.2 $\pm$ 1.6 |  54.1 $\pm$ 3.3 |  70.1 $\pm$ 4.6 |  62.5
Reason+OGPT4 | 74.9 $\pm$ 1.3 |  18.1 $\pm$ 2.3 | 59.9 $\pm$ 2.9 |  76.8 $\pm$ 4.2 | 74.9 $\pm$ 3.6 | 82.8 $\pm$ 3.7 |  45.7 $\pm$ 3.1 |  17.0 $\pm$ 1.7 | 39.9 $\pm$ 3.3 | 51.0 $\pm$ 5.0 |  59.1
Subtask+OGPT4 |  78.5 $\pm$ 1.2 |  18.5 $\pm$ 1.9 |  69.5 $\pm$ 3.0 |  85.9 $\pm$ 3.5 |  79.6 $\pm$ 3.5 |  85.9 $\pm$ 3.3 |  43.5 $\pm$ 2.9 |  16.5 $\pm$ 1.6 | 53.2 $\pm$ 3.5 |  62.9 $\pm$ 4.8 |  64.9
Table 7: nDCG@10 and Recall@5 for the benchmark datasets. Bold indicates
$p>0.05$ compared to the highest numerical value indicated in red. Error bars
are standard errors. The notation $+$O indicates task objective awareness. LM2
stands for Llama2 Touvron et al. (2023). SH stands for StripedHyena Poli et
al. (2023)

We use the GPT4-0613 checkpoint for all GPT4-based systems and access it via
OpenAI’s API. We use Together AI’s API to access the other LLMs’ checkpoints
Prakash et al. (2024).

Each GPT4-based system takes less than 12 hours to evaluate on BIRCO, and
costs approximately $500-$1000. Other LLM-based systems take less than 3 hours
to evaluate on BIRCO and cost less than $50.

All embedding models have 350M parameters or less. Promptagator models are
E5-v2-L checkpoint finetuned on promptagator-styled synthetic data, introduced
by Dai et al. (2022). No data for Clinical-Trial and RELIC is available for
Promptagator models. Other embedding models are not fine-tuned for IR tasks.

Finetuning for TART, MonoT5, and RankLLama was performed on MS MARCO.
Experiments on these models can be run on one node of an 8 NVIDIA H100 GPU
(80G) within one hour.

###  A.1 Baselines

For pretrained embedding models, we choose several recent state-of-the-art
models as well as several that have been extensively benchmarked on other
tasks: E5-Large-v2 Wang et al. (2022), SimCSE-Large Gao et al. (2021),
SPECTER-v2 Singh et al. (2022), ROBERTA-Large Liu et al. (2019), SPLADE-v2
Formal et al. (2021), and SPLADE++ Formal et al. (2022). There are several
encoder-decoder models specifically trained for information retrieval:
monoT5-3B Nogueira et al. (2020), which is supervised on MS MARCO, and
TART-1.5B Asai et al. (2023), which is instruction-tuned and can incorporate
task descriptions. For DORIS-MAE, ArguAna, and WhatsThatBook, we also report
results for E5-Large-v2 fine-tuned using synthetic data generated by
Promptagator Dai et al. (2022). Additionally, we evaluate a decoder-only model
RankLLaMA-7B Ma et al. (2023a) trained on MS MARCO. We also evaluate two other
generative models: Llama2 chat Touvron et al. (2023), which is one of the best
open-sourced LLMs, and StripedHyena-Nous-7B chat Poli et al. (2023), which is
a competitive alternative LLM with a hybrid architecture between multi-headed
attention and gated convolution.

###  A.2 Full Experiment Results

See Table 7 for full experiment results with more embedding models and
standard errors for each model’s performance.

####  A.2.1 Analysis for Non-LLM models

As shown, all embedding models are significantly worse than GPT4-based models.
On average, fine-tuned decoder models TART and MonoT5 are comparable with the
best pretrained embedding models. The larger RankLLaMA decoder model
outperforms most embedding models and other decoder models. However, based on
Promptagator’s performance on DORIS-MAE, ArguAna, and WhatsThatBook, this
synthetic data trained embedding model appears comparable to RankLLaMA. There
is a significant decrease in TART+O compared with TART, suggesting TART has
difficulty following complex instructions.

##  Appendix B Task Objectives for BIRCO

Figure 5: A framework for integrating LLMs into retrieval tasks Figure 6: An example query and LLM-generated decision criteria Dataset | Dataset Name | R/Q | Corpus Size | Relevancy | Recall@5 | Recall@20 | nDCG@10 | MRR@10 | MAP
---|---|---|---|---|---|---|---|---|---
BIRCO | DORIS-MAE | 18.2 | 100 | Multi-level | 14.7 | 42.2 | 72.0 | 14.3 | 40.5
ArguAna | 1.0 | 100 | Binary | 62.0 | 96.0 | 43.6 | 33.2 | 34.7
WhatsThatBook | 1.0 | 100 | Binary | 40.0 | 64.0 | 36.8 | 32.0 | 33.8
Clinical-Trial | 20.9 | 100 | 3-level | 10.6 | 37.8 | 29.4 | 34.4 | 37.4
RELIC | 1.0 | 100 | Binary | 15.0 | 43.0 | 11.2 | 8.9 | 12.2
| Average | 8.4 | 100 | - | 28.5 | 56.6 | 38.6 | 24.6 | 31.7
BEIR | MS MARCO | 1.1 | 8842K | Binary | - | - | 44.1 | - | -
TREC-COVID | 493.5 | 171K | 3-level | - | - | 78.3 | - | -
NFCorpus | 38.2 | 4K | 3-level | - | - | 36.1 | - | -
NQ | 1.2 | 2681K | Binary | - | - | 62.9 | - | -
HotpotQA | 2.0 | 5233K | Binary | - | - | 63.3 | - | -
FiQA | 2.6 | 58K | Binary | - | - | 38.6 | - | -
Touche | 19.0 | 383K | 3-level | - | - | 27.2 | - | -
CQADupStack | 1.4 | 457K | Binary | - | - | 39.4 | - | -
Quora | 1.6 | 523K | Binary | - | - | 88.2 | - | -
DBPedia | 38.2 | 4636K | 3-level | - | - | 42.4 | - | -
SCIDOCS | 4.9 | 26K | Binary | - | - | 20.1 | - | -
FEVER | 1.2 | 5417K | Binary | - | - | 65.0 | - | -
Climate-FEVER | 3.0 | 5417K | Binary | - | - | 22.4 | - | -
SciFact | 1.1 | 5K | Binary | - | - | 72.6 | - | -
| Average | 43.5 | 2418K | - | - | - | 50.0 | - | -
Table 8: Comparison between E5-v2’s performances on BIRCO and BEIR. R/D
indicates the average number of relevant documents per query. E5-v2’s original
paper only reports nDCG@10 for BEIR.

We report our prompts that describe task objectives. These prompts were
engineered on a small-scale development set for each dataset. The dev set for
DORIS-MAE has 40 queries. The dev set for Clinical-Trial has the rest 9
queries. All the other datasets have 50 queries in their dev sets. See
extraction procedure for triplets from these queries and their candidate pools
in Appendix D.

###  B.1 Task Objective for DORIS-MAE

"The query consists of users’ needs, leading to several research questions
that span a paragraph. Each candidate passage is an abstract from a scientific
paper. The objective of this information retrieval task is to identify the
abstract that most effectively meets the user’s needs in the query."

###  B.2 Task Objective for ArguAna

"This information retrieval (IR) task has a debate format where a topic is
given, and two directly opposing sides of arguments about this topic are
formed. A query is an argument that takes one side of this topic, focuses on a
particular point about this topic, and takes a stance (i.e., opinion,
position, view, perspective) about this particular point. A passage is an
argument that takes the opposing side of the same topic, focuses on the same
particular point about the same topic, and takes a directly opposing stance
that directly (i.e., no implying or inferring) refutes and attacks the query’s
stance regarding this particular point. Both query and passage might have
citations in them but these citations should not be considered in the scope of
this task. The overall goal of this specific information retrieval IR task is
to identify the central topic of the debate, to articulate the query’s stance,
and to find the passage that takes the opposing stance."

###  B.3 Task Objective for WhatsThatBook

"The query has this format: a user is trying to remember the name of a
specific book. The user only remembers some details about the book, such as
places, events, and some characters’ names. Some of the details are described
using informal language. The passage is a book description or summary of a
specific book. The passage typically describes the overall storyline of the
book and contains some details about the book. The objective of this
information retrieval IR task is for you to find the passage that has details
or components that holistically best match, explicitly or implicitly, the
details or components raised in the query. In other words, you need to find
the book description (i.e., the passage) that is most likely the book the user
is looking for in the query."

###  B.4 Task Objective for RELIC

"The query is a piece of literary analysis written by a scholar. In the query
(i.e., the excerpt from a literary analysis), one or more quotations from a
classic English novel is used as evidence to support the claims made by the
literary analysis. Quotations are identified by quotation marks. Now, one
quotation is being intentionally masked from the literary analysis (i.e., the
query), and replaced by the symbol [masked sentence(s)]. An important claim is
made in the preceding context and another important point is made in the
subsequent context surrounding the [masked sentence(s)]. The objective of this
information retrieval task is to find the most suitable passage that can be
used to **directly support** at least one claim made in the query (i.e., the
claim that is made in the preceding or the claim subsequent context
surrounding the [masked sentence(s)]) and is very natural to be plugged into
the [masked sentence(s)] part of the query. Obviously the most suitable
passage should **NOT REPEAT** or be contained in any part of the query. It
does not make sense to repeat the same or very similar things twice in
literary analysis."

###  B.5 Task Objective for Clinical-Trial

"The motivation of the Information Retrieval task is that clinical trials are
experiments conducted in the development of new medical treatments, drugs or
devices, and recruiting candidates for a trial is often a time-consuming and
resource-intensive effort. A query is a patient case report (either in the
form of electronic patient records or ad-hoc queries). A passage is a clinical
trial. This Information Retrieval task is to improve patient recruitment for
clinical trials. The overall goal of this specific information retrieval IR
task is to match eligible patients (the query) to clinical trials (the
passage) for recruitment."

##  Appendix C The Effects of Candidate Pool Construction

In order to make LLM-based systems’ evaluation on BIRCO cost-effective, we use
the mixture of ada-002 embedding model and BM25 algorithm to construct a
candidate pool of size 50 for each query in ArguAna, WhatsThatBook and RELIC.
We also randomly sample 100 queries from the above three datasets’
decontaminated set of queries, subject to a length constraint (i.e. queries
that have fewer than 50 tokens are filtered). For DORIS-MAE, we use the
original paper’s test set. For Clinical-Trial we use 50 decontaminated queries
and their original candidate pools as test set. The remaining 9 queries are
used as a dev set.

Please refer to Table 9 for the statistics about the test set and the whole
dataset. The reason that the ArguAna test set from BIRCO exhibits lower E5
performance than the entire ArguAna dataset is potentially due to shorter
queries being filtered out. Longer queries are possibly more challenging
queries. As a reference, the average query length for BIRCO’s ArguAna is 213
while the average query length for ArguAna’s whole set is 193.

Clinical-Trial test set is competitive with its complete set, which has 9 more
queries that include 4 contaminated queries. Potentially contaminated queries
(i.e. queries that GPT4 can answer without prior knowledge) are easier.

Conversely, on RELIC and WhatsThatBook, the test sets are easier, as indicated
by higher E5 performance. This is due to a significantly smaller candidate
pool size, which reduces the difficulty of the task.

Dataset | Test Set |  | Whole Dataset
---|---|---|---
| E5-v2 nDCG@10 | E5-v2 R@5 |  | E5-v2 nDCG@10 | E5-v2 R@5
ArguAna | 43.5 $\pm$ 3.2 | 62.0 $\pm$ 4.8 |  | 48.0 $\pm$ 1.0 | 59.7 $\pm$ 1.4
WhatsThatBook | 36.6 $\pm$ 4.0 | 39.9 $\pm$ 4.8 |  | 21.0 $\pm$ 2.8 | 24.7 $\pm$ 3.5
Clinical-Trial | 29.4 $\pm$ 2.7 | 10.5 $\pm$ 1.7 |  | 32.4 $\pm$ 2.6 | 13.1 $\pm$ 1.9
RELIC | 11.1 $\pm$ 2.6 | 14.9 $\pm$ 3.4 |  | 8.2 $\pm$ 0.4 | 10.1 $\pm$ 0.5
Table 9: Model performance on BIRCO’s test sets and the full original datasets

##  Appendix D Development Set

We construct the development set by extracting triplets from unused queries
and candidate pools, ensuring no overlaps between test and development set
queries and passages. The dev set for DORIS-MAE has 40 queries. The dev set
for Clinical-Trial has the rest 9 queries. All the other datasets have 50
queries in their dev sets. A triplet has the form of (query, relevant passage,
less relevant passage). The evaluation metric in development set is the number
of times a model can rank the relevant passage over the less relevant passage
given the query. In particular, for Clinical-Trial, we extract all (query,
score 2, score 1), (query, score 1, score 0), (query, score 2, score 0)
triplets, since it uses a 3-level relevancy scale. For DORIS-MAE, we extract
all (query, score x, score y) triplets such that $x-y>0.25$, $x,y>\textrm{max
score}/2$, since DORIS-MAE uses a multi-level, almost continuous relevancy
scale. The number of total possible triplets is reported in Table 1. In
practice, for time and cost reasons, we use less than 100 triplets from each
dataset for all prompt designs.

##  Appendix E Analysis of the Comparison between Datasets

Table 8 presents the performance of the E5-v2-L model on BIRCO and BEIR
datasets with respect to R@5, R@20, nDCG@10, MRR@10, and MAP. The maximum
possible Recall@k (i.e. theoretical upper-bound) is inversely proportional to
the number of relevant passages per query (R/D) if that number exceeds k.
Therefore, for the R@k metric to reach approximately 100%, the R/D for that
dataset should not exceed k. Consequently, for BIRCO, R@20 can reach a maximum
average of 94.3% and is suitable for coarse-grained comparison with other
datasets. In Table 2, we compare BIRCO with NQ with respect to R@20, as the
maximum R@20 for both benchmarks is near 100%. From Table 2, for example, we
see SimLM embedding model’s performance on NQ is 81.9% R@20, but only reaches
49.6% R@20, even though BIRCO’s max possible R@20 is 94%, this is still
conclusive evidence to indicate BIRCO is a much harder benchmark than NQ, at
least w.r.t SimLM.

For nDCG@10, the maximum value can always reach 100%. However, in a dataset
where many passages per query have a non-zero relevance score, such as DORIS-
MAE, which uses a fine-grained relevance scale, or TREC-COVID, where many
passages are relevant, the nDCG@10 value would be inflated. This is why the
random baseline on DORIS-MAE can achieve 49.9% nDCG@10.

Furthermore, a smaller candidate pool size (or corpus size if all queries
share the same candidate pool) would result in non-increasing task difficulty
and non-decreasing model performance.

We therefore conclude that the average nDCG@10 metric reported on BIRCO is
slightly elevated due to its smaller candidate pool size and its inclusion of
DORIS-MAE and Clinical-Trial. Similarly, the average nDCG@10 reported on BEIR
is slightly elevated due to several BEIR datasets having large R/D values.
E5-v2 achieves an average nDCG of 38.6% on BIRCO, which is significantly lower
than BEIR’s average 50.0% nDCG.

Table 8 also allows for the comparison of the nDCG@10 of any two datasets with
the same R/D value. For instance, RELIC only has 11.2% nDCG@10, suggesting it
is the most challenging task with respect to E5-v2, despite its smaller
candidate pool size.

##  Appendix F Definition of Facets in BIRCO Queries

This section elaborates on the definition of facets for each BIRCO query.

The DORIS-MAE dataset has a hierarchical, aspect-based structure. This
structure divides each query into a two-level tree of semantic components, or
facets. Consequently, we measure the degree of multi-facetedness by counting
the number of facets and sub-facets present in each query. For ArguAna, we use
GPT4 to count the number of unique arguments and their corresponding evidence
within a query. For WhatsThatBook, we use GPT4 to count the number of unique
details about a book provided in the user query. For Clinical-Trial, we use
GPT4 to count the number of unique symptoms reported in a patient’s case. For
RELIC, given that the [masked sentence] has both a preceding and a subsequent
context, we use GPT4 to determine if these two contexts convey the same idea.
If they do not, they are counted as two separate facets. To ensure the
accuracy of GPT4’s performance, we manually inspect its output. For full
results, see Table 4. For GPT4 prompts to calculate facets, see Appendix L.4.

##  Appendix G Scoring-based Framework Illustration

For clarity sake, see Figure 5 for an illustration of our proposed framework
and three distinct pipelines: Score+O (top), Reason+O (middle), Subtask+O
(bottom). See Figure 6 for an example set of decision criteria produced by
Reason+O.

Note, for each query, we select a single set of decision criteria which are
used across all candidate passages. An alternative, not investigated in this
paper, is to condition the decision criteria on both the query and the
passage, increasing the specificity of the criteria but also increasing their
variability.

##  Appendix H Error Analysis for Models

From manual inspection, we observed a few sources of common errors made by
LLMs. In the Reason+O pipeline, a low-quality set of decision criteria
typically led to low performance on this query.

From Table 7, we see Subtask+OGPT obtains better or comparable performance
compared to Reason+OGPT, indicating reducing task complexity and decomposing
task into sub-tasks offer more stability for GPT4. In particular, we see
significant improvements of Subtask+OGPT over Reason+OGPT on the ArguAna
dataset, showing that for certain complex tasks, a well-decomposed task
structure enables more accurate scoring and annotation.

###  H.1 Comparison vs Scoring

Both comparison-based approaches and scoring-based approaches provide
improvements in performance compared to previous models. The comparison-based
approach does not provide interpretable reasoning, but still achieves
comparable performances on DORIS-MAE and WhatsThatBook, and even slightly
better performances on RELIC, suggesting the potential value of exposing
multiple passages to LLM for comparison at the same time. On the other hand,
scoring-based approaches that annotate the decomposed sub-tasks offer more
stability and natural language reasoning across datasets.

###  H.2 Features that impact Task Difficulty

We examine several features of the datasets that may be contributing to the
difficulty of retrieval, with a focus on GPT4 with direct scoring. We
investigate various statistical features of the query to determine their
potential impacts on task difficulty. For each query in each dataset, we use
the average nDCG@10 of the query evaluated on Score+OGPT4, Reason+OGPT4, and
Subtask+OGPT4 as an indicator of the task difficulty in relation to this
query.

A higher average nDCG indicates that all three methods retrieved relevant
passages for the query, while a lower nDCG suggests these methods failed to do
so. The features we investigated include: query length, number of facets in a
query, the query’s lexical overlap with its relevant passage(s), the query’s
lexical overlap with all passages in the candidate pool, and the size of the
candidate pool.

We fit a linear regression for each feature and dataset. We found the
following statistically significant results: a positive linear relationship
between the lexical overlap of a query and its relevant passage(s), and the
nDCG scores for ArguAna (p<0.005) and WhatsThatBook (p<0.0005); a negative
linear relationship between the lexical overlap of a query and all passages,
and the NDCG scores for RELIC (p<0.0005). Note, even for these statistically
significant features, less than 15% variance could be explained by these
features, suggesting the complex nature of predicting task difficulty.

##  Appendix I Decontamination Filtering Procedure for BIRCO

The DORIS-MAE task requires identifying citations for specific computer
science papers. GPT4 was unable to do this successfully for any of the
queries. Similarly for ArguAna, the GPT4 generated counterargument does not
indicate it has seen or recalled the correct counterargument. For both cases,
human annotation is performed to determine whether contamination occurred.
First, GPT4 is prompted to respond to the DORIS-MAE query with a paper
abstract, and to respond to the ArguAna query with a counterargument. Then,
annotators analyze whether GPT4’s output paper abstract and counterargument
are semantically similar to the actual relevant abstract(s) and
counterargument. Annotators did not find any such feature to suggest GPT4’s
external knowledge can guide it to select relevant passages for queries in
DORIS-MAE and ArguAna.

For WhatsThatBook, we prompted GPT4 to generate its best guesses for the
forgotten book title based on the user query, and if any of GPT4’s generated
titles share significant textual similarity with the correct book title, the
query is removed. 38 queries out of 437 are removed. We also remove any
passage (i.e. book description) from the corpus if GPT4 can guess the book
title from the book description, suggesting it knows information beyond the
book description that could potentially help it answer the query. 4817 out of
8988 book descriptions are removed. Finally, we also remove a query if its
corresponding correct book description is removed, resulting in the removal of
an additional 238 queries.

For Clinical-Trial, we prompted GPT4 to generate the most relevant clinical
trial for the patient based on the patient’s case report. Human annotation is
conducted to compare GPT4’s answer with one of the relevant clinical studies.
In cases where there are multiple relevant passages, we choose the relevant
passage with the highest cosine similarity with GPT4’s result, calculated by
ada-002 embeddings. This is done to simplify and stabilize the human
annotation process. In 4 queries out of 59, we find knowledge similarity
between GPT4’s answer and the correct answer, i.e. they mention similar
treatment options. For example, GPT4 seems to have prior knowledge that
thyroid hormone replacement therapy is recommended to treat hypothyroidism.
These 4 queries are removed from the test set.

For RELIC, we prompted GPT4 to generate sentences that could be the masked
sentence based on the query which only contains the surrounding context of the
masked sentence. If any of the generated sentences share significant textual
similarity (i.e. a 5-gram overlap), we remove the query. 311 out of 3823
queries are removed.

We provide all prompts for decontamination in Appendix L.5

##  Appendix J Dataset License

  * •

DORIS-MAE: Provided under CC-BY-NC 4.0 license.

  * •

ArguAna: Provided under CC-BY-SA 4.0 license.

  * •

WhatsThatBook: Provided under ODC-BY license.

  * •

Clinical-Trial: Provided under CC-BY-SA 4.0 license.

  * •

RELIC: Provided under MIT license.

##  Appendix K Example (Query, Passage) Pairs from the Dataset

Please refer to Figure 7, 8, 9, 10, 11 for examples of query and passage pairs
from the datasets.

Figure 7: Example query and passage pair from DORIS-MAE Figure 8: Example
query and passage pair from ArguAna Figure 9: Example query and passage pair
from WhatsThatBook Figure 10: Example query and passage pair from Clinical-
Trial Figure 11: Example query and passage pair from RELIC

##  Appendix L All Prompting Strategies

This section includes all prompting templates used in the paper.

###  L.1 Prompts for Score+O

Please refer to Figure 12 for the prompt for Score+O.

Figure 12: Prompt for Score+O

###  L.2 Prompts for Reason+O

Please refer to Figure 13 for the prompt for Reason+O.

Figure 13: Prompt for Reason+O

###  L.3 Prompts for Score+O

Please refer to Figure 14 for the prompt for Subtask+O.

Figure 14: Prompt for Subtask+O

###  L.4 Prompts for Checking Multifacetedness

Please refer to Figure 15, 16, 17, 18 for prompts for checking the
multifacetedness in datasets.

Figure 15: Prompt for getting facets for ArguAna Figure 16: Prompt for getting
facets for WhatsThatBook Figure 17: Prompt for getting facets for Clinical-
trial Figure 18: Prompt for getting facets for RELIC

###  L.5 Prompts for Decontamination

Please refer to Figure 19, 20, 21, 22, 23 for prompts for decontamination in
BIRCO.

Figure 19: Prompt for DORIS-MAE decontamination Figure 20: Prompt for ArguAna
decontamination Figure 21: Prompts for WhatsThatBook decontamination Figure
22: Prompt for Clinical-trial decontamination Figure 23: Prompt for RELIC
decontamination

Please refer to Figure 24 for prompts for decontamination in BEIR.

Figure 24: Prompts for BEIR decontamination