#  BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information
Retrieval Models

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, Iryna
Gurevych
Ubiquitous Knowledge Processing Lab (UKP-TUDA)
Department of Computer Science, Technische Universität Darmstadt
www.ukp.tu-darmstadt.de

###### Abstract

Neural IR models have often been studied in homogeneous and narrow settings,
which has considerably limited insights into their generalization
capabilities. To address this, and to allow researchers to more broadly
establish the effectiveness of their models, we introduce beir (_Be
nchmarking_ _IR_), a _heterogeneous benchmark_ for information retrieval. We
leverage a careful selection of 17 datasets for evaluation spanning diverse
retrieval tasks including open-domain datasets as well as narrow expert
domains. We study the effectiveness of nine state-of-the-art retrieval models
in a _zero-shot_ evaluation setup on beir, finding that performing well
consistently across all datasets is challenging.

Our results show _BM25_ is a robust baseline and _Reranking_ -based models
overall achieve the best zero-shot performances, however, at high
computational costs. In contrast, _Dense_ -retrieval models are
computationally more efficient but often underperform other approaches,
highlighting the considerable room for improvement in their generalization
capabilities. In this work, we extensively analyze different retrieval models
and provide several suggestions that we believe may be useful for future work.
beir datasets and code are available at https://github.com/UKPLab/beir.

Figure 1: An overview of the diverse tasks and datasets present in beir.

##  1 Introduction

Many real-world NLP problems rely on a practical and efficient retrieval
component as a first step to find relevant information. Examples are open-
domain question-answering Chen et al. (2017), claim-verification Thorne et al.
(2018), and duplicate question detection Zhang et al. (2015). Traditionally,
retrieval has been dominated by lexical approaches like TF-IDF or BM25
Robertson and Zaragoza (2009). However, these approaches suffer from what is
known as lexical gap Berger et al. (2000) and only retrieve documents that
contain the keywords also present within the query. Further, queries and
documents are treated in a bag-of-words manner which does not take word
ordering into consideration.

Recently, deep learning and in particular pre-trained Transformer models like
BERT Devlin et al. (2018) have became popular in the information retrieval
space Lin et al. (2020). They overcome the lexical gap by mapping queries and
documents to a dense vector Guo et al. (2016); Lee et al. (2019); Karpukhin et
al. (2020); Guu et al. (2020); Gao et al. (2020); Liang et al. (2020); Ma et
al. (2021). The relevant documents for a given query are then retrieved using
(approximate) nearest neighbor search Johnson et al. (2017).

Another widely used approach involves re-ranking documents from the output of
a first-stage retrieval system Nogueira et al. (2019a, 2020); Nogueira and Cho
(2020); Khattab and Zaharia (2020). While dense retrieval approaches try to
overcome the (potential) lexical gap, re-ranking approaches aim to create a
better comparison of the retrieved documents. Different approaches can also be
combined together Ding et al. (2020); Gao et al. (2020); Luan et al. (2021).

Previous approaches were commonly trained on rather large datasets like the
Natural Questions (NQ) dataset Kwiatkowski et al. (2019) containing around
133k training examples or the MSMARCO dataset Nguyen et al. (2016) with more
than 500k training examples. Existing approaches have been shown to perform
well when evaluated in-domain or for similar tasks Nogueira and Cho (2020);
Karpukhin et al. (2020); Ding et al. (2020). However, large training corpora
are not available for most tasks and domains. As creating a large training
corpus can be expensive, it is not feasible to create such for most tasks and
domains. Hence, in most scenarios, we apply retrieval models in a zero-shot
setup, i.e. pre-trained models are applied out-of-the-box across new tasks and
domains. In this crucial setting, it remains unclear whether and how well
these retrieval models transfer or generalize to new tasks or domains.

In this work, we establish a new heterogeneous benchmark for Information
Retrieval called beir (_Be nchmarking_ _IR_) consisting of a broad range of
domains and tasks. Through beir, we systematically study the zero-shot
generalization capabilities of multiple neural retrieval approaches. Existing
benchmarks Guo et al. (2020); Petroni et al. (2020) have issues of a
comparatively narrow evaluation focusing either only on a single task or on a
certain domain. beir overcomes these shortcomings by covering 9 diverse
retrieval tasks with 17 datasets including a diverse set of domains, query and
document types.

We use beir to evaluate nine diverse state-of-the-art retrieval approaches.
From our analysis, we find that no approach consistently outperforms all
others. Further, we notice that the in-domain performance of a model does not
correlate with its generalization capabilities: models fine-tuned with the
same training data might generalize differently in beir.

In beir, we observe a trade-off between the zero-shot performances and the
computational cost: computationally expensive re-ranking models overall
perform the best. More computationally efficient dense models can, depending
on the task and domain, substantially underperform traditional lexical models
like BM25. In fact, BM25 is the third best performing approach on our
benchmark, beaten only by neural re-ranking approaches.

With beir, we take an important step towards broadly establishing and
improving the zero-shot capabilities of neural retrieval systems. Our
benchmark attempts to help the community better to understand when a retrieval
system could work or fail for a certain domain or task. Further, our benchmark
attempts to help researchers develop a universal neural retriever that
performs well across diverse tasks and domains.

We publicly release beir and an integration of various retrieval models in a
well-documented, easy to use and extensible open-source implementation. The
implementation is designed to allow easy integration of new tasks, datasets
and retrieval models.

##  2 Background and Related Work

To our knowledge, beir is the first zero-shot information retrieval benchmark.
Existing benchmarks do not evaluate retrieval in a zero-shot setting in depth,
they either focus over a single task, small corpora or on a certain domain.
This setting hinders for investigation of model generalization across diverse
set of domains and task types.

###  2.1 Existing Benchmarks

MultiReQA Guo et al. (2020) consists of eight Question-Answering (QA) datasets
and evaluates sentence-level answer retrieval given a question.MultiReQA only
tests a single task (question to answer-sentence retrieval) and five out of
eight datasets are on retrieval from Wikipedia. Further, MultiReQA evaluates
retrieval over rather small corpora: six out of eight tasks have less than
100k candidate sentences, which benefits dense retrieval over lexical as shown
previously in Reimers and Gurevych (2020). KILT Petroni et al. (2020) consists
of five knowledge-intensive tasks including a total of eleven datasets. The
tasks involve retrieval, but it is not the primary task. Further, KILT
retrieves documents only from Wikipedia.

###  2.2 Neural Retrieval

Information retrieval is the process of searching and returning relevant
documents for a query from a collection. In our paper, we focus on text
retrieval and use document as a cover term for text of any length in the given
collection and query for the user input, which can be of any length as well.

Traditionally, lexical approaches like TF-IDF and BM25 Robertson and Zaragoza
(2009) have dominated textual information retrieval. Recently, there is a
strong interest in using neural networks to improve or replace these lexical
approaches. In this section, we highlight a few neural-based approaches and we
refer the reader to Lin et al. (2020) for a detailed survey on recent
developments in neural retrieval.

Retriever-based Lexical approaches suffer from the lexical gap Berger et al.
(2000). To overcome this, earlier techniques proposed to improve lexical
retrieval systems with neural networks. Doc2query Nogueira et al. (2019b)
utilized a deep encoder-decoder architecture and generated possible queries
for a given document. These queries were appended to the document. DeepCT Dai
and Callan (2020) utilized BERT to learn relevant terms in a document and
repeats those terms at the end of the document, to achieve a higher term-
frequency weight. More recently, dense retrieval approaches have been proposed
to map queries and documents in a shared, dense vector space Gillick et al.
(2018). Here, a dual-encoder neural architecture based on pre-trained
Transformers showed strong performance for various tasks Guo et al. (2020);
Karpukhin et al. (2020); Liang et al. (2020); Ma et al. (2021). This dense
approach was recently extended by hybrid lexical-dense approaches which aims
to combine the strengths of both approaches Gao et al. (2020); Seo et al.
(2019); Luan et al. (2021). In contrast to the dense approaches, SPARTA Zhao
et al. (2020) learns sparse representations by combining contextualized and
non-contextualized embeddings.

Reranking-based Neural reranking approaches use the output of a first-stage
retrieval system, often BM25, and re-ranks the documents based on the
estimated relevancy between the query and the document. Significant
improvement in performance was achieved with the cross-attention mechanism of
BERT Nogueira and Cho (2020). A disadvantage of reranking with cross-attention
is the high computational overhead Reimers and Gurevych (2019). In contrast,
ColBERT Khattab and Zaharia (2020) computes contextualized embeddings
independently for queries and documents and uses an efficient maximum-
similarity operation for the relevance estimation.

##  3 BEIR Benchmark

Split ($\rightarrow$) |  | Train | Dev | Test | (Train + Dev + Test)
---|---|---|---|---|---
Task ($\downarrow$) | Domain ($\downarrow$) | Dataset ($\downarrow$) | Title | Relevancy | #Pairs | #Query | #Query | #Corpus | Avg. Docs / Q | Avg. Q Len | Avg. Doc Len
Passage-Retrieval | Misc. | MSMARCO | ✗ | Binary | 532,761 | —- | 6,980 | 8,841,823 | 1.1 | 5.96 | 55.98
Bio-Medical | Bio-Medical | (1) TREC-COVID | ✓ | 3-level | —- | —- | 50 | 171,332 | 493.5 | 10.60 | 160.77
Information | Bio-Medical | (2) NFCorpus | ✓ | 3-level | 110,575 | 324 | 323 | 3,633 | 38.2 | 3.30 | 232.26
Retrieval (IR) | Bio-Medical | (3) BioASQ | ✓ | Binary | 32,916 | —- | 500 | 14,914,602 | 4.7 | 8.05 | 202.61
Question | Wikipedia | (4) NQ | ✓ | Binary | 132,803 | —- | 3,452 | 2,681,468 | 1.2 | 9.16 | 78.88
Answering | Wikipedia | (5) HotpotQA | ✓ | Binary | 170,000 | 5,447 | 7,405 | 5,233,329 | 2.0 | 17.61 | 46.30
(QA) | Finance | (6) FiQA-2018 | ✗ | Binary | 14,166 | 500 | 648 | 57,638 | 2.6 | 10.77 | 132.32
Tweet-Retrieval | Twitter | (7) Signal-1M (RT) | ✗ | 3-level | —- | —- | 97 | 2,866,316 | 19.6 | 9.30 | 13.93
News-Retrieval | News | (8) TREC-NEWS | ✓ | 5-level | —- | —- | 57 | 594,977 | 19.6 | 11.14 | 634.79
Argument | Misc. | (9) ArguAna | ✓ | Binary | —- | —- | 1,406 | 8,674 | 1.0 | 192.98 | 166.80
Retrieval | Misc. | (10) Tóuche-2020 | ✓ | 6-level | —- | —- | 49 | 382,545 | 49.2 | 6.55 | 292.37
Duplicate-Question | StackEx. | (11) CQADupStack | ✓ | Binary | —- | —- | 13,145 | 457,199 | 1.4 | 8.59 | 129.09
Retrieval | Quora | (12) Quora | ✗ | Binary | —- | 5,000 | 10,000 | 522,931 | 1.6 | 9.53 | 11.44
Entity-Retrieval | Wikipedia | (13) DBPedia | ✓ | 3-level | —- | 67 | 400 | 4,635,922 | 38.2 | 5.39 | 49.68
Citation-Prediction | Scientific | (14) SCIDOCS | ✓ | Binary | —- | —- | 1,000 | 25,657 | 4.9 | 9.38 | 176.19
| Wikipedia | (15) FEVER | ✓ | Binary | 140,085 | 6,666 | 6,666 | 5,416,568 | 1.2 | 8.13 | 84.76
Fact Checking | Wikipedia | (16) Climate-FEVER | ✓ | Binary | —- | —- | 1,535 | 5,416,593 | 3.0 | 20.13 | 84.76
| Scientific | (17) SciFact | ✓ | Binary | 920 | —- | 300 | 5,183 | 1.1 | 12.37 | 213.63

Table 1: Statistics of all the tasks, domains and datasets included in beir.
Few datasets contain documents without titles. Relevancy column indicates the
relation between the query and document: binary (relevant, irrelevant) or
further graded into sub-levels. Avg. Docs/Query column indicates the average
relevant documents per question.

The selection of diverse datasets is an integral part of a benchmark.
Including a diverse set of challenging datasets is essential to infer broadly
the applicable findings from the observed benchmark results. Furthermore, the
benchmark should cover challenging tasks to support future work in that area
Gehrmann et al. (2021).

To collect the tasks and datasets with the desired properties, a selection
methodology is crucial. For beir, the methodology is motivated by the
following three factors:

##### (1) Diversity in Tasks

Information retrieval is a broad task and the lengths of queries and indexed
documents can differ extremely between tasks. Sometimes, queries are short,
like a keyword, while in other cases, they can be long like a complete news
article. Similarly, indexed documents can sometimes be long, and for other
tasks, short like a tweet. In beir, we aim to include vastly different tasks
with different properties for queries and documents.

##### (2) Diversity in Domains

Information retrieval systems are applied in all types of textual-based
domains. From broad and generic domains, like Wikipedia, to highly specialized
ones such as scientific publications of a specialised field. Hence, we include
tasks or datasets which provide a representation of real-world problems and
are diverse ranging from generic to specialized domains.

##### (3) Challenging Datasets

The difficulty of a dataset has to be sufficient. If a task is easily solved
by any algorithm, it will not be useful to differentiate the various models
used for evaluation. We evaluate several datasets based on existing model
performances and literature. We select tasks which we believe are challenging
and are not yet fully solved with existing approaches and where model
performances could be further improved.

###  3.1 Evaluation Tasks and Datasets

Following the selection criteria, we include seventeen English111We find it
also necessary to benchmark multilingual and cross-lingual tasks in beir,
which we keep for future work. and evaluation datasets that span over nine
heterogeneous tasks. As the majority of the evaluated approaches have been
trained on the MSMARCO dataset, we also report performances on this dataset,
but don’t include the outcome in our zero-shot comparison.

Figure 1 shows each task, dataset and domain together with the type of query
and document. Table 1 shows detailed statistics of the tasks and datasets in
beir. A majority of the retrieval tasks are binary, i.e. a search result can
be relevant or irrelevant, and a few contain fine-grained relevancy
judgements. Some datasets have few (< 2) annotated relevant documents for a
question, while others contain multiple relevant documents. Not all datasets
include a training or a development split denoting the practical importance
for zero-shot retrieval techniques and model performances.

In Appendix C, we motivate each task and dataset included within the
benchmark. We would refer the reader to this section for a detailed overview
on the tasks and datasets in the benchmark. We additionally provide
downloadable links (Table 6) and examples (Table 8) for all datasets.

###  3.2 Dataset and Diversity Analysis

As shown in Table 1, the datasets present in beir are from various domains
including Wikipedia, scientific publications, Twitter, News, online user
communities, and more. To measure the domain similarity between the datasets,
we compute a pairwise weighted Jaccard similarity Ioffe (2010) on unigram word
overlap between the document collections. For understanding how the similarity
score was calculated, refer to Appendix D. The pairwise results can be found
in Figure 5 in the appendix. We use these pairwise similarity scores and
cluster the datasets using the force-directed placement algorithm. The result
is shown in Figure 2. Nodes close in this graph have a high word overlap,
while nodes far away in the graph have a low overlap. From Figure 5, we
observe a rather low weighted Jaccard word overlap between the different
domains, indicating that beir is a challenging benchmark and to perform well
on it, retrieval approaches must generalize well to diverse text types.

Figure 2: A 2D-visualization of the pairwise weighted jaccard similarity
scores (Figure 5) using the force-directed placement algorithm with NetworkX
Hagberg et al. (2008). We color datasets differently for different domains.

##  4 Experimental Setup

We now describe our experimental setup which we use to benchmark different
state-of-the-art retrieval architectures and models.

###  4.1 Retrieval Models

In our work, we include a broad spectrum of retrieval models which we believe
cover diverse settings and architectures. We focus on selecting models based
on its performance, contemporaneity and popularity. We try to include publicly
available pre-trained checkpoints from a practical point of view, which could
lead to different model configurations. Unless specified otherwise, we
concatenate the title and passage together with a single white-space across
the models. We group the models based on their retrieval architecture: we
select one lexical, one sparse, six dense and two re-ranking models, leading
to a total of nine different retrieval approaches. Besides the included ones,
in future, models can be easily incorporated and compared within the
benchmark. All model checkpoints can be found in Table 7.

####  4.1.1 Lexical Model

##### BM25

Robertson and Zaragoza (2009) is a commonly-used bag-of-words retrieval
function based on token-matching between two high-dimensional sparse vectors
with TF-IDF token weights. Due to its strong and efficient performance, it is
the de-facto industrial standard. We use the implementation available on
Elasticsearch222https://www.elastic.co with the default settings. We index the
title (if available) and passage of each document as separate text fields for
retrieval.

####  4.1.2 Sparse Model

##### SPARTA

Zhao et al. (2020) computes the similarity between the non-contextualized word
piece embeddings from BERT and the contextualized word piece embeddings for a
given document. Due to the finite size of 30k non-contextualized embeddings,
the similarity scores can be pre-computed for a given document, which results
in a 30k dimensional sparse vector. As the original implementation is not
publicly available, we re-implement the approach. We fine-tune a DistilBERT
Sanh et al. (2020) model on MSMARCO Nguyen et al. (2016) and use sparse-
vectors with 2,000 non-zero entries.

Model ($\rightarrow$) | Lexical | Sparse | Dense / Neural | Reranking
---|---|---|---|---
Dataset ($\downarrow$) | BM25 | SPARTA | USE-QA | DPR | ANCE | SBERT | GenQ | BM25+CE | ColBERT
MSMARCO | 0.218 | 0.351‡ | 0.259 | 0.177 | 0.388‡ |  0.389‡ |  0.389‡ | 0.384‡ | 0.425‡
TREC-COVID | 0.616 | 0.538 | 0.528 | 0.332 | 0.654 | 0.482 | 0.554 | 0.667 | 0.677
BioASQ | 0.514 | 0.351 | 0.093 | 0.127 | 0.306 | 0.295 | 0.351 | 0.489 | 0.474
NFCorpus | 0.297 | 0.301 | 0.252 | 0.189 | 0.237 | 0.257 | 0.293 | 0.303 | 0.305
NQ | 0.310 | 0.398 | 0.180 | 0.474‡ | 0.446 | 0.450 | 0.360 | 0.516 | 0.524
HotpotQA | 0.601 | 0.492 | 0.258 | 0.391 | 0.456 | 0.513 | 0.497 | 0.701 | 0.593
FiQA-2018 | 0.239 | 0.198 | 0.264 | 0.112 | 0.295 | 0.258 | 0.284 | 0.326 | 0.317
Signal-1M (RT) | 0.388 | 0.252 | 0.241 | 0.155 | 0.249 | 0.261 | 0.257 | 0.308 | 0.274
TREC-NEWS | 0.371 | 0.258 | 0.340 | 0.161 | 0.382 | 0.367 | 0.369 | 0.430 | 0.393
ArguAna | 0.441 | 0.279 | 0.048 | 0.175 | 0.415 | 0.429 | 0.517 | 0.313 | 0.233
Tóuche-2020 | 0.605 | 0.231 | 0.252 | 0.127 | 0.284 | 0.249 | 0.226 | 0.378 | 0.275
CQADupStack | 0.316 | 0.257 | 0.236 | 0.153 | 0.296 | 0.306 | 0.339 | 0.348 | 0.350
Quora | 0.742 | 0.630 | 0.767 | 0.248 | 0.852 | 0.855 | 0.853 | 0.778 | 0.854
DBPedia | 0.288 | 0.314 | 0.238 | 0.263 | 0.281 | 0.339 | 0.320 | 0.380 | 0.392
SCIDOCS | 0.156 | 0.126 | 0.104 | 0.077 | 0.122 | 0.133 | 0.148 | 0.154 | 0.145
FEVER | 0.648 | 0.596 | 0.546 | 0.562 | 0.669 | 0.670 | 0.641 | 0.793 | 0.771
Climate-FEVER | 0.179 | 0.082 | 0.113 | 0.148 | 0.198 | 0.205 | 0.220 | 0.246 | 0.184
SciFact | 0.620 | 0.582 | 0.312 | 0.318 | 0.507 | 0.531 | 0.592 | 0.524 | 0.671
AVERAGE | 0.419 | 0.346 | 0.280 | 0.233 | 0.391 | 0.388 | 0.401 | 0.447 | 0.437

Table 2: In-domain and Zero-shot retrieval performances on beir datasets. All
scores denote nDCG@10. The best retrieval performance on a given dataset is
marked in bold, and the second best performance is underlined. Corresponding
Recall@100 performances can be found in Table 9. $\ddagger$ indicates the in-
domain performances.

####  4.1.3 Dense Models

##### USE-QA

Yang et al. (2020) is a two-tower bi-encoder encoding the query and document
separately. The model has been fine-tuned on multiple online question-
answering forums and the SNLI Bowman et al. (2015) corpus. The documents are
encoded as a sentence-level input with additional context information and are
retrieved using cosine-similarity. We utilize the English USE-QA
model333https://tfhub.dev/google/universal-sentence-encoder-qa/3. We
substitute the title as the input; passage as the surrounding context for the
original response encoder. For datasets without document titles, we substitute
the passage in both the fields.

##### DPR

Karpukhin et al. (2020) is a two-tower bi-encoder. The model uses [CLS] token
pooling and retrieves similar documents using dot-product. We experimented and
found the Multi model444https://huggingface.co/transformers/model_doc/dpr.html
to perform better compared to the NQ model fine-tuned only on Natural
Questions Kwiatkowski et al. (2019). The Multi-DPR model was fine-tuned on
four QA datasets: (NQ; Kwiatkowski et al., 2019), (TriviaQA; Joshi et al.,
2017), (WebQuestions; Berant et al., 2013) and (CuratedTREC; Baudiš and
Šedivỳ, 2015). The model requires a title for retrieval and concatenates the
title and passage together with a [SEP] token. However, we keep the title
field empty for evaluation if unavailable for a dataset.

##### ANCE

Xiong et al. (2020) is a siamese bi-encoder encoding the query and document
together in a common embedding space. The model uses [CLS] token pooling and
retrieves similar documents using dot-product. ANCE constructed negatives from
an Approximate Nearest Neighbor (ANN) index of the corpus, which in parallel
was updated to select negative training instances during fine-tuning of the
model. We use the open-sourced RoBERTa Liu et al. (2019)
model555https://github.com/microsoft/ANCE fine-tuned on MSMARCO Nguyen et al.
(2016) for 600K steps.

##### SBERT

Reimers and Gurevych (2019) is a siamese bi-encoder using mean pooling for
encoding and cosine-similarity for retrieval. We fine-tune a DistilBERT Sanh
et al. (2020) model666https://www.sbert.net/docs/pretrained_models.html on
MSMARCO Nguyen et al. (2016). During fine-tuning, we include in-batch
negatives Gillick et al. (2018) together with hard-negatives as provided by
MSMARCO. Then, for all train queries we retrieve the top-20 most similar
documents from a collection of 3 million MSMARCO passages. We use an ELECTRA
Clark et al. (2020) cross-encoder model fine-tuned on MSMARCO (subsubsection
4.1.4) to filter out false positives and only keep (query, passage) pairs with
a relevancy score below 0.1. We use these pairs as additional hard negatives
and fine-tune the model for 10 more epochs.

##### GenQ

Similar to past work Liang et al. (2020); Ma et al. (2021), we propose an
unsupervised domain-adaption approach for dense retrieval models using
synthetic queries. First, we fine-tune a T5-base Raffel et al. (2020) model to
generate queries given a passage. We use the MSMARCO dataset and train for 2
epochs. Then, for a target corpus we generate 5 queries for each document
using a combination of top-k and nucleus-sampling (top-k: 25; top-p: 0.95).
Due to resource constraints, we cap the maximum number of target documents in
each dataset to 100K. We found the T5 model to perform better compared to BART
Lewis et al. (2020) and our decoding setting better as compared to beam-
search. For retrieval, we continue to fine-tune the SBERT model (subsubsection
4.1.3) on the synthetic queries and document pairs. Note, this creates an
independent model for each task.

####  4.1.4 Reranking Models

##### BM25 + CE

We retrieve the top-100 relevant documents using BM25 and rerank them using a
ELECTRA cross-encoder model Clark et al. (2020). The model has been fine-tuned
on MSMARCO Nguyen et al. (2016) using the approach described by Nogueira and
Cho (2020).

##### ColBERT

Khattab and Zaharia (2020) independently encodes the query and the document
into a bag of token embeddings. The model first top-k retrieves candidates
through approximate nearest neighbor search using
faiss777https://github.com/facebookresearch/faiss Johnson et al. (2017) and
then reranks those by computing the sum of maximum cosine similarity between
the query and document token embeddings. We use the provided
implementation888https://github.com/stanford-futuredata/ColBERT and retrieve
the top-100 relevant documents using faiss. For reranking, we use a bert-base-
uncased model fine-tuned on the MSMARCO dataset for 300K steps.

###  4.2 Evaluation Metric

Depending upon the nature and requirements of real-world applications,
retrieval tasks can be either be precision or recall focused. To obtain
comparable results across models and datasets in beir, we argue that it is
important to leverage a single evaluation metric that can be computed
comparably across all tasks. Decision support metrics such as Precision and
Recall which are both rank unaware are not suitable. Binary rank-aware metrics
such as MRR (Mean Reciprocal Rate) and MAP (Mean Average Precision) fail to
evaluate tasks with graded relevance judgements. We find that Normalised
Cumulative Discount Gain (nDCG@k) provides a good balance suitable for both
tasks involving binary and graded relevance judgements.999Note that we
additionally provide other metrics like Precision, Recall and MAP (Mean
Average Precision) in beir. We refer the reader to Wang et al. (2013) for
understanding the theoretical advantages of the metric. For our experiments,
we utilize the Python interface of the official TREC evaluation
tool101010https://github.com/cvangysel/pytrec_eval Van Gysel and de Rijke
(2018) and compute nDCG@10 for all datasets.

Figure 3: Comparison of zero-shot neural retrieval performances with BM25.
Only Reranking-based models, i.e., BM25+CE and ColBERT outperform BM25 on more
than half of the evaluation datasets in beir.

##  5 Zero-Shot Evaluation Results

In Table 2, we report the nDCG@10 results for the retrieval approaches on a
single in-domain dataset (MSMARCO) and 17 zero-shot datasets. Figure 3 shows
on how many datasets the neural approaches outperform (green) and underperform
(red) our BM25 baseline in the zero-shot evaluation setup. Overall, we observe
that in-domain performance is not a good indicator of the generalization
capability as discussed below:

Lexical Model Although BM25 underperforms neural approaches in the in-domain
evaluation setting, the zero-shot results indicate that it is a strong
baseline for generalization. It even achieves the best performance in 4 out of
17 tasks.

Sparse Model SPARTA outperforms BM25 by a large margin on the in-domain
dataset, but performs poorly on beir datasets. It beats BM25 only for 3 out of
17 datasets with strong performance drops for other datasets.

Dense Models Dense retrieval approaches perform well only for certain
datasets, while for others, there are strong performance drops compared to
BM25. DPR and USE-QA perform poorly on nearly all the datasets. In contrast,
SBERT and ANCE perform well for 6 and 7 datasets respectively. As we will show
in section 7, these approaches fine-tuned on the MSMARCO dataset perform well
if the domain is similar to the training dataset. Interestingly, GenQ, which
further fine-tunes SBERT on synthetically generated query data is able to
capture vital domain information. It outperforms SBERT on specialised domains
such as scientific, finance or StackExchange domains. However, it
underperforms SBERT on tasks over broad domains such as Wikipedia. We
intuitively believe this is due to catastrophic forgetting during fine-tuning
of GenQ with synthetic queries.

Reranking Models The experiments show strong results for both the re-ranking
strategies: BM25 + CE and ColBERT. Both techniques achieve the best
performances. They both outperform BM25 in 11 out of 17 datasets. This
indicates re-ranking methods involving a cross-encoder generalize well to
unseen datasets and domains, which is in line with the findings of other
recent work (Akkalyoncu Yilmaz et al., 2019; Rücklé et al., 2020; Thakur et
al., 2020).

DBPedia (1 Million) | Retrieval Latency | Index
---|---|---
Rank | Model | Dim. | GPU | CPU | Size
(1) | BM25+CE | 768 | 550ms | 7100ms | 0.4GB
(2) | ColBERT | 128 | 350ms | – | 20GB
(3) | BM25 | – | – | 20ms | 0.4GB
(4) | GenQ | 768 | 14ms | 125ms | 3GB
(5) | ANCE | 768 | 20ms | 275ms | 3GB
(6) | SBERT | 768 | 14ms | 125ms | 3GB
(7) | SPARTA | 2000 | – | 20ms | 12GB
(8) | USE-QA | 512 | 35ms | 75ms | 2GB
(9) | DPR | 768 | 19ms | 230ms | 3GB

Table 3: Estimated retrieval latency for a single query and index sizes for 1
million documents in DBPedia. Ranked from best to worst zero-shot performance.
Lower the latency and size is better.

##  6 Retrieval Latency and Index Sizes

In this section, we compare different retrievers from a practical viewpoint.
Models need to potentially compare a single query against millions of
documents, hence, a high computational speed for retrieving results in real-
time is desired. Pre-computation and storing documents in indexes is vital at
inference or during deployment on production systems. For our comparison, we
randomly sample 1M documents from DBPedia Hasibi et al. (2017). We evaluate
retrieval for dense models using exact-search with an exception for ColBERT
where we followed the original implementation Khattab and Zaharia (2020). For
both reranking approaches we evaluate performances for top-100 hits. CPU
performances were measured with an 8 core Intel Xeon Platinum 8168 CPU @
2.70GHz. GPU performances were measured using a single Nvidia Tesla V100, CUDA
11.0 and cuDNN. Results are shown in Table 3.

##### Retrieval Latency

The good generalization capabilities of the both reranking approaches come at
the cost of high computational overhead, being slowest at inference. High
latency of 0.4 and 0.6 seconds are neither desirable nor practical for many
applications. In contrast, dense retrieval models are about 20-30x faster
compared to the reranking models, enabling real-time responses. On CPU, the
simpler model architectures such as SPARTA and BM25 dominate in terms of
speed. Note, we did not specifically speed-optimize the implementations and
used them as provided. With further optimizations, latency can potentially be
reduced for all approaches, however the observed behaviour of the approaches
will generally be the same.

##### Index Sizes

Besides speed, the size of the index is a critical factor as well since it is
usually stored in memory. Out of all the models, BM25 requires the smallest
memory to store the index. SPARTA has the second largest index sizes, as it
stores a 2000 dimensional sparse vector for each document, and ColBERT has the
largest index size, as it stores a 128 dimensional dense vector for each
document token. Index size is especially relevant for larger document
collections: ColBERT stores an index of 900GB size for BioASQ with around 15M
documents, while BM25 uses 18GB to store its index.

##  7 Analysis

This section attempts to analyse a few retrieval techniques in depth and
uncover interesting insights observed in our zero-shot experiments.

##### Dataset Annotation Biases

The biggest performance drops in both dense and re-ranking techniques compared
to BM25 are found in BioASQ, Signal-1M (RT) and Tóuche-2020 datasets. One
possible reason lies in the construction of the datasets itself. In
BioASQ111111Please see details at http://participants-
area.bioasq.org/general_information/Task8b/, document candidates are retrieved
for annotation via term-matching with boosting tags Tsatsaronis et al. (2015).
Further, the annotation depth is shallow (approximately 5 relevant articles
per query), whereas the total number of documents is around 15M, which favours
lexical term-matching systems like BM25 Liang et al. (2020). Creation of
Signal-1M (RT), involved retrieving tweets for a query using 8 different
retrieval systems for annotation. 7 out of these 8 techniques relied upon
term-matching signals and we suspect this causes a bias towards lexical
annotated documents Suarez et al. (2018). With Tóuche-2020, we believe one of
the reasons lies in the skewed length distribution of relevant documents as
shown in Figure 4. All neural approaches have limitations with document
lengths as they have limit of 512 word pieces. On the other hand BM25 greatly
profits from longer documents as more keywords occur in them thereby
increasing the chances of lexical overlap.

Figure 4: Annotated relevant document lengths (in words) for Tóuche-2020
Bondarenko et al. (2020). Majority of the highly relevant documents are
greater than 300-350 words providing a better score with BM25.

##### Domain Shift

The MSMARCO Nguyen et al. (2016) dataset is used for fine-tuning six out of
the nine retrieval models. Similar to Shah et al. (2018) setup, we calculate
the proportion of n-grams shared between the source (MSMARCO) and the target
datasets present in beir. We calculate the overlap using a Weighted Jaccard
Similarity score Ioffe (2010). From Table 4, we observe that _dense_ models
are unable to generalize on target datasets with a small weighted Jaccard
similarity score with MSMARCO, thereby under-performing _BM25_ on these
datasets. To analyse the correlation, we calculate the Spearman’s rank
correlation ($\rho$) between the difference in performances of the dense and
lexical-based models ranked accordingly from high to low unigram
overlap.121212We drop BioASQ, Signal-1M (RT) and Tòuche-2020 datasets in the
calculation of Spearman’s rank correlation due to annotation biases as
discussed in section 7. We achieve a correlation score ($\rho$) of 0.77 for
SBERT and 0.71 for ANCE denoting a high correlation between domain overlap and
the performance of dense retrievers.

Dataset | Avg. Jaccard Sim (%) | nDCG@10
---|---|---
Target ($\downarrow$) | Unigrams | Bigrams | BM25 | ANCE | SBERT
NQ | 47.27 | 20.49 | 0.310 | 0.446 | 0.450
Quora | 39.75 | 8.69 | 0.742 | 0.852 | 0.855
TREC-NEWS | 38.90 | 14.77 | 0.371 | 0.382 | 0.367
Tóuche-2020 | 37.02 | 14.05 | 0.605 | 0.284 | 0.249
FiQA-2018 | 35.95 | 9.78 | 0.239 | 0.295 | 0.258
FEVER | 34.79 | 14.99 | 0.648 | 0.669 | 0.670
Climate-FEVER | 34.79 | 14.99 | 0.179 | 0.198 | 0.205
ArguAna | 32.92 | 4.78 | 0.441 | 0.415 | 0.429
Signal-1M (RT) | 32.15 | 8.64 | 0.388 | 0.249 | 0.261
HotpotQA | 30.87 | 12.55 | 0.601 | 0.456 | 0.513
CQADupStack | 30.64 | 10.16 | 0.316 | 0.296 | 0.306
DBPedia | 30.16 | 11.88 | 0.288 | 0.281 | 0.339
SCIDOCS | 27.92 | 5.31 | 0.156 | 0.122 | 0.133
BioASQ | 27.10 | 10.03 | 0.514 | 0.306 | 0.295
TREC-COVID | 26.80 | 7.36 | 0.616 | 0.654 | 0.482
NFCorpus | 23.45 | 3.09 | 0.297 | 0.237 | 0.257
SciFact | 22.16 | 2.92 | 0.620 | 0.507 | 0.531

Table 4: Domain shift analysis with MSMARCO Nguyen et al. (2016). Datasets are
sorted from high to low unigram weighted Jaccard similarity Ioffe (2010). ANCE
and SBERT outperform BM25 (green) more often when MSMARCO and the target
dataset share a high domain overlap.

##### Dot Product vs. Cosine Similarity

Dense models require a similarity function to retrieve relevant documents for
a given query within an embedding space. This similarity function is also used
during training dense models with the InfoNCE van den Oord et al. (2019) loss:

$\mathcal{L}_{q}=-\log\frac{\exp(\tau\cdot\text{sim}(q,d_{+}))}{\sum_{i=0}^{n}\exp(\tau\cdot\text{sim}(q,d_{i}))}$

using $n$ in-batch negatives for each query $q$ and a scaling factor $\tau$.
where $d_{+}$ denotes the relevant (positive) document for query $q$. Commonly
used similarity functions ($\text{sim}(q,d)$) are cosine-similarity (used by
USE-QA and SBERT) and dot-product (used by ANCE and DPR).

As far as we know, the differences between the two similarity functions were
not systematically studied before. The performances of SBERT and ANCE with in-
domain evaluation on MSMARCO are nearly identical. However, in the zero-shot
evaluation setup, the two models can perform vastly different: on TREC-COVID,
ANCE outperforms SBERT by 17.2 points, whereas SBERT outperforms ANCE by 9.7
points on HotpotQA.

We re-ran the training of SBERT (subsubsection 4.1.3) and only changed the
similarity function from cosine-similarity to dot-product. As shown in Table 5
and Table 10 in the appendix, we observe significant performance differences
for some datasets. For TREC-COVID, the model trained with dot-product achieves
the biggest improvement with 15.3 points, while for a majority of other
datasets, it performs worse than the cosine-similarity model.

We observe that these (nearly) identical models retrieve documents with vastly
different lengths as shown in the violin plots in Table 5 and Figure 6 in the
appendix. For all datasets, we find the cosine-similarity model to prefer
shorter documents over longer ones. This is especially severe for TREC-COVID:
a large fraction of the scientific papers (approx. 42k out of 171k) consist
only of publication titles without an abstract. The cosine-similarity model
prefers retrieving these documents. In contrast, the dot-product model
primarily retrieves longer documents, i.e., publications with an abstract.
Cosine-similarity uses vectors of unit length, thereby having no notion of the
encoded text length. In contrast, for dot-product, longer documents can result
in vectors with higher magnitudes which can yield higher similarity scores for
a query.

As we previously saw in Figure 4, relevance scores are not uniformly
distributed over document lengths: for some datasets, longer documents are
annotated with higher relevancy scores, while in others, shorter documents
are. This can be either due to the annotation process, e.g., the candidate
selection method prefers short or long documents, or due to the task itself,
where shorter or longer documents could be more relevant to the user
information need. Hence, it can be more advantageous to train a model with
either cosine-similarity or dot-product depending upon the nature and needs of
the specific task.

TREC-COVID | Signal-1M (RT) | FEVER
---|---|---
Cosine-Sim. | Dot-Prod. | Cosine-Sim. | Dot-Prod. | Cosine-Sim. | Dot-Prod.
0.482 | 0.635 | 0.261 | 0.243 | 0.670 | 0.685

Table 5: Violin plots Hintze and Nelson (1998) of document lengths for the
top-10 retrieved hits and nDCG@10 scores using SBERT (subsubsection 4.1.3)
model trained with either cosine similarity (blue, top) or dot product
(orange, bottom).

##  8 Conclusion

In this work, we presented beir: a heterogeneous benchmark for information
retrieval. In contrast to the existing benchmarks, we provided a broader
selection of target tasks ranging from narrow expert domains to open domain
datasets. We included 9 different retrieval tasks spanning 17 diverse
datasets. beir marks the first large scale benchmark for the evaluation of
zero-shot transfer capabilities of retrieval models to a wide range of
settings.

We broadly studied the effectiveness of nine different retrieval models on
beir. We show that _BM25_ is a strong baseline for zero-shot evaluation. We
demonstrated, that in-domain performance cannot predict how well an approach
will generalize in a zero-shot setup. Overall, we demonstrate _Reranking_
-based models generalize well across all the datasets, at the cost of slower
retrieval times. _Dense_ models outperform _BM25_ when a high domain overlap
is present between the training dataset and the evaluated target dataset.
_Dense_ models, similar to BM25, are fast in retrieval and produce light-
weight indexes.

##  9 Limitations of BEIR

Even though we cover a wide range of tasks and domains in beir, no benchmark
is perfect and has its limitations. Making those explicit is a critical point
in understanding the results on the benchmark and, for future work, to improve
up-on the benchmark.

##### Multilingual Tasks

Currently, our benchmark is focused on English tasks. We would like to extend
beir to include multi- and cross-lingual tasks and models.

##### Long Document Retrieval

Most of our tasks have average document lengths up-to a few hundred words
roughly equivalent to a few paragraphs. Including tasks that require the
retrieval of longer documents would be highly relevant. However, as
transformer-based approaches often have a length limit of 512 word pieces, a
fundamental different setup would be required to compare approaches.

##### Multi-factor Search

Until now, we focused on pure textual search in beir. In many real-world
applications, further signals are used to estimate the relevancy of documents,
such as PageRank Page et al. (1999), recency Dong et al. (2010), authority
score Kleinberg (1999) or user-interactions such as click-through rates
Radlinski et al. (2008). The integration of such signals in the tested
approaches is often not straight-forward and is an interesting direction for
research.

##### Multi-field Retrieval

Retrieval can often be performed over multiple fields. For example, for
scientific publication we have the title, the abstract, the document body, the
authors list, and the journal name. So far we focused only on datasets that
have one or two fields.

##### Personalised Search

Document relevancy can vary with a user personal interest or preference Ghorab
et al. (2013). We currently rank relevant documents from a single global
annotated relevant or not-relevant label. In future, we would like to
incorporate personal preferences within the benchmark.

##  10 Future Work

In this work, we presented beir together with results and analysis for nine
different retrieval methods. However, beir offers plenty more opportunities
for future work. Some of the most interesting are: Understanding why
approaches generalize differently across tasks and how to improve the zero-
shot performance for approaches? Further, it would be interesting to be able
to predict when a certain pre-trained model performs well and when it does
not? As we have shown, re-ranking approaches generally perform better than
other approaches, but come at a higher computational cost: Understanding how
to reduce the computational overhead while maintaining the good generalization
capability would be of high practical value.

## Acknowledgement

This work has been supported by the German Federal Ministry of Education and
Research (BMBF) under the promotional reference 03VP02540 (ArgumenText), by
the German Research Foundation through the German-Israeli Project Cooperation
(DIP, grant DA 1600/1-1 and grant GU 798/17-1) and has been funded by the
German Federal Ministry of Education and Research and the Hessian Ministry of
Higher Education, Research, Science and the Arts within their joint support of
the National Research Center for Applied Cybersecurity ATHENE. We would like
to thank Tilman Beck, Kexin Wang and Jorge Cardona for their feedback.

## References

  * Ahmad et al. (2019) Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019.  ReQA: An evaluation for end-to-end answer retrieval models.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 137–146, Hong Kong, China. Association for Computational Linguistics.
  * Akkalyoncu Yilmaz et al. (2019) Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019.  Cross-domain modeling of sentence-level evidence for document retrieval.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019)_ , pages 3490–3496. Association for Computational Linguistics.
  * Baudiš and Šedivỳ (2015) Petr Baudiš and Jan Šedivỳ. 2015.  Modeling of the question answering task in the yodaqa system.  In _International Conference of the Cross-Language Evaluation Forum for European Languages_ , pages 222–228. Springer.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
  * Berger et al. (2000) Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. 2000.  Bridging the lexical chasm: statistical approaches to answer-finding.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 192–199.
  * Bondarenko et al. (2020) Alexander Bondarenko, Maik Fröbe, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, and Matthias Hagen. 2020.  Overview of Touché 2020: Argument Retrieval.  In _Working Notes Papers of the CLEF 2020 Evaluation Labs_ , volume 2696 of _CEUR Workshop Proceedings_.
  * Boteva et al. (2016) Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016.  A full-text learning to rank dataset for medical information retrieval.  In _Proceedings of the 38th European Conference on Information Retrieval (ECIR 2016)_ , pages 716–722.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020.  SPECTER: Document-level representation learning using citation-informed transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2270–2282, Online. Association for Computational Linguistics.
  * Corney et al. (2016) Davind Corney, Dyaa Albakour, Miguel Martinez, and Samir Moussa. 2016.  What do a million news articles look like? In _Proceedings of the First International Workshop on Recent Trends in News Information Retrieval co-located with 38th European Conference on Information Retrieval (ECIR 2016)_ , pages 42–47.
  * Dai and Callan (2020) Zhuyun Dai and Jamie Callan. 2020.  Context-aware term weighting for first stage passage retrieval.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, page 1533–1536, New York, NY, USA. Association for Computing Machinery.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Diggelmann et al. (2020) Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020.  Climate-fever: A dataset for verification of real-world climate claims.
  * Ding et al. (2020) Yingqi Qu Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2020.  Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering.
  * Dong et al. (2010) Anlei Dong, Ruiqiang Zhang, Pranam Kolari, Jing Bai, Fernando Diaz, Yi Chang, Zhaohui Zheng, and Hongyuan Zha. 2010.  Time is of the essence: Improving recency ranking using twitter data.  In _Proceedings of the 19th International Conference on World Wide Web_ , WWW ’10, page 331–340, New York, NY, USA. Association for Computing Machinery.
  * Gao et al. (2020) Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2020.  Complementing lexical retrieval with semantic residual embedding.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021.  The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics.
  * Ghorab et al. (2013) M Rami Ghorab, Dong Zhou, Alexander O’connor, and Vincent Wade. 2013.  Personalised information retrieval: survey and classification.  _User Modeling and User-Adapted Interaction_ , 23(4):381–443.
  * Gillick et al. (2018) Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018.  End-to-end retrieval in continuous space.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016.  A deep relevance matching model for ad-hoc retrieval.  In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_ , CIKM ’16, page 55–64, New York, NY, USA. Association for Computing Machinery.
  * Guo et al. (2020) Mandy Guo, Yinfei Yang, Daniel Cer, Qinlan Shen, and Noah Constant. 2020.  Multireqa: A cross-domain evaluation for retrieval question answering models.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.
  * Hagberg et al. (2008) Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. 2008.  Exploring network structure, dynamics, and function using networkx.  In _Proceedings of the 7th Python in Science Conference_ , pages 11 – 15, Pasadena, CA USA.
  * Hasibi et al. (2017) Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017.  Dbpedia-entity v2: A test collection for entity search.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’17, pages 1265–1268. ACM.
  * Hintze and Nelson (1998) Jerry L. Hintze and Ray D. Nelson. 1998.  Violin Plots: A Box Plot-Density Trace Synergism.  _The American Statistician_ , 52(2):181–184.
  * Hoogeveen et al. (2015) Doris Hoogeveen, Karin M Verspoor, and Timothy Baldwin. 2015.  Cqadupstack: A benchmark data set for community question-answering research.  In _Proceedings of the 20th Australasian document computing symposium_ , pages 1–8.
  * Ioffe (2010) Sergey Ioffe. 2010.  Improved consistent sampling, weighted minhash and l1 sketching.  In _2010 IEEE International Conference on Data Mining_ , pages 246–255. IEEE.
  * Ji et al. (2010) Ming Ji, Yizhou Sun, Marina Danilevsky, Jiawei Han, and Jing Gao. 2010.  Graph regularized transductive classification on heterogeneous information networks.  In _Machine Learning and Knowledge Discovery in Databases_ , pages 570–586, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Jiang and Zhai (2007) Jing Jiang and ChengXiang Zhai. 2007.  An empirical study of tokenization strategies for biomedical information retrieval.  _Information Retrieval_ , 10(4-5):341–363.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017.  Billion-scale similarity search with gpus.  _arXiv preprint arXiv:1702.08734_.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over bert.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, page 39–48, New York, NY, USA. Association for Computing Machinery.
  * Kleinberg (1999) Jon M. Kleinberg. 1999.  Authoritative sources in a hyperlinked environment.  _J. ACM_ , 46(5):604–632.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Liang et al. (2020) Davis Liang, Peng Xu, Siamak Shakeri, Cicero Nogueira dos Santos, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020.  Embedding-based zero-shot retrieval through query generation.
  * Lin et al. (2020) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020.  Pretrained transformers for text ranking: Bert and beyond.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, dense, and attentional representations for text retrieval.
  * Ma et al. (2021) Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021.  Zero-shot neural passage retrieval via domain-targeted synthetic question generation.
  * Maia et al. (2018) Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018.  Www’18 open challenge: Financial opinion mining and question answering.  In _Companion Proceedings of the The Web Conference 2018_ , WWW ’18, page 1941–1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _choice_ , 2640:660.
  * Nogueira and Cho (2020) Rodrigo Nogueira and Kyunghyun Cho. 2020.  Passage re-ranking with bert.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 708–718, Online. Association for Computational Linguistics.
  * Nogueira et al. (2019a) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019a.  Multi-stage document ranking with bert.
  * Nogueira et al. (2019b) Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019b.  Document expansion by query prediction.
  * Page et al. (1999) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999.  The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab.  Previous number = SIDL-WP-1999-0120.
  * Petroni et al. (2020) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2020.  Kilt: a benchmark for knowledge intensive language tasks.
  * Radlinski et al. (2008) Filip Radlinski, Madhu Kurup, and Thorsten Joachims. 2008.  How does clickthrough data reflect retrieval quality? In _Proceedings of the 17th ACM Conference on Information and Knowledge Management_ , CIKM ’08, page 43–52, New York, NY, USA. Association for Computing Machinery.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  The curse of dense low-dimensional information retrieval for large index sizes.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends in Information Retrieval_ , 3(4):333–389.
  * Rücklé et al. (2020) Andreas Rücklé, Jonas Pfeiffer, and Iryna Gurevych. 2020.  MultiCQA: Zero-shot transfer of self-supervised text matching models on a massive scale.  In _Proceedings of The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)_ , pages 2471–2486. Association for Computational Linguistics.
  * Sanh et al. (2020) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
  * Seo et al. (2019) Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019.  Real-time open-domain question answering with dense-sparse phrase index.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4430–4441, Florence, Italy. Association for Computational Linguistics.
  * Shah et al. (2018) Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov. 2018\.  Adversarial domain adaptation for duplicate question detection.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1056–1063, Brussels, Belgium. Association for Computational Linguistics.
  * Soboroff et al. (2019) Ian Soboroff, Shudong Huang, and Donna Harman. 2019.  Trec 2019 news track overview. In _TREC_.
  * Suarez et al. (2018) Axel Suarez, Dyaa Albakour, David Corney, Miguel Martinez, and Jose Esquivel. 2018\.  A data collection for evaluating the retrieval of related tweets to news articles.  In _40th European Conference on Information Retrieval Research (ECIR 2018), Grenoble, France, March, 2018._ , pages 780–786.
  * Thakur et al. (2020) Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2020.  Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015.  An overview of the bioasq large-scale biomedical semantic indexing and question answering competition.  _BMC bioinformatics_ , 16(1):138.
  * van den Oord et al. (2019) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019.  Representation learning with contrastive predictive coding.
  * Van Gysel and de Rijke (2018) Christophe Van Gysel and Maarten de Rijke. 2018.  Pytrec_eval: An extremely fast python interface to trec_eval.  In _SIGIR_. ACM.
  * Voorhees et al. (2021) Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021.  Trec-covid: Constructing a pandemic information retrieval test collection.  _SIGIR Forum_ , 54(1).
  * Wachsmuth et al. (2017) Henning Wachsmuth, Martin Potthast, Khalid Al-Khatib, Yamen Ajjour, Jana Puschmann, Jiani Qu, Jonas Dorsch, Viorel Morari, Janek Bevendorff, and Benno Stein. 2017.  Building an Argument Search Engine for the Web.  In _4th Workshop on Argument Mining (ArgMining 2017) at EMNLP_ , pages 49–59. Association for Computational Linguistics.
  * Wachsmuth et al. (2018) Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018.  Retrieval of the best counterargument without prior topic knowledge.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 241–251. Association for Computational Linguistics.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, Online. Association for Computational Linguistics.
  * Wang et al. (2020) Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex Wade, Kuansan Wang, Nancy Xin Ru Wang, Chris Wilhelm, Boya Xie, Douglas Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020.  Cord-19: The covid-19 open research dataset.
  * Wang et al. (2013) Yining Wang, Liwei Wang, Yuanzhi Li, Di He, Wei Chen, and Tie-Yan Liu. 2013.  A theoretical analysis of ndcg ranking measures.  In _Proceedings of the 26th annual conference on learning theory (COLT 2013)_ , volume 8, page 6.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.
  * Yang et al. (2020) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, et al. 2020.  Multilingual universal sentence encoder for semantic retrieval.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 87–94.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Zhang et al. (2015) Yun Zhang, David Lo, Xin Xia, and Jian-Ling Sun. 2015.  Multi-factor duplicate question detection in stack overflow.  _Journal of Computer Science and Technology_ , 30(5):981–997.
  * Zhao et al. (2020) Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2020.  Sparta: Efficient open-domain question answering via sparse transformer matching retrieval.

##  Appendix A Appendices

In this Appendix, we mention the following sections in detail: Training and
in-domain evaluation (B), descriptions of all zero-shot tasks and datasets in
beir (C). We further explain how we calculate weighted jaccard similarity (D)
and capped recall at k (E) metrics.

##  Appendix B Training and In-domain Evaluation

We use the MSMARCO Passage Ranking dataset Nguyen et al. (2016), which
contains 8.8M Passages and an official training set of 532,761 query-passage
pairs for fine-tuning for a majority of retrievers. The dataset contains
queries from Bing search logs with one text passage from various web sources
annotated as relevant. We find the dataset useful for training, in terms of
covering a wide variety of topics and providing the highest number of training
pairs. It has been extensively explored and used for fine-tuning dense
retrievers in recent works Nogueira and Cho (2020); Gao et al. (2020); Ding et
al. (2020). We use the official MSMARCO development set for our in-domain
evaluation which has been widely used in prior research Nogueira and Cho
(2020); Gao et al. (2020); Ding et al. (2020). It has 6,980 queries. Most of
the queries have only 1 document judged relevant; the labels are binary.

##  Appendix C Zero-shot Evaluation Tasks

Following the selection criteria mentioned in section 3, we include 17
evaluation datasets that span over 9 heterogeneous tasks. Each dataset
contains a document corpus denoted by $\mathbf{T}$ and test queries for
evaluation denoted by $\mathbf{Q}$. We additionally provide website-links in
Table 6 and intuitive examples in Table 8. We now summarise each task and
describe each dataset included for evaluation in beir:

###  C.1 Bio-Medical Information Retrieval

Bio-medical information retrieval is the task of searching relevant scientific
documents such as research papers or blogs for a given scientific query in the
biomedical domain Jiang and Zhai (2007). We consider a scientific query as
input and retrieve bio-medical documents as output.

##### TREC-COVID

Voorhees et al. (2021) is an ad-hoc search challenge based on the CORD-19
dataset containing scientific articles related to the COVID-19 pandemic Wang
et al. (2020). We include the July 16, 2020 version of
CORD-19131313https://www.semanticscholar.org/cord19 as corpus $\mathbf{T}$ and
use the final cumulative judgements of the original task as queries
$\mathbf{Q}$.

##### NFCorpus

Boteva et al. (2016) contains natural language queries harvested from
NutritionFacts141414https://nutritionfacts.org (NF). We use the original
splits provided alongside all content sources from NF (videos, blogs, and Q&A
posts) as queries $\mathbf{Q}$ and annotated medical documents from
PubMed151515https://pubmed.ncbi.nlm.nih.gov as corpus $\mathbf{T}$.

##### BioASQ

Tsatsaronis et al. (2015) Task 8b is a biomedical semantic QA challenge. We
use the original train and test splits provided in Task 8b as queries
$\mathbf{Q}$ and collect around 15M articles from PubMed provided in Task 8a
as our corpus $\mathbf{T}$.

###  C.2 Question Answering

Retrieval in open domain question answering Chen et al. (2017) is the task of
retrieving the correct answer for a question, without a predefined location
for the answer. In open-domain tasks, model must retrieve over an entire
knowledge source (such as Wikipedia). We consider the question as input and
the passage containing the answer as output.

##### Natural Questions

Kwiatkowski et al. (2019) contains Google search queries and documents with
paragraphs and answer spans within Wikipedia articles. We did not use the NQ
version from ReQA Ahmad et al. (2019) as it focused on queries having a short
answer. As a result, we parsed the HTML of the original NQ dataset and include
more complex development queries that often require a longer passage as answer
compared to ReQA. We filtered out queries without an answer, or having a table
as an answer, or with conflicting Wikipedia pages. We retain 2,681,468
passages as our corpus $\mathbf{T}$ and 3452 test queries $\mathbf{Q}$ from
the original dataset.

##### HotpotQA

Yang et al. (2018) contains questions which require reasoning over multiple
paragraphs to find the correct answer. We include the full-wiki task setting:
utilizing processed Wikipedia passages as corpus $\mathbf{T}$. We held out
randomly sampled 5447 queries from training as development split. We use the
original task’s development set as our test queries $\mathbf{Q}$.

##### FiQA-2018

Maia et al. (2018) Task 2 consists of opinion-based QA. We include financial
data by crawling StackExchange posts under the Investment topic from 2009-2017
as our corpus $\mathbf{T}$. We randomly sample out 500 and 648 as queries
$\mathbf{Q}$ from the training split as development and test splits
respectively.

###  C.3 Tweet Retrieval

Twitter is a popular micro-blogging website on which people post real-time
messages (i.e. tweets) about their opinions on a variety of topics and discuss
current issues. We consider a news headline as input and retrieve relevant
tweets as output.

##### Signal-1M Related Tweets

Suarez et al. (2018) includes 97 news articles from the Signal-1M dataset
Corney et al. (2016) as queries $\mathbf{Q}$. We construct our twitter corpus
$\mathbf{T}$ by manually scraping tweets from given tweet-ids using
Tweepy161616https://www.tweepy.org.

###  C.4 News Retrieval

##### TREC-NEWS

Soboroff et al. (2019) 2019 track involves background linking: Given a news
headline, we retrieve relevant news articles that provide important context or
background information. We include the shared task data as our test queries
$\mathbf{Q}$ and the TREC Washington Post171717https://ir.nist.gov/wapo/ as
our corpus $\mathbf{T}$. We convert the original exponential gain relevant
judgements to linear labels for simplicity.

###  C.5 Argument Retrieval

Argument retrieval is the task of ranking argumentative texts in a collection
of focused arguments (output) in order of their relevance to a textual query
(input) on different topics.

##### ArguAna Counterargs Corpus

Wachsmuth et al. (2018) involves the retrieval of the best counterargument to
an argument. It contains pairs of argument and counterargument scraped from
the online debate portal181818https://idebate.org as corpus $\mathbf{T}$. We
consider the original arguments present in the original tasks test split as
our queries $\mathbf{Q}$.

##### Touchè-2020

Bondarenko et al. (2020) Task 1 is a conversational argument retrieval task.
We use the conclusion as title and premise for arguments present in
args.me191919https://zenodo.org/record/4139439/ Wachsmuth et al. (2017) as
corpus $\mathbf{T}$. We include the shared task data as our test queries
$\mathbf{Q}$. The original relevance judgements (qrels) file also included
negative judgements (-2), but for simplicity we substitute them as zero.

###  C.6 Duplicate Question Retrieval

Duplicate question retrieval is the task of identifying duplicate questions
asked in community question answering (cQA) forums. A given query is the input
and the duplicate questions are the output.

##### CQADupStack

Hoogeveen et al. (2015) is a benchmark dataset for community question-
answering (cQA) research. The corpus $\mathbf{T}$ comprises of queries from 12
different StackExchange subforums. We utilize the predefined test split for
our queries $\mathbf{Q}$. We evaluate each subforum separately and report the
overall mean score.

##### Quora

Duplicate Questions dataset identifies whether two questions are duplicates.
Quora originally released containing 404,290 question pairs. We add transitive
closures to the original dataset. Further, we split it into train, dev, and
test sets with a ratio of about 85%, 5% and 10% of the original pairs. We
remove all overlaps between the splits and ensure that a question in one split
of the dataset does not appear in any other split to mitigate the transductive
classification problem Ji et al. (2010). We finally achieve 522,931 unique
queries as our corpus $\mathbf{T}$ and 5,000 dev and 10,000 test queries
$\mathbf{Q}$ respectively.

###  C.7 Entity Retrieval

Entity retrieval involves retrieving unique Wikipedia pages to entities
mentioned in the query. This is crucial for tasks involving Entity Linking
(EL). The entity-bearing query is the input and the entity abstract and title
are retrieved as output.

##### DBPedia-Entity-v2

Hasibi et al. (2017) is an established entity retrieval dataset. It contains a
set of heterogeneous entity-bearing queries $\mathbf{Q}$ and retrieves
entities from the english part of DBpedia corpus $\mathbf{T}$ from October
2015. We randomly sample out 67 queries from the test split as development.

###  C.8 Citation Prediction

Citations are a key signal of relatedness between scientific papers Cohan et
al. (2020). In this task, the model attempts to retrieve cited papers (output)
for a given query paper title (input).

##### SCIDOCS

Cohan et al. (2020) contains a corpus $\mathbf{T}$ of 30K held-out pool of
scientific papers. We consider the original direct-citations task as our
retrieval task. It includes 1k papers as queries $\mathbf{Q}$ and 5 relevant
papers and 25 (randomly selected) uncited papers for each query.

###  C.9 Fact Checking

Fact checking verifies a claim against a big collection of evidence Thorne et
al. (2018). The task requires knowledge about the claim and reasoning over
multiple documents. We consider the claim as input and the relevant document
passage verifying the claim as output.

##### FEVER

Thorne et al. (2018) The Fact Extraction and VERification dataset is collected
to facilitate the automatic fact checking. We utilize the original paper
splits as queries $\mathbf{Q}$ and retrieve evidences from the pre-processed
Wikipedia Pages (June 2017 dump) as our corpus $\mathbf{T}$.

##### Climate-FEVER

Diggelmann et al. (2020) is a dataset for verification of real-world climate
claims. We include the claims as queries $\mathbf{Q}$ and retrieve evidences
from the same FEVER Wiki corpus $\mathbf{T}$.

##### SciFact

Wadden et al. (2020) retrieves abstracts from the research literature
containing evidence for a given scientific claim. We use the original dev
split from the task containing 300 queries as our test queries $\mathbf{Q}$,
and include documents from the original scientific corpus $\mathbf{T}$.

##  Appendix D Weighted Jaccard Similarity

The weighted Jaccard similarity $J(S,T)$ Ioffe (2010) is intuitively
calculated as the unique word overlap for all words present in both the
datasets. More formally, the normalized frequency for an unique word $k$ in a
dataset is calculated as the frequency of word $k$ divided over the sum of
frequencies of all words in the dataset.

${S_{k}}$ is the normalized frequency of word $k$ in the source dataset $S$
and ${T_{k}}$ for the target dataset $T$ respectively. The weighted Jaccard
similarity between $S$ and $T$ is defined as:

$J(S,T)=\frac{\sum_{k}\min(S_{k},T_{k})}{\sum_{k}\max(S_{k},T_{k})}$

where the sum is over all unique words $k$ present in datasets $S$ and $T$.

##  Appendix E Capped Recall@k Score

Recall at $k$ is calculated as the fraction of the relevant documents that are
successfully retrieved within the top $k$ extracted documents. More formally,
the $R@k$ score is calculated as:

$R@k=\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{|\max_{k}(A_{i})\cap
A_{i}^{\star}|}{|A_{i}^{\star}|}$

where $Q$ is the set of queries, $A_{i}^{\star}$ is the set of relevant
documents for the $i$th query, and $A_{i}$ is a scored list of documents
provided by the model, from which top $k$ are extracted.

However measuring recall can be counterintuitive, if a high number of relevant
documents ($>k$) are present within a dataset. For example, consider a
hypothetical dataset with 500 relevant documents for a query. Retrieving all
relevant documents would produce a maximum $R@100$ score = 0.2, which is quite
low and unintuitive. To avoid this we cap the recall score ($R\\_cap@k$) at k
for datasets if the number of relevant documents for a query greater than k.
It is defined as:

$R\\_cap@k=\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{|\max_{k}(A_{i})\cap
A_{i}^{\star}|}{\min(k,|A_{i}^{\star}|)}$

where the only difference lies within the denominator where we compute the
minimum of k and $|A_{i}^{\star}|$, instead of $|A_{i}^{\star}|$ present in
the original recall.

Figure 5: Heatmap showing pairwise weighted jaccard similarity Ioffe (2010) on
unigram word overlap between the document collections for all datasets in
beir..

Dataset | Website (Link)
---|---
MSMARCO | https://microsoft.github.io/msmarco/
TREC-COVID | https://ir.nist.gov/covidSubmit/index.html
NFCorpus | https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/
BioASQ | http://bioasq.org
NQ | https://ai.google.com/research/NaturalQuestions
HotpotQA | https://hotpotqa.github.io
FiQA-2018 | https://sites.google.com/view/fiqa/
Signal-1M (RT) | https://research.signal-ai.com/datasets/signal1m-tweetir.html
TREC-NEWS | https://trec.nist.gov/data/news2019.html
ArguAna | http://argumentation.bplaced.net/arguana/data
Touchè-2020 | https://webis.de/events/touche-20/shared-task-1.html
CQADupStack | http://nlp.cis.unimelb.edu.au/resources/cqadupstack/
Quora | https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs
DBPedia-Entity | https://github.com/iai-group/DBpedia-Entity/
SCIDOCS | https://allenai.org/data/scidocs
FEVER | http://fever.ai
Climate-FEVER | http://climatefever.ai
SciFact | https://github.com/allenai/scifact

Table 6: Original dataset website (link) for all datasets present in beir.

Model | Model Checkpoints (URL)
---|---
USE-QA | https://tfhub.dev/google/universal-sentence-encoder-qa/3
SBERT (Cos-Sim.) | https://huggingface.co/sentence-transformers/msmarco-distilbert-base-v3
SBERT (Dot-Prod.) | https://huggingface.co/sentence-transformers/msmarco-distilbert-base-dot-prod-v3
ELECTRA (CE) | https://huggingface.co/sentence-transformers/ce-ms-marco-electra-base
SPARTA | https://huggingface.co/BeIR/sparta-msmarco-distilbert-base-v1
ANCE | https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/msmarco-roberta-base-ance-fristp.zip
DPR (Query) | https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/facebook-dpr-question_encoder-multiset-base.zip
DPR (Context) | https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/facebook-dpr-ctx_encoder-multiset-base.zip
ColBERT | https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/models/ColBERT/msmarco.psg.l2.zip

Table 7: Neural model and checkpoints used for evaluation in beir.

Dataset | Query | Relevant-Document
---|---|---
MSMARCO |  what fruit is native to australia |  <Paragraph> Passiflora herbertiana. A rare passion fruit native to Australia. Fruits are green-skinned, white fleshed, with an unknown edible rating. Some sources list the fruit as edible, sweet and tasty, while others list the fruits as being bitter and inedible. assiflora herbertiana. A rare passion fruit native to Australia…
TREC-COVID |  what is the origin of COVID-19 |  <Title> Origin of Novel Coronavirus (COVID-19): A Computational Biology Study using Artificial Intelligence <Paragraph> Origin of the COVID-19 virus has been intensely debated in the community…
BioASQ |  What is the effect of HMGB2 loss on CTCF clustering |  <Title> HMGB2 Loss upon Senescence Entry Disrupts Genomic Organization and Induces CTCF Clustering across Cell Types. <Paragraph> Processes like cellular senescence are characterized by complex events giving rise to heterogeneous cell populations. However, the early molecular events driving this cascade remain elusive….
NFCorpus |  Titanium Dioxide & Inflammatory Bowel Disease |  <Title> Titanium Dioxide Nanoparticles in Food and Personal Care Products <Paragraph> Titanium dioxide is a common additive in many food, personal care, and other consumer products used by people, which after use can enter the sewage system, and subsequently enter the environment as treated effluent discharged to surface waters or biosolids applied to agricultural land, or incinerated wastes…
NQ |  when did they stop cigarette advertising on television? |  <Title> Tobacco advertising <Paragraph> The first calls to restrict advertising came in 1962 from the Royal College of Physicians, who highlighted the health problems and recommended stricter laws…
HotpotQA |  Stockely Webster has paintings hanging in what home (that serves as the residence for the Mayor of New York)? |  <Title> Stokely Webster <Paragraph> Stokely Webster (1912 – 2001) was best known as an American impressionist painter who studied in Paris. His paintings can be found in the permanent collections of many museums, including the Metropolitan Museum of Art in New York, the National Museum…
FiQA-2018 |  What is the PEG ratio? How is the PEG ratio calculated? How is the PEG ratio useful for stock investing? |  <Paragraph> PEG is Price/Earnings to Growth. It is calculated as Price/Earnings/Annual EPS Growth. It represents how good a stock is to buy, factoring in growth of earnings, which P/E does not. Obviously when PEG is lower, a stock is more undervalued, which means that it is a better buy, and more likely…
Signal-1M (RT) |  Genvoya, a Gentler Anti-HIV Cocktail, Okayed by EU Regulators |  <Paragraph> All people with #HIV should get anti-retroviral drugs: @WHO, by @kkelland via @Reuters_Health #AIDS #TasP
TREC-NEWS |  Websites where children are prostituted are immune from prosecution. But why? |  <Title> Senate launches bill to remove immunity for websites hosting illegal content, spurred by Backpage.com <Paragraph> The legislation, along with a similar bill in the House, sets the stage for a battle between Congress and some of the Internet’s most powerful players, including Google and various free-speech advocates, who believe that Congress shouldn’t regulate Web content or try to force websites to police themselves more rigorously…
ArguAna |  Sexist advertising is subjective so would be too difficult to codify. Effective advertising appeals to the social, cultural, and personal values of consumers. Through the connection of values to products, services and ideas, advertising is able to accomplish its goal of adoption… |  <Title> media modern culture television gender house would ban sexist advertising <Paragraph> Although there is a claim that sexist advertising is to difficult to codify, such codes have and are being developed to guide the advertising industry. These standards speak to advertising which demeans the status of women, objectifies them, and plays upon stereotypes about women which harm women and society in general. Earlier the Council of Europe was mentioned, Denmark, Norway and Australia as specific examples of codes or standards for evaluating sexist advertising which have been developed.
Tóuche-2020 |  Should the government allow illegal immigrants to become citizens? |  <Title> America should support blanket amnesty for illegal immigrants. <Paragraph> Undocumented workers do not receive full Social Security benefits because they are not United States citizens " nor should they be until they seek citizenship legally. Illegal immigrants are legally obligated to pay taxes…
CQADupStack |  Command to display first few and last few lines of a file |  <Title> Combing head and tail in a single call via pipe <Paragraph> On a regular basis, I am piping the output of some program to either ‘head‘ or ‘tail‘. Now, suppose that I want to see the first AND last 10 lines of piped output, such that I could do something like ./lotsofoutput | headtail…
Quora |  How long does it take to methamphetamine out of your blood? |  <Paragraph> How long does it take the body to get rid of methamphetamine?
DBPedia |  Paul Auster novels |  <Title> The New York Trilogy <Paragraph> The New York Trilogy is a series of novels by Paul Auster. Originally published sequentially as City of Glass (1985), Ghosts (1986) and The Locked Room (1986), it has since been collected into a single volume.
SCIDOCS |  CFD Analysis of Convective Heat Transfer Coefficient on External Surfaces of Buildings |  <Title> Application of CFD in building performance simulation for the outdoor environment: an overview <Paragraph> This paper provides an overview of the application of CFD in building performance simulation for the outdoor environment, focused on four topics…
FEVER |  DodgeBall: A True Underdog Story is an American movie from 2004 |  <Title> DodgeBall: A True Underdog Story <Paragraph> DodgeBall: A True Underdog Story is a 2004 American sports comedy film written and directed by Rawson Marshall Thurber and starring Vince Vaughn and Ben Stiller. The film follows friends who enter a dodgeball tournament…
Climate-FEVER |  Sea level rise is now increasing faster than predicted due to unexpectedly rapid ice melting. |  <Title> Sea level rise <Paragraph> A sea level rise is an increase in the volume of water in the world ’s oceans, resulting in an increase in global mean sea level. The rise is usually attributed to global climate change by thermal expansion of the water in the oceans and by melting of Ice sheets and glaciers…

Table 8: Examples of queries and relevant documents for all datasets included
in beir. (<Title>) and (<Paragraph>) are used to distinguish the title
separately from the paragraph within a document in the table above. These
tokens were not passed to the respective models.

Model ($\rightarrow$) | Lexical | Sparse | Dense / Neural | Reranking
---|---|---|---|---
Dataset ($\downarrow$) | BM25 | SPARTA | USE-QA | DPR | ANCE | SBERT | GenQ | BM25+CE | ColBERT
MSMARCO | 0.621 | 0.793‡ | 0.720 | 0.552 |  0.852‡ | 0.847‡ | 0.847‡ | 0.621‡ | 0.865‡
TREC-COVID | 0.447⋆ | 0.409⋆ | 0.339⋆ | 0.212⋆ |  0.457⋆ | 0.344⋆ | 0.442⋆ | 0.447⋆ | 0.464⋆
BioASQ | 0.716 | 0.351 | 0.145 | 0.256 | 0.463 | 0.466 | 0.577 | 0.716 | 0.645
NFCorpus | 0.196 | 0.243 | 0.273 | 0.208 | 0.232 | 0.235 | 0.272 | 0.196 | 0.254
NQ | 0.753 | 0.787 | 0.605 |  0.880‡ | 0.836 | 0.858 | 0.845 | 0.753 | 0.912
HotpotQA | 0.757 | 0.651 | 0.398 | 0.591 | 0.578 | 0.637 | 0.624 | 0.757 | 0.748
FiQA-2018 | 0.505 | 0.446 | 0.601 | 0.342 | 0.581 | 0.540 | 0.609 | 0.505 | 0.603
Signal-1M (RT) | 0.376 | 0.270 | 0.266 | 0.162 | 0.239 | 0.263 | 0.271 | 0.376 | 0.283
TREC-NEWS | 0.401 | 0.262 | 0.381 | 0.215 | 0.398 | 0.367 | 0.395 | 0.401 | 0.367
ArguAna | 0.931 | 0.893 | 0.454 | 0.751 | 0.937 | 0.945 | 0.979 | 0.931 | 0.914
Tóuche-2020 | 0.463 | 0.257 | 0.257 | 0.171 | 0.307 | 0.300 | 0.273 | 0.463 | 0.309
CQADupStack | 0.588 | 0.521 | 0.546 | 0.403 | 0.579 | 0.596 | 0.672 | 0.588 | 0.624
Quora | 0.948 | 0.896 | 0.978 | 0.470 | 0.987 | 0.989 | 0.989 | 0.948 | 0.989
DBPedia | 0.384 | 0.411 | 0.281 | 0.349 | 0.319 | 0.403 | 0.396 | 0.384 | 0.461
SCIDOCS | 0.346 | 0.297 | 0.255 | 0.219 | 0.269 | 0.296 | 0.334 | 0.346 | 0.344
FEVER | 0.908 | 0.843 | 0.850 | 0.840 | 0.900 | 0.914 | 0.920 | 0.908 | 0.934
Climate-FEVER | 0.383 | 0.227 | 0.346 | 0.390 | 0.445 | 0.448 | 0.517 | 0.383 | 0.444
SciFact | 0.838 | 0.863 | 0.631 | 0.727 | 0.816 | 0.849 | 0.892 | 0.838 | 0.878
AVERAGE | 0.587 | 0.523 | 0.463 | 0.430 | 0.566 | 0.572 | 0.603 | 0.587 | 0.613

Table 9: In-domain and zero-shot retrieval performance on beir datasets.
Scores denote Recall@100. The best retrieval performance on a given dataset is
marked in bold, and the second best performance is underlined. $\ddagger$
indicates in-domain retrieval performance. ⋆ indicates the capped recall
score: R_cap@100 (Appendix E).

Figure 6: Violin plots Hintze and Nelson (1998) of document lengths for top-10
retrieved hits for SBERT (subsubsection 4.1.3) trained with either cosine-
similarity or dot-product. The x-axis shows the retrieved document length,
capped at a max length of 512 words. The dashed lines within the plot denotes
the corresponding (25%, 50% and 75%) quantiles. The plot shows the cosine-
similarity (top, blue) model across all datasets retrieves on average shorter
length documents compared to the dot product model (bottom, orange).

MSMARCO TREC-COVID BioASQ NFCorpus Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-Prod.
Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-Prod. 0.389 0.389 0.482 0.635 0.295
0.280 0.257 0.267 NQ HotpotQA FiQA-2018 Signal-1M (RT) Cosine-Sim. Dot-Prod.
Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-Prod. 0.450 0.442
0.513 0.477 0.258 0.255 0.261 0.243 TREC-NEWS ArguAna Tóuche-2020 Quora
Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-Prod. Cosine-Sim.
Dot-Prod. 0.389 0.389 0.482 0.635 0.295 0.280 0.855 0.833 CQADupStack DBPedia
SCIDOCS FEVER Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-
Prod. Cosine-Sim. Dot-Prod. 0.306 0.283 0.339 0.315 0.133 0.113 0.670 0.685
Climate-FEVER SciFact Cosine-Sim. Dot-Prod. Cosine-Sim. Dot-Prod. 0.205 0.169
0.531 0.511

Table 10: nDCG@10 scores for SBERT (subsubsection 4.1.3) trained with either
cosine-similarity (Cosine-Sim.) or dot-product (Dot-Prod.). For few datasets,
we find the dot-product model to outperform cosine-sim. model by a large
margin. For a majority of datasets, we find the cosine-sim. model to
marginally outperform the dot-product model.


ILTJ0909+4502 | 09:09:31.06 | +45:02:01.81 | 1.7 | HET
ILTJ0933+5215 | 09:33:42.71 | +52:15:17.83 | 1.0 | HET
ILTJ0933+5248 | 09:33:59.67 | +52:48:02.84 | 1.7 | HET
ILTJ0950+3440 | 09:50:21.02 | +34:40:29.06 | 1.0 | HET
ILTJ1004+3424 | 10:04:10.77 | +34:24:34.12 | 0.7 | Subaru
ILTJ1010+4234 | 10:10:54.32 | +42:34:39.90 | 1.4 | Subaru
ILTJ1011+5757 | 10:11:55.86 | +57:57:36.55 | 0.4 | Subaru
ILTJ1012+6532 | 10:12:17.72 | +65:32:10.90 | 1.4 | Subaru
ILTJ1020+5909 | 10:20:01.89 | +59:09:01.96 | 0.9 | HET
ILTJ1039+4936 | 10:39:35.52 | +49:36:24.22 | 0.7 | Subaru
ILTJ1056+3112 | 10:56:15.94 | +31:12:32.83 | 0.6 | Subaru
ILTJ1056+4759 | 10:56:23.66 | +47:59:06.56 | 0.7 | Subaru
ILTJ1059+5516 | 10:59:22.43 | +55:16:21.10 | 0.7 | Subaru
ILTJ1101+6507 | 11:01:31.65 | +65:07:50.99 | 0.8 | Subaru
ILTJ1124+5141 | 11:24:58.70 | +51:41:16.79 | 0.6 | HET
ILTJ1132+3651 | 11:32:13.29 | +36:51:49.93 | 0.1 | HET
ILTJ1135+3336 | 11:35:04.90 | +33:36:39.98 | 1.2 | HET
ILTJ1354+4234 | 13:54:51.40 | +42:34:01.85 | 1.3 | HET
ILTJ1357+3354 | 13:57:30.24 | +33:54:36.17 | 0.7 | HET
ILTJ1402+2914 | 14:02:02.45 | +29:14:14.17 | 1.1 | HET
ILTJ1519+4750 | 15:19:09.56 | +47:50:26.97 | 1.5 | HET
ILTJ1535+6145 | 15:35:46.82 | +61:45:59.39 | 0.9 | Keck
ILTJ1755+3937 | 17:55:45.66 | +39:37:44.23 | 0.5 | HET
ILTJ1801+3141 | 18:01:08.08 | +31:41:13.48 | 2.0 | HET
ILTJ2157+2108 | 21:57:22.41 | +21:08:12.90 | 0.3 | VLT
ILTJ2210+2449 | 22:10:21.05 | +24:49:52.49 | 0.7 | Subaru
ILTJ2232+1840 | 22:32:16.06 | +18:40:04.28 | 0.6 | HET
ILTJ2235+1802 | 22:35:25.17 | +18:02:37.67 | 1.2 | Subaru
ILTJ2244+2140 | 22:44:48.02 | +21:40:54.13 | 1.7 | Subaru
ILTJ2259+2018 | 22:59:57.36 | +20:18:49.44 | 0.9 | HET
ILTJ2308+1925 | 23:08:30.46 | +19:25:59.74 | 0.4 | Subaru
ILTJ2316+3142 | 23:16:09.78 | +31:42:40.18 | 1.4 | Subaru
ILTJ2327+2002 | 23:27:56.67 | +20:02:50.45 | 1.9 | HET

##  Appendix B Cutouts of newly discovered quasars

Figure 11: 20”$\times$20” Legacy DR8 (Dey et al. 2019), AllWISE (Cutri et al.
2021), and LOFAR (LoTSS-DR2; Shimwell et al. 2022) cutouts for the newly
confirmed quasars.