# Domain Adaptation of Multilingual Semantic Search - Literature Review

Anna Bringmann<EMAIL_ADDRESS>Georg-August University
GöttingenGermany and  Anastasia Zhukova<EMAIL_ADDRESS>Georg-August
University GöttingenGermany

###### Abstract.

This literature review gives an overview of current approaches to perform
domain adaptation in a low-resource and approaches to perform multilingual
semantic search in a low-resource setting. We developed a new typology to
cluster domain adaptation approaches based on the part of dense textual
information retrieval systems, which they adapt, focusing on how to combine
them efficiently. We also explore the possibilities of combining multilingual
semantic search with domain adaptation approaches for dense retrievers in a
low-resource setting.

multilingual semantic search, cross-lingual dense retriever, domain
adaptation, low resource, zero-shot, dense retrieval

††copyright: none

##  1\. Introduction

Text information retrieval (tIR) describes acquiring the relevant textual
information from all available textual information, given a user-entered
natural language query. The query and the text are a word sequence divided
into several tokens from a specific language (Zhao et al., 2022).

Traditionally, tIR was performed using lexical methods. These suffer from the
lexical gap since they do not include semantic information and, thus, cannot
recognise synonyms or distinguish ambiguous words. Nowadays, to close the gap,
natural language processing (NLP) is widely applied to tIR tasks. The most
common models are dense retrievers, and their invention resulted in the
concept of semantic search. Dense retrieval models map the user query and the
available textual information to a shared dense vector space. The generated
representations include semantic information and are closer to one another in
the shared vector space if they are semantically similar. Thus, dense
retrievers utilise approximate nearest neighbour search to fetch relevant
information. However, large-scale labelled data is needed to train dense
retrievers, and they are sensitive to domain shifts (Thakur et al., 2021b).

We define a domain as a coherent type of text corpus, i.e. the specific
dataset used for training. The type is a variety of latent factors, e.g. topic
(chemistry vs. sports), genre (social media vs. news article) or style (formal
vs. informal) (Ramponi and Plank, 2020). For most specialised domains, the
availability of labelled data is scarce and collecting it is resource-
intensive. Doing that manually for every language and every domain is not
feasible (Shen et al., 2022). Thus, automated processes for domain adaptation
in low-resource setups are required and have been subject to extensive
research in the past years (Shen et al., 2022; Ren et al., 2023b; Zhao et al.,
2022). Nonetheless, most domain-adaptation research focuses on monolingual
dense retrieval, while the field of multilingual semantic search is rapidly
advancing at the same time, holding the promise of breaking down language
barriers and expanding information access across diverse linguistic contexts
(Yang et al., 2019; Litschko et al., 2021; Zhang et al., 2022b; Feng et al.,
2022; Reimers and Gurevych, 2020; Bonifacio et al., 2021).

However, the possibility of combining domain-adaptation techniques with
multilingual approaches to semantic search has yet to be explored. Therefore,
this literature review introduces a typology to explore the possibility of
utilising domain-adaptation approaches in a multilingual setup. Our research
question: ”How to perform domain adaptation for multilingual semantic search?”
formalises this objective.

The rest of this paper is structured as follows: In Section 2, we introduce
the methodology used to retrieve relevant literature. Afterwards, in Section
3, we present related work to give an overview of the research field,
exploring the possibilities of adapting multilingual semantic search to
specialised domains. That is followed by Section 5 in which we review existing
approaches for adapting dense tIR systems to specialised domains in a low-
resource setting. Subsequently, in Section 6, we examine existing approaches
to performing multilingual semantic search. In Section 7, we summarise the
strengths and weaknesses of the different approaches and discuss their
compatibility. Then, we finalise our work by giving an outlook and drawing a
conclusion in Section 8.

##  2\. Method

We applied a process consisting of four steps to find literature relevant to
our research question. The first step was a keyword search, from which we
selected papers based on different criteria. The keywords we used to describe
semantic search were ”semantic search” and ”dense retrieval.” We then combined
these with either the keywords to describe domain adaptation in low resources
settings (”domain adaptation”, ”low resource”, ”few-shot”, and ”zero-shot”) or
the keywords to describe multilingual semantic search (”multilingual”,
”language agnostic” and ”cross-lingual”). Thus, exemplary keyword searches
conducted by us were ”multilingual semantic search” or ”low resource dense
retrieval”.111applying dense retrieval to languages for which available data
is scarce is also considered a low-resource setting. Therefore, this keyword
combination also yields results for adapting semantic search to different
languages.

The criteria for selecting relevant papers from our initial keyword search(es)
was finding existing literature reviews. We found one recent general review of
dense textual retrieval (Zhao et al., 2022), which contained information about
applying dense retrievers to low-resource and multilingual settings. We
selected this as an initial paper. Then, we continued with the two remaining
steps.

The second step was thoroughly reading the selected papers to decide whether
they were relevant to our research question and whether we should keep them in
our collection. If we considered a paper relevant, the third and fourth steps
followed. The third step was reviewing the paper’s references and adding the
papers we missed in our keyword search to our collection of relevant papers.

The fourth step was searching for the relevant paper on connected papers,
retrieving the papers connected to the paper we initially deemed relevant but
published after it. The reasoning behind only retrieving the papers published
after the paper we initially considered relevant was that relevant papers
published before are already part of the initial paper’s references, and we
thus retrieved them in the third step. Using connected papers allowed us to
discover recent developments in the field, which the third step could not
bring forth.

Then, we read the selected papers’ abstracts, deciding which were relevant,
then read the relevant ones thoroughly and performed the third and fourth
steps again until nearly no new papers could be retrieved.

Afterwards, we conducted another keyword search, using the initial keywords,
to check whether we missed any relevant, recent papers. Then, we performed
steps two to four for the new papers. With 250 papers in our collection of
relevant papers, we stopped retrieving new literature.

To make this literature review comprehensive, independent of readers’
knowledge levels, we included a short history of performing textual
information retrieval in Appendix A. We also recap the architecture of
information retrieval systems and dense retrievers in Section 4 and explain
the training procedure of bi-encoders for first-stage retrieval, including the
different options for negative selection.

##  3\. Related Work

A growing body of research describes dense information retrieval systems
(Karpukhin et al., 2020; Zhao et al., 2022; Luan et al., 2021) and the topic
of adapting dense retrievers to new domains in low-resource settings gained
attention within the last years (Li and Gaussier, 2022; Wang et al., 2022; Xu
et al., 2022; Lee et al., 2021). Some literature reviews summarising state-of-
the-art approaches for domain-adaptation of dense retrievers exist (Ren et
al., 2023b; Shen et al., 2022). For example, Ren et al. (Ren et al., 2023b)
analysed the influence of training data properties on the zero-shot
performance of dense retrievers and compared different existing methods for
domain adaptation in zero-shot settings. The same is true for multilingual
semantic search (Mao et al., 2022; Feng et al., 2022; Reimers and Gurevych,
2020; Yang et al., 2019; Litschko et al., 2021; Cer et al., 2017; Zhang et
al., 2022b; Asai et al., 2021a). While (Hedderich et al., 2021a) considers
multilingual dense retrieval and adapting dense retrievers to domains with
specialised language in low-resource setups, they do not investigate the
possibility of combining both. Our focus on the possibility of adapting
multilingual dense retrievers to another low-resource domain is, therefore, a
new perspective on existing research.

To explain how to enable multilingual dense retrieval in specialised domains,
we first need to look at the structure of information retrieval systems and
the model architectures used for their different parts since the different
approaches for domain adaptation of multilingual semantic search adapt
different parts of that architecture. We will do so in the following section.

##  4\. Information Retrieval systems

We formalise the Aim of text IR systems as:
Given the natural language query $q$ and a collection of $m$ documents $D=$
$d_{i}$${}^{m}_{i=1}$ the IR system returns a list
$L=[d_{2},d_{2},....,d_{n}]$ of $n$ relevant documents sorted by the retrieval
model’s relevance scores.

To calculate these relevance scores, sparse information retrieval models
perform lexical matching, while dense retrievers perform semantic matching.
All information retrieval systems consist of pipelines that combine multiple
steps as visualised in Figure 1.

Figure 1. Information Retrieval Systems

The first stage performs the retrieval. Thus, the model selects several
candidate documents relevant to the user query. The model which implements
this stage is called the retriever. During the second, the re-ranking stage,
the selected candidates are ordered by importance. The model which implements
this stage is called the reranker. TIR system can utilise one to many re-
ranking stages to refine the results. A model called the reader performs the
third stage of textual information retrieval. We only add it if the
information retrieval task is more fine-grained than document-level retrieval.
The reader analyses the documents the retriever returned and localises the
response to the query (Zhao et al., 2022).

###  4.1. Dense Retrievers

Based on the formalisation of tIR systems, a way of formalising dense
retrievers arises:
Dense retrieval models encode the query and the document corpus as dense
vectors. Thus, they compute relevance scores through some similarity function
between these dense vectors:

(1) $\text{Rel}(q,d)=f_{sim}(\phi(q),\psi(d))$

where $\phi(\cdot)\in\mathbb{R}^{l}$ und $\psi(\cdot)\in\mathbb{R}^{l}$ are
functions that map the query and the documents into $l$-dimensional vector
space. A deep neural network performs this mapping, and the similarity
$f_{sim}$ is measured by, for example, the inner product or the cosine
similarity. Dense retrievers, thus, measure the semantic interaction of query
and text based on the learned representation of both in latent semantic space.
The closer the query and text are to one another in latent space, the more
similar they are (Zhao et al., 2022). Apart from this formal notation, dense
retrievers can be divided into two mainstream architecture types: cross- and
bi-encoders. We introduce both in the following subsection.

###  4.2. Bi-encoders, Cross-encoders or Both?

First, we introduce bi-encoders and, afterwards, cross-encoders. However,
Figure 2 allows a direct, visual comparison of both architecture types. We
finalise this section by analysing the advantages and disadvantages of both
architectures.

Figure 2. Dense Retriever Architecture Types

As shown in Figure 2, bi-encoders are based on a two-tower architecture,
utilising PLM-based encoders. They perform the self-attention mechanism for
query and document separately, mapping each to the same dense vector space and
then utilising a similarity measurement to measure the distance between both
(Wang et al., 2022; Zhao et al., 2022; Huang et al., 2013). Their two-tower
architecture makes bi-encoders flexible since document and query encoder
architecture can differ. It resulted in two different bi-encoder structures.
The first is called single-representation bi-encoder. It embeds the query and
text separately using two different pre-trained language models (Karpukhin et
al., 2020; Qu et al., 2021; Xiong et al., 2020). We place the special [CLS]
token at the beginning of the query, and the text and their encodings
represent their semantic meaning. The relevance score for a query text
combination is then computed through some similarity function using the [CLS]
token embeddings. The downside of the single-representation bi-encoder
structure is that it does not accurately capture the semantic information of
query and text. The second bi-encoder structure is called multi-representation
bi-encoder. It aims at enhancing the semantic information stored in the
embeddings by combining multiple query and document embeddings to measure
their similarity from different ”semantic viewpoints”. These contextualised
embeddings are learned and stored during training as well as indexing. This
second bi-encoder structure improves the retrieval performance compared to the
single-representation structure. However, storing the multiple generated views
makes their index large, leading to higher computational and memory costs.
Examples for multi-representation bi-encoders are poly-encoder (Humeau et al.,
2020), ME-BERT (Luan et al., 2021), ColBERT (Khattab and Zaharia, 2020),
ColBERTer (Hofstätter et al., 2022), MVR (Zhang et al., 2022a) and MADRM (Kong
et al., 2022).

In contrast to bi-encoders, cross-encoders receive the concatenated query and
text pair, distinguished by a separation token [SEP], as input as shown in
Figure 2(Qiao et al., 2019). They utilise a cross-attention mechanism to
compute the interaction between any two tokens in the input and encode every
token in vector space. Thus, cross-encoders enable the tokens to interact
across queries and text (Guo et al., 2017; Xiong et al., 2017). We can then
compute match representations for the query-text pair using the learned
representations. Primarily, the encoding of the [CLS] token is used for this
semantic matching (Zhao et al., 2022; Wang et al., 2022). Nevertheless, we can
also average over all token embeddings to calculate the similarity measurement
(Reimers and Gurevych, 2019). The cross-encoder’s output is a fine-grained
relevance score for the query text pair, and a higher score indicates higher
relevance of the text, given the query.

Utilising a bi-encoder or a cross-encoder has different up- and down-sides.
Their two-tower architecture makes bi-encoders computationally more efficient
than cross-encoders since approximate nearest neighbour search enables fast
recall of large-scale vectors, and only the query, but not the textual
information, must be encoded during query time. Since cross-encoders calculate
relevance scores for every possible query and text pair, end-to-end
information retrieval is not possible, as they do not create independent
representations of query and text. Thus, the encoding must be recomputed every
time for every query text pair (Zhao et al., 2022; Wang et al., 2022).
However, bi-encoders need more training data than cross-encoders since they
independently map the documents to vector space. They also reach lower
retrieval accuracy.

Due to these pros and cons, bi- and cross-encoders are suited for different
tasks in information retrieval systems (Zhao et al., 2022). Bi-encoders are
used in first-stage retrieval to fetch the candidate documents, while Cross-
encoders are adapted as re-rankers or readers. In general, cross-encoders
perform better if applied to zero-shot retrieval tasks on out-of-domain data,
and multiple methods of domain adaptation dense retriever in low-resource
setups rely on cross-encoder architectures (Wang et al., 2022). However, some
tricks can be applied during bi-encoder training for first-stage retrieval to
improve their ability to generalise to new domains in a low-resource setup.
Therefore, the following section will give an overview of how to train bi-
encoders.

###  4.3. Training bi-encoders for first stage dense retrieval

First, we outline how to train a first-stage dense retriever with a bi-encoder
architecture. Afterwards, we summarise how to select negative documents for
that setting.

We jointly optimise a query and a document-oriented loss function while
training first-stage retrievers.

The query-oriented loss function is the exact negative log-likelihood
(Karpukhin et al., 2020). It maximises the probability of the relevant
(positive) document with respect to the query. The formula for this positive
query-document pair is:

(2)
$L(q_{i},d_{i}^{+})=-log\frac{e^{f(\phi(q_{i}),\psi(d_{i}^{+}))}}{e^{f(\phi(q_{i}),\psi(d_{i}^{+}))}+\sum_{d^{\prime}\in
D^{-}}e^{f(\phi(q_{i}),\psi(d^{\prime}))}}$

where $d_{i}^{+}$ is a relevant document for query $q_{i}$, $\phi(\cdot)$ and
$\psi(\cdot)$ are the query and text encoder and $f(\cdot)$ measures the
similarity between query embedding $\phi(q_{i})$ as well as text embedding
$\psi(d_{i}^{+})$. $D^{-}$ is the set of all documents except the positive
one(s) Since the formula iterates over all indexed documents in the
normalisation term computing, it is very time-consuming. The negative sampling
trick was introduced to the negative log-likelihood to tackle that issue
(Karpukhin et al., 2020; Qu et al., 2021). It reduces computational costs
compared to the exact formulation by sampling a set of negatives from all
documents. Its objective can be summarised as increasing the likelihood of
positive documents while decreasing the likelihood of sampled negative
documents. We formalise it as:

(3)
$L(q_{i},d_{i}^{+})=-log\frac{e^{f(\phi(q_{i}),\psi(d_{i}^{+}))}}{e^{f(\phi(q_{i}),\psi(d_{i}^{+}))}+\sum_{d^{\prime}\in
N_{q_{i}}}e^{f(\phi(q_{i}),\psi(d^{\prime}))}}$

where $N_{q_{i}}$ is a small set of negative samples for the query $q_{i}$. It
was derived from the InfoNCE loss (Oord et al., 2018), which contrasts a
positive pair of examples with randomly sampled examples.

The document-oriented loss function, which we jointly optimised with the
query-oriented loss function to train first-stage retrievers, is the negative
log-likelihood oriented at text (Xu et al., 2022):

(4)
$L_{T}(q_{i},d_{i}^{+})=-log\frac{e^{f(\psi(d_{i}^{+}),\phi(q_{i}))}}{e^{f(\psi(d_{i}^{+}),\phi(q_{i}))}+\sum_{q^{-}\in
Q^{-}}e^{f(\psi(d_{i}^{+}),\phi(q^{-}))}}$

where $Q^{-}$ is a set of sampled negative queries. In the following sections,
we abbreviate the combination of the query and document-oriented loss
functions, including the negative sampling trick as ”NLL+NS”.

To make the differentiation between positive and semantically close negative
query-document pairs more precise. We can add a text-text similarity
constraint to the loss of the bi-encoder performing the first-stage retrieval
(Ren et al., 2021). The constraint ensures that the similarity between a
positive document and the respective query is bigger than between a positive
document and a negative document. This objective can be formalised as:

(5) $f_{sim}(d^{+},q_{i})>f_{sim}(d^{+},d_{-})$

The complete loss function with the constraint is:

(6)
$L_{TT}(q_{i},d_{i}^{+})=-log\frac{e^{f(\psi(d_{i}^{+}),\phi(q_{i}))}}{e^{f(\psi(d_{i}^{+}),\phi(q_{i}))}+\sum_{d^{\prime}\in
N_{q_{i}}}e^{f(\psi(d_{i}^{+}),\psi(d^{\prime}))}}$

The sum in the normalisation term incorporates the text-text similarity.

One of the main training issues of tIR systems arises from the NLL+NS loss. It
is based on the large-scale candidate document space, while only a few are
relevant to the query. Computational restrictions only allow us to sample a
few negative documents for every query during training. Thus, the candidate
space of documents shifts during training compared to testing for which we use
the entire collection. Therefore, selecting negative documents during training
is crucial to retrieval performance as it influences convergence for the
gradient norms. Negative selection also impacts the capability of a trained
retrieval model to adapt to out-of-domain data distributions (Karpukhin et
al., 2020; Qu et al., 2021; Xiong et al., 2020). Therefore, we will now
introduce three prevalent negative selection strategies.

The first strategy selects in-batch negatives (IB) (Henderson et al., 2017;
Karpukhin et al., 2020). It uses all documents, except the positive one, per
batch as negatives. The main reason to apply this strategy is that dense
retrievers are often optimised on GPU in batches. Thus, we cannot access the
whole document corpus during training since keeping that in memory is
unfeasible. IB outperforms random sampling of negatives from the whole
document corpus before training since performing the sampling per batch during
training leads to more different negatives.

However, applying the second approach for negative selection, called cross-
batch negative selection, enables the first stage retriever to train on even
more negatives (Qu et al., 2021; Gao et al., 2021b). We train the model in
parallel on the different GPUs in multi-GPU settings. To utilise the negatives
across GPUs. We first compute the text embeddings on all GPUs, then
communicate those between GPUs and use them as negative documents for the
query. However, cross-batch negative selection can also be used in a single
GPU setting if gradient cashing is applied, accumulating mini-batches for
negative selection (Gao et al., 2021b). However, the larger candidate document
space for negative selection provided by cross-batch negative selection comes
at the cost of a higher computational effort than in-batch negative selection.

While in-batch and cross-batch negatives only increase the number of
negatives, the third strategy, hard negative selection, tries to increase the
quality of selected negatives. Hard negatives are negative documents
semantically similar to positive examples but irrelevant to the query. Three
different ways of selecting hard negatives exist. The first way generates
static hard negatives. Thus, during training, the negative selector is fixed.
Different methods can be used for the selection of static hard negatives. We
can use randomly selected top results from another sparse or dense retriever,
for example, the BM25 results, which are lexically similar but not relevant to
the query (Karpukhin et al., 2020)). Another option is to mix multiple types
of negatives. For example, retrieved negatives from BM25 and heuristic-based
context negatives (Lu et al., 2020). The combination performed in the second
method leads to better retrieval performance than the first. We can also apply
different fusion strategies for differently generated hard negatives to
improve retrieval performance even further (Lu et al., 2021b). Hofstätter et
al. (Hofstätter et al., 2021b) proposed topic-aware sampling of hard
negatives, an in-batch sampling technique that implicitly generates hard
negatives. They first cluster the queries, leading to higher similarity of
samples in a batch, and then sample the negative documents from one cluster
for every batch.

The second way to select hard negatives is to update them periodically during
training. Since the training of dense retrievers happens iteratively, updating
the hard negatives alongside the iterations is more suitable than fixing them.
Periodically updating the hard negatives generates more informative negatives
for training the discriminator, and they allow to weaken the train-test
discrepancy in the document candidate space since more informative negatives
are ”seen” during training (Zhao et al., 2022). However, dense retrievers must
be warmed up before training with dynamic hard negatives, increasing the
computational costs associated with this method (Xiong et al., 2020).
Different strategies for utilising dynamic hard negatives have been explored.
ANCE (Xiong et al., 2020) samples hard negatives from the top results returned
by the in-training retriever itself (global hard negatives). The advantage of
this approach is fast convergence. The disadvantage is that indexed document
embeddings must be updated after updating the model parameters, which is time-
consuming. A solution is to use an asynchronous index refresh strategy that
periodically updates the index for each $m$ batches. Another strategy is
called ADORE (Xiong et al., 2020). It samples documents from retrieval results
based on the in-training query encoder. The text encoder and text embedding
index are fixed. Since the query encoder for negative selection is optimised
during training, adaptive negatives for the same query are generated.

The third way of selecting hard negatives uses denoising strategies to ensure
that the selected hard negatives contain fewer false negatives. Again,
multiple methods applying this third strategy exist. RocketQA (Qu et al.,
2021) uses a cross-encoder to sample the top documents likely to be false
negatives. They use the hard negatives for the bi-encoder as candidate
documents for the cross-encoder to enable faster computation, thus refining
the hard negatives selected to train the bi-encoder. SimANS (Zhou et al.,
2022) introduces another method. The authors use ambiguous negatives. Thus,
negatives ranked close to positives, with moderate similarity. They prove
these are less likely to be false negatives and more informative.

Empirical studies showed that in-batch negatives are not informative enough to
train first-stage dense retrievers (Xiong et al., 2020; Lu et al., 2021b,
2020). Since the method only considers a small subspace of candidate documents
as possible negatives during training, it does not sample from the entire
distribution. This smaller candidate space results in the probability of
sampling informative negatives being close to 0 (Xiong et al., 2020). However,
according to Zhan et al. (Zhan et al., 2021), hard negative and random
negative sampling optimise different retrieval objectives. Random negative
sampling minimises the total pairwise error, possibly overemphasising learning
to handle complex queries. In contrast, hard negative sampling minimises the
top pairwise error, which is more suitable for optimising top rankings.
Combining both strategies during training is a valuable approach already used
to adapt dense retrievers to new domains, which we discuss in the next
section.

##  5\. Domain Adaptation of Semantic Search

Multiple strategies have been devised to facilitate the integration of dense
tIR systems into new domains with limited resources. To comprehensively
explore these approaches, we categorised them based on the components of the
IR system they modify. This categorisation is particularly valuable, as the
potential for combining distinct strategies becomes more feasible when they
target different aspects of the IR pipeline.

The approaches that adapt the data for training a tIR system either generate
(pseudo) query document pairs - utilising query generation or contrastive
learning - or enhance the quality of available data through semi-supervised
knowledge distillation. Other approaches directly alter the dense retrievers
by scaling up the model size or utilising a more capable model. The approaches
that adapt the training of models used for tIR employ a multi-task learning
approach, aim to create a domain-invariant embedding space or utilise
parameter-efficient learning to use pre-training techniques efficiently. An
approach that directly modifies the document ranking output by tIR systems
integrates sparse retrieval to enhance the system’s term-matching capability.
In the following, we will discuss these approaches in detail and close this
section by analysing their compatibility.

###  5.1. Data Adaptation

The main problem of domain adapting dense tIR systems is that, often, no
labelled data for the target domain is available. However, unlabelled
documents are easier to obtain. Thus, generating query-document pairs from
unlabelled text data is one approach to the domain adaptation of dense IR
systems. Forming these pairs can be done by query generation or contrastive
learning.

####  5.1.1. Query Generation

Query generation approaches form positive query-document pairs by generating
queries not found in the documents, utilising a data generation model (Liang
et al., 2020; Wang et al., 2022; Reddy et al., 2021; Ma et al., 2021b; Dai et
al., 2022b). The query generation is achieved by combining a Query Generator
(QG) with a filtering mechanism to enhance the quality of the generated
queries. Three different categories of QGs can be distinguished. Rule-based
QGs apply handcrafted features and templates to generate queries. However,
they are time-consuming to design, domain-specific and only generate certain
question forms (Pandey, 2013; Rakangor and Ghodasara, 2015). Prompt-based QGs
utilise generative PLMs, which can be presented with documents and the prompt
to generate a query (Bonifacio et al., 2022; Sachan et al., 2023; Dai et al.,
2022a, b). Supervised QG use a pre-trained language model, fine-tuned on in-
or out-of-domain data.

These supervised GQs can be trained end-to-end. Then, they directly generate
query-text pairs as training data for the dense retriever. Shakeri et al.
(Shakeri et al., 2020) used this approach. They utilised a pre-trained
language model and reframed the task of question-answer generation to machine
reading comprehension. Based on that, they trained a seq2seq network that
generates a question-answer pair given an input text. Reddy et al. (Reddy et
al., 2021) adapted this idea and added a selection step to the approach,
enabling them to generate better question-answer pairs.

The second approach to training supervised QGs utilises a pipeline consisting
of multiple dense retrievers and generative models. Strategies that employ
this approach are QGen (Ma et al., 2021b), which uses a query generator
trained on general domain data to generate domain queries for the target
corpus. Then, a dense retrieval model is trained on this (pseudo) labelled
data from scratch. GPL (Wang et al., 2022) extends the QGen approach by using
(pseudo) labels from a cross-encoder combined with hard negatives. Alberti et
al. (Alberti et al., 2019) used a large text corpus to construct question-
answer-pairs in three stages: First answer extraction, second question
generation and third roundtrip filtering. Lewis et al. (Lewis et al., 2021)
extended this approach by adding passage selection and global filtering. The
supervised QGs generate the best queries, while the other GQs often generate
inadequate results (Shen et al., 2022).

The query generators can handle different inputs. If only a document is
passed, the QG can attend to different document spans to generate a query and
thus even generate multiple queries (Du and Cardie, 2017; Kumar et al., 2019).
Alternatively, an answer span from the document and the document can be passed
to the QG, generating a query conditioned on both (Alberti et al., 2019;
Shakeri et al., 2020). Some QGs can also handle more fine-grained information
like question types to generate (Cao and Wang, 2021; Gao et al., 2022). Other
QGs can process further context to make queries less ambiguous, reducing the
question entropy, thereby making learning easier for the QG while increasing
the probability of error propagation (Liu et al., 2022; Zhang and Bansal,
2019).

The second part of the query generation approach is a filtering mechanism to
enhance the quality of the generated queries. Different choices of filters
were explored. Round trip consistency can filter out low-quality generated
queries (Alberti et al., 2019; Dong et al., 2019). It uses a pre-trained QA
system to generate an answer based on the generated queries. The query is only
kept if the generated answer is the same as the answer used to generate the
query. An alternative filtering mechanism manually sets an acceptance
threshold to relax this rather strict condition. Possible metrics for
thresholding are the probability score provided by the QA system which
generated the answer from the (pseudo) query (Zhang and Bansal, 2019; Lewis et
al., 2021), the LM score from the QG generator itself (Shakeri et al., 2020;
Liang et al., 2020), or an entailment score which can be acquired from a model
trained on question-context-answer pairs (Liu et al., 2022). Other filtering
mechanisms are influence functions to estimate the effect of including the
synthetic query on the validation loss (Yang et al., 2020b), ensemble
consistency for which an ensemble of QA models are trained and only the
questions onto which most QA systems agree are kept (Bartolo et al., 2021). If
some in-domain labelled samples are available, possible filter mechanisms are
reweighting the (pseudo) samples based on the validation loss (Sun et al.,
2021) or selecting samples that lead to validation performance gains (Yue et
al., 2022b).

####  5.1.2. Contrastive Learning

However, training another model to generate query document pairs is
computationally expensive. Self-contrastive learning presents another avenue
for constructing (pseudo) query-document pairs. It involves utilising various
parts of documents or external knowledge as (pseudo) queries. Different
strategies have been proposed for constructing positive query document pairs.

Perturbation-based methods introduce augmentations to the original text,
treating the augmented and original text as positive pairs (Shen et al.,
2022). Possible augmentation methods are word deletion, substitution or
permutation (Zhu et al., 2021; Yu et al., 2022), adding drop out in the
representation layers (Gao et al., 2021a) or passing sentences through
multiple language models (Carlsson et al., 2021).

Summary-based methods for constructing positive pairs involve extracting
document segments that likely summarise the content and using them as (pseudo)
queries. Suitable document parts are the document title (MacAvaney et al.,
2017, 2019; Mass and Roitman, 2020), a random sentence from the first document
section (Chang et al., 2020), randomly sampled n-grams from the document
(Chang et al., 2020) or generating several keywords from the document (Ma et
al., 2021a).

Proximity-based methods form positive pairs based on proximity positions in
documents. Different approaches are ICT (Lee et al., 2019a), which uses a
random sentence from a passage as the query and the same passage, but without
the selected sentence, as the positive document. ICT can be combined with
noise, for example, drop-out masks (Xu et al., 2022) or word chopping/
deletion (Izacard et al., 2022) to improve robustness. Other proximity-based
approaches are selecting random spans from the same document as positive pairs
(Gao and Callan, 2021b; Ma et al., 2022), selecting sentences from the same
paragraph and paragraphs from the same document as positive pairs (Di Liello
et al., 2022).

Co-occurrence-based methods utilise sentences containing a co-occurrent span
for constructing positive query document pairs (Ram et al., 2021). One can,
for example, use random sentences from the document corpus to construct a
(pseudo) query, using a term in the sentence as the answer and replacing it
with a unique token. Then, using the answer term to retrieve passages with
BM25, these passages are the (pseudo) positive documents (Glass et al., 2020).
Alternatively, one can use a token span and its surrounding context to form a
(pseudo) query and a passage that contains the same span as the (pseudo)
document (Ram et al., 2022). However, using the second approach to construct
the training data biases it towards sparse retrieval due to the high lexical
overlay in query-document pairs. Thus, it is unsuitable for comparing dense
and sparse retrieval approaches.

Hyperlink-based methods construct the positive query document pairs for
contrastive learning by exploiting hyperlink information, often capitalising
on anchor-document relations. They assume that anchor-document relations
approximate query-document relevance and use these relations to build the
positive query-document pairs (Zhang et al., 2020a; Ma et al., 2021b). An
example is to use a sentence from the first section in a document as the
(pseudo) query since this section often describes or summarises the topic and
a sentence from another document that contains a hyperlink to the first
document as the (pseudo) positive document (Chang et al., 2020). Another
example is replacing one token with a w-question phrase to form the (pseudo)
query. A passage containing the replaced token/entity from a document
containing a hyperlink to the first document can be used as the (pseudo)
relevant document (Yue et al., 2022a). The hyperlink approach performs best in
practice, but documents are only sometimes linked. Thus, it is not always
applicable (Sun et al., 2021). Utilising reinforcement learning to select
suitable (pseudo) query document pairs automatically is another option instead
of settling for one of the five approaches. However, it makes the training
procedure even slower (Zhang et al., 2020b). Apart from generating (pseudo)
query document pairs, contrastive learning can also be used as task-adaptive
pre-training. Zhao et al. (Zhao et al., 2022) performed a more in-depth
discussion of dense retrieval pre-training.

####  5.1.3. Knowledge Distillation

The two query generation methods discussed so far only need unlabelled text
data as input and aim at generating (pseudo) query document pairs. The
following method, knowledge distillation, requires independent documents and
queries from the target domain, which must still be linked as pairs. Knowledge
distillation automatically establishes the missing link, forming (pseudo)
query-document-pairs and generating relevance judgments.

The general definition of knowledge distillation in machine learning is to
transfer knowledge from a more capable model (teacher) to a less capable model
(student). For dense retrieval, we use it to improve the bi-encoder’s
(student) performance by utilising a more capable model (teacher). Knowledge
distillation is a form of distant supervision (Thakur et al., 2021a; Chen et
al., 2022a; Wang et al., 2022). The domain adaptation of the dense tIR system
is performed by using the teacher to generate (more) precise (pseudo) relevant
judgments for the document. Afterwards, we use the generated (pseudo) labelled
data to train the student model.

Different models can be used as teachers in a knowledge distillation setting.
If a pre-trained cross-encoder is used, we can either train it independently
from the student bi-encoder or incorporate information interaction of teacher
and student model when training the teacher (Qu et al., 2021). We do so by
randomly sampling the top-ranked texts of the student network as negatives for
training the teacher. Thus, the cross-encoder can adapt to bi-encoder results.
Another approach utilises two cross-encoders as teachers and trains them using
in-batch and pairwise negatives to improve the student model’s retrieval
performance (Hofstätter et al., 2021b). Both using a single teacher as well as
an ensemble of teachers improves the performance of the student model
(Hofstätter et al., 2021a). An enhanced bi-encoder (e.g. ColBERT (Khattab and
Zaharia, 2020)) can also be used as a teacher model (Lin et al., 2020).
Sometimes, even using a sparse retriever as the teacher network is desirable
since dense retriever (DR) and sparse retriever (SR) have complementary
strengths and weaknesses. SRs are better at exact matching and retrieving long
documents as well as generalising to out-of-domain distributions, while DRs
can perform semantic matching (Chen et al., 2022b; Luan et al., 2021; Thakur
et al., 2021a; Chen et al., 2022a). Utilising an SR as the teacher network for
the bi-encoder enhances its term-matching capabilities. The resulting model
reaches a better performance than only BM25 on test data. It enables the
student model to match rare entities, thus improving the zero-shot out-of-
domain performance (Chen et al., 2022a). Knowledge distillation might be more
efficient if the gap between the student and the teacher network is smaller
(Rezagholizadeh et al., 2022; Zhou et al., 2021). Thus, progressive
distillation adaption is helpful if the teacher model is way more capable than
the student model (Zeng et al., 2022; Lu et al., 2022; Lin et al., 2023a). It
can be implemented in two ways: either gradually making the teacher model more
capable at different stages of distillation to mimic the student model’s
learning (Lu et al., 2022; Lin et al., 2023a) or fixing the strong teacher
model, but gradually increasing the difficulty of the distilled knowledge
(Zeng et al., 2022; Ren et al., 2023a).

The outputs generated by the teacher model can be passed to the student as
either hard or soft labels. Hard labels are generated by directly setting the
relevance scores generated by the teacher model as binary data labels. To
exclude false positives, one can use thresholding methods (e.g. all documents
with a higher score than 0.9 = positives, with ¡0.1 = negatives, discard rest)
(Qu et al., 2021). In this approach, the finer-grained output of the teacher
model is ignored, but it utilises the same training objective as standard DR
training. Soft labels are generated by tuning the student model to approximate
the teacher’s output using different distillation functions. This approach
allows utilising the fine-grained output generated by the teacher model. The
MSE loss minimises the distance between the relevance scores of student and
teacher model:

(7) $L^{KD}_{MSE}=\frac{1}{2}\sum_{q\in Q}\sum_{d\in
D}(r_{q,d}^{(t)}-r_{q,d}^{(s)})^{2}$

where $Q$ is the set of all queries and $D$ is the set of all documents and
$r_{q,d}^{(t)}$ are the relevance scores of document $d$ with respect to query
$q$ as assigned by the teacher model, while $r_{q,d}^{(s)}$ are the relevance
scores of document $d$ with respect to query $q$ as assigned by the student
model.
The KL-divergence loss normalises the relevance scores of the candidate
documents as probability distributions per query and then reduces their KL-
divergence:

(8) $L^{KD}_{KL}=-\sum_{q\in Q,d\in
D}(log\tilde{r}_{q,d}^{(s)}-log\tilde{r}_{q,d}^{(t)})$

where $\tilde{r}_{q,d}^{(s)}$ is the normalised probability distribution of
the candidate documents by query of the student model’s relevance scores and
$\tilde{r}_{q,d}^{(t)}$ is the normalised probability distribution of the
candidate documents by query of the teacher model’s relevance scores.
The Max-Margin loss penalises the inversions of rankings generated by the
retriever:

(9)
$L^{KD}_{MM}=\sum_{q,d_{1},d_{2}}\text{max}(0,\gamma-\text{sign}(\Delta^{(t)}_{q,d_{1},d_{2}}\Delta^{(s)}_{q,d_{1},d_{2}})$

where $q\in Q$ and $d_{1},d_{2}\in D$, while
$\Delta^{(t)}_{q,d_{1},d_{2}}=r_{q,d_{1}}^{(t)}-r_{q,d_{2}`}^{(t)}$ and
$\Delta^{(s)}_{q,d_{1},d_{2}}=r_{q,d_{1}}^{(s)}-r_{q,d_{2}`}^{(s)}$
$\gamma$ is the margin, $sign(\cdot)$ is the sign function, which indicates
whether the value is positive, negative or zero.
The Margin MSE loss minimises the margin difference of a positive-negative
document pair between the teacher and student model

(10)
$L^{KD}_{M-MSE}=MSE(r_{q,d^{+}}^{(t)}-r_{q,d^{-}}^{(t)},r_{q,d^{+}}^{(s)}-r_{q,d^{-}}^{(s)})$

Izacard and Grave (Izacard and Grave, 2022) find that the KL divergence loss
leads to better distillation than MSE and max-margin loss for question-
answering tasks but that the Margin-MSE loss is most effective compared to the
other options. This result might be attributed to margin-based losses, making
the model less likely to overfit the synthetic training data. Thus, using
those should be preferred if the teacher’s output contains significant noise.

Knowledge distillation aims to adapt the bi-encoder model to the target-domain
data distribution. Its bottleneck is the quality of the teacher model. If the
teacher’s output contains much noise, the student model will overfit the
noise. A possible solution for that problem, if the teacher’s output scores
display actual output confidence, is to use confidence-based filtering as a
noise-resistant algorithm (Mukherjee and Awadallah, 2020; Yu et al., 2021).
However, the teacher’s output scores are less or un-reliable in challenging
tasks (Zhu et al., 2023). Another issue of knowledge distillation is its’
computational costs (Xu et al., 2020; Rao et al., 2022). However, if enough
computational power is available, knowledge distillation can be used to
generate more fine-grained relevance labels, enhancing the quality of the
training data, and it can be combined with query generation methods.

All three methods augmenting the tIR system’s data level have the same
problem. The generated labels tend to be noisy, meaning they contain more
errors than manually labelled data. Training on such noisy data can hurt the
model’s performance. Thus, two approaches to handle the noise have been
explored (Hedderich et al., 2021a). The first is noise filtering. Hard noise
filtering removes instances that are incorrectly labelled with a high
probability. We can find the incorrect labels through a probability threshold
(Jia et al., 2019), a binary classifier (Adel and Schütze, 2018; Onoe and
Durrett, 2019; Huang and Du, 2019) or a reinforcement-based agent (Yang et
al., 2018; Nooralahzadeh et al., 2019). Soft noise filter reweights the
labelled query-document-pairs according to their probability of being
incorrect (Le and Titov, 2019) or through attention measurements (Hu et al.,
2019). The second approach to handle the noise in the data is noise modelling.
It can be performed using correlation coefficients to estimate the
relationship between clean and noisy labels (Chen et al., 2019; Hedderich et
al., 2021b). Then, a noise model is applied to shift the distribution of the
noisy data to the (unseen) distribution of the clean data. Afterwards, we
train the dense retrievers on the data with the new ”clean” distribution. Chen
et al. (Chen et al., 2020a) use multiple reinforcement agents to re-label
noisy instances. Ren et al. (Ren et al., 2020) apply multiple sources of
distant supervision and learn how to combine them. Most of these denoising
approaches were suggested for named entity recognition tasks. However, their
strategies also apply to the data adaptation methods introduced in this
section.

###  5.2. Model Adaptation

Apart from tackling the problem of domain-adapting dense retrievers in low-
resource scenarios on the data level, we can also adapt the models themselves.
Zhan et al. (Zhan et al., 2022) showed that cross-encoders are more capable of
generalising to out-of-domain test data than bi-encoders and sparse
retrievers. However, their high computational costs during inference make
their practical application impractical. Ni et al. (Ni et al., 2021) scaled up
the size of a T5 dense-retriever through multi-stage training and reached a
better zero-shot retrieval performance on several BEIR (Thakur et al., 2021b)
datasets. While this approach also leads to higher computational costs, the
increasing availability of computational resources might make both approaches
feasible.

###  5.3. Training Adaptation

However, Instead of directly altering the models to improve their zero-shot
performance, we can use multi-task, domain-invariant or parameter-efficient
learning to adapt the model training in the dense tIR pipeline.

Multi-task learning involves jointly training self-supervised tasks, dense
retrieval, and extractive question answering (Fun et al., 2021). This approach
can significantly improve the zero-shot setting and enhance the model’s
generalisation capabilities (Aghajanyan et al., 2021).

The joint training across multiple tasks mitigates the model’s propensity to
overfit a specific task. The multi-task learning paradigm improves
performance, even when a substantial amount of labelled training data from the
target domain is available (Maillard et al., 2021). Several approaches have
been proposed within this framework: One suggests fine-tuning a universal
retriever on multiple, diverse tasks, effectively training the model to handle
various domains and tasks simultaneously. Another task-specific retriever
ensembling approach entails training separate retrievers for distinct tasks
and aggregating their predictions. As demonstrated by Li et al. (Li et al.,
2021), this is particularly advantageous when dealing with document corpora
that display inconsistencies. To further expedite domain adaptation, combining
multi-task learning with meta-learning techniques has shown promise (Finn et
al., 2017; Qian and Yu, 2019; Dai et al., 2020; Park et al., 2021).

Domain-invariant learning is another approach to adapting the model training
in dense tIR pipelines. It aims to equip models with representations that
remain invariant across different domains to enable better generalisation from
a source domain to a target domain (Ganin and Lempitsky, 2015; Ganin et al.,
2016). The core training objective in domain-invariant learning consists of
two components, represented as follows:

(11) $L=L_{rank}+\lambda L_{DiL}$

where $\lambda$ is a tunable hyperparameter, $L_{rank}$ is the ranking loss
typically applied to out-of-domain trainings data:

(12) $-\min_{R}\mathbb{E}_{q,d^{+},d_{1\sim n}^{-}\in Q_{o}\times
D_{o}}\mathcal{O}(R,q,d^{+},d^{-}_{1\sim n})$

The second loss function component, $L_{DiL}$, is the domain-invariant
learning loss, designed to minimise the distance between the embedding spaces
of the source and target domain:

$\displaystyle L_{DiL}=\min_{E_{1},E_{2}}\mathbb{E}_{q_{t}\in Q_{t},d_{t}\in
D_{t},q_{0}\in Q_{o},d_{o}\in D_{o}}$ $\displaystyle
D(E1(q_{o})||E1(q_{t}))+D(E2(d_{o})||E2(d_{t}))$

where $D$ is a discrepancy function, $E_{1}$ corresponds to the query encoder,
and $E_{2}$ is the document encoder. Two popular choices for the discrepancy
function, $D$, have emerged. The first choice, Maximum-Mean Discrepancy (MMD),
quantifies the distance between distributions in probability space. It
utilises the norm of the difference between the first moments of both domain
embedding distributions (Gretton et al., 2009; Tzeng et al., 2014; Long et
al., 2015).

(13) $D_{MMD}(P,Q)=||\mu_{P}-\mu_{Q}||_{H}$

where $P$ and $Q$ are the distributions of two different domains and, $\mu$ is
the distribution mean, estimated by the empirical mean over batches (Tran et
al., 2019) and $H$ is the corresponding reproducing Kernel Himert space
(Berlinet and Thomas-Agnan, 2004). Additional regularisation techniques, such
as minimising second moments, have been applied to enhance domain invariance
(Chen et al., 2020b; Liu et al., 2022). However, only focusing on the
distributions’ first and second moments does not guarantee their domain
invariance. Another approach randomly projects both embedding probability
distributions, making them one-dimensional and Gaussian-like. Afterwards,
minimising the first two moments makes the whole distributions similar to one
another (Tran et al., 2019).

The second choice for the discrepancy function $D$ is adversarial learning. It
uses a discriminator $f$ to estimate the density ratio of two distributions
(Goodfellow et al., 2014b):

(14) $\min_{f}\mathbb{E}_{x\sim P/Q}-logf(l_{x}|x)$

where $l_{x}$ is the label indicating from which distribution ($P$ or $Q$) $x$
stems. If $f$’s training objective is modified, it can be utilised to estimate
distance metrics between two distributions, like the Jesson-Shannon or
Wasserstein divergence (Goodfellow et al., 2014a; Arjovsky et al., 2017):

(15) $D_{adv}(P,Q)=\mathbb{E}_{x\sim P/Q}L(U|f(l_{x}|x))$

where $U$ is the uniform distribution, and $L$ is either a cross-entropy or a
KL divergence loss function. The model is trained iteratively, optimising the
first and second functions from above to make it impossible for $f$ to
distinguish from which domain encoded embeddings stem since both domains share
the same distribution space. Adversarial learning methods have been applied in
various natural language processing tasks. Lee et al. (Lee et al., 2019b) show
that adversarial learning can be used for question-answering tasks and
performs better than a baseline model. Wang et al. (Wang et al., 2019) apply
adversarial learning to synthetically generated queries. Shrivastava et al.
(Shrivastava and Wang, 2022) introduce the annealing trick, which prevents the
discriminator from providing misleading signals during early training stages.
Xin et al. (Xin et al., 2022) apply domain invariant learning to dense
retriever training. They propose maintaining a momentum queue averaging over
past batches, so feeding all data points to the classifier at every batch is
not needed. The authors aim to balance the efficiency and robustness of the
domain classifier and show that the performance is better than that of a
baseline model. However, Xin et al. do not compare domain invariant learning
to other domain adaptation techniques.

Balancing the ranking and domain-invariant learning loss requires careful
parameter tuning. When adversarial learning is employed, adjusting the update
rates of the dense retriever and discriminator becomes even more challenging
(Wang et al., 2020; Shrivastava and Wang, 2022). The training process for
domain-invariant learning is generally more complex than other approaches.
While domain-adaptive training is a promising approach, its potential when
combined with other techniques or its practicality in low-resource settings
must be explored further (Shen et al., 2022).

Parameter-efficient learning is the third approach to improving the zero-shot
performance by adapting the model training in the tIR pipeline. It is based on
modifying only a subset of the model’s parameters, allowing it to sustain
general knowledge learnt in pre-training and thus better generalisation in
low-resource setup. The different pre-training strategies this approach can
enhance are task-adaptive pre-training, which mimics the retrieval task in a
self-supervised way (Lee et al., 2019a; Chang et al., 2020; Ram et al., 2022).
Retrieval augmented pre-training which aims to improve the modelling capacity
of pre-trained language models by integrating a so-called knowledge retriever
that provides referring context for the Masked Language Modelling task (Guu et
al., 2020) and representation enhanced pre-training, which aims to enhance the
amount of information stored in the embedding of the ”[CLS]” token.
Representation enhanced pre-training is performed by autoencoder enhanced pre-
training (Gao and Callan, 2021a; Wang et al., 2021a; Gao and Callan, 2021b; Lu
et al., 2021a; Wang et al., 2023; Xiao et al., 2022; Liu and Yang, 2022) or
contrastive learning enhanced pre-training (Gao et al., 2021a; Yan et al.,
2021; Xu et al., 2022; Ma et al., 2022). All these pre-training techniques can
aid domain adaptation of dense retrievers in low-resource setups in varying
ways. Possible approaches to applying parameter-efficient learning - to
sustain the general knowledge a model acquires during pre-training - are:
adapters (Houlsby et al., 2019) or prompt-tuning (Li and Liang, 2021; Liu et
al., 2021b). Tam et al. (Tam et al., 2022) even use parameter-efficient prompt
tuning for retrieval tasks in-domain, cross-domain and cross-topic while only
fine-tuning 0.1% of the model’s parameters, leading to better generalisation
performance than in the case of fine-tuning all parameters.

###  5.4. Ranking Adaptation

Ranking adaptation of dense retrievers is performed by integrating sparse-
retrieval to combine the strengths of both approaches. Either the dense and
sparse retriever relevance scores are directly combined (Wang et al., 2021b;
Xu et al., 2022; Karpukhin et al., 2020) or an unsupervised retriever is
improved by interpolating BM25 and DR relevance scores (Ram et al., 2022; Ren
et al., 2023b). Another approach utilises RM3 (Jaleel et al., 2004) built on
top of the sparse results for selecting deep retriever results as the final
list (Kuzi et al., 2020) and Chen et al. (Chen et al., 2022b) use robust rank-
based scores leading to significant zero-shot performance improvements.
Another option is to use dense retrievers to generate term weights for
improving sparse retrieval results (Dai and Callan, 2020) or to apply a
classifier to decide whether to use dense, sparse or hybrid retrieval results
for each query (Arabzadeh et al., 2021). However, while integrating sparse
retrieval into dense retrieval systems enhances their term-matching
capabilities, this approach requires maintaining two indexing systems
simultaneously, which is often not feasible in practice. Nevertheless, some
approaches that do not require a dual index but still enhance the term-
matching capacities of dense retrieval systems exist. It is, for example,
possible to learn low-dimensional dense lexical representations (Lin et al.,
2023b; Lin and Lin, 2021) or to train a dense lexical retriever based on
weakly supervised data constructed by BM25 (Chen et al., 2022a). Another
option are pre-trained language models which have lexical awareness (like
SPLADE (Formal et al., 2021)) (Zhang et al., 2023). They are trained using
lexicon-augmented contrastive learning and rank-consistent regularisation.

###  5.5. Discussion

Apart from parameter-efficient learning, all the introduced methods lead to
higher computational costs for building dense tIR systems while improving
their zero-shot capabilities. They also require different data inputs, as
shown in Table 1.

Table 1. Domain-Adaptation Approaches

Adaptation Level Method Data Requirements Target Domain Data Query Generation
Only Text Contrastive Learning Only Text Knowledge Distillation Text and
Queries Model Size Labelled Data Capability Labelled Data Training Multi-task
Labelled Data Domain-Invariant Labelled Data Parameter-Efficient Labelled Data
Ranking Integrating Sparse Retrieval Labelled Data

Theoretically, all methods listed in Table 1 can be combined. Nevertheless,
combining methods further increases the computational costs of training the
system. Also, utilising contrastive learning and query generation in the same
tIR system does not make sense since they have the same objective of
generating (pseudo) query-document pairs. Thus, adding one or the other is
sufficient. However, combining knowledge distillation with query generation or
contrastive learning can generate fine-grained (pseudo) relevance judgments
for the generated (pseudo) query document pairs, mitigating the requirement of
independent queries and documents for knowledge distillation.

Ren et al. (Ren et al., 2023b) empirically compared different data-level
approaches to adapt dense retrievers to a target domain in a low-resource
setting. They found that LaPraDoR (Xu et al., 2022) outperformed other
methods. It combines contrastive learning with the integration of sparse
retrieval. However, if the BM25 part is removed, the performance on target
domain sets with considerable vocabulary overlap drops. Thus, integrating
lexical matching can improve retrieval performance on datasets with high
vocabulary overlap. For target data with low vocabulary overlap, the
improvement was lower. GPL (Wang et al., 2022) achieved competitive
performance since it improved vocabulary overlap and query type distribution
similarity between target and source data. It also increases the available
(pseudo) labelled data through query generation.

##  6\. Multilingual Semantic Search

Since most available labelled datasets are in English, developing multilingual
semantic search systems or even monolingual dense retrievers for low-resource
languages remains challenging. Two exceptions are mMARCO (Bonifacio et al.,
2021) \- a multilanguage version of MS MARCO, comprised of 13 languages,
created by machine translation - and Mr. TYDI (Zhang et al., 2021) \-
comprised of 11 languages.

Multilingual dense retrievers are trained on monolingual text representations
from multiple languages, generating multilingual text representations that
enable a single model to perform in multiple languages (Lange et al., 2020;
Devlin et al., 2018; Conneau et al., 2019). In zero-shot settings, they can be
fine-tuned on the target task in a high-resource language and then applied to
unseen target languages. Their performance is improved if we add a small
amount of target task and target language data. This transfer between
languages is achieved through creating a shared multilingual embedding space
by mapping between the different embedding spaces of the two languages,
resulting in similar feature vectors for embeddings in the two vectors spaces
(Lauscher et al., 2020; Hedderich et al., 2020; Mikolov et al., 2013; Joulin
et al., 2018). With this method, single languages can be differently embedded
by the same model and mapped to the same vector space afterwards (Cao et al.,
2020).

Recently, Feng et al. (Feng et al., 2022) introduced LaBSE, which creates
sentence embeddings with 768 dimensions and trains 471M parameters. LaBSE
performs well on sentence alignment tasks, partly attributed to the
multilingual model pre-training. However, the high dimensional embeddings and
the many parameters make adapting LaBSE to downstream tasks computationally
expensive. As an alternative, LEALLA (Mao and Nakagawa, 2023) is a distilled
encoder model to create multilingual sentence embeddings with lower
dimensions, reducing the computational overhead for down-stream tasks as well
as creating the embeddings themselves while sustaining competitive performance
on benchmark datasets for parallel sentence alignment. Before LEALLA, no
multilingual pre-trained lightweight language models were available. Thus, for
training LEALLA, the authors used LaBSE as a teacher in knowledge
distillation, combining feature and logit distillation.

In contrast to these two approaches, LASER (Artetxe and Schwenk, 2019) trains
an encoder-decoder LSTM model utilising a translation task. This translation-
based training enables LASER to perform well on exact translation tasks.
Nevertheless, LASER’s ability to accurately recognise the similarity of
sentences that are not exact translations could be improved. Another approach
is the Multilingual Universal Sentence Encoder (mUSE) (Chidambaram et al.,
2019; Yang et al., 2020a). MUSE was trained on general domain data in multiple
languages, utilising a multi-task setup, namely a translation ranking task in
which, given several incorrect and one correct translation, the latter needs
to be identified. However, this task requires hard negatives to enable state-
of-the-art performance, leading to a significant computational overhead (Guo
et al., 2018). Reimers and Gurevych (Reimers and Gurevych, 2020) further
explore the application of knowledge distillation to make monolingual pre-
trained language models multilingual.

Another application of multilingual language models is cross-lingual
retrieval. For example, XOR QA (Asai et al., 2021a) answers a question in one
language based on a document in another language, and Asai et al. (Asai et
al., 2021b) utilise iterative training to fine-tune a multilingual PLM to
build a cross-lingual dense passage retriever which can retrieve candidate
passages from a multilingual document collection.

##  7\. Discussion

An interesting approach to explore further would be utilising cross-lingual
models for query generation on texts from a specialised target domain. Would
they outperform existing machine translation systems? Furthermore, the
approach to use a capable, multilingual teacher in knowledge distillation, as
already investigated by Mao et al. (Mao and Nakagawa, 2023) as well as Reimers
and Gurevych (Reimers and Gurevych, 2020) might be expandable to performing
domain-adaptation if a pre-trained student model utilising parameter efficient
learning is applied. Moreover, domain-invariant learning and the training of
multilingual models share a similar objective. Therefore, it would be
interesting to see whether issues arising from the complex loss formulation
for domain-invariant learning could be mitigated by applying a mapping
function to create a joint space for source- and target-domain embeddings.
Additionally, would it be possible to extend a shared source- and target
domain vector space to include multilingual embeddings?

##  8\. Outlook and Conclusion

This literature review explored how to perform domain adaptation for
multilingual semantic search. We introduced a systematic clustering of domain
adaptation methods in a low resource setup, based on parts of the tIR system
they alter, summarised in Table 1. Focusing on tIR system parts and data
requirements of the different approaches helps clarify their combination
possibilities. Afterwards, we gave an overview of recent advances in
multilingual semantic search, followed by a discussion on combining it with
the introduced domain adaptation approaches. The domain adaptation approaches
we introduced focus on monolingual semantic search. However, some multilingual
models already adapted knowledge distillation (Mao and Nakagawa, 2023; Reimers
and Gurevych, 2020). Combining other domain adaptation techniques with
multilingual semantic search is an exciting research direction to investigate
further. The questions we raised in Section 7 might spark some ideas in that
regard. A limitation of this work is that we did not distinguish sentence
encoders from other sequence encoding strategies in our analysis. Doing so in
future work would be fruitful.

## References

  * (1)
  * Adel and Schütze (2018) Heike Adel and Hinrich Schütze. 2018.  CIS at TAC Cold Start 2015: Neural Networks and Coreference Resolution for Slot Filling.  Online.  arXiv:1811.02230
  * Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021.  Muppet: Massive Multi-task Representations with Pre-Finetuning.  Online.  arXiv:2101.11038
  * Aizawa (2003) Akiko Aizawa. 2003.  An information-theoretic perspective of tf–idf measures.  _Information Processing and Management_ 39, 1 (jan 2003), 45–65.  https://doi.org/10.1016/s0306-4573(02)00021-3
  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019.  Synthetic QA Corpora Generation with Roundtrip Consistency.  Online.  arXiv:1906.05416
  * Arabzadeh et al. (2021) Negar Arabzadeh, Xinyi Yan, and Charles L. A. Clarke. 2021.  Predicting Efficiency/Effectiveness Trade-offs for Dense vs. Sparse Retrieval Strategy Selection.  Online.  arXiv:2109.10739
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017.  Wasserstein Generative Adversarial Networks. In _Proceedings of the 34th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 70)_ , Doina Precup and Yee Whye Teh (Eds.). PMLR, 214–223.  https://proceedings.mlr.press/v70/arjovsky17a.html
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond.  _Transactions of the Association for Computational Linguistics_ 7 (nov 2019), 597–610.  https://doi.org/10.1162/tacl_a_00288
  * Asai et al. (2021a) Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021a.  XOR QA: Cross-lingual Open-Retrieval Question Answering. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 547–564.  https://doi.org/10.18653/v1/2021.naacl-main.46
  * Asai et al. (2021b) Akari Asai, Xinyan Yu, Jungo Kasai, and Hannaneh Hajishirzi. 2021b.  One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval. In _Advances in Neural Information Processing Systems_ , A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (Eds.).  https://openreview.net/forum?id=e8blYRui3j
  * Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.  Online.  arXiv:1611.09268
  * Bartolo et al. (2021) Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021.  Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 8830–8848.  https://doi.org/10.18653/v1/2021.emnlp-main.696
  * Berlinet and Thomas-Agnan (2004) Alain Berlinet and Christine Thomas-Agnan. 2004.  _Reproducing Kernel Hilbert Space in Probability and Statistics_.  Springer New York, NY.  https://doi.org/10.1007/978-1-4419-9096-9
  * Bonifacio et al. (2022) Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022.  InPars: Data Augmentation for Information Retrieval using Large Language Models.  Online.  arXiv:2202.05144
  * Bonifacio et al. (2021) Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. 2021.  mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset.  Online.  arXiv:2108.13897
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  Online.  arXiv:2005.14165
  * Cao et al. (2020) Steven Cao, Nikita Kitaev, and Dan Klein. 2020.  Multilingual Alignment of Contextual Word Representations.  Online.  arXiv:2002.03518
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  Controllable Open-ended Question Generation with A New Question Type Ontology. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 6424–6439.  https://doi.org/10.18653/v1/2021.acl-long.502
  * Carlsson et al. (2021) Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021.  Semantic Re-tuning with Contrastive Tension. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=Ov_sMNau-PF
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017.  SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation.  Online.  arXiv:1708.00055
  * Chang et al. (2020) Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020.  Pre-training Tasks for Embedding-based Large-scale Retrieval.  arXiv:2002.03932
  * Chen et al. (2020a) Daoyuan Chen, Yaliang Li, Kai Lei, and Ying Shen. 2020a.  Relabel the Noise: Joint Extraction of Entities and Relations via Cooperative Multiagents.  Online.  arXiv:2004.09930
  * Chen et al. (2019) Junfan Chen, Richong Zhang, Yongyi Mao, Hongyu Guo, and Jie Xu. 2019.  Uncover the Ground-Truth Relations in Distant Supervision: A Neural Expectation-Maximization Framework.  Online.  arXiv:1909.05448
  * Chen et al. (2022b) Tao Chen, Mingyang Zhang, Jing Lu, Michael Bendersky, and Marc Najork. 2022b.  Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models.  In _Lecture Notes in Computer Science_. Springer International Publishing, 95–110.  https://doi.org/10.1007/978-3-030-99736-6_7
  * Chen et al. (2022a) Xilun Chen, Kushal Lakhotia, Barlas Oğuz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen tau Yih. 2022a.  Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?  Online.  arXiv:2110.06918
  * Chen et al. (2020b) Zhihong Chen, Rong Xiao, Chenliang Li, Gangfeng Ye, Haochuan Sun, and Hongbo Deng. 2020b.  ESAM: Discriminative Domain Adaptation with Non-Displayed Items to Improve Long-Tail Performance. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM.  https://doi.org/10.1145/3397271.3401043
  * Chidambaram et al. (2019) Muthu Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yunhsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model. In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_. Association for Computational Linguistics, Florence, Italy, 250–259.  https://doi.org/10.18653/v1/W19-4330
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the Properties of Neural Machine Translation: Encoder-Decoder Approaches.  Online.  arXiv:1409.1259
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised Cross-lingual Representation Learning at Scale.  Online.  arXiv:1911.02116
  * Dai et al. (2020) Yinpei Dai, Hangyu Li, Chengguang Tang, Yongbin Li, Jian Sun, and Xiaodan Zhu. 2020.  Learning Low-Resource End-To-End Goal-Oriented Dialog for Fast and Reliable System Deployment. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 609–618.  https://doi.org/10.18653/v1/2020.acl-main.57
  * Dai and Callan (2020) Zhuyun Dai and Jamie Callan. 2020.  Context-Aware Term Weighting For First Stage Passage Retrieval. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Virtual Event, China) _(SIGIR ’20)_. Association for Computing Machinery, New York, NY, USA, 1533–1536.  https://doi.org/10.1145/3397271.3401204
  * Dai et al. (2022a) Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. 2022a.  Dialog Inpainting: Turning Documents into Dialogs.  Online.  arXiv:2205.09073
  * Dai et al. (2022b) Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2022b.  Promptagator: Few-shot Dense Retrieval From 8 Examples.  Online.  arXiv:2209.11755
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  Online.  https://doi.org/10.48550/ARXIV.1810.04805 arXiv:1810.04805
  * Di Liello et al. (2022) Luca Di Liello, Siddhant Garg, Luca Soldaini, and Alessandro Moschitti. 2022.  Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 11806–11816.  https://doi.org/10.18653/v1/2022.emnlp-main.810
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified Language Model Pre-training for Natural Language Understanding and Generation.  Online.  arXiv:1905.03197
  * Du and Cardie (2017) Xinya Du and Claire Cardie. 2017.  Identifying Where to Focus in Reading Comprehension for Neural Question Generation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Copenhagen, Denmark, 2067–2073.  https://doi.org/10.18653/v1/D17-1219
  * Fan et al. (2022) Yixing Fan, Xiaohui Xie, Yinqiong Cai, Jia Chen, Xinyu Ma, Xiangsheng Li, Ruqing Zhang, and Jiafeng Guo. 2022.  Pre-training Methods in Information Retrieval.  arXiv:2111.13853 [cs.IR]
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic BERT Sentence Embedding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 878–891.  https://doi.org/10.18653/v1/2022.acl-long.62
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.  Online.  arXiv:1703.03400
  * Formal et al. (2021) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021.  SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.  Online.  arXiv:2107.05720
  * Fun et al. (2021) Hengxin Fun, Sunil Gandhi, and Sujith Ravi. 2021.  Efficient Retrieval Optimized Multi-task Learning.  Online.  arXiv:2104.10129
  * Ganin and Lempitsky (2015) Yaroslav Ganin and Victor Lempitsky. 2015.  Unsupervised Domain Adaptation by Backpropagation.  Online.  arXiv:1409.7495
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016.  Domain-Adversarial Training of Neural Networks.  Online.  arXiv:1505.07818
  * Gao and Callan (2021a) Luyu Gao and Jamie Callan. 2021a.  Condenser: a Pre-training Architecture for Dense Retrieval. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.  https://doi.org/10.18653/v1/2021.emnlp-main.75
  * Gao and Callan (2021b) Luyu Gao and Jamie Callan. 2021b.  Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval.  Online.  arXiv:2108.05540
  * Gao et al. (2022) Lingyu Gao, Debanjan Ghosh, and Kevin Gimpel. 2022.  “What makes a question inquisitive?” A Study on Type-Controlled Inquisitive Question Generation. In _Proceedings of the 11th Joint Conference on Lexical and Computational Semantics_. Association for Computational Linguistics, Seattle, Washington, 240–257.  https://doi.org/10.18653/v1/2022.starsem-1.22
  * Gao et al. (2021b) Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021b.  Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup.  Online.  arXiv:2101.06983
  * Gao et al. (2021a) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021a.  SimCSE: Simple Contrastive Learning of Sentence Embeddings.  Online.  arXiv:2104.08821
  * Glass et al. (2020) Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, and Avi Sil. 2020.  Span Selection Pre-training for Question Answering. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 2773–2782.  https://doi.org/10.18653/v1/2020.acl-main.247
  * Goodfellow et al. (2014a) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014a.  Generative Adversarial Nets. In _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2_ (Montreal, Canada) _(NIPS’14)_. MIT Press, Cambridge, MA, USA, 2672–2680.
  * Goodfellow et al. (2014b) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014b.  Generative Adversarial Networks.  Online.  arXiv:1406.2661
  * Gretton et al. (2009) A Gretton, AJ Smola, J Huang, M Schmittfull, and B Borgwardt, KM.and Schölkopf. 2009.  _Covariate shift and local learning by distribution matching_.  MIT Press, Cambridge, MA, USA, 131–160.
  * Guo et al. (2017) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2017.  A Deep Relevance Matching Model for Ad-hoc Retrieval.  Online.  arXiv:1711.08611
  * Guo et al. (2019) Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, and Xueqi Cheng. 2019.  A Deep Look into Neural Ranking Models for Information Retrieval.  Online.  arXiv:1903.06902
  * Guo et al. (2018) Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Effective Parallel Corpus Mining using Bilingual Sentence Embeddings. In _Proceedings of the Third Conference on Machine Translation: Research Papers_. Association for Computational Linguistics, Brussels, Belgium, 165–176.  https://doi.org/10.18653/v1/W18-6317
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  REALM: Retrieval-Augmented Language Model Pre-Training.  Online.  arXiv:2002.08909
  * Hedderich et al. (2020) Michael A. Hedderich, David Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, and Dietrich Klakow. 2020.  Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 2580–2591.  https://doi.org/10.18653/v1/2020.emnlp-main.204
  * Hedderich et al. (2021a) Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021a.  A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios.  Online.  arXiv:2010.12309
  * Hedderich et al. (2021b) Michael A. Hedderich, Lukas Lange, and Dietrich Klakow. 2021b.  ANEA: Distant Supervision for Low-Resource Named Entity Recognition.  Online.  arXiv:2102.13129
  * Henderson et al. (2017) Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017.  Efficient Natural Language Response Suggestion for Smart Reply.  Online.  arXiv:1705.00652
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ 9, 8 (nov 1997), 1735–1780.  https://doi.org/10.1162/neco.1997.9.8.1735
  * Hofstätter et al. (2021a) Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2021a.  Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation.  Online.  arXiv:2010.02666
  * Hofstätter et al. (2022) Sebastian Hofstätter, Omar Khattab, Sophia Althammer, Mete Sertkan, and Allan Hanbury. 2022.  Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction.  Online.  arXiv:2203.13088
  * Hofstätter et al. (2021b) Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021b.  Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling.  Online.  arXiv:2104.06967
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-Efficient Transfer Learning for NLP.  Online.  arXiv:1902.00751
  * Hu et al. (2019) Linmei Hu, Luhao Zhang, Chuan Shi, Liqiang Nie, Weili Guan, and Cheng Yang. 2019.  Improving Distantly-Supervised Relation Extraction with Joint Label Embedding. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 3821–3829.  https://doi.org/10.18653/v1/D19-1395
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013.  Learning deep structured semantic models for web search using clickthrough data. In _Proceedings of the 22nd ACM international conference on Conference on information & knowledge management - CIKM '13_. ACM Press.  https://doi.org/10.1145/2505515.2505665
  * Huang and Du (2019) Yuyun Huang and Jinhua Du. 2019.  Self-Attention Enhanced CNNs and Collaborative Curriculum Learning for Distantly Supervised Relation Extraction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 389–398.  https://doi.org/10.18653/v1/D19-1037
  * Humeau et al. (2020) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020.  Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring.  Online.  arXiv:1905.01969
  * Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.  Unsupervised Dense Information Retrieval with Contrastive Learning.  Online.  arXiv:2112.09118
  * Izacard and Grave (2022) Gautier Izacard and Edouard Grave. 2022.  Distilling Knowledge from Reader to Retriever for Question Answering.  Online.  arXiv:2012.04584
  * Jaleel et al. (2004) Nasreen Abdul Jaleel, James Allan, W. Bruce Croft, Fernando Diaz, Leah S. Larkey, Xiaoyan Li, Mark D. Smucker, and Courtney Wade. 2004.  UMass at TREC 2004: Novelty and HARD. In _Text Retrieval Conference_.  https://api.semanticscholar.org/CorpusID:16221853
  * Jia et al. (2019) Wei Jia, Dai Dai, Xinyan Xiao, and Hua Wu. 2019.  ARNOR: Attention Regularization based Noise Reduction for Distant Supervision Relation Classification. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 1399–1408.  https://doi.org/10.18653/v1/P19-1135
  * Joulin et al. (2018) Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Herve Jegou, and Edouard Grave. 2018.  Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion.  Online.  arXiv:1804.07745
  * Joyce and Needham (1958) T. Joyce and R. M. Needham. 1958.  The thesaurus approach to information retrieval.  _American Documentation_ 9, 3 (aug 1958), 192–197.  https://doi.org/10.1002/asi.5090090305
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 6769–6781.  https://doi.org/10.18653/v1/2020.emnlp-main.550
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.  Online.  arXiv:2004.12832
  * Kong et al. (2022) Weize Kong, Swaraj Khadanga, Cheng Li, Shaleen Kumar Gupta, Mingyang Zhang, Wensong Xu, and Michael Bendersky. 2022.  Multi-Aspect Dense Retrieval. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ (Washington DC, USA) _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 3178–3186.  https://doi.org/10.1145/3534678.3539137
  * Kumar et al. (2019) Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, and Preethi Jyothi. 2019.  Cross-Lingual Training for Automatic Question Generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 4863–4872.  https://doi.org/10.18653/v1/P19-1481
  * Kuzi et al. (2020) Saar Kuzi, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. 2020.  Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach.  Online.  arXiv:2010.01195
  * Lange et al. (2020) Lukas Lange, Heike Adel, and Jannik Strötgen. 2020.  On the Choice of Auxiliary Languages for Improved Sequence Tagging. In _Proceedings of the 5th Workshop on Representation Learning for NLP_. Association for Computational Linguistics, Online, 95–102.  https://doi.org/10.18653/v1/2020.repl4nlp-1.13
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 4483–4499.  https://doi.org/10.18653/v1/2020.emnlp-main.363
  * Le and Titov (2019) Phong Le and Ivan Titov. 2019.  Distant Learning for Entity Linking with Automatic Noise Detection. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 4081–4090.  https://doi.org/10.18653/v1/P19-1400
  * Lee et al. (2021) Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021.  Learning Dense Representations of Phrases at Scale.  Online.  arXiv:2012.12624
  * Lee et al. (2019a) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019a.  Latent Retrieval for Weakly Supervised Open Domain Question Answering.  Online.  arXiv:1906.00300
  * Lee et al. (2019b) Seanie Lee, Donggyu Kim, and Jangwon Park. 2019b.  Domain-agnostic Question-Answering with Adversarial Training.  Online.  arXiv:1910.09342
  * Lewis et al. (2021) Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021.  PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them.  Online.  arXiv:2102.07033
  * Li (2011) Hang Li. 2011.  _Learning to Rank for Information Retrieval and Natural Language Processing_.  Morgan & Claypool Publishers.  https://doi.org/10.2200/S00348ED1V01Y201104HLT012
  * Li and Gaussier (2022) Minghan Li and Eric Gaussier. 2022.  Domain Adaptation for Dense Retrieval through Self-Supervision by Pseudo-Relevance Labeling.  Online.  arXiv:2212.06552
  * Li et al. (2021) Minghan Li, Ming Li, Kun Xiong, and Jimmy Lin. 2021.  Multi-Task Dense Retrieval via Model Uncertainty Fusion for Open-Domain Question Answering. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 274–287.  https://doi.org/10.18653/v1/2021.findings-emnlp.26
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-Tuning: Optimizing Continuous Prompts for Generation.  Online.  arXiv:2101.00190
  * Liang et al. (2020) Davis Liang, Peng Xu, Siamak Shakeri, Cicero Nogueira dos Santos, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020.  Embedding-based Zero-shot Retrieval through Query Generation.  Online.  arXiv:2009.10270
  * Lin et al. (2023b) Sheng-Chieh Lin, Minghan Li, and Jimmy Lin. 2023b.  Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval.  Online.  arXiv:2208.00511
  * Lin and Lin (2021) Sheng-Chieh Lin and Jimmy Lin. 2021.  Densifying Sparse Representations for Passage Retrieval by Representational Slicing.  Online.  arXiv:2112.04666
  * Lin et al. (2020) Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2020.  Distilling Dense Representations for Ranking using Tightly-Coupled Teachers.  Online.  arXiv:2010.11386
  * Lin et al. (2023a) Zhenghao Lin, Yeyun Gong, Xiao Liu, Hang Zhang, Chen Lin, Anlei Dong, Jian Jiao, Jingwen Lu, Daxin Jiang, Rangan Majumder, and Nan Duan. 2023a.  PROD: Progressive Distillation for Dense Retrieval.  Online.  arXiv:2209.13335
  * Litschko et al. (2021) Robert Litschko, Ivan Vulić, Simone Paolo Ponzetto, and Goran Glavaš. 2021.  On Cross-Lingual Retrieval with Multilingual Text Encoders.  Online.  arXiv:2112.11031
  * Liu and Yang (2022) Alexander Liu and Samuel Yang. 2022.  Masked Autoencoders As The Unified Learners For Pre-Trained Sentence Representation.  Online.  arXiv:2208.00231
  * Liu et al. (2021a) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  Online.  arXiv:2101.00190
  * Liu (2011) Tie-Yan Liu. 2011.  _Learning to Rank for Information Retrieval_.  Springer Berlin Heidelberg.  https://doi.org/10.1007/978-3-642-14267-3
  * Liu et al. (2022) Weiming Liu, Xiaolin Zheng, Mengling Hu, and Chaochao Chen. 2022.  Collaborative Filtering with Attribution Alignment for Review-based Non-overlapped Cross Domain Recommendation.  Online.  arXiv:2202.04920
  * Liu et al. (2021b) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b.  GPT Understands, Too.  Online.  arXiv:2103.10385
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  Online.  arXiv:1907.11692
  * Long et al. (2015) Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. 2015.  Learning Transferable Features with Deep Adaptation Networks.  Online.  arXiv:1502.02791
  * Lu et al. (2020) Jing Lu, Gustavo Hernandez Abrego, Ji Ma, Jianmo Ni, and Yinfei Yang. 2020.  Neural Passage Retrieval with Improved Negative Contrast.  Online.  arXiv:2010.12523
  * Lu et al. (2021b) Jing Lu, Gustavo Hernandez Abrego, Ji Ma, Jianmo Ni, and Yinfei Yang. 2021b.  Multi-stage Training with Improved Negative Contrast for Neural Passage Retrieval. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 6091–6103.  https://doi.org/10.18653/v1/2021.emnlp-main.492
  * Lu et al. (2021a) Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021a.  Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2780–2791.  https://doi.org/10.18653/v1/2021.emnlp-main.220
  * Lu et al. (2022) Yuxiang Lu, Yiding Liu, Jiaxiang Liu, Yunsheng Shi, Zhengjie Huang, Shikun Feng Yu Sun, Hao Tian, Hua Wu, Shuaiqiang Wang, Dawei Yin, and Haifeng Wang. 2022.  ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval.  Online.  arXiv:2205.09153
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, Dense, and Attentional Representations for Text Retrieval.  _Transactions of the Association for Computational Linguistics_ 9 (2021), 329–345.  https://doi.org/10.1162/tacl_a_00369
  * Ma et al. (2021b) Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021b.  Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation.  Online.  arXiv:2004.14503
  * Ma et al. (2022) Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. 2022.  Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM.  https://doi.org/10.1145/3477495.3531772
  * Ma et al. (2021a) Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2021a.  PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_. ACM.  https://doi.org/10.1145/3437963.3441777
  * Ma et al. (2019) Xiaofei Ma, Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang. 2019.  Universal Text Representation from BERT: An Empirical Study.  Online.  https://doi.org/10.48550/ARXIV.1910.07973 arXiv:1910.07973
  * MacAvaney et al. (2017) Sean MacAvaney, Kai Hui, and Andrew Yates. 2017.  An Approach for Weakly-Supervised Deep Information Retrieval. In _SIGIR 2017 Workshop on Neural Information Retrieval_.  https://arxiv.org/abs/1707.00189v2
  * MacAvaney et al. (2019) Sean MacAvaney, Andrew Yates, Kai Hui, and Ophir Frieder. 2019.  Content-Based Weak Supervision for Ad-Hoc Re-Ranking. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM.  https://doi.org/10.1145/3331184.3331316
  * Maillard et al. (2021) Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen tau Yih, Barlas Oğuz, Veselin Stoyanov, and Gargi Ghosh. 2021.  Multi-task Retrieval for Knowledge-Intensive Tasks.  Online.  arXiv:2101.00117
  * Mao et al. (2022) Zhuoyuan Mao, Chenhui Chu, and Sadao Kurohashi. 2022.  EMS: Efficient and Effective Massively Multilingual Sentence Representation Learning.  Online.  arXiv:2205.15744
  * Mao and Nakagawa (2023) Zhuoyuan Mao and Tetsuji Nakagawa. 2023.  LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation.  Online.  arXiv:2302.08387
  * Mass and Roitman (2020) Yosi Mass and Haggai Roitman. 2020.  Ad-hoc Document Retrieval using Weak-Supervision with BERT and GPT2. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 4191–4197.  https://doi.org/10.18653/v1/2020.emnlp-main.343
  * Mikolov et al. (2013) Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013.  Exploiting Similarities among Languages for Machine Translation.  Online.  arXiv:1309.4168
  * Mitra and Craswell (2017) Bhaskar Mitra and Nick Craswell. 2017.  Neural Models for Information Retrieval.  Online.  arXiv:1705.01509
  * Mukherjee and Awadallah (2020) Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020.  Uncertainty-aware Self-training for Text Classification with Few Labels.  Online.  arXiv:2006.15315
  * Ni et al. (2021) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hern’andez ’Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021.  Large Dual Encoders Are Generalizable Retrievers. In _Conference on Empirical Methods in Natural Language Processing_.
  * Nooralahzadeh et al. (2019) Farhad Nooralahzadeh, Jan Tore Lønning, and Lilja Øvrelid. 2019.  Reinforcement-based denoising of distantly supervised NER with partial annotation. In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_. Association for Computational Linguistics, Hong Kong, China, 225–233.  https://doi.org/10.18653/v1/D19-6125
  * Onoe and Durrett (2019) Yasumasa Onoe and Greg Durrett. 2019.  Learning to Denoise Distantly-Labeled Data for Entity Typing.  Online.  arXiv:1905.01566
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation Learning with Contrastive Predictive Coding.  Online.  arXiv:2004.09930
  * Pandey (2013) Shiva Ram Pandey. 2013.  Automatic Question Generation Using Software Agents for Technical Institutions.  https://api.semanticscholar.org/CorpusID:6306809
  * Park et al. (2021) Cheonbok Park, Yunwon Tae, Taehee Kim, Soyoung Yang, Mohammad Azam Khan, Eunjeong Park, and Jaegul Choo. 2021.  Unsupervised Neural Machine Translation for Low-Resource Domains via Meta-Learning.  Online.  arXiv:2010.09046
  * Qian and Yu (2019) Kun Qian and Zhou Yu. 2019.  Domain Adaptive Dialog Generation via Meta Learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 2639–2649.  https://doi.org/10.18653/v1/P19-1253
  * Qiao et al. (2019) Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019.  Understanding the Behaviors of BERT in Ranking.  Online.  arXiv:1904.07531
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 5835–5847.  https://doi.org/10.18653/v1/2021.naacl-main.466
  * Rakangor and Ghodasara (2015) Sheetal Rakangor and Dr. Y. R. Ghodasara. 2015.  Literature Review of Automatic Question Generation Systems.  https://api.semanticscholar.org/CorpusID:45541311
  * Ram et al. (2021) Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021.  Few-Shot Question Answering by Pretraining Span Selection.  Online.  arXiv:2101.00438
  * Ram et al. (2022) Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. 2022.  Learning to Retrieve Passages without Supervision.  Online.  arXiv:2112.07708
  * Ramponi and Plank (2020) Alan Ramponi and Barbara Plank. 2020.  Neural Unsupervised Domain Adaptation in NLP—A Survey. In _Proceedings of the 28th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Barcelona, Spain (Online), 6838–6855.  https://doi.org/10.18653/v1/2020.coling-main.603
  * Rao et al. (2022) Jun Rao, Xv Meng, Liang Ding, Shuhan Qi, and Dacheng Tao. 2022.  Parameter-Efficient and Student-Friendly Knowledge Distillation.  Online.  arXiv:2205.15308
  * Reddy et al. (2021) Revanth Gangi Reddy, Vikas Yadav, Md Arafat Sultan, Martin Franz, Vittorio Castelli, Heng Ji, and Avirup Sil. 2021.  Towards Robust Neural Retrieval Models with Synthetic Pre-Training.  Online.  arXiv:2104.07800
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  Online.  arXiv:1908.10084
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 4512–4525.  https://doi.org/10.18653/v1/2020.emnlp-main.365
  * Ren et al. (2021) Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021.  PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics.  https://doi.org/10.18653/v1/2021.findings-acl.191
  * Ren et al. (2023a) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2023a.  RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking.  Online.  arXiv:2110.07367
  * Ren et al. (2023b) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qifei Wu, Yuchen Ding, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2023b.  A Thorough Examination on Zero-shot Dense Retrieval.  Online.  arXiv:2204.12755
  * Ren et al. (2020) Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. 2020.  Denoising Multi-Source Weak Supervision for Neural Text Classification. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 3739–3754.  https://doi.org/10.18653/v1/2020.findings-emnlp.334
  * Rezagholizadeh et al. (2022) Mehdi Rezagholizadeh, Aref Jafari, Puneeth S.M. Saladi, Pranav Sharma, Ali Saheb Pasand, and Ali Ghodsi. 2022.  Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher. In _Proceedings of the 29th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 4714–4727.  https://aclanthology.org/2022.coling-1.418
  * Robertson (2004) Stephen Robertson. 2004.  Understanding inverse document frequency: on theoretical arguments for IDF.  _Journal of Documentation_ 60, 5 (oct 2004), 503–520.  https://doi.org/10.1108/00220410410560582
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The Probabilistic Relevance Framework: BM25 and Beyond.  _Foundations and Trends® in Information Retrieval_ 3, 4 (2009), 333–389.  https://doi.org/10.1561/1500000019
  * Robertson et al. (1994) Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994.  Okapi at TREC-3. In _Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994_ _(NIST Special Publication, Vol. 500-225)_ , Donna K. Harman (Ed.). National Institute of Standards and Technology (NIST), 109–126.  http://trec.nist.gov/pubs/trec3/papers/city.ps.gz
  * Sachan et al. (2023) Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2023.  Improving Passage Retrieval with Zero-Shot Question Generation.  Online.  arXiv:2204.07496
  * Salton (1962) Gerard Salton. 1962.  Some experiments in the generation of word and document associations. In _Proceedings of the December 4-6, 1962, fall joint computer conference on - AFIPS ’62 (Fall)_. ACM Press.  https://doi.org/10.1145/1461518.1461544
  * Salton and Buckley (1988) Gerard Salton and Christopher Buckley. 1988.  Term-weighting approaches in automatic text retrieval.  _Information Processing & Management_ 24, 5 (jan 1988), 513–523.  https://doi.org/10.1016/0306-4573(88)90021-0
  * Salton et al. (1975) G. Salton, A. Wong, and C. S. Yang. 1975.  A vector space model for automatic indexing.  _Commun. ACM_ 18, 11 (nov 1975), 613–620.  https://doi.org/10.1145/361219.361220
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  Online.  arXiv:1910.01108
  * Shakeri et al. (2020) Siamak Shakeri, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020.  End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 5445–5460.  https://doi.org/10.18653/v1/2020.emnlp-main.439
  * Shen et al. (2022) Xiaoyu Shen, Svitlana Vakulenko, Marco del Tredici, Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. 2022.  Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey.  Online.  arXiv:2208.03197
  * Shrivastava and Wang (2022) Shubham Shrivastava and Kaiyue Wang. 2022.  QAGAN: Adversarial Approach To Learning Domain Invariant Language Features.  Online.  arXiv:2206.12388
  * Sun et al. (2021) Si Sun, Yingzhuo Qian, Zhenghao Liu, Chenyan Xiong, Kaitao Zhang, Jie Bao, Zhiyuan Liu, and Paul Bennett. 2021.  Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision.  Online.  arXiv:2012.14862
  * Tam et al. (2022) Weng Lam Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Xingjian Zhang, Yuxiao Dong, Jiahua Liu, Maodi Hu, and Jie Tang. 2022.  Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers.  Online.  arXiv:2207.07087
  * Thakur et al. (2021a) Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021a.  Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks.  Online.  arXiv:2010.08240
  * Thakur et al. (2021b) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021b.  BEIR: Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In _Proceedings of the 2021 Neural Information Processing Systems (NeurIPS-2021): Track on Datasets and Benchmarks_.  https://arxiv.org/abs/2104.08663
  * Tran et al. (2019) Brandon Tran, Maryam Karimzadehgan, Rama Kumar Pasumarthi, Mike Bendersky, and Don Metzler. 2019.  Domain Adaptation for Enterprise Email Search. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_.
  * Tzeng et al. (2014) Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. 2014.  Deep Domain Confusion: Maximizing for Domain Invariance.  Online.  arXiv:1412.3474
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  Online.  arXiv:1706.03762
  * Wang et al. (2019) Huazheng Wang, Zhe Gan, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, and Hongning Wang. 2019.  Adversarial Domain Adaptation for Machine Reading Comprehension.  Online.  arXiv:1908.09209
  * Wang et al. (2021a) Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021a.  TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 671–688.  https://doi.org/10.18653/v1/2021.findings-emnlp.59
  * Wang et al. (2022) Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022.  GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Seattle, United States, 2345–2360.  https://doi.org/10.18653/v1/2022.naacl-main.168
  * Wang et al. (2020) Liqiang Wang, Xiaoyu Shen, Gerard de Melo, and Gerhard Weikum. 2020.  Cross-Domain Learning for Classifying Propaganda in Online Contents.  Online.  arXiv:2011.06844
  * Wang et al. (2023) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2023.  SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval.  Online.  arXiv:2207.02578
  * Wang et al. (2021b) Shuai Wang, Shengyao Zhuang, and Guido Zuccon. 2021b.  BERT-Based Dense Retrievers Require Interpolation with BM25 for Effective Passage Retrieval. In _Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval_ (Virtual Event, Canada) _(ICTIR ’21)_. Association for Computing Machinery, New York, NY, USA, 317–324.  https://doi.org/10.1145/3471158.3472233
  * Xiao et al. (2022) Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022.  RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder.  Online.  arXiv:2205.12035
  * Xin et al. (2022) Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. 2022.  Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations. In _Findings of the Association for Computational Linguistics: ACL 2022_. Association for Computational Linguistics, Dublin, Ireland, 4008–4020.  https://doi.org/10.18653/v1/2022.findings-acl.316
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-End Neural Ad-hoc Ranking with Kernel Pooling. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM.  https://doi.org/10.1145/3077136.3080809
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval.  Online.  arXiv:2007.00808
  * Xu et al. (2022) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2022.  LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval.  Online.  arXiv:2203.06169
  * Xu et al. (2020) Guodong Xu, Ziwei Liu, and Chen Change Loy. 2020.  Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup.  Online.  arXiv:2012.09413
  * Yan et al. (2021) Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021.  ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer.  Online.  arXiv:2105.11741
  * Yang et al. (2019) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Multilingual Universal Sentence Encoder for Semantic Retrieval.  Online.  arXiv:1907.04307
  * Yang et al. (2020a) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020a.  Multilingual Universal Sentence Encoder for Semantic Retrieval. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Online, 87–94.  https://doi.org/10.18653/v1/2020.acl-demos.12
  * Yang et al. (2018) Yaosheng Yang, Wenliang Chen, Zhenghua Li, Zhengqiu He, and Min Zhang. 2018.  Distantly Supervised NER with Partial Annotation Learning and Reinforcement Learning. In _Proceedings of the 27th International Conference on Computational Linguistics_. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 2159–2169.  https://aclanthology.org/C18-1183
  * Yang et al. (2020b) Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020b.  Generative Data Augmentation for Commonsense Reasoning. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 1008–1025.  https://doi.org/10.18653/v1/2020.findings-emnlp.90
  * Yu et al. (2022) Changlong Yu, Hongming Zhang, Yangqiu Song, and Wilfred Ng. 2022.  CoCoLM: COmplex COmmonsense Enhanced Language Model with Discourse Relations.  Online.  arXiv:2012.15643
  * Yu et al. (2021) Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. 2021.  Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach.  Online.  arXiv:2010.07835
  * Yue et al. (2022a) Xiang Yue, Xiaoman Pan, Wenlin Yao, Dian Yu, Dong Yu, and Jianshu Chen. 2022a.  C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References.  Online.  arXiv:2203.08928
  * Yue et al. (2022b) Xiang Yue, Ziyu Yao, and Huan Sun. 2022b.  Synthetic Question Value Estimation for Domain Adaptation of Question Answering.  Online.  arXiv:2203.08926
  * Zeng et al. (2022) Hansi Zeng, Hamed Zamani, and Vishwa Vinay. 2022.  Curriculum Learning for Dense Retrieval Distillation.  Online.  arXiv:2204.13679
  * Zhai (2009) ChengXiang Zhai. 2009.  _Statistical Language Models for Information Retrieval_.  Springer International Publishing.  https://doi.org/10.1007/978-3-031-02130-5
  * Zhan et al. (2021) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021.  Optimizing Dense Retrieval Model Training with Hard Negatives.  Online.  arXiv:2104.08051
  * Zhan et al. (2022) Jingtao Zhan, Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2022.  Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models.  Online.  arXiv:2204.11447
  * Zhang et al. (2023) Kai Zhang, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, and Daxin Jiang. 2023.  LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval.  Online.  arXiv:2208.13661
  * Zhang et al. (2020b) Kaitao Zhang, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2020b.  Selective Weak Supervision for Neural Information Retrieval. In _Proceedings of The Web Conference 2020_. ACM.  https://doi.org/10.1145/3366423.3380131
  * Zhang and Bansal (2019) Shiyue Zhang and Mohit Bansal. 2019.  Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering.  Online.  arXiv:1909.06356
  * Zhang et al. (2022a) Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang, and Nan Duan. 2022a.  Multi-View Document Representation Learning for Open-Domain Dense Retrieval.  Online.  arXiv:2203.08372
  * Zhang et al. (2021) Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021.  Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval.  Online.  arXiv:2108.08787
  * Zhang et al. (2022b) Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. 2022b.  Towards Best Practices for Training Multilingual Dense Retrieval Models.  Online.  arXiv:2203.08926
  * Zhang et al. (2020a) Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020a.  An Unsupervised Sentence Embedding Method by Mutual Information Maximization. In _Conference on Empirical Methods in Natural Language Processing_.  https://api.semanticscholar.org/CorpusID:221948991
  * Zhao et al. (2022) Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2022.  Dense Text Retrieval based on Pretrained Language Models: A Survey.  Online.  arXiv:2211.14876
  * Zhou et al. (2021) Chunting Zhou, Graham Neubig, and Jiatao Gu. 2021.  Understanding Knowledge Distillation in Non-autoregressive Machine Translation.  Online.  arXiv:1911.02727
  * Zhou et al. (2022) Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Majumder, Ji-Rong Wen, Nan Duan, and Weizhu Chen. 2022.  SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval.  Online.  arXiv:2210.11773
  * Zhu et al. (2023) Dawei Zhu, Xiaoyu Shen, Michael A. Hedderich, and Dietrich Klakow. 2023.  Meta Self-Refinement for Robust Learning with Weak Supervision.  Online.  arXiv:2205.07290
  * Zhu et al. (2021) Yutao Zhu, Jian-Yun Nie, Zhicheng Dou, Zhengyi Ma, Xinyu Zhang, Pan Du, Xiaochen Zuo, and Hao Jiang. 2021.  Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. ACM.  https://doi.org/10.1145/3459637.3482243
  * Zobel and Moffat (2006) Justin Zobel and Alistair Moffat. 2006.  Inverted files for text search engines.  _Comput. Surveys_ 38, 2 (jul 2006), 6.  https://doi.org/10.1145/1132956.1132959
  * Zobel et al. (1998) Justin Zobel, Alistair Moffat, and Kotagiri Ramamohanarao. 1998.  Inverted files versus signature files for text indexing.  _ACM Transactions on Database Systems_ 23, 4 (dec 1998), 453–490.  https://doi.org/10.1145/296854.277632

##  Appendix A History of textual information retrieval

The textual information retrieval task was theorised as early as 1950 by T.
Joyce and R. M. Needham (Joyce and Needham, 1958). They asked how texts can be
indexed and picked representative terms to retrieve relevant information.
Following that idea, the ”bag-of-words” assumption led to a vector space model
that encoded the query and the document in sparse term-based vectors (Salton,
1962; Salton et al., 1975). Many different approaches to constructing these
sparse vectors exist. A first approach, called tf-idf, allows for a relevance
measure based on the text vector’s and query vector’s lexical similarity
(Salton and Buckley, 1988; Aizawa, 2003; Robertson, 2004). As a second
approach, inverted indexing utilises the same representation idea as tf-idf
but makes text retrieval more efficient by organising documents in term-
oriented posting lists (Zobel and Moffat, 2006; Zobel et al., 1998).
Probabilistic relevance frameworks are a third approach to solving the text
retrieval task and modelling the relevance of documents. They enable a more
in-depth understanding of the retrieval mechanisms. One example of such a
framework is BM25 (Robertson et al., 1994; Robertson and Zaragoza, 2009), but
multiple more (statistical) language modelling approaches exist (Zhai, 2009).
The rise of machine learning approaches led to the use of learning to rank
algorithms for text retrieval tasks (Liu, 2011; Li, 2011). These algorithms
apply supervised machine learning models to rank the relevance of retrieved
documents but require human feature engineering.
The availability of more computational power and the application of stochastic
gradient descent for neural networks resulted in neural information retrieval,
a deep learning approach to text retrieval tasks. This approach no longer
required hand-designed feature engineering (Huang et al., 2013; Guo et al.,
2017; Mitra and Craswell, 2017; Guo et al., 2019). Neural information
retrieval models map the query and the text corpus into low-dimensional
vectors in latent representation space. Thus, they encode the textual input in
dense vectors. This process is also called embedding. The similarity between
the query and the document vector estimates the relevance of a text for a
query. Thus, the relevance of information is no longer estimated by lexical
but by semantic similarity. Contrary to this approach, the sparse vectors
constructed by classical vector space models encode explicit term dimensions.
In contrast, the dense vectors constructed by neural IR models encode the
latent semantic characteristics of language. The so-called transformer
architecture was the subsequent historical development in the textual
information retrieval task after neural IR. Nowadays, transformers are the
basis for most NLP tasks. They were initially proposed to model any sequence
data by utilising the self-attention mechanism through which every token
attends to every other token in the sequence (Vaswani et al., 2017; Hochreiter
and Schmidhuber, 1997; Cho et al., 2014). Their two significant advantages are
that they can be trained in parallel and scaled up easily. The use of
transformers for NLP tasks, as well as the availability of large-scale
labelled retrieval datasets like MS MARCO(Bajaj et al., 2016) gave rise to
pre-trained language models (Devlin et al., 2018; Liu et al., 2019; Brown et
al., 2020; Fan et al., 2022). The pre-trained language models use different
self-supervised loss functions and are trained on large-scale general document
corpora. Fine-tuning the models enables their transfer to downstream tasks
(Liu et al., 2021a; Zhao et al., 2022). They facilitated a broader
representation of semantics and language in general. The most common pre-
trained language model is called BERT (Devlin et al., 2018). It employs a deep
bi-directional architecture and word masking to improve text encoding into
dense vectors. BERT can encode the general English language. The recent
developments after BERT can be categorised into three different research
fields. Firstly, different pre-training approaches are explored (Liu et al.,
2019). Secondly, the bi-directional representation is refined (Ma et al.,
2019), and thirdly, a compressed and thus more lightweight version of BERT was
developed, called distilBERT (Sanh et al., 2019). Nowadays, pre-trained dense
retrievers are the ”gold standard” for solving tIR tasks due to their
astonishing representation of documents and ability to answer more complex
queries (Zhao et al., 2022). However, most pre-trained dense retrievers were
trained on English data containing general, informal knowledge. Transforming
them to other languages or specialised domains requires further fine-tuning on
labelled data in the desired domain and language. Such data, as mentioned in
Section 1, is rarely available.