# Explainable Information Retrieval: A Survey

Avishek Anand<EMAIL_ADDRESS>0000-0002-0163-0739 ,  Lijun Lyu
<EMAIL_ADDRESS>0000-0002-7268-4902 Delft University of TechnologyP.O. Box
1212DelftThe Netherlands ,  Maximilian Idahl<EMAIL_ADDRESS>0000-0003-1866-6436
,  Yumeng Wang<EMAIL_ADDRESS>0000-0002-2105-8477 ,  Jonas Wallat<EMAIL_ADDRESS>0000-0003-1239-2067 and  Zijian Zhang<EMAIL_ADDRESS>0000-0001-9000-4678 L3S
Research Center, Leibniz University HannoverAppelstr. 9aHannoverLower
SaxonyGermany

###### Abstract.

Explainable information retrieval is an emerging research area aiming to make
transparent and trustworthy information retrieval systems. Given the
increasing use of complex machine learning models in search systems,
explainability is essential in building and auditing responsible information
retrieval models. This survey fills a vital gap in the otherwise topically
diverse literature of explainable information retrieval. It categorizes and
discusses recent explainability methods developed for different application
domains in information retrieval, providing a common framework and unifying
perspectives. In addition, it reflects on the common concern of evaluating
explanations and highlights open challenges and opportunities.

††copyright: rightsretained

##  1\. Introduction

Information retrieval (IR) systems are one of the most user-centric systems on
the Web, in digital libraries, and enterprises. Search engines can be general-
purpose (e.g., Web search) to specialized expert systems that are geared
towards expert consumption or support, including legal and patent retrieval IR
(Chakraborty et al., 2021), historical search (Holzmann and Anand, 2016;
Holzmann et al., 2017a), and scholarly search (Hanauer, 2006; Singh et al.,
2016b). On the one hand, riding on the recent advances of complex machine
learning (ML) models trained on large amounts of data, IR has seen impressive
performance gains over classical models (Lin, 2019). On the other hand,
complex models also tend to be opaque and less transparent than their
classical and arguably simpler counterparts. Therefore, towards an important
goal of ensuring a reliable and trustworthy IR systems, recent years have seen
increased interest in the area of explainable information retrieval (ExIR).

###  1.1. Motivation

Firstly, in IR, there has been sufficient evidence of how user interaction
data from search engines can be a source of biases, especially associated with
gender and ethnicity (Mowshowitz and Kawaguchi, 2005; Rekabsaz and Schedl,
2020; Bigdeli et al., 2022). When undetected and unidentified, the users of an
IR system too are exposed to stereotypical biases that reinforce known yet
unfair prejudices. Secondly, model retrieval models based on transformer-style
over-parameterized models can be brittle and sensitive to small adversarial
errors (Wang et al., 2022). Recently developed inductive biases, pre-training
procedures, and transfer learning practices might lead these statistical over-
parameterized models to learn shortcuts (Geirhos et al., 2020). Consequently,
shortcuts that do not align with human understanding results in learning
patterns that are right for the wrong reasons. Finally, expert users using
specialized search systems – in legal search, medicine, journalism, and patent
search – need control, agency, and lineage of the search results. For all the
above IR-centric reasons, among many other general reasons – like utility for
legal compliance, scientific investigation, and model debugging – the field of
ExIR provides the tools/primitives to examine learning models and the
capability to build transparent IR systems.

###  1.2. The Landscape of Explainable Information Retrieval

Although interpretability in IR is a fairly recent phenomenon, there has been
a large amount of growing yet unorganized work that covers many tasks and
aspects of data-driven models in IR. This survey aims to collect, organize and
synthesize the progress in ExIR in the last few years. ExIR has quite a
diverse landscape owing to the continued and sustained interest in the last
few years. The initial approaches in ExIR were adaptations of widely popular
feature-attribution approaches (e.g., LIME (Ribeiro et al., 2016) and SHAP’s
(Lundberg et al., 2018)). However, in the following years, there has been a
multitude of approaches that tackle specific problems in IR. We cover a wide
range of approaches, from post-hoc approaches (cf. Sections 3, LABEL:, 5,
LABEL: and 4), grounding to axiomatic approaches (cf. Section 6), to
interpretable-by-design methods (cf. Section 8 and Section 9).

###  1.3. Methodology and Scope

Before we started our literature review, we needed to collect a corpus of
relevant papers for ExIR and delineate the boundaries of the review.

####  1.3.1. Corpus Creation.

We started with very first works in ExIR (e.g., (Singh and Anand, 2020, 2019;
Cummins and O’Riordan, 2007)), to build up an initial pool of papers. We did
then forward search from this initial set of papers that mention terms
“(explain* OR interpretab* OR explanation* OR transparen*)” AND “(retriev* OR
rank*”. Secondly, we limited our search to articles published in the past five
years (2018 – 2022) to provide a representative window into current best
practices that have emerged since the inception of the earliest works in ExIR
in the following IR venues – ACM Special Interest Group on Information
Retrieval (SIGIR), International Conference on the Theory of Information
Retrieval (ICTIR), International Conference on Web Search and Data Mining
(WSDM), Conference on Information and Knowledge Management (CIKM), the ACM Web
Conference (TheWebConf). In total, after filtering, we ended up with $68$
papers that we consider in this review that are partially relevant. A subset
of $32$ papers of those partially relevant papers find more detailed treatment
in this survey.

####  1.3.2. Scope.

We note that many of the methods in ExIR have methodological overlap with
those invented in ML, natural language processing (NLP), and recommender
systems (RS) communities. In fact, most of the approaches in ExIR are based on
seminal papers in these communities. We only focus on core-IR issues in this
survey and, wherever possible, clearly spell out the distinctions from similar
approaches in NLP, RS and ML in general. Rationale-based models have been
heavily investigated in NLP. We cover only the methods popularized in IR-
centric or venues. Our survey focuses on rationale-based models, i.e.,
document-ranking tasks, in learning-to-rank (LTR), and tasks that rely on a
retrieval component. Also, RS have a lot of work and even surveys in
explainability (Zhang and Chen, 2020). We only survey those approaches that
are useful for query modeling in query-based systems. The papers on the topics
of personalization search or explainable RS, although they can be considered
as user modeling applications of ExIR, were not selected due to either lack of
specific interpretability methods or being more suitable to be classified into
a relatively independent field of study. We also exclude IR approaches dealing
with image or multi-modal data.

##  2\. Notions and Categorization

qtree/.style=for tree=parent anchor=south, child
anchor=north,align=center,inner sep=3pt, fill=nodefill, rounded corners=2pt ,
baseline, qtree [Explainable IR [Post-hoc [Feature
Attribution §3] [Free-text
Explanations §4] [Adversarial
Examples §5] ] [Grounding to IR Properties [Axiomatic
Analysis §6] [Probing §7
] ] [Interpretable by-design [Explainable
Architectures §8] [Rationale-based
Methods §9] ] ]

Figure 1. Categorization of explainable IR approaches, where § indicates the
section the approach is discussed.

We start the survey by first introducing the notions and terminologies that
are commonly used in ExIR. Note that most of the terminologies in ExIR are
adapted from the general area of interpretable machine learning (Molnar,
2022), explainable vision (Samek et al., 2019), natural language processing
(Søgaard, 2021), and recommendation systems (Zhang and Chen, 2020). We
harmonize the differences in the categorizations used in these areas to
distill a specific method-centric classification of all approaches used in
ExIR in Figure 1. Our classification permeates the binary divides of post-hoc
and interpretable-by-design approaches by covering IR-specific dimensions of
axiomatic characterization and free-text explanations.

###  2.1. Notions in Explainable Information Retrieval

Explanations are the outputs of an interpretable machine learning procedure or
an interpretability method. In general machine learning, explanations vary in
scope and type. The scope of an explanation can be a single instance or the
entire dataset. The type of explanation refers to the style or form of the
explanation. Notions in ExIR share commonalities for the most part with
general XAI. However, there are some variations due to different tasks,
inputs, and output types in IR. In the following, we describe these IR-
specific notions pertaining to explainability.

####  2.1.1. Local vs global interpretability

Local interpretability refers to per-instance interpretability. For the task
of document ranking, an individual query is usually considered as a single
instance even though multiple decisions might be involved (e.g., multiple
query-document pairs and multiple preference pairs). Specifically, local
interpretability aims to explain the model decisions in the locality of a
specific query. On the other hand, global interpretability refers to the case
when there is no distinction across instances/queries in terms of model
parameters, input spaces, etc.

####  2.1.2. Pointwise, Pairwise, Listwise.

Ranking models output a ranked candidate list for a given query. Therefore,
the explanation of pointwise methods can only explain the models’ decision of
a single element in the list; while pairwise methods intend to explain the
model’s preference of a candidate pair. The explanation of listwise methods,
however, aims to cover all individual decisions in the entire ranking list.

####  2.1.3. Type of Explanations

A model decision can be explained differently in terms of input features,
training data, model parameters, or human-understandable decision structures.
When an explanation method measures the contribution of each feature in the
input instance leading to a specific decision, the generated explanation can
be a feature attribution. On the one hand, feature attributions can be soft
masks, i.e., real numbers denoting feature importance. On the other hand, they
can also be presented as boolean or hard masks where a feature is either
present or absent in the explanation. An explanation is understandable to
humans or users based not only if the feature space is understandable but also
if the explanation is small. An attribution over a feature space of hundreds
of dimensions is hard to interpret, even if it is over words and phrases that
are themselves understandable. In IR, we typically deal with long text
documents, and using feature attributions and sparsity is a key design
criterion. Explanation procedures can enforce sparsity constraints to have
short extractive attributions or generate a small set of words or terms called
free-text explanation. Unlike feature-based explanations, explanations can be
in terms of input instances. Contrastive explanations are such types of
explanations where the objective is to generate example instances with minor
differences from the input example but with contrasting predictions. The value
of contrastive examples as explanations is grounded in social sciences
(Miller, 2019). Therefore, using contrastive explanations to understand model
behavior is one crucial aspect of gaining more transparency into the model’s
decision-making process. Finally, rules are also one of the prevalent
explanations. We denote the explicit decision-making rules as hard-rule, such
as a decision-tree path and the well-established IR principles (axioms). On
the other hand, a soft-rule refers to those that partially impact the model
decision.

###  2.2. Post-hoc Interpretability

Post-hoc interpretability methods explain the decisions of already trained
machine learning models. Post-hoc approaches are either _model-agnostic_
(black-box) where the interpretability approach has no access to the trained
model parameters (Ribeiro et al., 2016; Lundberg and Lee, 2017), or _model
introspective_ (white-box) which have full access to the parameters of the
underlying model (Sundararajan et al., 2017; Shrikumar et al., 2017) or data
(Koh and Liang, 2017). In this survey, we will review approaches for both
white-box and black-box settings. Moreover, specifically in IR, we make a
distinction between a strongly- and weakly-agnostic setting depending on if we
are provided only access to a ranking of documents or also the score of a
document given a query. Most of the work in the existing literature only
considers our definition of a weakly agnostic model.

####  2.2.1. Methods of post-hoc interpretability.

A dominant class of post-hoc explanation approaches output what is known as
feature attributions or saliency maps. Most of the white-box approaches adapt
gradient-based attribution approaches with task-specific calibrations. For
black-box approaches, explanation methods use words/sentences/passages in text
retrieval and ranking, and numeric and categorical features in LTR for
modeling the feature space. We discuss methods in detail about feature
attribution in Section 3, free-text explanations in Section 4, and adversarial
examples in Section 5.

###  2.3. Interpretability by Design

A common problem with post-hoc approaches is that it is often unclear how much
the model behavior is indeed understood. In fact, Rudin (2019) advocates using
Interpretable-by-design (IBD) models as much as possible specifically for
high-stakes decision-making. However, building an IBD model that is indeed
fully transparent and meanwhile maintaining competitive performance is
challenging, especially for complex non-linear and over-parameterized neural
models. We note that most proposals in literature are partially interpretable,
instead of exhibiting full transparency.

####  2.3.1. Explainable by Architecture vs Rationales

Many approaches brand themselves as IBD methods, when in fact they are
partially interpretable. On one hand, some methods have only interpretable
feature interactions and score compositions (Khattab and Zaharia, 2020; Formal
et al., 2021). On the other hand, methods choose extractive input sequences as
explanations while the models themselves are non-interpretable (Zhang et al.,
2021a; Leonhardt et al., 2021). In this survey, we firstly subdivide the
family of IBD approaches by explainable by architecture (cf. Section 8) where
components of the model architecture are partially or fully interpretable.
Secondly, IBD methods that enforce input feature sparsity are detailed in
Section 9 as rationale-based methods.

Figure 2. Example ranking result showing top-5 ranked documents with predicted
relevance scores for the query “can you do yoga from a chair”. Query and
Documents are selected from TREC-DL (2021) (Craswell et al., 2021) and MS
MARCO (Nguyen et al., 2016), respectively.

###  2.4. Grounding to Information Retrieval Principles

There is a long-standing history of building text ranking models in IR. Most
of the well-known and robust approaches for understanding relevance are based
on establishing closed-formed relevance equations based on probabilistic
(Ponte and Croft, 1998) or axiomatic foundations (Bruza and Huibers, 1994). A
possible improve way to improve the transparency of data-driven complex ML
models is to determine if the learned models adhere to well-understood IR
principles. Towards this, there are two streams of research efforts that
attempt to ground the predictions of learned ranking models into axioms or
probing models for known relevance factors of matching, term proximity, and
semantic similarity. We review these approaches in Sections 6 and 7. Note that
the methods utilizing IR principles can be implemented in both post-hoc and
IBD manner.

###  2.5. Evaluation of Explanations

Evaluation of interpretability or explainability approaches has long been an
arduous and challenging task. There is no agreed-upon set of experimental
protocols leading to various design decisions due to a lack of ground truths
and differences in the perceived utility, stakeholders, and forms. Doshi-Velez
and Kim (2017) classify evaluation approaches as application-grounded, human-
grounded, and functionality-grounded. The difference between application- and
human-grounded evaluations is using experts and non-experts as evaluation
subjects. Functionality-grounded evaluation does not involve humans and relies
on a closed-form definition of interpretability that serves as a proxy to
evaluate the explanation quality. We introduce the following three classes of
evaluation strategies employed in ExIR.

####  2.5.1. Human evaluation

Most current papers in ExIR involve human evaluation, but primarily do not
differentiate between expert- and non-expert users. Evaluations can be simply
anecdotal. In this case, example explanations are shown to users, and
typically binary judgments regarding the goodness of the explanations are
elicited. A surprising number of ExIR papers claim interpretability of their
approaches but conduct simple anecdotal experiments. A more fine-grained human
evaluation is to ask users to solve specific tasks with the assistance of
explanations. Such an approach evaluates the utility of the explanations or
answers the question – how helpful are the explanations in the context of a
given application?

####  2.5.2. Fidelity-based Evaluation

Fidelity measures to which degree the explanations can replicate the
underlying model decisions. Fidelity is measured by generating a second
prediction and computing the agreement between the actual and the generated
prediction. The second prediction could be derived from either 1) using a part
of the input, 2) using a surrogate model, or 3) generating a counterfactual or
adversarial example. A more fine-grained category of fidelity can include
evaluating the comprehensiveness, sufficiency, etc. We will further discuss
the detailed metrics when we come to specific methods.

####  2.5.3. Reference-based Evaluation

The lack of ground truths for explanations is a central problem in explainable
AI. Whenever the ground-truth explanations are available, we can use them as
the reference to compare with the generated explanations. In case of a lack of
ground truth explanations, some methods choose a well-understood and fully
explainable/transparent model as a reference model. In such cases, we can
evaluate the truthfulness of the explanation methods by comparing the
explanations generated by the reference model and the explanation method.

##  3\. Feature Attribution

Feature attribution methods, also known as feature importance or saliency
methods, typically generate explanations for individual predictions by
attributing the model output to the input features. A scalar representing the
importance is assigned to each input feature or groups of input features.
These scores are then commonly visualized using a heatmap or a bar chart,
informing the user about which features the model’s prediction is most
sensitive to. Figure 3 demonstrates example feature attributions for the top-2
ranked documents, following the example from Figure 2.

Figure 3. A fictive example using a heatmap to visualize feature attributions
for the top-2 ranked documents for the query “can you do yoga from a chair”.
Feature importance is highlighted in orange.

Feature attribution methods have been found to be the most popular explanation
technique and are used in many domains (Bhatt et al., 2020). However, as is
common for interpretation techniques, most feature attribution methods were
originally designed to explain the predictions of classification models.

Recent work explores how such methods can be applied or adapted to explain the
output of ranking models, where feature importance scores can be computed for
the query or document input individually, or for both, as shown in Figure 3.
Following our categorization (Section 2), we differentiate between model-
agnostic and model-introspective feature attribution methods.

###  3.1. Model-agnostic Feature Attribution

A variety of feature attribution methods generate explanations in a model-
agnostic way by perturbing input features and observing the change in the
model output. The underlying model is treated as a black box.

####  3.1.1. Feature Ablation

Feature ablation is a simple perturbation-based approach to computing
importance scores. Individual (or groups of) input features are removed one at
a time, and an importance score is assigned based on the observed difference
between the model predictions.

To interpret a BERT-based ranking model, Qiao et al. (2019) compute the
importance of tokens through feature ablation. To produce feature importance
scores, they compare the ranking score of an unmodified document with the
ranking score for the same document when removing a randomly chosen input
token. Specifically, they only remove input tokens corresponding to regular
words and keep all tokens that are special tokens or correspond to stopwords.
They find that the ranking score produced by a BERT model depends on only a
few tokens in each document. The ranking score often decreases significantly
when these tokens are removed. When manually examining the important tokens,
the authors find that they often correspond to exact match terms, i.e., terms
that also appear in the input query, and terms in close semantic context. In
contrast, when examining token importance scores for a neural ranker based on
convolutions and interactions (Dai et al., 2018) that soft-matches n-grams for
ad-hoc search, the most important terms appear to be rather loosely related to
the input query.

####  3.1.2. Surrogate Models

Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et al., 2016)
is an interpretability method that generates explanations by training a
surrogate model on a dataset of perturbed samples to locally approximate the
behavior of the underlying black-box model. Typically, a linear model,
preferably sparse, is chosen as the interpretable surrogate model since the
weights directly specify the importance of each feature. Using LIME to
generate feature attributions, Singh and Anand (2019) propose EXS, an
explainable search system that provides explanations to users through feature
attribution. Specifically, EXS aims to provide information on three questions:
1) Why is a document relevant to the query, 2) Why is a document ranked higher
than another document, and 3) What the intent of the query is according to the
ranker? LIME is designed to explain the output of a classifier, and EXS casts
the output of a pointwise ranker into a classification problem by transforming
query-document scores into class probabilities. A binary classification
problem is created by considering the top-$k$ documents in an input ranking as
relevant and the rest as irrelevant, essentially considering document ranking
as a classification problem where the black box ranker is considered as a
classifier. Polley et al. (2021) compare EXS with their evidence-based
explainable document search system, ExDocS, which performs reranking using
interpretable features. In a user study, they found that EXS is on par with
the ExDocS system in completeness and transparency metrics, although users
rated ExDocS as more interpretable compared to EXS. At the same time, the use
of ExDocS resulted in a drop in ranking performance, whereas the use of EXS
does not affect performance at all.

Similarly, Verma and Ganguly (2019) adapt LIME to create locally interpretable
ranking model explanations (LIRME). In contrast to EXS, LIRME trains the local
surrogate model directly on the query-document scores and does not transform
them into class probabilities. Instead, they experiment with different
strategies to sample documents in the neighborhood of the document to be
explained. In their experiments, they create explanations for the output of a
Jelinek-Mercer smoothed language model on the TREC-8 dataset and find that
uniform or TF-IDF-biased term replacement strategies produce better
explanations than replacement strategies that use term position information.

Instead of training a local surrogate model to generate explanations for
individual examples, Singh and Anand (2018) distill an already trained black-
box LTR model into an interpretable global surrogate model that is used to
generate explanations. This global surrogate model only operates on the
interpretable subset of features and is trained to mimic the predictions of
the black-box ranker. For training, they create numerous artificial training
examples In their experiments, they validate whether it is possible to train
an interpretable model that approximates a complex model. On the LTR datasets
(Qin and Liu, 2013) they find that a faithful interpretable ranker can only be
learned for certain query localities. This showcases the limitation that
simple models, even when trained with a much larger quantity of training data,
are not able to faithfully explain all localities of the decision boundary of
a complex model and that using local surrogate models can be advantageous.

####  3.1.3. Searching for Explanations

An alternative to the above approaches is to search the space of all possible
explanations, optimizing for a metric of choice. For LTR models, Singh et al.
(2021) propose a simple, yet effective greedy search-based approach to find
explanations. Their approach aims to find a subset of explanatory features
that maximizes two measures, validity and completeness. The validity of an
explanation is defined as the amount of predictive capacity contained in a
subset of explanatory features. The idea is that the explanatory features
should be sufficient to produce the original output ranking. In fact, this
measure aligns with the sufficiency metric introduced by DeYoung et al.
(2020). The completeness metric measures whether removing explanatory features
from the input significantly changes the output. When all explanatory features
are removed, it should not be possible to produce the original output ranking.
Kendall’s tau rank correlation measures differences in output rankings; the
underlying model is treated as a black-box.

###  3.2. Model-introspective Feature Attribution

In contrast to model-agnostic methods, model-introspective feature attribution
methods require white-box access to the model being explained. Model-
introspective methods typically rely on gradients or other properties of the
model to compute feature importance scores.

####  3.2.1. Gradient-based Methods

Many feature attribution methods generate an explanation by computing the
gradient with respect to the input features. This gradient reflects how a
small change in the input features affects the prediction. The vanilla
gradient method can produce noisy explanations and suffers from a saturation
problem. A variety of methods aim to remedy these issues. For example,
Integrated Gradients (Sundararajan et al., 2017) accumulates gradients on a
path between a baseline input and the actual input. While this resolves the
saturation problem, the baseline input is a hyperparameter to be chosen
carefully. It is unclear what baseline is best, and each baseline makes
assumptions about the distribution of the data and the concept of missingness
in the feature space (Sturmfels et al., 2020). Other gradient-based feature
attribution methods, such as Layer-wise Relevance Propagation (Bach et al.,
2015), Guided Backpropagation (Springenberg et al., 2014), or DeepLIFT
(Shrikumar et al., 2017) back-propagate custom relevance scores using
modified, sometimes layer-specific, rules.

Fernando et al. (2019) apply DeepSHAP (Lundberg and Lee, 2017), a combination
of SHAP (Lundberg and Lee, 2017) and DeepLIFT (Shrikumar et al., 2017), to
neural retrieval models. Specifically, they investigate the sensitivity of the
explanations to different choices for constructing a baseline input document.
Generating explanations for a subset of queries from the TREC Robust04 test
collection and the corresponding top-3 ranked documents, they find that the
explanations are indeed sensitive to the baseline input. The DeepSHAP
explanations are also compared to explanations produced by LIME, and while for
some baseline inputs there is high overlap in the most important features,
there is a lack of overlap for others.

Purpura et al. (2021) use simple gradient-based feature attribution to find
the most important features used by LTR models. They generate a saliency map
for each instance in a training dataset and select feature groups by
thresholding the normalized importance values. Feature selection is then
performed by counting how often each feature group is considered important
across all extracted saliency maps.

Zhan et al. (2020) use Integrated Gradients (Sundararajan et al., 2017) to
obtain feature attributions for a BERT-based ranking model. As a baseline
input, they create an empty query and an empty document input by replacing the
corresponding tokens with the special padding token “[PAD]”. An example of
feature attributions for BERT-style input is visualized in Figure 4.

Figure 4. Example visualization of feature attributions for a single query-
document pair using the BERT-style input format, which is “[CLS] query [SEP]
document [SEP]”. Important tokens are highlighted in orange.

####  3.2.2. Attention-based Methods

Instead of using gradients, attention-based feature attribution methods use
the attention weights contained in attention layers, which are a core building
block of transformer models. The attention weights can be used to explain what
part of the input a model attends to when making a prediction, for example, by
visualizing the attention weights at certain layers (Vig, 2019). However,
whether attention weights actually provide explanations is subject to an
ongoing debate (Bibal et al., 2022; Bastings and Filippova, 2020).

Qiao et al. (2019) analyze the learned attentions of BERT-based ranking
models, using attention weights to measure the importance of features. They
group input tokens into three categories, as visualized in Figure 5: Regular
Words, Stopwords, and Markers, which are the special tokens “[CLS]” and
“[SEP]”. In their experiments on the MS MARCO passage reranking dataset
(Nguyen et al., 2016), they find that marker tokens receive the highest
attention. The importance of the marker tokens is confirmed by observing a
strong decrease in model performance when they are removed from the inputs.
Stopwords appear to be as important as regular words; however, removing them
does not appear to affect the ranking performance. Additionally, they observe
that the attention scores spread more uniformly across the input sequence in
deeper layers of BERT, as the embeddings become more contextualized.

Figure 5. Example bar chart visualization of feature attributions for
different groups of tokens.

In addition to Integrated Gradients, Zhan et al. (2020) also use attention
weights to obtain feature attributions for a BERT-based ranking model. With an
experimental setup similar to Qiao et al. (2019), they compute attribution
scores for different groups of input tokens: The special “[CLS]” and “[SEP]”
tokens, the query tokens, the document tokens, and the period token. While
confirming that a significant amount of attention weight is distributed to the
special tokens and the period token, the authors also find that the
attributions produced using attention weights are negatively correlated with
the attributions produced by Integrated Gradients. Based on their results, the
authors speculate that these tokens receive high attention weights due to
their high document frequency. They argue that the model dumps redundant
attention on these tokens, while these actually carry little relevance
information.

###  3.3. Evaluating Feature Attributions

Input feature attributions can be evaluated in many ways. However, there is
little agreement on which evaluation strategy is best. Sanity-checks (Adebayo
et al., 2018; Tomsett et al., 2020) test functionally grounded assumptions
behind feature attributions. Whether feature attributions are faithful to the
model that is explained can also be evaluated by removing important features
and re-evaluating model performance, either with or without retraining (Hooker
et al., 2019; Madsen et al., 2021; Rong et al., 2022). However, if the model
is not retrained, removing or replacing features can result in out-of-
distribution inputs. Other works propose shortcut, artifact, or spurious
correlation detection tasks to evaluate feature attributions (Yang and Kim,
2019; Bastings et al., 2021; Idahl et al., 2021; Adebayo et al., 2022), where
bugs are added to a model on purpose and then used as ground-truth for
explanation evaluation. Feature attribution methods that rely on surrogate
models need to evaluate their fidelity, that is, how well the surrogate model
approximates the black box model being explained. Unfortunately, the
evaluation of feature attributions in IR is often limited to anecdotal
examples. Singh and Anand (2019) neither evaluate the explanation quality of
EXS nor the fidelity of the local surrogate models used to generate
explanations. Verma and Ganguly (2019) evaluate LIRME by comparing the
explanations to a reference of important terms obtained from relevance
judgments but also do not explicitly evaluate faithfulness. Fernando et al.
(2019) include an analysis of the faithfulness of LIME explanations for neural
ranking models by measuring accuracy and mean-squared error of the local
surrogate model. To evaluate explanations produced by DeepSHAP, they use LIME
explanations as a reference. Directly optimizing explanations based on
evaluation metrics, as done by Singh et al. (2021), seems advantageous, but
does not provide any guarantees of finding a good explanation. Based on the
limited work on evaluating feature attributions in IR, we argue that claims
and hypotheses based on insights from feature attribution explanations should
be handled with caution unless the explanation methodology has been evaluated
rigorously.

Approach | Task | Explanation | Evaluation
---|---|---|---
EXS (Singh and Anand, 2019) | Text Ranking | Feature Attribution | Anecdotal
LIRME (Verma and Ganguly, 2019) | Text Ranking | Feature Attribution | Anecdotal/Reference
DeepSHAP (Fernando et al., 2019) | Text Ranking | Feature Attribution | Reference
Attention (Qiao et al., 2019; Zhan et al., 2020) | Text Ranking | Feature Attribution | Visualization
Global Surrogate Model (Singh and Anand, 2018) | LTR | Global Feature Attribution | Faithfulness
Greedy Search (Singh et al., 2021) | LTR | Feature Attribution | Sufficiency/Completeness
Gradient Saliency (Purpura et al., 2021) | LTR | Feature Attribution | Faithfulness
Intent Modeling (Singh and Anand, 2020) | Text Ranking | Terms/Words | Faithfulness/Reference
CtrsGen (Zhang et al., 2020a) | Text Ranking | Free-text | Reference
GenEx (Rahimi et al., 2021) | Text Ranking | Free-text | Reference/Human
LiEGe (Yu et al., 2022a) | Text Ranking | Topic Words | Reference
Universal Adv. Triggers (Wang et al., 2022) | Text Ranking | Trigger | Anecdotal/Visualization
Table 1. Overview of post-hoc explanation methods. The evaluation of post-hoc
methods can be anecdotal, visualized, or can be intrinsically measured by a
corresponding faithfulness measure. “Reference” refers to comparison with
ground-truth explanations, an interpretable model, or another attribution
method.

##  4\. Free-text Explanations

Free-text explanations methods aim to generate explanations using natural
language and are thus also called natural language explanations. Compared to
feature attributions, the explanations can be more expressive, as they are not
limited to words that already contain the input. Typical free-text
explanations are not more than a few sentences long, and sometimes even
limited to a few words. This form of explanation is popular for both textual
and visual-textual tasks, for which a variety of datasets have been collected
or expanded to include explanations (Wiegreffe and Marasovic, 2021). However,
apart from a few question-answering datasets, none of them are closely related
to IR. Instead, this explanation style is commonly used for tasks that involve
reasoning. Since for such tasks, the information contained in the inputs is
often insufficient to achieve good task performance, the explanations must
also contain external information apart from what is contained in the inputs.
In fact, many datasets that include free-text explanations are used to improve
the task performance of the model. The idea is that a model will generalize
better if it can also explain its predictions (Camburu et al., 2018; Kumar and
Talukdar, 2020; Liu et al., 2019; Rajani et al., 2019).

Approaches to generating free-text explanations for text ranking models focus
either on interpreting the query intent as understood by a ranking model or on
producing a short text summary to explain why an individual document or a list
of documents is relevant.

###  4.1. Explaining Query Intent

Satisfying the information need of a user that issues a search query is a key
concept in IR. Explaining the intent as understood by black box ranking models
can be useful to examine whether complex ranking models perform in accordance
with a user’s intent.

####  4.1.1. Query Expansion

Singh and Anand (2020) propose a model-agnostic approach to interpret a query
intent as understood by a black-box ranker. Given a single query and a set of
expansion terms as input, they fit an interpretable term-based ranking model
to mimic the complex model to be interpreted. The goal is to identify a set of
query expansion terms such that most of the pairwise preferences in the output
ranking are preserved. Query expansion terms are selected by optimizing the
preference pair coverage using greedy search. The expanded query terms act as
an explanation for the intent perceived by the black-box ranking model, as
Figure 6 demonstrates.

Figure 6. Example of query expansion terms (green) as explanations. The
expansion terms are chosen such that an interpretable term-based ranker best
approximates the ranking of documents produced by a black-box ranking model.

In experiments with a variety of ranking models, including RM3 (Lavrenko and
Croft, 2001), DESM (Nalisnick et al., 2016), DRMM (Guo et al., 2016), P-DRMM
(McDonald et al., 2018), trained on the Robust04 collection (Voorhees, 2006),
they show that this approach can produce explanations with high fidelity.

####  4.1.2. Generating Query Descriptions

Zhang et al. (2020a) introduce a Query-to-Intent-Description task for query
understanding. Given a query and a set of both relevant and irrelevant
documents, the goal is to generate a natural language intent description. To
solve this task, they propose CtrsGen, a contrastive generation model that
generates a query intent description by contrasting the relevant and
irrelevant documents. The training data for CtrsGen consists of multiple TREC
and SemEval (Chatterjee et al., 2019) collections that already include query
descriptions. Although not explicitly discussed by Zhang et al. (2020a),
CtrsGen can be used to explain query intents as understood by a black-box
ranker by selecting relevant and irrelevant documents based on the output of
the ranking model. However, it has not yet been examined whether the
generations of CtrsGen explain the underlying ranking model faithfully.

###  4.2. Explaining Document Relevance

A recent line of work in explainable document retrieval aims to explain why a
document or a set of documents is considered relevant to a query by generating
free-text explanations. Compared to other model-agnostic explanation methods,
free-text explanations are not limited to explaining document relevance using
features that are already contained in the input. A user study by (Rahimi et
al., 2021) suggests that adding free-text document relevance explanations to
search engine result pages can help users identify relevant documents faster
and more accurately.

####  4.2.1. Pointwise Explanations

Rahimi et al. (2021) generate document relevance explanations for individual
query-document pairs. They propose GenEx, a transformer-based model that
outputs free-text document relevance explanations. Given a query-document
pair, GenEx learns to generate a text sequence that explains why the document
is relevant to the query. The explanations consist of only a few words instead
of whole snippets, and explicitly avoid reusing the terms already contained in
the query. The model uses an encoder-decoder architecture, with the decoder
being extended by a query-masking mechanism to decrease the probability of
generating tokens that are already contained in the query. The training data
consists of query-document-explanation triplets and is automatically
constructed from Wikipedia articles and the ClueWeb09 dataset (Callan et al.,
2009).

####  4.2.2. Listwise Explanations

Yu et al. (2022a) argue that explaining documents independently is inherently
limited. Per-document explanations do not explain differences between
documents, and a single document can potentially cover multiple query aspects
at the same time. As a solution, they propose a listwise explanation generator
(LiEGe) that for a given query jointly explains all the documents contained in
a ranked result list. LiEGe is based on an encoder-decoder transformer
architecture and uses pre-trained weights from BART (Lewis et al., 2020). The
authors introduce two settings for search result explanations: 1)
comprehensive explanation generation, where the explanation contains all query
aspects covered by each document, and 2) novelty explanation generation, where
the explanation contains a description of the relevant information of a
document that is novel, considering all the preceding documents in the ranked
list. Two weakly labeled datasets are constructed from Wikipedia to train
LiEGe for these two settings, the evaluation dataset is constructed using
query logs from the MIMICS dataset (Zamani et al., 2020).

###  4.3. Evaluation of Free-text Explanations

The evaluation of free-text explanations is generally based on the
availability of ground-truth explanations. Although explanations are not
included in most IR datasets, proxy explanations can be created from query
descriptions, query aspect annotations, topic annotations, or click logs
(Zhang et al., 2020a; Rahimi et al., 2021; Yu et al., 2022a). BLEU (Papineni
et al., 2002) and ROUGE (Lin, 2004), two metrics commonly used to evaluate
text summarization and machine translation tasks, can be used to compare
generated free-text explanations with reference explanations. Furthermore,
Rahimi et al. (2021) and Yu et al. (2022a) use BERTScore (Zhang et al., 2020b)
to measure semantic coherence. However, human-annotated but model-independent
ground-truth explanations can only be used to evaluate the plausibility of
generated explanations. Whether the generated explanations are faithful to the
ranking model being explained remains an open question. Only Singh and Anand
(2020) evaluate the faithfulness of their query intent explanations since they
have to ensure that the interpretable ranker used during optimization closely
mimics the black-box ranking model being explained. To examine whether GenEx
explanations actually help users, Rahimi et al. (2021) conduct a user study.
Specifically, they collect explanation preferences, linguistic quality
ratings, and relevance judgments from crowd-workers, comparing GenEx
explanations with different baseline explanations.

##  5\. Adversarial Examples

Adversarial examples are commonly used to demonstrate the fragility or
robustness of machine learning models. However, they can also serve as
explanations and provide valuable insight. In fact, adversarial examples are
closely related to counterfactual examples, but instead of providing
actionable recourse, the goal is to fool machine learning models. Given an
individual input to a model, a corresponding adversarial example is crafted by
applying small deliberate perturbations to deceive a model into making a wrong
prediction. The resulting adversarial examples inform about the minimal input
changes required to change a prediction and thus provide insight into the
decision behavior of the model. Specifically, the adversarial perturbations
indicate which input features have to change by how much to alter a predicted
outcome. Compared to feature attributions (Section 3), adversarial
explanations are contrastive explanations, since the adversarial example is
always compared to the unmodified input example. From the perspective of
social science, Miller (2019) argues that such contrastive explanations can be
considered more human-grounded.

###  5.1. Adversarial Examples in Ranking

Most of the work on adversarial examples is concerned with classification
tasks, where a wrong prediction is defined by comparing the predicted label
with a target label. For ranking tasks, the main objective of an adversarial
perturbation is to cause a relatively large rank promotion or rank demotion of
a document. For example, a company aiming to optimize search engines could
leverage adversarial attacks to promote a specific web page to the top of a
search result page with minor changes in the page content itself.

Raval and Verma (2020) generate adversarial examples for black-box retrieval
models that lower the position of a top-ranked document with minimal changes
to the document text. Given the non-differentiability of replacing discrete
tokens, they optimize adversarial examples using a stochastic evolutionary
algorithm with a one-token-at-a-time replacement strategy. Wu et al. (2022)
take a different approach by training a surrogate model based on pseudo-
relevance feedback, which is used to approximate the gradient of the
underlying black box ranking model. This approximated gradient is then used to
find adversarial perturbations that promote a target document. Additionally,
the adversarial perturbations are restricted by semantic similarity to the
original document. The authors argue that the perturbations are imperceptible
and evade spam detection when constraining the perturbations to semantic
synonyms. Goren et al. (2020) craft adversarial examples for the LambdaMART
LTR model. For a given query, they use past rankings to create perturbations
by replacing passages in the target document with passages from other high-
ranked documents. Wang et al. (2022) use gradient-based optimization to
generate adversarial examples for BERT-based ranking models. They add or
replace a few tokens in documents that cause significant rank promotions and
demotions.

###  5.2. Universal Adversarial Triggers

While adversarial examples focus on input perturbations that change the
prediction of individual inputs, universal adversarial triggers (Wallace et
al., 2019) are input-agnostic perturbations that lead to a model making a
specific prediction whenever the trigger is concatenated to any input.
Starting from an initial sequence of tokens, a trigger is optimized via a
gradient-based search algorithm that iteratively replaces tokens. The effect
of replacing a discrete token is usually approximated using HotFlip (Ebrahimi
et al., 2018). Since the resulting triggers transfer across input examples,
they can be used to explain the global behavior of a model and can reveal
global patterns.

####  5.2.1. Universal Triggers for Text Ranking

Wang et al. (2022) adapt universal adversarial triggers for text-based ranking
models. They propose a global ranking attack to find trigger tokens that are
adversarial to all queries contained in a dataset. Specifically, they optimize
a fixed-length trigger so that any document to which it is concatenated will
be demoted (or promoted) as much as possible for any given query. In their
experiments with BERT-based ranking models fine-tuned on ClueWeb09 (Callan et
al., 2009) and MS MARCO (Nguyen et al., 2016), they discover topical patterns
within and between datasets and expose potential dataset and model biases. For
example, the trigger

hinduism earthquakes childbirth tornadoes Wikipedia

promotes a document by 63 ranks on average, and the trigger

acceptable competition rayon favour ##kei

demotes a document by 84 ranks on average across all queries. In general,
finding triggers for which highly relevant documents get demoted appears
easier than finding triggers for which low-ranked documents are promoted.

##  6\. Axiomatic Analysis of Text Ranking Models

Unlike current data-driven, parameterized models for relevance estimation,
traditional IR approaches to ranking involve probabilistic models of relevance
such as BM25 (Amati and Van Rijsbergen, 2002) and axiomatic approaches. Both
approaches have a top-down defined notion of relevance, allowing for some sort
of interpretability. Yet, the probabilistic models are currently dominant and
axiomatic approaches less popular. In contrast to the recent development of
neural, and therefore less interpretable, rankers, _Axiomatic IR_ postulates
and formalizes the properties of principled rankers. The term _axiom_ in IR
was first coined by Bruza and Huibers (1994), who proposed to describe
retrieval mechanisms using axioms expressed through concepts in the field of
IR.

Figure 7. Example of applying the TFC1 (Fang et al., 2004) axiom to rank two
documents. Query terms are highlighted. $d_{i}$ is ranked higher than $d_{j}$
because it contains more query terms.

An example axiom is TFC1 (Fang et al., 2004) which proposes to prefer
documents having more query terms occurrences (Figure 7). Formally, given a
query $q={t}$ and two documents $d_{1},d_{2}$ with $|d_{1}|=|d_{2}|$, TFC1 is
defined as

(1) $tf(t,d_{1})>tf(t,d_{2})\Rightarrow d_{1}>_{TFC1}d_{2}.$

Similarly, a large set of axioms has been proposed in recent decades, ranging
over different aspects of relevance such as term frequency (Fang et al., 2004,
2011), document length (Fang et al., 2004), semantic similarity (Fang and
Zhai, 2006), or term proximity (Hagen et al., 2016) among others (see Table
2). For a more detailed description of the various axioms, we refer to an
overview by Bondarenko et al. (2022).

Property | Axiom | Details
---|---|---
Term Frequency |  TFC1 (Fang et al., 2004) |  | Prefer documents with more query term occurrences
---
Given $Q={q},|D_{1}|=|D_{2}|$, $tf(q,D_{1})>tf(q,D_{2})$ $\Rightarrow
D_{1}>_{TFC1}D_{2}$
Document Length |  LNC1 (Fang et al., 2004) |  | Penalize long documents for non-relevant terms
---
Given $t\notin Q$, arbitrary term $w$, $tf(t,D_{1})>tf(t,D_{2})\land$
$\forall_{w\neq t}tf(w,D_{1})=tf(w,D_{2})$ $\Rightarrow D_{1}>_{LNC1}D_{2}$
Semantic Similarity |  STMC1 (Fang and Zhai, 2006) |  | Prefer terms more similar to query terms
---
Given $Q={q},D_{1}={t_{1}},D_{2}={t_{2}},q\neq t_{1},q\neq t_{2}$,
$sim(q,t_{1})>sim(q,t_{2})$
$\Rightarrow D_{1}>_{STMC1}D_{2}$
Query Aspect |  AND (Zheng and Fang, 2010) |  | Prefer documents containing all query terms
---
Given $Q={q_{1},q_{2}},td(q_{1})\geq td(q\\_2)$, $tf(q_{1},D_{1})=1\land
tf(q_{2},D_{1})=1$
$\land tf(q_{1},D_{2})>1\land tf(q_{2},D_{2})=0$ $\Rightarrow
D_{1}>_{AND}D_{2}$
Table 2. Selection of proposed retrieval axioms. Adapted from a more complete
list of axioms available in (Bondarenko et al., 2022).

Axioms are human-understandable concepts. This is in stark contrast to neural
networks, which have been shown time and time again to learn spurious
correlations (Geirhos et al., 2020) and to be susceptible to adversarial
attacks (Wang et al., 2022). Although not yet achieved, a long-term goal of
axiomatic IR could be a concept of relevance built on axioms. This
conceptualization of relevance would then be robust to attacks, generalize to
novel distributions, and be interpretable for humans.

Although there is no general model of relevance yet, previous work aggregated
axioms to build axiomatic rankers (Section 6.1), analyze and explain existing
neural ranking approaches by aligning them to known axioms (Section 6.2), and
use axioms to regularize the training of neural rankers (Section 6.3). An
overview of this classification and papers in this section can be found in
Table 3.

Paper | Task | Approach | Dataset | Evaluation
---|---|---|---|---
Hagen et al. (2016) | LTR | IBD | TREC Web tracks 2009-2014 | -
Rennings et al. (2019) | Text Ranking | Post-hoc | WikiPassageQA | -
Câmara and Hauff (2020) | Text Ranking | Post-hoc | TREC 2019 DL | -
Völske et al. (2021) | Text Ranking | Post-hoc | Robust04, MS MARCO | Fidelity
Rosset et al. (2019) | Text Ranking | Regularization | MS MARCO | -
Cheng and Fang (2020) | Text Ranking | Regularization | WikiQA, MS MARCO | -
Chen et al. (2022) | Text Ranking | Regularization | MS MARCO, TREC 2019 DL | Anecdotal
Table 3. Classification of axiomatic methods. The evaluation w.r.t.
interpretability can be anecdotal or intrinsically measured by a corresponding
faithfulness measure.

###  6.1. Interpretable Axiomatic Rankers

Hagen et al. (2016) is one of the first to operationalize retrieval axioms to
perform axiomatic re-ranking. By learning the importance of individual axioms,
they aggregate the axioms’ partial orderings. Despite being inherently more
interpretable, they evaluate their axiomatic re-ranking step with a selection
of retrieval models, showing that for most of them the performance
significantly increases. Given that the axioms and the aggregation method are
fully interpretable, the resulting re-ranking is also fully interpretable.
Bondarenko et al. (2022) proposed a utility library called _ir_axioms_ that
allows experimenting with a collection of 25 different axioms and allows one
to add new axioms. This library can be used for axiomatic result re-ranking
and diagnostic experiments to explain neural ranking models.

###  6.2. Axioms for Model Diagnostics

More directly related to the classical post-hoc interpretability work is a
line of recent works diagnosing and explaining ranking models using axioms.
Rennings et al. (2019) constructed diagnostic datasets based on existing
axioms and checked whether classical neural ranking models are in agreement
with the axiomatic rules. They find that out-of-the-box neural rankers conform
with the axiomatic rankings to only a limited extent. However, they
hypothesize that including diagnostic datasets in the training process could
boost this conformity. Câmara and Hauff (2020) extend this work and apply
diagnostic datasets similarly to ad-hoc retrieval with BERT. They find that
BERT does not align with most of the ranking axioms but significantly
outperforms other neural and classical approaches. The authors conclude that
the current set of axioms is insufficient to understand BERT’s notion of
relevance. Last in this line of work is an approach to produce axiomatic
explanations for neural ranking models by Völske et al. (2021). Similar to
existing work on axiomatic re-ranking (Hagen et al., 2016) and diagnosing
neural rankers (Rennings et al., 2019; Câmara and Hauff, 2020), this study
investigates whether neural rankings can be explained by the combination of
existing axioms. To do so, they train a small random forest explanation model
on the axioms’ partial orderings to reconstruct the ranking list produced by
the neural ranking model. They find that axiomatic explanations work well in
cases where the ranking models are confident in their relevance estimation.
However, these explanations fail for pairs with similar retrieval scores and
conclude that more axioms are needed to close this gap.

###  6.3. Axioms for Regularizing Neural Rankers

Recently, a variety of approaches for axiomatic regularization of neural
ranking models has been proposed (Rosset et al., 2019; Cheng and Fang, 2020;
Chen et al., 2022). These approaches aim to regularize opaque neural rankers
to incentivize learning of the principled, axiomatic notions of relevance.
This has the benefits of faster convergence (Rosset et al., 2019), improved
performance (Cheng and Fang, 2020) or generalization ability (Rosset et al.,
2019; Chen et al., 2022), and improved interpretability (Chen et al., 2022).
The method by which the ranking models are regularized varies from adding a
regularization term to the loss function (Rosset et al., 2019; Chen et al.,
2022) to axiomatically perturbing the training data to amplify desirable
properties (Cheng and Fang, 2020). An example of such a regularization term is
applied by Chen et al. (2022) who add a relevance loss to their final loss
function that checks how well the model’s relevance judgments coincides with
the axioms’. Cheng and Fang (2020) extend the training dataset by randomly
sampling instances and perturbing them according to three document length
normalization axioms, such as by adding noise terms. Then, these more noisy
documents are assigned a lower relevancy value. From such perturbed data
examples, the model is expected to understand the corresponding normalization
axiom based on document length. While current regularization methods offer
only limited (perceived) interpretability, the approach similar to the neuro-
symbolic approaches (Sarker et al., 2021) marry the benefits of both axioms
and data-driven models.

###  6.4. Evaluation

IR axioms have been applied in various works over the past decades, and many
revolve around interpretability. However, little formal evaluation of the
insights gained through the axioms has been done from an interpretability
perspective. One exception is Chen et al. (2022), who give anecdotal examples
of their axiomatically regularized model’s input attribution being more sparse
and focused on relevant tokens. In addition, only Völske et al. (2021) use
established interpretability evaluation metrics and measure the fidelity of
their generated (post-hoc) explanations. From the interpretability
perspective, two steps are needed for upcoming work: 1), proposing new axioms
or methods to better explain neural ranking models and 2), rigorously
evaluating the produced explanations with established metrics and eventually
human acceptance studies.

##  7\. Probing and Parametric Analysis of Text Ranking Models

Probing is a method to analyze the content of latent embeddings. It allows us
to understand the information encoded in the model’s representations. Usually,
probing includes training a small classifier to predict the property of
interest (e.g., part-of-speech tags or question types) directly from the
embeddings (Tenney et al., 2019b, a; Voita and Titov, 2020; Belinkov, 2022).

###  7.1. The Probing Methodology

Figure 8 shows an example in which we test whether a ranking model encodes
information on different question types.

Figure 8. Example of the probing paradigm. A small classifier (the probe
model) is used to predict properties (in this case the question type) from a
ranker’s frozen representations.

To do so, we need a small, labeled dataset of questions and their respective
question types. We then train the probing classifier to recover the question
type information from the ranker’s frozen embeddings. Originally, the model
would be considered to encode the property of interest if the classifier can
better predict it than a majority classifier. However, depending on the task’s
difficulty, dataset size, and classifier complexity, large portions of the
resulting performance must be attributed to the classifier. Therefore, a large
set of improvements to the probing paradigm have been proposed – from
introducing suitable baselines (Zhang and Bowman, 2018) and control tasks
(Hewitt and Liang, 2019), over varying the classifier complexity (Pimentel et
al., 2020), to measuring the target property’s ease of extraction from the
embeddings (Voita and Titov, 2020). For a more comprehensive overview of the
initial probing paradigm and the proposed improvements, we refer to the paper
by Belinkov (2022).

Paper | Task | Concept under Investigation | Architectural component
---|---|---|---
Choi et al. (2022) | Text Ranking | IDF | Attention
Zhan et al. (2020) | Text Ranking | Attention, ¡Q, D¿ Interactions | Attention, Embeddings
Formal et al. (2022) | Text Ranking | Lexical Matching | Behavioral
Formal et al. (2021) | Text Ranking | Matching, Term Importance | Behavioral
Sen et al. (2020) | Text Ranking | TF, IDF, Document Length | Behavioral
MacAvaney et al. (2020) | Text Ranking | Matching, Manipulation, Style | Embeddings
Fan et al. (2021) | Various IR Tasks | Relevance Modeling | Embeddings
van Aken et al. (2019) | QA | QA Subtasks | Embeddings
Cai et al. (2020) | RC | MRC Subtasks | –
Wallat et al. (2020) | Various NLP Tasks | Factual Knowledge | Embeddings
Petroni et al. (2021) | <Benchmark> | Factual Knowledge | <Benchmark>
Table 4. Classification of the probing literature (Section 7). These papers
usually investigate whether models trained on a downstream (IR) task encode a
concept (such as lexical matching) in different architectural components
(e.g., the attention maps). Behavioral studies do not probe a specific model
component but investigate the model’s general behavior.

###  7.2. Probing Ranking Models

Several variations of the probing paradigm have also been applied to various
IR tasks and models. An overview of the papers, together with a
classification, can be found in Table 4. As a text-ranking model, the approach
of Zhan et al. (2020) investigates the attention patterns of BERT after fine-
tuning on the document ranking task. Their experiments show that large parts
of the attention are off-loaded to low information tokens such as punctuation,
which might lead to increased susceptibility to adversarial attacks.
Similarly, a recent study by Choi et al. (2022) probes the attention maps of a
BERT ranker, finding that inverse document frequency is captured. As discussed
in Section 6, the existing ranking axioms are insufficient to explain rankings
produced by BERT-based models. Therefore, Formal et al. (2021) investigate the
ColBERT regarding its term-matching mechanism. By stratifying on IDF bins,
they show that ColBERT indeed captures a notion of term importance, which is
enhanced by fine-tuning. However, the results suggest that estimating term
importance is limited when no exact matches are available. Given the limited
ability of current neural retrieval models to generalize to new datasets,
Formal et al. (2022) question whether this is caused by their inability to
perform lexical matching in the out-of-domain scenario. While general lexical
matching ability is present in neural retrievers (such as TAS-B or ColBERT),
the understanding of which terms are important to match seems to be missing in
the out-of-domain setting. Sen et al. (2020) aim to attribute relevance
prediction performance to term frequency, document frequency, or document
length. To do so, they train a linear model using these aspects to approximate
the ranking model. The resulting coefficients are then used to understand the
importance of the corresponding aspects. The resulting explanations confirm
that the model behavior follows certain constraints used in axiomatic IR
(Section 6). MacAvaney et al. (2020) also further investigate the hidden
abilities of neural rankers that lead to their good ranking performance. They
attribute the model’s matching ability to three properties (concepts),
relevance, document length, and term frequency. They devise a behavioral-
probing setup that verifies to what extent the model could capture these
concepts. For manipulation-sensitivity analysis, they test the effect of
shuffled words, sentences, or typos on the model performance. Lastly,
MacAvaney et al. (2020) create probing sets for writing style concepts such as
fluency, formality, or factuality. Their results suggest that neural rankers
are biased toward factually correct articles and that appending irrelevant
text can improve the relevance scores. Similarly, the work by Fan et al.
(2021) strives to understand the relevance-modeling of IR models. They also
propose to probe for a large set of lexical, syntactic, and semantic concepts
such as named entities or coreference resolution ability. By comparing the
performance of their fine-tuned models to a pre-trained BERT, they find that
these IR models generally seem to sacrifice small parts of their ability to
perform lexical and syntactic tasks and improve especially in semantic
matching (e.g., identifying synonyms). Furthermore, causal intervention
analysis is applied to the model parameters, input features, and training
objectives, resulting in suggesting that a careful intervention on linguistic
properties can improve the performance of downstream IR models.

###  7.3. Probing other Information Retrieval Models

In addition to the core ranking objective, models for other IR-related tasks
have been probed. van Aken et al. (2019) investigate BERT embeddings of a QA
model and how do they interact over the layers when answering questions.
Specifically, they probed a pre-trained BERT and a QA model, finding that
training the model for QA improves the performance on related tasks such as
question type classification or identification of supporting facts. The
question of how BERT reacts to fine-tuning has also been investigated in
several studies (Wallat et al., 2020; Fan et al., 2021; van Aken et al.,
2019). Cai et al. (2020) probe MRC (machine reading comprehension) models for
relevant subtasks (synonyms, abbreviations, coreference, as well as question
type classification). They find that only for core MRC subtasks, the token
representation varies in the later layers of the MRC model. The core MRC
subtasks include tasks such as coreference, question type classification, and
answer boundary detection. However, for tasks like synonym and abbreviation
detection, the representations are only moderately different from the pre-
trained BERT representations. Wallat et al. (2020) probe models fine-tuned for
various tasks to assess the effect of fine-tuning on (factual) knowledge
retention. In their layer-wise experiments, they find the ranking model to be
specifically knowledgeable, dropping the least amount of knowledge compared to
the question-answering and named entity recognition models. Additionally,
large parts, though not all, of the factual knowledge seem to be captured in
the latter layers. Petroni et al. (2021) identify the requirement of world
knowledge for many IR tasks such as open-domain question-answering, slot
filling, entity linking, or fact-checking. To understand to what extent do
current models capture real-world knowledge, Petroni et al. (2021) propose a
benchmark containing knowledge-intensive tasks (QA, slot filling, entity
linking, fact-checking, among others) all derived from a single Wikipedia
corpus.

###  7.4. Evaluation

In the past, probing results have been evaluated differently by the
interpretability community than other post hoc methods. Whereas other methods
such as feature attributions have been rigorously evaluated concerning metrics
such as fidelity or faithfulness, this has not been the case in the probing
literature. As suggested by Belinkov (2022), a standard probing setting can
answer the question: _What information can be decoded from the model’s
embeddings?_ It does not offer a human-centered explanation for a specific
data instance, but rather provides general information about the model.
Thereafter, it does not offer interpretability for users but for model
developers, although the probing methodology has been scrutinized and extended
in various works (Sturmfels et al., 2020; Hewitt and Liang, 2019; Voita and
Titov, 2020). Given the correct baselines and a tightly controlled setup, it
might be able to shed light on the question of _What information is learned by
training on a specific task?_ or _How easily extractable is information about
a concept from the model?_ (Voita and Titov, 2020). However, it is unclear
whether this information is actually being used by the model at inference time
(Belinkov, 2022). To resolve this, recent studies borrow ideas from causality
research to understand whether a specific concept is utilized during the
inference using counterfactual representations, where the concept is voided
(Elazar et al., 2021; Lasri et al., 2022). The model is proven to have used
the concept if the counterfactual representations result in worse task
performance. In conclusion, while there has been an in-depth evaluation of the
probing paradigm by the NLP and interpretability community and many
improvements have been proposed, little of that found its way into IR-related
probing studies. Future probing studies in IR will need to include learnings
and best practices from established research and use them to evaluate and
validate the findings for IR models.

Figure 9. Types of IBD models. Green and gray color refers to mostly
interpretable/non-interpretable components, respectively.

##  8\. Explainable-by-Architecture Models

We refer to the first family of IBD models as explainable-by-architecture
models. Those models can be viewed as a modular framework of multiple
components (see Figure 9). The general architecture of these models involves
intermediate feature extraction (that might involve feature attributions), and
a task-specific decision structure (that might involve feature interactions).
Pragmatically speaking, not all components are fully interpretable to ensure
competitive task performance. Therefore, most of the IBD resort to making only
specific components interpretable or transparent. In the following, we look at
two major use cases of such models in text ranking and LTR tasks.

###  8.1. Explainable Text Rankers

In text ranking, the need for interpretability is based on large input sizes
and complex feature interactions. Since documents can be long, it is hard to
ascertain what sections of text the query terms interact with within a complex
model. This problem is particularly acute in the case of contextual models
with transformers, where the self-attention mechanism essentially considers
all pairs of interactions between the query and the document terms. Therefore,
one strategy of the IBD models in the text ranking family focuses on building
interpretable query-document interaction functions and, in turn, leading to a
more transparent decision-making path. In this setup, the query and the
document are encoded separately by two individual models and each token (or
word) is represented by a fixed-size embedding vector. Note that this encoding
process remains opaque for both context-free and contextualized embeddings. A
(partially) explainable model employs human-understandable functions to
measure the degree of query-document interactions, which essentially indicates
the similarity of the query and the document. The final relevance judgment can
then be made based on the interactions. Another line of IBD text rankers is
focusing on reducing the large input space, which we refer to as rationale-
based methods. The idea is to use a small set of explicit words or sentences
as input leading to the final prediction, whereas how the input is selected,
and how the prediction is made, remains agnostic. There are extensive works in
building such sorts of models, to highlight the popularity, we will further
discuss this method family in Section 9.

####  8.1.1. Feature Interaction

We summarize three ranking models, which utilize two BERT/Transformer-style
encoders to generate the vectorized representations for query and document
individually. In the following paragraphs, we emphasize on their interaction
and decision-making processes, showing how the relevance decision can be
explained.

Colbert (Khattab and Zaharia, 2020) follows the conventional term-matching
strategy. For each query token, it computes the cosine similarity scores with
each token from the document and keeps the maximum similarity score. The final
document relevance is computed by simply summing up the maximum scores of all
query tokens. Essentially, Colbert measures the semantic similarity between
the query and the document, and a document is deemed more relevant if it
contains more terms that are semantically closer to the query. Boytsov and
Kolter (2021) propose NeuralModel1, which adds an explainable layer, namely
Model1 (Berger and Lafferty, 1999) on top of the input embedding.
Specifically, the non-parametric Model1 layer maintains pairwise similarity
statistics between query-document tokens, which are learned/computed from
parallel datasets beforehand. The final document relevance is combined from
all query-document similarity scores by the product-of-sum formula. This
approach is very similar to Colbert, where the cosine similarity computation
can also be viewed as an explainable layer. NeuralModel1 experimented with
slightly more comprehensive similarity learning, resulting in lower
interpretability. Nevertheless, with a more complex interaction mechanism,
NeuralModel1 achieves better balance in terms of ranking performance and
efficiency.

Transformer-Kernel (Hofstätter et al., 2020) maintains a matching matrix,
where each row represents the cosine similarity scores between a particular
query token and all document tokens. In contrast to Colbert, which simply
takes the maximum similarity from each row and sums them up to obtain the
query-document relevance, Transformer-Kernel transforms the matching matrix to
a set of isomorphic matrices with RBF-kernels (Xiong et al., 2017) and each
kernel focuses on a specific similarity range. This interaction shares a
similar idea as the similarity histogram in DRMM (Guo et al., 2016) model but
employs the kernel-pooling technique to solve the non-differentiation of the
hard histogram. The final relevance score is learned by a feed-forward layer,
given the semantic matrices as the input. Therefore, the interaction in
Transformer-Kernel can be viewed as smoothed semantic similarity, and the
relevance decision is learned via a neural layer, making the Transformer-
Kernel less interpretable in comparison to Colbert and NeuralModel1.

###  8.2. Explainable Learning-to-Rank

For LTR task dealing with smaller numerical input features, there are works
relying on explicitly aggregating feature contribution for relevance
measurement, or a fully transparent tree model. The goal of LTR is to predict
a relevance score for a fixed-size input vector. Because of the smaller and
structured input features, it is more practical to build a transparent model
in input space or the entire decision path. In the following paragraphs, we
introduce one LTR model with explicit feature contribution and one transparent
decision-tree model incorporated with reduced input space.

####  8.2.1. Explicit Feature Contribution.

Different from the previously discussed feature-attribution methods, explicit
feature contribution indicates a simple and transparent correlation between
each input feature and the relevance prediction, in addition to showing
importance heatmaps.

NeuralGAM (Zhuang et al., 2021) is based on Generalized Additive Models
(GAMs). For each individual feature, NeuralGAM employs an isolated black-box
(e.g., neural networks) model to generate a score indicating the contribution
(or importance) of the feature. The relevance of the input is aggregated by
simply summing up all contribution scores. NeuralGAM is explainable in terms
of feature contribution, as the relevance is aggregated from the feature
importance score directly by a simple sum operation. Nevertheless, it remains
opaque how each feature importance score is generated by the black-box model.

####  8.2.2. Explainable Tree Models

The main challenge of interpreting tree models is the over-complex decision
path caused by the massive number of features and their interactions. Thus, an
explainable tree model should have a limited number of features and
interactions and, in turn, be able to provide a simple and understandable
decision-making path.

ILMART (Lucchese et al., 2022) shares a similar structure as GAM, while using
LambdaMART as the sub-model. ILMART starts from learning a set of trees, with
each dealing with one single distinct feature only. This step enables ILMART
to identify a small yet crucial set of features and exclude the rest. Then,
ILMART enforces a new ensemble of trees to explore the interactions between
every two remaining features only. This design can effectively reduce the
model’s complexity. Finally, ILMART combines trees from the previous two steps
and learns a much smaller and simpler ensemble-tree model with the input space
hugely reduced.

###  8.3. Evaluation

A key attribute of interpretable models is, it does not just highlight the
importance of input snippets/dimensions (e.g., tokens in a query or document),
but also suggest why those snippets lead to the decision. Namely, a set of
rules can be implicitly inferred from the explanations, even when only the
input features are presented. This is the usual case when the audience group
of explanation is system developers or domain experts. One explanation example
for Colbert can be a small set of tokens in the query and document, together
with their cosine similarity degree. We denote this type of explanation as
soft-rule, to distinguish from the hard-rule of an explicit path in a tree
model. NeuralGAM presents feature attribution scores (similar to Section 3) as
explanations and moreover, the relevance decision can be explicitly induced
from the scores.

Except for Colbert, all methods evaluate the goodness of explanations by
showing anecdotal examples. Additionally, NeuralGAM compares the features to a
referenced tree-model, and justifies the faithfulness of explanations by a
similar trend. A summary of methods can be found in Table 5.

Method | Task | Components | Explanation | Dataset | Evaluation
---|---|---|---|---|---
Colbert (Khattab and Zaharia, 2020) | Text Ranking | Interaction | Soft-rule | MS MARCO | -
Transformer-Kernel (Hofstätter et al., 2020) | Text Ranking | Interaction | Soft-rule | MS MARCO | Anecdotal
NeuralModel1 (Boytsov and Kolter, 2021) | Text Ranking | Interaction | Soft-rule | MS MARCO | Anecdotal
NeuralGAM (Zhuang et al., 2021) | LTR | Input | Feature Attr. | Yahoo | Reference
ILMART (Lucchese et al., 2022) | LTR | Fully | Hard-rule | Yahoo | Anecdotal
Table 5. Explainable-by-architecture Methods. Components indicate which
component of the model architecture is explainable. Note that Colbert did not
discuss or evaluate explainability. More similar datasets are used in each
paper, and we choose one as representative.

##  9\. Rationale-based Methods

The second class of IBD methods deals to enhance the interpretability of IR
models by generating rationales as an intermediate sparse input representation
(see Figure 9). A rationale is defined as an extractive piece of the input
text that is responsible for the decision of the model. A rationale-based
method performs the task prediction two-stage. In the first feature-extraction
phase, a model learns to extract the rationale from the input text. In the
subsequent prediction phase, another independent task model predicts the task
output solely based on the extractive explanation. Note that in such a setup,
each prediction can be unambiguously attributed to the generated rationale
that is both human-understandable and acts as an explanation. Examples of
rationales are provided below in Figure 10.

Figure 10. Example of a binary rationale selection. First, a subset of tokens
is selected. Then, a prediction is made based on the selected rationale
tokens. Selected rationales are highlighted in orange.

We summarize the approaches in this section in Table 6. The feature extraction
stage in rationale-based models is sometimes called the selection or
rationale-generation stage (Zhang et al., 2021a; Leonhardt et al., 2021). The
major challenge in rationale-based methods is training the rationale-
extraction module due to the discrete rationale output of this stage. There
are essentially two types of rationale-based methods based on the optimization
styles – pipeline or end-to-end.

###  9.1. Pipeline Approaches

A rationale-based model is a pipeline model if the rationale-extraction module
is trained separately from the task prediction module. Most pipeline methods
require the presence of human-annotated, extractive rationale data to train
the rationale-extraction network. The first pipeline model was proposed by
Lehman et al. (2019). Their approach was proposed for the analysis of clinical
trial reports, where the aim is to predict whether the clinical trial causes a
significant effect compared with the control group. The reports are themselves
annotated by human experts, where experts do annotate not only the
significance of the trial but also the snippet of the reports as the rationale
supporting such prediction. This constitutes the training data for the
rationale-extraction module. During the inference, the prediction model takes
the output of the rationale-extraction module as its input.

Method | Task | Training | Dataset | Evaluation
---|---|---|---|---
Expred (Zhang et al., 2021a) | Knowledge Intensive | Pipeline |  ERASER Benchmark(DeYoung et al., 2020) | PRF Score, C/S
Wojtas and Chen (2020) | Medical Search | Pipeline | GM12878 | Feature Ranking
Jiang’s IB (Jiang et al., 2021) | Text Ranking | End-to-End | MS MARCO | –
Select and Rank(Leonhardt et al., 2021) | Text Ranking | End-to-End |  TrecDL, Core17, CW09 | Anecdotal
IDCM(Hofstätter et al., 2021) | Text Ranking | Pipeline | MS MARCO | –
IOT-Match(Yu et al., 2022b) | Legal Case Matching | Pipeline | ELAM, eCAIL | PRF Scores

Table 6. Rationale-based approaches. The C/S score in the evaluation refers to
comprehensive and sufficiency.

Later in the same year, DeYoung et al. (2020) released a benchmark called
ERASER to evaluate rationale-based interpretability. The ERASER benchmark
consists of a large variety of knowledge-intensive tasks that presupposes an
IR system, like question answering (QA) and fact-checking. Despite the
reasonable performance benefits of such select-and-predict approaches, they
suffer from a crucial deficiency. That is, the rationale-extraction module
could “cheat” to overfit the pattern of the rationale sub-sequences instead of
selecting the rationales based on their semantic meaning (Jacovi and Goldberg,
2021; Bastings et al., 2021). To this end, another pipeline approach ExPred
(Zhang et al., 2021a) was proposed. The main idea of ExPred was to make the
rationale-extraction phase task-aware by training it in a multitask fashion
with the downstream task. By doing so, they use an auxiliary output to force
the rationale selector to learn the semantics of the inputs with respect to
the classification task.

The pipeline models introduced above contain only one extractor-predictor
training cycle. Wojtas and Chen (2020), however, propose to train the
rationale extractor and the task predictor alternatively for multiple rounds
and select the masks using a genetic algorithm. The alternative training cycle
is initialized by training the classifier on multiple randomly sampled
rationales and keeping the best rationale mask, resulting in the best
classification performance. Then they optimize the rationale-extractor and the
task-predictor alternatively.

For document ranking tasks, Hofstätter et al. (2021) propose the IDCM (Intra-
Document Cascading Model) approach to overcome the input length limitations of
modern transformer-based rankers. IDCM is a pipeline approach whose rationale
extractor is an effectively simple model (student model) trained to mimic the
passage-selection behavior of a more complex model (teacher model). The
student extractor model selects important passages as rationales from the huge
amount of documents before calculating the ranking score of selected passages
with respect to the current query using another complex model. Evaluated on
the MS MARCO dataset (Nguyen et al., 2016), IDCM turns to outperform
traditional ranking models.

Finally, another pipeline model called IOT-match (Yu et al., 2022b) focuses on
the case-matching problem. The case-matching problem is defined as: given two
different legal cases, the model should predict whether the two cases are
related. They try to solve the problem using optimal transport theory. The
intuition behind their algorithm is that the predicted sentence matching
matrix is also an optimal transport matrix that minimizes the transport
distance given the sentence discrepancy matrix. The sentence matching matrix
is a selection matrix that selects sentence pairs from both cases, where the
sum of their discrepancies is minimized (similarity maximized). After
selecting the most similar sentences from both cases, they utilize the task
prediction model to predict whether the two cases are related based on matched
sentence pairs as the rationales.

###  9.2. End-to-End Approaches

As its name suggests, we can train both the task and the rationale-generation
module jointly using the gradients from the task supervision signal. The major
technical challenge in this setting is that of gradient estimation for the
rationale generation parameters. Lei et al. (2016) is the first work that
proposes an end-to-end approach for the problem of rationale-based models in
the context of vanilla text classification. The rationale extraction module
parameterizes a per-token distribution of relevance. The output of this layer
is a hard binary mask as rationales. The parameters of this module are
optimized by estimated gradients, using a REINFORCE-style algorithm.
Additionally, they also applied constraints like the continuity of the
selected rationales and the sparsity of the rationales to further enhance the
sparsity. Extensions of this approach include (Lehman et al., 2019) and
(Bastings et al., 2019) that also focus on text classification, albeit using
reparameterization trick for better numerical stability and convergence rate
when training the rationale-extractor.

The first work to propose end-to-end methods for ranking tasks is (Leonhardt
et al., 2021). Their approach is called select and rank and is based on the
observation that only a few sentences in a related document are relevant given
a query. In the rationale-selection phase, they first select relevant
sentences from a document with respect to the query input. The selected
rationales act as an extractive summary of the document. After that, only
these rationales are used in the re-ranking phase with the query in a cross-
encoder ranker. Both the selector and the rankers are trained end-to-end using
a combination of the Gumbel-Softmax and reservoir sampling to ensure a user-
specified $k$-sentences to be sampled.

Unlike previous rationale-based models, Chen et al. (2018) use a mutual-
information-based formulation. Their theory is to select the rationales
containing the most mutual information with the final prediction. Jiang et al.
(2021) and Bang et al. (2021) further extend this information-theory-based
approach by considering the information bottleneck (IB) as the rationale.
Specifically, the information bottleneck $\mathbf{T}$ is an intermediate
(usually latent) representation that maximizes the mutual information between
$\mathbf{T}$ and the prediction $\mathbf{Y}$, while its mutual information
with the input $\mathbf{X}$ is minimized, i.e.,
$\mathbf{I}(\mathbf{Y};\mathbf{T})-\beta I(\mathbf{X};\mathbf{T})\text{,}$
where $\beta$ is a hyper-parameter that balances both terms. Specifically, the
information bottleneck can be seen as a rationale mask applied to the input,
i.e. $T=\mathbf{m}$. The approach from Bang et al. (2021) leverages the
Gumbel-Softmax trick to sub-sample $k$ tokens as the rationale, while Jiang et
al. (2021)’s approach predicts the probability of being the rationale for each
feature individually and obtain the rationale mask by rounding the
probability.

###  9.3. Evaluation

Evaluation regimes to evaluate rationale-based models typically trade off task
performance and the interpretability achieved. The desirable objective for
these approaches is: a good IBD approach should provide a task-prediction
model that performs at least no worse, if not better, than its non-explainable
comparators, and delivers valuable rationales.

The quality of the rationales can be measured by the degree of their agreement
with the ground-truth rationales. Essentially, they try to answer the
interpretability question: To what degree do the rationales agree to what
humans consider as true reasons? Benchmarks like (DeYoung et al., 2020)
collect multiple human-annotated datasets in IR ranging from sentiment
analysis, and fact-checking to entailment prediction. Therefore, given the
human-annotated rationale data, one can also evaluate the rationales output by
the rationale-extractor by calculating their similarity to the human
annotations. The similarity metrics include but are not restricted to the
accuracy, precision, recall, and F1 score of the rationale selection.

Apart from correspondence with human reasoning, DeYoung et al. (2020) also
introduces C/S scores, two evaluation metrics that evaluate semantic
attribution of selected rationales: “comprehensiveness” and “sufficiency”. For
an arbitrary input $\mathbf{x}$ with its corresponding label $l$ on a fine-
tuned model $M$, the comprehensiveness of a rationale-selection mask
$\mathbf{m}$ is defined as the difference between the model prediction made
based on the whole input and on all-but-rationale tokens, i.e.:

(2)
$\mathtt{comprehensiveness}(M,\mathbf{m},(\mathbf{x},l)):=P(y=l|\mathbf{x};M)-P(y=l|\mathbf{x}\odot\bar{\mathbf{m}};M)\text{,}$

where $\bar{\mathbf{m}}$ is the inverse mask and $P(\cdot)$ indicates the
predicted probability. The sufficiency is defined as the difference between
the full-input prediction and the prediction based on the rationale-only
tokens, i.e.:

(3)
$\mathtt{sufficiency}(M,\mathbf{m},(\mathbf{x},l)):=P(y=l|\mathbf{x};M)-P(M(\mathbf{x}\odot\mathbf{m})=l)\text{.}$

Furthermore, Bang et al. (2021) evaluate their information-bottleneck model
with the fidelity of the rationales. They define the fidelity similar to the
sufficiency score introduced above, i.e., to answer how well does the
rationale-based prediction match the prediction on the full input?

##  10\. Limitations and Open Questions

In this section, we will discuss the limitations, challenges, and open
questions in the area of explainable information retrieval. We have reviewed
many interpretability methods and approaches that cover various aspects and
tasks in IR. However, there are many unanswered questions, use cases, and
scenarios that need further research. We feel that most interpretability
approaches have focussed on the functional aspect of the central IR tasks of
ranking items. There are, however, many more IR tasks that employ learning
systems. Similarly, an IR system has different stakeholders – most
prominently, the benefactor of the IR system is the user, but much of the work
has focused on the system developer as the most likely stakeholder. Finally,
most of the explanation methods have relied on feature attributions as the
dominant type of explanations. However, explanations can be in terms of
training instances, adversarial examples, rules, etc.

###  10.1. Limitations

There are multiple limitations and challenges in facilitating and developing
interpretable approaches for information retrieval tasks. For the common task
of document retrieval, we discussed early heard that we require listwise or
pairwise explanations instead of pointwise explanations.

####  10.1.1. Limiting Assumptions.

The underlying assumption for surrogate models is that a simple model can
locally approximate the behavior of a complex black-box ranker. However, the
ranked output from a complex retrieval model can involve multiple relevance
factors. While one document in the ranking might rely on term matching with
the query, another document in the same ranking might be deemed relevant by
the same ranking model due to the proximity of query terms in the document.
Therefore, rankings with multiple and sometimes conflicting relevance factors
for a single simple surrogate model might not be able to provide high
fidelity.

####  10.1.2. Disentangling Explanations.

Many of the feature attribution methods provide one explanation, but complex
machine learning models learn multiple features for the same behavior, which
are also difficult to disentangle. This problem is exaggerated when it is
coupled with the problem of correlated reasons. Specifically, many relevance
factors are known to be correlated. A document that exhibits high semantic
similarity with the query might also have a high term-level matching score. In
these cases, it is likely that the methods covered in this survey (for
example, probing approaches) will not be able to disentangle the effects of
the underlying relevant factors from each other.

###  10.2. Open Questions

Now we turn to some of the open questions in the area of explainable
information retrieval. We divide the questions into three main categories -
types of explanations, explaining user models, evaluation of explanations,
causal explanations and the utility of explanations.

####  10.2.1. Explanation by Example

As discussed before, most of the explanation methods have been limited to
explaining the feature space – words, sentences, or numerical features in
learning to rank tasks. Prominent among these are attribution methods and hard
masking techniques. However, data attribution or instance attributes and
methods have not been explored in the context of information retrieval tasks.
Current papers that deal with explanation by examples are limited to finding
adversarial examples of generated text documents that result in errors of
contextual rankers. Instance attribution methods attribute the rationale or
reason behind the prediction back to the input instances in the training data.
Examples of instance attribution methods include influence functions and data
poisoning methods. The interpretability question that instance attribution
method methods answer is which of the input instances in the training data is
responsible for training the model in a certain way to cause the following
decision.

For a document retrieval task, the interpretability question could be “which
of the queries in the training set affect a certain test query ?”. The output
of instance attribution tasks can result in isolating mislabelled training
instances, identifying dataset biases, and providing insights into query
representation of the complex encoders. Other types of explanations can be
generated explanations for the stakeholders who are end users. These
generative explanations can take the form of fully-fledged natural language
that is composed of extractive explanations, feature attributions, or even
instance attribution methods.

####  10.2.2. Explaining User Models

Personalized ranking models tailor the search result list according to a
user’s profile as inferred by the search engine. While it is useful, modern
personalization techniques cause growing anxiety in their users – “Why am I
seeing these search results? What does the search engine believe my interests
are?”

Search engines have recently come under increased scrutiny due to their
influence on people’s decision-making in critical scenarios such as health and
elections. Search personalization typically involves tailoring the ranking of
results for individual users based on models of their past preferences and
interests. Consequently, there is a growing concern in users due to the
possible negative effects of personalization that include bias (Hannak et al.,
2013; Mustafaraj et al., 2020), filter bubbles (Pariser, 2011; Haim et al.,
2018; Flaxman et al., 2016) and increased opacity of the ranking mechanism.
Modern personalization techniques are based on learning an effective
representation of a user by mining sensitive behavioral data like click-
throughs (Joachims, 2002), query logs (Carman et al., 2010) and topical
interests (Harvey et al., 2013) from social media. Given today’s landscape of
partisan news coupled with the fact that commercial search engines do not
highlight personalized results, there is a real need to help us better
understand what a search engine infers about its users. Specifically, an
interesting interpretability question to ask is _what does the search engine
perceive the user as when they issue a certain query?_ This manner of post-hoc
auditing of retrieval models can be useful in a variety of downstream bias
detection and validation applications.

####  10.2.3. Evaluation of Explanations

Evaluation of explanations is a general problem in the area of interpretable
machine learning. There is a large spectrum of evaluation methods, starting
from functionally grounded evaluations to Human-centered evaluation in the
wider domain of machine learning and natural language processing. However, in
information retrieval, most of the explanation evaluation techniques have
focused on functionally grounded evaluation. Approaches that we reviewed in
this paper propose and evaluate explanation methods by their fidelity,
validity, completeness, and human congruence. We refer to these methods as
intrinsic methods.

A deeper problem lies in the absence of ground truth for evaluating or
validating the accuracy of output explanations of post-hoc methods.
Unfortunately, this leads to a chicken-and-egg problem that is hard to fix –
_to evaluate an explanation, one needs a mechanism to generate or collect
ground truth, which in the first place is the objective of the
interpretability task_. If we indeed have a procedure to create ground-truth
explanations from a black-box model, that is, to determine what exactly the
model pays attention to, then we would have solved the problem. Note that this
is in stark contrast to standard ML tasks, where the ground-truth are indeed
the observed variables that are explicitly specified in the data. While
intrinsic methods in the absence of ground-truth explanations are reasonable
proxies, they still do not answer the utility question of explanations – that
is, to what extent do the explanations assist the end-user in performing a
given task. Examples of tasks depend upon the stakeholder. For a machine
learning expert, the task can be explanation-based model debugging, while for
an end-user the question would be why the machine learning model ranks an
obviously relevant document lower than an irrelevant document. Apart from
these open questions, we believe that there is ample opportunity for
explainable IR methods to many vertical search applications like medical
search (Hanauer, 2006), high-recall search (Chakraborty et al., 2021),
scholarly and historical search (Holzmann et al., 2017b, a; Holzmann and
Anand, 2016). Apart from specialized search application, explainable IR has
direct applications in knowledge intensive tasks that use an information
retrieval component like fact checking (Pratapa et al., 2020; Zhang et al.,
2021b; Fetahu et al., 2016), question answering (Roy and Anand, 2021), entity
addition (Singh et al., 2016a).

##  11\. Conclusion

We provided an extensive investigation into the state of ExIR research. We
fill a distinct gap in the IR literature to curate, organize, and synthesize
works relating to explainability of learning systems. Our analysis reveals
that while post-hoc interpretability was heavily researched in the initial
years, current efforts are trying to propose approaches that are interpretable
by design (IBD). Due to a variety of design choices in IBD models, we find
that authors are often vague about the extent and style of interpretability in
their IBD approaches. We explored the feature-attribution, free-text
generation, and adversarial examples for post-hoc interpretability. Moreover,
we summarize methods that make use of well-established IR principles to
explain and probe ranking models. Finally, we explored the two major subtypes
of IBD methods for IR tasks. Based on our findings, we reflect on the design
trade-offs and experimental protocols that are used in evaluating ExIR
approaches. In the end, we present some limitations and open questions that we
foresee as the next steps toward building transparent, trustworthy search
systems.

###### Acknowledgements.

We acknowledge all the helpful comments from the anonymous reviewers, and
funding from DFG AN 996/1-1.

## References

  * (1)
  * Adebayo et al. (2018) Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. 2018.  Sanity Checks for Saliency Maps. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 9525–9536.  https://proceedings.neurips.cc/paper/2018/hash/294a8ed24b1ad22ec2e7efea049b8737-Abstract.html
  * Adebayo et al. (2022) Julius Adebayo, Michael Muelly, Harold Abelson, and Been Kim. 2022\.  Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.  https://openreview.net/forum?id=xNOVfCCvDpM
  * Amati and Van Rijsbergen (2002) Gianni Amati and Cornelis Joost Van Rijsbergen. 2002\.  Probabilistic Models of Information Retrieval Based on Measuring the Divergence from Randomness.  _ACM Trans. Inf. Syst._ 20, 4 (2002), 357–389.  https://doi.org/10.1145/582415.582416
  * Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ 10, 7 (2015), e0130140.
  * Bang et al. (2021) Seojin Bang, Pengtao Xie, Heewook Lee, Wei Wu, and Eric Xing. 2021.  Explaining A Black-box By Using A Deep Variational Information Bottleneck Approach.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 13 (2021), 11396–11404.  https://doi.org/10.1609/aaai.v35i13.17358
  * Bastings et al. (2019) Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019.  Interpretable Neural Predictions with Differentiable Binary Variables. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 2963–2977.  https://doi.org/10.18653/v1/P19-1284
  * Bastings et al. (2021) Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. 2021\.  ”Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification.  _ArXiv preprint_ abs/2111.07367 (2021).  https://arxiv.org/abs/2111.07367
  * Bastings and Filippova (2020) Jasmijn Bastings and Katja Filippova. 2020.  The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?. In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_. Association for Computational Linguistics, Online, 149–155.  https://doi.org/10.18653/v1/2020.blackboxnlp-1.14
  * Belinkov (2022) Yonatan Belinkov. 2022\.  Probing Classifiers: Promises, Shortcomings, and Advances.  _Comput. Linguistics_ 48, 1 (2022), 207–219.  https://doi.org/10.1162/coli_a_00422
  * Berger and Lafferty (1999) Adam Berger and John Lafferty. 1999.  Information retrieval as statistical translation. In _Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval_. 222–229.
  * Bhatt et al. (2020) Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. 2020.  Explainable Machine Learning in Deployment. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ (Barcelona, Spain) _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 648–657.  https://doi.org/10.1145/3351095.3375624
  * Bibal et al. (2022) Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François, and Patrick Watrin. 2022.  Is Attention Explanation? An Introduction to the Debate. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 3889–3900.  https://doi.org/10.18653/v1/2022.acl-long.269
  * Bigdeli et al. (2022) Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Morteza Zihayat, and Ebrahim Bagheri. 2022\.  Gender Fairness in Information Retrieval Systems. In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022_. ACM, 3436–3439.  https://doi.org/10.1145/3477495.3532680
  * Bondarenko et al. (2022) Alexander Bondarenko, Maik Fröbe, Jan Heinrich Reimer, Benno Stein, Michael Völske, and Matthias Hagen. 2022.  Axiomatic Retrieval Experimentation with ir_axioms. In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022_. ACM, 3131–3140.  https://doi.org/10.1145/3477495.3531743
  * Boytsov and Kolter (2021) Leonid Boytsov and Zico Kolter. 2021.  Exploring classic and neural lexical translation models for information retrieval: Interpretability, effectiveness, and efficiency benefits. In _European Conference on Information Retrieval_. Springer, 63–78.
  * Bruza and Huibers (1994) Peter Bruza and Theo W. C. Huibers. 1994.  Investigating Aboutness Axioms using Information Fields. In _Proceedings of SIGIR Forum 1994_. 112–121.
  * Cai et al. (2020) Jie Cai, Zhengzhou Zhu, Ping Nie, and Qian Liu. 2020\.  A Pairwise Probe for Understanding BERT Fine-Tuning on Machine Reading Comprehension. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (Eds.). ACM, 1665–1668.  https://doi.org/10.1145/3397271.3401195
  * Callan et al. (2009) Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. 2009\.  Clueweb09 data set.  http://lemurproject.org/clueweb09/
  * Câmara and Hauff (2020) Arthur Câmara and Claudia Hauff. 2020.  Diagnosing BERT with Retrieval Heuristics. In _Proceedings of ECIR 2020_ , Vol. 12035. Springer, 605–618.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-SNLI: Natural Language Inference with Natural Language Explanations. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_. 9560–9572.  https://proceedings.neurips.cc/paper/2018/hash/4c7a167bb329bd92580a99ce422d6fa6-Abstract.html
  * Carman et al. (2010) Mark James Carman, Fabio Crestani, Morgan Harvey, and Mark Baillie. 2010.  Towards query log based personalization using topic models. In _Proceedings of the 19th ACM Conference on Information and Knowledge Management, CIKM 2010, Toronto, Ontario, Canada, October 26-30, 2010_. ACM, 1849–1852.  https://doi.org/10.1145/1871437.1871745
  * Chakraborty et al. (2021) Manajit Chakraborty, David Zimmermann, and Fabio Crestani. 2021\.  PatentQuest: A User-Oriented Tool for Integrated Patent Search. In _Proceedings of the 11th International Workshop on Bibliometric-enhanced Information Retrieval co-located with 43rd European Conference on Information Retrieval (ECIR 2021), Lucca, Italy (online only), April 1st, 2021_ _(CEUR Workshop Proceedings, Vol. 2847)_. CEUR-WS.org, 89–101.  http://ceur-ws.org/Vol-2847/paper-09.pdf
  * Chatterjee et al. (2019) Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. 2019.  Semeval-2019 task 3: Emocontext contextual emotion detection in text. In _Proceedings of the 13th international workshop on semantic evaluation_. 39–48.
  * Chen et al. (2022) Jia Chen, Yiqun Liu, Yan Fang, Jiaxin Mao, Hui Fang, Shenghao Yang, Xiaohui Xie, Min Zhang, and Shaoping Ma. 2022.  Axiomatically Regularized Pre-training for Ad hoc Search. In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022_. ACM, 1524–1534.  https://doi.org/10.1145/3477495.3531943
  * Chen et al. (2018) Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. 2018.  Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , Jennifer G. Dy and Andreas Krause (Eds.), Vol. 80. PMLR.  http://proceedings.mlr.press/v80/chen18j.html
  * Cheng and Fang (2020) Zitong Cheng and Hui Fang. 2020.  Utilizing Axiomatic Perturbations to Guide Neural Ranking Models. In _ICTIR ’20: The 2020 ACM SIGIR International Conference on the Theory of Information Retrieval, Virtual Event, Norway, September 14-17, 2020_. ACM, 153–156.  https://doi.org/10.1145/3409256.3409828
  * Choi et al. (2022) Jaekeol Choi, Euna Jung, Sungjun Lim, and Wonjong Rhee. 2022\.  Finding Inverse Document Frequency Information in BERT.  _ArXiv preprint_ abs/2202.12191 (2022).  https://arxiv.org/abs/2202.12191
  * Craswell et al. (2021) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021.  Overview of the TREC 2020 deep learning track.  _CoRR_ abs/2102.07662 (2021).  arXiv:2102.07662 https://arxiv.org/abs/2102.07662
  * Cummins and O’Riordan (2007) Ronan Cummins and Colm O’Riordan. 2007.  An Axiomatic Comparison of Learned Term-Weighting Schemes in Information Retrieval: Clarifications and Extensions.  _Artif. Intell. Rev._ 28, 1 (2007), 51–68.
  * Dai et al. (2018) Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018\.  Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018_. ACM, 126–134.  https://doi.org/10.1145/3159652.3159659
  * DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020.  ERASER: A Benchmark to Evaluate Rationalized NLP Models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 4443–4458.  https://doi.org/10.18653/v1/2020.acl-main.408
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards A Rigorous Science of Interpretable Machine Learning.  https://doi.org/10.48550/ARXIV.1702.08608
  * Ebrahimi et al. (2018) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018\.  HotFlip: White-Box Adversarial Examples for Text Classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Melbourne, Australia, 31–36.  https://doi.org/10.18653/v1/P18-2006
  * Elazar et al. (2021) Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021.  Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals.  _Transactions of the Association for Computational Linguistics_ 9 (2021), 160–175.  https://doi.org/10.1162/tacl_a_00359
  * Fan et al. (2021) Yixing Fan, Jiafeng Guo, Xinyu Ma, Ruqing Zhang, Yanyan Lan, and Xueqi Cheng. 2021\.  A Linguistic Study on Relevance Modeling in Information Retrieval. In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_. ACM / IW3C2, 1053–1064.  https://doi.org/10.1145/3442381.3450009
  * Fang et al. (2004) Hui Fang, Tao Tao, and ChengXiang Zhai. 2004.  A Formal Study of Information Retrieval Heuristics. In _Proceedings of SIGIR 2004_. 49–56.
  * Fang et al. (2011) Hui Fang, Tao Tao, and ChengXiang Zhai. 2011.  Diagnostic Evaluation of Information Retrieval Models.  _ACM Trans. Inf. Syst._ 29, 2 (2011), 7:1–7:42.
  * Fang and Zhai (2006) Hui Fang and ChengXiang Zhai. 2006.  Semantic Term Matching in Axiomatic Approaches to Information Retrieval. In _Proceedings of SIGIR 2006_. 115–122.
  * Fernando et al. (2019) Zeon Trevor Fernando, Jaspreet Singh, and Avishek Anand. 2019\.  A study on the Interpretability of Neural Retrieval Models using DeepSHAP. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_. ACM, 1005–1008.  https://doi.org/10.1145/3331184.3331312
  * Fetahu et al. (2016) Besnik Fetahu, Katja Markert, Wolfgang Nejdl, and Avishek Anand. 2016.  Finding news citations for wikipedia. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_. 337–346.  https://doi.org/10.1145/2983323.2983808
  * Flaxman et al. (2016) Seth Flaxman, Sharad Goel, and Justin M Rao. 2016\.  Filter bubbles, echo chambers, and online news consumption.  _Public opinion quarterly_ 80, S1 (2016), 298–320.
  * Formal et al. (2021) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021\.  A White Box Analysis of ColBERT. In _Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II_ _(Lecture Notes in Computer Science, Vol. 12657)_. Springer, 257–263.  https://doi.org/10.1007/978-3-030-72240-1_23
  * Formal et al. (2022) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2022\.  Match Your Words! A Study of Lexical Matching in Neural Information Retrieval. In _Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part II_ _(Lecture Notes in Computer Science, Vol. 13186)_. Springer, 120–127.  https://doi.org/10.1007/978-3-030-99739-7_14
  * Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020.  Shortcut learning in deep neural networks.  _Nat. Mach. Intell._ 2, 11 (2020), 665–673.  https://doi.org/10.1038/s42256-020-00257-z
  * Goren et al. (2020) Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2020.  Ranking-Incentivized Quality Preserving Content Modification. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_. ACM, 259–268.  https://doi.org/10.1145/3397271.3401058
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016\.  A Deep Relevance Matching Model for Ad-hoc Retrieval. In _Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016_. ACM, 55–64.  https://doi.org/10.1145/2983323.2983769
  * Hagen et al. (2016) Matthias Hagen, Michael Völske, Steve Göring, and Benno Stein. 2016.  Axiomatic Result Re-Ranking. In _Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016_. ACM, 721–730.  https://doi.org/10.1145/2983323.2983704
  * Haim et al. (2018) Mario Haim, Andreas Graefe, and Hans-Bernd Brosius. 2018\.  Burst of the filter bubble? Effects of personalization on the diversity of Google News.  _Digital Journalism_ 6, 3 (2018), 330–343.
  * Hanauer (2006) David A Hanauer. 2006\.  EMERSE: the electronic medical record search engine. In _AMIA annual symposium proceedings_ , Vol. 2006. American Medical Informatics Association, 941\.
  * Hannak et al. (2013) Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013\.  Measuring personalization of web search. In _22nd International World Wide Web Conference, WWW ’13, Rio de Janeiro, Brazil, May 13-17, 2013_. International World Wide Web Conferences Steering Committee / ACM, 527–538.  https://doi.org/10.1145/2488388.2488435
  * Harvey et al. (2013) Morgan Harvey, Fabio Crestani, and Mark James Carman. 2013\.  Building user profiles from topic models for personalised search. In _22nd ACM International Conference on Information and Knowledge Management, CIKM’13, San Francisco, CA, USA, October 27 - November 1, 2013_. ACM, 2309–2314.  https://doi.org/10.1145/2505515.2505642
  * Hewitt and Liang (2019) John Hewitt and Percy Liang. 2019.  Designing and Interpreting Probes with Control Tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 2733–2743.  https://doi.org/10.18653/v1/D19-1275
  * Hofstätter et al. (2021) Sebastian Hofstätter, Bhaskar Mitra, Hamed Zamani, Nick Craswell, and Allan Hanbury. 2021\.  Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking. In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_. ACM, 1349–1358.  https://doi.org/10.1145/3404835.3462889
  * Hofstätter et al. (2020) Sebastian Hofstätter, Markus Zlabinger, and Allan Hanbury. 2020\.  Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking.  In _ECAI 2020_. IOS Press, 513–520.
  * Holzmann and Anand (2016) Helge Holzmann and Avishek Anand. 2016.  Tempas: Temporal archive search based on tags. In _Proceedings of the 25th International Conference Companion on World Wide Web_. 207–210.  https://doi.org/10.1145/2872518.2890555
  * Holzmann et al. (2017a) Helge Holzmann, Wolfgang Nejdl, and Avishek Anand. 2017a.  Exploring web archives through temporal anchor texts. In _Proceedings of the 2017 ACM on Web Science Conference_. 289–298.
  * Holzmann et al. (2017b) H. Holzmann, W. Nejdl, and A. Anand. 2017b.  Exploring web archives through temporal anchor texts. In _Proceedings of the 2017 ACM on Web Science Conference_. 289–298.  https://doi.org/10.1145/3091478.3091500
  * Hooker et al. (2019) Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019.  A Benchmark for Interpretability Methods in Deep Neural Networks. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_. 9734–9745.  https://proceedings.neurips.cc/paper/2019/hash/fe4b8556000d0f0cae99daa5c5c5a410-Abstract.html
  * Idahl et al. (2021) Maximilian Idahl, Lijun Lyu, Ujwal Gadiraju, and Avishek Anand. 2021.  Towards Benchmarking the Utility of Explanations for Model Debugging. In _Proceedings of the First Workshop on Trustworthy Natural Language Processing_. Association for Computational Linguistics, Online, 68–73.  https://doi.org/10.18653/v1/2021.trustnlp-1.8
  * Jacovi and Goldberg (2021) Alon Jacovi and Yoav Goldberg. 2021.  Aligning Faithful Interpretations with their Social Attribution.  _Transactions of the Association for Computational Linguistics_ 9 (2021), 294–310.  https://doi.org/10.1162/tacl_a_00367
  * Jiang et al. (2021) Zhiying Jiang, Raphael Tang, Ji Xin, and Jimmy Lin. 2021\.  How Does BERT Rerank Passages? An Attribution Analysis with Information Bottlenecks. In _Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 496–509.  https://doi.org/10.18653/v1/2021.blackboxnlp-1.39
  * Joachims (2002) Thorsten Joachims. 2002\.  Optimizing search engines using clickthrough data. In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 133–142.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_. ACM, 39–48.  https://doi.org/10.1145/3397271.3401075
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017.  Understanding Black-box Predictions via Influence Functions. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ _(Proceedings of Machine Learning Research, Vol. 70)_. PMLR, 1885–1894.  http://proceedings.mlr.press/v70/koh17a.html
  * Kumar and Talukdar (2020) Sawan Kumar and Partha Talukdar. 2020.  NILE : Natural Language Inference with Faithful Natural Language Explanations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 8730–8742.  https://doi.org/10.18653/v1/2020.acl-main.771
  * Lasri et al. (2022) Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022\.  Probing for the Usage of Grammatical Number. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_. Association for Computational Linguistics, 8818–8831.  https://doi.org/10.18653/v1/2022.acl-long.603
  * Lavrenko and Croft (2001) Victor Lavrenko and W. Bruce Croft. 2001.  Relevance Based Language Models. In _Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ (New Orleans, Louisiana, USA) _(SIGIR ’01)_. Association for Computing Machinery, New York, NY, USA, 120–127.  https://doi.org/10.1145/383952.383972
  * Lehman et al. (2019) Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019\.  Inferring Which Medical Treatments Work from Reports of Clinical Trials. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 3705–3717.  https://doi.org/10.18653/v1/N19-1371
  * Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016.  Rationalizing Neural Predictions. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, 107–117.  https://doi.org/10.18653/v1/D16-1011
  * Leonhardt et al. (2021) Jurek Leonhardt, Koustav Rudra, and Avishek Anand. 2021\.  Learnt Sparsity for Effective and Interpretable Document Ranking.  _ArXiv preprint_ abs/2106.12460 (2021).  https://arxiv.org/abs/2106.12460
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 7871–7880.  https://doi.org/10.18653/v1/2020.acl-main.703
  * Lin (2004) Chin-Yew Lin. 2004\.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.  https://aclanthology.org/W04-1013
  * Lin (2019) Jimmy Lin. 2019\.  The neural hype, justified!: a recantation.  _SIGIR Forum_ 53, 2 (2019), 88–93.  https://doi.org/10.1145/3458553.3458563
  * Liu et al. (2019) Hui Liu, Qingyu Yin, and William Yang Wang. 2019.  Towards Explainable NLP: A Generative Explanation Framework for Text Classification. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 5570–5581.  https://doi.org/10.18653/v1/P19-1560
  * Lucchese et al. (2022) Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, and Alberto Veneri. 2022\.  ILMART: Interpretable Ranking with Constrained LambdaMART. In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022_. ACM, 2255–2259.  https://doi.org/10.1145/3477495.3531840
  * Lundberg et al. (2018) Scott M. Lundberg, Gabriel G. Erion, and Su-In Lee. 2018\.  Consistent Individualized Feature Attribution for Tree Ensembles.  _ArXiv preprint_ abs/1802.03888 (2018).  https://arxiv.org/abs/1802.03888
  * Lundberg and Lee (2017) Scott M. Lundberg and Su-In Lee. 2017.  A Unified Approach to Interpreting Model Predictions. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_. 4765–4774.  https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html
  * MacAvaney et al. (2020) Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, and Arman Cohan. 2020.  ABNIRML: Analyzing the Behavior of Neural IR Models.  _ArXiv preprint_ abs/2011.00696 (2020).  https://arxiv.org/abs/2011.00696
  * Madsen et al. (2021) Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. 2021.  Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining.  _ArXiv preprint_ abs/2110.08412 (2021).  https://arxiv.org/abs/2110.08412
  * McDonald et al. (2018) Ryan McDonald, George Brokos, and Ion Androutsopoulos. 2018\.  Deep Relevance Ranking Using Enhanced Document-Query Interactions. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 1849–1860.  https://doi.org/10.18653/v1/D18-1211
  * Miller (2019) Tim Miller. 2019\.  Explanation in artificial intelligence: Insights from the social sciences.  _Artif. Intell._ 267 (2019), 1–38.  https://doi.org/10.1016/j.artint.2018.07.007
  * Molnar (2022) Christoph Molnar. 2022\.  _Interpretable Machine Learning_ (2 ed.).  https://christophm.github.io/interpretable-ml-book
  * Mowshowitz and Kawaguchi (2005) Abbe Mowshowitz and Akira Kawaguchi. 2005.  Measuring search engine bias.  _Inf. Process. Manag._ 41, 5 (2005), 1193–1205.  https://doi.org/10.1016/j.ipm.2004.05.005