# Quantifying the Reliance of Black-Box Decision-Makers on Variables of
Interest

Daniel Vebman
Cornell University
<EMAIL_ADDRESS>
(May 2023)

## Abstract

This paper introduces a framework for measuring how much black-box decision-
makers rely on variables of interest. The framework adapts a permutation-based
measure of variable importance from the explainable machine learning
literature. With an emphasis on applicability, I present some of the
framework’s theoretical and computational properties, explain how reliance
computations have policy implications, and work through an illustrative
example. In the empirical application to interruptions by Supreme Court
Justices during oral argument, I find that the effect of gender is more muted
compared to the existing literature’s estimate; I then use this paper’s
framework to compare Justices’ reliance on gender and alignment to their
reliance on experience, which are incomparable using regression coefficients.

Keywords: econometrics, interpretability, fairness, explainable machine
learning, gender.

##  1 Introduction

Decision-makers routinely choose among some menu of options without explaining
why. We are interested in understanding which observed variables are important
to the decision-maker: does the judge rely on sex, being more lenient towards
women [Goulette et al., 2015]? Does the doctor rely on race, assuming that
Black people have higher pain tolerance [Staton et al., 2007]? Or, as the
Supreme Court recently decided, do (Harvard) admissions officers discriminate
against Asians [SFFA v. Harvard, 2022]? Understanding a decision-making
process has inherent value and direct policy implications. To correct an
iniquity, do judges need to be trained about their cognitive biases, do
medical textbooks need to be rewritten, or do admissions practices need to
change?

The difficulty is that the decision-maker is a black box, whom we cannot query
for new data points. Human brains are figuratively black boxes and literally
neural nets. Combining the CS literature on ML interpretability with
econometric work on the prediction of counterfactual choices—including in
partially identified settings—presents a novel way to understand and, if
needed, adjust how decision-makers make choices in the real world.

This paper introduces a framework for measuring how much black-box decision-
makers rely on variables of interest. The approach is inspired by a
permutation-based measure of variable importance from the explainable machine
learning literature; Breiman [2002] originally presented this method in his
study of random forests, and Fisher, Rudin, and Dominici [2019] generalized it
to arbitrary models. This paper’s contributions are:

  1. 1.

I generalize Fisher et al.’s approach beyond ML, to human behavior.

  2. 2.

I prove in proposition 6 that this framework encapsulates and can test for
conditional statistical parity, a fairness metric from the explainable machine
learning literature.

  3. 3.

Propositions 6 and 10 theoretically justify Fisher et al.’s normalization by a
baseline.

  4. 4.

Studying interruptions by Supreme Court Justices in section 5, I find smaller
effects than the existing literature. I also rank each Justice’s reliance on
gender, alignment, and experience, which were previously not compared.

##  2 Related Work

###  2.1 Variable Importance Techniques from Economics and the Social Sciences

Importance analysis quantifies how much certain variables contribute to
uncertainty around the outcome [Coyle et al., 2003]. Correlation- or variance-
based measures are the most common importance measures. Simple correlation
coefficients and contributions to variance fall in this category.
Probability-, elasticity-, and information/entropy-based measures are also
common in the economics and social sciences literatures [Coyle et al., 2003].

Variable importance is distinct from and offers advantages over typical
alternatives. First, consider a linear model. Different units prevent direct
comparisons of coefficients to each other. For example, as explored in the
empirical application in section 5, consider a model for the number of times a
Supreme Court Justice interrupts an advocate during oral argument,
$InterruptionRate=\beta_{0}+\beta_{1}Gender+\beta_{2}Experience+\beta_{3}JusticeAndAdvocateAgree$.
We can directly compare $\beta_{1}$ to $\beta_{3}$ since the covariates they
multiply are both binary. It is nonsensical, however, to compare a marginal
increase in $Experience$ to a change in $Gender$ from male to female – the
units are completely different. The variable-importance framework presented
here overcomes this limitation and allows direct comparisons of any two
variables.

A second advantage of variable importance is that it is sensitive to the
relationship between the different inputs. As illustrated in the school
admissions example in section 4, the coefficients on the model do not alone
dictate how much a variable affects the model’s output. More generally, the
distinction between statistical and economic significance suggests an
important analogue to variable importance for decision-makers: even though a
variable might have a large effect on a hypothetical input to the model, it
might only have limited importance for a typical input, in practice.

###  2.2 Variable Importance Techniques from Explainable Machine Learning

The ML explainability literature studies post-hoc methods to understand why a
model arrives at its outputs [Barredo Arrieta et al., 2020]. It includes the
general class of variable importance (or feature relevance) measures, which
quantify how much a prediction model’s accuracy depends on the information in
each covariate. This class contains Shapley values [e.g., Lundberg and Lee,
2017], perturbation-based measures [e.g., Robnik-Šikonja and Bohanec, 2018],
and permutation-based measures [e.g., Breiman, 2001]. Perturbation-based
measures observe the change in a loss function for a small or infinitesimal
change to the inputs. Permutation-based measures observe the change in loss
after shuffling independent observations together to sever the correlation
between the variable of interest and the other covariates/label.

Permutation-based measures are well-suited to black-box models because they
only require a mapping from inputs to outputs. They do not need to rely on
gradients or any other knowledge of the model. Indeed, model-agnostic
explanations are increasingly popular because they are widely applicable
[Balagopalan et al., 2022]. Proposed by Marco Tulio et al. [2016], Local
Interpretable Model-agnostic Explanations (LIME) and its variations are among
the most well-known such approaches. LIME is a local technique, which explains
how the model made a single decision by approximating a small region of the
decision boundary. In contrast, global methods approximate the entire function
with a more interpretable surrogate (like a tree-based or sparse linear model)
or otherwise summarize the entire decision model.

This paper centers on a permutation-based measure of variable importance.
Breiman [2001, 2002] first developed this approach in his study of random
forests. Gregorutti et al. [2017] use this method in a variable-selection
algorithm, specifically for random forests that model the conditional
expectation function. Fisher et al. [2019] suggest this permutation-based
approach as a generic measure of variable importance for any model and with
arbitrary loss function; they further study how to bound this measure for sets
of models that perform roughly equally well on the same prediction problem.

Like Gregorutti et al., I study the conditional expectation function only;
however, whereas they study random forests and typically use the square loss,
I generalize to any estimator and arbitrary loss. I use almost the same
generic definition of reliance as Fisher et al.; but, unlike them, I study the
conditional expectation function in the context of human decision-making and
discuss how reliance values could be used in various public-policy settings.

##  3 Framework

###  3.1 Setup

A decision-maker chooses an alternative $Y\in\mathcal{Y}$ depending on
covariates, which we partition into $X_{1}\in\mathcal{X}_{1}$ and
$X_{2}\in\mathcal{X}_{2}$. In this part, we define a formal measure of the
decision-maker’s reliance on $X_{1}$.

We define reliance by rephrasing our question as a prediction problem: the
decision rule induces a joint distribution $(Y,X_{1},X_{2})\sim P$ on the
choice and covariates. We have an oracle model $\mathbb{E}[Y|X_{1},X_{2}]$,
where $(Y,X_{1},X_{2})\sim P$, which tells us the decision-maker’s choice. The
oracle allows us to make counterfactual queries to the choice function.

By definition, the oracle assumes that its inputs and outputs are distributed
according to $P$. To measure the oracle’s reliance on $X_{1}$, we therefore
observe how much the oracle errs when we replace $X_{1}$ with noise.
Specifically, we make $X_{1}$ completely uninformative of $Y$ and $X_{2}$,
while preserving their marginal distributions. We make this intuitive notion
precise in the next section.

###  3.2 Definition of Reliance

Let $(Y^{a},X_{1}^{a},X_{2}^{a})$ and $(Y^{b},X_{1}^{b},X_{2}^{b})$ be two
independent draws from $P$. Splice $X_{1}^{b}$ into the $a$ draw to create the
coupling $(Y^{a},X_{1}^{b},X_{2}^{a})$. $X_{1}^{b}$ and $(Y^{a},X_{2}^{a})$
have the same marginal distributions as before, but they are now independent.
We want to compare $Y^{a}$ to the oracle’s prediction for the pair
$(X_{1}^{b},X_{2}^{a})$.

######  Definition 1 (Oracle’s prediction).

The oracle’s prediction for the pair
$(x_{1},x_{2})\in\mathcal{X}_{1}\times\mathcal{X}_{2}$ is

$f(x_{1},x_{2})=\mathbb{E}[Y\mid X_{1}=x_{1},X_{2}=x_{2}],$

where $(Y,X_{1},X_{2})\sim P$ as before.

We want to measure how much the oracle errs when we feed it an $X_{1}$ that is
completely uninformative of $Y$. To quantify this change, we need a loss
function $L$. We also require the following technical assumption:

######  Assumption 2.

Assume that the coupling $P_{X_{1}^{b},X_{2}^{a}}$ is absolutely continuous
with respect to $P_{X_{1}^{a},X_{2}^{a}}$. Recall
$P_{X_{1}^{b},X_{2}^{a}}=P_{X_{1}}P_{X_{2}}$ and
$P_{X_{1}^{a},X_{2}^{a}}=P_{X_{1},X_{2}}$. The subscripts on $P$ refer to the
respective marginal distributions.

######  Remark.

This assumption is necessary so that the oracle’s prediction is well-defined
over the shuffling. For example, if $X_{1}$ and $X_{2}$ are binary but are
never equal, then this assumption fails. When we try to shuffle $X_{1}$, we
will fail to compute $\mathbb{E}[Y\mid X_{1}=0,X_{2}=0]$ since the conditioned
event has probability 0.

We can now formally define model reliance:

######  Definition 3 (Reliance on $X_{1}$).

Given a loss function $L:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}$,
and a partition $(X_{1},X_{2})$ of the covariates, the reliance on $X_{1}$ is

$r=\mathbb{E}_{Y^{a},X_{1}^{b},X_{2}^{a}}L(Y^{a},\,f(X_{1}^{b},X_{2}^{a}))$

where $(Y^{a},X_{2}^{a})\sim P_{Y,X_{2}}$ and $X_{1}^{b}\sim P_{X_{1}}$ are
independent.

In general, we might require that the loss function $L$ admit some or all of
the following kinds of statements:

  1. 1.

Rankings of variables within one distribution: The Justice relies more on
gender than on experience.

  2. 2.

Rankings of one variable across distributions: Justice P relies more on gender
than Justice Q does.

  3. 3.

Rankings of different variables across distributions: Justice P relies more on
gender than Justice Q relies on experience.

The theoretical exposition will justify that these three statements are
sensical, and the applications will demonstrate that they are natural and
valuable. We will make all these kinds of statements in the application to
Supreme Court Justices in section 5. First, I provide a few examples of loss
functions and explain their properties.

######  Example 4 (Square Loss).

A simple choice for $L$ is the square loss,

$L(y,\hat{y})=(y-\hat{y})^{2}.$

That is,

$r=\mathbb{E}_{Y^{a},X_{1}^{b},X_{2}^{a}}(Y^{a}-f(X_{1}^{b},X_{2}^{a}))^{2}.$

We will show in the next proposition that, by using the square loss, we can
interpret reliance values with respect to a baseline. Further, the reliance
and the baseline are equal if and only if the decision $Y$ is conditionally
mean independent of $X_{1}$ given $X_{2}$. Thus, this reliance measure can
test conditional statistical parity, a fairness metric from the machine
learning literature.

######  Assumption 5.

Let $(\mathcal{Y}\times\mathcal{X}_{1}\times\mathcal{X}_{2},\mathcal{F},P)$ be
a probability space. In what follows, assume the random vector
$(Y,X_{1},X_{2})$ on this space is $L^{2}$. As before, $(Y,X_{1},X_{2})$,
$(Y^{a},X_{1}^{a},X_{2}^{a})$ and $(Y^{b},X_{1}^{b},X_{2}^{b})$ are
independent and identically distributed.

######  Proposition 6.

To simplify notation, let $f(x_{2})=\mathbb{E}[Y\mid X_{2}=x_{2}]$, where
$(Y,X_{2})\sim P_{Y,X_{2}}$. Define the baseline reliance,

$b=\mathbb{E}_{Y^{a},X_{1}^{a},X_{2}^{a}}(Y^{a}-f(X_{1}^{a},X_{2}^{a}))^{2},$

where we do not shuffle $X_{1}$. Then, $r\geq b$. Furthermore, the following
are equivalent:

  1. 1.

$r=b$;

  2. 2.

$f(X_{1}^{a},X_{2}^{a})=f(X_{1}^{b},X_{2}^{a})$ a.s. with respect to
$(X_{1}^{a},X_{2}^{a},X_{1}^{b})\sim P_{X_{1},X_{2}}P_{X_{1}}$;

  3. 3.

$f(X_{1}^{a},X_{2}^{a})=f(X_{2}^{a})$ a.s. with respect to
$(X_{1}^{a},X_{2}^{a})\sim P_{X_{1},X_{2}}$; and

  4. 4.

$Y$ is conditionally mean independent of $X_{1}$ given $X_{2}$ almost surely.

###### Proof.

Proof in appendix.∎

######  Remark.

We prove the chain 1 $\Leftrightarrow$ 2 $\Leftrightarrow$ 3, and 4 simply
rephrases 3. 2 $\Leftrightarrow$ 3 is not trivial because the conditions hold
almost surely over related but different distributions.

Because $r\geq b$ always, we justifiably call $b$ the baseline reliance of $Y$
on $X_{1}$. We can interpret $b$ as the intrinsic noise of $P$ — it is the
loss that even the best predictor incurs. Fisher et al. [2019] broadly define
their reliance measure as the ratio $r/b$, but for arbitrary loss function.
However, it is generally false that $b$ minimizes the loss for arbitrary $L$,
which challenges $b$’s use as a baseline; furthermore, as demonstrated in
section 3.3, this definition unnecessarily complicates comparisons within the
same distribution. We return to the idea of partialling out the intrinsic
noise in example 7 and in section 3.3 on comparing reliance across multiple
choice distributions.

This proposition also provides strong intuition for how to interpret reliance
values: in the context of binary decisions, conditional mean independence is
equivalent to conditional statistical parity, a fairness metric from the
explainable machine learning literature that captures “fairness through
blindness” [see Corbett-Davies et al., 2017]. In the fairness setting, $X_{1}$
are the sensitive attributes and $X_{2}$ are the legitimate attributes. Fisher
et al. [2019, Section 10.2] only identify that reliance is “related” to this
fairness metric. Contributing to the explainable machine learning literature,
this proposition proves that $r=b$ is equivalent to and therefore can test for
conditional statistical parity.

######  Example 7 (Cross-Entropy Loss).

In a typical binary classification setting where the oracle returns
probabilities in the range $[0,1]$, it is unclear how to interpret the actual
values taken by the square loss. The cross-entropy loss is commonly used in
the machine learning literature in such settings:

$\displaystyle\mathbb{E}_{Y^{a},X_{1}^{b},X_{2}^{a}}[Y^{a}$
$\displaystyle\log\mathbb{E}[Y\mid X_{1}=X_{1}^{b},X_{2}=X_{2}^{a}]$
$\displaystyle+(1-Y^{a})\log(1-\mathbb{E}[Y\mid
X_{1}=X_{1}^{b},X_{2}=X_{2}^{a}])]$

This equals the expected cross-entropy
$H(Y\mid(X_{2}=X_{2}^{a}),\;Y\mid(X_{1}=X_{1}^{b},X_{2}=X_{2}^{a}))$. A neat
interpretation follows from the relation to the Kullback-Leibler divergence:

$\displaystyle
H\left(Y\mid(X_{2}=X_{2}^{a}),\;Y\mid(X_{1}=X_{1}^{b},X_{2}=X_{2}^{a})\right)$
$\displaystyle=H(Y\mid(X_{2}=X_{2}^{a}))$
$\displaystyle\qquad+D_{KL}\left(Y\mid(X_{2}=X_{2}^{a})\mid\mid
Y\mid(X_{1}=X_{1}^{b},X_{2}=X_{2}^{a})\right)$ (cross-entropy = entropy +
divergence)

The Kraft-McMillan theorem establishes the optimal number of bits to code
messages that follow a specific distribution. Returning to our setting, this
theorem implies the following interpretations:

  1. 1.

The entropy term equals the optimal expected number of bits needed to code a
draw from $Y\mid(X_{2}=X_{2}^{a})$.

  2. 2.

The cross-entropy term equals the optimal expected number of bits needed to
code a draw from $Y\mid(X_{2}=X_{2}^{a})$ if we wrongly assume that our draws
come from $Y\mid(X_{1}=X_{1}^{b},X_{2}=X_{2}^{a})$.

  3. 3.

The divergence term equals the excess number of bits we need to code a draw
from $Y\mid(X_{2}=X_{2}^{a})$ if we mistakenly assume that it is drawn from
$Y\mid(X_{1}=X_{1}^{b},X_{2}=X_{2}^{a})$.

It a mistake to assume that the data are drawn from
$Y\mid(X_{1}=X_{1}^{b},X_{2}=X_{2}^{a})$ because $X_{1}^{b}\perp X_{2}^{a}$.
Reliance measures the cost of this mistake.

######  Example 8 (Context-Specific Loss).

However, for public-policy questions, it may be difficult to translate such an
information-based interpretation into the language of policy. More broadly,
how does one evaluate the loss once it is computed? For example, consider the
scenario where judges decide if to detain ($y=1$) or release ($y=0$) a
defendant pretrial based on some covariates $x$. Suppose we assert that all
judges should maximize the utility function

$u(y,x)=-yP(S=0\mid X=x)-\lambda(1-y)P(S=1\mid X=x)$

where $S=1$ indicates the event that the defendant with characteristics $X=x$
would skip trial if released, and $S=0$ means that they would not skip if
released. $\lambda$ is a policy-preference weight. According to this $u$,
judges should minimize the cost of a mistake. Next, adjust our reliance
measure to be

$r=\mathbb{E}_{Y^{a},X_{1}^{a},X_{2}^{a},X_{1}^{b}}L(Y^{a},\mathbb{E}[Y\mid
X_{1}=X_{1}^{b},X_{2}=X_{2}^{a}];\;(X_{1}^{a},X_{2}^{a}))$

where

$L(y,\hat{y};x)=u(y,x)-u(\hat{y},x).$

This captures how much the judge relies on $X_{1}$ to attain his solution to
the maximum-utility problem. If $r>0$, then the judge uses information in
$X_{1}$ to increase utility, i.e., minimize mistakes. If $r<0$, then the
judge’s reliance on $X_{1}$ lowers utility, i.e., causes more mistakes.

###  3.3 Comparing Reliance

As defined, we can already compare a decision-maker’s reliance on a variable
$X_{1}$ to their reliance on another variable $X_{1}^{\prime}$ because the
expected values are taken over the same distribution. However, two decision-
makers impose distinct joint distributions over their choices and the
covariates. For example in section 5, which investigates interruptions by
Supreme Court Justices during oral argument, each Justice is a separate
decision-maker.

We can compute a normalized measure of reliance across multiple distributions
by joining all of the decision-makers’ distributions. In particular, consider
$n$ decision-makers. For each $1\leq i\leq n$, we have a choice $Y_{i}$, a
partition of the covariates $(X_{1i},X_{2i})$, and a joint distribution
$P_{i}$ over choices and covariates. Furthermore, let $\mathcal{P}$ denote the
(arbitrary) coupling of all the $P_{i}$’s. In the example of multiple Supreme
Court Justices interrupting the same advocate, the $P_{i}$’s are not
independent.

######  Definition 9 (Cross-Distribution Reliance).

Fix some loss function
$\mathcal{L}:(\mathcal{Y}_{1}\times\cdots\times\mathcal{Y}_{n})^{2}\rightarrow\mathbb{R}$,
and let
$f_{i}:\mathcal{X}_{1i}\times\mathcal{X}_{2i}\rightarrow\mathcal{Y}_{i}$ be
the prediction function for decision-maker $i$, as in definition 1. The cross-
distribution reliance of decision-maker $k$ on $X_{1k}$ is

$r_{k}^{\times}=\mathbb{E}\mathcal{L}\left((Y_{i}^{a})_{i=1}^{n},\;(\,f_{i}(s_{k}(X_{1i}^{a},X_{1i}^{b}),X_{2i}^{a})\,)_{i=1}^{n}\right),$

where the shuffle function $s_{k}(X_{1i}^{a},X_{1i}^{b})$ equals $X_{1i}^{b}$
if $i=k$ and $X_{1i}^{a}$ if $i\neq k$. The expectation is over the
independent draws $(Y_{i}^{a},X_{1i}^{a},X_{2i}^{a})_{i=1}^{n}$ and
$(Y_{i}^{b},X_{1i}^{b},X_{2i}^{b})_{i=1}^{n}$ from $\mathcal{P}$.

Observe that the cross-distribution reliance coincides with the original
definition of reliance if $n=1$. In effect, we have stacked all the decision-
makers into one super-decision-maker. A special case is when $\mathcal{L}$ is
additively separable with respect to $i$:

######  Proposition 10 (Equivalent Ranking).

Suppose the cross-distribution loss function $\mathcal{L}$ is additively
separable, i.e.,

$\mathcal{L}((y_{i})_{i=1}^{n},(\hat{y}_{i})_{i=1}^{n})=\sum_{i=1}^{n}L_{i}(y_{i},\hat{y}_{i}),$

where $L_{i}:\mathcal{Y}_{i}\times\mathcal{Y}_{i}\rightarrow\mathbb{R}$ for
each $i$. For each decision-maker $i$, define the baseline reliance:

$b_{i}=\mathbb{E}_{Y_{i}^{a},X_{1i}^{a},X_{2i}^{a}}L_{i}\left(Y_{i}^{a},\;f_{i}(X_{1i}^{a},X_{2i}^{a})\right),$

where the expectation is taken with respect to
$(Y_{i}^{a},X_{1i}^{a},X_{2i}^{a})\sim P_{i}$ for each $i$, as indicated.
Then,

$r_{j}^{\times}<r_{k}^{\times}\Leftrightarrow r_{j}-b_{j}<r_{k}-b_{k},$

where $r_{k}$ is the reliance on $X_{k}$ as in definition 3. That is, the
ranking by $r_{k}-b_{k}$ is equivalent to the ranking by $r_{k}^{\times}$.

###### Proof.

Proof in appendix.∎

######  Remark.

Letting $L_{i}$ be the cross-entropy as in example 7, the difference
$r_{k}-b_{k}$ equals the KL divergence. Thus, we normalize the reliance
measures by partialling out each distribution’s intrinsic noise.

Proposition 10 is an important result because $r_{k}-b_{k}$ is much simpler to
compute than $r_{k}^{\times}$. Furthermore, the equivalence rigorously
justifies Fisher et al.’s normalization by the un-shuffled expected loss. In
section 5, we apply this proposition to determine which Justices rely the most
on gender when interrupting advocates.

###  3.4 Acting on Reliance Values

Beyond providing insight into the decision-making process, reliance values are
an actionable metric for a wide range of real-world problems.

######  Example 11 (Enforcing Conditional Statistical Parity).

Consider for example the admissions decision setting, like in the lawsuit SFFA
v. Harvard, 2022. We might assert that the admissions decision should not
directly rely on race ($X_{1}$). As proved in proposition 6, we could test the
equivalent condition, $r=b$. If we reject the hypothesis, then we would have
evidence that the admissions decision relies on race. The benefit of this
metric over something like a coefficient on the race variable is that this
approach is agnostic to the specific modelling assumptions on the conditional
expectation function; therefore, it can better accommodate flexible ML methods
that excel in high-dimensional settings.

######  Example 12 (Preventing Manipulation).

Still in the admissions setting, we might want to ensure a ranking among some
subset of the covariates. For example, in order to prevent manipulation of the
admissions process, we might require that the admissions decision is less
sensitive to self-reported community service hours than to exam scores. In
this case, we would test $r(\text{community service hours})<r(\text{exam
score})$. An advantage of this framework is that it enables us to compare any
two variables, regardless of their units.

######  Example 13 (Idealized Baselines).

We might instead want to see how the admissions officer’s behavior compares to
an idealized decision rule that we cook up ourselves. For example, we might
calibrate a simple admissions rule to our own preferences and generate a
ranking of variables by their importance. We would then do the same for the
observed admissions data and check for deviations between the two rankings. We
could also apply proposition 10 to directly verify that the admissions officer
relies less on a sensitive variable like race than the idealized rule does.

###  3.5 Alternative Formulations

I briefly mention two alternatives to the formulation of reliance in
definition 3. These alternatives share a similar motivation to the original
definition, but their definition of ‘noise’ differs.

First, in our motivation for reliance, we made $X_{1}$ noise by making it
completely uninformative of $(Y,X_{2})$. That is, we shuffled in
$X_{1}^{b}\perp(Y^{a},X_{2}^{a})$. We might instead assert that $X_{1}$ adds
no additional information on $Y$ given $X_{2}$. In other words, we might only
need $X_{1}^{b}\perp Y^{a}\mid X_{2}^{a}$. This conditional reliance measure
instead takes one draw $(Y^{a},X_{1}^{a},X_{2}^{a})\sim P$ and picks
$X_{2}^{b}\sim P_{X_{2}^{b}\mid X_{1}^{b}=X_{2}^{a}}$. Fisher et al. [2019]
present this definition too in section 8.2.

Second, we might measure the worst-case reliance

$r=\sup_{x_{1}^{b}\in\mathcal{X}_{1}}\mathbb{E}L(Y^{a},f(x_{1}^{b},X_{2}^{a}))$

where the expectation is over $(Y^{a},X_{2}^{a})\sim P_{Y,X_{2}}$. The max
replaces $X_{1}$ with the noise that creates the largest loss. Compared to
these two alternatives, the one in definition 3 is much easier to compute.
Furthermore, the conditional reliance measure will suffer in small samples and
high-dimensional settings since it is less likely to find multiple
observations with the same $X_{2}$ to shuffle together.

###  3.6 Estimating Reliance

If we can estimate counterfactual queries to the oracle, then we can define a
plug-in estimator for $r$:

######  Definition 14 (Estimator for $r$).

Given an estimator
$\hat{f}:\mathcal{X}_{1}\times\mathcal{X}_{2}\rightarrow\mathcal{Y}$ for $f$,
define the estimator

$\hat{r}=\frac{1}{n(n-1)}\sum_{i=1}^{n}\sum_{j\neq
i}L(y_{i},\hat{f}(x_{1j},x_{2i}))$

over the data $\\{(y_{i},x_{1i},x_{2i})\\}_{i=1}^{n}$.

######  Remark.

Assuming that the loss has finite variance, $\hat{r}$ has a normal limiting
distribution with mean $\mathbb{E}[L(Y^{a},\hat{f}(X_{1}^{b},X_{2}^{a}))]$ by
the central limit theorem. $\hat{r}$’s distribution depends on that of
$\hat{f}$.

The double sum can make this object computationally expensive: if evaluating
$\hat{f}$ is $O(1)$, then directly computing this double sum is $O(n^{2})$.
However, if $X_{1}$ realizes few values, we can compute $\hat{r}$ much more
efficiently:

######  Proposition 15.

Let $C=\\{x_{1i}\\}_{i=1}^{n}$ be the distinct observed values that $X_{1}$
takes in the data. Then,

$\hat{r}=\frac{1}{n(n-1)}\sum_{i=1}^{n}\sum_{c\in
C}(n_{c}-1\\{x_{1i}=c\\})L(y_{i},\hat{f}(c,x_{2i})),$

where $n_{c}=|\\{i:x_{1i}=c\\}|$.

###### Proof.

Proof in appendix.∎

######  Remark.

We can implement this second formulation of $\hat{r}$ in $O(n|C|)$ time
complexity, assuming that evaluating $\hat{f}$ is $O(1)$. To do this,
precompute $n_{c}$ for each $c\in C$ by looping through the $x_{1i}$’s and
tracking counts in a hash table whose keys are $C$; this step is $O(n)$.

######  Remark.

As a corollary, if $X_{1}$ is binary, i.e., $C=\\{0,1\\}$, then computing
$\hat{r}$ using the formula in proposition 15 is only $O(n)$. In fact, it is
$O(n)$ as long as $X_{1}$ has finite support (though the hidden constant can
be quite large).

The oracle asks what the decision-maker would have done were he presented with
a specific vector of covariates. We rely on the oracle because we cannot
directly query the decision-maker, but this counterfactual inference incurs
the cost of additional assumptions and statistical uncertainty. In fact, this
object might only be partially identified: black-box decision-makers often
rely on private information, so $X_{2}$ may be only partially observed, or
some data might be systematically missing. If counterfactual queries are only
partially identified, then we may need to settle for bounds on $r$:

######  Proposition 16.

Suppose $\mathcal{Y}=[0,1]$, but $y_{i}\in\\{0,1\\}$ for all $i$, and assume
that $L(y,\hat{y})$ increases monotonically with $|y-\hat{y}|$. As a
normalization, assume $L(y,\hat{y})=0$ if $|y-\hat{y}|=0$. Then, given bounds
$\hat{f}_{min},\hat{f}_{max}$ on $\hat{f}$, we can obtain the conservative
bounds

$\hat{r}_{min}=\frac{1}{n(n-1)}\sum_{i=1}^{n}\sum_{j\neq
i}\min\\{L(y_{i},\hat{f}_{min}(x_{1j},x_{2i})),L(y_{i},\hat{f}_{max}(x_{1j},x_{2i}))\\}$

and $\hat{r}_{max}$ by replacing $\min$ with $\max$. That is,
$\hat{r}\in[\hat{r}_{min},\hat{r}_{max}]$.

###### Proof.

Proof in appendix.∎

##  4 Example: School Admissions

###  4.1 Setup

This worked example shows how this framework could be applied end-to-end: an
admissions officer decides if to admit a student depending on his or her race,
sex, and test score. The choice $Y\mid X$ is characterized by the decision
rule

$Y=1\\{-2X_{1}+X_{2}+X_{3}/5-2.2\geq 0\\}$

where $X_{1}\sim Bernoulli(.5)$ and $X_{2}\sim Bernoulli(.3)$. Also, we define
$X_{3}=A+E$, rounded to the nearest integer between 0 and 10, where innate
ability $A\sim N(6,1)$ and study effort $E\sim N(1,1)$.

Students self-report if they were admitted. However, they only respond to the
survey if $Z=1\mid X,E$, where

$Z=1\\{X_{1}+3X_{2}+X_{3}/8+E-3.5\geq 0\\}.$

The researchers want to estimate the admissions officer’s reliance on
$X_{1},X_{2}$ and $X_{3}$. Importantly, the researchers know the survey
response rate ($E[Z]$) and assume $Y\mid Z,X_{1},X_{2},X_{3}$ is roughly
logistically distributed; however, they do not make distributional assumptions
about the noise, nor do they know that, in fact, $Y\perp Z\mid
X_{1},X_{2},X_{3}$.

###  4.2 Identification

To compute these reliance values, we must estimate $P(Y=1\mid X)$, but, given
the researchers’ knowledge, there is confounding in the missing data problem
due to $E$. Therefore, this estimand is only partially identified. We only
observe $Y,X$ when $Z=1$ and we have $E[Z]$, so our identification result is

$E[Y\mid X]\in[E[Y\mid X,Z=1]P(Z=1),\;E[Y\mid X,Z=1]P(Z=1)+P(Z=0)].$ (1)

The interval is smaller when $P(Z=1)$ is larger, and the interval is just a
point when $P(Z=1)=1$.

######  Proposition 17.

The identification region in equation 1 is tight.

###### Proof.

Proof in appendix.∎

###  4.3 Estimation

In this simulation, $10,000$ students apply. 8,313 students respond to the
survey, and their acceptance rate is 13%.

Estimate $f_{1}(x)=E[Y\mid X=x,Z=1]$ by logistic regression of the observed
$Y$ on $X$; denote this estimator by $\hat{f}_{1}$. Even though $Y\mid X,Z=1$
isn’t necessarily logistic, over 99.8% of the sample is classified correctly,
and this approximation will suffice for the sake of this example. We can now
use our identification result in equation 1 to define lower and upper bounds
for $f(x)=E[Y\mid X=x]$:

$\displaystyle f_{min}(x)$ $\displaystyle=f_{1}(x)P(Z=1)\quad\text{and}$
$\displaystyle f_{max}(x)$ $\displaystyle=f_{1}(x)P(Z=1)+P(Z=0).$

Then, estimate $\hat{f}_{min}$ and $\hat{f}_{max}$ by plugging in
$\hat{f}_{1}$ for $f_{1}$.

###  4.4 Compute Reliance

For measuring the reliance on $X_{k}$, we plug these estimators into the
result from proposition 16 to obtain the functions:

$\displaystyle\hat{r}_{min}^{1}(k)$
$\displaystyle=\frac{1}{n(n-1)}\sum_{i=1}^{n}\sum_{j\neq
i}\min\\{L(y_{i},\hat{f}_{min}(s(i,j,k))),\,\,L(y_{i},\hat{f}_{max}(s(i,j,k)))\\}$
$\displaystyle\hat{r}_{max}^{1}(k)$
$\displaystyle=\frac{1}{n(n-1)}\sum_{i=1}^{n}\sum_{j\neq
i}\max\\{L(y_{i},\hat{f}_{min}(s(i,j,k))),\,\,L(y_{i},\hat{f}_{max}(s(i,j,k)))\\}$

where the shuffling function $s(i,j,k)$ returns row $i$ with $x_{jk}$ shuffled
into the $k$th slot. However, these bounds are not bounds on $r$. For example,
for $k=1$, the observed data are $X_{1}\mid Z=1$ and $(Y,X_{2},X_{3})\mid
Z=1$, hence the superscript 1 on the variable names. That is,
$[\hat{r}_{min}^{1}(1),\hat{r}_{max}^{1}(1)]$ bounds how much the admissions
officer depends on race for students who respond to the survey.

Race ($X_{1}$)
---
Sex ($X_{2}$)
---
Score ($X_{3}$)
---
0.00
---
0.05
---
0.10
---
0.15
---
0.20
---

Reliance
---

Note: Blue lines indicate true reliance values.

Figure 1: Reliance for Responding Students

Using the square loss $L(y,\hat{y})=(y-\hat{y})^{2}$, we obtain the bands in
figure 1. Observe that the bands for race and sex are completely above the
band for test score, but the bands for race and sex overlap. Thus, we might
conclude that, for students who ultimately respond to the survey, the
admissions officer relies more on race and sex than on test scores. But, it is
inconclusive if she relies more on race or on sex. That is, for an average
student who responds to the survey, replacing race or sex with noise changes
the admissions decision more than replacing the test score with noise. If we
believe, however, that test score should be the most important variable for
this (self-selecting) cohort, then we might call for a closer look at or
revision of admissions practices.

Note that the ranking of reliance values differs from that of the coefficients
in the admissions decision rule, $Y=1\\{-2X_{1}+X_{2}+X_{3}/5-2.2\geq 0\\}$.
Here, $X_{1}$ has coefficient $-2$, which is larger in absolute value than the
coefficient of $1$ on $X_{2}$, but the reliance on $X_{2}$ is higher.
Furthermore, we cannot compare the coefficients on race or sex to the
coefficient on test score; we can, however, directly compare the reliance on
them.

##  5 Example: Interruptions During Supreme Court Oral Argument

###  5.1 Introduction

Using Supreme Court oral argument transcripts since 1982 [Chang et al., 2020,
Danescu-Niculescu-Mizil et al., 2012], Cai et al. [2023] measure gender’s
effect on how often Justices interrupt advocates. An oral argument is
comprised of a sequence of utterances, each with one speaker. The authors
extract chunks from each oral argument. A chunk is a contiguous dialogue of
four or more utterances between exactly one advocate and one Justice. For
example, one chunk from Comcast Corp. v. National Association of African
American-Owned Media is:

> Erwin Chemerinsky: If at the end the plaintiff concedes that he or she would
> have never gotten the contract anyway, I believe, at the end, under the
> standard adopted in Patterson versus McLean, the plaintiff would not
> prevail.
>
> Justice John G. Roberts Jr.: So that the --
>
> Erwin Chemerinsky: But that doesn’t --
>
> Justice John G. Roberts Jr.: I’m sorry. Go ahead.
>
> Erwin Chemerinsky: I was going to say but that doesn’t tell us what’s
> required at the pleading stage or at the prima facie case stage.

This chunk has 5 utterances, Erwin Chemerinsky is the advocate, and Justice
Roberts is the speaker. As Cai et al. note, these transcripts are manually
typed and consistently formatted, and interruptions are indicated with either
two dashes (as in this chunk) or two dots at the end of an utterance. In this
chunk, the advocate says 62 words (advocate tokens) and is interrupted by
Justice Roberts twice. In the entire dataset of 677,294 chunks, there is a
mean of 59 tokens per chunk, and the median is 28.

Cai et al. seek to estimate the effect of gender $G_{i}$ on the token-
normalized interruption rate $Y_{i\mid j}$ of chunk $i$ with Justice $j$:

$Y_{i\mid j}=\frac{\text{number of interruptions by Justice $j$ in chunk
$i$}}{(\text{number of advocate tokens in chunk $i$})/1000},$ (2)

or the number of times the Justice would interrupt the advocate if the
advocate spoke 1,000 words, which is about 4 pages of 12-point font, double-
spaced text. The Justices’ median interruption rates range from Justice
Blackmun’s 2.2 interruptions per thousand tokens to Justice Breyer’s 11.0. The
median interruption rate overall is 6.8, and the mean is 10.7.

The authors assume that the interruption rates of all chunks, i.e., the
observations, are independent and that “there is no unmeasured confounding.”
They consider but ultimately decline to control for the ideological alignment
between the Justice and the advocate’s argument, the advocate’s stylistic
quality, and the advocate’s experience. Reformulated in Rubin’s potential
outcomes framework, $(Y_{i\mid j}^{(0)},Y_{i\mid j}^{(1)})\perp G_{i}$, where
$Y_{i\mid j}^{(g)}$ is the interruption rate of chunk $i$ by Justice $j$ if
the advocate’s gender in that chunk were $g$. Given these assumptions, the
authors compute the difference between the mean male and female interruption
rates for each Justice and conclude that there is a “clear and consistent
gender effect.”

My analysis contributes new results: First, recognizing the presence of
outliers and heteroskedasticity and controlling for argumentative alignment
and advocate experience, I find that the effect of gender is more muted by
estimating a robust liner regression. Second, I compute cross-distribution
reliance values for these three covariates, as in definition 9, allowing us to
compare the importance of otherwise incommensurable covariates and
illustrating the value of this paper’s framework.

###  5.2 Estimate the Effect of Gender

Like Cai et al., I divide each oral argument since 1982 into chunks of four or
more utterances with one Justice and one advocate, and I compute each chunk’s
token-normalized interruption rate as in equation 2.

Like them, I determine each advocate’s gender by checking the honorific with
which the Justices address that advocate in the oral argument. For example,
for Erwin Chemerinsky, I check if the Justices ever say “Mr. Chemerinsky” or
“Ms. Chemerinsky.” If neither honorific matches, for example because the
speaker is addressed as “General,” I use the confident classifications of an
open-source gender guesser [Pérez, 2016]. I manually resolve a few names that
match both honorifics; for example in Pierce v. Underwood, Justice Rhenquist
accidentally calls advocate Mary Burdick “Mr. Burdick.” I drop any advocates
without a matched gender. The full pipeline is available online [Vebman,
2023].

I define experience as the number of oral arguments since 1982 in which a
particular advocate appears. I also define alignment as whether or not the
Justice ultimately votes for that advocate’s side. Judges, clerks, and
scholars doubt that oral argument actually changes decisions in all but the
closest cases [Wolfson, 2002, Duvall, 2007, Coleman, 2023], mitigating the
possibility that argumentative quality affects both interruptions and the
Justice’s ultimate decision. Even though alignment is weakly correlated with
gender (–0.00375), it may still be relevant because Justices use oral
arguments to refine their opinions. Ultimately, the identification assumption
is

$(Y_{i\mid j}^{(0)},Y_{i\mid
j}^{(1)})\perp\textit{Gender}_{i}\mid\textit{Experience}_{i},\textit{Alignment}_{i}.$

0
---
200
---
400
---
600
---
800
---
1000
---
0
---
10
---
20
---
30
---
40
---
50
---
60
---

Advocate Mean Interruption Rate
---

Male Advocates
---
0
---
50
---
100
---
150
---
200
---
250
---
0
---
10
---
20
---
30
---
40
---
50
---
60
---
70
---
80
---
Female Advocates
---
Number of Chunks with Advocate
---
Note: Each point represents a distinct advocate. An advocate’s mean
interruption rate is the average of the interruption rates over all chunks
with that advocate.

Figure 2: Advocate Mean Interruption Rate vs. Number of Chunks

I use a robust regression because the data are heteroskedastic and contain
outliers. A White test for heteroskedasticity rejects the null hypothesis at
the 5% level for 6 out of 21 Justices. Furthermore, figure 2, which shows each
advocate’s mean interruption rate over the number of chunks with that
advocate, helps reveal that there are outliers. There is significantly more
variance among advocates who appear in fewer chunks, and there are clear
outliers among them. For example, Lisa Corkran is the advocate with the
highest mean interruption rate of 81 interruptions per thousand tokens;
however, she appears in only 16 chunks, including one with Justice Breyer and
an interruption rate of 200 and two with Justice Roberts and an interruption
rate of 500.

–15
---
–10
---
–5
---
0
---
5
---
10
---
Interruptions per 1,000 Tokens
---
Clarence Thomas
---
Lewis Powell
---
Brett Kavanaugh
---
Sonia Sotomayor
---
Neil Gorsuch
---
Warren Burger
---
Elena Kagan
---
Byron White
---
Thurgood Marshall
---
Samuel Alito
---
Ruth Ginsburg
---
William Rehnquist
---
John Stevens
---
Harry Blackmun
---
David Souter
---
Stephen Breyer
---
Anthony Kennedy
---
John Roberts
---
Antonin Scalia
---
Sandra Day O’Connor
---
William Brennan
---
Note: Huber-estimator robust regression coefficient in blue and non-robust
(vanilla OLS) in gray. Both regressions control for alignment and experience.
95% confidence intervals shown. Observed difference in means in red.

Figure 3: Average Treatment Effect of Being Female

Figure 3 displays the results. For each Justice, we estimate a robust
regression of each chunk’s interruption rate on the advocate’s gender,
experience, and argumentative alignment. The average treatment effect of
gender computed by the robust regression is in blue, and that computed with
vanilla OLS is in gray. The observed difference between mean male and female
interruption rates is in red. The observed difference is often closer to 0
compared to the vanilla OLS ATE, and the robust estimates are consistently
closer to 0 than both. Notably, the sign of Justice Marshall’s estimate flips
when using the robust estimate.

Overall, 10 out of 21 Justices have robust confidence intervals that are
entirely greater than 0. That is, for 10 Justices, being a woman increases the
number of interruptions at the 5% significance level. For Justice Powell only,
being a woman decreases the number of interruptions at the 5% significance
level. Although statistically significant, the magnitudes are modest: the
robust ATE of gender is less than 1 interruption per thousand tokens in
magnitude for 12 Justices. For 19 Justices, being female adds (or, in
Kavanaugh and Powell’s case, subtracts) fewer than 2 interruptions per
thousand tokens.

###  5.3 Compute Reliance Values

We now directly apply this paper’s definitions and results to compute each
Justice’s reliance on gender, experience, and alignment. We use the square
loss $L(y,\hat{y})=(y-\hat{y})^{2}$ and let $\hat{f}_{j}$ be the prediction
function of the Huber estimator [Huber, 1981] for Justice $j$ used to compute
the robust ATE above. We compute the within-distribution reliance for each
Justice as in proposition 15, which provides massive performance improvements
over the formulation in definition 14. To allow comparisons across Justices,
we must normalize the within-distribution reliance values. If we define an
additively separable total loss function over the $J$ Justices

$\mathcal{L}((y_{j})_{j=1}^{J},(\hat{y}_{j})_{j=1}^{J})=\sum_{j=1}^{J}L(y_{j},\hat{y}_{j})$

then we can apply proposition 10 to obtain cross-distribution rankings of
reliance values by subtracting off an estimate of the baseline reliance,

$\hat{b}_{j}=\frac{\sum_{\text{chunk }i\text{ with Justice }j}L(y_{i\mid
j},\hat{f}(\textit{Gender}_{i},\textit{Experience}_{i},\textit{Aignment}_{i}))}{\text{number
of chunks with Justice $j$}}.$

0
---
1
---
2
---
3
---
4
---
5
---
6
---
Reliance
---
John Roberts
---
Antonin Scalia
---
Lewis Powell
---
Stephen Breyer
---
Harry Blackmun
---
Ruth Ginsburg
---
Clarence Thomas
---
William Brennan
---
Sandra Day O’Connor
---
Anthony Kennedy
---
David Souter
---
Elena Kagan
---
Thurgood Marshall
---
Brett Kavanaugh
---
William Rehnquist
---
John Stevens
---
Samuel Alito
---
Neil Gorsuch
---
Byron White
---
Warren Burger
---
Sonia Sotomayor
---
Alignment < Experience < Gender
---
Experience < Alignment < Gender
---
Alignment < Gender < Experience
---
Gender < Alignment < Experience
---
Experience < Gender < Alignment
---
Gender < Experience < Alignment
---
Note: Reliance on gender (red), alignment (green), and experience (blue).
Justices in the same horizontal panel rely on these variables in the same
order. Colored lines connect points for ease of viewing.

Figure 4: Reliance on Gender, Alignment, and Experience

The results are summarized in figure 4; the raw reliance values and Huber
estimate coefficients are also in table 1 in the appendix. In example 4, we
showed that $r\geq b\geq 0$, and hence our reliance values, $r-b$, must be
non-negative. The presence of negative estimated reliance indicates that
$\hat{f}$ produces errors, which propagate through $\hat{r}$ and $\hat{b}$.

The Huber-ATE and reliance (both recorded in table 1) provide competing
measures of how much gender and alignment matter to each Justice: for 10 out
of 21 Justices, the ATE of gender is greater than the ATE of alignment. These
same 10 Justices, in addition to Kagan, Marshall, and Ginsburg, rely more on
gender than on alignment (these are the top 3 panels in figure 4). That being
said, the point estimates of reliance on gender and alignment are very close
for many Justices.

We cannot directly compare the coefficients on gender and alignment to the
coefficient on experience. Using reliance, however, we can directly compare
the importance of any variables:

  * •

11 Justices rely the most on gender, 7 rely the most on alignment, and 3 rely
the most on experience.

  * •

13 Justices rely more on gender than on experience when ‘deciding’ if to
interrupt an advocate, but 8 Justices rely more on experience.

  * •

Justices Brennan, Roberts, and Scalia rely the most on gender among all the
Justices. Justices Burger, Sotomayor, and Marshall rely the least. (Thomas’s
estimate is unreliable because he very rarely interrupts.)

  * •

Of the Justices currently on the Court, only Roberts (and Thomas) rely the
most on gender. Alito, Gorsuch, and Sotomayor rely the least on gender among
all the Justices; instead, they all rely the most on alignment. Kagan and
Kavanaugh rely the most on experience.

  * •

Sotomayor relies more on alignment than Roberts relies on gender, even though
he relies the most on gender among any Justice on the Court (besides Thomas).

##  6 Conclusion

This paper expands and expounds on the permutation-based measure of variable
importance, inspired by Fisher et al. [2019] and earlier used by Breiman
[2001, 2002] for random forests. For economics and the social sciences, the
approach discussed in this paper introduces a flexible and interpretable
framework to quantify how much black-box decision-makers rely on variables of
interest. I discuss some of the considerations in implementing such a
framework, rigorously connect it to the machine learning fairness literature,
explain how reliance computations can have policy implications, and present
illustrative and applied examples.

This work also contributes to the machine learning explainability literature
by incorporating counterfactual estimation, including partial identification
of counterfactual queries, from economics. Partial identification is
particularly new. Settling for bounds instead of point-estimates can allow for
more credible assumptions about how black boxes operate under the hood. As
machine learning models become more broadly deployed, including in proprietary
contexts, auditors’ access to these models will decrease. With credible
assumptions, it might be possible to estimate counterfactual queries when
directly querying the black box is impossible.

This paper equips the analyst with a new, flexible, and intuitive method for
understanding how decision-makers think. By helping us explain opaque
decisions, it has the potential to ensure fairness, confirm priorities, and
improve outcomes across a huge array of contexts.

## Appendix

###### Proof of proposition 6.

First, prove $r\geq b$ and 1 $\Leftrightarrow$ 2. We prove the claim by
reformulating $b$ as the solution to the optimization of mean squared
prediction error (MSPE) over the vector
$(Y^{a},X_{1}^{a},X_{1}^{b},X_{2}^{a})$, which is the coupling where
$(Y^{a},X_{1}^{a},X_{2}^{a})\sim P$ is independent of $X_{1}^{b}\sim
P_{X_{1}}$:

$\displaystyle r$
$\displaystyle=\mathbb{E}_{Y^{a},X_{1}^{b},X_{2}^{a}}(Y^{a}-f(X_{1}^{b},X_{2}^{a}))^{2}$
$\displaystyle=\mathbb{E}_{Y^{a},X_{1}^{a},X_{1}^{b},X_{2}^{a}}(Y^{a}-f(X_{1}^{b},X_{2}^{a}))^{2}$
$\displaystyle\geq\min_{\begin{subarray}{c}g:\mathcal{X}_{1}^{2}\times\mathcal{X}_{2}\rightarrow\mathbb{R}\\\
\text{measurable}\end{subarray}}\mathbb{E}_{Y^{a},X_{1}^{a},X_{1}^{b},X_{2}^{a}}(Y^{a}-g(X_{1}^{a},X_{1}^{b},X_{2}^{a}))^{2}.$

The conditional expectation minimizes MSPE, so this minimum is attained by

$\displaystyle g(x_{1}^{a},x_{1}^{b},x_{2}^{a})$
$\displaystyle=\mathbb{E}[Y^{a}\mid
X_{1}^{a}=x_{1}^{a},X_{1}^{b}=x_{1}^{b},X_{2}^{a}=x_{2}^{a}]$
$\displaystyle=\mathbb{E}[Y^{a}\mid X_{1}^{a}=x_{1}^{a},X_{2}^{a}=x_{2}^{a}]$
$\displaystyle=\mathbb{E}[Y\mid X_{1}=x_{1}^{a},X_{2}=x_{2}^{a}]$
$\displaystyle=f(x_{1}^{a},x_{2}^{a}).$

The second equality holds because $(Y^{a},X_{1}^{a},X_{2}^{a})\perp
X_{1}^{b}$. The third equality holds because $(Y,X_{1},X_{2})$ and
$(Y^{a},X_{1}^{a},X_{2}^{a})$ are both distributed according to $P$. Thus:

$r\geq\mathbb{E}_{Y^{a},X_{1}^{a},X_{1}^{b},X_{2}^{a}}(Y^{a}-f(X_{1}^{a},X_{2}^{a}))^{2}=b.$

The second claim follows because the minimizer of MSPE is unique almost
surely.

Second, prove 2 $\Leftrightarrow$ 3. We will prove the forward implication
(only if) by the contrapositive: Suppose
$f(X_{1}^{a},X_{2}^{a})=f(X_{1}^{b},X_{2}^{a})$ does not hold almost surely.
Therefore, there exists $U\subseteq\mathcal{X}_{1}^{2}\times\mathcal{X}_{2}$
such that $P_{X_{1}^{a},X_{1}^{b},X_{2}^{a}}(U)>0$ and
$f(x_{1}^{a},x_{2}^{a})\neq f(x_{1}^{b},x_{2}^{a})$ for all
$(x_{1}^{a},x_{1}^{b},x_{2}^{a})\in U$. Decompose $U$ into two potentially
overlapping subsets

$U^{a}=\\{(x_{1}^{a},x_{1}^{b},x_{2}^{a})\in U:f(x_{1}^{a},x_{2}^{a})\neq
f(x_{2}^{a})\\}$

and

$U^{b}=\\{(x_{1}^{a},x_{1}^{b},x_{2}^{a})\in U:f(x_{1}^{b},x_{2}^{a})\neq
f(x_{2}^{a})\\}.$

Observe that for $u=(x_{1}^{a},x_{1}^{b},x_{2}^{a})\in U$, if $u\notin
U^{a}\cup U^{b}$ then

$f(x_{1}^{a},x_{2}^{a})=f(x_{2}^{a})=f(x_{1}^{b},x_{2}^{a}),$

contradicting that $u\in U$. Thus, $U=U^{a}\cup U^{b}$. Therefore,
$P_{X_{1}^{a},X_{1}^{b},X_{2}^{a}}(U)>0$ implies
$P_{X_{1}^{a},X_{1}^{b},X_{2}^{a}}(U^{a})>0$ or
$P_{X_{1}^{a},X_{1}^{b},X_{2}^{a}}(U^{b})>0$ (or both).

  1. a.

If $P_{X_{1}^{a},X_{1}^{b},X_{2}^{a}}(U^{a})>0$, then define
$V=\\{(x_{1}^{a},x_{2}^{a}):(x_{1}^{a},x_{1}^{b},x_{2}^{a})\in U\\}$ by
dropping $x_{1}^{b}$ from the vector. We thus have
$P_{X_{1}^{a},X_{2}^{a}}(V)>0$, so $f(X_{1}^{a},X_{2}^{a})=f(X_{2}^{a})$ does
not hold almost surely.

  2. b.

If $P_{X_{1}^{a},X_{1}^{b},X_{2}^{a}}(U^{b})>0$, then define
$V=\\{(x_{1}^{b},x_{2}^{a}):(x_{1}^{a},x_{1}^{b},x_{2}^{a})\in U\\}$ by
dropping $x_{1}^{a}$ from the vector. We thus have
$P_{X_{1}^{b},X_{2}^{a}}(V)>0$, which, by absolute continuity, implies
$P_{X_{1}^{a},X_{2}^{a}}(V)>0$. Hence, $f(X_{1}^{a},X_{2}^{a})=f(X_{2}^{a})$
does not hold almost surely.

Thus, we have proved the forward direction by the contrapositive.

Now, prove the reverse implication (if) also by the contrapositive: Suppose
that $f(X_{1}^{a},X_{2}^{a})=f(X_{2}^{a})$ does not hold almost surely.
Therefore, there exists a set $A\subseteq\mathcal{X}_{1}\times\mathcal{X}_{2}$
such that $P_{X_{1}^{a},X_{2}^{a}}(A)>0$ and for all $(x_{1}^{a},x_{2}^{a})\in
A$,

$\displaystyle f(x_{1}^{a},x_{2}^{a})$ $\displaystyle\neq f(x_{2}^{a})$
$\displaystyle=\mathbb{E}[Y\mid X_{2}=x_{2}^{a}]$
$\displaystyle=\mathbb{E}_{X_{1}}[\mathbb{E}[Y\mid X_{1},X_{2}=x_{2}^{a}]\mid
X_{2}=x_{2}^{a}]$ $\displaystyle=\mathbb{E}_{X_{1}}[f(X_{1},x_{2}^{a})\mid
X_{2}=x_{2}^{a}].$

The second equality holds by the law of iterated expectation. Furthermore,

$B(x_{1}^{a},x_{2}^{a})=\\{x_{1}^{b}\in\mathcal{X}_{1}:f(x_{1}^{a},x_{2}^{a})\neq
f(x_{1}^{b},x_{2}^{a})\\}$

satisfies

$P_{X_{1}\mid X_{2}=x_{2}^{a}}(B(x_{1}^{a},x_{2}^{a}))>0,$

otherwise the inequality would fail for a given $(x_{1}^{a},x_{2}^{a})$ pair.
Therefore, each $B(x_{1}^{a},x_{2}^{a})$ also has positive probability with
respect to the unconditioned distribution $P_{X_{1}}$; that is,
$P_{X_{1}}(B(x_{1}^{a},x_{2}^{a}))>0$. Since $P_{X_{1}^{b}}=P_{X_{1}}$, we
have

$P_{X_{1}^{b}}(B(x_{1}^{a},x_{2}^{a}))>0$

for all $(x_{1}^{a},x_{2}^{a})\in S$. Now, combine $A$ and each
$B(\cdot,\cdot)$ to produce the set

$U=\\{(x_{1}^{a},x_{1}^{b},x_{2}^{a}):(x_{1}^{a},x_{2}^{a})\in A\wedge
x_{1}^{b}\in B(x_{1}^{a},x_{2}^{a})\\}.$

Then,

$P_{X_{1}^{a},X_{1}^{b},X_{2}^{a}}(U)=P_{X_{1}^{a},X_{2}^{a}}(A)P_{X_{1}^{b}}\left(\bigcup_{(x_{1}^{a},x_{2}^{a})\in
A}B(x_{1}^{a},x_{2}^{a})\right)$

because $(X_{1}^{a},X_{2}^{a})\perp X_{1}^{b}$. The first term has positive
probability by assumption. The second term also has positive probability
because the union is nonempty by assumption and each $B(x_{1}^{a},x_{2}^{a})$
has positive probability.

There thus exists a set $U\subseteq\mathcal{X}_{1}^{2}\times\mathcal{X}_{2}$
such that $P_{X_{1}^{a},X_{1}^{b},X_{2}^{a}}(U)>0$ and
$f(x_{1}^{a},x_{2}^{a})\neq f(x_{1}^{b},x_{2}^{a})$ for all
$(x_{1}^{a},x_{1}^{b},x_{2}^{a})\in U$. Hence,
$f(X_{1}^{a},X_{2}^{a})=f(X_{1}^{b},X_{2}^{a})$ does not hold almost surely,
and we have proved the second implication by the contrapositive. ∎

###### Proof of Proposition 10.

Because $L$ is additively separable and the expected value commutes with
addition, we can decompose cross-distribution reliance $r_{k}^{\times}$ into

$\displaystyle r_{k}^{\times}$
$\displaystyle=\sum_{i=1}^{n}\mathbb{E}L_{i}(Y_{i}^{a},f_{i}(s_{k}(X_{1i}^{a},X_{1i}^{b}),X_{2i}^{a}))$
$\displaystyle=\mathbb{E}L_{k}(Y_{k}^{a},f_{k}(X_{1k}^{b}),X_{2k}^{a})+\sum_{\begin{subarray}{c}i=1\\\
i\neq
k\end{subarray}}^{n}\mathbb{E}L_{i}(Y_{i}^{a},f_{i}(X_{1i}^{a},X_{2i}^{a}))$
$\displaystyle=r_{k}+\sum_{\begin{subarray}{c}i=1\\\ i\neq
k\end{subarray}}^{n}b_{i}.$

Subtracting $\sum_{i=1}^{n}b_{i}$ from both sides gives

$r_{k}^{\times}-\sum_{i=1}^{n}b_{i}=r_{k}-b_{k}$

Thus, $r_{k}-b_{k}$ gives the same ranking as
$r_{k}^{\times}-\sum_{i=1}^{n}b_{i}$. This ranking shifts all
$r_{k}^{\times}$’s by the same constant $\sum_{i=1}^{n}b_{i}$, and hence gives
the same ranking as $r_{k}^{\times}$. ∎

###### Proof of Proposition 15.

Taking the double sum from definition 14:

$\displaystyle\sum_{i=1}^{n}\sum_{j\neq i}L(y_{i},\hat{f}(x_{1j},x_{2i}))$
$\displaystyle=\sum_{i=1}^{n}\sum_{c\in C}\sum_{\begin{subarray}{c}j\neq i\\\
x_{1j}=c\end{subarray}}L(y_{i},\hat{f}(x_{1j},x_{2i}))$
$\displaystyle=\sum_{i=1}^{n}\sum_{c\in C}\sum_{\begin{subarray}{c}j\neq i\\\
x_{1j}=c\end{subarray}}L(y_{i},\hat{f}(c,x_{2i}))$
$\displaystyle=\sum_{i=1}^{n}\sum_{c\in
C}\left(L(y_{i},\hat{f}(c,x_{2i}))\sum_{\begin{subarray}{c}j\neq i\\\
x_{1j}=c\end{subarray}}1\right)$

Note that

$\sum_{\begin{subarray}{c}j\neq i\\\ x_{1j}=c\end{subarray}}1=|\\{j:j\neq
i,x_{1j}=c\\}|=n_{c}-1\\{x_{1i}=c\\},$

which gives the desired result. ∎

###### Proof of Proposition 16.

Recall from definition 14 that

$\hat{r}=\frac{1}{n(n-1)}\sum_{i=1}^{n}\sum_{j\neq
i}L(y_{i},\hat{f}(x_{1j},x_{2i})).$

For each $i\neq j$, denote the shuffled pair of covariates
$x=(x_{1j},x_{2i})$. Since $y_{i}\in\\{0,1\\}$ and
$0\leq\hat{f}_{min}(x)\leq\hat{f}(x)\leq\hat{f}_{max}(x)\leq 1$, either
$\hat{f}_{min}(x)$ or $\hat{f}_{max}(x)$ is farther from $y_{i}$ than
$\hat{f}(x)$ is. Therefore,

$L(y_{i},\hat{f}(x))\leq
L_{max}\equiv\max\\{L(y_{i},\hat{f}_{min}(x)),L(y_{i},\hat{f}_{max}(x))\\}$

because $L$ increases monotonically with $|y-\hat{y}|$.

To compute the lower bound $L_{min}$, note either
$y_{i}\in[\hat{f}_{min}(x),\hat{f}_{max}(x)]$ or $y_{i}$ is outside of this
range. If $y_{i}$ is outside of the range, then by the same logic as above,
$L(y_{i},\hat{f}(x))\geq\min\\{L(y_{i},\hat{f}_{min}(x)),L(y_{i},\hat{f}_{max}(x))\\}$.
If, however, $y_{i}$ is within this range, then $\hat{f}_{min}(x)=0$ or
$\hat{f}_{max}(x)=1$ since $y_{i}\in\\{0,1\\}$ and $\mathcal{Y}=[0,1]$.
Therefore, $\min\\{L(\hat{f}_{min}(x)),L(\hat{f}_{max}(x))\\}=0$. Thus, either
way,

$L(y_{i},\hat{f}(x))\geq
L_{min}\equiv\max\\{L(y_{i},\hat{f}_{min}(x)),L(y_{i},\hat{f}_{max}(x))\\}$

Hence, $L(y_{i},\hat{f}(x))\in[L_{min},L_{max}]$, so the desired result
follows directly. ∎

###### Proof of Proposition 17.

To show that the lower bound is tight, consider the model

$Y=1\\{X_{1}+X_{2}\geq 1\\},\,Z=X_{2}.$

That is, $Y=1\\{X_{1}+Z\geq 1\\}$. Then, by the law of iterated expectation,

$\displaystyle E[Y\mid X_{1}=0]$ $\displaystyle=E_{Z}[E[Y\mid X_{1}=0,Z]]$
$\displaystyle=E[Y\mid X_{1}=0,Z=1]P(Z=1)+E[Y\mid X_{1}=0,Z=0]P(Z=0)$
$\displaystyle=E[Y\mid X_{1}=0,Z=1]P(Z=1)$

since $E[Y\mid X=0,Z=0]=0$. Thus, the lower bound is tight.

Similarly, to see that the upper bound is tight, consider the model

$Y=1\\{X_{1}-X_{2}\geq 0\\},\,Z=X_{2}.$

That is, $Y=1\\{X_{1}-Z\geq 0\\}$, and

$\displaystyle E[Y\mid X_{1}=0]$ $\displaystyle=E_{Z}[E[Y\mid X_{1}=0,Z]]$
$\displaystyle=E[Y\mid X_{1}=0,Z=1]P(Z=1)+E[Y\mid X_{1}=0,Z=0]P(Z=0)$
$\displaystyle=E[Y\mid X_{1}=0,Z=1]P(Z=1)+P(Z=0)$

since $E[Y\mid X_{1}=0,Z=0]=1$. Thus, both bounds are tight. ∎

Reliance Huber Estimate Coefficient Gender Experience Alignment Gender
Experience Alignment Justice Clarence Thomas 6.194115 0.914656 1.074321
-4.066875 0.045477 -0.967887 William Brennan 2.024609 -0.012736 0.830177
2.069208 0.007237 0.953094 John Roberts 1.028896 0.124092 -0.179966 1.231796
0.013199 -0.665908 Antonin Scalia 0.720167 0.214415 -0.038943 1.240160
0.010162 -0.116746 Lewis Powell 0.673571 0.242610 0.019986 -1.705089 -0.015079
-0.063667 Sandra Day O’Connor 0.543206 0.090279 0.133488 1.336867 0.011487
-0.851091 Anthony Kennedy 0.521350 0.101282 0.437427 1.079679 -0.003048
-0.948730 Stephen Breyer 0.452215 0.155556 0.153670 1.030942 -0.004663
-0.779379 David Souter 0.337769 -0.008472 0.024470 1.030264 0.001045 -0.996237
William Rehnquist 0.244351 0.105431 0.445602 0.737906 0.012149 -0.908850 John
Stevens 0.232185 0.052248 0.900550 0.799289 0.012740 -1.347174 Brett Kavanaugh
0.191025 1.165301 0.924661 -0.339167 -0.014464 -0.696940 Harry Blackmun
0.182901 0.130280 -0.002453 0.829614 -0.009206 -0.474451 Ruth Ginsburg
0.143509 0.114229 -0.219453 0.522730 -0.004648 -0.834549 Elena Kagan 0.129975
0.209616 -0.342419 0.159647 -0.005856 -1.090435 Samuel Alito 0.036339 0.045138
0.279884 0.397898 -0.003541 -0.768242 Neil Gorsuch 0.005193 0.118988 0.425556
0.058726 0.015541 -0.688322 Byron White 0.003446 0.028180 0.234361 0.210519
0.006359 -0.573768 Warren Burger 0.001972 0.003847 0.274191 0.101611 0.001788
-0.712125 Sonia Sotomayor -0.001649 0.023033 1.127161 0.032598 -0.004273
-1.648903 Thurgood Marshall -0.050600 0.772543 -0.121786 0.235322 -0.028233
-0.423940

Table 1: Reliance and Coefficients

## Acknowledgments

This research was supported by funding from the Cornell University College of
Arts and Sciences.

## References

  * Goulette et al. [2015] Natalie Goulette, John Wooldredge, James Frank, and Lawrence Travis.  From initial appearance to sentencing: Do female defendants experience disparate treatment?  _Journal of Criminal Justice_ , 43(5):406–417, 2015.  ISSN 0047-2352.  doi: https://doi.org/10.1016/j.jcrimjus.2015.07.003.  URL https://www.sciencedirect.com/science/article/pii/S0047235215000665.
  * Staton et al. [2007] Lisa J. Staton, Mukta Panda, Ian Chen, Inginia Genao, James Kurz, Mark Pasanen, Alex J. Mechaber, Madhusudan Menon, Jane O’Rorke, JoAnn Wood, Eric Rosenberg, Charles Faeslis, Tim Carey, Diane Calleson, and Sam Cykert.  When race matters: disagreement in pain perception between patients and their physicians in primary care.  _Journal of the National Medical Association_ , 99(5):532–538, 2007.
  * sff [2022] Students for Fair Admissions v. President and Fellows of Harvard College, 2022\.  URL www.oyez.org/cases/2022/20-1199.
  * Breiman [2002] Leo Breiman.  Manual on setting up, using, and understanding random forests v3. 1.  page 14, 2002.
  * Fisher et al. [2019] Aaron Fisher, Cynthia Rudin, and Francesca Dominici.  All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously.  _Journal of Machine Learning Research_ , 20, 12 2019.  URL http://arxiv.org/abs/1801.01489.
  * Coyle et al. [2003] Douglas Coyle, Martin J Buxton, and Bernie J O’Brien.  Measures of importance for economic analysis based on decision modeling.  _Journal of Clinical Epidemiology_ , 56(10):989–997, 10 2003.  ISSN 08954356.  doi: 10.1016/S0895-4356(03)00176-8.
  * Barredo Arrieta et al. [2020] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  _Information Fusion_ , 58:82–115, 6 2020.  ISSN 15662535.  doi: 10.1016/j.inffus.2019.12.012.
  * Lundberg and Lee [2017] Scott M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 4768–4777, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Robnik-Šikonja and Bohanec [2018] Marko Robnik-Šikonja and Marko Bohanec.  _Perturbation-Based Explanations of Prediction Models_ , pages 159–175.  Springer International Publishing, Cham, 2018.  ISBN 978-3-319-90403-0.  doi: 10.1007/978-3-319-90403-0_9.  URL https://doi.org/10.1007/978-3-319-90403-0_9.
  * Breiman [2001] Leo Breiman.  Random forests.  _Machine learning_ , 45:5–32, 2001.
  * Balagopalan et al. [2022] Aparna Balagopalan, Haoran Zhang, Kimia Hamidieh, Thomas Hartvigsen, Frank Rudzicz, and Marzyeh Ghassemi.  The road to explainability is paved with bias: Measuring the fairness of explanations.  page 1194–1206. ACM, 6 2022.  ISBN 978-1-4503-9352-2.  doi: 10.1145/3531146.3533179.  URL https://dl.acm.org/doi/10.1145/3531146.3533179.
  * Marco Tulio et al. [2016] Ribeiro Marco Tulio, Sameer Singh, and Carlos Guestrin.  Why should I trust you?: Explaining the predictions of any classifier.  In _ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 1135–1144. ACM, 2016.
  * Gregorutti et al. [2017] Baptiste Gregorutti, Bertrand Michel, and Philippe Saint-Pierre.  Correlation and variable importance in random forests.  _Statistics and Computing_ , 27(3):659–678, 5 2017.  ISSN 0960-3174, 1573-1375.  doi: 10.1007/s11222-016-9646-1.
  * Corbett-Davies et al. [2017] Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq.  Algorithmic decision making and the cost of fairness.  _CoRR_ , abs/1701.08230, 2017.  URL http://arxiv.org/abs/1701.08230.
  * Chang et al. [2020] Jonathan P. Chang, Caleb Chiam, Liye Fu, Andrew Z. Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil.  Convokit: A toolkit for the analysis of conversations.  _CoRR_ , abs/2005.04246, 2020.  URL https://arxiv.org/abs/2005.04246.
  * Danescu-Niculescu-Mizil et al. [2012] Cristian Danescu-Niculescu-Mizil, Lillian Lee, Bo Pang, and Jon Kleinberg.  Echoes of power: Language effects and power differences in social interaction.  In _Proceedings of WWW_ , pages 699–708, 2012.
  * Cai et al. [2023] Erica Cai, Ankita Gupta, Katherine Keith, Brendan O’Connor, and Douglas R Rice.  "let me just interrupt you": Estimating gender effects in supreme court oral arguments, 2 2023.  URL osf.io/preprints/socarxiv/4dngy.
  * Pérez [2016] Israel Saeta Pérez.  Gender guesser.  https://github.com/lead-ratings/gender-guesser, 2016.
  * Vebman [2023] Daniel Vebman.  Scorpus.  https://github.com/danielVebman/scorpus, 2023.
  * Wolfson [2002] Warren D Wolfson.  Oral argument: Does it matter?  _Indiana Law Review_ , 35:451–456, 2002.  URL https://mckinneylaw.iu.edu/ilr/pdf/vol35p451.pdf.
  * Duvall [2007] Michael Duvall.  When is oral argument important? a judicial clerk’s view of the debate.  _The Journal of Appellate Practice and Process_ , 9:121–130, 2007.  URL https://lawrepository.ualr.edu/appellatepracticeprocess/vol9/iss1/5.
  * Coleman [2023] Robert Coleman, Jr.  The vanishing oral argument: Why it matters and what to do about it.  _American Bar Association Appellate Issues_ , 2023.  URL https://www.americanbar.org/groups/judicial/publications/appellate_issues/2020/winter/the-vanishing-oral-argument/.
  * Huber [1981] Peter J. Huber.  _Regression_ , pages 153–198.  John Wiley & Sons, Ltd, 1981.  ISBN 9780471725251.  doi: https://doi.org/10.1002/0471725250.ch7.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/0471725250.ch7.