HEX: Human-in-the-loop Explainability via Deep Reinforcement Learning

Michael T. Lash

School of Business
University of Kansas
Lawrence, KS 66045

None Yet Assigned

The use of machine learning (ML) models in decision-making contexts, particularly those used in high-stakes decision-making, are fraught with issue and peril since a person – not a machine – must ultimately be held accountable for the consequences of the decisions made using such systems. Machine learning explainability (MLX) promises to provide decision-makers with prediction-specific rationale, assuring them that the model-elicited predictions are made for the right reasons and are thus reliable. Few works explicitly consider this key human-in-the-loop (HITL) component, however. In this work we propose HEX, a human-in-the-loop deep reinforcement learning approach to MLX. HEX incorporates 0-distrust projection to synthesize decider-specific explanation-providing policies from any arbitrary classification model. HEX is also constructed to operate in limited or reduced training data scenarios, such as those employing federated learning. Our formulation explicitly considers the decision boundary of the ML model in question, rather than the underlying training data, which is a shortcoming of many model-agnostic MLX methods. Our proposed methods thus synthesize HITL MLX policies that explicitly capture the decision boundary of the model in question for use in limited data scenarios.

explainability, human-in-the-loop, deep reinforcement learning, federated learning, interpretability

§ INTRODUCTION

Over the years machine learning has become increasingly adopted and incorporated into decision-focused platforms. Such platforms provide predictions that entail some follow-up action depending on the outcome predicted. For instance, Infervision provides deep learning models to hospitals to detect the presence of cancer in CT image scans. These are clinically employed in China to help supplement the radiological workforce deficit [Marr, 2019]. In this example, if the system detects that a patient has cancer, then a course of treatment is prescribed (follow-up action). Otherwise, no further action is taken. In either case, an incorrect prediction produces serious consequences.

Ultimately, however, machines cannot be held accountable for such high-stakes decisions [Neri et al., 2020, Doshi-Velez et al., 2017]. When a decision based on a machine-elicited prediction goes awry, an individual (i.e., person) is necessarily held accountable for the consequences of that decision. Such human-machine accountability has consequently given rise to a variety of human-in-the-loop (HITL) machine learning paradigms. The idea of such paradigms is to incorporate, augment, or aid the human decision-maker by providing machine-elicited insights that are readily interpreted by the decision-maker.

One such HITL paradigm is machine learning explainability (MLX), otherwise referred to as artificial intelligence explainability (AIX). In this context decision platform-embedded machines not only provide predictions (classifications) as to the outcome of the decision task at hand, but a readily interpretable explanation as to why such a prediction was made as well. The explanation provided by the machine provides a contextual lens that allows the decision-maker to understand why the machine made the prediction that it did. If the rationale seems reasonable to the decision-maker they can choose to trust the prediction. If, on the other hand, the rationale does not make sense or seems counter-intuitive, the decision-maker can choose to disregard the prediction and seek solutions and insights elsewhere (using a different model, their own expertise, etc.).

Many MLX and AIX methods have been proposed over the years, including SHAP [Lundberg and Lee, 2017], LIME [Ribeiro et al., 2016], TREPAN [Craven and Shavlik, 1995], MUSE [Lakkaraju et al., 2019], and MAPLE [Plumb et al., 2018], to name just a few. Relatively few works, however, explicitly incorporate the human element [Hitzler and Sarker, 2022, Druce et al., 2021, De et al., 2021]. Furthermore, many works rely on training a local surrogate model [Ribeiro et al., 2016, Lundberg and Lee, 2017, Lakkaraju et al., 2019, Plumb et al., 2018] or some other computationally complex procedure for each instance in need of explanation. These surrogate methods therefore model, and thus provide explanations for the training data, rather than ML model in question, a major short-coming. Finally, many sensitive decision scenarios may employ more recent privacy-preserving techniques, such as federated learning [Liu et al., 2021, Salehkaleybar et al., 2021], which may greatly reduce the data points available to train local, instance-specific models.

In this work, we propose a novel model-agnostic human-in-the-loop classification explanation method, HEX, that addresses the above-mentioned challenges. We contextualize our method around the search for an instance-specific explanatory point – the point nearest the instance in question that lies on the decision boundary, a new mode of explanation that is similar to, yet distinct from that of typical counterfactual explanation methods. The explanatory point promises to be superior since the decision boundary of the ML model in question is explicitly considered. We frame our search for such a point as a Markov Decision Process (MDP), thus allowing us to adopt a deep reinforcement learning (DRL) approach, from which we synthesize explanatory point-producing policies. We adopt two state-of-the-art actor-critic methods, DDPG [Lillicrap et al., 2016] and TD3 [Fujimoto et al., 2018], subsequently augmenting these with several HITL and federated learning-motivated innovations and improvements.

In particular, we propose and define two related phenomena, which we refer to as buffer degeneracy and policy degeneracy. Buffer degeneracy states that a distribution of sub-par actions, derived from a fixed policy, produces a distribution of sub-par rewards. Policy degeneracy, on the other hand, states that sub-par policies, synthesize from a fixed distribution of training instances, also produce sub-par rewards. We prove that buffer degeneracy implies policy degeneracy, using this proof as a basis for a greedy policy-learning strategy.

Furthermore, we incorporate a HITL element into our proposed method by considering the preferred explanatory features of a HITL decider and learning an MLX policy that is specific to these features (i.e., a decider-specific MLX policy). We do so by defining the domain of the decider's preferred features as the feasible action region, projecting the elicited actions onto this region during policy synthesis. We term this 0-distrust project and can thus guarantee that all policy-produced actions are intuitive to the decider in question.

Finally, we incorporate federated learning considerations into our method. First, our MDP conceptualization, coupled with the exploratory, off-policy behavior of our adopted DRL methods ensures that the probability space is explored beyond the training instances initially supplied to the learning process. Thus, the limited-data problem is naturally mitigated. Class imbalance may still permit the issue to exist, however. We therefore further augment our method with synthetic class balancing behavior to mitigate this additional concern.

We evaluate our proposed methods, HEX-DDPG and HEX-TD3, against their out-of-the-box DRL counterparts and two competing state-of-the-art explainability methods on two scenarios, one that is HITL decider-free and another that explicitly consider a HITL decider. Our investigation in the context of these scenarios spans five real-world decision-focused datasets and five machine learning (ML) models trained on each. Our experiments show that our methods perform well in both scenarios, particularly the HITL decider scenario.

§ OUR PROBLEM SETTING AND RELATED WORKS

To further clarify the disposition of this research we provide a brief discussion of related works, highlighting where our proposed method fits in to the broader MLX landscape as the discussion progresses. First, however, we provide some preliminary notation to clarify our problem setting and facilitate discussion of related work.

Our problem setting considers an arbitrary classification model $f$. We assume that $f$ takes an instance $\bx \in \mathcal{X}=[0,1]^p$ as input and produces a probability estimate or classification score $\hat{\mathbb{p}}$ as to the class $c \in \mathcal{C}$ the instance belongs to. To simplify our discussion we will assume w.l.o.g. that $f$ outputs probabilities[If an adopted $f$ does not natively produce probabilities, Platt Scaling [Platt, 1999] can be employed to obtain such outputs.]. This work specifically considers binary classification problems – i.e., $\mathcal{C} = \{0,1\}$[Future work will consider multi-class formulations, which are discussed in the conclusions and future work section.]. The model-elicited probabilities are used with a decision rule $\omega$ and decision function $\Omega$ to produce classifications:

\begin{align}
\label{eq:decisionfunc}
\hat{c} = \Omega\left(f\left(\bx \right),\omega \right) \triangleq
\begin{cases}
\text{if } f\left(\bx \right) < \omega: & 0 \\
\text{else:} & 1
\end{cases}.
\end{align}

This work proposes to learn an explanation-producing policy $\pi:\mathcal{X} \rightarrow \mathcal{Z}_x$, $\mathcal{Z}_x =\{\bz:\min\left(\mathcal{X}\right)-\bx \leq \bz \leq \max\left(\mathcal{X}\right)-\bx\}=\{\bz:\mathbf{0}-\bx \leq \bz \leq \mathbf{1}-\bx\}$, that considers $\bx$ and produces $\bz$ such that, when $\bz$ is additively applied to $\bx$, an explanatory point $\bx^{\prime}$ is produced – i.e., $\bx^{\prime} = \bx + \bz$. Definition <ref>, below, formalizes the notion of an explanatory point.

The explanatory point $\bx^{\prime}$ of $\bx$ is

\begin{align}
\bx^{\prime} \triangleq \argmin_{\bx^{\prime} \in \mathcal{X}}\{\Vert \bx^{\prime} - \bx \Vert : f(\bx^{\prime}) = \omega\},
\end{align}

where $\Vert \cdot \Vert$ is  the Euclidean norm. In other words, $\bx^{\prime}$ is the point nearest $\bx$ that falls on decision boundary $\omega$.

The explanatory point does not itself provide the explanation of instance $\bz$, but functions as a decision boundary reference. The explanation is provided by $\bz$. The policy $\pi$ thus directly produces explanations. To further clarify how $\bz$ provides an explanation of $\bx$ we provide two examples in Figure <ref>, with Figure <ref> showing the explanation of a positive classification and Figure <ref> showing a negative classification.

Positive example.

Negative example.

Two toy explanatory point examples.

Explanations are provided by sorting the values $\vert \bz \vert$ from highest to lowest magnitude and then displaying these to a human-in-the-loop decider $k$. To ensure that $k$ is not overwhelmed and the explanation is simple, only the largest $q$ values need be actually displayed (in the form of a bar chart) [Ribeiro et al., 2016]. Ideally these features will align with $k$'s intuition. We describe how our method provides decider-specific explanations, which guarantee this, in Section <ref>. The benefits of using the proposed explanatory point over other modes of explanation are elucidated on at the end of the next subsection.

§.§ Explainability Methods

The general idea of MLX is to provide some type of explanation as to why $f$ and $\omega$ produce the classification(s) that they do. Both the methods that provide such explanations and the manner in which the explanations are themselves provided are varied and can thus be decomposed and discussed along a variety of different dimensions.

One such dimension is local vs. global explainability. Methods that provide global explanations explain the model as a whole, often doing so through the lens of the entire (global) dataset. Such methods include iForest [Zhao et al., 2018], Partial Dependence Plots (PDP) [Goldstein et al., 2015], Gradient Feature Auditing (GFA) [Adler et al., 2018], and Golden Eye [Henelius et al., 2014]. On the other hand, local explainability methods provide explanations as to why individual instances are classified as they are. This paper focuses on local explanations since our problem setting specifically considers decisions made with respect to individual predictions. Furthermore, our method learns a so-called surrogate explainer model, a separate model that provides explanations of $f$, $\Omega$, and $\omega$, which is a popular approach to the MLX problem. Local surrogate explanation methods include LIME [Ribeiro et al., 2016], SHAP [Lundberg and Lee, 2017] and ShapNet [Wang et al., 2021], PatternNet [Kindermans et al., 2017], MUSE [Lakkaraju et al., 2019], MAPLE [Plumb et al., 2018], and Growing Spheres [Laugel et al., 2018], among many others.

Global surrogate methods are a special case of local explanation methods that attempt to approximate black box models through an interpretable surrogate [Hall et al., 2017, Andrzejak et al., 2013, Hinton and Frosst, 2017, Hara and Hayashi, 2018, Ribeiro et al., 2016, Schetinin et al., 2007]. Our method is similar to these in that we attempt to globally approximate $f$ wrt. decision boundary $\omega$. However, rather than learn an interpretable approximation of $f$, our method learns a policy $\pi$ that directly produces explanations $\bz$. [Samoilescu et al., 2021] also use RL to produce counterfactual explanations (discussed in the next paragraph), adopting an encoder-decoder approach to reduce the dimensionality of the feature space and focusing on the incorporation and modeling of various data modalities.

Local explanations are elicited in a variety of ways. Gradient-based methods explain differentiable models using extracted gradient information (or linear model coefficient) [Ribeiro et al., 2016, Adler et al., 2018, Lash et al., 2017, Cortez and Embrechts, 2011, Shrikumar et al., 2017]. Gradient-based methods are not totally model-agnostic, however, since the model must be differentiable, thus precluding decision trees, random forests, etc. Propagation- and sensitivity-based methods work by propagating the prediction backward through the model and examining the relevant inputs [Datta et al., 2016, Montavon et al., 2018, Goldstein et al., 2015, Hall et al., 2017]. Perturbation-based methods manipulate the inputted feature values of an instance, examining the corresponding output as a means of explanation [Smilkov et al., 2017, Lash et al., 2017, Samek et al., 2017]. Similarly, counterfactual methods provide information about how the classification can be changed, thus providing an explanation through a counterexample [Laugel et al., 2018, Martens and Provost, 2014, Samoilescu et al., 2021, Tsirtsis et al., 2021]. Prototype methods display (training) data points that are similar to the instance in need of explanation [Kim et al., 2014, Kim et al., 2016]. This paper proposes the use of an explanatory point, which explicitly considers the decision boundary $\omega$. The explanatory point therefore explicitly considers classification model $f$, thus providing explanations of the ML model itself, rather than the training data, which is a common approach of other model-agnostic methods.

With the above in mind, our proposed method can simultaneously be considered a prototypical, counterfactual, and perturbation-based method. Readers interested in a more in-depth discussion of MLX and AIX methods and approaches may wish to consult [Burkart and Huber, 2021, Linardatos et al., 2020, Beaudouin et al., 2020, Wang and Wang, 2022, Biecek and Burzykowski, 2021], among other well-written pieces on the subject.

§.§ Markov Decision Processes and Reinforcement Learning Methods

This work proposes to learn a policy $\pi$ that produces an explanatory point $\bz$ provided some inputted instance $\bx$, arbitrary classification function $f$, decision function $\Omega$, and decision boundary $\omega$. To learn such a policy we take a Markov Decision Process (MDP) approach, ultimately adopting deep reinforcement learning methodologies to learn such a policy[Note that this is not a reinforcement learning explainability (XRL) method/work. XRL methods seek to explain reinforcement learning models themselves. In this work, we use RL as a means of providing classification explanations.]. We adopt this approach for five primary reasons:

* The explanatory point generating process is tantamount to approximating $f(\bx) = \omega: \bx \in \mathcal{X}$. Reinforcement learning is particularly suitable for function approximation [Sutton et al., 1999, Melo et al., 2008, Xu et al., 2014].

* Deep reinforcement learning uses neural networks, which are universal function approximators [Hornik et al., 1989], further strengthening the rationale of (1), above.

* The Markov Property is appropriate for the MLX problem setting since we are attempting to explain individual data points independent of one another (local explanations).

* Reinforcement learning methods belong to a broader class of approximate dynamic programming methods. Among this broader class of methods, reinforcement learning does not rely on limiting distributional assumptions, which is beneficial since we wish to make as few assumptions about $\mathcal{X}$ and $f$ as possible.

* By learning a policy that produces explanatory points, we avoid the computational overhead associated with many MLX methods that algorithmically construct an explanation for each explained instance [Ribeiro et al., 2016, Lundberg and Lee, 2017, Lakkaraju et al., 2019, Samoilescu et al., 2021].

MDPs are cast in terms depicted by Figure <ref> and include a state space $\mathcal{S}$, action space $\mathcal{A}$, environment $E$, and reward function $R$.

Standard MDP problem setting.

The current state $\bs_t \in \mathcal{S}$, observed at iteration $t$, is a numerical representation describing the “current, observable state of affairs” of the phenomena being modeled. Some form of so-called agent considers $\bs_t$ and produces actions $\ba_t$. These actions are then applied via a state transition function to produce $\bs_{t+1}$ and subsequently observed in environment $E$. The so-called goodness of taking these actions is assessed via a defined reward function $R$, which produces reward $r_{t+1}$.

The agent is represented by some function or algorithmic process. Function approximation methods attempt to approximate either the policy, value, or both, which then dictate how the agent takes actions. Policy approximation, often referred to as actor-only methods, learn an action-producing policy directly [Schulman et al., 2015, Sutton et al., 1998, Sutton et al., 1999]. Value approximation methods, on the other hand, attempt to approximate the so-called value of being in a particular state from which a policy can be indirectly inferred [Watkins and Dayan, 1992, Mnih et al., 2013, Sutton et al., 1998]. Some methods estimate both the policy (actor) and value (critic) and are thus referred to as actor-critic methods [Konda and Tsitsiklis, 2000, Lillicrap et al., 2016, Fujimoto et al., 2018]. This work adopts an actor-critic approach for reasons that will be discussed shorty in the next section.

§ THE PROPOSED METHOD

Our problem setting considers continuously-valued state and action spaces. We define $\mathcal{X}$ as our state space and $\mathcal{Z}_x$ as our action space[Future work may wish to consider discrete and ordinal explainability problems.]. Further, we define $f$ as our environment and element-wise vector addition as our transition function – i.e., $\bx_{t+1} = \bx_{t} + \bz_{t}$. Figure <ref> clarifies the conceptualization of our problem setting as an MDP.

MLX conceptualized as an MDP.

§.§ An MLX Reward Function

The reward $r_{t+1}$ of Figure <ref> is assessed using a defined explanatory point-based MLX reward function $R(\cdot)$, based on Definition <ref>. Our reward function is disclosed in (<ref>), below:

\begin{align}
\label{eq:exp-reward}
R\left(\mathbf{x}_t,\mathbf{z}_t \right) \triangleq & -\alpha \left[f\left(\mathbf{x}_{t+1}\right)- \left(\omega+ \epsilon_{x} \right)\right]^2 + \beta \left[ \Omega\left(f\left(\mathbf{x}_{t+1}\right),\omega \right) -\Omega\left(f\left(\mathbf{x}_{t}\right), \omega \right)\right]^2 \\ \nonumber
& - \Vert \mathbf{z}_t \Vert_2 - \frac{1}{p}\vert \mathbf{z}_t \vert_0,
\end{align}

where $\alpha,\beta,\epsilon_x > 0$. The first term of (<ref>) encourages solutions that are as close to $\omega$ as possible. Solutions that are further from $\omega$ incur larger penalties. The additive term $\epsilon_x$ is included to facilitate discovery of the decision boundary $\omega$ by encouraging solutions that slightly crest this threshold. Experimentally, we found that the quality of our solutions improved once $\epsilon_x$ was included. While a variety of strategies and values were explored, we set $\epsilon_x$ according to:

\begin{align}
\label{eq:eps-add}
\epsilon_x = \left \{
\begin{array}{cc}
.01 & \text{if }\Omega(f(\mathbf{x}_t), \omega) = 0  \\
-.01 &  \text{otherwise}
\end{array} \right.
\end{align}

While the first term, which maximizes $R$ at $f(\bx_{t}+\bz_{t}) = \omega+\epsilon_x$, penalizes solutions that are further from the decision boundary, the second term of (<ref>) rewards solutions that flip the classification of the instance $\bx_{t}$. This term works in conjunction with the $\epsilon_x$ value of the first term and is motivated by the objective function of [Laugel et al., 2018]. The inclusion of this term vastly improved the policy-learning process since it rewards, rather than penalizes discovery of the decision boundary[This was observed anecdotally as the reward function and methods were developed.]. $R$ is maximized through this term at $\Omega\left(f\left(\mathbf{x}_{t+1}\right),\omega \right) -\Omega\left(f\left(\mathbf{x}_{t}\right), \omega \right)=1$. The third term penalizes solutions that are further from $\bx_{t}$ and has an immediate correspondence to Definition <ref>, since we want the point nearest $\bx_t$ that falls on the decision boundary. This term is also included in [Laugel et al., 2018, Samoilescu et al., 2021] and maximizes $R$ at $\Vert \bz_{t} \Vert_2 \rightarrow 0$. The fourth (last) term $\frac{1}{p}\vert \bz_{t} \vert_0$ penalizes solutions that are less sparse, and therefore less interpretable, through the $0$-norm [Laugel et al., 2018, Samoilescu et al., 2021]. Like the preceding term, $R$ is maximized as $\vert \bz_{t} \vert_0 \rightarrow 0$.

In our experiments, we set $\alpha=4, \beta=10$, and $\epsilon_x$ according to (<ref>) after exploring a variety of values and observing that these worked well and produced stable results across several datasets.

§.§ Learning an Explanatory Point-producing Policy

To learn an explanatory point-producing policy $\pi$, we embed our reward function $R(\cdot)$ in an actor-critic reinforcement learning framework, adopting two well-known deep reinforcement learning methods, DDPG [Lillicrap et al., 2016] and TD3 [Fujimoto et al., 2018]. We tailor these to the MLX problem setting and then further propose several additional MLX-specific innovations which account for HITL deciders and federated learning scenarios.

Figure <ref> illustrates the adopted actor-critic framework in terms of our MLX MDP. Actor-critic methods work by approximating both the policy (actor) and the value (critic). The actor is responsible for estimating the policy and thus produces actions, while the critic is responsible for estimating the goodness of the actions produced by the policy. Since the critic and actor are connected in this manner, if we can accurately assess/estimate action quality, we can learn an accurate policy.

More concretely, the actor and critic each have their own parameter sets – i.e., they are two separate deep learning models – but have a joint learning objective that begins with the critic. To elicit value estimates, the critic uses the well-known Q function, which is based on the Bellman Equation, in conjunction with our specific MLX reward function $R(\cdot)$. The learning objective of the critic is to therefore minimize the following loss function:

\begin{align}
\label{eq:exp-loss}
L\left(Q^{\theta}\right) \triangleq & \hspace{.2cm} \mathbb{E}_{\mathbf{x}_t \sim \rho^{\pi}, \mathbf{z}_t \sim \pi^{\theta}, r_t \sim R} \left[\left(Q \left(\mathbf{x}_t,\mathbf{z}_t \vert \theta^{Q} \right) - y_t^m \right)^2 \right],
\end{align}

where $\rho^{\pi}$ denotes a policy-dependent data distribution, $m\in\{DDPG,TD3\}$ such that

\begin{align}
\label{eq:disc-fut-rewards-ddpg}
y_t^{DDPG} = & \hspace{.2cm} R \left(\mathbf{x}_t,\mathbf{z}_t \right) + \gamma Q\left(\mathbf{x}_{t+1}, \check{\pi}\left( \mathbf{x}_{t+1} \vert \theta^{\pi}\right)\vert \theta^{Q} \right),
\end{align}

when the DDPG method is used and

\begin{align}
\label{eq:disc-fut-rewards-td3}
y_t^{TD3} = & \hspace{.2cm} R \left(\mathbf{x}_t,\mathbf{z}_t \right) + \gamma  \min_{i=1,2} Q\left(\mathbf{x}_{t+1}, \check{\pi}\left( \mathbf{x}_{t+1} \vert \theta^{\pi}\right)\vert \theta^{Q}_i \right),
\end{align}

when TD3 is used, and where

\begin{align}
\label{eq:off-policy}
\check{\pi}\left(\mathbf{x}_t \vert \theta \right) = & \pi\left(\mathbf{x}_t \vert \theta \right) + \mathcal{N}\left(\mathbf{0},\pmb{\epsilon} \right).
\end{align}

Put briefly, the loss is defined as the estimate elicited from the critic, parameterized by $\theta^Q$, minus the discounted, future rewards expressed by either (<ref>) or (<ref>), where $0 < \gamma < 1$ is a discount parameter imposed on the estimate of future rewards. Note that the primary distinction between DDPG and TD3 is the use of two critics by TD3, over the single critic of DDPG. TD3 selects the minimum of the two estimates to help regularize the value estimation. Additionally, TD3 applies soft parameter updates periodically, rather than at every iteration – we discuss this shortly.

To optimize the parameters $\theta^{\pi}$ of the actor the chain rule is applied to (<ref>):

\begin{align}
\label{eq:grad-pi}
\nabla_{\theta^{\pi}}J \approx & \hspace{.2cm} \nabla_{\theta^{\pi}}L\left(Q^{\theta}\right)\\\nonumber
= & \hspace{.2cm} \mathbb{E}\left[\nabla_{\theta^{\pi}}Q \left(\tilde{\bx},\tilde{\bz} \vert \theta^{Q} \right)\vert_{\tilde{\bx}=\bx_t,\tilde{\bz}=\pi\left(\bx_t \vert \theta^{\pi} \right)} \right]\\\nonumber
= & \hspace{.2cm}\mathbb{E}\left[\nabla_{\tilde{\bz}}Q \left(\tilde{\bx},\tilde{\bz} \vert \theta^{Q} \right)\vert_{\tilde{\bx}=\bx_t,\tilde{\bz}=\pi\left(\bx_t\right)} \nabla_{\theta^{\pi}} \pi\left(\tilde{\bx} \vert \theta^{\pi} \right)\vert_{\tilde{\bx}=\bx_t} \right]\nonumber
\end{align}

This result follows that of [Lillicrap et al., 2016], who also point out that [Silver et al., 2014] proves this is the gradient that optimizes the policy (i.e., policy gradient). Note that the deterministic policy $\pi$ is used rather than the exploratory policy $\check{\pi}$.

To avoid the pitfalls associated with the use of Q Learning with deep neural networks, so-called target Q and policy networks, $\theta^{Q^{\prime}}$ and $\theta^{\pi^{\prime}}$, are used. These networks are updated gradually using the parameters $\theta^{Q}$ and $\theta^{\pi}$ along with a scaling factor $\tau$. These are called soft updates and can be expressed as:

\begin{align}
\label{eq:soft-update}
\theta^{Q^{\prime}} \leftarrow & \tau \theta^{Q} + (1-\tau) \theta^{Q^{\prime}} \\\nonumber
\theta^{\pi^{\prime}} \leftarrow & \tau \theta^{\pi} + (1-\tau) \theta^{\pi^{\prime}}.
\end{align}

To compute the loss and optimize the parameters of both networks a sampling procedure is used in conjunction with a so-called experience replay buffer [Mnih et al., 2013], which we denote using $\pmb{\rho}$. The replay buffer stores tuples of the form $\left(\bx_t,\bz_t,r_t,\bx_{t+1}\right)$ across training episodes ($e=1,\dots,E$) and inner training iterations ($t=1,\dots,T$) according to some specified buffer size $\xi$:

\begin{align}%change j to something else (it is used later to index features)
\label{eq:buffer}
\pmb{\rho}_{l \text{ mod } \xi} = \left(\bx_t,\bz_t,r_t,\bx_{t+1}\right)_l,
\end{align}

where $l = e \cdot t$. When $l > \xi$ previous buffer entries are overwritten. Therefore, if all observations are to be stored, a buffer of size $E \cdot T$ is needed. Note that we refer to distributions of specific sub-elements of $\pmb{\rho}$ using an index that refers to the element in question– e.g., $\rho^{\pi}$ is the policy-dependent distribution.

At each update iteration the buffer is sampled uniformly $N$ times:

\begin{align}
\left(\bx_t,\bz_t,r_t,\bx_{t+1}\right)_i \sim \mathcal{U}\left(\pmb{\rho}\right):i=1,\dots,N,
\end{align}

where the $N$ samples are used to compute the loss expressed in (<ref>). To illustrate the training procedure we provide Algorithm <ref>, highlighting the buffering of experience tuples in red. Projection onto the feasible action region (Line 6 of Algorithm <ref>) is achieved by

\begin{align}
\label{eq:proj}
Proj_{\mathcal{Z}_x}\left(\bz\right) \triangleq & \min\left(\max\left(\mathbf{l}_z,\bz \right),\mathbf{u}_z\right)
\end{align}

where $\mathbf{l}_z$ and $\mathbf{u}_z$ are the lower and upper bounds of $\mathcal{Z}_x$
Initial MLX policy learning procedure

Initialize buffer $\mathcal{R} \gets \{\}$

Execute a cold start procedure to provide initial samples to $\pmb{\rho}$.

$\bx \sim \mathcal{D}$ (Sample an instance from policy-learning dataset $\mathcal{D}$)

$\bz_{t} \gets Proj_{\mathcal{Z}_x}\left(\check{\pi}\left(\mathbf{x}_t \vert \theta \right)\right)$ (Using (<ref>) and (<ref>)).

Estimate $r_t$ according to (<ref>).

$\bx_{t+1} \gets \bx_{t} + \bz_{t}$ (Transition to the next state.)

$\pmb{\rho}_{l \mod \xi} \gets (\bx_{t},\bz_{t},r_t,\bx_{t+1})_l$ (Buffer the sample.)

$m = DDPG$
Sample $\pmb{\rho}$ and compute the loss according to (<ref>).

Backpropagate the loss to update $\theta^Q$ and use (<ref>) to update $\theta^\pi$.

Apply soft updates according to (<ref>).

$m = TD3$ and $t \mod 2 = 0$
Sample $\pmb{\rho}$ and compute the loss according to (<ref>).

Backpropagate the loss to update $\theta^Q$ and use (<ref>) to update $\theta^\pi$.

Apply soft updates according to (<ref>).

§.§ Buffer and Policy Degeneracy

The buffer is critically important to the successful learning of a policy that produces explanatory points. If the buffer fills up with “bad” examples, then the quality of the policy will be low: learning what is bad does not guarantee that the actions obtained from such a policy will be good. Furthermore, since the buffer depends in part on the policy itself, a negative feedback loop may be created such that the learning process becomes degenerate. To better characterize this phenomenon we first define instance degeneracy, expanding this definition to characterize buffer degeneracy and policy degeneracy. We then provide a theorem and proof to show the linkage between the two.

Instance degeneracy is provided in Definition <ref>.

Instance degeneracy. A sampled instance $\bx \sim \rho^{\pi}_l$ from the policy-dependent data distribution at $l$ is said to experience degeneracy if

\begin{align}
y_l > & y_{l^{\prime}}: l^{\prime} > l,
\end{align}

where $y_l$ is computed on (<ref>) or (<ref>) using $\pi_l^{\theta}$ and $\pi_{l^{\prime}}^{\theta}$ to produce actions and $Q_l^{\theta}$ and $Q_{l^{\prime}}^{\theta}$ to assess future discounted rewards.

Definition <ref> state that an instance received a more favorable action at an earlier training iteration $l \leq l^{\prime}$ and forms the basis and motivation for our definition of both buffer degeneracy and policy degeneracy. Buffer degeneracy is characterized by Definition <ref>.

Buffer degeneracy. The buffer $\pmb{\rho}_{l^{\prime}}$ at iteration $l^{\prime}$ is to be degenerate if, under a fixed policy $\pi_{l}^{\theta}$ and fixed Q function $Q^{\theta}_l$,

\begin{align}
\mathbb{E}_{y_l \sim \rho^{y}_l}\left[y_l\right] > & \mathbb{E}_{y_{l^{\prime}} \sim \rho^{y}_{l^{\prime}}}\left[y_{l^{\prime}}\right]: l^{\prime} > l,
\end{align}

where $\rho^{y}_l = \{y:\mathbf{x} \sim \rho^{\pi}_l, \mathbf{z} \sim \pi^{\theta}_l, r \sim R\}$ and  $\rho^{y}_{l^{\prime}} = \{y: \mathbf{x} \sim \rho^{\pi}_{l^{\prime}}, \mathbf{z} \sim \pi^{\theta}_{l}, r \sim R\}$.

The definition of buffer degeneracy states that the expectation over the future discounted reward distribution, formed by assessing the buffered samples at $l$ using the policy and value (Q) function at $l$, is larger than using the same policy and value function on the buffered samples at $l^{\prime}$. This immediately gives rise to our characterization of policy degeneracy, provided in Definition <ref>, below.

Policy degeneracy. A policy $\pi_{l^{\prime}}$ at iteration $l^{\prime}$ is to be degenerate if

\begin{align}
\mathbb{E}_{y_l \sim \rho^{y}_l}\left[y_l\right] > & \mathbb{E}_{\tilde{y}_{l^{\prime}} \sim \tilde{\rho}^{y}_{l^{\prime}}}\left[\tilde{y}_{l^{\prime}}\right]: l^{\prime} > l,
\end{align}

where $\tilde{\rho}^{y}_{l^{\prime}} = \{y: \mathbf{x} \sim \rho^{\pi}_{l}, \mathbf{z} \sim \pi^{\theta}_{l^{\prime}}, r \sim R\}$.

Policy degeneracy occurs when the expectation taken over the future discounted reward distribution, formed using actions elicited from a policy at $l$ is better than the expectation over the distribution from from actions elicited from the policy at $l^{\prime}$. With the definitions of buffer degeneracy and policy degeneracy in mind, we propose Theorem <ref>, below.

(Buffer degeneracy $\implies$ policy degeneracy)

\begin{align}
\mathbb{E}_{y_l \sim \rho^{y}_l}\left[y_l\right] > \mathbb{E}_{y_{l^{\prime}} \sim \rho^{y}_{l^{\prime}}}\left[y_{l^{\prime}}\right] \implies \mathbb{E}_{y_l \sim \rho^{y}_l}\left[y_l\right] > \mathbb{E}_{\tilde{y}_{l^{\prime}} \sim \tilde{\rho}^{y}_{l^{\prime}}}\left[\tilde{y}_{l^{\prime}}\right]: l^{\prime} > l
\end{align}

The proof of Theorem <ref> is in Appendix A. The theorem and proof show the relationship between buffer degeneracy and policy degeneracy and map intuitively onto the actor-critic policy-learning ecosystem. When the buffer fills up with low-reward examples, the policy synthesized from these samples consequently produces actions that have lower rewards. This behavior is attributable to the loss of (<ref>), and consequent gradient updates of (<ref>), which is minimized by learning such a policy. Note that the adopted off-policy approach may either help prevent or help encourage degenerate learning. On the one hand, off-policy learning can help prevent degeneracy during early learning phases. On the other hand, a policy that has learned to produce good rewards may become degenerate if the off-policy component produces enough degenerate action-having experience replay tuples. This issue, however, is mitigated by having a sufficiently large buffer.

§.§.§ Selective Buffering

To inhibit buffer degeneracy and thus prevent policy degeneracy, we propose a simple greedy procedure that selectively adds samples to the buffer. In essence, we add the experience tuple observed to have the highest reward over a prescribed number of iterations. This procedure is described by Algorithm <ref>, below.
Selective Buffering

Initialize $r_{best} \gets -\infty$

$r_t > r_{best}$
$best \gets (\bx_{t},\bz_{t},r_t,\bx_{t+1})$

$t \mod w = 0$
Add $best$ to $\pmb{\rho}$

$r_{best} \gets -\infty$

Since Algorithm <ref> screens samples according to their reward, the buffer is prevented from filling up with increasingly worse samples, thereby mitigating the buffer degeneracy issue. Note that, $w$ can be set lower or higher to tailor the procedure to be more or less greedy. We show how Algorithm <ref> fits into an updated MLX policy-learning procedure at the end of this section with all ensuing considerations added in.

§.§ Data Corruption and Federate Learning Considerations

The optimized policy $\pi^{\theta^*}$ is synthesized by sampling instances from a dataset. This dataset may not be the same as that used to induce $f$ for a variety of reasons. First, instances may be lost or corrupted over time. Second, some data may not be accessible due to federated learning or other privacy-preserving considerations. This may give rise to two related issues, illustrated by Figure <ref>:

* The first, illustrated by Figure <ref> is that there may be unsupported, and therefore unrepresented areas of the probability space, leading to an unreliable approximation of $f$.

* A second, related artifact, illustrated by Figure <ref>, also leading to an unreliable approximation of $f$, is class imbalance, where substantially fewer samples of one class are available for learning $\pi^{\theta^*}$. The latter may also be attributable to class imbalance within the training data itself. In this case, a reliable policy may be learned for either $c=0$ or $c=1$, but not the other.

Unsupported areas of the probability space.

Class imbalance.

Two examples of unsupported probability space regions, shaded in gray. The decision boundary, which we are trying to approximate, is represented by the dark gray dotted line and the available policy-learning training samples are represented by the black dots.

The “unsupported probability space” issue is naturally mitigated as a consequence of our conceptualization of the problem and adopted off-policy RL methods. Off-policy RL explores the environment in question (see (<ref>)) – i.e., the probability space – when searching for the explanatory point of each instance during policy synthesis. Thus, if a sufficiently large number of policy training iterations are executed, the probability space is naturally explored, mitigating this concern.

To alleviate the class imbalance issue, which amounts to under-exploration of one side of the probability space wrt. the decision boundary in question, we can apply any well-known class-balancing technique to the policy-learning dataset. Such methods include the various majority class under-sampling [Liu et al., 2008, Tsai et al., 2019, Yen and Lee, 2009] and minority class oversampling [Chawla et al., 2002, Barua et al., 2012] techniques. Since we wish to keep as many organic instances as possible, we focus on oversampling the minority class, adopting the well-known synthetic minority oversampling technique (SMOTE) [Chawla et al., 2002].

§.§ Human-in-the-loop Feature Trustworthiness

An intrinsic component of MLX is the HITL element [Lipton, 2018]. It is this element that necessitates elicitation of an explanation in the first place. In this paper we consider a HITL decision-making scenario, where an individual $k$ interacts with $f(\bx)$ and $\Omega(\bx,\omega)$ by inputting $\bx$ and obtaining a probability estimate $\hat{\mathbb{p}}$ and a discrete prediction $\hat{c}$ of the outcome. The decider $k$ then also obtains an explanation $\bz$ of this prediction from $\pi^{\theta^*}$. Subsequently, $k$ considers $\bz$ in order to decide whether the prediction was made for the right reasons – i.e., the prediction is trustworthy/reliable  [Ribeiro et al., 2016]. Recall that $\bz = \bx - \bx^{\prime}$, where $\bx^\prime$ is the explanatory point nearest $\bx$ on the class-separating decision boundary. Therefore, $\bz$ tells $k$ which features (and by how much) “pushed” the instance towards the classification that was produced and, if those features seem reasonable, $k$ can choose to trust and subsequently act on the prediction (purchase the stock, treat the patient, etc.). To better formalize this notion, let $\bz_k \in \{0,1\}^{p}$ denote an acceptability vector for decider $k$, where

\begin{align}
z_k^{(j)} =
\begin{cases}
1 & \text{if }k \text{ trusts feature }j \\
0 & \text{Otherwise}
\end{cases}.
\end{align}

Further, denote $\tilde{\bz}$ to be an indicator vector of $\bz$ such that

\begin{align}
\tilde{z}^{(j)} =
\begin{cases}
1 & \text{if }z^{(j)} \neq 0\\
0 & \text{Otherwise}
\end{cases}.
\end{align}

In other words, $\tilde{\bz}$ indicates the non-zero entries in $\bz$ that were used to explain $\bx$. Using $\tilde{\bz}$ and $\bz_k$ we can compute an explanation-decider disagreement score as

\begin{align}
\label{eq:agree-score}
\zeta_k^{z}= \sum_{j=1}^{p}\mathbb{1}_{ (\tilde{z}^{(j)} - z_k^{(j)}) > 0}.
\end{align}

Note that $(\tilde{z}^{(j)} - z_k^{(j)}) > 0$ increases the score when a non-trusted feature is used to make an explanation, but not when a trusted feature is not used as part of the explanation. This is intuitive since there may be times when only a subset of trustworthy features are used to provide an explanation. The explanation-decider disagreement score is used to motivate the UEP assessment measure, defined in the Experiments section. Finally, note that this measure can be considered a measure of explanation fidelity, similar to those proposed in [Yeh and Ravikumar, 2021].

§.§.§ Individual-specific Explanatory Policies via 0-Distrust Projection

Ideally, the learned explanatory policy $\pi^{\theta^*}$ will be sensitive to an individual $k$'s preferred explanatory features, i.e.,  the non-zero entries of $\bz_k$. To learn a policy that is sensitive and specific to $k$ we propose the use of what we term 0-Distrust Projection. To present this idea, we define an instance-decider action feasibility domain $\mathcal{Z}^{(k)}_x$ as

\begin{align}
\label{eq:0-distrust}
\mathcal{Z}_x^{(k,j)} =
\begin{cases}
\mathcal{Z}_x^{(j)} & \text{if }z_k^{(j)} = 1\\
\{0\} & \text{Otherwise}
\end{cases} j=1,\dots,p.
\end{align}

We subsequently update (<ref>) to project onto instance-decider domain $\mathcal{Z}^{(k)}_x$ if we wish to synthesize a policy that is sensitive to $k$'s preferences, thus ensuring $\zeta_k^{z}=0$.

With selective buffering, federated learning/data corruption, and 0-distrust (i.e., HITL) considerations in mind, we provide an updated MLX policy-learning procedure in Algorithm <ref>, below.
HEX MLX policy learning procedure

Initialize buffer $\pmb{\rho} \gets \{\}$ and $r_{best} \gets -\infty$

Execute a cold start procedure to provide initial samples $\pmb{\rho}$.

$\bx \sim \mathcal{D}$ (Sample an instance from policy-learning dataset $\mathcal{D}$)

$\bz_{t} \gets Proj_{\mathcal{Z}_x^{(k)}}\left(\check{\pi}\left(\mathbf{x}_t \vert \theta \right)\right)$ (Using (<ref>) and (<ref>) with (<ref>)).

$\bz_{t} \gets Proj_{\mathcal{Z}_x}\left(\check{\pi}\left(\mathbf{x}_t \vert \theta \right)\right)$ (Using (<ref>) and (<ref>)).

Estimate $r_t$ according to (<ref>).

$\bx_{t+1} \gets \bx_{t} + \bz_{t}$ (Transition to the next state.)

$r_t > r_{best}$
$best \gets (\bx_{t},\bz_{t},r_t,\bx_{t+1})$

$t \mod w = 0$
$\pmb{\rho}_{l/w \mod \xi} \gets best$

$r_{best} \gets -\infty$

$m = DDPG$
Sample $\pmb{\rho}$ and compute the loss according to (<ref>).

Backpropagate the loss to update $\theta^Q$ and use (<ref>) to update $\theta^\pi$.

Apply soft updates according to (<ref>).

$m = TD3$ and $t \mod 2 = 0$
Sample $\pmb{\rho}$ and compute the loss according to (<ref>).

Backpropagate the loss to update $\theta^Q$ and use (<ref>) to update $\theta^\pi$.

Apply soft updates according to (<ref>).

§ EXPERIMENTS

To assess our proposed method we adopt five real-world datasets – Bank, MIMIC, Movie, News, and Student – spanning a variety of decision-focused domains. Descriptions of these datasets and summary statistics are found in Appendix B. We learn five well-known machine learning algorithms, logistic regression, SVM, neural networks, decision trees, and random forest on each dataset. The procedure for training each of these models on each of our datasets is found in Appendix C, along with predictive performance assessments of each model on each dataset.

Our first experiments do not consider a HITL decider (i.e., are decider-free). To begin assessing explainability method performance, we first examine the policy learning curves of each of our proposed HEX methods on each dataset and model against out-of-the-box DDPG and TD3 methods, which we henceforth refer to only as DDPG and TD3. Policies are learned by setting $E=1000$, corresponding to 1000 sampled training instances, and $T=300$. Inner training iterations terminate early when an explanatory point is discovered. Each method learns 10 policies (10 trials) for each dataset and method to ensure the results obtained are robust. Episode-wise averages are taken over the 10 trials, with a rolling window average subsequently taken over these averages (window size $= 40$) and plotted to produce learning curves. Subsequently, we examine the performance of these methods at producing explanations on held-out sets of data, each consisting of $100$ instances, comparing to two well-established model-agnostic explanation methods, LIME [Ribeiro et al., 2016] and Growing Spheres [Laugel et al., 2018], which we display as “Grow” when reporting results. The learning curves are shown in terms of average explanatory reward, while the test set evaluations are shown in terms of average explanatory point decision boundary deviance (DBD), defined as follows:

\begin{align}
DBD =& \hspace{.1cm} \vert  f(\bx^{\prime}) - \omega \vert.
\end{align}

$\omega=0.50$ for all ML models for all datasets.

In our second set of experiments, we repeat the above for a scenario that explicitly considers a human-in-the-loop decider who is sensitive to some subset of features. To simulate this scenario we randomly select 10, 50, and 90% of features and deem those selected as being explanation-unacceptable to the simulated decider (i.e., $90,50,10$% are acceptable). We refer to these percentages as the unacceptable percentage, or UAP for short. We vary this random selection across each of the 10 trials, holding each selected feature set consistent for each MLX method being evaluated. Subsequently, we examine the average DBD on a held-out set of test data, comparing our HEX methods against the non-HEX DDPG, TD3, LIME, and Grow benchmark methods. Furthermore, we show the percentage of undesirable features used in explaining predictions, averaging across the 10 trials. We refer to this percentage as the unacceptable explanatory percentage, or UEP for short. This measure is motivated by our defined explanation-decider disagreement score disclosed in (<ref>). The UEP of an explanatory point is calculated as:

\begin{align}
UEP = & \hspace{.1cm} \frac{\sum_{j=1}^{p} \mathbb{1}_{x^{\prime}_j > 0 \hspace{0.1cm} \cap \hspace{0.1cm} \mathcal{Z}_x^{(j)} = \{0\}}}{p \times UAP}
\end{align}

§ RESULTS

We present the results of our experiments below, stratifying their presentation into “decider-free' and “HITL decider” scenarios.

§.§ Decider-free Scenario

This subsection presents the results of our decider-free evaluations, where we do not consider or simulate a HITL decider. This scenario is intended to establish our proposed methods as being comparable or superior to the state-of-the-art, agnostic of the specific HITL problem setting considered by this work. We begin by examining the learning curves of the four DRL methods: DDPG, TD3, HEX-DDPG, and HEX-TD3. Subsequenntly, we examine the performance of all methods on held-out test data.

§.§.§ Learning Curves

The learning curves of the four DRL methods are depicted in the table of Figures, below (Table <ref>). Generally speaking, all DRL methods were able to learn explanatory point-producing policies on all datasets for all ML methods considered. That said, several trends are apparent.

Bank     MIMIC     Movie     News     Student

Learning curves of each RL method on each dataset and ML model for the decider-free scenario. Plotted values are episode-wise averages taken over ten trials with a rolling window average (window size $=40$) of these averages plotted.

First, our HEX-TD3 method is universally able to synthesize higher reward-producing policies across datasets and ML methods. This is, of course, training set performance (more on this shortly). Second, DDPG almost universally learns the lowest reward-producing policies across datasets and ML methods (also more on this shortly). TD3 and HEX-DDPG are often very comparable, falling between HEX-TD3 and DDPG in terms of average reward performance, although generally closer to HEX-TD3 – i.e., DDPG tends to be well beneath the other three methods.

§.§.§ Testing

Table <ref> shows the average DBD by explainability method, dataset, and ML model. Table <ref> in Appendix E shows the average rewards of these results. There are 25 total evaluations (# datasets $\times$ # of ML models). Of these a DRL method performs the best on 22/25. LIME performs the best on 2/25 and Grow on 1/25. Curiously, the 3/25 models that Grow and LIME perform the best on are all DT models.

2|c|     5c|Model
1|l|Dataset     1c|Method     NN     SVM     RF     LR     DT
1|c|6*Bank     DDPG     0.28     0.173     0.091     0.339     0.31

1|c|     HEX-DDPG     0.277     0.173     0.111     0.33     0.331

1|c|     TD3     0.298     0.212     0.09     0.376     0.337

1|c|     HEX-TD3     0.3     0.227     0.108     0.366     0.343

1|c|     LIME     0.473     0.461     0.303     0.499     0.446

1|c|     Grow     0.476     0.462     0.335     0.495     0.427
1|c|6*MIMIC     DDPG     0.271     0.104     0.046     0.215     0.386

1|c|     HEX-DDPG     0.277     0.119     0.059     0.223     0.373

1|c|     TD3     0.308     0.119     0.055     0.251     0.392

1|c|     HEX-TD3     0.313     0.146     0.063     0.246     0.388

1|c|     LIME     0.427     0.338     0.154     0.426     0.321

1|c|     Grow     0.483     0.404     0.236     0.489     0.342
1|c|6*Movie     DDPG     0.24     0.316     0.076     0.454     0.438

1|c|     HEX-DDPG     0.207     0.335     0.097     0.424     0.448

1|c|     TD3     0.247     0.356     0.072     0.457     0.433

1|c|     HEX-TD3     0.235     0.381     0.092     0.452     0.45

1|c|     LIME     0.462     0.5     0.314     0.5     0.49

1|c|     Grow     0.45     0.416     0.217     0.49     0.331
1|c|6*Student     DDPG     0.246     0.176     0.092     0.374     0.381

1|c|     HEX-DDPG     0.198     0.212     0.082     0.247     0.372

1|c|     TD3     0.239     0.178     0.092     0.371     0.385

1|c|     HEX-TD3     0.192     0.206     0.083     0.246     0.363

1|c|     LIME     0.31     0.195     0.101     0.37     0.225

1|c|     Grow     0.42     0.304     0.126     0.449     0.234
1|c|6*News     DDPG     0.367     0.425     0.05     0.26     0.493

1|c|     HEX-DDPG     0.382     0.385     0.067     0.274     0.493

1|c|     TD3     0.387     0.429     0.056     0.3     0.496

1|c|     HEX-TD3     0.377     0.383     0.068     0.278     0.492

1|c|     LIME     0.496     0.448     0.294     0.494     0.498

1|c|     Grow     0.497     0.482     0.271     0.483     0.436

Decider-free scenario. Average DBD. Bold indicates the lowest average DBD obtain on each dataset and ML model.

The success of the DDPG and TD3 methods, particularly DDPG, is interesting provided the learning curve results. The out-of-the-box methods perform the best on 11/25 experiments and our HEX methods also perform the best on 11/25. More concretely, DDPG performed the best on 9/25 evaluations, TD3 performed the best on 2/25, HEX-DDPG performed the best on 7/25, and HEX-TD3 performed the best on 4/25. Recall that DDPG produced the lowest learning curves across all datasets and methods and HEX-TD3 produced the highest. We suspect that the added greediness of our methods may, at times, lead to overfitting. Nevertheless, these results show there are benefits to our proposed additions even in non-HITL scenarios. We again stress that a DRL approach outperforms other state-of-the-art methods on 22/25 evaluations.

These results also reveal several additional artifacts worth mentioning. First, the linear model, logistic regression (LR), tends to have larger DBD values across the various datasets and explainability methods. This is interesting because the method itself is naturally interpretable (due to it's linear nature). Second, random forest (RF), a highly non-linear model, generally has lower DBD values across the various explainability methods, particularly the DRL methods. This is ideal since explainability methods are adopted to overcome the unexplainable nature of nonlinear, black box models in the first place.

§.§ HITL Decider Scenario

This subsection presents the results of the HITL decider scenario. We consider three simulated decider scenarios using 10, 50, and 90% of features as the UAPs for each, respectively (i.e., 90, 50, and 10% of features are preferred by the decider). We randomly select a different set of features for each UAP scenario, for each of the 10 trials, holding the selection consistent across each explainability method and ML model. We begin by examining the learning curves of the various DRL methods, subsequently examining the test set performance of all models first in terms of average DBD and then in terms of average UEP. Note that DDPG and TD3, along with LIME and Grow, do not explicitly consider a HITL. We therefore only report one set of results for each of these explainability methods.

§.§.§ Learning Curves

The DRL learning curves for the HITL decider scenario are shown in Table <ref>. Note that there are three learning curves for each HEX method, but only one for the out-of-the-box methods, since the out-of-the-box methods do not consider the decider's preferred features.

Bank     MIMIC     Movie     News     Student

Learning curves of each RL method on each dataset and ML model for the HITL decider scenario. Plotted values are per-episode averages over ten trials with a rolling window average (window size $=40$) of these averages plotted.

First, we notice that DDPG and TD3 have persistently lower learning curves than either of the HEX methods, even across different UAP values. In particular, DDPG has the lowest learning curve with the exception of the MIMIC DT, MIMIC RF, and Movie RF. On the other hand, the highest learning curves tend to be the HEX models with low unacceptable feature percentage (i.e., $UAP=0.10$), although this varies somewhat by dataset and method. Surprisingly, however, even when UAP is high, the HEX models are still relatively comparable and at times even outperform models with lower UAP values. This is particularly encouraging since ideally high quality explanations will still be obtained even when considering a decider's preferred features. This may, however, indicate some level of overfitting, which is better assessed via test set performance, provided and discussed in the next subsection.

§.§.§ Testing

We assess test performance of the various explainability methods in two different ways in this simulated, HITL decider scenario. Similar to the decider-free scenario, we first look at average DBD. Second, we examine the average UEP of the various methods to better orient the lens through which we view the average DBD results.

Table <ref> shows the average DBD across various datasets, ML methods, UAPs, and explainability methods; Table <ref> in Appendix F shows the average rewards. We bolden the lowest (best) DBD value among our HEX methods by dataset and ML model, also boldening the lowest DBD value among non-HEX methods. If one of the HEX methods has a lower DBD than the non-HEX methods the result is colored red. If, on the other hand, a non-HEX method has a lower DBD, the result is displayed in blue. Note that DDPG, TD3, LIME, and Grow, do not consider a HITL decider. Evaluations are therefore only shown once per dataset and ML method. Further note that because these methods do not consider a HITL decider they return explanations using all features. This is why we also show the average UEP in Table <ref>, which we discuss next.
3|c|     5c|Model
1|c|Dataset     1c|UAP     1c|Method     NN     SVM     RF     LR     DT
1|c|     1c|     HEX-DDPG     0.305     0.197     0.138     0.36     0.321

1|c|     1c|-2*0.3     HEX-TD3     0.311     0.236     0.137     0.379     0.331

1|c|     1c|     HEX-DDPG     0.303     0.202     0.15     0.35     0.327

1|c|     1c|-2*0.5     HEX-TD3     0.316     0.231     0.152     0.382     0.334

1|c|     1c|     HEX-DDPG     0.335     0.237     0.181     0.372     0.334

1|c|     1c|-2*0.7     HEX-TD3     0.343     0.258     0.179     0.397     0.338
1|c|     1c|     DDPG      0.28      0.173     0.091      0.339      0.31

1|c|     1c|     TD3     0.298     0.212      0.09     0.376     0.337

1|c|     1c|     LIME     0.473     0.461     0.303     0.499     0.446

1|c|-10*Bank     1c|-4*None     Grow     0.476     0.462     0.335     0.495     0.427
1|c|     1c|     HEX-DDPG     0.263     0.103     0.053      0.209     0.376

1|c|     1c|-2*0.3     HEX-TD3     0.293     0.131     0.055     0.226     0.379

1|c|     1c|     HEX-DDPG      0.247      0.081     0.066     0.22     0.376

1|c|     1c|-2*0.5     HEX-TD3     0.268     0.104     0.065     0.231     0.372

1|c|     1c|     HEX-DDPG     0.254     0.09     0.056     0.221      0.366

1|c|     1c|-2*0.7     HEX-TD3     0.265     0.118     0.059     0.221      0.366
1|c|     1c|     DDPG     0.271     0.104      0.046     0.215     0.386

1|c|     1c|     TD3     0.308     0.119     0.055     0.251     0.392

1|c|     1c|     LIME     0.427     0.338     0.154     0.426     0.321

1|c|-10*MIMIC     1c|-4*None     Grow     0.483     0.404     0.236     0.489     0.342
1|c|     1c|     HEX-DDPG     0.183     0.297     0.082     0.419     0.439

1|c|     1c|-2*0.3     HEX-TD3     0.203     0.337     0.076     0.435     0.439

1|c|     1c|     HEX-DDPG     0.173     0.288     0.077     0.416     0.437

1|c|     1c|-2*0.5     HEX-TD3     0.199     0.321     0.081     0.43     0.439

1|c|     1c|     HEX-DDPG      0.172      0.245      0.065      0.388      0.423

1|c|     1c|-2*0.7     HEX-TD3     0.186     0.286     0.066     0.406     0.424
1|c|     1c|     DDPG     0.24     0.316     0.076     0.454     0.438

1|c|     1c|     TD3     0.247     0.356     0.072     0.457     0.433

1|c|     1c|     LIME     0.462     0.5     0.314     0.5     0.49

1|c|-10*Movie     1c|-4*None     Grow     0.45     0.416     0.217     0.49     0.331
1|c|     1c|     HEX-DDPG     0.195     0.205      0.084     0.228     0.366

1|c|     1c|-2*0.3     HEX-TD3     0.195     0.205     0.087     0.229     0.366

1|c|     1c|     HEX-DDPG      0.193     0.206     0.089     0.22     0.341

1|c|     1c|-2*0.5     HEX-TD3     0.195     0.203     0.086      0.218     0.342

1|c|     1c|     HEX-DDPG     0.207     0.21     0.097     0.249     0.336

1|c|     1c|-2*0.7     HEX-TD3     0.209     0.212     0.096     0.249     0.338
1|c|     1c|     DDPG     0.246      0.176     0.092     0.374     0.381

1|c|     1c|     TD3     0.239     0.178     0.092     0.371     0.385

1|c|     1c|     LIME     0.31     0.195     0.101     0.37      0.225

1|c|-10*Student     1c|-4*None     Grow     0.42     0.304     0.126     0.449     0.234
1|c|     1c|     HEX-DDPG     0.384     0.373     0.065     0.285     0.488

1|c|     1c|-2*0.3     HEX-TD3     0.387     0.371     0.067     0.273     0.49

1|c|     1c|     HEX-DDPG     0.381     0.363     0.073     0.286     0.477

1|c|     1c|-2*0.5     HEX-TD3     0.387     0.368     0.072     0.282     0.478

1|c|     1c|     HEX-DDPG     0.381      0.35     0.073     0.288     0.476

1|c|     1c|-2*0.7     HEX-TD3     0.383     0.356     0.072     0.283      0.475
1|c|     1c|     DDPG      0.367     0.425      0.05      0.26     0.493

1|c|     1c|     TD3     0.387     0.429     0.056     0.3     0.496

1|c|     1c|     LIME     0.496     0.448     0.294     0.494     0.498

1|c|-10*News     1c|-4*None     Grow     0.497     0.482     0.271     0.483     0.436
Average DBD for our HITL decider scenario. Bold indicates the lowest DBD among all HEX methods and all non-HEX method on each dataset and ML model and all (HEX/non-HEX considered separately). Blue indicates that a non-HEX method received the lowest overall within-ML method and dataset DBD, while red indicates that a HEX method received the lowest overall within-method and dataset DBD.

First, we are surprised how often one of the HEX methods outperforms the non-HEX methods since the non-HEX methods are free to use the entire feature space to produce explanations, while the HEX methods use 0-distrust projection and are thus limited to certain features. In fact, of the 25 model-dataset combinations, one of the HEX models performs better on 14. Second, among the various HEX results, we are surprised that the lowest UAP (UAP$=0.10$) does not always produce the lowest DBD result. In fact, of the 25 model-dataset experiments, a UAP$=0.10$ model only produces the lowest within-HEX DBD value on 8 of these. This finding suggests that, at times, more human-interpretable solutions may naturally produce better explanatory points.

To further contextualize and orient the lens through which the DBD results of Table <ref> are viewed, we provide Table <ref>, below, which shows the average UEP used to generate explanations. We show the average UEP of each non-HEX method for each UAP scenario generated. The bottom of the table shows the HEX method results for all datasets and UAP values. Since the HEX methods use 0-distrust projection, they never use unacceptable features (i.e., these values are always 0).
3|c|     5c|Model
1|l|Dataset     1c|UAP     Method     NN     SVM     RF     LR     DT
1|c|     1c|     DDPG     0.84     0.84     0.836     0.835     0.835

1|c|     1c|     TD3     0.834     0.836     0.836     0.838     0.837

1|c|     1c|     LIME     0.572     0.572     0.572     0.572     0.572

1|c|     1c|-4*0.3     Grow     0.744     0.744     0.744     0.744     0.744
1|c|     1c|     DDPG     0.757     0.766     0.759     0.755     0.755

1|c|     1c|     TD3     0.755     0.762     0.757     0.761     0.759

1|c|     1c|     LIME     0.651     0.65     0.65     0.65     0.65

1|c|     1c|-4*0.5     Grow     0.616     0.616     0.616     0.616     0.616
1|c|     1c|     DDPG     0.7     0.703     0.693     0.69     0.69

1|c|     1c|     TD3     0.7     0.702     0.699     0.703     0.7

1|c|     1c|     LIME     0.763     0.763     0.763     0.763     0.763

1|c|-12*Bank     1c|-4*0.7     Grow     0.518     0.518     0.518     0.518     0.518
1|c|     1c|     DDPG     0.762     0.763     0.764     0.763     0.764

1|c|     1c|     TD3     0.766     0.765     0.765     0.764     0.765

1|c|     1c|     LIME     0.517     0.517     0.516     0.517     0.516

1|c|     1c|-4*0.3     Grow     0.754     0.754     0.754     0.754     0.754
1|c|     1c|     DDPG     0.612     0.613     0.613     0.613     0.613

1|c|     1c|     TD3     0.612     0.613     0.612     0.612     0.613

1|c|     1c|     LIME     0.6     0.601     0.6     0.6     0.6

1|c|     1c|-4*0.5     Grow     0.6     0.6     0.6     0.6     0.6
1|c|     1c|     DDPG     0.544     0.546     0.546     0.546     0.545

1|c|     1c|     TD3     0.541     0.542     0.542     0.543     0.543

1|c|     1c|     LIME     0.754     0.754     0.754     0.754     0.754

1|c|-12*MIMIC     1c|-4*0.7     Grow     0.515     0.515     0.515     0.515     0.515
1|c|     1c|     DDPG     0.8     0.797     0.799     0.797     0.797

1|c|     1c|     TD3     0.817     0.806     0.806     0.805     0.805

1|c|     1c|     LIME     0.539     0.539     0.539     0.539     0.538

1|c|     1c|-4*0.3     Grow     0.742     0.742     0.742     0.742     0.742
1|c|     1c|     DDPG     0.701     0.697     0.698     0.693     0.693

1|c|     1c|     TD3     0.725     0.709     0.707     0.706     0.705

1|c|     1c|     LIME     0.633     0.633     0.634     0.634     0.633

1|c|     1c|-4*0.5     Grow     0.613     0.613     0.613     0.613     0.613
1|c|     1c|     DDPG     0.618     0.615     0.617     0.611     0.612

1|c|     1c|     TD3     0.651     0.633     0.631     0.63     0.627

1|c|     1c|     LIME     0.755     0.754     0.755     0.754     0.755

1|c|-12*Movie     1c|-4*0.7     Grow     0.512     0.512     0.512     0.512     0.512
1|c|     1c|     DDPG     0.842     0.842     0.841     0.842     0.843

1|c|     1c|     TD3     0.823     0.83     0.828     0.827     0.83

1|c|     1c|     LIME     0.523     0.523     0.523     0.523     0.523

1|c|     1c|-4*0.3     Grow     0.756     0.756     0.756     0.756     0.756
1|c|     1c|     DDPG     0.745     0.745     0.747     0.748     0.75

1|c|     1c|     TD3     0.722     0.737     0.734     0.735     0.736

1|c|     1c|     LIME     0.651     0.651     0.651     0.651     0.651

1|c|     1c|-4*0.5     Grow     0.616     0.616     0.616     0.616     0.616
1|c|     1c|     DDPG     0.67     0.67     0.67     0.671     0.673

1|c|     1c|     TD3     0.642     0.654     0.648     0.649     0.651

1|c|     1c|     LIME     0.77     0.77     0.77     0.77     0.77

1|c|-12*Student     1c|-4*0.7     Grow     0.493     0.493     0.493     0.493     0.493
1|c|     1c|     DDPG     0.869     0.844     0.845     0.846     0.84

1|c|     1c|     TD3     0.844     0.839     0.839     0.839     0.838

1|c|     1c|     LIME     0.491     0.491     0.491     0.491     0.491

1|c|     1c|-4*0.3     Grow     0.744     0.744     0.744     0.744     0.744
1|c|     1c|     DDPG     0.793     0.756     0.759     0.761     0.752

1|c|     1c|     TD3     0.766     0.754     0.753     0.754     0.752

1|c|     1c|     LIME     0.61     0.61     0.61     0.61     0.61

1|c|     1c|-4*0.5     Grow     0.61     0.61     0.61     0.61     0.61
1|c|     1c|     DDPG     0.73     0.684     0.688     0.69     0.678

1|c|     1c|     TD3     0.69     0.676     0.676     0.678     0.676

1|c|     1c|     LIME     0.744     0.744     0.744     0.744     0.744

1|c|-12*News     1c|-4*0.7     Grow     0.491     0.491     0.491     0.491     0.491
1|l|All     1l|All     HEX-DDPG      0      0      0      0      0

1|l|All     1l|All     HEX-TD3      0      0      0      0      0
Average UEP by dataset, ML model, and UAP scenario.

First, Table <ref> shows that the non-HEX methods rely heavily on the UAP-selected features, highlighting the benefits of our proposed HEX methods. Second, a curious artifact discovered in presenting and examining Table <ref> is the uniformity of feature reliance in generating explanations across different machine learning models among both LIME and Growing Spheres – i.e., the UEP is consistent across ML models within dataset and UAP scenario. This result is unsurprising, however, provided how these methods operate. While both LIME and Growing Spheres are model agnostic, they do not explicitly consider the ML model when producing explanations, instead relying on the underlying training data. This finding highlights two things. First, the lack of explicit model consideration is a short-coming of these methods since explanatory models should explain the predictions made by an ML model, rather than the training data. Second, in a federated learning or other limited or reduced training data availability scenario, exploration of the ML model-learned probability space is key to reliably explaining the model. We suspect that UEP uniformity of LIME and Growing Spheres across ML models is partially attributable to the limited, underlying training data and lack of probability space exploration.

§ CONCLUSIONS

In this work we propose a human-in-the-loop (HITL) model-agnostic classification explainability method, adopting a markov decision process (MDP) approach, thus permitting the use of deep reinforcement learning (DRL) methods to synthesize explanatory policies. We augment two state-of-the-art DRL methods, DDPG and TD3, with both HITL and federated learning considerations, further updating the methods based on the relationship we prove to exist between buffer and policy degeneracy. We compare our proposed methods, HEX-DDPG and HEX-TD3, against out-of-the-box DRL methods and two competing state-of-the-art explainability methods, LIME and Growing Spheres. We conduct these evaluations across two different scenarios, one that does not consider a HITL and another that explicitly considers a HITL decider. Both scenarios consider five different well-known machine learning models each of which are trained on five different decision-focused datasets spanning business, medicine, and education. Our experiments show that HEX-DDPG and HEX-TD3 are competitive with and often outperform both baseline DRL methods and competing state-of-the-art explainability methods. Furthermore, our methods frequently outperform both baseline DRL methods, LIME, and Growing Spheres in our HITL decider scenario, even using our proposed $0$-distrust projection method, which restricts explanations to using only decider-trusted features.

§ APPENDIX A: PROOF OF THEOREM <REF>

Proof sketch: The theorem is of the form

\begin{align}
A > B \implies A > B^{\prime}.
\end{align}

By showing that $B^{\prime}$ is directly derived from $B$ – i.e., $B \implies B^{\prime}$ – we show $ \left ( A > B \cap B \implies B^{\prime} \right) \implies A > B^{\prime}$ to be true vacuously. The proof is below.

Assume Buffer Degeneracy holds.
Let $\pi^{\theta^*}_l = \argmin_{\pi}\{ \mathbb{E}_{\mathbf{x} \sim \rho^{\pi}_l, \mathbf{z} \sim \pi^{\theta}, r \sim R} \left[\left(Q \left(\mathbf{x},\mathbf{z} \vert \theta^{Q} \right) - y \right)^2 \right] \}$ and $\pi^{\theta^*}_{l^{\prime}} = \argmin_{\pi}\{ \mathbb{E}_{\mathbf{x} \sim \rho^{\pi}_{l^{\prime}}, \mathbf{z} \sim \pi^{\theta}, r \sim R} \left[\left(Q \left(\mathbf{x},\mathbf{z} \vert \theta^{Q} \right) - y \right)^2 \right] \}$. Let $\rho^{y}_l = \{y:\mathbf{x} \sim \rho^{\pi}_l, \mathbf{z} \sim \pi^{\theta^*}_l, r \sim R\}$ and $\tilde{\rho}^{y}_{l^{\prime}} = \{y: \mathbf{x} \sim \rho^{\pi}_{l}, \mathbf{z} \sim \pi^{\theta^*}_{l^{\prime}}, r \sim R\}$. Since buffer degeneracy holds and $\pi^{\theta^*}_{l^{\prime}}$ is the minimizer of the loss on the degenerate buffer, then

\begin{align}
\mathbb{E}_{y_l \sim \rho^{y}_l}\left[y_l\right] > & \mathbb{E}_{\tilde{y}_{l^{\prime}} \sim \tilde{\rho}^{y}_{l^{\prime}}}\left[\tilde{y}_{l^{\prime}}\right]: l^{\prime} > l,
\end{align}

as desired.

§ APPENDIX B: DATASET DETAILS

To assess our proposed method we select several datasets spanning a variety of decision-focused domains, as well as a dataset that is commonly used to evaluate MLX methods. We provide a brief description of each as follows:

* Bank: A bank solicited customers via a phone call marketing campaign to entice them into subscribing to a term deposit. Predictive models derived from this data can be used to prioritize solicitation of potential customers in subsequent campaigns. The raw data is available at: <https://archive.ics.uci.edu/ml/datasets/bank+marketing> [Moro et al., 2014]; $n=41188$, $p=61$,  $c\in\{\text{subscribe},\text{didn't subscribe}\}$, $\text{count}(c=\text{subscribe})=4640$.

* MIMIC: A healthcare dataset consisting of patients experiencing sepsis. The models derived from this dataset can be used to help categorize patients as low/high risk and thus aid in treatment decisions. The raw data is available at: <https://physionet.org/content/mimiciii-demo/1.4/> [Johnson et al., 2016]; $n=1122$, $p=25$, $c \in \{\text{alive},\text{dead}\}$, $\text{count}(c=\text{dead})=243$.

* Movie: A movie dataset that measures various film characteristics available only during the pre-production phase of development; box office profitability as the outcome. ML models can be used to help potential investors decided whether to invest a film or not. Obtained from [Lash and Zhao, 2016]; $n=2506$, $p=119$, $c \in \{\text{profitable},\text{not profitable}\}$, $\text{count}(c=\text{profitable})=757$.

* News: A text-based dataset consisting of different newsgroup posts. We adopt the posts from the Christianity and atheism newsgroups as in [Ribeiro et al., 2016]. Features are the 300 most common corpus terms. The raw data is available at: <https://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups> [Joachims, 1996]; $n=1796$, $p=300$, $c \in \{\text{Christianity},\text{atheism}\}$, $\text{count}(c=\text{Christianity})=997$.

* Student: A dataset consisting of Portuguese high school students with final grades as outcomes. Predictive models can be used to prioritize/decide which students may need extra assistance. The raw data is available at: <https://archive.ics.uci.edu/ml/datasets/student+performance> [Cortez and Silva, 2008]; $n=650$, $p=43$, $c \in \{\text{good grade},\allowbreak \text{not good grade}\}$, $\text{count}(c=\text{not good grade})=455$.

§ APPENDIX C: MACHINE LEARNING MODEL TRAINING AND EVALUATION

We conduct our experiments using five different machine learning models, each of which is trained on each of our selected datasets. We partition each dataset randomly into training, validation, and testing, where 70%, 15%, and 15% of the full dataset are allocated to each partition, respectively. The training set is used to train models, the validation set is used for parameter selection and training the reinforcement learning models, and the test set is used to assess explainability.

We perform a grid search to optimize predictive model performance, depending on the model (i.e., if hyperparameters are available). We provide the following details for each model and summarize the search space for each, where relevant:

* LR: Optimized using ADAM with an initial learning rate of $0.01$, a batch size of 64, a maximum of 4000 epochs, with early stopping and a patience value of 200.

* NN: Fully connected layers, with relu activation, a single hidden layer, a batch size of 64, a maximum of 4000 epochs, with early stopping and a patience value of 200, optimized used ADAM with an initial learning rate of $0.01$. We search across the following hidden node numbers: 3, 5, 10, 25, 50, 100.

* SVM: We perform a search over the kernel, where $kernel \in \{\text{linear},\text{RBF},\text{polynomial}\}$. A degree of 2 was used with the polynomial kernel and the gamma parameter of the RBF kernel was set to $\frac{1}{p}$. We further impose miss-classification penalties on the loss according to proportional class imbalance – i.e., the penalty imposed on positive miss-classifications is $p_{c^{+}}=\frac{\vert \mathbf{c} \vert }{\sum_{c \in \mathbf{c}^-} 1}$ and $p_{c^{-}}= p_{c^{+}}^{-1}$.

* DT: We employ CART, using the gini index as the splitting criteria with a minimum number of leaf nodes of 2. We search across the following maximum depth values: 5, 10 , 30, 50. We also impose miss-classification penalties in a similar, but slightly different way to that of SVMs, where $p_{c^{+}}=\frac{\sum_{c \in \mathbf{c}^-} 1}{\vert \mathbf{c} \vert }$ and $p_{y^{-}}= 1 - p_{y^{+}}$.

* RF: We use a bag of CART classifiers described by DT, above, performing a full grid search across two parameters: maximum depth and the number of weak learners. We search across the same max depth values as that of DT and the following number of weak learners: 50, 100, 200, 400, 600. We also impose the same miss-classification penalties as that of DT.

We report the training, validation, and test set performance on each of the optimized classification models, below.

1l|2*     2c|Train                    2c|Validation              2cTest
1l|                      Accuracy      1l|AUC          Accuracy     1l|AUC          Accuracy        AUC
1l|LR                    0.9726986     1l|0.911175     0.973934     1l|0.912431     0.972485        0.912269

1l|NN                    0.9787762     1l|0.917349     0.978133     1l|0.912269     0.97627         0.912269

1l|DT                    0.8480855     1l|0.881607     0.84202      1l|0.87335      0.849142        0.880499

1l|RF                    0.8788152     1l|0.912631     0.865976     1l|0.884968     0.870346        0.876768

1l|SVM                   0.9423904     1l|0.785564     0.905147     1l|0.677491     0.909032        0.687833

1l|2*     2c|Train                   2c|Validation              2cTest
1l|                      Accuracy     1l|AUC          Accuracy     1l|AUC          Accuracy        AUC
1l|LR                    0.941238     1l|0.877551     0.91674      1l|0.840237     0.955639        0.905325

1l|NN                    0.949067     1l|0.882653     0.920311     1l|0.846154     0.944155        0.905325

1l|DT                    0.859694     1l|0.863633     0.757396     1l|0.65991      0.757396        0.679361

1l|RF                    0.936224     1l|0.91462      0.828402     1l|0.734541     0.881657        0.836712

1l|SVM                   0.922194     1l|0.829096     0.840237     1l|0.693489     0.87574         0.77457

1l|2*     2c|Train                   2c|Validation              2cTest
1l|                      Accuracy     1l|AUC          Accuracy     1l|AUC          Accuracy        AUC
1l|LR                    0.900309     1l|0.812999     0.903491     1l|0.819149     0.873911        0.829787

1l|NN                    0.912744     1l|0.831243     0.849536     1l|0.789894     0.805978        0.726064

1l|DT                    0.855188     1l|0.829737     0.829787     1l|0.793625     0.816489        0.776651

1l|RF                    0.8626       1l|0.816247     0.848404     1l|0.787164     0.837766        0.784485

1l|SVM                   0.794185     1l|0.676512     0.81117      1l|0.698507     0.803191        0.692782

1l|2*     2c|Train                   2c|Validation              2cTest
1l|                      Accuracy     1l|AUC          Accuracy     1l|AUC          Accuracy        AUC
1l|LR                    0.99691      1l|0.974522     0.989095     1l|0.948148     0.986927        0.937037

1l|NN                    0.998119     1l|0.984873     0.989376     1l|0.951852     0.98011         0.937037

1l|DT                    0.944268     1l|0.938628     0.925926     1l|0.92         0.892593        0.883333

1l|RF                    0.984076     1l|0.982111     0.959259     1l|0.955833     0.92963         0.921667

1l|SVM                   0.999204     1l|0.999106     0.977778     1l|0.976667     0.948148        0.944167

1l|2*     2c|Train                   2c|Validation              2cTest
1l|                      Accuracy     1l|AUC          Accuracy     1l|AUC          Accuracy        AUC
1l|LR                    0.865161     1l|0.769231     0.841747     1l|0.742268     0.79966         0.742268

1l|NN                    0.887303     1l|0.782418     0.831013     1l|0.752577     0.786003        0.71134

1l|DT                    0.683516     1l|0.751095     0.701031     1l|0.757099     0.628866        0.666075

1l|RF                    0.828571     1l|0.856652     0.804124     1l|0.82074      0.690722        0.680527

1l|SVM                   0.87033      1l|0.821052     0.71134      1l|0.665568     0.670103        0.576826

§ APPENDIX D: ADDITIONAL EXPERIMENT MODEL PARAMETERIZATION DETAILS

Several additional experiment details are provided below, including parameterization of the different models attempted. All code was written and executed in Python 3.6.

* DRL Models: We generally found two hidden layers with 50 hidden units each to work well across our various datasets and DRL methods and therefore adopted this parameterization for all DRL models. Each model was optimized using the Adam optimizer with default parameterization. Models were constructed, trained, etc. using Tensorflow 2.4.

* LIME: We obtained the code from the [Ribeiro et al., 2016] Github repository at <https://github.com/marcotcr/lime> (installed using PyPI). We found the following parameterized to work well and adopted it for use with all models and datasets: $num\_samples=300, num\_features=p$.

* Grow: We obtained the code from [Laugel et al., 2018] Github repository at <https://github.com/thibaultlaugel/growingspheres>. We found the following parameterization to work well and adopted it for use with all ML models and datasets: $n\_in\_layer=200$, $first\_radius=1.1$, $dicrease\_radius=2.0$.

§ APPENDIX E: DECIDER-FREE TESTING REWARDS

Table <ref>, below, shows the average reward over 10 trials, with 100 test instances in each trial, by explainability method, ML method, and dataset for our “decider free“ explainability scenario. Bold shows the highest reward obtain by an explainability method for each ML model-dataset combination.
2|c|     5c|Model
1|l|Dataset     1c|Method     1cNN     1cSVM     1cRF     1cLR     1c|DT
1|c|6*Bank     DDPG     8.185     8.291     8.188     7.934     8.137

1|c|     Our DDPG     8.453     8.54     8.756     8.207     8.367

1|c|     TD3     7.941     8.006     8.11     7.642     7.874

1|c|     Our TD3     8.216     8.271     8.564     7.959     8.104

1|c|     LIME     -1.392     -0.982     -0.693     -1.197     -1.113

1|c|     Grow     -3.535     -2.451     -1.751     -3.718     -2
1|c|6*MIMIC-3     DDPG     8.072     7.982     8.545     8.004     7.92

1|c|     Our DDPG     7.936     8.474     8.545     7.902     7.976

1|c|     TD3     7.943     7.899     8.367     7.955     7.735

1|c|     Our TD3     7.892     8.255     8.359     7.95     7.765

1|c|     LIME     -1.452     -0.734     0.1     -1.158     1.463

1|c|     Grow     -3.445     -2.151     -1.538     -3.725     -1.724
1|c|6*Movie     DDPG     6.114     7.752     8.174     2.782     7.465

1|c|     Our DDPG     7.837     7.876     8.312     7.5     7.577

1|c|     TD3     6.139     7.291     7.831     2.937     7.134

1|c|     Our TD3     7.623     7.411     7.982     7.121     7.256

1|c|     LIME     -0.737     -1.82     -1.167     -2.287     -1.21

1|c|     Grow     -4.156     -1.995     -1.32     -3.019     -1.63
1|c|6*Student     DDPG     1.774     2.786     4.342     0.635     7.889

1|c|     Our DDPG     5.166     2.306     5.869     5.005     6.575

1|c|     TD3     1.775     2.789     4.568     0.712     7.708

1|c|     Our TD3     5.365     2.505     5.898     5.139     6.769

1|c|     LIME     1.416     -0.713     -1.117     2.086     -0.249

1|c|     Grow     -4.186     -2.402     -1.853     -3.589     -1.813
1|c|6*News     DDPG     7.003     4.149     7.717     7.24     6.93

1|c|     Our DDPG     4.786     4.6     7.831     5.928     6.985

1|c|     TD3     6.473     4.122     7.298     6.766     6.396

1|c|     Our TD3     5.08     4.657     7.509     6.179     6.659

1|c|     LIME     1.971     0.671     2.378     2.402     3.205

1|c|     Grow     -5.433     -3.544     -2.133     -5.382     -2.19
Average test rewards. Bold shows the largest average reward on each dataset-ML model combination.

§ APPENDIX F: HITL DECIDER TESTING REWARDS

Table <ref>, below, shows the results of our HITL decider scenario in terms of average reward taken over 10 trials, with 100 test instances in each trial, by explainability method, ML method, and dataset. Results for our HEX methods are also  shown by UAP scenario. Results for the out-of-the-box DRL, LIME, and Grow methods are shown without considering the UAP since these methods do not handle such considerations.
3|c|     5c|Model
1|l|Dataset     1l|% Corrupt     Method     NN     SVM     RF     LR     DT
1|c|     1c|     GS-DDPG     6.718     7.117     6.75     5.991     6.896

1|c|     1c|-2*0.3     GS-TD3     6.597     6.834     6.627     5.84     6.812
1|c|     1c|     GS-DDPG     6.358     6.755     5.961     5.558     6.122

1|c|     1c|-2*0.5     GS-TD3     6.207     6.707     5.91     5.301     6.016
1|c|     1c|     GS-DDPG     4.618     5.266     4.022     3.863     4.757

1|c|     1c|-2*0.7     GS-TD3     4.548     5.221     4.065     3.669     4.727
1|c|     1c|     DDPG      8.185      8.291      8.188      7.934      8.137

1|c|     1c|     TD3     7.941     8.006     8.11     7.642     7.874

1|c|     1c|     LIME     -1.392     -0.982     -0.693     -1.197     -1.113

1|c|-10*Bank     1c|-4*None     Grow     -3.535     -2.451     -1.751     -3.718     -2
1|c|     1c|     GS-DDPG      8.074     8.787     8.222     7.788     7.702

1|c|     1c|-2*0.3     GS-TD3     8.062     8.593     8.186     7.86     7.654
1|c|     1c|     GS-DDPG     7.623     8.978     5.788     6.888     6.597

1|c|     1c|-2*0.5     GS-TD3     7.623     8.841     5.845     6.93     6.556
1|c|     1c|     GS-DDPG     7.838      9.074     7.504     7.458     6.818

1|c|     1c|-2*0.7     GS-TD3     7.933     8.952     7.527     7.714     6.832
1|c|     1c|     DDPG     8.072     7.982      8.545      8.004      7.92

1|c|     1c|     TD3     7.943     7.899     8.367     7.955     7.735

1|c|     1c|     LIME     -1.452     -0.734     0.1     -1.158     1.463

1|c|-10*MIMIC     1c|-4*None     Grow     -3.445     -2.151     -1.538     -3.725     -1.724
1|c|     1c|     GS-DDPG     8.077     8.218      8.26     7.725     7.443

1|c|     1c|-2*0.3     GS-TD3     7.999     7.841     8.021     7.508     7.269
1|c|     1c|     GS-DDPG     8.237     8.387     8.146     8.012     7.197

1|c|     1c|-2*0.5     GS-TD3     8.039     8.072     7.925     7.742     7.023
1|c|     1c|     GS-DDPG      8.352      8.622     7.76      8.081     6.609

1|c|     1c|-2*0.7     GS-TD3     8.323     8.305     7.756     7.842     6.492
1|c|     1c|     DDPG     6.114     7.752     8.174     2.782      7.465

1|c|     1c|     TD3     6.139     7.291     7.831     2.937     7.134

1|c|     1c|     LIME     -0.737     -1.82     -1.167     -2.287     -1.21

1|c|-10*Movie     1c|-4*None     Grow     -4.156     -1.995     -1.32     -3.019     -1.63
1|c|     1c|     GS-DDPG     5.299     2.521      5.51     5.386     6.12

1|c|     1c|-2*0.3     GS-TD3      5.347     2.461     5.427     5.469     6.28
1|c|     1c|     GS-DDPG     5.269     2.394     4.671     5.445     5.249

1|c|     1c|-2*0.5     GS-TD3     5.233     2.444     4.883      5.553     5.15
1|c|     1c|     GS-DDPG     4.503     1.915     4.268     4.283     4.612

1|c|     1c|-2*0.7     GS-TD3     4.527     2.003     4.224     4.308     4.661
1|c|     1c|     DDPG     1.774     2.786     4.342     0.635      7.889

1|c|     1c|     TD3     1.775      2.789     4.568     0.712     7.708

1|c|     1c|     LIME     1.416     -0.713     -1.117     2.086     -0.249

1|c|-10*Student     1c|-4*None     Grow     -4.186     -2.402     -1.853     -3.589     -1.813
1|c|     1c|     GS-DDPG     4.436     4.811      7.626     5.675     6.812

1|c|     1c|-2*0.3     GS-TD3     4.594      4.905     7.515      6.005     6.678
1|c|     1c|     GS-DDPG     3.98     4.743     6.902     5.439     6.324

1|c|     1c|-2*0.5     GS-TD3     3.979     4.692     7.026     5.448     6.326
1|c|     1c|     GS-DDPG     3.733     4.674     6.57     5.156     5.123

1|c|     1c|-2*0.7     GS-TD3     3.871     4.724     6.65     5.332     5.138
1|c|     1c|     DDPG      7.003     4.149     7.717      7.24      6.93

1|c|     1c|     TD3     6.473     4.122     7.298     6.766     6.396

1|c|     1c|     LIME     1.971     0.671     2.378     2.402     3.205

1|c|-10*News     1c|-4*None     Grow     -5.433     -3.544     -2.133     -5.382     -2.19
HITL scenario test rewards. Bold indicates the lowest reward among all HEX methods and all non-HEX methods by dataset and ML model (HEX/non-HEX considered separately). Blue indicates that a non-HEX method received the lowest overall within-ML method reward, while red indicates that a HEX method received the lowest overall within-method reward.

[Adler et al., 2018]
Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck,
Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian.
Auditing black-box models for indirect influence.
Knowledge and Information Systems, 540 (1):0
95–122, 2018.

[Andrzejak et al., 2013]
Artur Andrzejak, Felix Langner, and Silvestre Zabala.
Interpretable models from distributed data via merging of decision
In 2013 IEEE Symposium on Computational Intelligence and Data
Mining (CIDM), pages 1–9. IEEE, 2013.

[Barua et al., 2012]
Sukarna Barua, Md Monirul Islam, Xin Yao, and Kazuyuki Murase.
Mwmote–majority weighted minority oversampling technique for
imbalanced data set learning.
IEEE Transactions on Knowledge and Data Engineering,
260 (2):0 405–425, 2012.

[Beaudouin et al., 2020]
Valérie Beaudouin, Isabelle Bloch, David Bounie, Stéphan
Clémençon, Florence d'Alché Buc, James Eagan, Winston Maxwell,
Pavlo Mozharovskyi, and Jayneel Parekh.
Flexible and context-specific ai explainability: a multidisciplinary
Available at SSRN 3559477, 2020.

[Biecek and Burzykowski, 2021]
Przemyslaw Biecek and Tomasz Burzykowski.
Explanatory model analysis: Explore, explain and examine
predictive models.
Chapman and Hall/CRC, 2021.

[Burkart and Huber, 2021]
Nadia Burkart and Marco F Huber.
A survey on the explainability of supervised machine learning.
Journal of Artificial Intelligence Research, 70:0
245–317, 2021.

[Chawla et al., 2002]
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.
Smote: Synthetic minority over-sampling technique.
Journal of Artificial Intelligence Research, 16:0
321–357, 2002.

[Cortez and Embrechts, 2011]
Paulo Cortez and Mark J Embrechts.
Opening black box data mining models using sensitivity analysis.
In 2011 IEEE Symposium on Computational Intelligence and Data
Mining (CIDM), pages 341–348. IEEE, 2011.

[Cortez and Silva, 2008]
Paulo Cortez and Alice Maria Gonçalves Silva.
Using data mining to predict secondary school student performance.

[Craven and Shavlik, 1995]
Mark Craven and Jude Shavlik.
Extracting tree-structured representations of trained networks.
In Advances in Neural Information Processing Systems
(NeurIps), volume 8, 1995.

[Datta et al., 2016]
Anupam Datta, Shayak Sen, and Yair Zick.
Algorithmic transparency via quantitative input influence: Theory and
experiments with learning systems.
In 2016 IEEE Symposium on Security and Privacy (SP), pages
598–617. IEEE, 2016.

[De et al., 2021]
Abir De, Nastaran Okati, Ali Zarezade, and Manuel Gomez-Rodriguez.
Classification under human assistance.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 35, pages 5905–5913, 2021.

[Doshi-Velez et al., 2017]
Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David
O'Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, et al.
Accountability of ai under the law: The role of explanation.
arXiv preprint arXiv:1711.01134, 2017.

[Druce et al., 2021]
Jeff Druce, James Niehaus, Vanessa Moody, David Jensen, and Michael L Littman.
Brittle ai, causal confusion, and bad mental models: Challenges and
successes in the xai program.
arXiv preprint arXiv:2106.05506, 2021.

[Fujimoto et al., 2018]
Scott Fujimoto, Herke Hoof, and David Meger.
Addressing function approximation error in actor-critic methods.
In Proceedings of the 35th International Conference on Machine
Learning (ICML), pages 1587–1596. PMLR, 2018.

[Goldstein et al., 2015]
Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin.
Peeking inside the black box: Visualizing statistical learning with
plots of individual conditional expectation.
Journal of Computational and Graphical Statistics, 240
(1):0 44–65, 2015.

[Hall et al., 2017]
Patrick Hall, Navdeep Gill, Megan Kurka, and Wen Phan.
Machine learning interpretability with h2o driverless ai.
H2O. ai, 2017.

[Hara and Hayashi, 2018]
Satoshi Hara and Kohei Hayashi.
Making tree ensembles interpretable: A bayesian model selection
In International Conference on Artificial Intelligence and
Statistics (AISTATS), pages 77–85. PMLR, 2018.

[Henelius et al., 2014]
Andreas Henelius, Kai Puolamäki, Henrik Boström, Lars Asker, and
Panagiotis Papapetrou.
A peek into the black box: exploring classifiers by randomization.
Data Mining and Knowledge Discovery, 280 (5):0
1503–1529, 2014.

[Hinton and Frosst, 2017]
Geoffrey Hinton and Nicholas Frosst.
Distilling a neural network into a soft decision tree.
In Comprehensibility and Explanation in AI and ML (CEX), 2017.

[Hitzler and Sarker, 2022]
P Hitzler and MK Sarker.
Human-centered concept explanations for neural networks.
Frontiers in Artificial Intelligence and Applications,
342:0 337, 2022.

[Hornik et al., 1989]
Kurt Hornik, Maxwell Stinchcombe, and Halbert White.
Multilayer feedforward networks are universal approximators.
Neural Networks, 20 (5):0 359–366, 1989.

[Joachims, 1996]
Thorsten Joachims.
A probabilistic analysis of the rocchio algorithm with tfidf for text
Technical report, Dept of Computer Science, Carnegie Mellon
University, Pittsburgh, PA, 1996.

[Johnson et al., 2016]
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng,
Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and
Roger G Mark.
Mimic-iii, a freely accessible critical care database.
Scientific Data, 30 (1):0 1–9, 2016.

[Kim et al., 2014]
Been Kim, Cynthia Rudin, and Julie A Shah.
The bayesian case model: A generative approach for case-based
reasoning and prototype classification.
volume 27, 2014.

[Kim et al., 2016]
Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo.
Examples are not enough, learn to criticize! criticism for
In Advances in Neural Information Processing Systems
(NeurIps), 2016.

[Kindermans et al., 2017]
Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert
Müller, Dumitru Erhan, Been Kim, and Sven Dähne.
Learning how to explain neural networks: Patternnet and
arXiv preprint arXiv:1705.05598, 2017.

[Konda and Tsitsiklis, 2000]
Vijay R Konda and John N Tsitsiklis.
Actor-critic algorithms.
In Advances in Neural Information Processing Systems
(NeurIps), pages 1008–1014. Citeseer, 2000.

[Lakkaraju et al., 2019]
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec.
Faithful and customizable explanations of black box models.
In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics,
and Society, pages 131–138, 2019.

[Lash and Zhao, 2016]
Michael T Lash and Kang Zhao.
Early predictions of movie success: The who, what, and when of
Journal of Management Information Systems, 330
(3):0 874–903, 2016.

[Lash et al., 2017]
Michael T Lash, Qihang Lin, Nick Street, Jennifer G Robinson, and Jeffrey
Generalized inverse classification.
In Proceedings of the 2017 SIAM International Conference on
Data Mining (SDM), pages 162–170. SIAM, 2017a.

[Lash et al., 2017]
Michael T Lash, Qihang Lin, W Nick Street, and Jennifer G Robinson.
A budget-constrained inverse classification framework for smooth
In 2017 IEEE International Conference on Data Mining Workshops
(ICDMW), pages 1184–1193. IEEE, 2017b.

[Laugel et al., 2018]
Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and
Marcin Detyniecki.
Comparison-based inverse classification for interpretability in
machine learning.
In International Conference on Information Processing and
Management of Uncertainty in Knowledge-Based Systems, pages 100–111.
Springer, 2018.

[Lillicrap et al., 2016]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom
Erez, Yuval Tassa, David Silver, and Daan Wierstra.
Continuous control with deep reinforcement learning.
In 2016 International Conference on Learning Representations
(ICLR), 2016.

[Linardatos et al., 2020]
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis.
Explainable ai: A review of machine learning interpretability
Entropy, 230 (1):0 18, 2020.

[Lipton, 2018]
Zachary C. Lipton.
The mythos of model interpretability: In machine learning, the
concept of interpretability is both important and slippery.
Queue, 160 (3):0 31–57, jun 2018.
ISSN 1542-7730.
URL <https://doi.org/10.1145/3236386.3241340>.

[Liu et al., 2008]
Xu-Ying Liu, Jianxin Wu, and Zhi-Hua Zhou.
Exploratory undersampling for class-imbalance learning.
IEEE Transactions on Systems, Man, and Cybernetics, Part B
(Cybernetics), 390 (2):0 539–550, 2008.

[Liu et al., 2021]
Yang Liu, Tao Fan, Tianjian Chen, Qian Xu, and Qiang Yang.
Fate: An industrial grade platform for collaborative learning with
data protection.
Journal of Machine Learning Research, 220
(226):0 1–6, 2021.

[Lundberg and Lee, 2017]
Scott M Lundberg and Su-In Lee.
A unified approach to interpreting model predictions.
In Advances in Neural Information Processing Systems
(NeurIps), volume 30, 2017.

[Marr, 2019]
Bernard Marr.
Artificial intelligence in practice: how 50 successful
companies used AI and machine learning to solve problems.
John Wiley & Sons, 2019.

[Martens and Provost, 2014]
David Martens and Foster Provost.
Explaining data-driven document classifications.
Management Information Systems Quarterly, 380
(1):0 73–100, 2014.

[Melo et al., 2008]
Francisco S Melo, Sean P Meyn, and M Isabel Ribeiro.
An analysis of reinforcement learning with function approximation.
In Proceedings of the 25th International Conference on Machine
Learning (ICML), pages 664–671, 2008.

[Mnih et al., 2013]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis
Antonoglou, Daan Wierstra, and Martin Riedmiller.
Playing atari with deep reinforcement learning.
In Advances in Neural Information Processing Systems
(NeurIps), 2013.

[Montavon et al., 2018]
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.
Methods for interpreting and understanding deep neural networks.
Digital Signal Processing, 73:0 1–15, 2018.

[Moro et al., 2014]
Sérgio Moro, Paulo Cortez, and Paulo Rita.
A data-driven approach to predict the success of bank telemarketing.
Decision Support Systems, 62:0 22–31, 2014.

[Neri et al., 2020]
Emanuele Neri, Francesca Coppola, Vittorio Miele, Corrado Bibbolino, and
Roberto Grassi.
Artificial intelligence: Who is responsible for the diagnosis?
La Radiologia Medica, 125:0 517–521, 2020.

[Platt, 1999]
John Platt.
Probabilistic outputs for support vector machines and comparisons to
regularized likelihood methods.
Advances in Large Margin Classifiers, 100
(3):0 61–74, 1999.

[Plumb et al., 2018]
Gregory Plumb, Denali Molitor, and Ameet S Talwalkar.
Model agnostic supervised local explanations.
In Advances in Neural Information Processing Systems
(NeurIps), volume 31, 2018.

[Ribeiro et al., 2016]
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.
“why should i trust you?” explaining the predictions of any
In Proceedings of the 22nd ACM International Conference on
Knowledge Discovery and Data Mining (KDD), pages 1135–1144, 2016.

[Salehkaleybar et al., 2021]
Saber Salehkaleybar, Arsalan Sharifnassab, and S Jamaloddin Golestani.
One-shot federated learning: theoretical limits and algorithms to
achieve them.
Journal of Machine Learning Research, 220
(189):0 1–47, 2021.

[Samek et al., 2017]
Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller.
Explainable artificial intelligence: Understanding, visualizing and
interpreting deep learning models.
arXiv preprint arXiv:1708.08296, 2017.

[Samoilescu et al., 2021]
Robert-Florian Samoilescu, Arnaud Van Looveren, and Janis Klaise.
Model-agnostic and scalable counterfactual explanations via
reinforcement learning.
arXiv preprint arXiv:2106.02597, 2021.

[Schetinin et al., 2007]
Vitaly Schetinin, Jonathan E Fieldsend, Derek Partridge, Timothy J Coats,
Wojtek J Krzanowski, Richard M Everson, Trevor C Bailey, and Adolfo
Confident interpretation of bayesian decision tree ensembles for
clinical applications.
IEEE Transactions on Information Technology in Biomedicine,
110 (3):0 312–319, 2007.

[Schulman et al., 2015]
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp
Trust region policy optimization.
In Proceedings of the 32nd International Conference on Machine
Learning (ICML), pages 1889–1897. PMLR, 2015.

[Shrikumar et al., 2017]
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.
Learning important features through propagating activation
In Proceedings of the 34th International Conference on Machine
Learning (ICML), pages 3145–3153. PMLR, 2017.

[Silver et al., 2014]
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and
Martin Riedmiller.
Deterministic policy gradient algorithms.
In Proceedings of the 31st International Conference on Machine
Learning (ICML), pages 387–395. PMLR, 2014.

[Smilkov et al., 2017]
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin
Smoothgrad: removing noise by adding noise.
In 2017 International Conference on Machine Learning Workshop
on Visualization for Deep Learning, 2017.

[Sutton et al., 1998]
Richard S Sutton, Andrew G Barto, et al.
Introduction to reinforcement learning, volume 135.
MIT press Cambridge, 1998.

[Sutton et al., 1999]
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.
Policy gradient methods for reinforcement learning with function
Advances in Neural Information Processing Systems (NeurIps),
12, 1999.

[Tsai et al., 2019]
Chih-Fong Tsai, Wei-Chao Lin, Ya-Han Hu, and Guan-Ting Yao.
Under-sampling class imbalanced datasets by combining clustering
analysis and instance selection.
Information Sciences, 477:0 47–54, 2019.

[Tsirtsis et al., 2021]
Stratis Tsirtsis, Abir De, and Manuel Rodriguez.
Counterfactual explanations in sequential decision making under
Advances in Neural Information Processing Systems, 34, 2021.

[Wang et al., 2021]
Rui Wang, Xiaoqian Wang, and David I Inouye.
Shapley explanation networks.
arXiv preprint arXiv:2104.02297, 2021.

[Wang and Wang, 2022]
Yipei Wang and Xiaoqian Wang.
A unified study of machine learning explanation evaluation metrics.
arXiv preprint arXiv:2203.14265, 2022.

[Watkins and Dayan, 1992]
Christopher JCH Watkins and Peter Dayan.
Machine Learning, 80 (3-4):0 279–292, 1992.

[Xu et al., 2014]
Xin Xu, Lei Zuo, and Zhenhua Huang.
Reinforcement learning algorithms with function approximation: Recent
advances and applications.
Information Sciences, 261:0 1–31, 2014.

[Yeh and Ravikumar, 2021]
Chih-Kuan Yeh and Pradeep Ravikumar.
Objective criteria for explanations of machine learning models.
Applied AI Letters, page e57, 2021.

[Yen and Lee, 2009]
Show-Jane Yen and Yue-Shi Lee.
Cluster-based under-sampling approaches for imbalanced data
Expert Systems with Applications, 360 (3):0
5718–5727, 2009.

[Zhao et al., 2018]
Xun Zhao, Yanhong Wu, Dik Lun Lee, and Weiwei Cui.
iforest: Interpreting random forests via visual analytics.
IEEE Transactions on Visualization and Computer Graphics,
250 (1):0 407–416, 2018.