2021

[1]Maryam Hashemi

[1]Computer Science and Engineering, University of New South Wales,
Kensington, Sydney, 2033, NSW, Australia

#  Who wants what and how: a Mapping Function for Explainable Artificial
Intelligence

<EMAIL_ADDRESS>*

###### Abstract

The increasing complexity of AI systems has led to the growth of the field of
explainable AI (XAI), which aims to provide explanations and justifications
for the outputs of AI algorithms. These methods mainly focus on feature
importance and identifying changes that can be made to achieve a desired
outcome. Researchers have identified desired properties for XAI methods, such
as plausibility, sparsity, causality, low run-time, etc.

The objective of this study is to conduct a review of existing XAI research
and present a classification of XAI methods. The study also aims to connect
XAI users with the appropriate method and relate desired properties to current
XAI approaches. The outcome of this study will be a clear strategy that
outlines how to choose the right XAI method for a particular goal and user and
provide a personalized explanation for users.

###### keywords:

Explainable AI, Counter Factual Explanations, and Users.

##  1 Introduction

Artificial Intelligence algorithms have great potential to be used in our
daily lives. Because of these algorithms’ complexity and black-box nature,
there have been numerous studies in the field of explainable AI (XAI). The
motivation behind XAI is to explain how an AI algorithm works in a human-
understandable way. Although XAI is highly in demand, there is no study on
knowing the users of AI, the difference between various methods, and mapping
each user to a specific method.

This work is an observational study on users of explainable AI (XAI), desired
properties for XAI methods, and missions that XAI has. Through this paper, we
want to answer this question: who wants what from XAI, and how should we
deliver it? To do so, we start by exploring this field’s roots and milestones.
It will help us to see shortages better and have a comprehensive review.

To explain the AI black box, two general strategies have been selected. The
first approach is replacing the black-box algorithm with a white-box
algorithm. Since white-box algorithms are much simpler than black-box ones,
they usually have lower performance. The second approach is opening the black-
box algorithm and justifying the system’s output. This work is dedicated to
the second approach. We categorized research in the XAI into three categories:
philosophy, practice, and theory. Philosophy tries to define an explanation
and how a human mind processes and understands an explanation. The theory part
will provide the math behind the explanation and formulate this problem.
Practice studies challenges of XAI in practice.

###  1.1 Paper Outlines

The next section will review the previous works from three perspectives:
philosophy, theory, and practice. In section 3, we discuss the objectives and
motivations behind using XAI. Section 4 is dedicated to explaining our mapping
function and is our main contribution. The last section is the conclusion and
future work.

##  2 Previous Works

###  2.1 Philosophy

In this category of studies, scholars discussed the concept of explanations
and tried to set a definition for the terms ”explanations”, ”transparency”,
and ”counterfactual explanation” from the computer science perspective. In the
following, we will review the works that answer basic questions in this area:
1- What is an explanation? 2- What is explainable AI? 3- What are
counterfactual explanations?

Xu et al. xu2019explainable  responded to the basic question of what an
explanation from a scientific perspective is? They named two parts for a valid
explanation: the object we want to explain and the content regarding the
object that explains it.

Doran et al. https://doi.org/10.48550/arxiv.1710.00794  separated four
concepts that have been used in this field together: opaque systems,
interpretable systems, comprehensible systems, and explainable systems. They
defined truly explainable systems as having automated reasoning as a part of
output without needing human post-processing.

Hoffman et al. https://doi.org/10.48550/arxiv.1812.04608  took one step beyond
explanations and tried to define mental models such as user understanding for
explainable AI. Their study is focused on metrics that assure us users
understand explanations well.

Byrne byrne2019counterfactuals  searched for the nature of counterfactual
explanations. The author discussed the counterfactual structure, relations,
content, and inferences in this paper. For example, it has been mentioned that
counterfactuals can be additive or subtractive. In additive CFs (Counter
Factual), ”a new situation that could happen but didn’t” will be added to
explanations. For example, in the case of auto-drive cars, we can say, “if the
car had detected the pedestrian earlier and braked, the passenger would not
have been injured”. On the other hand, the subtractive counterfactual delete a
fact that already happened to reach the desired output, “if the car had not
swerved and hit the wall, the passenger would not have been injured”. The
other concept that the author has pointed out is that counterfactual and
causal inferences are related to each other, and they called it ”two sides of
the same coin” hume1978treatise . The author believes that CFs intensify the
causal link between an action and its outcome. In the case of CF content, the
author says that in CFs, we are making changes in actions rather than
inactions. Consider two clients: one has shares in company A, and wants to
move to another company, name it B, and switch to it, and she loses $1,000;
the other is in Company B, considering going to Company A, but he decides to
stay in company A, and also loses $1,000. People created counterfactuals
focused on the person who acted, “if only she hadn’t switched…”Daniel .

Byrne byrne2019counterfactuals  did the argument to conclude we have various
types of explanations with different impacts. Still, to maximize the effect of
CFs, computer science needs to ask for the help of cognitive science.

In 2020, Amann et al. Amann2020  published a paper on the case of the need for
XAI in medicine from the following perspectives. The legal perspective,
medical perspective, patient perspective, and ethical perspective. In 2022,
Amann et al. Amann2022  reviewed opinions to support or reject the
explainability of the AI-powered Clinical Decision Support System (CDSS). They
reviewed statements based on technical considerations, human factors, and the
designated system’s role in decision-making.

The next work tries to bridge between the philosophical side and the computer
science side and review recent works in this field to find shortages
keane2021if . The authors mentioned that Counter Factual Explanations (CFEs)
can convey more information than other XAI methods. However, there are
significant shortages in generating CFEs so far. The first one is that most of
the works didn’t assess their proposed algorithms on real-world systems. Keane
et al. keane2021if  reported less than 21% of the papers tested their system
on real-world users, and Adadi et al. adadi2018peeking  said this number is
less than 5%. The second shortage is that most papers claim that their CFEs
are plausible; however, it is not supported by user tests. The authors named
covering coverage and comparative testing as other shortages in this field.

In conclusion, many works have focused on defining ”explanations”,
”counterfactual”, and the need for XAI in AI algorithms. In the following
sections, we will review explanations from the theory and practice
perspectives.

###  2.2 Practice

In this part, we take some distance from the philosophy of the XAI and review
the actual use of XAI in the real world. We talk about what is a ”good
explanation” and a ”bad explanation”, and enumerate properties that make an
explanation good. We also talk about approaches to applying the XAI in the
real world more effectively.

Herm et al. herm2021don  conducted a user study to evaluate the explainability
of XAI and comprehensibility of XAI. They concluded that XAI-transferred ANN
(XANN) has the highest explainability and comprehensibility rather than other
explainable models, such as linear regression.

Vermeire et al. https://doi.org/10.48550/arxiv.2107.04427  evaluated the needs
of stakeholders for XAI. They recommended a specific method of explainability
based on the needs and desired properties of the AI model.

Another work evaluated the customer’s preferences for XAI in Swedish Credit
Scoring Industry matz2021explaining . They concluded that customers prefer
rule-based explanations. They tested three prototypes based on Shapley
Additive Explanations (SHAP), Local Interpretable Model-Agnostic Explanations
(LIME), and counterfactual method. They reported that users could trust SHAP
more than LIME and counterfactuals. The least system understanding was for
counterfactuals, and the most usefulness was for SHAP.

In 2020, Verma et al. verma2020counterfactual  published a paper and
enumerated some desired properties for CFEs that we mention in the following.
It is worth mentioning that these properties can be applied to other XAI
approaches too.

  1. 1.

Actionability (Plausibility): This property guarantees that the CF explanation
will not advise changing immutable features (e.g., race, gender), and only
mutable features will change (e.g., education, income).

  2. 2.

Sparsity: This property is about the trade-off between the number of features
changed and the total amount of change made to obtain the CFE. A CFE should
change a smaller number of features. It has been reasoned that people find it
easier to understand shorter explanations miller2019explanation .

  3. 3.

Data Manifold closeness: This property assurance that CF explanations respect
the data distribution. It means CF explanations will not recommend changing
something unlikely to change. For example, consider a patient at high risk for
heart disease. The system wants to suggest changes to change his
classification to low risk. The system should not recommend losing 60 kg
weight in one year since someone is unlikely to lose this weight in that time.

  4. 4.

Causality: Features in a dataset are rarely independent; therefore, changing
one feature in the real world impacts others. For example, increasing job
experience will be with increasing age. So we cannot ask someone to increase
his experience while his age does not change. A counterfactual should maintain
any known causal relations between features to be realistic and actionable.

  5. 5.

Amortized Inference: Generating a counterfactual is expensive since we need an
optimization function for each data point. Amortized inference uses generative
techniques. Generative techniques help the algorithm compute a counterfactual
(or several) for any new input without solving an optimization problem.

  6. 6.

Alternative Methods: This property is about having guaranteed and optimum
answers by mixed-integer programming, or SMT solvers. However, it limits us to
classifiers with linear (or piece-wise linear) structures.

In 2021, Verma et al. verma2021counterfactual  identified obstacles impeding
CFE deployment in the industry. In this work, they added two other desired CF
properties:

  1. 1.

Model-agnostic CFEs: An algorithm that can generate CFE for all black-box
models, or in other words, a CF algorithm that is not restricted to a specific
AI algorithm.

  2. 2.

Fast CFEs: This feature is about speed and is easy to use in practice. Fast
CFEs refer to algorithms that can generate CFEs for multiple data points after
a single optimization.

The authors mentioned some concerns in the industry. CFEs should be an
interactive service; moreover, CFEs say what should change, but they also
should say what should not change. Finally, CFEs should capture personal
preferences.

The mentioned properties are debated in the CFE setting. However, all
properties can be evaluated for other methods of explanation. For example, the
model agnostic property is not only preferred for CFE but also the desired
property for other approaches. In the following sections, we will talk about
this concern in detail.

Brennen talked with many stakeholders to realize what they want from XAI and
found two main concerns brennen2020people . (1) There needs to be consistent
terminology for Explainable AI. (2) There are multiple diverse use cases for
Explainable AI, including debugging models, understanding bias, and building
trust, and each case deals with a different user category. So, it requires
different explanation strategies that are not developed yet.

Alufaisan et al. alufaisan2020does  conducted an experiment. The authors
compared objective human decision accuracy without AI with an AI prediction
(no explanation) and AI prediction with an explanation. They found that
providing AI prediction improves user decision accuracy, but no definitive
evidence exists that explainable AI has a meaningful impact. To make this
conclusion, they considered the accuracy of participants’ decisions,
confidence rating, and reaction time.

The authors in gerlings2020reviewing  mentioned the basic reasons we need XAI.
1- Generate Trust, Transparency and Understanding. 2- GDPR Law. 3- Social
Responsibility, Fairness, and Risk Avoidance. 4- Generate Accountable and
Reliable models. 5- Minimize Biases. 6- Being Able to Validate Models.

In 2021, Chromik et al. chromik2021think  studied how non-technical users make
their global understanding of the model when we give them local explanations.
The authors searched for answers to the following questions by an empirical
study: 1- How robust is a self-reported global understanding gained from local
explanations when examined? 2- How non-technical XAI users construct a global
understanding from local explanations? They used accuracy of decisions,
confidence, and Perceived Understanding metrics to evaluate their empirical
study. They reported that the perception of the understanding of non-technical
users decreased when researchers examined non-technical user perception.

To sum up, there are three main lessons that we can learn from reviewing works
in the practice category.

  1. 1.

Lesson one:

We can see that the most important properties that have been mentioned in the
past works are as follow: actionability, sparsity, causality, data manifold
closeness, and model-agnostic algorithms.

  2. 2.

Lesson two:

There are specific needs and reasons to have XAI. Moral values (transparency
and trust), debugging, eliminating bias, model validation, intervention and
changing the output, and law requirement (GDPR).

  3. 3.

Lesson three:

Users of XAI can be classified into three groups: stakeholders, expert users,
and non-expert users.

In the next section, we will talk about the theory and math behind XAI.

###  2.3 Theory

From the computer science perspective, the theory behind XAI is an essential
part of it. In this category of papers, researchers formulate the problem,
explain the math behind it, and find an optimum answer for the problem.

We categorized XAI methods into two general categories feature importance-
based approaches and counterfactuals. In the feature importance-based
approaches, researchers proposed algorithms that explain the importance or
contribution of each feature. In counterfactual explanations, algorithms have
been developed to find changes that will alter the output.

The first method of XAI that we discuss here is Local Interpretable Model-
agnostic Explanations (LIME) ribeiro2016should . This approach is a feature
importance-based method for explaining local samples. The intuition of this
method is that in a nonlinear model, LIME estimates a linear classifier in a
local place where the sample is located. This idea can be formulated as below:

$\xi(x)=\underset{g\in
G}{\operatorname{argmin}}\mathcal{L}\left(f,g,\pi_{x}\right)+\Omega(g)$ (1)

where $\mathcal{L}\left(f,g,\pi_{x}\right)$ is a measure of how unfaithful $g$
is in approximating $f$ in the locality defined by $\pi_{x}$ and $\Omega(g)$
is the complexity of model.

$\mathcal{L}\left(f,g,\pi_{x}\right)=\sum_{z,z^{\prime}\in\mathcal{Z}}\pi_{x}(z)\left(f(z)-g\left(z^{\prime}\right)\right)^{2}$
(2)

$G$ is the class of linear models, such that
$g(z^{{}^{\prime}})=w_{g}.z^{{}^{\prime}}$. They used the locally weighted
square loss as $\mathcal{L}$, and $\pi_{x}(z)=$
$\exp\left(-D(x,z)^{2}/\sigma^{2}\right)$ is an exponential kernel and $D$ is
distance function with width $\sigma$.

In the line of feature importance approaches, Lundberg et al.
lundberg2017unified  proposed SHAP (SHapley Additive exPlanations). The idea
of this approach comes from game theory, where each feature is an agent, and
we want to distribute output fairly among agents. It means on a scale of 1,
the feature with a bigger role gets a bigger proportion. Same as LIME, SHAP
provides us with local explanations. SHAP defines an explanation as follows:

$g\left(z^{\prime}\right)=\phi_{0}+\sum_{i=1}^{M}\phi_{i}z_{i}^{\prime}$ (3)

where $g$ is the explanation model, $z^{\prime}\in\\{0,1\\}^{M}$ is coalition
vector for agents, $M$ is the maximum coalition size, and
$\phi_{i}\in\mathbb{R}$ is feature $i$ contribution . $\phi_{i}$ can be
calculated as follow:

$\phi_{i}=\sum_{S\subseteq F\backslash\\{i\\}}\frac{\mbox{{}\rm\sf\leavevmode\hbox{}\hbox{}S\/} !(\mbox{{}\rm\sf\leavevmode\hbox{}\hbox{}F}-\mbox{{}\rm\sf\leavevmode\hbox{}\hbox{}S}-1) !}{\mbox{{}\rm\sf\leavevmode\hbox{}\hbox{}F} !}\left[f_{S \cup\\{i\\}}\left(x_{S \cup\\{i\\}}\right)-f_{S}\left(x_{S}\right)\right] \end{equation} \par$F$ is the set of all features and $S$ is a subset of features ($S\in F$). $f_{S\cup\\{i\\}}$ is the model trained with the feature present, and $f_{S}$ is trained with the feature withheld. Models are compared on the current input $f_{S\cup\\{i\\}}\left(x_{S\cup\\{i\\}}\right)-f_{S}\left(x_{S}\right)$, where $x_{S}$ represents the values of the input features in the set $S$. \par\par The next approaches that attracted much attention are PDP (partial dependence plot) \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{friedman2001greedy}{\@@citephrase{(}}{\@@citephrase{)}}}, and ALE (Accumulated local effects) \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{apley2020visualizing}{\@@citephrase{(}}{\@@citephrase{)}}}. Unlike previous methods, ALE is global, which means it shows the effect of features on the prediction in general, not only based on one sample. The idea of PDP and ALE is explained below. \par PDP evaluates the effect of one feature on output (consider it $x_{1}$, and we call the rest of features $x_{2}$). $Y$ is a scalar response variable and $X=(X_{1},X_{2},...,X_{d})$ is the feature vector. $i$ is indicators of our $i$-th sample in training set ($\\{y_{i},x_{i}=(x_{i,1},x_{i,2},...,x_{i,d}):i=1,2,...,n\\}$) and $p_{2}(\cdot)$ denotes the marginal distribution of $X_{2}$. \par\begin{equation}f_{1,PD}\left(x_{1}\right)\equiv\mathbb{E}\left[f\left(x_{1},X_{2}\right)\right]=\int p_{2}\left(x_{2}\right)f\left(x_{1},x_{2}\right)dx_{2}\end{equation} \par ALE is calculated as follow: \par\par\begin{equation}\begin{aligned} f_{1,ALE}\left(x_{1}\right)&\equiv\int_{x_{\min,1}}^{x_{1}}\mathbb{E}\left[f^{1}\left(X_{1},X_{2}\right)\mid X_{1}=z_{1}\right]dz_{1}-\text{ constant }\\\ &=\int_{x_{\min,1}}^{x_{1}}\int p_{2\mid 1}\left(x_{2}\mid z_{1}\right)f^{1}\left(z_{1},x_{2}\right)dx_{2}dz_{1}-\text{ constant }\end{aligned}\end{equation} \par\par where $f^{1}\left(x_{1},x_{2}\right)\equiv\frac{\partial f\left(x_{1},x_{2}\right)}{\partial x_{1}}$ represents the local effect of $x_{1}$ on $f(\cdot)$ at $\left(x_{1},x_{2}\right)$, and $x_{\min,1}$ is some value chosen near the lower bound of the effective support of $p_{1}(\cdot)$, e.g., just below the smallest observation $\min\left\\{x_{i,1}:i=1,2,\ldots,n\right\\}$. Choice of $x_{\min,1}$ is not important, as it only affects the vertical translation of the ALE plot of $f_{1,ALE}\left(x_{1}\right)$ versus $x_{1}$, and the constant in equation \ref{ALEEQ} will be chosen to vertically center the plot. \par\par\par\par\par\par\par All the methods mentioned above are feature-based. Counterfactual explanations, unlike others, are not based on feature importance. Historically, one of the first people who formulated CF explanations was Watcher et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{wachter2017counterfactual}{\@@citephrase{(}}{\@@citephrase{)}}} in 2017. Watcher formulates the problem as follows: \par\begin{equation}arg\min_{w}l(f_{w}(x_{i}),y_{i})+p(w)\end{equation} \par\begin{equation}arg\min_{x^{{}^{\prime}}}\max_{\lambda}\lambda(f_{w}(x^{{}^{\prime}})-y^{{}^{\prime}})^{2}+d(x_{i},x^{{}^{\prime}})\end{equation} \par\par Where $w$ indicates the weight, $y_{i}$ is the label for data point $x_{i}$, and $p(.)$ is a regularizer over the weights in an AI algorithm. We wish to find a counterfactual $x^{{}^{\prime}}$ as close to the original point $x_{i}$ as possible such that $f_{w}(x^{{}^{\prime}})$ is our desired output $y^{{}^{\prime}}$. Here, $d$ is a distance function that measures how far the counterfactual $x^{{}^{\prime}}$ is from the original data point $x_{i}$. Watcher considers this distance as follows, which is the median absolute deviation of feature $k$ over the set of points $P$: \par\begin{equation}d(x_{i},x^{{}^{\prime}})=\sum_{k\in F}\dfrac{\|X_{i,k}-x^{{}^{\prime}}_{k}\|}{median_{j\in P}(\|X_{j,k}-median_{l\in P}(X_{l,k})\|)}\end{equation} \par\par One year later, in 2018, Ustun et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{ustun2019actionable}{\@@citephrase{(}}{\@@citephrase{)}}} published a new work in the field of CFEs. In their work, they referred to CF as actionable recourse. The main difference between Ustun's work with Watcher \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{wachter2017counterfactual}{\@@citephrase{(}}{\@@citephrase{)}}} was that Ustun used percentile as the distance function, and they guaranteed feasibility. By feasibility, we mean that the method should not consider immutable features, such as gender, etc., to change. So, they formulated their problem as follows: \par\@@amsalign&\min cost(a\mathchar 24635\relax\;x)\\\ &s.t.\quad f(x+a)=1\\\ &a\in A(x) \begin{equation}cost(x+a\mathchar 24635\relax\;x)=\max_{j\in J_{A}}\|Q_{j}(x_{j}+a_{j})-Q_{j}(x_{j})\|\end{equation} \par where $x$ is the current data point, $x+a$ is the counterfactual point, $f(x)$ is the AI function, $A(x)$ is a set of feasible actions, and $Q_{j}(\cdot)$ is the CDF of $x_{j}$ in the target population. \par In \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{lucic2019focus}{\@@citephrase{(}}{\@@citephrase{)}}}, the authors framed the problem of finding CFEs as a gradient-based optimization task. For non-differentiable models such as tree ensembles, they used probabilistic model approximations in the optimization framework. They mentioned Euclidean, Cosine, and Manhattan distances as potential distances in the cost function. \par\par In 2020, we saw a surge in XAI publications, especially CFEs. Karimi et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{karimi2020model}{\@@citephrase{(}}{\@@citephrase{)}}} presented a model agnostic method to find CFEs and named it MACE. The strength of his work was that the method was model agnostic and could be applied to neural networks, decision trees, and all other AI algorithms. He also considered the feasibility property (referred to as plausibility in this work). They evaluated their method on three famous datasets in this field: Loan approval (adult) \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}, Credit dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{YEH20092473}{\@@citephrase{(}}{\@@citephrase{)}}}, and Pretrial bail (COMPAS) dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Larson}{\@@citephrase{(}}{\@@citephrase{)}}}. The drawback of his proposed method is in high runtime compared with other methods that later have been introduced. \par Karimi et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{karimi2020algorithmic}{\@@citephrase{(}}{\@@citephrase{)}}} also provided a method to generate CFEs with limited knowledge about the causality between features and used a gradient-based optimization method to find the optimum CF point. \par Another work considered causal reasoning \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{karimi2021algorithmic}{\@@citephrase{(}}{\@@citephrase{)}}} to generate CFEs. The authors shifted from recourse via the nearest CF to recourse through minimal interventions, or in other words, shifting the focus from explanations to interventions. \par In 2021, Mohammadi et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{mohammadi2021scaling}{\@@citephrase{(}}{\@@citephrase{)}}} proposed a framework based on Mixed-Integer Programming (MIP). This framework calculates the nearest counterfactual explanations for the outcomes of neural networks, and it guarantees an answer and an optimal solution compared to the gradient-based approaches. \par The authors in \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{karimi2021survey}{\@@citephrase{(}}{\@@citephrase{)}}} elaborated on the concept of CFEs, recourse definition, causality, plausibility, actionability, diversity, and sparsity. They named four essential criteria for CF explanations optimality, perfect coverage, efficient runtime, and access. \par Mothilal et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{mothilal2020explaining}{\@@citephrase{(}}{\@@citephrase{)}}} proposed a techniques to find CFE named DICE in 2020. They named two desired properties for their proposed framework, feasibility and diversity. They added these two properties to their cost function to maximize feasibility and diversity. Finally, the approach has been tested on four datasets: Adult-Income dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}, LendingClub dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Kevin2015}{\@@citephrase{(}}{\@@citephrase{)}}}, German-Credit dataset\cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}, and COMPAS dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Larson}{\@@citephrase{(}}{\@@citephrase{)}}}. \par\par\par Another work proposed the skyline CFEs that define the skyline of CFs as all non-dominated changes \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{wang2021skyline}{\@@citephrase{(}}{\@@citephrase{)}}}. They solved this problem as multi-objective optimization over actionable features. Basically, they searched for a space of CF instead of just a data point. To verify their algorithm, they used three datasets 1- UCI Adult Dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}, 2- Give Me Some Credit (GMSC), and 3- HELOC Dataset . The authors did not discuss the runtime for this approach. \par\par Researchers introduced a new objective function that evaluates a pair of actions and orders them based on feature interaction \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{kanamori2021ordered}{\@@citephrase{(}}{\@@citephrase{)}}}. The authors clarified that there is an asymmetric interaction among features, such as causality. So, the total cost of the action depends on feature interaction. Therefore, practical CF methods must provide an appropriate order of changing features. They applied this idea by considering an interaction matrix to find the causality between features. \par\par Hada et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{hada2021exploring}{\@@citephrase{(}}{\@@citephrase{)}}} focused on explaining classification and regression trees among AI algorithms and formulated the counterfactual explanation as an optimization problem. \par\par\par Verma et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{verma2021amortized}{\@@citephrase{(}}{\@@citephrase{)}}} published an article that proposed a stochastic-control-based approach to generate sequential Algorithmic Recourses (ARs) or the same CF. This algorithm permits the data point to move stochastically and sequentially across intermediate states to a final state of the cf point. The authors named their approach FASTAR and mentioned the following desiderata for their algorithm. 1- Actionability (or feasibility). 2- Sparsity: this property searches for smaller explanations since smaller explanations are more understandable to humans \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{miller2019explanation}{\@@citephrase{(}}{\@@citephrase{)}}}. 3- Data manifold: This property ensures that the explanation respects the training data manifold \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{dhurandhar2019model}{\@@citephrase{(}}{\@@citephrase{)}}}, \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{kanamori2020dace}{\@@citephrase{(}}{\@@citephrase{)}}}. 4- Causal constraints: this property considers the causality between features \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{mahajan2019preserving}{\@@citephrase{(}}{\@@citephrase{)}}}. 5- Model-agnostic. 6- Amortized: an amortized approach can generate ARs for several data points without optimizing separately for each of them \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{mahajan2019preserving}{\@@citephrase{(}}{\@@citephrase{)}}}. They tested their proposed approach on three datasets, German Credit, Adult income, and Credit Default \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}. \par\par We summarized all important approaches in XAI and information related to them in Table \ref{summary}. \par\par\par\begin{table}\centering \@@toccaption{{\lx@tag[ ]{{1}}{Summary of Recent Related Works in CFEs.}}}\@@caption{{\lx@tag[: ]{{Table 1}}{Summary of Recent Related Works in CFEs.}}}\leavevmode\resizebox{416.27809pt}{177.06458pt}{ \begin{tabular}[]{ | p{4cm}| p{3cm} | p{6cm}| p{4cm} | p{2cm} | }\hline\cr General Category of Explanation&Paper Name&Main Contribution&Dataset&Is The Code Public?\\\ \hline\cr Feature Importance-based&Ribeiro et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{ribeiro2016should}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing Local Interpretable Model-agnostic Explanations (LIME)&1- Books dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{blitzer2007biographies}{\@@citephrase{(}}{\@@citephrase{)}}}&\href https://github.com/marcotcr/lime\\\ &&&2- DVDs Dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{blitzer2007biographies}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ \hline\cr Feature Importance-based&Lundberg et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{lundberg2017unified}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing SHAP (SHapley Additive exPlanations)&1- MNIST digit dataset&\href https://shap.readthedocs.io/en/latest/index.html\\\ \hline\cr Feature Importance-based&Apley et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{apley2020visualizing}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing ALE (Accumulated Local Effect)&2- bike sharing rental counts \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{bike}{\@@citephrase{(}}{\@@citephrase{)}}}&\href https://pypi.org/project/PyALE/\\\ \hline\cr Counterfactual&Watcher et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{wachter2017counterfactual}{\@@citephrase{(}}{\@@citephrase{)}}}&Formulationg CFEs for the first time&No dataset&No\\\ \hline\cr Counterfactual&Ustun et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{ustun2019actionable}{\@@citephrase{(}}{\@@citephrase{)}}}&Using percentile as cost function + added feasibility to CFE&Credit dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{YEH20092473}{\@@citephrase{(}}{\@@citephrase{)}}}&\href https://github.com/ustunb/actionable-recourse\\\ \hline\cr&&&1- Wine Quality \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ &&&2- HELOC \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{FICO1}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ Counterfactual&Lucic et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{lucic2019focus}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing an approach for non-differentiable tree ensembles&3- COMPAS \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Ofer2017}{\@@citephrase{(}}{\@@citephrase{)}}}&\href https://github.com/a-lucic/focus\\\ &&&4- Shopping \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ \hline\cr&&&1- Loan approval (adult) \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ Counterfactual&Karimi et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{karimi2020model}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing an approach that is model-agnostic + actionable&2- Credit dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{YEH20092473}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ &&&3- Pretrial bail (COMPAS) dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Larson}{\@@citephrase{(}}{\@@citephrase{)}}}&\href https://github.com/amirhk/mace\\\ \hline\cr Counterfactual&Karimi et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{karimi2020algorithmic}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing an approach based on Limited Knowledge of causality and gradient based optimization&No dataset&No\\\ \hline\cr Counterfactual&Karimi et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{karimi2021algorithmic}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing an approach that shifted the focus from explanations to interventions&No dataset&No\\\ \hline\cr&&&1- Adult&\\\ Counterfactual&Mohammadi et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{mohammadi2021scaling}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing a framework based on Mixed-Integer Programming (MIP)&2- COMPAS&\href https://github.com/amirhk/mace\\\ &&&3- Credit dataset&\\\ \hline\cr&&&1- Adult-Income dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ &&&2- LendingClub dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Kevin2015}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ Counterfactual&Mothilal et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{mothilal2020explaining}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing DICE algorithm that provides feasible + divers CFEs&3- German-Credit dataset\cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\href https://github.com/interpretml/DiCE\\\ &&&4- COMPAS dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Larson}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ \hline\cr&&&1- UCI Adult Dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ Counterfactual&Wang et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{wang2021skyline}{\@@citephrase{(}}{\@@citephrase{)}}}&Introducing the skyline (n space) for CFEs&2- Give Me Some Credit (GMSC)&No\\\ &&&3- HELOC Dataset &\\\ \hline\cr&&&1- FICO \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{FICO1}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ &&Introducing a new objective function that evaluates a pair of actions and&2- German-Credit dataset\cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ Counterfactual&Kanamori et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{kanamori2021ordered}{\@@citephrase{(}}{\@@citephrase{)}}}&orders them based on feature interaction for CFEs&3- Wine Quality \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\href https://github.com/kelicht/ordce\\\ &&&4- Diabetes \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ \hline\cr&&&1- Student Portuguese grades \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ Counterfactual&Hada et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{hada2021exploring}{\@@citephrase{(}}{\@@citephrase{)}}}&Providing an algorithm to generate CFEs for regression trees&2- German Credit \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&No\\\ &&&3- Boston housing dataset \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ \hline\cr&&&1- German Credit \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ Counterfactual&Verma et al. \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{verma2021amortized}{\@@citephrase{(}}{\@@citephrase{)}}}&Proposing a stochastic control-based approach to generate sequential&2- Adult income \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&No\\\ &&Algorithmic Recourses (AR)&3- Credit Default \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}}&\\\ \hline\cr\end{tabular} } \@add@centering\end{table} \par\par In conclusion, from the theoretical perspective computing the CFE for an AI algorithm is minimizing a cost function between the current point $x$ and CF point $x^{{}^{\prime}}$ subject to the output of the AI algorithm function for our CF point is the desired output. Different works proposed different cost functions or how to find the answer to this optimization problem. For feature importance-based methods, each method proposes a new way of finding the importance or contribution of a feature in ML algorithm prediction. Also, by noticing papers, we can say German Credit, Adult income, and Credit Default \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{Dua:2019}{\@@citephrase{(}}{\@@citephrase{)}}} are the three most popular datasets in this field to test the explanation-generating algorithms. \par\par\par\@@numbered@section{section}{toc}{Objectives and Concerns in XAI} \par So far, we know that because of the black-box nature of AI algorithms, we cannot see vividly why the algorithm concluded a specific output. To address this problem, researchers developed explainable AI. XAI explains how an algorithm works. In the previous section, we reviewed some papers in the field of XAI and presented state-of-the-art approaches in this field. This section will debate shortages and concerns in this field and propose potential ways to address them. We show each concern as an objective as follows: \par\par\par\@@numbered@section{subsection}{toc}{Objective 1} \par The general theme in the recent papers is presenting a new approach to generating explanations. The concern here is that it needs to be clarified who is the user of each explanation. Is that an expert, stakeholder, or non-expert user such as a patient? So we can say the users of XAI are not well-defined in the literature. This concern is critical since explanations will differ for each user, and we should consider each group differently. But {how} the explanations would be different for each group, and {what} should change for each group of users is not clear. \par\par\par So far, we know that various categories of people will use XAI, or in other words, different people want XAI for different purposes. We need a mapping function between users, XAI methods, and purposes to clarify this relationship. For users, we have three general groups of \\{experts, non-experts, and stakeholders\\}. For XAI methods, we can consider \\{LIME, SHAP, ALE, PDP, CFE, etc.\\}, and for purposes, we have \\{debugging, trustworthiness, validations, etc.\\}. For instance, we can map that stakeholders want CFEs to earn customers' trust. So we have the following set \\{stakeholders, CFE, trustworthiness\\}. On the other hand, non-expert users never seek CFE to debug; they ask for an explanation to change the output or trust the system, \\{non-expert, CFE, trustworthiness\\}. \par A further study can clarify this mapping function and use it to find the proper explanation with the suitable properties for each user (Personalized explanations for each category of users). \par\par\par\par\@@numbered@section{subsection}{toc}{Objective 2} \par One issue with CFEs and XAI is that different properties are often considered desirable in different contexts. For example, sparsity implies a preference for shorter sentences over longer ones. However, in the context of a medical application, where the users are doctors and experts, sparsity may not be a desired property. Doctors may prefer longer sentences that provide more information for decision making. Further research is necessary to determine the optimal level of sparsity for this particular application and how it can best inform users. \par\par\@@numbered@section{subsection}{toc}{Objective 3} \par Another concern in XAI is the lack of study on user preferences for explanations. For instance, a doctor may prefer a detailed explanation, while a patient may prefer a simpler, shorter answer. Additionally, user preferences over the features used in a counterfactual explanation should also be taken into account. For example, in a loan application scenario, an 80-year-old applicant may prefer increasing their savings, while a 30-year-old applicant may prefer earning a higher degree to increase their loan eligibility. It is important to acknowledge that different individuals have different preferences for different features and to use these preferences to generate personalized explanations. \par\par\par\@@numbered@section{subsection}{toc}{Objective 4} \par The fourth missing point of this puzzle is that many researchers reported that experiments showed current explanations have no meaningful impact on human decision-making \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{alufaisan2020does}{\@@citephrase{(}}{\@@citephrase{)}}}. However, it is still not studied that {why} XAI are not working well in practice? {what changes} are needed to make them effective? Is this "no meaning full impact" is the same for all kinds of users and all kinds of XAI methods? We can imagine that some explanations are more informative for experts rather than non-expert ones. So, the meaningful impact may be different for various categories of users. A case study may prove or reject this hypothesis. \par Another important issue to consider is the possibility of XAI providing misleading information. What constitutes misleading information in XAI and how can it be classified and recognized? For example, is it possible that a counterfactual explanation generated by an algorithm led to a wrong action by the user? If so, can misleading information be classified as directly misleading (e.g. the algorithm recommended a change in the wrong direction or to the wrong feature) or indirectly misleading (e.g. the explanation caused the user to make incorrect conclusions about other features in their profile)? As an example, consider a job applicant who was rejected for a position and was provided with two explanations for how to increase their chances of success in the future. The first explanation was to increase their expected salary, while the second was to increase their job experience and obtain more recommendations from past employers. The first explanation represents a direct mistake as they should decrease expected salary, while the second could be considered indirectly misleading, as the user may conclude that all other aspects of their profile are satisfactory and that they only need to focus on improving their job experience. \par\par\par\@@numbered@section{subsection}{toc}{Objective 5} \par The final challenge in this area is defining immutable features in XAI. Past research has proposed algorithms that ensure that the explanations do not alter certain features, but it's not clear how these features are determined. For example, in a loan application, features such as gender, savings, race, age, and debt may be easily identifiable, but in other real-world applications such as medicine, the list of features may be unknown. To overcome this challenge, one potential solution is using time-series data monitoring. By observing the data and features over a sufficient period, we can identify which features remain unchanged and classify them as immutable (satisfying the actionability criterion), and which ones change infrequently (meeting the requirement of closeness to the data distribution). \par\par\par\par\@@numbered@section{section}{toc}{Who wants what and how: a Mapping Function} \par This section will answer the following questions. Who are the users of XAI? What are their purposes for using XAI? What properties are useful for each user and goal, and finally, which methods are usable for different goals. This section's importance is that researchers didn't consider the ultimate goal of each explanation and which property each explanation should have. Many XAI properties have been proposed without considering whether this characteristic is helpful for a specific purpose or not. This section aims to study the users of XAI, their motivation to apply XAI, and the potential properties of XAI for each user and goal. The mapping function proposed in this section will help all users of XAI to select the match method and properties for their purposes. Finally, the output of this section is XAI methods that are generated regarding the task that they suppose to do and are chosen carefully to work better in practice. \par\par\par\par In general, Explainable Artificial Intelligence (XAI) has been used for the following purposes: 1- XAI is turning to customers' rights (e.g., GDPR Law). 2- For finding algorithm mistakes (debugging). 3- For finding the bias in the system. 4- Increase the trustworthiness of the system. 5- To determine whether we should trust the system or not (system validation). 6- Help users modify the output toward the desired output. 7- Moral values. \par\par\par In the case of users, we have three categories of users. 1- Stakeholders such as banks, hospitals, and institutions. 2- Experts, such as doctors. 3-Not expert users, such as an applicant for loan applications or patients. \par\par In the case of desired properties for XAI, we have plausibility (actionability), sparsity, run time, being model agnostic, data manifold closeness, causality, and fairness. \par\par In the case of approaches to provide explanations, we have the following approaches. Counterfactual explanations, Saliency maps, Partial Dependence Plot (PDP), Accumulated Local Effects(ALE), Individual Conditional Expectation (ICE), Local Interpretable Model-agnostic Explanations (LIME), Shapley Additive exPlanations (SHAP), Layer-wise relevance propagation (LRP), Class Activation Mapping (CAM), Gradient-weighted Class Activation Mapping (Grad-CAM), and CLass-Enhanced Attentive Response (CLEAR). \par\par In general, we have three ways to present explanations: 1- Visual explanations such as plots. 2- Textual explanations such as text. 3- Mathematical or numerical explanations. \par\par Explanations can be provided for classification, regression, segmentation, etc., tasks. \par\par Finally, the last property is the format of data. Data that has been used in AI algorithms can be tabular or numerical data, image data, and text data. For example, some explainability approaches are proposed for Artificial Neural Networks (ANNs) that are mainly used for image data. In the following, we study each classification in detail. \par\par\@@numbered@section{subsection}{toc}{Who Wants XAI?} \par The first category of users is stakeholders, who refer to individuals or organizations that are not experts in AI but are implementing AI to provide a service. They can be policymakers, institutions, banks, employees, investors, etc. These stakeholders benefit from XAI by using it for moral values, to validate the systems provided by engineers, to meet legal requirements, and to gain the trust of their users. \par However, stakeholders are not directly involved in fixing the system, eliminating bias, or changing its output as they do not have enough knowledge to do so. The company hiring people is an example of a stakeholder. By providing explanations to applicants who were not hired, the company demonstrates its commitment to transparency and fairness while also gaining the trust of its applicants. \par\par\par Experts are another category of AI users who use XAI to achieve a specific goal. They may not be directly using AI themselves but interact with it to make decisions or complete tasks. Examples of experts include doctors, pathologists, AI technicians, researchers, and police officers. \par Experts can use XAI to verify or validate a system, debug the system, eliminate bias, and ensure compliance with moral values and legal requirements. They can also use XAI to trust in the AI system they are using. \par Experts are not usually interested in changing the AI-based decision as it is not directly related to them. For example, a doctor may use counterfactual explanations to understand how a patient can be classified as low-risk for heart disease instead of high-risk. In this situation, the doctor is using the explanation as a proxy for the patient, rather than for personal benefit. \par\par The last category of XAI customers is non-expert users. Non-expert users are users who are not experts in AI and any other field combined with AI, but they are under the direct effect of AI decisions. They can be patients in hospitals, loan applicants, job applicants, etc. Non-expert XAI users want to intervene and change the output made by AI algorithms, use the rights that the law gave them, and trust systems (trustworthiness). Non-expert users are not in the position that debugs the system or eliminate the bias and don't have enough knowledge to validate the system. We believe this category of users is more vulnerable than other users. They can easily be misled or confused by XAI and have less information about AI. To illustrate this vulnerability, consider an AI system that classifies patients into two groups that immediately need cardiac surgery and those that do not need it. This algorithm classifies a patient as someone who does not need surgery while he is in an emergency situation and needs immediate surgery. Since the classification of the AI algorithm is wrong, the explanation will be wrong too. Now consider two groups of people who see this explanation, doctors and patients. Since doctors know medicine, they are more immune from false information provided by explanations. However, this misleading XAI information can damage patients irrecoverably. \par In table \ref{users} we summarized all XAI users and the potential purposes that they can have. \par\begin{table}\centering\@@toccaption{{\lx@tag[ ]{{2}}{XAI Users and Their Purposes for Using XAI.}}}\@@caption{{\lx@tag[: ]{{Table 2}}{XAI Users and Their Purposes for Using XAI.}}}\begin{tabular}[]{ |l | l| }\hline\cr User&Goal\\\ \hline\cr&System Validation\\\ Stakeholders&Moral Values\\\ &Law Requirement\\\ &Trustworthiness\\\ \hline\cr&System Validation\\\ &System Debugging\\\ Experts&Eliminate Bias\\\ &Moral Values\\\ &Law Requirement\\\ &Trustworthiness\\\ \hline\cr&Intervention and Changing the Output\\\ Non-expert Users&Moral Values\\\ &Law Requirement\\\ &Trustworthiness\\\ \hline\cr\end{tabular} \par \@add@centering\end{table} \par\par\par\par\@@numbered@section{subsection}{toc}{What Properties Should XAI Have?} \par So far, we clearly know why each user of XAI is using it. Now we can discuss the desired properties for each goal for using XAI. \par Plausibility is the most basic property that makes an explanation valid. Plausibility is a required condition for all purposes of XAI. It's hard to imagine that people will trust AI systems that make not-actionable recommendations. \par\par Model-agnostic is another desirable property for all XAI purposes. Validating systems based on XAI methods that are designed for all kinds of ML algorithms will be time-saving and more trustable. \par\par If we are using XAI for reasons like system validation, debugging, and bias elimination, then the run time will not be an issue. The reason is that we do not use XAI as a real-time task for the mentioned purposes. So, time should not be considered as a side of this tradeoff. In the mentioned cases, better performance is superior to low run time. It is worth noting that by claiming run time is not an issue, we are not talking about infinite run time (e.g., some brute force algorithms). If non-expert users are using XAI for intervention purposes, then run-time is an issue simply because faster systems are more desirable. Moral values prefer less time-taking explanations. There is no clear study about the connection between trustworthiness and run-time in XAI systems. But, in general, providing explanations in less time seems more successful in earning trust. There is no law requirement currently about the run time of explanations. For example, GPDR law clarifies that "information must be concise, transparent, intelligible and easily accessible, and use clear and plain language" \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{eu-269-2014}{\@@citephrase{(}}{\@@citephrase{)}}}. As we see, there is no referring to the run-time of explanations. A threshold for the needed time for generating explanations can be a consideration. \par\par With the same logic, we can say sparsity is not an on-demand property for debugging, validating, and eliminating bias in XAI systems. We can imagine cases where longer sentences can give us more information about algorithm performance. For example, in a counterfactual explanation, an explanation recommends changing ten features in a patient to be classified as a low-risk heart disease patient rather than a high-risk patient. Eight features out of 10 are related to the patient's job life, such as decreasing working hours and stress level. With this long explanation that is not meeting the sparsity property, we can say the system overweighted patients' job features rather than recommending changing patients' weight, eating habits, etc. For intervention purposes, sparsity is a desirable property. In general, sparsity is a preferred property when a non-expert user is involved; otherwise, it is not. Since shorter explanations are more understandable, they will earn more trust. Same as before, there is no law requirement regarding sparsity, but it can be applied as a requirement. \par Causality and correlation between features is an important property that many XAI methods currently do not possess. This property is crucial for changing the output of AI systems and for complete validation of the system. XAI should be able to explain the causality between features to ensure that the AI algorithm understands it correctly. While other validation methods, such as checking for bias, can be useful, examining causality is essential for complete validation. Therefore, the ability to explain causality is a desirable property for all AI applications. \par\par In the context of detecting and eliminating bias in AI systems, being loyal to the data distribution (data manifold closeness) is not always the desired property. For example, in a dataset where most of the people hold a B.Sc. degree, if the goal is to detect bias for M.Sc. degrees, the desired explanations should focus on the M.Sc. degree and reveal the AI algorithm's strategy for this specific degree. In this case, being loyal to the data distribution is not the appropriate property. On the other hand, data manifold closeness may be a suitable property for other objectives such as intervention or moral values. \par In Table \ref{property} we summarized desired properties for each XAI goal. \par\par\begin{table}[ht] \centering\@@toccaption{{\lx@tag[ ]{{3}}{Purposes for Using XAI and Desired Properties.}}}\@@caption{{\lx@tag[: ]{{Table 3}}{Purposes for Using XAI and Desired Properties.}}}\begin{tabular}[]{ | l | l| }\hline\cr Goal&Property\\\ \hline\cr&Plausibility\\\ System Validation&Model- agnostic\\\ &Causality\\\ &Data Manifold Closeness\\\ \hline\cr&Plausibility\\\ System Debugging&Model- agnostic\\\ &Causality\\\ &Data Manifold Closeness\\\ \hline\cr&Plausibility\\\ Eliminate Bias&Model- agnostic\\\ &Causality\\\ \hline\cr&Plausibility\\\ &Model- agnostic\\\ System Intervention&Sparsity\\\ &Causality\\\ &Run Time\\\ &Data Manifold Ccloseness\\\ \hline\cr&Plausibility\\\ &Model- agnostic\\\ Moral Values&Sparsity\\\ &Causality\\\ &Run Time\\\ &Data Manifold Closeness\\\ \hline\cr&Plausibility\\\ &Model- agnostic\\\ Trustworthiness&Sparsity\\\ &Causality\\\ &Run Time\\\ &Data Manifold Closeness\\\ \hline\cr&Plausibility\\\ &Model- agnostic\\\ Law requirement&Sparsity\\\ &Causality\\\ &Run Time\\\ &Data Manifold Closeness\\\ \hline\cr\end{tabular} \@add@centering\end{table} \par\par\par\@@numbered@section{subsection}{toc}{Which XAI Method Should We Use?} \par So far, we have mapped different users to different goals and mapped various purposes to various properties. In the following, we will map different methods to generate XAI to the goals and desired properties. \par\par As we discussed earlier, counterfactual explanations suggest a set of changes to achieve the desired output. Based on this definition, this method can be used by experts and non-experts that want to intervene and change the output. Since counterfactual explanations can not provide much information about how the system is working and which features are making the biggest contribution to the decision-making process, this method is not the best for system validation, debugging, and eliminating bias purposes. All the before-mentioned properties for this explanation are valid and needed. \par\par Many XAI approaches, such as LIME, SHAP, etc., are based on feature importance. The intuition behind feature importance in XAI is finding the contribution of each feature in the prediction and justifying the output based on the importance of each feature. Features in these approaches can be numbers in tabular data, pixels in images, and words in text data. \par\par\par The second XAI method is saliency map \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{simonyan2013deep}{\@@citephrase{(}}{\@@citephrase{)}}}. Saliency maps visualize pixels that are more important for the ML algorithm. This method is usually applied to image datasets. The saliency map does not explain how the network made a specific decision. Thus, saliency maps don't give us enough information to intervene and change the output. We can use saliency maps in system validation, debugging, eliminating the bias, earning the trust, and meeting the law requirements. Plausibility, sparsity, and data manifold closeness do not apply to the saliency map, but model-agnostic and low run-time are desired properties. In the case of causality, in image data, we have a connectivity matrix that shows the correlation between pixels. However, the saliency map does not visualize that. \par\par Partial Dependence Plots (PDPs) are a method for evaluating feature importance in AI algorithms. It illustrates the contribution of each feature in the decision-making process by assuming that features are independent. However, this independence assumption can make it difficult to understand correlations and causality between features, and it is not suitable for making changes to the output. PDPs allow us to observe how the prediction changes by modifying a single feature, helping us assess whether the system is using the correct features and controlling bias. They can be challenging for non-experts to interpret, as they often produce complex plots. Plausibility and sparsity are not applicable to this method. This method does not consider causality. But it can be a desired property. Low run time and being model agnostic is desirable too. In the case of closeness to manifold data, this method considers all feature constant except one and changes this feature in a wide range. For some amount, this feature will be unlikely, or it will not be close to the dataset distribution center. For example, suppose we want to consider the education feature in job applicants. In that case, this method will consider high school education, B.Sc., M.Sc., and Ph.D. for education, while Ph.D. and high school are not close to the center of the education distribution. \par\par Accumulated Local Effect (ALE) is another approach based on feature importance and is close to PDP. Basically, this approach plots the feature's influence on algorithm prediction. ALE considers the features correlations. Based on the presented explanation about ALE, we can say that stakeholders and experts can use this approach. Because of the visual nature of this approach, it is usable for non-experts too. Still, since this algorithm does not consider causality, it is unsuitable for intervention and changing the output purposes. Plausibility and sparsity do not apply to this approach. Run time and model-agnostic are valid desirable properties. For manifold data closeness, the same discussion in PDP is right here. In ALE, the algorithm will consider all values in the dataset for feature, and some of them will be unloyal to data distribution. \par\par\par Individual Conditional Expectation (ICE) \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{goldstein2015peeking}{\@@citephrase{(}}{\@@citephrase{)}}} shows how the output will change if we change a feature in an instance in the input set. So, we can consider it a plot showing how each sample prediction will change if we change a feature in that sample. ICE can "uncover heterogeneous relationships" \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{goldstein2015peeking}{\@@citephrase{(}}{\@@citephrase{)}}}. However, it does not show causality and is not suitable for intervention tasks. ICE information is helpful for stakeholders, experts, and non-experts. In the case of potential properties, it is the same as PDP and ALE. \par\par Local Interpretable Model-agnostic Explanations (LIME) is a method for providing local explanations for nonlinear AI algorithms. It works by creating a linear model that approximates the selected instance in the nonlinear algorithm, with the linear model being more interpretable. This approach can be used to validate and debug an AI system, reduce bias, and improve the transparency of the algorithm's decision-making process. LIME makes the reasoning behind the algorithm's decision clear to stakeholders, experts, and non-experts, increasing trust in the system. \par For the purpose of intervention, LIME presents a simplified decision boundary, indicating that changing the input so it falls on the opposite side of the estimated boundary may modify the output. However, it does not specify which features to change or by how much. Thus, it is not well-suited for system intervention. The concepts of plausibility, sparsity, and data manifold closeness cannot be considered in this context. Additionally, LIME does not take into account causality. It is desirable for LIME to have a low runtime and to be model-agnostic. \par\par SHAP (SHapley Additive exPlanations) is another visualization method that indicates the contribution of each feature to the output in a specific instance. This approach can be used for system validation, debugging, and bias elimination since it helps experts to see whether the suitable feature is making a significant contribution or not. For example, we want to provide explanations for the rejected job applicant. Based on this approach, we can check whether the gender or race of the applicant has a high role in output or not. Still, since it does not show the correlation and causality between features, intervention is not a possible option for this method of explanation. Having low run time and being model-agnostic is an expected property for this algorithm. \par\par LRP (Layer-Wise Relevance Propagation) is an explanation method by \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{bach2015pixel}{\@@citephrase{(}}{\@@citephrase{)}}} to explain how neural networks work. This method is based on the backward propagation of a prediction to find the relevance of each feature to the predicted output. The drawback of this approach is that this approach is mainly applicable to images, not other types of data. Researchers used LRP in applications of bias elimination \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{lapuschkin2019unmasking}{\@@citephrase{(}}{\@@citephrase{)}}} and system verification \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{arbabzadah2016identifying}{\@@citephrase{(}}{\@@citephrase{)}}}. The output of this method is a heatmap that indicates the relevance of each pixel to the predicted output. Since this approach does not clarify the correlation and causality between pixels, known as the connectivity matrix, it is inappropriate for intervention and changing the output. \par\par Class Activation Map (CAM) \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{zhou2016learning}{\@@citephrase{(}}{\@@citephrase{)}}} is another approach to explaining CNNs. This method is mostly used on images. The output of CAM is highlighted pixels that are effective on the CNN classification output. As it can show which pixels are more critical for CNN, it can be used for system verification, debugging, and eliminating bias. The goal of trustworthiness also can be achieved by this method. Still, this approach does not clarify connectivity and correlation between pixels, so it does not provide enough information to recourse and change the output. \par\par The concept behind Class-Enhanced Attentive Response (CLEAR) is similar to previous methods. CLEAR visualizes the level of interest of a CNN in a classification task, as described in \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{kumar2017explaining}{\@@citephrase{(}}{\@@citephrase{)}}}. It can be useful for system verification, debugging, and reducing bias, which can increase user trust. However, it does not give us information to change the output. Model-agnosticism and low run-time are preferred in this context. \par The summary of methods, their potential use, and properties are presented in Table \ref{methods}. \par\par Explanations can be presented in three formats: visual, text, and numerical. Each format has its own pros and cons. Visual information can convey more information quickly, making it easier to understand than text or numerical data. However, numerical data is more accurate than other formats. \par\par The format of data used in AI algorithms is another important factor. Explanation approaches such as LRP and CAMP have been developed mainly for images, while counterfactual explanations have been applied to both tabular and image data and have limited use in text data \cite[cite]{\@@bibref{Authors Phrase1YearPhrase2}{yang2020generating}{\@@citephrase{(}}{\@@citephrase{)}}}. Feature importance and saliency maps are mainly used for image data but have some applications in text data. PDP, ALE, and ICE are used for tabular data. LIME and SHAP methods have been applied to tabular, image, and text data. LRP and CAM were originally developed to explain neural networks dealing with image data, but there have been limited efforts to apply them to text and tabular data. CLEAR, which was introduced for image data, has also been used for tabular data. \par\par In general, the ability of an XAI algorithm to address causality and correlation between features is crucial in allowing users to change the output. However, our study showed that many algorithms do not adequately consider causality or have not developed it properly. This raises questions about how we can fully understand AI systems if we don't know how one feature affects other features. \par\par Another discussion related to this topic is the difference between the {"ability"} of a method to do a task and the {"superiority"} of a method to do a task. For example, PDP, ALE, and LIME can be used for system debugging. All the mentioned methods have the "ability" of debugging, as mentioned in Table \ref{methods}. However, among these three methods, one of them is superior to the others. The superiority of a method cannot be discussed unless we know the application, data type, AI algorithm, users, etc. For example, for an object image classification task based on a random forest, non-expert users can underestand a saliency map more easily than ALE. \par\par\begin{table}\centering\@@toccaption{{\lx@tag[ ]{{4}}{Mapping Function Table to Map Approaches in XAI to Users and Desired Properties.}}}\@@caption{{\lx@tag[: ]{{Table 4}}{Mapping Function Table to Map Approaches in XAI to Users and Desired Properties.}}}\leavevmode\resizebox{390.25534pt}{202.3529pt}{ \begin{tabular}[]{ | l | l| l|l|l|}\hline\cr Approach&Potential Users&Potential goals&Potential Properties&Possible Data Format\\\ \hline\cr Counterfactual Explanations&Stakeholders&System Intervention&Plausibility&Tabular\\\ &Expert Users&Moral Values&Model-agnostic&Image\\\ &Non-expert Users&Trustworthiness&Sparsity&Text\\\ &&Law Requirement&Causality&\\\ &&&Run Time&\\\ &&&Data Manifold Closeness&\\\ \hline\cr Saliency Map&Stakeholders&System Validation&Model-agnostic&Tabular\\\ &Expert Users&System Debugging&Run Time&Image\\\ &Non-expert Users&Eliminate Bias&&Text\\\ &&Moral Values&&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr Partial Dependence Plot (PDP)&Stakeholders&System Validation&Causality&Tabular\\\ &Expert Users&System Debugging&Model- agnostic&\\\ &Non-expert Users&Eliminate Bias&Run Time&\\\ &&Moral Values&Data Manifold Closeness&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr Accumulated Local Effect (ALE)&Stakeholders&System Validation&Causality&Tabular\\\ &Expert Users&System Debugging&Model-agnostic&\\\ &Non-expert users&Eliminate Bias&Run Time&\\\ &&Moral Values&Data Manifold Closeness&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr Individual Conditional Expectation (ICE)&Stakeholders&System Validation&Causality&Tabular\\\ &Expert users&System Debugging&Model-agnostic&\\\ &Non-expert users&Eliminate Bias&Run Time&\\\ &&Moral Values&Data Manifold Closeness&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr Local Interpretable Explanations (LIME)&Stakeholders&System Validation&Causality&Tabular\\\ Model-agnostic&Expert Users&System Debugging&Model- agnostic&Image\\\ &Non-expert Users&Eliminate Bias&Run Time&Text\\\ &&Moral Values&&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr SHAP (SHapley Additive exPlanations)&Stakeholders&System Validation&Causality&Tabular\\\ &Expert Users&System Debugging&Model-agnostic&Image\\\ &Non-expert Users&Eliminate Bias&Run Time&Text\\\ &&Moral Values&&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr Layer-wise Relevance Propagation (LRP)&Stakeholders&System Validation&Causality&Tabular\\\ &Expert users&System Debugging&Model-agnostic&Image\\\ &Non-expert users&Eliminate Bias&Run Time&Text\\\ &&Moral Values&&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr Class Activation Map (CAM)&Stakeholders&System Validation&Causality&Tabular\\\ &Expert users&System Debugging&Model-agnostic&Image\\\ &Non-expert users&Eliminate Bias&Run Time&Text\\\ &&Moral Values&&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr CLass-Enhanced Attentive Response (CLEAR)&Stakeholders&System Validation&Causality&Tabular\\\ &Expert Users&System Debugging&Model-agnostic&Image\\\ &Non-expert Users&Eliminate Bias&Run Time&\\\ &&Moral Values&&\\\ &&Trustworthiness&&\\\ &&Law Requirement&&\\\ \hline\cr\end{tabular} } \@add@centering\end{table} \par\par In this section, we reviewed the most important methods in XAI and evaluated each method's ability to do a task and desired properties for each method. The output of this section is methods that will work better in practice. XAI system developers can use the provided information to select the match method based on their purpose of using XAI and develop properties that are aligned with their intentions. \par\par\par\par\par\par\par\@@numbered@section{section}{toc}{Conclusion} \par\par This paper explores previous works in the field of XAI from a new perspective and categorizes them into three categories: philosophy, practice, and theory. In section 2, several shortcomings were identified, including: \par\begin{enumerate} \par\enumerate@item@The lack of well-defined users for XAI, which results in a lack of understanding about what is desired from XAI. \par\enumerate@item@The mapping between the desired properties of XAI and the different applications that require them has not yet been established. \par\enumerate@item@Current works do not take into account individual preferences for features and explanations. \par\enumerate@item@The reasons for the poor performance of XAI in practice are not yet fully understood. \par\enumerate@item@It is not yet clear how to identify immutable and mutable features when no information about the feature set is available. \par\end{enumerate} \par Each of the mentioned objectives can be a new research direction. \par In the third section, we debated XAI users and why each user is using XAI. After that, various methods and their desired properties have been evaluated. The final conclusion of this section was a mapping function that can map each user to different goals and map different goals to suitable properties. Also, we mapped XAI methods to their expected properties and related goals. \par\par\backmatter\par\par\par\par\thebibliography\bibcommenthead\@@lbibitem{xu2019explainable}\NAT@@wrout{1}{}{}{}{(1)}{xu2019explainable}\lx@bibnewblock Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., Zhu, J.: Explainable ai: A brief survey on history, research areas, approaches and challenges. In: CCF International Conference on Natural Language Processing and Chinese Computing, pp. 563--574 (2019). Springer \par\@@lbibitem{https://doi.org/10.48550/arxiv.1710.00794}\NAT@@wrout{2}{}{}{}{(2)}{https://doi.org/10.48550/arxiv.1710.00794}\lx@bibnewblock Doran, D., Schulz, S., Besold, T.R.: What Does Explainable AI Really Mean? A New Conceptualization of Perspectives. arXiv (2017). \url{https://doi.org/10.48550/ARXIV.1710.00794}. \url{https://arxiv.org/abs/1710.00794} \par\@@lbibitem{https://doi.org/10.48550/arxiv.1812.04608}\NAT@@wrout{3}{}{}{}{(3)}{https://doi.org/10.48550/arxiv.1812.04608}\lx@bibnewblock Hoffman, R.R., Mueller, S.T., Klein, G., Litman, J.: Metrics for Explainable AI: Challenges and Prospects. arXiv (2018). \url{https://doi.org/10.48550/ARXIV.1812.04608}. \url{https://arxiv.org/abs/1812.04608} \par\@@lbibitem{byrne2019counterfactuals}\NAT@@wrout{4}{}{}{}{(4)}{byrne2019counterfactuals}\lx@bibnewblock Byrne, R.M.: Counterfactuals in explainable artificial intelligence (xai): Evidence from human reasoning. In: IJCAI, pp. 6276--6282 (2019) \par\@@lbibitem{hume1978treatise}\NAT@@wrout{5}{}{}{}{(5)}{hume1978treatise}\lx@bibnewblock Hume, D.: A Treatise of Human Nature. Clarendon Press, ??? (1896) \par\@@lbibitem{Daniel}\NAT@@wrout{6}{}{}{}{(6)}{Daniel}\lx@bibnewblock Daniel, K., Tversky, A.: The Psychology of Preferences, pp. 160--73. Scientific American 246.1, ??? (1982) \par\@@lbibitem{Amann2020}\NAT@@wrout{7}{}{}{}{(7)}{Amann2020}\lx@bibnewblock Amann, J., Blasimme, A., Vayena, E., et al.: Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Med Inform Decis Mak {20}, 310 (2020). \url{https://doi.org/10.1186/s12911-020-01332-6} \par\@@lbibitem{Amann2022}\NAT@@wrout{8}{}{}{}{(8)}{Amann2022}\lx@bibnewblock Amann, J., Vetter, D., Blomberg, S.N., et al.: To explain or not to explain?—artificial intelligence explainability in clinical decision support systems. PLOS Digit Health {1(2)} (2022). \url{https://doi.org/10.1371/journal.pdig.0000016} \par\@@lbibitem{keane2021if}\NAT@@wrout{9}{}{}{}{(9)}{keane2021if}\lx@bibnewblock Keane, M.T., Kenny, E.M., Delaney, E., Smyth, B.: If only we had better counterfactual explanations: Five key deficits to rectify in the evaluation of counterfactual xai techniques. arXiv preprint arXiv:2103.01035 (2021) \par\@@lbibitem{adadi2018peeking}\NAT@@wrout{10}{}{}{}{(10)}{adadi2018peeking}\lx@bibnewblock Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access {6}, 52138--52160 (2018) \par\@@lbibitem{herm2021don}\NAT@@wrout{11}{}{}{}{(11)}{herm2021don}\lx@bibnewblock Herm, L.-V., Wanner, J., Seubert, F., Janiesch, C.: I don’t get it, but it seems valid! the connection between explainability and comprehensibility in (x) ai research. In: European Conference on Information Systems (ECIS), Virtual Conference, AIS (2021) \par\@@lbibitem{https://doi.org/10.48550/arxiv.2107.04427}\NAT@@wrout{12}{}{}{}{(12)}{https://doi.org/10.48550/arxiv.2107.04427}\lx@bibnewblock Vermeire, T., Laugel, T., Renard, X., Martens, D., Detyniecki, M.: How to choose an Explainability Method? Towards a Methodical Implementation of XAI in Practice. arXiv (2021). \url{https://doi.org/10.48550/ARXIV.2107.04427}. \url{https://arxiv.org/abs/2107.04427} \par\@@lbibitem{matz2021explaining}\NAT@@wrout{13}{}{}{}{(13)}{matz2021explaining}\lx@bibnewblock Matz, F., Luo, Y.: Explaining Automated Decisions in Practice: Insights from the Swedish Credit Scoring Industry (2021) \par\@@lbibitem{verma2020counterfactual}\NAT@@wrout{14}{}{}{}{(14)}{verma2020counterfactual}\lx@bibnewblock Verma, S., Dickerson, J., Hines, K.: Counterfactual explanations for machine learning: A review. arXiv preprint arXiv:2010.10596 (2020) \par\@@lbibitem{miller2019explanation}\NAT@@wrout{15}{}{}{}{(15)}{miller2019explanation}\lx@bibnewblock Miller, T.: Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence {267}, 1--38 (2019) \par\@@lbibitem{verma2021counterfactual}\NAT@@wrout{16}{}{}{}{(16)}{verma2021counterfactual}\lx@bibnewblock Verma, S., Dickerson, J., Hines, K.: Counterfactual explanations for machine learning: Challenges revisited. arXiv preprint arXiv:2106.07756 (2021) \par\@@lbibitem{brennen2020people}\NAT@@wrout{17}{}{}{}{(17)}{brennen2020people}\lx@bibnewblock Brennen, A.: What do people really want when they say they want" explainable ai?" we asked 60 stakeholders. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1--7 (2020) \par\@@lbibitem{alufaisan2020does}\NAT@@wrout{18}{}{}{}{(18)}{alufaisan2020does}\lx@bibnewblock Alufaisan, Y., Marusich, L.R., Bakdash, J.Z., Zhou, Y., Kantarcioglu, M.: Does explainable artificial intelligence improve human decision-making? arXiv preprint arXiv:2006.11194 (2020) \par\@@lbibitem{gerlings2020reviewing}\NAT@@wrout{19}{}{}{}{(19)}{gerlings2020reviewing}\lx@bibnewblock Gerlings, J., Shollo, A., Constantiou, I.: Reviewing the need for explainable artificial intelligence (xai). arXiv preprint arXiv:2012.01007 (2020) \par\@@lbibitem{chromik2021think}\NAT@@wrout{20}{}{}{}{(20)}{chromik2021think}\lx@bibnewblock Chromik, M., Eiband, M., Buchner, F., Kr{\"{u}}ger, A., Butz, A.: I think i get your point, ai! the illusion of explanatory depth in explainable ai. In: 26th International Conference on Intelligent User Interfaces, pp. 307--317 (2021) \par\@@lbibitem{ribeiro2016should}\NAT@@wrout{21}{}{}{}{(21)}{ribeiro2016should}\lx@bibnewblock Ribeiro, M.T., Singh, S., Guestrin, C.: " why should i trust you?" explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135--1144 (2016) \par\@@lbibitem{lundberg2017unified}\NAT@@wrout{22}{}{}{}{(22)}{lundberg2017unified}\lx@bibnewblock Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. Advances in neural information processing systems {30} (2017) \par\@@lbibitem{friedman2001greedy}\NAT@@wrout{23}{}{}{}{(23)}{friedman2001greedy}\lx@bibnewblock Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Annals of statistics, 1189--1232 (2001) \par\@@lbibitem{apley2020visualizing}\NAT@@wrout{24}{}{}{}{(24)}{apley2020visualizing}\lx@bibnewblock Apley, D.W., Zhu, J.: Visualizing the effects of predictor variables in black box supervised learning models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) {82}(4), 1059--1086 (2020) \par\@@lbibitem{wachter2017counterfactual}\NAT@@wrout{25}{}{}{}{(25)}{wachter2017counterfactual}\lx@bibnewblock Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL \& Tech. {31}, 841 (2017) \par\@@lbibitem{ustun2019actionable}\NAT@@wrout{26}{}{}{}{(26)}{ustun2019actionable}\lx@bibnewblock Ustun, B., Spangher, A., Liu, Y.: Actionable recourse in linear classification. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 10--19 (2019) \par\@@lbibitem{lucic2019focus}\NAT@@wrout{27}{}{}{}{(27)}{lucic2019focus}\lx@bibnewblock Lucic, A., Oosterhuis, H., Haned, H., de Rijke, M.: Focus: Flexible optimizable counterfactual explanations for tree ensembles. arXiv preprint arXiv:1911.12199 (2019) \par\@@lbibitem{karimi2020model}\NAT@@wrout{28}{}{}{}{(28)}{karimi2020model}\lx@bibnewblock Karimi, A.-H., Barthe, G., Balle, B., Valera, I.: Model-agnostic counterfactual explanations for consequential decisions. In: International Conference on Artificial Intelligence and Statistics, pp. 895--905 (2020). PMLR \par\@@lbibitem{Dua:2019}\NAT@@wrout{29}{}{}{}{(29)}{Dua:2019}\lx@bibnewblock Dua, D., Graff, C.: UCI Machine Learning Repository (2017). \url{http://archive.ics.uci.edu/ml} \par\@@lbibitem{YEH20092473}\NAT@@wrout{30}{}{}{}{(30)}{YEH20092473}\lx@bibnewblock Yeh, I.-C., Lien, C.-h.: The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications {36}(2, Part 1), 2473--2480 (2009). \url{https://doi.org/10.1016/j.eswa.2007.12.020} \par\@@lbibitem{Larson}\NAT@@wrout{31}{}{}{}{(31)}{Larson}\lx@bibnewblock Larson, J., Mattu, S., Kirchner, L., Angwin, J. \url{https://doi.org/https://github.com/propublica/compas-analysis.} \par\@@lbibitem{karimi2020algorithmic}\NAT@@wrout{32}{}{}{}{(32)}{karimi2020algorithmic}\lx@bibnewblock Karimi, A.-H., Von~{}K{\"{u}}gelgen, J., Sch{\"{o}}lkopf, B., Valera, I.: Algorithmic recourse under imperfect causal knowledge: a probabilistic approach. Advances in Neural Information Processing Systems {33}, 265--277 (2020) \par\@@lbibitem{karimi2021algorithmic}\NAT@@wrout{33}{}{}{}{(33)}{karimi2021algorithmic}\lx@bibnewblock Karimi, A.-H., Sch{\"{o}}lkopf, B., Valera, I.: Algorithmic recourse: from counterfactual explanations to interventions. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 353--362 (2021) \par\@@lbibitem{mohammadi2021scaling}\NAT@@wrout{34}{}{}{}{(34)}{mohammadi2021scaling}\lx@bibnewblock Mohammadi, K., Karimi, A.-H., Barthe, G., Valera, I.: Scaling guarantees for nearest counterfactual explanations. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 177--187 (2021) \par\@@lbibitem{karimi2021survey}\NAT@@wrout{35}{}{}{}{(35)}{karimi2021survey}\lx@bibnewblock Karimi, A.-H., Barthe, G., Sch{\"{o}}lkopf, B., Valera, I.: A survey of algorithmic recourse: contrastive explanations and consequential recommendations. ACM Computing Surveys (CSUR) (2021) \par\@@lbibitem{mothilal2020explaining}\NAT@@wrout{36}{}{}{}{(36)}{mothilal2020explaining}\lx@bibnewblock Mothilal, R.K., Sharma, A., Tan, C.: Explaining machine learning classifiers through diverse counterfactual explanations. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607--617 (2020) \par\@@lbibitem{Kevin2015}\NAT@@wrout{37}{}{}{}{(37)}{Kevin2015}\lx@bibnewblock Davenport, K.: Lending Club Data Analysis Revisited with Python,. \url{https://doi.org/http://kldavenport.com/lending-club-data-analysis-revisted-with-python/} \par\@@lbibitem{wang2021skyline}\NAT@@wrout{38}{}{}{}{(38)}{wang2021skyline}\lx@bibnewblock Wang, Y., Ding, Q., Wang, K., Liu, Y., Wu, X., Wang, J., Liu, Y., Miao, C.: The skyline of counterfactual explanations for machine learning decision models. In: Proceedings of the 30th ACM International Conference on Information \& Knowledge Management, pp. 2030--2039 (2021) \par\@@lbibitem{kanamori2021ordered}\NAT@@wrout{39}{}{}{}{(39)}{kanamori2021ordered}\lx@bibnewblock Kanamori, K., Takagi, T., Kobayashi, K., Ike, Y., Uemura, K., Arimura, H.: Ordered counterfactual explanation by mixed-integer linear optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11564--11574 (2021) \par\@@lbibitem{hada2021exploring}\NAT@@wrout{40}{}{}{}{(40)}{hada2021exploring}\lx@bibnewblock Hada, S.S., Carreira-Perpin{\'{a}}n, M.: Exploring counterfactual explanations for classification and regression trees. In: XKDD (ECML PKDD International Workshop) (2021) \par\@@lbibitem{verma2021amortized}\NAT@@wrout{41}{}{}{}{(41)}{verma2021amortized}\lx@bibnewblock Verma, S., Hines, K., Dickerson, J.P.: Amortized generation of sequential counterfactual explanations for black-box models. arXiv preprint arXiv:2106.03962 (2021) \par\@@lbibitem{dhurandhar2019model}\NAT@@wrout{42}{}{}{}{(42)}{dhurandhar2019model}\lx@bibnewblock Dhurandhar, A., Pedapati, T., Balakrishnan, A., Chen, P.-Y., Shanmugam, K., Puri, R.: Model agnostic contrastive explanations for structured data. arXiv preprint arXiv:1906.00117 (2019) \par\@@lbibitem{kanamori2020dace}\NAT@@wrout{43}{}{}{}{(43)}{kanamori2020dace}\lx@bibnewblock Kanamori, K., Takagi, T., Kobayashi, K., Arimura, H.: Dace: Distribution-aware counterfactual explanation by mixed-integer linear optimization. In: IJCAI, pp. 2855--2862 (2020) \par\@@lbibitem{mahajan2019preserving}\NAT@@wrout{44}{}{}{}{(44)}{mahajan2019preserving}\lx@bibnewblock Mahajan, D., Tan, C., Sharma, A.: Preserving causal constraints in counterfactual explanations for machine learning classifiers. arXiv preprint arXiv:1912.03277 (2019) \par\@@lbibitem{blitzer2007biographies}\NAT@@wrout{45}{}{}{}{(45)}{blitzer2007biographies}\lx@bibnewblock Blitzer, J., Dredze, M., Pereira, F.: Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 440--447 (2007) \par\@@lbibitem{bike}\NAT@@wrout{46}{}{}{}{(46)}{bike}\lx@bibnewblock Fanaee-T, H., Gama, J.: Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, 1--15 (2013). \url{https://doi.org/10.1007/s13748-013-0040-3} \par\@@lbibitem{FICO1}\NAT@@wrout{47}{}{}{}{(47)}{FICO1}\lx@bibnewblock FICO, Explainable Machine Learning Challenge. \url{URLhttps://community.fico.com/s/explainable-machine-learning-challenge.} \par\@@lbibitem{Ofer2017}\NAT@@wrout{48}{}{}{}{(48)}{Ofer2017}\lx@bibnewblock Ofer, D.: COMPAS Dataset. \url{https://www.kaggle.com/danofer/compass.} \par\@@lbibitem{eu-269-2014}\NAT@@wrout{49}{}{}{}{(49)}{eu-269-2014}\lx@bibnewblock {Council of European Union}: Council regulation ({EU}) no 269/2014. \newline \url{http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1416170084502&uri=CELEX:32014R0269} (2014) \par\@@lbibitem{simonyan2013deep}\NAT@@wrout{50}{}{}{}{(50)}{simonyan2013deep}\lx@bibnewblock Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013) \par\@@lbibitem{goldstein2015peeking}\NAT@@wrout{51}{}{}{}{(51)}{goldstein2015peeking}\lx@bibnewblock Goldstein, A., Kapelner, A., Bleich, J., Pitkin, E.: Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. journal of Computational and Graphical Statistics {24}(1), 44--65 (2015) \par\@@lbibitem{bach2015pixel}\NAT@@wrout{52}{}{}{}{(52)}{bach2015pixel}\lx@bibnewblock Bach, S., Binder, A., Montavon, G., Klauschen, F., M{\"{u}}ller, K.-R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one {10}(7), 0130140 (2015) \par\@@lbibitem{lapuschkin2019unmasking}\NAT@@wrout{53}{}{}{}{(53)}{lapuschkin2019unmasking}\lx@bibnewblock Lapuschkin, S., W{\"{a}}ldchen, S., Binder, A., Montavon, G., Samek, W., M{\"{u}}ller, K.-R.: Unmasking clever hans predictors and assessing what machines really learn. Nature communications {10}(1), 1--8 (2019) \par\@@lbibitem{arbabzadah2016identifying}\NAT@@wrout{54}{}{}{}{(54)}{arbabzadah2016identifying}\lx@bibnewblock Arbabzadah, F., Montavon, G., M{\"{u}}ller, K.-R., Samek, W.: Identifying individual facial expressions by deconstructing a neural network. In: German Conference on Pattern Recognition, pp. 344--354 (2016). Springer \par\@@lbibitem{zhou2016learning}\NAT@@wrout{55}{}{}{}{(55)}{zhou2016learning}\lx@bibnewblock Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921--2929 (2016) \par\@@lbibitem{kumar2017explaining}\NAT@@wrout{56}{}{}{}{(56)}{kumar2017explaining}\lx@bibnewblock Kumar, D., Wong, A., Taylor, G.W.: Explaining the unexplained: A class-enhanced attentive response (clear) approach to understanding deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 36--44 (2017) \par\@@lbibitem{yang2020generating}\NAT@@wrout{57}{}{}{}{(57)}{yang2020generating}\lx@bibnewblock Yang, L., Kenny, E.M., Ng, T.L.J., Yang, Y., Smyth, B., Dong, R.: Generating plausible counterfactual explanations for deep transformers in financial text classification. arXiv preprint arXiv:2010.12512 (2020) \par\endthebibliography \par\par\@add@PDF@RDFa@triples\par\end{document}$ (13)