# Interpretable Machine Learning for Discovery: Statistical Challenges &
Opportunities

Genevera I. Allen1,2, Luqin Gan2, and Lili Zheng3

###### Abstract

New technologies have led to vast troves of large and complex datasets across
many scientific domains and industries. People routinely use machine learning
techniques to not only process, visualize, and make predictions from this big
data, but also to make data-driven discoveries. These discoveries are often
made using Interpretable Machine Learning, or machine learning models and
techniques that yield human understandable insights. In this paper, we discuss
and review the field of interpretable machine learning, focusing especially on
the techniques as they are often employed to generate new knowledge or make
discoveries from large data sets. We outline the types of discoveries that can
be made using Interpretable Machine Learning in both supervised and
unsupervised settings. Additionally, we focus on the grand challenge of how to
validate these discoveries in a data-driven manner, which promotes trust in
machine learning systems and reproducibility in science. We discuss validation
from both a practical perspective, reviewing approaches based on data-
splitting and stability, as well as from a theoretical perspective, reviewing
statistical results on model selection consistency and uncertainty
quantification via statistical inference. Finally, we conclude by highlighting
open challenges in using interpretable machine learning techniques to make
discoveries, including gaps between theory and practice for validating data-
driven-discoveries.

Keywords: machine learning; interpretability; explainability; data-driven
discoveries; validation; stability; selection consistency; uncertainty
quantification

11footnotetext: Departments of Electrical and Computer Engineering and
Computer Science, Rice University, Houston, TX, USA 77005; Neurological
Research Institute, Baylor College of Medicine, Houston, TX, USA 77030; Email:
<EMAIL_ADDRESS>Department of Statistics, Rice University,
Houston, TX, USA 77005.33footnotetext: Department of Electrical and Computer
Engineering, Rice University, Houston, TX, USA 77005.

##  1 Introduction

Machine learning systems have gained widespread use in science, technology,
and society. Given the increasing number of high-stakes machine learning
applications and the growing complexity of machine learning models, many have
advocated for interpretability and explainability to promote understanding and
trust in machine learning results (Rasheed et al. ,, 2022, Toreini et al. ,,
2020, Broderick et al. ,, 2023). In response, there has been a recent
explosion of research on Interpretable Machine Learning (IML), mostly focusing
on new techniques to interpret black-box systems; see Molnar, (2022), Lipton,
(2018), Guidotti et al. , (2018), Doshi-Velez & Kim, (2017), Du et al. ,
(2019), Murdoch et al. , (2019), Carvalho et al. , (2019) for recent reviews
of the IML and explainable artificial intelligence literature. While most of
these interpretability techniques were not necessarily designed for this
purpose, they are increasingly being used to mine large and complex data sets
to generate new insights (Roscher et al. ,, 2020). These so-called data-driven
discoveries are especially important to advance data-rich fields in science,
technology, and medicine. While prior reviews focus mainly on IML techniques,
we primarily review how IML methods promote data-driven discoveries,
challenges associated with this task, and related new research opportunities
at the intersection of machine learning and statistics.

In the sciences and beyond, IML techniques are routinely employed to make new
discoveries from large and complex data sets; to motivate our review on this
topic, we highlight several examples. First, feature importance and feature
selection in supervised learning are popular forms of interpretation that have
led to major discoveries like discovering new genomic biomarkers of diseases
(Guyon et al. ,, 2002), discovering physical laws governing dynamical systems
(Brunton et al. ,, 2016), and discovering lesions and other abnormalities in
radiology (Borjali et al. ,, 2020, Reyes et al. ,, 2020). While most of the
IML literature focuses on supervised learning (Molnar,, 2022, Lipton,, 2018,
Guidotti et al. ,, 2018, Doshi-Velez & Kim,, 2017), there have been many major
scientific discoveries made via unsupervised techniques and we argue that
these approaches should be included in any discussion of IML. For example, one
of the earliest and most important machine learning findings in medicine was
the discovery of genomic subtypes of breast cancer using hierarchical
clustering of gene expression data (Perou et al. ,, 2000), which led to new
ways to diagnose and treat cancer based on a patient’s specific genomic
subtype and ushered in an era of personalized medicine (Hassan et al. ,,
2022). Clustering techniques have also been used to discover galaxies from
astronomical surveys (Materne,, 1978) and discover communities with similar
political affiliations (Ozer et al. ,, 2016). Other major unsupervised
discoveries include discovering major climate patterns like EL Nino and their
localized effects via dimension reduction (Houghton & Wilson,, 2020) and
discovering the functional organization of the brain via network models
(Rubinov & Sporns,, 2010). These are just a few of many examples of how IML
techniques have led to new scientific discoveries. As the size and complexity
of scientific data continues to grow, IML techniques will be ever more
valuable for mining this data to generate new findings and advance science,
hence motivating our review on this topic.

In this article, we review IML for the purpose of generating new data-driven
discoveries. We also discuss several challenges that come with using IML for
discovery, review statistical and other research that has sought to address
these challenges, and highlight many associated open research opportunities.
We organize this article by first reviewing the extensive IML literature in
Section 2. Next in Section 3, we review IML techniques, but instead of
organizing this according to technique-type as in most other IML reviews, we
discuss IML techniques as they are used to generate different discovery types.
Our discussion includes both supervised and unsupervised techniques, given the
importance of the latter for making discoveries. In order for IML findings to
lead to accurate discoveries, however, we need them to be replicable and
reliable (Yu & Kumbier,, 2020), which also promotes trust in machine learning
results (Rasheed et al. ,, 2022, Toreini et al. ,, 2020, Broderick et al. ,,
2023). In other words, we need approaches to validate IML discoveries. But
unfortunately, validation for IML is not widely discussed or applied in
practice as it presents many more challenges than validating ML predictions.
In Section 4, we discuss the grand challenge of validating IML discoveries and
review several practical validation strategies with examples. Then in Section
5, we approach validation from a theoretical perspective and review
statistical theory and statistical inference approaches that can help
determine when IML techniques will find the desired discovery with high
probability (Section 5.1) as well as help quantify the uncertainty in IML
discoveries via confidence intervals and statistical hypothesis testing
(Section 5.2). We finally conclude with a discussion of the major open
problems and opportunities in IML for discovery in Section 6.

##  2 Interpretable Machine Learning: Definitions, Rationale, & Categories

Before focusing on IML for making discoveries, we review the growing
literature on IML. We discuss definitions, reasons for using IML, and
taxonomies that provide a systematic way to describe IML techniques. These are
summarized in Figure 1.

Figure 1: Overview of Interpretable Machine Learning.

###  2.1 What is Interpretable Machine Learning?

Many have discussed IML, yet there is not a universally accepted consensus
definition (Murdoch et al. ,, 2019, Roscher et al. ,, 2020, Du et al. ,, 2019,
Rudin,, 2014, Barredo Arrieta et al. ,, 2020). Imprecise definitions have
likely led to a lack of consensus on how to study and validate IML techniques,
a major concern when these methods are used to make data-driven discoveries
(Rudin et al. ,, 2022, Gilpin et al. ,, 2018). We adopt a broad definition of
interpretable machine learning: Interpretable machine learning is the use of
machine learning techniques to generate human-understandable insights into
data, the learned model, or the model output. In other words, interpretable
machine learning is very general and provides an understanding of any aspect
of the machine learning process: the model inputs (data), the model insides or
model guts (the model parameters or learned model, or even how the model
interacts with data), and the model outputs (predictions or decisions based on
the data and model). As many have noted, what is considered a human-
understandable insight depends on the intended audience and the domain area;
thus, interpretations in machine learning are domain, problem, and audience
specific (Murdoch et al. ,, 2019, Roscher et al. ,, 2020).

###  2.2 Why Interpretability?

Why do we need interpretability in machine learning? Many have proposed a
number of reasons and uses for interpretable machine learning (Du et al. ,,
2019, Murdoch et al. ,, 2019, Molnar,, 2022, Lipton,, 2018, Guidotti et al. ,,
2018, Carvalho et al. ,, 2019, Doshi-Velez & Kim,, 2017, Roscher et al. ,,
2020) which we briefly review here.

Model Validation. When fitting complex machine learning systems, the modeler
may need to check that the model is performing and behaving in the desired
manner, or perform model validation. One may ask: Does this model make sense?
Is this model consistent with my prior expectations or knowledge about the
system? This form of human validation requires human interpretable machine
learning models.

Model Debugging. When something goes wrong in a machine learning system, how
can one diagnose a problem in a machine learning system if they don’t
understand the model and how it interacts with the data? Interpreting and
understanding machine learning systems are critical for diagnosing, debugging
and fixing systems (Koh & Liang,, 2017).

Transparency, Accountability & Trust. Interpretable machine learning
approaches often help to make black-box and other machine learning systems
easier for humans to understand and hence more transparent. This transparency
is critical for promoting accountability and trust of machine learning systems
which are necessary for their utilization in high-stakes societal applications
(Rudin,, 2019, Samek & Müller,, 2019, Xu et al. ,, 2019).

Ethics. There has been an increasing focus on ensuring that machine learning
algorithms are fair and ethical (Doshi-Velez & Kim,, 2017). Due to biases that
exist in our society, machine learning algorithms that are trained on possibly
biased data can often exacerbate these biases leading to unfair predictions
that are discriminatory (Guidotti et al. ,, 2018). Understandable machine
learning techniques are needed to both assess and improve the fairness of
machine learning in critical societal applications.

Data Exploration. John Tukey coined the term Exploratory Data Analysis and
promoted this as the critical first stage of data analysis (Tukey,, 1977).
Human-interpretable techniques can help gain insights into major patterns,
trends, groups, or artifacts of the data. These data exploration insights are
then used to clean and prepare data for modeling, make downstream modeling
decisions, and visualize and interpret model outputs (Murdoch et al. ,, 2019,
Berkhin,, 2006).

Discovery. As data has grown in size and complexity, we often rely on machine
learning techniques to make discoveries, or in other words, find rare signals
in a sea of data. Using interpretable machine learning techniques to make
data-driven discoveries is the main focus of this review.

###  2.3 A Taxonomy of IML Techniques

Recently, many have discussed interpretable machine learning techniques and
proposed various categorizations to systematize discussion and evaluation of
the approaches (Molnar,, 2022, Lipton,, 2018, Guidotti et al. ,, 2018, Doshi-
Velez & Kim,, 2017). While there is not complete agreement in the literature
on these categories, we discuss three main dimensions or axes along which most
interpretable machine learning techniques lie and give examples of methods
falling under each designation. We also discuss how these categories of
techniques relate to the task of using IML methods for generating new
discoveries.

####  2.3.1 Intrinsic vs. Post-hoc Interpretability

A major axis that differentiates IML techniques is intrinsic versus post-hoc
interpretability. Intrinsic interpretations are understandings that are
inherent in the fitted model itself. In other words, the user needs to simply
fit a model to produce the desired interpretation. Examples include trees,
additive models, or regularization approaches which make the fitted model more
understandable by adding constraints like sparsity or smoothness. More
recently in deep learning, many have proposed models that are more
intrinsically interpretable by constraining the final layer in a deep neural
network to follow certain prototypes or interpretability constraints (Dong et
al. ,, 2017, Rudin,, 2019). In contrast, post-hoc interpretations require a
secondary technique to be applied to the fitted model or model outputs for the
sole purpose of interpretation. Examples of post-hoc interpretations include
backpropagation-related methods which traverse the learned neural network
architecture to assign importance scores to each feature and local
interpretable model explanations (LIME) which fits a second, simple, and
interpretable model approximating the black-box model at a particular input
(Molnar,, 2022). Additionally, most supervised model-agnostic interpretations,
discussed subsequently, are post-hoc in nature. Very little attention has been
paid to unsupervised learning techniques in the context of IML. But, we argue
that all unsupervised learning techniques are naturally intrinsically
interpretable as their objective is to find some meaningful structure that
helps the user gain insights into the data, hence falling under our definition
of IML. One can still use post-hoc interpretations of unsupervised findings,
however. Consider that after clustering, one may perform a secondary analysis
to determine which features are most responsible for separating the clusters
(Satija et al. ,, 2015).

Many have argued that intrinsic interpretations are preferable to post-hoc
interpretations (Lipton,, 2018, Varshney,, 2019, Rudin,, 2019). For the
purpose of making data-driven discoveries, however, there is not a particular
preference as long as the interpretations accurately capture the discovery of
interest. For intrinsic interpretations, this means the model must fit the
data well and closely approximate the true generating model for the
interpretations to reflect true discoveries. In linear regression, for
example, the intrinsic interpretation of feature importance based on estimated
coefficients will only be accurate if the true underlying model is linear or
approximately linear. For post-hoc interpretations, on the other hand, both
the original model and the secondary analysis must accurately capture the
data-generating process to yield accurate interpretations. If a deep learning
model fits the data well, but a secondary analysis with LIME does not
sufficiently capture the original deep learning model, then interpretations
and resulting discoveries will not be accurate (Zhang et al. ,, 2019).

####  2.3.2 Model-Specific vs. Model-Agnostic Interpretations

Another dimension along which we can categorize IML techniques is by whether
they are model-specific or model-agnostic. Model-specific interpretations are
tailored to the model and cannot generalize across models. Model-agnostic
interpretations can be applied to any model and interpreted in a similar
manner for all models. There are many forms of model-specific feature
importance; these include coefficients in generalized linear or additive
models, feature importance scores for trees, or the plethora of deep learning
specific techniques for feature attribution like backpropagation methods
(Molnar,, 2022). On the other hand, there are several model-agnostic feature
importance methods that can be used for any supervised model; these include
Shapley values, feature permutations, feature occlusion, and LIME (Molnar,,
2022). Note that model-specific interpretations are not necessarily intrinsic
interpretations; consider that feature importance scores for trees and guided
backpropagation feature attribution are both model-specific but post-hoc. In
contrast, most model-agnostic interpretations are post-hoc in nature.

For the same reason that many prefer intrinsic interpretations to post-hoc
interpretations, many have argued that model-specific interpretations are
preferable (Ribeiro et al. ,, 2016, Lundberg & Lee,, 2017). Yet, we point out
that there are several advantages to model-agnostic interpretations.
Importantly, model-agnostic interpretations can be understood in the same way
across all models. This is particularly useful for model comparisons; for
feature importance, for example, it allows one to compare importance scores
for each feature derived from several model families. We argue that this
advantage is especially important for the task of making data-driven
discoveries where validating the interpretations is critical. One easy
approach is to try many different model families and check if the
interpretations are the same across all of these models. This type of
validation is easier with model-agnostic interpretations that can be directly
compared. On the other hand, it is typically easier to study model-specific
interpretations theoretically to understand under what conditions the
resulting discoveries accurately recover some aspect of the true model, a
topic we will discuss further in Section 5.

####  2.3.3 Global vs. Local Interpretations

A final major dimension along which we can categorize IML techniques is based
on whether the approach offers a local or a global interpretation. Global
interpretations reveal the overall structure of the fitted model. On the other
hand, local interpretations only yield model insights based on sub-parts of
the model input space; these could include local interpretations about a
single observation or a subset of the domain. To make these distinctions
concrete, again consider the example of feature importance in supervised
learning. Here, methods previously mentioned like coefficients in linear or
additive models, tree-based feature importance, and backpropagation-based
feature attribution are all global interpretations that capture the relevance
of each feature for all model predictions. In contrast, methods like LIME and
saliency maps, highlight the important features of a single new test instance
or observation (Ribeiro et al. ,, 2016, Molnar,, 2022). Similarly, in
unsupervised learning, consider the task of dimension reduction. Methods like
principal components analysis and spectral embedding yield global
interpretations, revealing global patterns represented in all observations in
each of the factors. In contrast, local embedding and neighborhood embedding
methods, like t-SNE and UMAP, highlight local interpretations through patterns
and relationships amongst particular neighborhoods.

When using interpretable machine learning to make discoveries, global
interpretations are more commonly employed as they reveal discoveries
reflective of all the input data and model landscape. Yet, local
interpretations are increasingly important to make discoveries amongst
subgroups of observations. An example application of this is in precision
medicine where we might seek to discover important genomic biomarkers for each
individual patient or sub-groups of similar patients.

##  3 Types of IML Discoveries & Techniques

Recent research in IML has produced an abundance of interpretability
techniques, as thoroughly reviewed in Molnar, (2022). But these works focus on
the types of techniques and not the types of data-driven discoveries that
various techniques can make. Here, we organize this section to highlight the
major types of discoveries achieved through interpretations of machine
learning models. Importantly and distinct from the IML literature, we place
great emphasis on unsupervised techniques, which are popularly used throughout
the sciences to make discoveries from unlabeled data.

Figure 2: Overview of the broad types of unsupervised and supervised data-
driven discoveries that can be made using interpretable machine learning
techniques.

###  3.1 Unsupervised Discoveries

Most work on IML has focused on supervised models and interpreting the results
of predictive systems (Rudin et al. ,, 2022). In scientific domains, however,
some of the most widespread uses of machine learning are in unsupervised
settings; hence in this section, we review major types of unsupervised
discoveries and highlight which types of IML techniques are employed to
generate these discoveries.

####  3.1.1 Groups

Uncovering hidden group structures in large datasets is a common and popular
type of unsupervised discovery. There are many well-established clustering
techniques used for this task including $K$-means, hierarchical clustering,
mixture modeling, and spectral clustering, among many others (Hennig et al. ,,
2015). Beyond group membership, other types of interpretations related to
clustering include uncovering groups of both observations and features
simultaneously via biclustering, discovering nested group structure via
hierarchical clustering, detecting localized important regions via spatial
clustering, and finding a subset of features that distinguish groups of
observations via sparse clustering (Witten & Tibshirani,, 2010). Clustering
has been applied broadly and is a nearly ubiquitous technique in unsupervised
and exploratory analysis; groups found via clustering have also led to several
major scientific discoveries such as finding gene expression patterns and/or
genomic subtypes of diseases like cancer (Perou et al. ,, 2000).

####  3.1.2 Patterns & Trends

When conducting unsupervised analyses, a typical first task is to visualize
and explore the data to look for major patterns and trends. Often, important
unsupervised discoveries can be made through these visual inspections of the
data. For large multivariate data, dimension reduction approaches reduce the
data down to a smaller number of components that retain important structure,
or patterns, in the data. There are a plethora of dimension reduction
techniques including linear approaches like principal components analysis
(PCA), non-negative matrix factorization and independent component analysis,
or non-linear approaches like spectral embedding, multi-dimensional scaling,
isomap, t-SNE, UMAP, or autoencoders; see Fodor, (2002) for a recent review of
such approaches. Each of these approaches is optimized to find slightly
different types of patterns. For example, PCA finds variance-maximizing
patterns that preserve the global structure whereas t-SNE finds localized
patterns that preserve neighborhood and group structure.

####  3.1.3 Associations

Discovering associations, or important relationships amongst features, is
another widely used type of unsupervised discovery. Most typically find linear
or nonlinear associations by exploring all possible pairwise interactions
amongst features, using correlation, mutual information, or other such
metrics. Recently, there has been a surge of interest to explore feature
relationships using graphical models (Lauritzen,, 1996). In Markov Networks,
or undirected graphical models, for example, the goal of structural learning
is to estimate conditional dependencies between features; structural learning
in Bayesian Networks or Directed Acyclic Graphs (DAG) seeks to learn directed
relationships and is an important part of causal discovery (Drton & Maathuis,,
2017).

####  3.1.4 Anomalies & Prototypes

Other types of unsupervised discoveries that are perhaps less commonly used
are finding anomalies (rare entities) or prototypes (typical entities).
Anomalies are rare but noteworthy observations. Techniques for anomaly
detection are similar to those for outlier detection; these include distance-
based approaches which often employ dimension reduction, clustering approaches
like single-linkage hierarchical clustering or DBSCAN, the one-class support
vector machine, or the isolation forest (Hodge & Austin,, 2004).
Alternatively, sometimes one seeks to find the most representative
observations, or prototypes. Adaptions of other unsupervised approaches,
especially dimension reduction and clustering, are typically employed for this
task (Bien & Tibshirani,, 2011).

###  3.2 Supervised Discoveries

Supervised learning has been the focus of the vast majority of the
interpretable machine learning literature (Molnar,, 2022, Lipton,, 2018,
Guidotti et al. ,, 2018, Doshi-Velez & Kim,, 2017). This occurs as some of the
best performing predictive models, such as deep learning and tree-based
ensembles, are essentially black-boxes that are not intrinsically
interpretable and difficult to decipher. Thus, interpretations of these
predictive models are critical for generating new insights and making data-
driven discoveries.

####  3.2.1 Feature Importance & Feature Selection

Perhaps the most common and popular form of interpretation in supervised
models is understanding how each feature influences a model’s predictions,
often referred to as feature importance. Related to this is feature selection
which finds the best subset of features that maximize predictive accuracy.
Importantly, feature importance and feature selection in supervised learning
offers a form of multivariate or conditional feature interpretation: given all
other features in the model, what is the added benefit of including a
particular feature? This conditional feature interpretation is much stronger
than marginally assessing how each feature relates to an outcome and has been
used extensively to discover important features. Let us review the many types
of methods for interpreting features in supervised learning through the
context of our IML taxonomies from Section 2.3. First, consider global and
model-specific feature importance metrics. For linear or generalized additive
models, the feature weight (or parameter or coefficient) can be directly
interpreted as the conditional feature importance, offering intrinsic feature
interpretability. Tree-based ensembles offer post-hoc interpretability by the
feature importance scores based on loss difference for each split. In deep
learning, post-hoc approaches are popular and include several feature
importance scores calculated via gradient-based methods which traverse the
fitted neural network to attribute relevance to each input feature (Samek et
al. ,, 2021). Local, post-hoc, and model-specific methods are popular in
computer vision where measures such as saliency maps and GradCAM highlight
which pixels in a specific image were used to generate the predicted label
(Samek et al. ,, 2021). There are also several model-agnostic metrics that can
be used with any supervised learning model, including methods that yield
global interpretations, like feature occlusion, feature permutation, and
Shapley values, as well as local interpretations such as LIME. See Molnar,
(2022) on these methods. There is an equally impressive literature on feature
selection for supervised learning; most of these strategies offer intrinsic
and model-specific interpretations by working with the empirical risk
minimizer or loss function. As finding the best subset of features is a
combinatorially hard optimization problem, people typically turn to greedy
step-wise methods, like Recursive Feature Elimination, or regularization
strategies that relax the best subset constraint. Popular approaches to the
latter include the $\ell_{1}$ or the Lasso penalty that encourages sparsity in
the feature weights (Tibshirani,, 1996). The Lasso and other regularization
approaches are routinely employed across all areas of machine learning to aid
in interpreting features (Li et al. ,, 2022).

####  3.2.2 Feature Interactions & Feature Representations

Beyond the importance of each individual feature, one may want to understand
higher-order interactions or feature patterns that are important for a model’s
predictions. Decision trees and their extensions offer natural ways of
assessing model-specific feature interactions, but there are several model
agnostic approaches such as Friedman’s H-statistic, variable interaction
networks, and partial dependence functions; see Molnar, (2022) for further
details.

Going beyond pairwise feature interactions, many are interested in
understanding how more complex, higher-order, and non-linear feature patterns
contribute to a model’s predictions. This growing area is often called
representation learning and utilizes deep learning models like transformers to
encode complex feature relationships in an often lower-dimensional
representation space (Bengio et al. ,, 2013). While many of these feature
representations are not directly interpretable, there is an active area of
research to learn interpretable feature representations, especially in
computer vision (Bengio et al. ,, 2013).

####  3.2.3 Influential Points

We have discussed interpretations of features in supervised models, but one
can also interpret the observations through influential points, defined as
observations whose removal significantly changes a model’s prediction. There
are a few model-specific approaches that provide intrinsic interpretations of
influential points, like support vector machines, but most use model-agnostic
strategies to identify these points. Coming from classical statistics, one can
use strategies to detect outliers as well as measure the effect of removing
each single training point (Hodge & Austin,, 2004). But more recently in
machine learning, may have proposed using the influence function to
approximate parameter changes for individual points based on the change in the
gradient; these approaches have found widespread application in deep learning
models (Koh & Liang,, 2017).

##  4 Validating IML Discoveries

Interpretable Machine Learning techniques are being deployed across science
and beyond to generate new knowledge or make data-driven-discoveries. Yet one
may ask, is my discovery true? Or, have I discovered an artifact? How can I
tell the difference? In other words, how can we validate discoveries made via
Interpretable Machine Learning? While most research in the Interpretable
Machine Learning community has focused on developing new interpretability
techniques, there has been relatively little work on the critically important
problem of validation. We contend that validation is one of the grand
challenges in interpretable machine learning, and this is especially crucial
for making replicable, reliable, and trustworthy data-driven discoveries. In
this section, we motivate the necessity of validation for IML, discuss why
this is so challenging, and then discuss several practical approaches that can
be deployed with most IML techniques to help validate discoveries; we conclude
with recommendations for validating IML discoveries in practice.

###  4.1 Motivation & Challenges

####  4.1.1 Motivation: Replicability, Reliability & Trust

Interpretable machine learning techniques are designed to always produce the
desired interpretation, regardless of whether that interpretation or discovery
truly reflects the underlying structure of the data. For example, $K$-means
clustering always returns $K$ clusters whether there are groups in the data or
not; feature selection always returns a subset of features whether the
underlying true model is sparse or not. Then, how can we tell if the machine
learning interpretation generated a true discovery or is just an artifact in
the data? Further, there are a plethora of IML techniques and often each
technique produces a different interpretation. Then, which interpretation is
correct and represents a true discovery? These are perhaps unknowable,
epistemological questions. Science addresses this by continually replicating
and validating discoveries in follow-up studies until findings converge upon
an accepted truth. Indeed, reproducibility and replicability are cornerstones
of science (National Academies of Sciences, Engineering, and Medicine and
others,, 2019, Stodden,, 2020).

In machine learning, reproducibility means being able to obtain the exact same
results after the same computational steps are performed on the same data,
which is purely a computational concept (Willis & Stodden,, 2020, Fineberg et
al. ,, 2020) and is a prerequisite for validation. Replicability means being
able to obtain very similar results when two independent studies are performed
to answer the same scientific question; in machine learning, this could entail
performing the same or similar analysis on a new data set (Meng,, 2020,
Fineberg et al. ,, 2020). Replicability by itself, however, can not be the
ultimate goal of scientific discoveries, as replicable results can still be
wrong if the same mistakes are made in follow-up studies. Thus, going one step
beyond this, many have advocated for reliability in machine learning, saying
that predictions and findings should be robust to reasonable sensitivity tests
like small changes in the data or the model, out-of-sample prediction tests,
and consistency with domain knowledge (Meng,, 2020). Validation for machine
learning directly seeks to assess the replicability and reliability of
results. For predictive tasks, there are well-developed and routinely employed
validation strategies like data-splitting and cross-validation. For machine
learning interpretations, however, there are very few widely accepted
validation strategies and most employ IML techniques without any validation
whatsoever in practice. For some uses of IML, this practice might not be
terrible, but for the task of generating new discoveries, lack of validation
is extremely damaging and could lead to erroneous, irreplicable and unreliable
findings. Indeed, there has been much commentary over the past several decades
about a reproducibility and replicability crisis in science (Baker,, 2016).
Recently, several have suggested that failures to validate machine learning
findings could be contributing to this crisis (Beam et al. ,, 2020, McDermott
et al. ,, 2021, Gibney,, 2022). Validation is a crucial component of IML for
generating data-driven discoveries.

Beyond just the goal of utilizing best practices in science, replicability and
reliability are critical to promote trust in machine learning results. Many
have lamented a lack of trust in machine learning systems and recommended to
promote trust and societal acceptance of machine learning results by
generating understandable interpretations (Jacovi et al. ,, 2021, Toreini et
al. ,, 2020). But, can we trust these interpretations? If interpretations are
not replicable and reliable, then trust in these interpretations and
discoveries breaks down. Recently, Broderick et al. , (2023) discussed these
issues, among others, that cause trust to break down in probabilistic machine
learning. Further, they and several others have proposed various ways to
enhance trust in machine learning results, emphasizing the need for validation
strategies (Rasheed et al. ,, 2022, Toreini et al. ,, 2020, Broderick et al.
,, 2023).

####  4.1.2 Challenges

To better understand why validating machine learning interpretations and their
data-driven discoveries is so challenging, let us first discuss why a
discovery might fail to validate. First, the machine learning model could be a
poor fit to the data and hence any resulting interpretations would poorly
reflect the signal in the data. Next, even if the model fits the data well,
the interpretation approach could be a poor fit for the model, resulting in
problematic interpretations; this can especially be the case with some post-
hoc interpretability methods that fit a second model to generate the
interpretation (e.g. LIME) (Molnar,, 2022). In addition, there could be a
mismatch between the employed interpretation technique and the desired
discovery task. For example, one important application of feature selection is
to discover important biomarkers from high dimensional genomics data which is
known to be highly correlated. Many feature selection techniques such as the
Lasso are known to only select one feature out of a correlated set (Zou &
Hastie,, 2005) and hence would fail to identify some important but correlated
biomarkers. Next, IML techniques are typically designed to find the desired
interpretation in the data, regardless of whether that discovery truly exists
in data. For example, $K$-means clustering will always discover $K$ clusters.
Many machine learning techniques are so powerful that they can always detect
the rarest signals in large and complex data sets. For predictive tasks, we
call this overfitting. We also argue that machine learning interpretations can
be overfit to the training data, and are hence challenging to validate.

However, despite the fact that machine learning interpretations might fail to
validate for a number of reasons, its validation is a significant challenge
that has received surprisingly little attention in the literature (Rasheed et
al. ,, 2022). For prediction tasks, on the other hand, we have well-
established techniques for validation: we ensure the predictive model
generalizes to new, similar data. For instance, we can randomly split the
available data into a training set which is used for building the predictive
model, and a test set which is used for assessing the predictive accuracy of
the model. Similar to that of predictive models, we say that a machine
learning interpretation validates if the resulting data-driven discovery
generalizes well to new, similar data. Given this, one may ask: Can we simply
employ a training and test set to validate interpretations? We discuss this
possibility subsequently, but in short, this prospect becomes much more
complicated for interpretations. First, many interpretable machine learning
techniques are designed to make discoveries from the current (training) data
but cannot directly apply the discovery to new data (e.g. clustering, manifold
learning, feature importance ranking), and hence it is unclear how to assess
how well it generalizes. Second, unlike the well-established prediction error
metrics, there is no consensus on metrics for quantifying the accuracy of
interpretations. Different machine learning interpretation approaches can
yield very different discoveries and it is hard to come up with a fair metric
that determines which one is the best on a test set. Most have suggested to
assess machine learning interpretations via human evaluation of laypersons or
domain experts (Carvalho et al. ,, 2019, Molnar,, 2022, Doshi-Velez & Kim,,
2017), but this does not lend itself to an easy-to-quantify metric analog of
prediction accuracy.

###  4.2 Practical Approaches for Validating Interpretations

In this section, we review two practical validation strategies that can be
employed for almost any machine learning discovery. While there may be
additional validation approaches for specific interpretable machine learning
models, we highlight these as they are fairly general and can be applied for
both supervised and unsupervised discoveries.

####  4.2.1 Data-Splitting

As we previously discussed, randomly splitting the available data into a
training and test set is the established mechanism for validating machine
learning predictions. Similar strategies can also help validate machine
learning interpretations, but this also presents several challenges and
limitations. The key idea is to use IML on the training data to generate an
interpretation as well as construct a prediction model based on this learned
interpretation; then one can evaluate the model’s predictive performance on
the test data. For example, with projective dimension reduction techniques
like PCA, one can learn the projection from the training set and then evaluate
how the test data differs from its projection onto these components. Similarly
in clustering, one could discover clusters on the training set, develop a
classification model on the training set to discriminate these clusters, then
apply this to the test set to predict cluster labels; one could then compare
these predicted labels to those generated in an unsupervised manner by
clustering the test set (Lange et al. ,, 2004, Handl et al. ,, 2005). For
supervised discoveries like feature selection, one could discover important
features on the training set as well as build a predictive model that only
uses these features for the associated supervised learning task; then one can
evaluate the prediction error of this model on the test set. The same idea can
also be applied to feature interactions, feature patterns and other supervised
discoveries.

Even though data-splitting provides a direct approach to validating IML
discoveries, there remain many open questions and challenges. Although we
presented several examples of how this strategy can be used, it is unclear how
to define an appropriate prediction task for some other machine learning
interpretations, such as discovered associations and relationships between
features, or anomalies and prototypes. Next, this approach generates
predictions on the test set, but what should these predictions be compared to?
For selecting important features, often the prediction error of a sparse model
is not comparable to one with all the features. Hence, how do we know whether
the prediction error achieved by the smaller model is good enough to consider
the feature subset validated? Similar issues arise with many other IML
approaches. Finally, and arguably the most problematic aspect of data-
splitting, the resulting interpretation is found using only part of the data
and hence the interpretation might change with another randomly sampled
training data. This can be very troubling for replicability and in science.
Related, some would argue that since discovery is such a challenging task, one
needs to use all available data and data-splitting reduces the amount of data
available for the discovery stage.

####  4.2.2 Stability

Another popular strategy for directly assessing the reliability of machine
learning interpretations is the stability principle, which seeks to identify
interpretations that are stable subject to random data perturbations. This
idea was first introduced in Meinshausen & Bühlmann, (2010) in the context of
feature selection with the Lasso. They randomly subsampled the data, fit a
Lasso model to each subsample to select features, and the features that are
selected most frequently over all subsamples are viewed as stable discoveries;
importantly, they show that under certain assumptions this procedure controls
the expected number of falsely selected features (Meinshausen & Bühlmann,,
2010). There have also been many variants of this method studied in the
statistical machine learning literature for feature selection (Shah &
Samworth,, 2013), feature interactions (Basu et al. ,, 2018), graphical models
(Liu et al. ,, 2010), PCA (Taeb et al. ,, 2020), and clustering where it is
commonly called consensus clustering Monti et al. , (2003). Even though it has
not been widely applied in other areas of machine learning, the idea of
stability analysis is rather general and could be applied to any IML
procedure. We summarize the approach in Figure 3. First, the data is
repeatedly randomly perturbed through subsampling, bootstrapping, randomly
adding noise, or random data thinning (Neufeld et al. ,, 2023). Then, IML
procedures are used to make a discovery on each new random data set and
discoveries with high frequency are declared as stable discoveries. The core
idea of stability is that discoveries that are not consistent under random
data perturbations are more likely to be due to artifacts in the data or
sampling noise, and hence are not replicable and reliable. Indeed, stability
analysis has received widespread attention, and many have advocated using this
to validate discoveries and promote reproducibility in (data) science (Yu &
Kumbier,, 2020). It has also been widely used for solving many scientific
problems such as discovering biomarkers in genomics (He & Yu,, 2010).

Despite the appeal of stability analysis to directly assesses reliability and
overfitting, several challenges remain. First, stability analysis can be
computationally burdensome as it requires refitting the IML model many times;
this is especially problematic for huge data or with complex models like deep
learning. Next, it is not always clear what type of random perturbation is
appropriate and what quantitative criterion should by employed to determine
stable discoveries for a given IML model and discovery task. Consider
consensus clustering where co-cluster membership is recorded for each
subsample as there is not an easy way to record and ensemble the cluster
membership. Further, stability analysis could exacerbate mismatches between
the interpretation techniques and the discovery task. Stability with the Lasso
for feature selection, for example, is known to perform very poorly with
correlated features. This is due to the fact that the Lasso may only select
one among highly correlated features for each subsample, and hence none of
these features would be deemed as stable, even if they are all important.
Additionally, it is unclear whether the final stable discoveries are
consistent with each other since they might not correspond to a single
interpretation or set of interpretations from applying an IML model (e.g.
stable features may not correspond to any Lasso solution at a single
regularization parameter); many might consider this a disadvantage in
scientific domains. Finally, and perhaps most importantly, stability analysis
only assesses one form of reliability: the robustness of the discovery to
small changes in the data, but not the robustness to changes in the modeling
choice. Furthermore and as we discussed earlier, reliability also means
consistency with prior knowledge and out-of-sample predictive power, which are
not reflected in stability analysis (Yu & Kumbier,, 2020). To see how this
could be problematic, consider a scenario where a linear model is a poor fit
for the data; then stability selection with the Lasso might result in stable
features, but because of high model bias, these features will not be
predictive or reflective of true important features. Because of this, two
different IML methods might lead to completely different discoveries that are
each stable, which calls for further validation strategies. To summarize,
stability is necessary for indicating the reliability of a discovery, but it
is not sufficient.

Figure 3: Illustration of stability principle for determining reliable data-
driven discoveries.

####  4.2.3 Example: Validating Clusters

To illustrate data-splitting and stability analysis for validating IML, we
turn to two real clustering examples: the Author data set with $n=841$
observations and $p=69$ features measuring the stop word count of book
chapters from four English-language authors (Blake,, 1998), and the TCGA
PANCAN data set with $n=761$ subjects and $p=13,244$ genes measuring the bulk
RNA-seq gene expression for patients with five different types of tumors
(Weinstein et al. ,, 2013). We apply $K$-means clustering with $K=4$ and $K=5$
respectively, and seek to validate our discovered clusters using data-
splitting and stability. For data-splitting, we follow the predictive cluster
validation approach outlined in (Lange et al. ,, 2004, Handl et al. ,, 2005)
by randomly taking 70% of observations as a training set where we discover
clusters as well as build a random forest classifier to predict these cluster
labels. We then independently cluster the remaining 30% of observations in the
test set as well as apply the random forest classifier to predict the labels;
we measure the overlap between the predicted labels and the test set cluster
labels using the adjusted rand index (ARI), a metric between zero and one with
higher values indicating better cluster membership overlap. Results are shown
in the top panel of Figure 4 where we visualize the training and test set
clusters in principal component (PC) scatterplots and highlight the test set
observations where there is a mismatch between the predicted and cluster
labels as larger points. In the Author data, the training set predictions and
test set cluster labels have a high degree of overlap, indicating strong
validation of these four clusters. Clusters in the PANCAN data set do not
validate as well as shown by the lower ARI and the confusion of the cluster
labels for the blue and teal clusters.

Additionally, we apply the stability principle, which in clustering is often
called consensus clustering (Monti et al. ,, 2003), to validate these same
cluster findings. Specifically, we employ repeated data-splitting by
repeatedly subsampling a training set on which we discover clusters and record
the cluster co-membership; we then average these cluster co-memberships across
all data splits to yield the $n\times n$ consensus matrix taking values
between zero and one, with one indicating that the two observations were
always assigned to the same cluster. Heatmaps of the consensus matrix are
shown in the bottom left panels of Figure 4 with darker green indicating
values at or near one. We see that in both the Author and PANCAN data set,
consensus clustering validates that there are $K=4$ and $K=5$ clusters
respectively, exhibiting a clear block diagonal pattern. Consensus clustering
additionally allows us to inspect the uncertainty of cluster assignments for
individual observations. In the PC scatterplots on the bottom right of Figure
4, we show observations with point sizes inversely proportional to their
cluster assignment uncertainty. From this, we see that clusters in the Author
data set are fairly well separated, but again, the teal and the blue clusters
in the PANCAN data set exhibit a high degree of confusion. Overall, both data-
splitting and the stability principle can be used to validate cluster
discoveries, and both of these methods reveal similar findings in the two
examples we present.

Figure 4: Example of how to validate cluster discoveries using data-splitting
(top row) and the stability principle (bottom row) on two data sets. Results
from the Author data set (a) show four well-validated clusters whereas results
from the TCGA PANCAN data set (b) show that the blue and teal clusters are not
well-separated and do not validate as well.

####  4.2.4 Practical Recommendations

We have discussed two general, practical validation strategies, but each of
these has its own strength and limitations. Data-splitting can be a useful
strategy for checking whether the discovered interpretation fits the data
well, while stability analysis is most effective for evaluating whether the
discovery is induced by random noise. Here, we would like to recommend one
potential practical solution that leverages strengths of both data-splitting
and stability analysis to evaluate the appropriateness of the chosen IML model
(bias) and whether the ML technique overfits the noise (variance) in the IML
discovery respectively. Specifically, one could employ repeated random data-
splitting. On the training sets, one could apply the IML technique and record
the resulting discovery as well as build a predictive model to predict that
discovery. On the test sets, one could evaluate the predictions and record the
prediction error. Then, one could average both the discovery events and
prediction errors across all randomized data sets. This is essentially
repeated data splitting plus stability analysis, but importantly, allows one
to assess both bias and variance associated with the IML procedure and
discovery simultaneously. While we have not seen this technique explicitly
outlined or employed, we have found this to be fruitful in our own work and
thus recommend it as a possible validation strategy. Overall, validation is
critically important for IML and especially for generating data-driven
discoveries. But the practical validation strategies we discussed are often
constrained for certain types of discoveries; they often lack rigorous
theoretical guarantees or quantitative guidelines on how to employ the
approaches and which evaluation criterion to apply. Hence, there are many
opportunities for further research to develop and apply these practical
validation strategies, study their theoretical properties, and determine the
best validation approaches for many IML models and discovery types.

##  5 Statistical Theory & Inference for IML Discoveries

Validation of IML discoveries is critical in practice to promote
replicability, reliability and trust in data-driven discoveries. But,
theoretical guarantees and valid statistical inference offer a different
perspective related to validation and are equally necessary to help build
trust and promote replicability of IML discoveries (Rasheed et al. ,, 2022,
Broderick et al. ,, 2023). In this section, we review statistical theoretical
foundations for IML discoveries that address two pressing questions: (1) Under
what data-generating models and under what conditions does an IML technique
recover the true discovery with high probability? and (2) What is the
uncertainty in a discovery or what discoveries can be trusted with a
sufficient level of confidence? We discuss these in Sections 5.1 and 5.2
respectively.

###  5.1 Statistical Theory for Interpretable Machine Learning

Recently, Broderick et al. , (2023) argued that theoretical guarantees were
important to help build trust in machine learning. For IML discoveries, the
goal is to theoretically characterize the type of data-generating models and
the conditions under which IML techniques will make the desired discovery with
high probability tending to one. These types of theoretical guarantees largely
fall under the areas of statistical consistency and selection consistency; the
latter has received a huge amount of attention in the statistical machine
learning community over the past two decades (Wainwright,, 2019). Developing
such theoretical foundations can help guide practitioners to choose the
appropriate technique for their application and desired discovery task,
understand when certain IML techniques will perform well and when they will
not, and perhaps inspire the development of new IML techniques with improved
performance and theoretical guarantees.

Statistical consistency and selection consistency are well-studied for certain
types of statistical and machine learning models, but are perhaps not readily
applicable to other classes of machine learning methods leaving a gap in our
theoretical understanding of IML. For example, classical statistical theory
addresses the conditions under which parametric models like linear and
generalized linear models consistently estimate their coefficients, a measure
of intrinsic feature importance, in asymptotic and low-dimensional settings.
More recently, there has been a surge of interest in studying regularized
versions of these and other semi-parametric statistical machine learning
models in finite-sample and high-dimensional settings (Bühlmann & Van De
Geer,, 2011, Wainwright,, 2019). Perhaps the most widely studied has been the
Lasso, or $\ell_{1}$-regularized regression, for the IML task of feature
selection (Tibshirani,, 1996). For example, it is well established that the
Lasso achieves selection consistency, or correct selection of true features
with high-probability, under sparse linear regression models, when there is
sufficient sample size relative to the log number of features, when there is
sufficient signal in the true features, and under conditions like the
irrepresentable, restricted eignenvalue, or incoherence conditions that limit
the amount of correlation between features in the model Zhao & Yu, (2006).
Consistency and selection consistency has also been established for many
extensions of the Lasso, the Lasso in classification, semi-parametric models
and other settings, and for other sparse regularizers (Bühlmann & Van De
Geer,, 2011). Beyond feature selection and feature importance, several other
model-specific and intrinsically interpretable unsupervised IML techniques
have been studied theoretically under high-dimensional regimes. These include
statistical consistency guarantees for clustering under a Gaussian mixture
model (Löffler et al. ,, 2021), network clustering under the stochastic block
model (Abbe,, 2017), low-rank estimation via PCA under spiked covariance
models (Johnstone & Lu,, 2009), and graph selection or structural graph
learning for both Markov Networks (undirected graphs) and Directed Acyclic
Graphs for causal discovery (Drton & Maathuis,, 2017); we refer the reader to
Wainwright, (2019) for more details on many of these recent advances in high-
dimensional statistical theory.

These advances in statistical theory provide assurance and insights for
certain IML discoveries and certain techniques, but there remain are many
limitations of this type of theory as well as open questions. First, this
statistical theory assumes a true population model that generates the data. In
practice, the true data generating process is unknown and uncheckable; it is
often unclear how these IML techniques perform with mis-specified models.
Second, this type of theory is only applicable to model-specific and
intrinsically interpretable IML techniques, which are often limited to linear
or additive parametric or semi-parametric models. Thus, this theory does not
help us understand the performance of more flexible, non-linear modeling
strategies like tree-based ensembles and deep learning. Next, even when this
type of theory is applicable to a particular model, techniques, and discovery
task, the assumptions required to make the correct discovery with high
probability are often hard to interpret and impossible to check in practice.
For example, it is impossible to check the irrepresentable condition (Zhao &
Yu,, 2006) necessary for selection consistency of the Lasso for a particular
data set without knowing the true features. Thus while this theory helps us
understand the properties of certain IML techniques, this is unhelpful for
trying to assess the validity of a particular discovery made by an IML
technique on a particular data set. Finally, statistical theory is currently
very limited for interpretations of tree-based ensembles like random forests
and boosting, neural networks and deep learning, and model-agnostic
interpretations like Shapley values for feature importance. Such areas provide
many open research opportunities that would help us better understand these
popular IML approaches and further promote trust in their discoveries
(Broderick et al. ,, 2023).

###  5.2 Statistical Inference for Interpretable Machine Learning

While statistical theory highlights the assumptions required to make an
accurate discovery with high probability, another approach to validate
discoveries is through statistical inference which quantifies the uncertainty
associated with the discovery. Uncertainty quantification, typically through
confidence intervals and hypothesis testing, is crucial in discerning whether
a discovered pattern is due to random chance or is a genuine discovery. This
is especially important in high-stakes applications of IML where making
decisions based on discoveries with a high degree of uncertainty could have
devastating consequences; in science, this could lead to wasted resources and
irreplicable results. While uncertainty quantification for IML is a critically
important task, this presents many challenges. Note that the statistical
theory discussed previously, in Section 5.1, also quantifies errors for a
discovery, but these cannot readily be used for uncertainty quantification as
they depend on unknown parameters. Similarly, practical validation approaches
like data-splitting and stability, discussed in Section 4, give a sense of the
uncertainty in a discovery but cannot always be translated into rigorous
statistical uncertainty quantification. Nonetheless, uncertainty
quantification has been studied for many of the same statistical machine
learning models and IML tasks for which statistical theory has been developed;
and more recently, uncertainty quantification has been considered in model-
agnostic settings for specific IML tasks like feature importance and feature
selection. We briefly review these approaches.

Most statistical inference procedures are designed for model specific, global,
and intrinsically interpretable statistical models that cover only a narrow
range of IML techniques. Classical inference approaches, which are typically
asymptotic in nature, can be used for linear, generalized linear, or additive
parametric (sometimes semi-parametric) statistical models to quantify the
uncertainty in parameters. Non-parametric or semi-parametric methods like
bootstrap uncertainty quantification can also be used for many of the same
methods. More recently, several have developed inferential procedures for
regularization techniques like the Lasso in high-dimensional regimes. Such
approaches include debiasing techniques (van de Geer et al. ,, 2014), which
calculate the high-dimensional asymptotic distribution of the Lasso, and
selective inference (Taylor & Tibshirani,, 2015), which computes confidence
intervals and tests conditional on the Lasso solution. Several others have
recently employed similar strategies to quantify the uncertainty for
unsupervised statistical learning tasks like clustering (Gao et al. ,, 2022),
graphical models (Liu,, 2013), and principal components analysis (Koltchinskii
& Lounici,, 2016). This recent research on statistical inference for popular
statistical machine learning models in high-dimensional regimes represents
important advances in the field, but the approaches also have several
limitations. All of these approaches are model-specific and limited to
parametric (or perhaps semi-parametric) statistical models, hence precluding
application to popular non-linear machine learning models like kernels, tSNE,
tree-based ensembles, and deep learning. Further, these approaches assume the
data arises from a specific generating model, which is not checkable in
practice. They are less effective at quantifying the uncertainty in IML
discoveries when the model is mis-specified.

Given the limitations of model-specific approaches and following from recent
developments in distribution-free predictive inference, many have advocated
for model agnostic inference which can quantify the uncertainty associated
with any IML model. Thus far, such approaches have only been developed for
feature importance and feature selection. Among the first such approaches were
based on the model-X knock-off framework which generates knock-off features
with no relation to the response but that still retain the dependencies
structure amongst the features (Candès et al. ,, 2018, Barber & Candès,,
2019). This approach has been used to select features with false discovery
rate control (Barber & Candès,, 2019), conduct conditional independence
testing (Berrett et al. ,, 2018), and construct confidence intervals for
feature importance (Zhang & Janson,, 2020), among others. The fact that knock-
off approaches can be employed for any IML model is a major advantage, but
this comes at the expense of assuming that the distribution of the features is
known or can be closely approximated, a significant limitation in many
domains. Others have recently developed model agnostic inference approaches
for feature importance; some consider feature occlusion inference (Lei et al.
,, 2018, Williamson et al. ,, 2022, Gan et al. ,, 2022) which examines the
prediction loss when removing one feature, while others consider the feature
permutation test (Berrett et al. ,, 2018, Kim et al. ,, 2021) which randomly
permutes the feature of interest. While very general and widely applicable,
these approaches either perform inference for a random quantity that depends
on the training set or require limiting assumptions on the data distribution
or the consistency of the model employed. In fact, the fundamental difficulty
of distribution-free and model-agnostic feature importance inference has been
recently revealed by Shah & Peters, (2020), who shows that any conditional
independence test that is valid without further assumptions on the data
distribution or the model has no statistical power. Hence while model-agnostic
inference and uncertainty quantification for IML is critically important for
validating many popular IML models, further research is needed to understand
and work around limiting distributional and modeling assumptions. Finally,
there are many research opportunities to develop model agnostic inference
approaches for IML tasks beyond feature selection and importance.

While this review has not focused on Bayesian machine learning, it is
important to mention these techniques in the context of uncertainty
quantification. Indeed, many argue that one of the more appealing aspects of
Bayesian approaches is the built-in uncertainty quantification through
computing the posterior distribution and credible intervals; such approaches
have been developed for IML tasks like feature importance and selection,
graphical models, factor models, clustering, and more (Cortes et al. ,, 2017,
Vallejos et al. ,, 2015). Despite appealing uncertainty quantification
properties, there are several challenges when applying these techniques to
generate and validate IML discoveries. First, computing or sampling from the
exact posterior distribution is typically intractable or computationally
prohibitive in big data settings. Thus, people typically employ approximation
techniques like variational inference (Blei et al. ,, 2017), but there is
limited theory on how well these approaches work and how they affect the
uncertainty quantification of IML discoveries. Further, the IML discovery,
posterior distribution, and any uncertainty quantification depend strongly on
the prior employed. The posterior distribution does not reflect this
sensitivity to the prior and hence can underestimate the true uncertainty in
the IML discovery; further sensitivity tests and model checking are needed for
validation (Gelman & Shalizi,, 2013, Kruschke,, 2021). We refer the reader to
(van de Schoot et al. ,, 2021) for more details.

In summary, statistical inference for IML discoveries is critical for
validation and a growing area of research. There are a number of important
recent results in this field, especially for model agnostic inference, but
there are also many open questions and challenges that are ripe areas for
further research.

##  6 Discussion

In this paper, we provided an overview of IML techniques that can be used for
data-driven-discovery and discussed associated challenges and opportunities
like validation, statistical theory, and inference. But importantly, there are
many aspects that we did not cover in this review that warrant further
coverage and discussion in other works. This paper focused on fairly general
machine learning tasks and techniques, but there are an abundance of
techniques developed for specific areas and tasks like those in computer
vision, natural language processing and large language models, and
reinforcement learning Glanois et al. , (2021), among several others. Many of
these IML techniques can be used for discoveries and also share similar
validation challenges. Another important area that we only briefly covered but
deserves its own careful consideration is causality, which includes
interpretability via counterfactual explanations Mothilal et al. , (2020),
causal inference from interventional studies, and causal discovery from
observational data. The latter can be especially important in science for
discovering causal mechanisms, but perhaps faces even more challenges when it
comes to validation, theory, and uncertainty quantification. We also only
briefly discussed Bayesian machine learning and its associated uncertainty
quantification, but this growing area of research deserves further discussion
in the context of IML for generating new discoveries.

In this paper, we reviewed and discussed the grand challenge of how to
validate discoveries made using IML. We specifically discussed three aspects
of this grand challenge: (1) practical tools for validating interpretations,
(2) theoretical foundations of major IML techniques, and (3) uncertainty
quantification for machine learning interpretations. We presented two major
types of practical validation strategies, data-splitting and stability, but
each of these has their own caveats and limitations. Further research is
needed to combine the strengths of both approaches, elucidate a theoretical
basis for these approaches, or perhaps develop a connection with uncertainty
quantification via inference. Next, we have a strong theoretical understanding
of only a limited number of IML techniques, mainly those that are intrinsic,
global, and model-specific. This does not include interpretations of popular
machine learning methods like random forests and deep learning; further
research is needed to not only explain the strong predictive performance of
these approaches but also understand their interpretations and discoveries.
Finally, there has been growing interest in uncertainty quantification for
prediction, but quantifying the uncertainty of machine learning
interpretations is also another critical component of validation that deserves
further attention and research. In addition to challenges associated with
validation, there are also several other important questions that require
further consideration and research. Some of these include how to match the
appropriate IML technique to the desired discovery task, how to compare
different interpretations from different IML techniques, and how to marry
domain knowledge and expertise with IML to better develop, deploy, and
evaluate IML discoveries.

In summary, IML techniques hold great promise for making breakthroughs in
science and beyond by mining ever larger data sets to detect the rarest
signals. But at the same time, these IML discoveries should be interpreted
with caution without careful validation or uncertainty quantification. Solving
this grand challenge is critical for promoting replicable and reliable (data)
science as well as trustworthy machine learning; they also provide exciting
research opportunities at the intersection of statistics and machine learning.

## References

  * Abbe, (2017) Abbe, Emmanuel. 2017.  Community detection and stochastic block models: recent developments.  The journal of machine learning research, 18(1), 6446–6531.
  * Baker, (2016) Baker, Monya. 2016.  Reproducibility crisis.  Nature, 533(26), 353–66.
  * Barber & Candès, (2019) Barber, Rina Foygel, & Candès, Emmanuel J. 2019\.  A knockoff filter for high-dimensional selective inference.  The annals of statistics, 47(5), 2504 – 2537.
  * Barredo Arrieta et al. , (2020) Barredo Arrieta, Alejandro, Díaz-Rodríguez, Natalia, Del Ser, Javier, Bennetot, Adrien, Tabik, Siham, Barbado, Alberto, Garcia, Salvador, Gil-Lopez, Sergio, Molina, Daniel, Benjamins, Richard, Chatila, Raja, & Herrera, Francisco. 2020.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information fusion, 58, 82–115.
  * Basu et al. , (2018) Basu, Sumanta, Kumbier, Karl, Brown, James B., & Yu, Bin. 2018.  Iterative random forests to discover predictive and stable high-order interactions.  Proceedings of the national academy of sciences, 115(8), 1943–1948.
  * Beam et al. , (2020) Beam, Andrew L, Manrai, Arjun K, & Ghassemi, Marzyeh. 2020.  Challenges to the reproducibility of machine learning models in health care.  Jama, 323(4), 305–306.
  * Bengio et al. , (2013) Bengio, Yoshua, Courville, Aaron, & Vincent, Pascal. 2013.  Representation learning: A review and new perspectives.  Ieee transactions on pattern analysis and machine intelligence, 35(8), 1798–1828.
  * Berkhin, (2006) Berkhin, P. 2006\.  A survey of clustering data mining techniques.  Berlin, Heidelberg: Springer Berlin Heidelberg.  Pages 25–71.
  * Berrett et al. , (2018) Berrett, Thomas B., Wang, Yi, Barber, Rina Foygel, & Samworth, Richard J. 2018\.  The conditional permutation test for independence while controlling for confounders.  Journal of the royal statistical society: Series b (statistical methodology), 82.
  * Bien & Tibshirani, (2011) Bien, Jacob, & Tibshirani, Robert. 2011.  Prototype selection for interpretable classification.  The annals of applied statistics, 5(4), 2403 – 2424.
  * Blake, (1998) Blake, Catherine. 1998.  Uci repository of machine learning databases.  http://www. ics. uci. edu/~ mlearn/mlrepository. html.
  * Blei et al. , (2017) Blei, David M., Kucukelbir, Alp, & McAuliffe, Jon D. 2017\.  Variational inference: A review for statisticians.  Journal of the american statistical association, 112(518), 859–877.
  * Borjali et al. , (2020) Borjali, Alireza, Chen, Antonia F., Muratoglu, Orhun K., Morid, Mohammad A., & Varadarajan, Kartik M. 2020\.  Deep learning in orthopedics: How do we build trust in the machine?  Healthcare transformation, 0(0), null.
  * Broderick et al. , (2023) Broderick, Tamara, Gelman, Andrew, Meager, Rachael, Smith, Anna L, & Zheng, Tian. 2023.  Toward a taxonomy of trust for probabilistic machine learning.  Science advances, 9(7), eabn3999.
  * Brunton et al. , (2016) Brunton, Steven L., Proctor, Joshua L., & Kutz, J. Nathan. 2016.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems.  Proceedings of the national academy of sciences, 113(15), 3932–3937.
  * Bühlmann & Van De Geer, (2011) Bühlmann, Peter, & Van De Geer, Sara. 2011.  Statistics for high-dimensional data: methods, theory and applications.  Springer Science & Business Media.
  * Candès et al. , (2018) Candès, Emmanuel, Fan, Yingying, Janson, Lucas, & Lv, Jinchi. 2018.  Panning for Gold: ‘Model-X’ Knockoffs for High Dimensional Controlled Variable Selection.  Journal of the royal statistical society series b: Statistical methodology, 80(3), 551–577.
  * Carvalho et al. , (2019) Carvalho, Diogo V., Pereira, Eduardo M., & Cardoso, Jaime S. 2019\.  Machine learning interpretability: A survey on methods and metrics.  Electronics, 8(8).
  * Cortes et al. , (2017) Cortes, Adrian, Dendrou, Calliope A, Motyer, Allan, Jostins, Luke, Vukcevic, Damjan, Dilthey, Alexander, Donnelly, Peter, Leslie, Stephen, Fugger, Lars, & McVean, Gil. 2017.  Bayesian analysis of genetic association across tree-structured routine healthcare data in the uk biobank.  Nature genetics, 49(9), 1311–1318.
  * Dong et al. , (2017) Dong, Yinpeng, Su, Hang, Zhu, Jun, & Bao, Fan. 2017.  Towards interpretable deep neural networks by leveraging adversarial examples.  arxiv preprint arxiv:1708.05493.
  * Doshi-Velez & Kim, (2017) Doshi-Velez, Finale, & Kim, Been. 2017.  Towards a rigorous science of interpretable machine learning.  arxiv preprint arxiv:1702.08608.
  * Drton & Maathuis, (2017) Drton, Mathias, & Maathuis, Marloes H. 2017.  Structure learning in graphical modeling.  Annual review of statistics and its application, 4, 365–393.
  * Du et al. , (2019) Du, Mengnan, Liu, Ninghao, & Hu, Xia. 2019.  Techniques for interpretable machine learning.  Commun. acm, 63(1), 68–77.
  * Fineberg et al. , (2020) Fineberg, Harvey, Stodden, Victoria, & Meng, Xiao-Li. 2020.  Highlights of the US National Academies Report on “Reproducibility and Replicability in Science”.  Harvard data science review, 2(4).  https://hdsr.mitpress.mit.edu/pub/6an6ppum.
  * Fodor, (2002) Fodor, I K. 2002.  A survey of dimension reduction techniques.  5\.
  * Gan et al. , (2022) Gan, Luqin, Zheng, Lili, & Allen, Genevera I. 2022.  Model-agnostic confidence intervals for feature importance: A fast and powerful approach using minipatch ensembles.  arxiv preprint arxiv:2206.02088.
  * Gao et al. , (2022) Gao, Lucy L., Bien, Jacob, & Witten, Daniela. 2022.  Selective inference for hierarchical clustering.  Journal of the american statistical association, 0(0), 1–11.
  * Gelman & Shalizi, (2013) Gelman, Andrew, & Shalizi, Cosma Rohilla. 2013.  Philosophy and the practice of bayesian statistics.  British journal of mathematical & statistical psychology, 66(1), 8–38.
  * Gibney, (2022) Gibney, Elizabeth. 2022.  Is ai fuelling a reproducibility crisis in science.  Nature, 608, 250–251.
  * Gilpin et al. , (2018) Gilpin, Leilani H., Bau, David, Yuan, Ben Z., Bajwa, Ayesha, Specter, Michael A., & Kagal, Lalana. 2018.  Explaining explanations: An overview of interpretability of machine learning.  2018 ieee 5th international conference on data science and advanced analytics (dsaa), 80–89.
  * Glanois et al. , (2021) Glanois, Claire, Weng, Paul, Zimmer, Matthieu, Li, Dong, Yang, Tianpei, Hao, Jianye, & Liu, Wulong. 2021.  A survey on interpretable reinforcement learning.  arxiv preprint arxiv:2112.13112.
  * Guidotti et al. , (2018) Guidotti, Riccardo, Monreale, Anna, Ruggieri, Salvatore, Turini, Franco, Giannotti, Fosca, & Pedreschi, Dino. 2018.  A survey of methods for explaining black box models.  Acm comput. surv., 51(5).
  * Guyon et al. , (2002) Guyon, Isabelle, Weston, Jason, Barnhill, Stephen, & Vapnik, Vladimir. 2002\.  Gene selection for cancer classification using support vector machines.  Machine learning, 46, 389–422.
  * Handl et al. , (2005) Handl, Julia, Knowles, Joshua, & Kell, Douglas B. 2005.  Computational cluster validation in post-genomic data analysis.  Bioinformatics, 21(15), 3201–3212.
  * Hassan et al. , (2022) Hassan, Mubashir, Awan, Faryal Mehwish, Naz, Anam, deAndrés Galiana, Enrique J, Alvarez, Oscar, Cernea, Ana, Fernández-Brillet, Lucas, Fernández-Martínez, Juan Luis, & Kloczkowski, Andrzej. 2022.  Innovations in genomics and big data analytics for personalized medicine and health care: A review.  International journal of molecular sciences, 23(9), 4645.
  * He & Yu, (2010) He, Zengyou, & Yu, Weichuan. 2010.  Stable feature selection for biomarker discovery.  Computational biology and chemistry, 34(4), 215–225.
  * Hennig et al. , (2015) Hennig, Christian, Meilă, Marina, Murtagh, Fionn, & Rocci, Roberto. 2015\.  Handbook of cluster analysis.
  * Hodge & Austin, (2004) Hodge, Victoria, & Austin, Jim. 2004.  A survey of outlier detection methodologies.  Artificial intelligence review, 22(2), 85–126.
  * Houghton & Wilson, (2020) Houghton, Isabel A., & Wilson, James D. 2020\.  El niño detection via unsupervised clustering of argo temperature profiles.  Journal of geophysical research: Oceans, 125(9), e2019JC015947.
  * Jacovi et al. , (2021) Jacovi, Alon, Marasović, Ana, Miller, Tim, & Goldberg, Yoav. 2021.  Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai.  Pages 624–635 of: Proceedings of the 2021 acm conference on fairness, accountability, and transparency.
  * Johnstone & Lu, (2009) Johnstone, Iain M., & Lu, Arthur Yu. 2009.  On consistency and sparsity for principal components analysis in high dimensions.  Journal of the american statistical association, 104(486), 682–693.  PMID: 20617121.
  * Kim et al. , (2021) Kim, Ilmun, Neykov, Matey, Balakrishnan, Sivaraman, & Wasserman, Larry. 2021\.  Local permutation tests for conditional independence.  arxiv preprint arxiv:2112.11666.
  * Koh & Liang, (2017) Koh, Pang Wei, & Liang, Percy. 2017.  Understanding black-box predictions via influence functions.  Page 1885–1894 of: Proceedings of the 34th international conference on machine learning - volume 70.  ICML’17.  JMLR.org.
  * Koltchinskii & Lounici, (2016) Koltchinskii, Vladimir, & Lounici, Karim. 2016.  Asymptotics and concentration bounds for bilinear forms of spectral projectors of sample covariance.  Annales de l’institut henri poincaré, probabilités et statistiques, 52(4), 1976 – 2013.
  * Kruschke, (2021) Kruschke, John K. 2021.  Bayesian analysis reporting guidelines.  Nature human behaviour, 5(10), 1282–1291.
  * Lange et al. , (2004) Lange, Tilman, Roth, Volker, Braun, Mikio L, & Buhmann, Joachim M. 2004.  Stability-based validation of clustering solutions.  Neural computation, 16(6), 1299–1323.
  * Lauritzen, (1996) Lauritzen, Steffen L. 1996.  Graphical models.  Vol. 17.  Clarendon Press.
  * Lei et al. , (2018) Lei, Jing, G’Sell, Max, Rinaldo, Alessandro, Tibshirani, Ryan J., & Wasserman, Larry. 2018.  Distribution-free predictive inference for regression.  Journal of the american statistical association, 113(523), 1094–1111.
  * Li et al. , (2022) Li, Xiaoping, Wang, Yadi, & Ruiz, Rubén. 2022.  A survey on sparse learning models for feature selection.  Ieee transactions on cybernetics, 52(3), 1642–1660.
  * Lipton, (2018) Lipton, Zachary C. 2018\.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  Queue, 16(3), 31–57.
  * Liu et al. , (2010) Liu, Han, Roeder, Kathryn, & Wasserman, Larry. 2010.  Stability approach to regularization selection (stars) for high dimensional graphical models.  Advances in neural information processing systems, 23.
  * Liu, (2013) Liu, Weidong. 2013.  Gaussian graphical model estimation with false discovery rate control.  The annals of statistics, 41(6), 2948 – 2978.
  * Löffler et al. , (2021) Löffler, Matthias, Zhang, Anderson Y., & Zhou, Harrison H. 2021\.  Optimality of spectral clustering in the Gaussian mixture model.  The annals of statistics, 49(5), 2506 – 2530.
  * Lundberg & Lee, (2017) Lundberg, Scott M, & Lee, Su-In. 2017.  A unified approach to interpreting model predictions.  In: Guyon, I., Luxburg, U. Von, Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., & Garnett, R. (eds), Advances in neural information processing systems, vol. 30.  Curran Associates, Inc.
  * Materne, (1978) Materne, J. 1978.  The structure of nearby clusters of galaxies-hierarchical clustering and an application to the leo region.  Astronomy and astrophysics, 63, 401–409.
  * McDermott et al. , (2021) McDermott, Matthew BA, Wang, Shirly, Marinsek, Nikki, Ranganath, Rajesh, Foschini, Luca, & Ghassemi, Marzyeh. 2021.  Reproducibility in machine learning for health research: Still a ways to go.  Science translational medicine, 13(586), eabb1655.
  * Meinshausen & Bühlmann, (2010) Meinshausen, Nicolai, & Bühlmann, Peter. 2010.  Stability selection.  Journal of the royal statistical society: Series b (statistical methodology), 72(4), 417–473.
  * Meng, (2020) Meng, Xiao-Li. 2020.  Reproducibility, replicability, and reliability.  Harvard data science review, 2(4).
  * Molnar, (2022) Molnar, Christoph. 2022.  Interpretable machine learning. 2 edn.
  * Monti et al. , (2003) Monti, Stefano, Tamayo, Pablo, Mesirov, Jill, & Golub, Todd. 2003.  Consensus clustering: a resampling-based method for class discovery and visualization of gene expression microarray data.  Machine learning, 52(1-2), 91–118.
  * Mothilal et al. , (2020) Mothilal, Ramaravind K, Sharma, Amit, & Tan, Chenhao. 2020.  Explaining machine learning classifiers through diverse counterfactual explanations.  Pages 607–617 of: Proceedings of the 2020 conference on fairness, accountability, and transparency.
  * Murdoch et al. , (2019) Murdoch, W. James, Singh, Chandan, Kumbier, Karl, Abbasi-Asl, Reza, & Yu, Bin. 2019.  Definitions, methods, and applications in interpretable machine learning.  Proceedings of the national academy of sciences, 116(44), 22071–22080.
  * National Academies of Sciences, Engineering, and Medicine and others, (2019) National Academies of Sciences, Engineering, and Medicine and others. 2019\.  Reproducibility and replicability in science.  National Academies Press.
  * Neufeld et al. , (2023) Neufeld, Anna, Dharamshi, Ameer, Gao, Lucy L, & Witten, Daniela. 2023.  Data thinning for convolution-closed distributions.  arxiv preprint arxiv:2301.07276.
  * Ozer et al. , (2016) Ozer, Mert, Kim, Nyunsu, & Davulcu, Hasan. 2016.  Community detection in political twitter networks using nonnegative matrix factorization methods.  Pages 81–88 of: 2016 ieee/acm international conference on advances in social networks analysis and mining (asonam).
  * Perou et al. , (2000) Perou, Charles M, Sørlie, Therese, Eisen, Michael B, Van De Rijn, Matt, Jeffrey, Stefanie S, Rees, Christian A, Pollack, Jonathan R, Ross, Douglas T, Johnsen, Hilde, Akslen, Lars A, et al. . 2000.  Molecular portraits of human breast tumours.  nature, 406(6797), 747–752.
  * Rasheed et al. , (2022) Rasheed, Khansa, Qayyum, Adnan, Ghaly, Mohammed, Al-Fuqaha, Ala, Razi, Adeel, & Qadir, Junaid. 2022.  Explainable, trustworthy, and ethical machine learning for healthcare: A survey.  Computers in biology and medicine, 149, 106043.
  * Reyes et al. , (2020) Reyes, Mauricio, Meier, Raphael, Pereira, Sérgio, Silva, Carlos A, Dahlweid, Fried-Michael, Tengg-Kobligk, Hendrik von, Summers, Ronald M, & Wiest, Roland. 2020.  On the interpretability of artificial intelligence in radiology: challenges and opportunities.  Radiology: artificial intelligence, 2(3), e190043.
  * Ribeiro et al. , (2016) Ribeiro, Marco Tulio, Singh, Sameer, & Guestrin, Carlos. 2016.  ”why should i trust you?”: Explaining the predictions of any classifier.  Page 1135–1144 of: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining.  KDD ’16.  New York, NY, USA: Association for Computing Machinery.
  * Roscher et al. , (2020) Roscher, Ribana, Bohn, Bastian, Duarte, Marco F., & Garcke, Jochen. 2020\.  Explainable machine learning for scientific insights and discoveries.  Ieee access, 8, 42200–42216.
  * Rubinov & Sporns, (2010) Rubinov, Mikail, & Sporns, Olaf. 2010.  Complex network measures of brain connectivity: uses and interpretations.  Neuroimage, 52(3), 1059–1069.
  * Rudin, (2014) Rudin, Cynthia. 2014.  Algorithms for interpretable machine learning.  Page 1519 of: Proceedings of the 20th acm sigkdd international conference on knowledge discovery and data mining.  KDD ’14.  New York, NY, USA: Association for Computing Machinery.
  * Rudin, (2019) Rudin, Cynthia. 2019.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature machine intelligence, 1(5), 206–215.
  * Rudin et al. , (2022) Rudin, Cynthia, Chen, Chaofan, Chen, Zhi, Huang, Haiyang, Semenova, Lesia, & Zhong, Chudi. 2022.  Interpretable machine learning: Fundamental principles and 10 grand challenges.  Statistics surveys, 16(none), 1 – 85.
  * Samek & Müller, (2019) Samek, Wojciech, & Müller, Klaus-Robert. 2019.  Towards explainable artificial intelligence.  Cham: Springer International Publishing.  Pages 5–22.
  * Samek et al. , (2021) Samek, Wojciech, Montavon, Grégoire, Lapuschkin, Sebastian, Anders, Christopher J., & Müller, Klaus-Robert. 2021.  Explaining deep neural networks and beyond: A review of methods and applications.  Proceedings of the ieee, 109(3), 247–278.
  * Satija et al. , (2015) Satija, Rahul, Farrell, Jeffrey A, Gennert, David, Schier, Alexander F, & Regev, Aviv. 2015.  Spatial reconstruction of single-cell gene expression data.  Nature biotechnology, 33(5), 495–502.
  * Shah & Peters, (2020) Shah, Rajen D., & Peters, Jonas. 2020.  The hardness of conditional independence testing and the generalised covariance measure.  The annals of statistics, 48(3), 1514 – 1538.
  * Shah & Samworth, (2013) Shah, Rajen D, & Samworth, Richard J. 2013.  Variable selection with error control: another look at stability selection.  Journal of the royal statistical society: Series b (statistical methodology), 75(1), 55–80.
  * Stodden, (2020) Stodden, Victoria. 2020.  Theme Editor’s Introduction to Reproducibility and Replicability in Science.  Harvard data science review, 2(4).  https://hdsr.mitpress.mit.edu/pub/kincvy67.
  * Taeb et al. , (2020) Taeb, Armeen, Shah, Parikshit, & Chandrasekaran, Venkat. 2020.  False discovery and its control in low rank estimation.  Journal of the royal statistical society series b: Statistical methodology, 82(4), 997–1027.
  * Taylor & Tibshirani, (2015) Taylor, Jonathan, & Tibshirani, Robert J. 2015\.  Statistical learning and selective inference.  Proceedings of the national academy of sciences, 112(25), 7629–7634.
  * Tibshirani, (1996) Tibshirani, Robert. 1996.  Regression shrinkage and selection via the lasso.  Journal of the royal statistical society. series b (methodological), 58(1), 267–288.
  * Toreini et al. , (2020) Toreini, Ehsan, Aitken, Mhairi, Coopamootoo, Kovila, Elliott, Karen, Zelaya, Carlos Gonzalez, & van Moorsel, Aad. 2020.  The relationship between trust in ai and trustworthy machine learning technologies.  Page 272–283 of: Proceedings of the 2020 conference on fairness, accountability, and transparency.  FAT* ’20.  New York, NY, USA: Association for Computing Machinery.
  * Tukey, (1977) Tukey, John W. 1977\.  Exploratory data analysis.  Addison-Wesley.
  * Vallejos et al. , (2015) Vallejos, Catalina A, Marioni, John C, & Richardson, Sylvia. 2015.  Basics: Bayesian analysis of single-cell sequencing data.  Plos computational biology, 11(6), e1004333.
  * van de Geer et al. , (2014) van de Geer, Sara, Bühlmann, Peter, Ritov, Ya’acov, & Dezeure, Ruben. 2014.  On asymptotically optimal confidence regions and tests for high-dimensional models.  The annals of statistics, 42(3), 1166 – 1202.
  * van de Schoot et al. , (2021) van de Schoot, Rens, Depaoli, Sarah, King, Ruth, Kramer, Bianca, Märtens, Kaspar, Tadesse, Mahlet G, Vannucci, Marina, Gelman, Andrew, Veen, Duco, Willemsen, Joukje, et al. . 2021.  Bayesian statistics and modelling.  Nature reviews methods primers, 1(1), 1.
  * Varshney, (2019) Varshney, Kush R. 2019\.  Trustworthy machine learning and artificial intelligence.  Xrds, 25(3), 26–29.
  * Wainwright, (2019) Wainwright, Martin J. 2019\.  High-dimensional statistics: A non-asymptotic viewpoint.  Cambridge Series in Statistical and Probabilistic Mathematics.  Cambridge University Press.
  * Weinstein et al. , (2013) Weinstein, John N, Collisson, Eric A, Mills, Gordon B, Shaw, Kenna R Mills, Ozenberger, Brad A, Ellrott, Kyle, Shmulevich, Ilya, Sander, Chris, & Stuart, Joshua M. 2013.  The cancer genome atlas pan-cancer analysis project.  Nature genetics, 45(10), 1113–1120.
  * Williamson et al. , (2022) Williamson, Brian D., Gilbert, Peter B., Simon, Noah R., & Carone, Marco. 2022.  A general framework for inference on algorithm-agnostic variable importance.  Journal of the american statistical association, 0(0), 1–14.
  * Willis & Stodden, (2020) Willis, Craig, & Stodden, Victoria. 2020.  Trust but Verify: How to Leverage Policies, Workflows, and Infrastructure to Ensure Computational Reproducibility in Publication.  Harvard data science review, 2(4).  https://hdsr.duqduq.org/pub/f0obb31j.
  * Witten & Tibshirani, (2010) Witten, Daniela M, & Tibshirani, Robert. 2010.  A framework for feature selection in clustering.  Journal of the american statistical association, 105(490), 713–726.
  * Xu et al. , (2019) Xu, Feiyu, Uszkoreit, Hans, Du, Yangzhou, Fan, Wei, Zhao, Dongyan, & Zhu, Jun. 2019.  Explainable ai: A brief survey on history, research areas, approaches and challenges.  Pages 563–574 of: Tang, Jie, Kan, Min-Yen, Zhao, Dongyan, Li, Sujian, & Zan, Hongying (eds), Natural language processing and chinese computing.  Cham: Springer International Publishing.
  * Yu & Kumbier, (2020) Yu, Bin, & Kumbier, Karl. 2020.  Veridical data science.  Proceedings of the national academy of sciences, 117(8), 3920–3929.
  * Zhang & Janson, (2020) Zhang, Lu, & Janson, Lucas. 2020.  Floodgate: inference for model-free variable importance.  arxiv preprint arxiv:2007.01283.
  * Zhang et al. , (2019) Zhang, Yujia, Song, Kuangyan, Sun, Yiming, Tan, Sarah, & Udell, Madeleine. 2019.  ” why should you trust my explanation?” understanding uncertainty in lime explanations.  arxiv preprint arxiv:1904.12991.
  * Zhao & Yu, (2006) Zhao, Peng, & Yu, Bin. 2006.  On model selection consistency of lasso.  Journal of machine learning research, 7(90), 2541–2563.
  * Zou & Hastie, (2005) Zou, Hui, & Hastie, Trevor. 2005.  Regularization and variable selection via the elastic net.  Journal of the royal statistical society: Series b (statistical methodology), 67(2), 301–320.