# Meaningfully Debugging Model Mistakes using Conceptual Counterfactual
Explanations

Abubakar Abid  Mert Yuksekgonul  James Zou

###### Abstract

Understanding and explaining the mistakes made by trained models is critical
to many machine learning objectives, such as improving robustness, addressing
concept drift, and mitigating biases. However, this is often an ad hoc process
that involves manually looking at the model’s mistakes on many test samples
and guessing at the underlying reasons for those incorrect predictions. In
this paper, we propose a systematic approach, conceptual counterfactual
explanations (CCE), that explains why a classifier makes a mistake on a
particular test sample(s) in terms of human-understandable concepts (e.g. this
zebra is misclassified as a dog because of faint _stripes_). We base CCE on
two prior ideas: counterfactual explanations and concept activation vectors,
and validate our approach on well-known pretrained models, showing that it
explains the models’ mistakes meaningfully. In addition, for new models
trained on data with spurious correlations, CCE accurately identifies the
spurious correlation as the cause of model mistakes from a single
misclassified test sample. On two challenging medical applications, CCE
generated useful insights, confirmed by clinicians, into biases and mistakes
the model makes in real-world settings. The code for CCE is publicly available
at https://github.com/mertyg/debug-mistakes-cce.

##  1 Introduction

People who use machine learning (ML) models often need to understand why a
trained model is making a particular mistake. For example, upon seeing a model
misclassify an image, an ML practitioner may ask questions such as: Was this
kind of image underrepresented in my training distribution? Am I preprocessing
the image correctly? Has my model learned a spurious correlation or bias that
is hindering generalization? Answering this question correctly affects the
usability of a model and can help make the model more robust. Consider a
motivating example:

Example 1: Usability of a Pretrained Model. A pathologist downloads a
pretrained model to classify histopathology images. Despite a high reported
accuracy, he finds the model performing poorly on his images. He investigates
why that is the case, finding that the hues in his images are different than
in the original training data. Realizing the issue, he is able to transform
his own images with some preprocessing, matching the training distribution and
improving the model’s performance.

In the above example, we have a domain shift occurring between training and
test time, which degrades the model’s performance (Subbaswamy et al., 2019;
Koh et al., 2020). By explaining the cause of the domain shift, we are able to
easily fix the model’s predictions. On the other hand, if the domain shift is
due to more complex spurious correlations the model has learned, it might need
to be completely retrained before it can be used. During development,
identifying and explaining a model’s failure points can also make it more
robust (Abid et al., 2019; Kiela et al., 2021), as in the following example:

Example 2: Discovering Biases During Development. A dermatologist trains a
machine learning classifier to classify skin diseases from skin images
collected from patients at her hospital. She shares her trained model with a
colleague at a different hospital, who reports that the model makes many
mistakes with his own patients. She investigates why the model is making
mistakes, realizing that patients at her colleague’s hospital have different
skin colors and ages. Answering this question allows her to not only know her
own model’s biases but also guides her to expand her training set with the
right kind of data to build a more robust model.

Explaining a model’s mistakes is also useful in other settings where data
distributions may change, such as concept drift for deployed machine learning
models (Lu et al., 2018). Despite its usefulness, explaining a model’s
performance drop is often an ad hoc process that involves manually looking at
the model’s mistakes on many test samples and guessing at the underlying
reasons for those incorrect predictions. We present counterfactual conceptual
explanations (CCE), a systematic method for explaining model mistakes in terms
of meaningful human-understandable concepts, such as those in the examples
above (e.g. _hues_). In addition, CCE was developed with the following
criteria:

  * •

No training data or retraining: CCE does not require access to the training
data or model retraining.

  * •

Only needs the model: CCE only needs white-box access to the model. The user
can use any dataset to learn desired concepts.

  * •

High-level explanations: CCE provides high-level explanations of model
mistakes using concepts that are easy for users to understand.

Figure 1: Explaining with Conceptual Counterfactuals: (a) In generating
conceptual explanations, the first step is to define a concept library. After
defining a concept library, we then learn a concept activation vector (CAV)
(Kim et al., 2018) for each concept. (b) Given a misclassified sample, such as
the Zebra image shown here as an African hunting dog, we would like to
generate a conceptual counterfactual. Meaning that we would like to generate a
perturbation in the embedding space that would correct the model prediction,
using a weighted sum of our concept bank. (c) Our method assigns a score to
the set of concepts. A large positive score means that adding that concept to
the image (e.g. stripes) will increase the probability of correctly
classifying the image, as will removing or reducing a concept with a large
negative score (e.g. polka dots or dogness).

Fig. 1 provides an overview of our CCE method. To summarize our contributions,
we develop a novel method, CCE, that can systematically explain model mistakes
in terms of high-level concepts that are easy for users to understand. We show
that CCE correctly identifies spurious correlations in model training, and we
quantitatively validate CCE across different experiments over natural and
clinical images.

##  2 Related Works

CCE is inspired by prior efforts to explain machine learning models’
predictions. Two ideas are particularly relevant: counterfactual explanations
and concept activation vectors.

##### Counterfactual Explanations

Counterfactual explanations (Verma et al. (2020) provides a comprehensive
review) are a class of model interpretation methods that seek to answer: what
perturbations to the input are needed for a model’s prediction to change in a
particular way? A large number of counterfactual explanation methods have been
proposed with various desiderata such as sparse perturbations (Wachter et al.,
2017), perturbations that remain close to the data manifold (Dhurandhar et
al., 2018), and causality (Mahajan et al., 2019). What these methods share in
common is that they provide an interpretable link between model predictions
and perturbations to input features. There are several works focusing on
counterfactual explanations for image data(Goyal et al., 2019; Singla et al.,
2020) and many of these methods use distractor images or modify input images
to explain the model behavior. However, perturbations or saliency maps in the
input space are shown to be tricky to interpret and be adopted by
users(Alqaraawi et al., 2020; Adebayo et al., 2018).

To compute CCE, we use a similar approach, perturbing input data to change its
predictions, but for a complementary goal: to understand the limitations and
biases of our model and its training data, rather than to change a specific
prediction. Further improving upon existing work, we do this by directly using
human-interpretable concepts, without having access to training data.

##### Concept Activation Vectors

To explain an image classification model’s mistakes in a useful way, we need
to operate not with low-level features, but with more meaningful concepts.
Concept activation vectors (CAVs) (Kim et al., 2018) are a powerful method to
understand the internals of a network in human-interpretable concepts. CAVs
are linear classifiers trained in the bottleneck layers of a network and
correspond to concepts that are usually user-defined or automatically
discovered from data (Ghorbani et al., 2019). Unlike many prior interpretation
methods, explanations produced by CAVs are in terms of high-level, human-
understandable concepts rather than individual pixels (Simonyan et al., 2013;
Zhou et al., 2016) or training samples (Koh & Liang, 2017).

In previous literature, CAVs have been used to test the association between a
concept and the model’s prediction on a class of training data (Kim et al.,
2018) as well as provide visual explanations for predictions (Zhou et al.,
2018). Our approach extends CAVs by showing that perturbations along the CAV
can be used to change a model’s prediction on a specific test sample, e.g. to
correct a mistaken prediction, and thereby be used in a similar manner to
counterfactuals. Since samples (even of the same class) can be misclassified
for different reasons (see Fig. 5), our approach allows a more contextual
understanding of a model’s behavior and mistakes.

CoCoX(Akula et al., 2020) explored counterfactual explanations using an
approach called fault-lines. There are several key differences between our
approach and theirs. Our CCE method solves a constrained optimization problem
to ensure the validity (3.2) of the counterfactual concepts, which was not
done in CoCoX. CCE also allows analysis of batches of data, while CoCoX was
developed for individual data. Furthermore, these methods use different
approaches for capturing concepts. We provide several controlled experiments
to demonstrate the benefit of CCE in the real world.

##### Bias/Error Detection and Robustness

Understanding a model’s behavior and explaining its mistakes is critical for
building more robust and less biased models, as we have discussed in Section
1. While some other methods, such as FairML (Adebayo & Kagal, 2016) and the
What-If Tool (Wexler et al., 2020), can also be used to discover biases and
failure points in models, they are typically limited to tabular datasets where
sensitive features are explicitly designated, unlike our approach which is
designed for unstructured data and works with a more flexible and diverse set
of concepts. In a similar spirit to our work, (Singh et al., 2020) aim to
identify and mitigate contextual bias, however, they assume access to the
whole training setup and data; which is a limiting factor in practical
scenarios.

Our approach can also be used to explain a model’s performance degradation
with data drift. It is complementary to methods for out-of-distribution sample
detection (Hendrycks & Gimpel, 2016) and black-box shift detection (Lipton et
al., 2018), which can detect data drift, but cannot explain the underlying
reasons. There are also algorithmic approaches to improve model robustness
with data drift (Raghunathan et al., 2018; Nestor et al., 2019). Although
these methods can provide some guarantees around robustness for unstructured
data, they are in terms of low-level perturbations and do not hold with more
conceptual and natural changes in data distributions. There are several recent
methods for detecting clusters of mistakes made by a model (Kim et al., 2019;
d’Eon et al., 2021), which is useful for detecting disparity in the model, for
example. CCE differs in that instead of finding mistake clusters, CCE explains
errors from just one image (or a batch).

##  3 Methods

In this section, we detail the key steps of our method. Let us define basic
notations: let $f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{k}$ be a deep neural
network, let $\bm{x}\in\mathbb{R}^{d}$ be a test sample belonging to class
$y\in\\{1,\ldots k\\}$. We assume that that the model misclassifies $\bm{x}$,
meaning that $\operatorname*{arg\,max}_{i}f(\bm{x})_{i}\neq y$ or simply that
the model’s confidence in class $y$, $f(\bm{x})_{y}$, is lower than desired.
Let $m$ be the dimensionality of the bottleneck layer $L$, and
$b_{L}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}$ be the “bottom” of the
network, which maps samples from the input space to the bottleneck layer, and
$t_{L}:\mathbb{R}^{m}\rightarrow\mathbb{R}^{k}$ be the “top” of the network,
defined analogously. For readability, we will usually underline class names
and italicize concepts throughout this paper.

###  3.1 Learning Concepts

In generating conceptual explanations, the first step is to define a concept
library: a set of human-interpretable concepts $C=\\{c_{1},c_{2},...\\}$ that
occur in the dataset. For each concept, we collect positive examples,
$P_{c_{i}}$, that exhibit the concept, as well as negative examples,
$N_{c_{i}}$, in which the concept is absent. Unless otherwise stated, we use
$P_{c_{i}}=N_{c_{i}}=100$. These concepts can be defined by the researcher, by
non-ML domain experts, or even learned automatically from the data (Ghorbani
et al., 2019). Since concepts are broadly reusable within a data domain, they
can also be shared among researchers. For our experiments with natural images,
we defined 170 general concepts that include (a) the presence of specific
objects (e.g. mirror, person), (b) settings (e.g. street, snow) (c) textures
(e.g. stripes, metal), and (d) image qualities (e.g. blurriness, greenness).
Many of our concepts were adapted from the BRODEN dataset of visual concepts
(Fong & Vedaldi, 2018). Note that the data used to learn concepts can differ
from the data used to train the ML model that we evaluate.

After defining a concept library, we then learn an SVM and the corresponding
CAV for each concept. We follow the same procedure as in (Kim et al., 2018),
for the penultimate layer of a ResNet18 pretrained in ImageNet. This step only
needs to be done once for each model that we want to evaluate, and the CAVs
can then be used to explain any number of misclassified samples. We refer the
reader to Appendix A.1 for implementational details and Appendix B for a full
list of concepts. We denote the vector normal to the classification hyperplane
boundary as ${\bm{c}}_{i}$(normalized, i.e. $|{\bm{c}}_{i}|=1$) and the
intercept of the SVM as $\phi_{i}$. To measure whether concepts are
successfully learned, we keep a hold-out validation set and measure the
validation accuracy, disregarding concepts with accuracies below a threshold
(0.7 in our experiments, which left us with 168 of the 170 concepts). We
provide more details about the threshold in Appendix B.

###  3.2 Conceptual Counterfactual Explanations

Drawing inspiration from the counterfactual explanations literature (Verma et
al., 2020), we generate a perturbation for a given misclassified test sample
by varying the amount of different concepts in a way such that the
perturbation satisfies the following principles:

  1. 1.

Correctness: A counterfactual is considered correct if it achieves a desired
outcome. In our case, this would mean that the perturbed test point should be
classified as the correct label.

  2. 2.

Validity: We would like our counterfactuals to be valid, such that they would
not violate real-world conditions. In our case, this means ensuring that the
perturbed points contain realistic levels of each concept, as discussed below.

  3. 3.

Sparsity: The ultimate goal of generating explanations is communicating them
to users. Analyzing a large number of modifications and interactions may not
be trivial, so perturbations should change a small number of concepts.

Let $\mathcal{L}_{CE}$ denote the cross entropy loss. Using the concept
vectors and intercepts ($\bm{c}_{i},\phi_{i}$)(see 3.1), we build our concept
bank $\bm{C}\in\mathbb{R}^{N_{c}\times m},\bm{\phi}\in\mathbb{R}^{N_{c}}$
where $N_{c}$ is the number of concepts and $m$ is the output dimension of the
bottleneck layer. As our goal is to come up with a scoring scheme for
concepts, we need to define what it means to add a concept. For this purpose,
we use statistics of training samples. We compute the geometric margin to the
decision boundary of the SVM,
$d_{i}=\bm{c}_{i}b_{L}(\bm{x}_{i})^{T}+\phi_{i}$, for all of the training
examples. As different concepts have different embedding volumes, we scale
each concept by the maximum amount of the concept observed in the data used to
learn that concept. Let $d^{\max}_{i}$ denote the maximum margin in the
training distribution. We let
$\tilde{\mathbf{c}}_{i}=d^{\max}_{i}\mathbf{c}_{i}$, and using the scaled
concept vectors we construct our final concept bank $\tilde{\mathbf{C}}$. For
example, adding 1 unit of $\tilde{\mathbf{c}}_{\text{redness}}$ would mean
adding the maximum amount of redness seen before.

Our optimization problem is implemented as follows:

$\displaystyle\min_{\bm{w}}$
$\displaystyle{\color[rgb]{126,0,24}\mathcal{L_{\textrm{CE}}}(y,t_{L}(\mathbf{b}_{L}(\bm{x})+\bm{w}\tilde{\bm{C}}))}+{\color[rgb]{255,172,59}\alpha|\bm{w}|_{1}+\beta|\bm{w}|_{2}}$
(1) s.t.
$\displaystyle{\color[rgb]{69,2,112}\bm{w}^{\min}\leq\bm{w}\leq\bm{w}^{\max}}$

By minimizing the cross entropy loss, we aim to flip the label of the model
prediction to correct the misclassification, to ultimately achieve
correctness. We do this by adding a weighted sum of concept vectors, weighted
by the parameter $\bm{w}$. Additionally, we apply elastic net regularization
to introduce sparsity in the concept scores.

We further introduce validity constraints to make sure that the concept
additions are within a realistic range. Concretely, assume that we have a
concept that already exists in the image. We can query this fact by looking at
the prediction of our concept SVM. If the concept already exists in the image,
adding that concept to the image would be less meaningful. Similarly, if a
concept is already absent in the image, then removing it should not be a valid
action. Following these intuitions, we will use the bounds
[$\mathbf{w^{\min}}$, $\mathbf{w^{\max}}$] to guide the optimization.

First, we should not be able to add the concept to explain the model’s mistake
if the concept is already in the image, i.e. the score $w_{i}$ should not be
positive:

$w^{\max}_{i}=0\quad\textrm{
if}\quad\bm{c}_{i}b_{L}(\bm{x})^{T}+\phi_{i}>\kappa_{i}$ (2)

Here, $\kappa_{i}$ is an offset we use to predict the existence of the
concept. Setting $\kappa_{i}=0$ would mean using the SVM as the decision
boundary. As an alternative strategy, we can vary $\kappa_{i}$ to allow for
weaker forms of validity regularization (e.g. by setting it to the mean
positive margin over the training samples).

Moreover, we should be able to restrict the amount of concept we are adding to
the embedding, e.g. adding an infinite amount of redness may not be
meaningful. We first use the training samples to identify the maximum
geometric margin for the concept i, $d_{i}^{\max}$. Then we restrict $w_{i}$
such that the concept addition would result in a margin at most as large as
the maximum margin over the training samples. We want to have

$\bm{c}_{i}^{T}(b_{L}(\bm{x})+w_{i}\tilde{\bm{c}}_{i})+\phi_{i}\leq
d_{i}^{\max}$ (3)

and thus

$w_{i}\leq\frac{d_{i}^{\max}-\bm{c}_{i}^{T}b_{L}(\bm{x})-\phi_{i}}{\bm{c}_{i}^{T}\tilde{\bm{c}}_{i}}=\frac{d_{i}^{\max}-\bm{c}_{i}^{T}b_{L}(\bm{x})-\phi_{i}}{d_{i}^{\max}}$
(4)

Combining this with the Equation 2 would lead to our final constraints and
follow the same procedure to compute the lower bounds :

$w^{\max}_{i}=\begin{cases}0&\textrm{if}\quad\bm{c}_{i}b_{L}(\bm{x})^{T}+\phi_{i}>\kappa_{i}\\\
\frac{d^{\max}_{i}-\bm{c}_{i}^{T}b_{L}(\bm{x})-\phi_{i}}{d^{\max}_{i}}&\textrm{else}\end{cases}$
(5)

$w^{\min}_{i}=\begin{cases}0&\textrm{if}\quad\bm{c}_{i}^{T}b_{L}(\bm{x})^{T}+\phi_{i}<-\kappa_{i}\\\
\frac{d^{\min}_{i}-\bm{c}_{i}^{T}b_{L}(\bm{x})-\phi_{i}}{d^{\min}_{i}}&\textrm{else}\end{cases}$
(6)

Algorithm 1 Conceptual Counterfactual Explanations(CCE)

Input: $\bm{x},y,[\bm{w}^{\min},\bm{w}^{\max}],\tilde{\bm{C}}$

Hyperparameters: $\alpha,\beta,\gamma,\eta$

Output: $\bm{w}$

repeat

$\hat{y}\leftarrow t_{L}(b_{L}(\bm{x})+\bm{w}\tilde{\bm{C}})$

$\mathcal{L}_{\textrm{total}}\leftarrow\mathcal{L}_{\textrm{CE}}(\hat{y},y)+\alpha|\bm{w}|_{1}+\beta|\bm{w}|_{2}$

${\bm{w}}\leftarrow\text{G.Descent}(\mathcal{L}_{\textrm{total}},\bm{w},\textrm{lr}=\gamma,\textrm{momentum}=\eta)$

${\bm{w}}\leftarrow\textrm{clamp}(\bm{w},\bm{w}^{\min},\bm{w}^{\max})$

until $\bm{w}$ not converged

We solve the problem in Equation 1 using Projected Gradient Descent, where we
introduce projection steps to enforce the validity constraints. Namely, after
each gradient step, we clamp the values of the scores to remain within the
precomputed range. The final Conceptual Counterfactual Explanations (CCE)
algorithm is given in Algorithm 1. Throughout the experiments, unless
otherwise stated, we use
$\alpha=0.1,\beta=0.01,\gamma=0.01,\eta=0.9,\kappa_{i}=0$.

In summary, we propose that using our validity constraints and achieving a
correct counterfactual, we can explain the model mistakes and behavior in an
interpretable (sparse) manner. A large positive score means that adding that
concept to the image will increase the probability of correctly classifying
the image, as will removing or reducing a concept with a large negative score.
CCE provides us with an assessment of which concepts explain a misclassified
sample.

##  4 Results

In this section, we demonstrate how CCE can be used to explain model
limitations. First, we show that CCE reveals high-level spurious correlations
learned by the model. We then show analogous results for low-level image
characteristics. Finally, we show real-world medical applications where we are
able to identify biases and spurious correlations in the training dataset and
give users feedback about the image quality. In all scenarios, CCE (Alg. 1)
correctly identifies biases in the model and artifacts in the image as
explanations of model mistakes.

Figure 2: Validating CCE by Identifying Spurious Correlations: (a) First, we
train 5-class animal classification models on skewed datasets that contain
spurious correlations that occur in practice. For example, one model is
trained on images of dogs that are all taken with snow. This causes the model
to associate snow with dogs and incorrectly predicts dogs without snow to not
be dogs. We test whether our method can discover this spurious correlation.
(b) We repeat this experiment with 20 different models, each that has learned
a different spurious correlation, finding that in most cases the model
identifies the spurious correlation in more than 90% of the misclassified test
samples. For comparison, we also run CCE using a control model without the
spurious correlation, and we randomly select concepts as well. The random
performance is evaluated by sampling three concepts out of the available 150
and calculating the Precision@3.

###  4.1 CCE Reveals Spurious Correlations Learned by the Model

We start by demonstrating that CCE correctly identifies high-level spurious
correlations that models may have learned. For example, consider a training
dataset that consists of images of different animals in natural settings. A
model trained on such a dataset may capture not only the desired correlations
related to the class of animals but spurious correlations related to other
objects present in the images and the setting of the images. We use CCE to
identify these spurious correlations.

To systematically validate CCE, we need to know the ground-truth spurious
correlations that a model has learned. To this end, we train models with
intentional and known spurious correlations using the MetaDataset (Liang &
Zou, 2021), a collection of labeled datasets of animals in different settings
and with different objects. We construct 20 different training scenarios, each
consisting of 5 animal classes (cat, dog, bear, bird, elephant). We trained a
separate model for each scenario. In each scenario, one class is only included
with a specific confounding variable (e.g. all images of dogs are with snow),
inducing a spurious correlation in the model (images of dogs without snow will
be misclassified), which we can probe with CCE. We also train a control model
using random samples of animals across contexts, without intentional spurious
correlations. In Appendix A.8.2, we provide training distributions with less
severe spurious correlations, i.e. where instead of all images having the
confounding concept, we use a varying level of severity in the correlations.

Our experimental setup is shown in Fig. 2(a). We train models with $n=750$
images, fine-tuning a pretrained ResNet18 model. In all cases, we achieve a
validation accuracy of at least 0.7. We then present the models with 50 out-
of-distribution (OOD) images (randomly sampled from the entire MetaDataset),
i.e. images of the class without the confounding variable present during
training.

We then use CCE to recover the top 3 concepts that would explain a model’s
mistake on each of 50 OOD images that are misclassified, then we report if the
spurious concept is among these 3 concepts(Precision@3). For almost all
images, CCE identifies the ground-truth spurious correlation as one of the top
3 concepts (Fig. 2(b)). For example, in the model we trained with dogs
confounded with snow, CCE recovered the spurious correlation in 100% of test
samples. We compare these results to performing CCE using the control model,
in which the same test images are presented, as well as to picking concepts
randomly, both of which result in the spurious correlation being identified
much less frequently.

Additionally, we compare our approach to a (simpler) univariate version of
CCE, CCE(Univariate), Conceptual Sensitivity Score(CSS) (Kim et al., 2018),
and CoCoX (Akula et al., 2020). Our implementation details for the prior work
is shared in the A.3. Instead of running our optimization method over multiple
concepts simultaneously, we iterate over each CAV and quantify how a
perturbation in the direction of the concept ($d^{\max}_{i}\bm{c}_{i}$)
changes the prediction probability of the correct class $y$. Specifically, we
compute the CCE as the difference
$t_{L}(b_{L}(\bm{x})+d^{\max}_{i}\bm{c}_{i})_{y}-t_{L}(\bm{x})_{y}$, for each
concept, then we order the concepts by the change in the probability.

Method | Mean Prec@3 | Median Rank
---|---|---
Random | $0.02$ | $82.65(42.7,120.4)$
CSS | $0.003$ | $76.5(69.79,87.51)$
CoCoX | $0.73$ | $4.63(3.82,5.89)$
CCE(Control) | $0.04$ | $32.3(28.03,40.05)$
CCE(Univariate) | $0.91$ | $2.00(1.71,2.35)$
CCE | $0.95$ | $1.85(1.80,2.10)$
Table 1: Empirical evaluation for detecting spurious correlations in training
data. We report Precision@3 and ranks averaged over 20 scenarios, the complete
table is in Appendix Table 2. Distribution of the Precision@K metric as we
vary K can be found in Appendix A.9.

Table 1 provides results over the 20 scenarios and the detailed results can be
found in Appendix Table 2. We further provide the mean of the median rank of
the concept in each scenario, along with the mean of the first and third
quartiles of ranks. Both CCE(univariate) and CCE recover the ground-truth
spurious correlation as one of the top 3 concepts correctly across the
scenarios. CSS is not designed to inform the user about a given input, it is
not a metric for an individual sample but works with the aggregation step
(tCAV) when only the sign is used. CoCoX is comparable to our method, and the
complete (multivariate) CCE achieves the highest precision and the best median
rank.

Testing CCE in more challenging settings: In Appendix A.8 we provide results
in more challenging scenarios. First, as a practical scenario, we test when
the target spurious concept does not exist in the concept bank. In this case,
CCE identifies concepts that ‘hinting’ at (i.e. co-occuring with or similar
to) the target concepts, which gives reasonable information to users (A.8.1).
Second, we test when the correlations are less drastic, i.e. not all of the
images in the training dataset have the particular concept. CCE can identify
biases even when correlations are subtle (A.8.2). Third, we evaluate CCE in
the batch-setting: instead of a sample-by-sample analysis, we apply CCE
simultaneously to batch of data. Batch-Mode CCE successfully provides a
holistic understanding of a set of mistakes (A.8.3).

Figure 3: Validating CCE: (a) Here, we take an arbitrary test sample of a
Granny Smith apple that is originally correctly classified and perturb the
image by turning it gray. We compute CCE scores at each perturbed image and
observe that the score for greenness increases as the image is grayed.  (b) We
demonstrate the effectiveness of validity constraints in a qualitative
scenario. In the last two columns, we provide ranks of each concept when a
particular method is used. Without validity, methods can use concepts that
already exist in the image to explain model mistakes.

###  4.2 CCE Reveals Low-Level Image Artifacts

We show that CCE can capture low-level spurious correlations that models may
have learned. For example, the ImageNet(Deng et al., 2009) dataset on which
SqueezeNet was trained includes a class of green apples known as Granny Smith,
which were always colored images. Fig. 3(a) shows that as the image is grayed,
the probability of it being classified as Granny Smith decreases, while the
CCE for greenness increases. We provide details and examples about this
experiment in Appendix A.7.

###  4.3 Validity Constraint Improves Explanations

Here we demonstrate validity constraints are necessary to plausibly explain
model mistakes. In these experiments, instead of training the model with a
single concept that is associated with a class, we use two concepts. For
instance, we use 50 dog images with water and 50 with bed during training. In
Figure 3a, we show such a scenario. In all of these images, CCE(Univariate)
identifies both bed and water concepts to explain the model mistakes. However,
in the first image, there is already water and in the second image, there is
already a bed. Thus, using them in the counterfactual result in an invalid
explanation. However, when CCE with validity constraints is used, the score
for the water concept in the first image and the bed concept in the second
image drops. Ultimately, this would result in a plausible counterfactual
explanation. More examples are provided in Appendix A.4.

###  4.4 Evaluating CCE in the Wild

We run experiments where we evaluate CCE on real-world medical data and
models. We demonstrate that CCE helps identify learned biases and artifacts
that provide insight into a model’s mistakes. Throughout our evaluations, we
worked with a board-certified dermatologist and cardiologist to confirm the
clinical relevance of CCE explanations.

Dermatology - Skin Condition Classification:  We follow the model training
procedure described in (Groh et al., 2021) to train a ResNet18 model to
predict one of the 114 skin conditions using the Fitzpatrick17k dataset of
16,577 annotated skin images (Groh et al., 2021). This classifiation model
achieved 20% overall accuracy, closely matching the number reported in the
paper (see Appendix A.5 for details). To explain the model’s mistakes, in
addition to the 168 concepts used in Sec. 3.1, we also learned 8 clinically
relevant concepts: defocus blur, zoom blur, brightness, motion blur, contrast,
dark skin type, skin hair, and zoom. To learn each of these concepts, we use
25 pairs of positive and negative images (except the skin hair concept, for
which we used 10 images, where the skin hair images are obtained from the ISIC
(Rotemberg et al., 2021) dataset).

Figure 4: CCE explains model mistakes using learned biases and image quality
conditions. (a) CCE identifies dark skin type correlation with the allergic
contact dermatitis condition that exists in the training dataset. (b, c, d)
CCE identifies image artifacts that degrade the model performance.

In Figure 4, we observe several ways CCE guides our understanding of model
mistakes. In addition to the unbalanced fraction of skin type groups over the
whole dataset, we find that there are wider discrepancies when particular skin
conditions are considered. For example, for the allergic contact dermatitis
condition, there are 259 images from the lightest skin tones in the training
data, 137 images of intermediate skin tones, and only 24 images of dark skin
tones. In the test set, for 23 out of all 24 allergic contact dermatitis dark-
skin images where the model makes a mistake, CCE identifies the dark skin type
concept as one of the 3 concepts with the largest negative score. This
collectively reflects bias in the training data.

For fixed eruptions and mucinosis, CCE finds image qualities that would
increase the classification performance. Namely, CCE identified that the blur
in 2nd image in Fig. 4 caused the model’s mistake (reducing blur would correct
the mistake). In the 3rd image, CCE learned that the image is too zoomed out,
and increasing zoom would correct the mistake. For the 4th image, CCE
identified that too much skin hair in the image contributed to the model’s
mistake. This is consistent with other studies which show that presence of
skin hair can degrade the model performance (Okur & Turkan, 2018). In all
cases, CCE identifies relevant concepts from our expanded concept bank and
displays them to the user. These explanations were validated by a trained
dermatologist as plausible reasons why these images may have been difficult to
classify. In Appendix A.5, we describe details of this experiment and include
comments about concepts suggested by CCE that seem irrelevant.

##### Cardiology - Pneumothorax Classification from Chest X-Ray Images

Several studies have raised concerns about the biases learned by chest X-ray
models and deployability in novel domains (Seyyed-Kalantari et al., 2020;
Larrazabal et al., 2020; Wu et al., 2021). We investigate a cross-site
setting, where models are trained on Chest X-Ray images to classify the
pneumothorax condition. We take a binary classification model trained on a
dataset collected from the National Institutes of Health Clinical Center in
Bethesda (NIH)(Wang et al., 2017) and we test the model with images obtained
from the Stanford Health Care in Palo Alto (SHC)(Irvin et al., 2019). In this
experiment, we follow the protocol described in (Wu et al., 2021); see Section
A.6 for details. Notably, the NIH dataset consists of images from the frontal
(AP/PA) view positions, whereas in the SHC dataset, there are also images
obtained from the Lateral View, i.e. the model has not seen any images from
the lateral view during training. Examples of these images can be seen in
Appendix 9.We use 100 pairs of images to learn clinically relevant concepts:
Lateral View & AP View, Cardiomegaly & Atelectasis (comorbidities),  gender,
and age under 24.

From the SHC dataset, we randomly select 150 images taken from the lateral
view where the model makes a mistake. In $\mathbf{100\%}$ of these 150 test
images, the Lateral View concept received the largest negative score, meaning
that CCE finds removing the concept would increase the probability of
correctly classifying these images. This is consistent with cardiologist
expectation: a model that has not seen images from the lateral view may fail
to generalize to that category. With CCE, users can identify shortcomings and
help address the biases in the training dataset.

##  5 Conclusion and Future Work

We present a simple and intuitive method, CCE, that generates meaningful
insights into why machine learning models make mistakes on test samples. We
validated that CCE identifies spurious correlations learned by the model for
both natural images and medical images, where CCE’s explanations are confirmed
by clinicians. In medical applications, CCE was able to inform the end-user
about the image quality conditions and identify biases in the model’s training
data. CCE is fast: the concept bank just needs to be learned once using simple
SVMs and each test example takes $<0.3$ seconds on a single CPU. It can be
readily applied to any deep network without retraining and provides
explanations of mistakes in human-interpretable terms.

CCE can detect biases in training data that lead to model mistakes without
needing the training data. It requires a small number of labeled examples to
learn concepts. For instance, the dermatology experiments used 50 images to
learn each concept. The data used to learn concepts can come from datasets
different from the ones used to train the model. In the dermatology case
study, we learned concepts using the ISIC dataset (for skin hair) and still
correctly explained the mistakes of a model trained on the Fitzpatrick17k
data. This makes the CCE method more broadly useful. The quality of the
concept bank is a crucial component for the outcome of the explanations, as
demonstrated in AppendixA.8.1. It is important to seek guidance from the
experts of the problems when building concept libraries to obtain the best
results. Incorporating automatic concept learning(Ghorbani et al., 2019) is a
fruitful future direction, as it could further simplify the entire pipeline.

There are several areas where CCE can be extended. While we focused in this
paper on image classification tasks, CCE can be applied to other data
modalities and tasks such as text, audio, video data, regression and
segmentation. Finally, we seek to do user studies to understand how human
subjects respond to explanations with CCE and how it drives improvements in
model debiasing and robustness.

## Acknowledgements

We would like to thank Dr. Roxana Daneshjou for her feedback on the
dermatology experiments; Amirata Ghorbani for providing the initial version of
the Broden Concepts dataset; Eric Wu for his suggestions on the Cardiology
experiments; Kyle Swanson for giving valuable feedback on the manuscript;
Roxana Daneshjou, Zach Izzo, Bryan He, Kyle Swanson, Kailas Vodrahalli, Eric
Wu, and Kevin Wu for fruitful discussions on the project. This research is
supported by funding from NSF CAREER, Sloan Fellowship, and the Chan-
Zuckerberg Biohub.

## References

  * Abid et al. (2019) Abid, A., Abdalla, A., Abid, A., Khan, D., Alfozan, A., and Zou, J.  Gradio: Hassle-free sharing and testing of ml models in the wild.  _arXiv preprint arXiv:1906.02569_ , 2019.
  * Adebayo & Kagal (2016) Adebayo, J. and Kagal, L.  Iterative orthogonal feature projection for diagnosing bias in black-box models.  _arXiv preprint arXiv:1611.04967_ , 2016.
  * Adebayo et al. (2018) Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I. J., Hardt, M., and Kim, B.  Sanity checks for saliency maps.  In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , 2018.  URL https://proceedings.neurips.cc/paper/2018/hash/294a8ed24b1ad22ec2e7efea049b8737-Abstract.html.
  * Akula et al. (2020) Akula, A., Wang, S., and Zhu, S.-C.  Cocox: Generating conceptual and counterfactual explanations via fault-lines.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 2594–2601, 2020.
  * Alqaraawi et al. (2020) Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., and Bianchi-Berthouze, N.  Evaluating saliency map explanations for convolutional neural networks: a user study.  _Proceedings of the 25th International Conference on Intelligent User Interfaces_ , 2020.
  * Chattopadhay et al. (2018) Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V. N.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pp. 839–847, 2018.  doi: 10.1109/WACV.2018.00097.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * d’Eon et al. (2021) d’Eon, G., d’Eon, J., Wright, J. R., and Leyton-Brown, K.  The spotlight: A general method for discovering systematic errors in deep learning models.  _arXiv preprint arXiv:2107.00758_ , 2021.
  * Dhurandhar et al. (2018) Dhurandhar, A., Chen, P.-Y., Luss, R., Tu, C.-C., Ting, P., Shanmugam, K., and Das, P.  Explanations based on the missing: Towards contrastive explanations with pertinent negatives.  _arXiv preprint arXiv:1802.07623_ , 2018.
  * Fitzpatrick (1988) Fitzpatrick, T. B.  The Validity and Practicality of Sun-Reactive Skin Types I Through VI.  _Archives of Dermatology_ , 124(6):869–871, 06 1988.  ISSN 0003-987X.  doi: 10.1001/archderm.1988.01670060015008.  URL https://doi.org/10.1001/archderm.1988.01670060015008.
  * Fong & Vedaldi (2018) Fong, R. and Vedaldi, A.  Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8730–8738, 2018.
  * Ghorbani et al. (2019) Ghorbani, A., Wexler, J., Zou, J., and Kim, B.  Towards automatic concept-based explanations.  _arXiv preprint arXiv:1902.03129_ , 2019.
  * Goyal et al. (2019) Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., and Lee, S.  Counterfactual visual explanations.  In Chaudhuri, K. and Salakhutdinov, R. (eds.), _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 2376–2384. PMLR, 2019.  URL http://proceedings.mlr.press/v97/goyal19a.html.
  * Groh et al. (2021) Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., and Badri, O.  Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset.  _arXiv preprint arXiv:2104.09957_ , 2021.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hendrycks & Gimpel (2016) Hendrycks, D. and Gimpel, K.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_ , 2016.
  * Huang et al. (2017) Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4700–4708, 2017.
  * Iandola et al. (2016) Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., and Keutzer, K.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size.  _arXiv preprint arXiv:1602.07360_ , 2016.
  * Irvin et al. (2019) Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pp. 590–597, 2019.
  * Kiela et al. (2021) Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al.  Dynabench: Rethinking benchmarking in nlp.  _arXiv preprint arXiv:2104.14337_ , 2021.
  * Kim et al. (2018) Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , pp. 2668–2677. PMLR, 2018.
  * Kim et al. (2019) Kim, M. P., Ghorbani, A., and Zou, J.  Multiaccuracy: Black-box post-processing for fairness in classification.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pp. 247–254, 2019.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Koh & Liang (2017) Koh, P. W. and Liang, P.  Understanding black-box predictions via influence functions.  In _International Conference on Machine Learning_ , pp. 1885–1894. PMLR, 2017.
  * Koh et al. (2020) Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al.  Wilds: A benchmark of in-the-wild distribution shifts.  _arXiv preprint arXiv:2012.07421_ , 2020.
  * Larrazabal et al. (2020) Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., and Ferrante, E.  Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis.  _Proceedings of the National Academy of Sciences_ , 117(23):12592–12594, 2020.
  * Liang & Zou (2021) Liang, W. and Zou, J.  Metadataset: A dataset of datasets for evaluating distribution shifts and training conflicts.  _arXiv preprint_ , 2021.
  * Lipton et al. (2018) Lipton, Z., Wang, Y.-X., and Smola, A.  Detecting and correcting for label shift with black box predictors.  In _International conference on machine learning_ , pp. 3122–3130. PMLR, 2018.
  * Lu et al. (2018) Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., and Zhang, G.  Learning under concept drift: A review.  _IEEE Transactions on Knowledge and Data Engineering_ , 31(12):2346–2363, 2018.
  * Mahajan et al. (2019) Mahajan, D., Tan, C., and Sharma, A.  Preserving causal constraints in counterfactual explanations for machine learning classifiers.  _arXiv preprint arXiv:1912.03277_ , 2019.
  * Michaelis et al. (2019) Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A. S., Bethge, M., and Brendel, W.  Benchmarking robustness in object detection: Autonomous driving when winter is coming.  _arXiv preprint arXiv:1907.07484_ , 2019.
  * Nestor et al. (2019) Nestor, B., McDermott, M. B., Boag, W., Berner, G., Naumann, T., Hughes, M. C., Goldenberg, A., and Ghassemi, M.  Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks.  In _Machine Learning for Healthcare Conference_ , pp. 381–405. PMLR, 2019.
  * Okur & Turkan (2018) Okur, E. and Turkan, M.  A survey on automated melanoma detection.  _Engineering Applications of Artificial Intelligence_ , 73:50–67, 2018.
  * Raghunathan et al. (2018) Raghunathan, A., Steinhardt, J., and Liang, P.  Semidefinite relaxations for certifying robustness to adversarial examples.  _arXiv preprint arXiv:1811.01057_ , 2018.
  * Rotemberg et al. (2021) Rotemberg, V., Kurtansky, N., Betz-Stablein, B., Caffery, L., Chousakos, E., Codella, N., Combalia, M., Dusza, S., Guitera, P., Gutman, D., et al.  A patient-centric dataset of images and metadata for identifying melanomas using clinical context.  _Scientific data_ , 8(1):1–8, 2021.
  * Seyyed-Kalantari et al. (2020) Seyyed-Kalantari, L., Liu, G., McDermott, M., Chen, I. Y., and Ghassemi, M.  Chexclusion: Fairness gaps in deep chest x-ray classifiers.  In _BIOCOMPUTING 2021: Proceedings of the Pacific Symposium_ , pp. 232–243. World Scientific, 2020.
  * Simonyan et al. (2013) Simonyan, K., Vedaldi, A., and Zisserman, A.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Singh et al. (2020) Singh, K. K., Mahajan, D. K., Grauman, K., Lee, Y. J., Feiszli, M., and Ghadiyaram, D.  Don’t judge an object by its context: Learning to overcome contextual bias.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 11067–11075, 2020.
  * Singla et al. (2020) Singla, S., Pollack, B., Chen, J., and Batmanghelich, K.  Explanation by progressive exaggeration.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=H1xFWgrFPS.
  * Subbaswamy et al. (2019) Subbaswamy, A., Schulam, P., and Saria, S.  Preventing failures due to dataset shift: Learning predictive models that transport.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pp. 3118–3127. PMLR, 2019.
  * Verma et al. (2020) Verma, S., Dickerson, J., and Hines, K.  Counterfactual explanations for machine learning: A review.  _arXiv preprint arXiv:2010.10596_ , 2020.
  * Wachter et al. (2017) Wachter, S., Mittelstadt, B., and Russell, C.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.  _Harv. JL & Tech._, 31:841, 2017.
  * Wang et al. (2017) Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R.  Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _IEEE CVPR_ , volume 7, 2017.
  * Wexler et al. (2020) Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viégas, F., and Wilson, J.  The what-if tool: Interactive probing of machine learning models.  _IEEE Transactions on Visualization and Computer Graphics_ , 26(1):56–65, 2020.  doi: 10.1109/TVCG.2019.2934619.
  * Wu et al. (2021) Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., and Zou, J.  How medical ai devices are evaluated: limitations and recommendations from an analysis of fda approvals.  _Nature Medicine_ , 27(4):582–584, 2021.
  * Zhou et al. (2016) Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2921–2929, 2016.
  * Zhou et al. (2018) Zhou, B., Sun, Y., Bau, D., and Torralba, A.  Interpretable basis decomposition for visual explanation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 119–134, 2018.

##  Appendix A Appendix

###  A.1 Learning Concepts

Here we provide implementational details on how we learn the concept
activation vectors (CAVs). Concretely, we pick a bottleneck layer in our
model, which is the representation space in which we will learn our features.
Unless otherwise stated, we choose the penultimate layer in ResNet18 for
experiments in this paper. Let $m$ be the dimensionality of the bottleneck
layer $L$, and $b_{L}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}$ be the “bottom”
of the network, which maps samples from the input space to the bottleneck
layer, and $t_{L}:\mathbb{R}^{m}\rightarrow\mathbb{R}^{k}$ be the “top” of the
network, defined analogously. In generating conceptual explanations, the first
step is to define a concept library: a set of human-interpretable concepts
$C=\\{c_{1},c_{2},...\\}$ that occur in the dataset. For each concept, we
collect positive examples, $P_{c_{i}}$, that exhibit the concept, as well as
negative examples, $N_{c_{i}}$, in which the concept is absent. Then, we train
a support vector machine to classify $\\{b_{L}(\bm{x}):\bm{x}\in P_{c_{i}}\\}$
from $\\{b_{L}(\bm{x}):\bm{x}\in N_{c_{i}}\\}$, the same way as in Kim et al.
(2018). Overall pseudocode for the procedure can be found in Figure 2.

Algorithm 2 Learning concept vectors

Inputs:
f  trained network: model
L bottleneck layer (hyperparameter): int
concepts set of concepts: set[str]
P # positive examples per concept: dict[str, list[sample]]
N # negative examples per concept: dict[str, list[sample]]

Output: svms #Set of SVMs containing concept predictors.

b, t = f.layers[:L], f.layers[L:]  # Divide network $f(\cdot)$ into a bottom
$b$ (first $l$ layers) and top $t$ (remaining layers) so that
$f(\cdot)=t(b(\cdot))$

for c in concepts: # Per concept, learn an SVM to classify bottleneck
representations of positive and negative examples.

svms[c] = svm.train(b(P[c]), b(N[c])) # Filter out concepts that are not
learned well

if svms[c].acc < .7:

del svms[c]

###  A.2 MetaDataset Experiments

In Table 2, we provide results over 20 scenarios. In each of the scenarios, we
fine-tune only the classification layer of a ResNet18(He et al., 2016)
pretrained on ImageNet, which outputs a probability distribution over 5
animals(cat, dog, bear, bird, elephant). For instance, in the case of the
dog(snow) experiment, we train the model with images of dogs that are all
taken with snow. We replicate this experiment with 20 different animal &
concept combinations and report the results below.

In Table 3, for each scenario, we sample 50 images with and without the
concept and we report the accuracy over those images. We observe a start
difference between accuracies, which verifies that model learns to rely on the
correlation.

Experiment | CCE-Prec.@3 | CCE(Univariate)-Prec.@3 | CCE(Control)-Prec.@3
---|---|---|---
dog(chair) | 0.980 | 0.360 | 0
cat(cabinet) | 1 | 1 | 0.180
dog(snow) | 1 | 1 | 0
dog(car) | 1 | 1 | 0
dog(horse) | 1 | 1 | 0.500
bird(water) | 0.960 | 1 | 0.660
dog(water) | 0.980 | 0.980 | 0
dog(fence) | 1 | 0.980 | 0
elephant(building) | 1 | 0.821 | 0
cat(keyboard) | 0.860 | 0.800 | 0
dog(sand) | 0.760 | 0.900 | 0
cat(computer) | 0.980 | 1 | 0
dog(bed) | 1 | 1 | 0
cat(bed) | 0.980 | 1 | 0
cat(book) | 0.960 | 1 | 0
dog(grass) | 0.700 | 0.780 | 0
cat(mirror) | 0.900 | 0.900 | 0
bird(sand) | 0.960 | 1 | 0
bear(chair) | 0.940 | 0.720 | 0
cat(grass) | 0.940 | 0.900 | 0
Table 2: Empirical evaluation for detecting spurious correlations in the training data. We report results over 20 scenarios, where each class is associated with the concept in parenthesis during the training phase. Experiment | Accuracy for Images with the Concept | Accuracy for Images Without the Concept
---|---|---
dog(chair) | 0.74 | 0.54
cat(cabinet) | 0.8 | 0.62
dog(snow) | 0.76 | 0.12
dog(car) | 0.86 | 0.66
dog(horse) | 0.70 | 0.36
bird(water) | 0.78 | 0.42
dog(water) | 0.82 | 0.4
dog(fence) | 0.74 | 0.52
elephant(building) | 0.82 | 0.72
cat(keyboard) | 0.96 | 0.46
dog(sand) | 0.66 | 0.42
cat(computer) | 1.0 | 0.68
dog(bed) | 0.86 | 0.46
cat(bed) | 0.84 | 0.54
cat(book) | 0.9 | 0.64
dog(grass) | 0.96 | 0.56
cat(mirror) | 0.86 | 0.4
bird(sand) | 0.78 | 0.66
bear(chair) | 0.82 | 0.44
cat(grass) | 0.72 | 0.46
Table 3: Accuracy of the model for images tested with and without the
confounding variable. Figure 5: Different reasons of model mistakes for the
same class..

###  A.3 Baselines

####  A.3.1 Conceptual Sensitivity Score (Kim et al., 2018)

We follow (Kim et al., 2018) and define Conceptual Sensitivity Score (CSS) as
a baseline. Particularly, for a given sample ${\bm{x}}$ and concept vector
${\bm{c}}$, CSS is defined as

$\displaystyle S_{{\bm{c}},L}(\bm{x})$ $\displaystyle=$
$\displaystyle\lim\limits_{\epsilon\rightarrow
0}\frac{t_{L}(b_{L}({\bm{x}})+\epsilon{\bm{c}})-t_{L}(b_{L}(\bm{x}))}{\epsilon}$
(7) $\displaystyle=$ $\displaystyle\nabla
t_{L}(b_{L}(\bm{x}))\cdot{\bm{c}},\vspace*{-2ex}$

Namely, it is the derivative at ${\bm{x}}$ in the direction of ${\bm{c}}$.
This is a measure of sensitivity of the model prediction with respect to the
concept vector ${\bm{c}}$.

####  A.3.2 CocoX (Akula et al., 2020)

The original CoCoX repository does not contain the implementation of the
algorithm. We followed the paper to match their description. Further, their
manuscript does not contain the hyperparameters they used for the algorithm.
The implementation that reasonably worked for us is an instantiation of their
algorithm via a) A continuous relaxation of their integer programming
formulation b) Replacing FISTA with an SGD solver (the same as our method) c)
Not separating concepts into sets(in their paper, they choose a specific set
of concepts for each class, then only use the concepts for the predicted class
and the concepts for the ground truth label which we cannot assume).

###  A.4 Validity Examples

In Fig. 6, we provide additional examples on where validity improves
explanations. In a) we have a model trained with the dog-snow spurious
correlation and in b) we have the dog class associated with both the horse and
bed concepts. For both of these examples, we observe that in the images where
the proposed concept already exists, validity constraints help prevent the
model from explaining the mistake by adding that particular concept, which
results in valid explanations.

Figure 6: Validity constraint improves explanations. .

###  A.5 Dermatology Experiment with Fitzpatrick17k

We directly follow the experimental protocol in (Groh et al., 2021).
Fitzpatrick17k (Groh et al., 2021) is a dermatology dataset that contains
16,577 skin images with 114 different skin conditions. For these images, skin
type labels are provided using the Fitzpatrick system (Fitzpatrick, 1988).
(Groh et al., 2021) decomposes the dataset into three groups: The lightest
skin types (1 & 2) with 7,755 images, the middle skin types (3 & 4) with 6,089
images, the darkest skin types (5 & 6) with 2,168 images. We randomly
partition the dataset into training(80%) and testing(20%) sets. We use a
ResNet18 (He et al., 2016) backbone pretrained on ImageNet(Deng et al., 2009)
and we fine-tune a classification head to predict one of the 114 skin
conditions using Adam(Kingma & Ba, 2014) optimizer. The classification head
consists of 1) a fully-connected layer with 256 hidden units 2) relu
activation 3) dropout layer with a 40% probability of masking activations 4)
another fully-connected layer with the number of predicted categories. We
obtain the training script from the
repository111https://github.com/mattgroh/fitzpatrick17k for the paper (Groh et
al., 2021).

We are using the
imagecorruptions222https://github.com/bethgelab/imagecorruptions(Michaelis et
al., 2019) library to obtain positive samples of defocus blur(3), zoom
blur(2), brightness(3), motion blur(3), and contrast(2), where we used the
severity levels provided in parenthesis. Examples of positive images for these
concepts can be found in Fig. 7.

We learn the zoom concept by cropping the height and width of images to a
fourth of their sizes. We obtain positive images from the darkest skin types
(5&6) and negative images from the lightest skin types (1&2) to learn the dark
skin color concept. For all dermatology concepts except the skin hair concept,
we use 25 positive & negative pairs of images. For the skin hair concept, we
use 10 pairs of images from the ISIC dataset (Rotemberg et al., 2021).

####  A.5.1 Explanation Artifacts

It is important to note that sometimes concepts such as water or ashcan may be
suggested as explanations by CCE. Concepts are represented as vectors in the
embedding space, and there may be similarities among them. Hence, water or
ashcan concepts may be very similar to certain textures that are relevant to
the dermatology task, where it was not possible to obtain the true concept
itself. The practitioner using CCE to explain the mistake can easily discard
these artifacts suggested by CCE, where there is also concepts that are more
relevant and obtained larger scores. Secondly, as the user keeps using
concept-based methods, they can understand what type of concepts is needed, or
is relevant to the task. Accordingly, it is possible to update and obtain a
more informative bank. Generally, there are various interesting directions
that could be pursued in the human-machine interface that would improve the
real-life performance of explanation algorithms. We are also interested in
further understanding this, and think more about the deployment aspects of CCE
in future work.

Figure 7: Learning image quality concepts. Here we have the positive samples
for different image quality concepts. For each corruption type, we provide
examples for different levels of severity. The original image is obtained from
the Fitzpatrick dataset.

####  A.5.2 Testing GradCAM++ in the Fitzpatrick Setting

As we mention in Section 4.4, the allergic contact dermatitis condition has a
biased skin color distribution in the training data. Activation-map-based
methods such as GradCAM++(Chattopadhay et al., 2018) fail to communicate this
to the user. However, CCE can correctly identify this as a reason why the
model fails. In Figure 8 we show two examples where CCE is able to identify
the bias in the data that drives the model mistake; however, one of the
methods that are very commonly used to understand model predictions,
GradCAM++, fails to identify this reason.

Figure 8: CAM based methods fail to communicate model biases. Here we show two
examples where CCE is able to identify the bias in the data that drives the
model mistake. However, one of the methods that are very commonly used to
understand model predictions, GradCam++, fails to identify and communicate the
underlying reason.

###  A.6 Cardiology Experiment with Chest X-Rays

We investigate a cross-site evaluation setting, where models trained on Chest
X-Ray images are used to classify the pneumothorax condition which was
proposed in (Wu et al., 2021). Using pretrained models from (Wu et al., 2021),
we use a DenseNet-121(Huang et al., 2017) architecture pretrained on
ImageNet(Deng et al., 2009) and then fine-tuned on the NIH dataset(Wang et
al., 2017). As it is reported in their paper, this model achieves $0.779$ mean
AUC when tested on the (Irvin et al., 2019) dataset and $0.903$ mean AUC when
tested on the NIH dataset. We use CCE to explain mistake over a subset of the
SHC dataset. Specifically, we run CCE over the images taken from the lateral
view, which does not exist in the training dataset since NIH only contains
images from the frontal (Anterior-Posterior(AP) or Posterior-Anterior(PA))
view positions. This constitutes a real-life scenario of a distribution shift.
Some examples of these images can be seen in Figure 9. We sample 150 images
that were taken from the lateral view where the model makes a mistakes. For
$\mathbf{150/150}$ of those images, CCE identifies the lateral view concept as
the concept with the lowest score. Namely, CCE claims that removing the
lateral viewness concept would increase the probability of correctly
classifying the image.

Figure 9: Chest X-Ray images from different views. Here we provide several
lateral view and frontal view images from the SHC dataset. On the left, we see
images from the lateral view and on the right we have images from the frontal
view. NIH dataset does not have any lateral view images in the dataset, which
can explain why a model trained on the NIH dataset performs poorly when tested
on lateral view images.

###  A.7 Additional Examples Validating CCE through Low-Level Image
Perturbations

In Section 4.2, we report a case where CCE reveals low-level artifacts learned
by a SqueezeNet(Iandola et al., 2016) pretrained on ImageNet. Particularly,
the ImagetNet dataset on which SqueezeNet was trained includes a class of
green apples known as Granny Smith. These images were always colored in
ImageNet, meaning that the model misclassifies images of these apples in
grayscale. We can use this fact to gradually transform a natural image of a
Granny Smith apple, blending it with its grayscale version. At different
levels of the original image vs. its transformed version, we run it through
SqueezeNet to obtain a prediction and then calculate its CCE scores. The
results, shown in Fig. 3(a), show that as the image is grayed, the probability
of it being classified as Granny Smith decreases, while the CCE for greenness
increases. We repeat this experiment with 25 images of Granny Smith apples and
provide results in Appendix Fig. 10(b). Our results show that CCE is also
effective at explaining a model’s mistakes in terms of low-level visual
artifacts.

In Fig. 3, we showed an example in which CCE was successfully able to identify
why images that are being perturbed through low-level image transformations
are being misclassified. Here, we show another pair of examples, with a
different concept: redness (Fig. 11). Additionally, in Fig. 10 we show the
aggregated curve over 25 granny smith apple images.

Figure 10: Validating CCE Through Low-Level Image Perturbations: (a) Here, we
take an arbitrary test sample of a Granny Smith apple that is originally
correctly classified and perturb the image by turning it gray until it is
eventually misclassified (in this case, as mortar). We compute the CCE scores
at each perturbed input image and observe that the score for greenness
increases, corresponding to the degree to which we remove the green color from
the image.  (b) We repeat this for 25 different images of Granny Smith apples
(some of which are shown under the plot) and find that the same trends
generally hold true (each image is a gray line). Although a few images do not
follow this trend, the mean CCE score (bolded green line) does. Figure 11: In
an analogous manner to Fig. 3, we take images that were originally correctly
classified as robin and strawberry, and perturb them by removing red colors
until they are misclassified by the model (x-axis). The CCE scores correctly
identify the concept of _redness_ as the most important concept for correcting
the model’s mistakes.

###  A.8 Challenging scenarios for CCE

####  A.8.1 When the target concept is missing from the bank

What happens when the spuriously correlated concept is not in our concept
bank? In this section, we replicate our controlled experiments in Metadataset
by omitting the target concept from our concept bank. For instance, for the
dog(snow) experiment, we remove the concept snow from the set of 168 concepts
that we have and re-run our analysis.

In Table 4, we provide the Top-5 concepts suggested by CCE. We use 50 mistakes
for each scenario, evaluate the ranks for each concept in our concept bank
using CCE, and average the ranks over all the samples that we have. We report
the Top-5 concepts with the highest rank. In almost all of the scenarios, CCE
identifies concepts that are ‘hinting‘ to the target concepts. For example, in
the dog(bed) case, CCE reports Sofa, Dog, Bedclothes, Muzzle, Headboard as the
reasons for the mistake, which are either related to the target class or the
spuriously correlated concept. Generally, we see a similar pattern in all of
our experiments. This provides another piece of evidence that highlights the
importance of the richness of the concept bank. If we have a rich-enough bank,
then CCE helps us identify the spuriously correlated concept even if it is not
directly in the bank.

Experiment | Top5 Concepts
---|---
dog(bed) | Sofa, Dog, Bedclothes, Muzzle, Headboard
dog(chair) | Sofa, Book, Hand, Bedclothes, Dog
cat(cabinet) | Inside arm, Microwave, Chest of drawers, Door frame, Oven
dog(snow) | Mountain, Car, Dog, Minibike, Headlight
dog(car) | Bus, Headlight, Motorbike, Fence, Coach
dog(horse) | Cow, Muzzle, Motorbike, Bicycle, Grass
bird(water) | Bird, Airplane, Coach, Sand, Mountain
dog(water) | Muzzle, Dog, Sand, Airplane, Mountain
dog(fence) | Field, Grass, Coach, Muzzle, Dog
elephant(building) | Horse, Cow, Pedestal, Car, Bucket
cat(keyboard) | Cat, Computer, Inside arm, Paper, Faucet
dog(sand) | Water, Airplane, Horse, Blueness, Mountain
cat(computer) | Faucet, Keyboard, Cat, Inside arm, Microwave
cat(bed) | Cat, Pillow, Inside arm, Headboard, Back pillow
cat(book) | Computer, Cat, Bookcase, Oven, Chest of drawers
dog(grass) | Horse, Field, Muzzle, Bus, Motorbike
cat(mirror) | Inside arm, Headlight, Door frame, Countertop, Bathtub
bird(sand) | Water, Airplane, Bird, Mountain, Blind
bear(chair) | Food, Candlestick, Plate, Lamp, Fabric
cat(grass) | Tree, Mouth, Field, Path, Blind
Table 4: CCE suggestions when the target concept is missing from the concept
bank. We average the ranks for each concept over 50 mistakes in each
experiment, and report the Top-5 concepts with the highest rank.

####  A.8.2 When the correlations are less drastic

What happens when the correlations are less drastic? More concretely, in
Section 4.1, we assumed all images in the training dataset contains the
spuriously correlated concept, we call this situation 100% severity in this
section. Here, we evaluate when the correlation is less severe. Particularly,
what happens when we have a scenario with $50\%$ severity, i.e. when only
$50\%$ of the training images contain the spuriously correlated concept? In
Figure 12a & b, we report the CCE performance as we vary the severity. The
bolded line shows the median performance across 20 scenarios, and the
confidence levels show the first and third quantiles. We observe that starting
from relatively low levels of severity($\approx 20\%$), CCE is able to
identify the spuriously correlated concept in a majority of the scenarios.
Consequently, we see that CCE can perform well even in less-severe, and thus
more realistic scenarios.

Figure 12: (a, b) We report the performance of CCE as we vary the severity
level of the correlation.(K% severity means K% of the images in the training
dataset has the concept). The bolded line shows the median performance across
20 scenarios, and the confidence levels show the first and third quantiles. We
observe that starting from relatively low levels of severity($\approx 20\%$),
CCE is able to identify the spuriously correlated concept in a majority of the
scenarios. (c) We evaluate Batch Mode CCE over controlled experiments with
varying severity levels. The y-axis corresponds to the rank of the target
concepts, the x-axis gives the severity of the spurious correlation, the bold
line is the median performance across 20 scenarios and the confidence
intervals give the lower and upper quartiles. We observe that Batch Mode CCE
is able to provide an automated analysis of the model biases using a batch of
samples, given that it can identify the spuriously correlated concept as one
of the major causes over all model mistakes for the given class.

####  A.8.3 Batch evaluation for a set of mistakes

Here we extend CCE to a batch evaluation setting. Namely, instead of
explaining an individual mistake, we aim to explain a batch of mistakes using
conceptual counterfactuals. We do so by making a slight modification to our
optimization procedure:

$\displaystyle\min_{\bm{w}}$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}{{\color[rgb]{126,0,24}\mathcal{L_{\textrm{CE}}}(y_{i},t_{L}(\mathbf{b}_{L}(\bm{x}_{i})+\bm{w}\tilde{\bm{C}}))}}+{\color[rgb]{255,172,59}\alpha|\bm{w}|_{1}+\beta|\bm{w}|_{2}}$
(8) s.t.
$\displaystyle{\color[rgb]{69,2,112}\bm{w}^{\min}\leq\bm{w}\leq\bm{w}^{\max}}$

The optimization problem in Equation 8 outputs a single shared set of concept
scores that minimizes the proposed loss for a batch of mistakes. Furthermore,
we let $\bm{w}^{\min}=\frac{1}{N}\sum_{i=1}^{N}\bm{w}_{i}^{\min}$ and
$\bm{w}^{\max}=\frac{1}{N}\sum_{i=1}^{N}\bm{w}_{i}^{\max}$ to make the
validity constraints work in the batch-wise setting where $\bm{w}_{i}^{\max}$
and $\bm{w}_{i}^{\min}$ denotes the validity bounds for the sample $i$.

This formulation results in a more ‘holistic‘ understanding of the model bias
compared to the sample-by-sample analysis. In practice, using all of our
mistakes in the test dataset, we could run Batch-CCE analysis to provide a
global interpretation of the model biases. In Figure 12(c), we provide the
performance of Batch Mode CCE over 20 scenarios across different severity
levels. The y-axis corresponds to the rank of the target concepts, the x-axis
gives the severity of the spurious correlation, the bold line is the median
performance across 20 scenarios and the confidence intervals give the lower
and upper quartiles. We observe that Batch Mode CCE is able to provide an
automated analysis of the model biases using a batch of samples, given that it
can identify the spuriously correlated concept as one of the major causes over
all model mistakes for the given class.

Figure 13: (a) We observe how the performance changes with respect to K for
the metric Precision@K. (b) We provide the distribution of the validation
accuracies for each concept, as we vary the number of samples we use to learn
the Concept Activation Vectors.

###  A.9 Details on Precision@K

In Fig 13(a), we observe how the reported Precision@K metric changes for
various values of K. For $K<3$ the performance is relatively lower.
Empirically, we observe that this is mostly due to co-occuring concepts, e.g.
for the Dog(Snow) case, sometimes the first two concepts can turn out to be
muzzle or dog. Similarly, for the Dog(Car) case, concepts like Bus or Truck
can turn out to be the Top-2 concepts in the evaluation. We choose $K=3$ since
it is less noisy due to the explained reasons, and also more comprehensible in
terms of providing the user a small number of concepts that they can easily
work with.

##  Appendix B Concepts

In Table 5, we list the 170 concepts that we considered, along with their
validation accuracies. We kept 168 concepts which had a validation accuracy of
0.7 or greater (bolded). We choose the threshold 0.7 since it works well
empirically. For a more detailed analysis, in Fig 13(b) we provide the
distribution of validation accuracies for concepts, as we vary the number of
samples we use to learn the Concept Activation Vectors. As we increase the
number of samples, the validation accuracies move beyond the threshold we use
($0.7$). When we use $100$ samples per concept, most of the concepts move
beyond this value.

loudspeaker(0.86) | microwave(0.82) | arm(0.86) | exhaust hood(0.92)
---|---|---|---
airplane(0.96) | pedestal(0.78) | back(0.72) | mouse(0.68)
glass(0.84) | polka dots(1.00) | mouth(0.80) | keyboard(0.81)
inside arm(0.76) | bird(0.94) | bedclothes(0.82) | paper(0.82)
blind(0.86) | brick(0.84) | stairs(0.86) | countertop(0.92)
base(0.81) | person(0.94) | blueness(0.84) | bathroom s(0.84)
pane(0.96) | motorbike(1.00) | hair(0.84) | paw(0.84)
candlestick(0.88) | outside arm(0.92) | ceiling(0.88) | light(0.88)
street s(0.87) | column(0.82) | door frame(0.96) | granite(0.90)
cow(0.96) | sand(0.94) | bottle(0.84) | cup(0.88)
plate(0.98) | double door(0.92) | pillow(0.78) | plant(0.88)
doorframe(0.86) | eyebrow(0.88) | flower(0.90) | horse(0.98)
toilet(0.90) | ceramic(0.86) | greenness(0.90) | back pillow(0.86)
drawer(0.86) | coach(0.92) | metal(0.84) | lid(0.90)
bannister(0.86) | handle bar(0.78) | fan(0.79) | bush(0.92)
blotchy(0.97) | fireplace(0.96) | bowl(0.80) | nose(0.80)
leg(0.80) | door(0.82) | stripes(0.91) | apron(0.72)
oven(0.80) | pack(0.84) | body(0.86) | foot(0.80)
frame(0.74) | dining room s(0.88) | board(0.78) | bridge(0.82)
sofa(0.78) | bedroom s(0.86) | head(0.92) | blurriness(0.95)
footboard(0.80) | leather(0.86) | hand(0.90) | fluorescent(0.83)
tree(0.98) | knob(0.89) | headlight(0.91) | blackness(0.91)
house(0.82) | jar(0.94) | mirror(0.94) | pipe(0.82)
bathtub(0.96) | flag(0.70) | refrigerator(0.82) | curtain(0.92)
book(0.80) | coffee table(0.92) | field(0.90) | chandelier(0.94)
cap(0.82) | hill(0.92) | ashcan(0.88) | path(0.94)
counter(0.82) | cardboard(0.88) | desk(0.84) | balcony(0.88)
box(0.72) | napkin(0.79) | ear(0.94) | food(0.86)
manhole(0.82) | chest of drawers(0.88) | fence(0.82) | building(0.92)
figurine(0.77) | lamp(0.96) | basket(0.80) | eye(0.90)
ground(0.86) | car(0.90) | cat(1.00) | bookcase(0.94)
palm(0.96) | water(0.98) | bus(0.96) | laminate(0.82)
handle(0.78) | painted(0.88) | dog(0.98) | bumper(0.73)
concrete(0.82) | awning(0.66) | clock(0.90) | faucet(0.89)
headboard(0.90) | canopy(0.92) | bucket(0.74) | drinking glass(0.72)
armchair(0.84) | minibike(0.96) | carpet(0.78) | cabinet(0.76)
bed(0.77) | earth(0.88) | neck(0.84) | can(0.89)
bicycle(0.96) | bag(0.70) | chain wheel(0.89) | beak(0.72)
mountain(0.94) | redness(0.96) | air conditioner(0.85) | chair(0.86)
engine(0.90) | painting(0.80) | grass(0.92) | snow(0.84)
pillar(0.84) | chimney(0.73) | floor(0.78) | fabric(0.74)
computer(0.86) | flowerpot(0.78) | muzzle(0.94) | bench(0.72)
ottoman(0.80) | cushion(0.84) |  |
Table 5: List of concepts and validation accuracies for SVMs, for a ResNet18
pretrained on ImageNet.