# You Only Explain Once

David A. Kelly, Hana Chockler, Nathan Blake, Aditi Ramaswamy,
Melane Navaratnarajah, Aaditya Shivakumar
Department of Informatics
King’s College London
david.a.kelly, hana.chockler, nathan.blake, aditi.ramaswamy,
melane.navarathnarajah, aaditya.shivakumar AT kcl.ac.uk Daniel Kroening
AWS
<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a new black-box explainability algorithm and tool,
anon-rex, for efficient explanation of the outputs of object detectors. The
new algorithm computes explanations for all objects detected in the image
simultaneously. Hence, compared to the baseline, the new algorithm reduces the
number of queries by a factor of $10\times$ for the case of ten detected
objects. The speedup increases further with with the number of objects. Our
experimental results demonstrate that anon-rex can explain the outputs of yolo
with a negligible overhead over the running time of yolo. We also demonstrate
similar results for explaining ssd and faster r-cnn. The speedup is achieved
by avoiding backtracking by combining aggressive pruning with a causal
analysis.

##  1 Introduction

Neural networks are now a primary building block of most computer vision
systems. The opacity of neural networks creates demand for explainability
techniques, which attempt to provide insight into _why_ a particular input
yields a particular observed output. Beyond increasing a user’s confidence in
the output, as well as their trust in the AI system, these insights help to
uncover subtle recognition errors that are not detectable from the output
alone [7].

Explainability is especially important for object detection. Object detectors
are fast networks, localizing and identifying objects within a given image.
Object detectors are crucial for self-driving vehicles and advanced driver-
assistance systems, as well as many other applications, as they are fast
enough to compute results in real time. The state of the art in the field is
yolo [21], an object detector that localizes and labels objects on an input
image within one pass. Other notable object detectors include ssd [16], which
is also a one-pass object detector, and a number of two-passes object
detectors, most popular of which are the variants of r-cnn [12]. These are
slower in general, but recognize more classes of objects. While there is a
large body of work on explaining image classifiers, there is no published
research on how to explain their results, despite the prevalence of object
detection use-cases.

For image classifiers, the standard form for an explanation is a subset of
highest-ranked pixels that is sufficient for the original classification
outcome – typically a small number of contiguous areas. Figure
LABEL:fig:accept illustrates how this idea is adapted to the output of object
detectors: there must exist a separate explanation for each detected object,
contained within the bounding box computed by the object detector.

In this paper, we present anon-rex, an explainability algorithm for object
detectors such as yolo. Explaining the outputs of neural networks is a known
hard problem, and it is even harder to explain outputs of object detectors, as
they detect multiple objects in a given image and there are strong performance
constraints. We hence introduce aggressive pruning and biased partitioning of
the input, as well as the ability to construct explanations of multiple
objects in the image simultaneously in order to compute our explanations
efficiently.

As there is no existing black-box explainability tool for object detectors, we
construct a baseline tool for comparison from rex, which is an explanability
tool for image classifiers [7]. Our experimental results show that anon-rex
outperforms rex on yolo by at least an order of magnitude and is much more
scalable with respect to the number of objects detected on the image.
Furthermore, our experimental results demonstrate that anon-rex produces
significantly better explanations than the only existing white-box
explainability tool for object detectors, eigencam-yolo [18].

To demonstrate that our technique is broadly applicable beyond yolo, we also
present experimental results of anon-rex on ssd and faster r-cnn, both showing
similar levels of improvement we observe when using yolo. Full experimental
results, datasets, and the code of anon-rex are submitted as a part of the
supplementary material.

##  2 Related Work

Existing algorithms for explaining the output of image classifiers can be
roughly divided into black-box algorithms, which are agnostic to the structure
of the classifier, and white-box ones, which rely on the access to the
internals of the classifier. There are clear advantages to the black-box
tools, namely being agnostic to the internal structure and complexity of the
classifier. Yet, black-box tools are in general much less efficient than
white-box ones, as they rely on querying the classifier many times. Thus, they
cannot be used as-is to explain the outputs of object detectors, whose main
characteristic is efficiency.

There is a large body of work on algorithms for computing an explanation for a
given output of an image classifier. They can be largely grouped into two
categories: propagation and perturbation. Propagation-based explanation
methods back-propagate a model’s decision to the input layer to determine the
weight of each input feature for the decision [26, 28, 4, 25, 19]. grad-cam
only needs one backward pass and propagates the class-specific gradient into
the final convolutional layer of a DNN to coarsely highlight important regions
of an input image [23].

Perturbation-based explanation approaches introduce perturbations to the input
space directly in search for an explanation. shap (SHapley Additive
exPlanations) computes Shapley values of different parts of the input and uses
them to rank the features of the input according to their importance [17].
lime constructs a small neural network to label the original input and its
neighborhood of perturbed images and uses this network to estimate the
importance of different parts of the input [22, 10, 5, 20, 11]. Finally, rex
(previously called deepcover) ranks elements of the image according to their
importance for the classification and uses this ranking to greedily construct
a small explanation [27, 7, 8]. The deepcover ranking procedure in [27] uses
SFL, and is replaced in [7] by the approximate computation of causal
responsibility. The latest version, rex [8], computes multiple explanations
for a given input.

None of the tools mentioned above works with yolo, or any object detector,
natively. There exists a version of grad-cam for yolo [3]; unfortunately,
however, it is proprietary and not available for experiments. An open source,
white-box equivalent is eigencam-yolo [1], based on eigencam [18] and yolo v8.
The approach taken by eigencam-yolo works for some images, but often it yields
explanations that are too large, highlight the wrong sections of the image, or
differ significantly from plain yolo bounding boxes. A major disadvantage of
eigencam-yolo is its reliance on heatmaps as its main method of communicating
explanations: an image with many identifiable objects, such as the shelf of
wine bottles, becomes a red blur when run through eigencam-yolo’s explanation
algorithm, and the user obtains no further knowledge about the working of the
object detector.

##  3 Overview of rex

Causal Responsibility eXplanations (rex) [8, 7] (formerly deepcover) is an XAI
tool based on the solid foundations of causal reasoning [13]. Essentially, rex
constructs an approximation of a causal explanation [14, 13] by first ranking
the pixels of a given image according to their importance for the
classification, and then constructing an explanation greedily from the
saliency landscape of the pixels. The ranking is an approximate degree of
responsibility [6], computed on the coarsely partitioned image, and updated by
refining the areas of the input with high responsibility, until a sufficiently
fine partition is reached.

To compute the degree of responsibility, rex generates mutants – images with
some of the parts masked – and queries the classifier on these mutants. If a
non-masked area is sufficient to get the same classification as the original
image, it is a cause for the classification, with the degree of responsibility
of each part based on the number of parts in the area. For example, if a part
by itself is sufficient for the classification, the responsibility of each of
its pixels is $1$; if, however, the smallest set of parts needed to obtain the
same classification is of size $5$, then each pixel in each part of this set
has the responsibility $1/5$.

Essentially, the responsibility is in $[0,1]$, and the higher it is, the more
important a given pixel is for the classification. If a part has no influence
on the classification, the responsibility of each of its pixel is $0$, and it
is eliminated from further analysis. The process is repeated over a number of
different random partitions, and the result is averaged. The partitions are
randomly produced, but the precise nature of the random distribution is
flexible. rex supports a number of distributions natively, including discrete
uniform and beta-binomial. We use uniform for all of our experiments.

##  4 anon-rex Algorithm

anon-rex introduces a number of modifications to the original rex algorithm,
drastically reducing the number of calls to the object detector and building a
saliency landscape for multiple objects at the same time. These changes are
agnostic to the internals of the object detector, as they change the rex
algorithm only and do not depend on a particular classification or object
detection model. There is one caveat to this agnosticism in that different
object detector tools return their predictions in different formats. These
different formats need to be handled appropriately in order to provide
necessary information to anon-rex.

We present a high-level view of the algorithm in Figure 1 and its pseudocode
in Algorithm 1. Object detection tools return a set of labels and bounding
boxes for these labels. We use this initial output as a process queue, which
we update and refine during the algorithm. The algorithm maintains a copy of
the original set of predictions and bounding boxes separately from the process
queue. This allows us to refer back to the initial bounding boxes when
extracting explanations from the saliency landscape. An explanation should not
extend beyond the borders of the bounding box of the initial prediction (as we
see later in Figure 5, allowing explanations to bleed out of the original box
results in explanations which are difficult to interpret). The algorithm
averages the ranking of pixels over $k$ iterations (for a given parameter
$k$). in each iteration calculating the rank of each pixel in each bounding
box,

②generate_mutants③PruningAlgorithm _PRUNE_ causal_ranking④ _extract_ $k$
iterations of inner loop$\mathcal{E}$ Figure 1: A schematic depiction of our
algorithm, returning a set of explanations $\mathcal{E}$, one for each
bounding box in the original image (the black box stands for an object
detector). It creates masked mutants for all bounding boxes simultaneously ②,
queries the model, then the _pruning algorithm_ ③ selects mutants for further
investigation and removes all other mutants. anon-rex then generates a common
saliency landscape of pixels for all objects using the causality-based ranking
procedure. After $k$ iterations, ④ extracts the explanations from the original
bounding boxes and the saliency landscape, returning $\mathcal{E}$, the set of
explanations for all detected objects in the original image. Algorithm 1
$\mbox{{{anon}-{r}{e}{x}}}\,(\mathit{x},\mathcal{N})$

INPUT: an image $x$, an object detection model $\mathcal{N}$, number of
iterations $k$

OUTPUT: a set of explanations $\mathcal{E}$ for each bounding box in
$\mathcal{N}(x)$

1: $\mathcal{D}\leftarrow\mathcal{N}(x)$

2: $R\leftarrow$ map of $0$’s

3: for $i$ in $0\ldots k$ do

4: $\mathit{Q}\leftarrow\mathcal{D}$

5: initialize all cells of $updated$ to $-1$

6: $\mathcal{E}\leftarrow\emptyset$

7: while True do

8: $\mathds{P}\leftarrow\mathit{random\\_partitions(Q)}$

9: for all $p\in\mathds{P}$ do

10: $\mathit{m}\leftarrow\mathit{generate\\_mutant(Q,p)}$

11: $\mathit{preds}\leftarrow\mathcal{N}(m)$

12:
$(\mathit{updated},\mathit{Q})\leftarrow\mathit{prune}(preds,\mathcal{D},updated)$

13: if ($\forall u\in updated:u>=0$) then

14: break

15: else if ($\forall u\in updated:u=-1$) then

16: break

17: end if

18: end for

19: end while

20: $R\leftarrow R+\mathit{causal\\_ranking(queue,updated,\mathds{P})}$

21: end for

22: $\mathcal{E}\leftarrow\mathit{extract(R,D)}$

23: return $\mathcal{E}$

The _prune_ procedure, in combination with the _ranking_ procedure, adapted
from the one in [7], introduces _aggressive pruning_. Essentially, the
algorithm first partitions the input image into $s$ parts and then performs a
gradual refinement of the partition based on the _degree of causal
responsibility_ [6, 13] of each part. The original rex ranking procedure
computes the degree of responsibility of each part and then discards the parts
with responsibility $0$. In anon-rex, we compute the degree of responsibility
greedily, and stop at each level when we find a subset of parts whose union
(with the remainder of the image masked) elicits the original set of labels
from the object detector. All other parts of the bounding box are discarded,
and the algorithm progresses to the next refinement level.

The original rex assumes one object per image, whereas yolo can produce
multiple detections. The procedure Algorithm 2 mutates all bounding boxes
detected in the image at the same time. We add the appropriate set of parts of
each bounding box to the mutant. For example, assuming $4$ parts for each
bounding box, Algorithm 2 creates a mutant where each upper left box for every
bounding box is revealed in that mutant. We continue this for all combinations
of parts for each bounding box.

For reasons of efficiency, we group batches of mutants together. To be
concrete, we have the first group of combinations as $(0\ldots s)$, we have
each bounding box subdivided into $s$ partitions, numbered appropriately, and
we reveal partition $s_{i}$ for each bounding box to create one mutant. The
second group, if required, considers pairs of parts and so on. Grouping in
this fashion allows us to balance the benefits of batching (by sending more
mutants to the model) without doing too much unnecessary work, _i.e_.
considering combinations of partitions when we already have a complete set of
passing mutants.

To keep track of which combination of partitions satisfies the classification,
we maintain an array the same length as $\mathcal{D}$, which holds a
combination of parts that is sufficient for a classification ($-1$ means that
none were found). As soon as one such combination is found, we mask all other
parts and proceed to partition this combination further. The following claim
is straightforward from the procedure.

###### Claim 1

At any level, the non-masked parts of the image contain explanations for all
detected objects in the image.

Algorithm 2 $\mathit{generate\\_mutant(Q,p)}$

INPUT: current process queue $Q$, and a set of parts $p$ to be unmasked for
each bounding box

OUTPUT: a mutant $m$

1: $\mathit{m}\leftarrow\emptyset$

2: for all $q\in\mathit{queue}$ do

3: $\mathit{m}\,\leftarrow\mathit{m}\cup$ (unmask $p$ in bounding box of $q$)

4: end for

5: return $\mathit{m}$

There are various subtleties for which we must account. When we mask part of a
bounding box and query the model, we do not always get only one prediction in
return. For example, if we start with a top-level classification and bounding
box of “person” and mask part of that bounding box, we can get a prediction of
“person” but also a new bounding box which contains a different
classification. We ignore these new bounding boxes and only update the process
queue with the bounding box that best matches the prediction and partition
which spawned it, as shown in Algorithm 3.

Algorithm 3 $\mathit{prune}\,(\mathcal{P},\mathcal{D},\mathcal{F})$

INPUT: an array of predictions, $\mathit{preds}$, top-level target predictions
$\mathcal{D}$, a job queue _Q_ , an array $\mathit{updated}$

OUTPUT: $\mathit{updated}$, $\mathit{Q}$

1: for $\mathit{pred}$ in $\mathit{preds}$ do

2: if ($\mathit{pred}=$ original label in $\mathcal{D})$ and
$(\mathit{updated(pred)}\neq-1)$ then

3: remove all other parts

4: update location in $\mathit{updated}$

5: update $\mathit{Q}$ with $\mathit{pred}$

6: end if

7: end for

8: return $(\mathit{updated},\mathit{Q})$

After running a set number of iterations to generate the saliency landscape
(Figure 3), the explanation extraction procedure extracts explanations for all
objects by overlaying bounding boxes produced by yolo on the landscape and
adding in pixels, grouped by responsibility, under the bounding box, until the
explanations satisfy the model, as shown in Algorithm 4. We sort all unique
responsibility values in the bounding box $d$ under investigation in
descending order. We then iterate over these different degrees of
responsibility. The procedure _add_pixels_at()_ simply reveals all pixels in
the image $x$ which are inside the bounding box for $d$ with the appropriate
degree of responsibility. We query the model at each responsibility level
under the image passes the classifier. This will always succeed as the
original bounding box is a passing classification. We reach the original
bounding box when we include all pixels with responsibility $\geq 0$. This
situation occurs when the bounding box is already minimal for the yolo
classification.

Algorithm 4 $\mathit{extract}\,(R,\mathcal{D},\mathcal{N},$x$)$

INPUT: a saliency landscape $R$, image target detections $\mathcal{D}$, an
object detector $\mathcal{N}$ and an image $x$

OUTPUT: an array of explanations $\mathcal{E}$

1: $\mathcal{E}\leftarrow\emptyset$

2: for all $d\in D$ do

3: $\mathit{mask}\leftarrow\emptyset$

4: $\mathit{levels}\leftarrow\mathit{unique(R,box(d))}$ in descending order

5: for all $\mathit{level}\in\mathit{levels}$ do

6: $\mathit{mask}\leftarrow\mathit{add\\_pixels\\_at(x,level,box(d))}$

7: if $\mathcal{N}(\mathit{mask})=\mathit{class(d)}$ then

8: $\mathcal{E}\leftarrow\mathcal{E}\cup\mathit{mask}$

9: end if

10: end for

11: end for

12: return $\mathcal{E}$

In anon-rex, extracting explanations from the saliency landscape is
complicated not only by presence of bounding boxes, but also by the existence
of overlapping bounding boxes. If a smaller bounding box overlaps a larger
bounding box and has high responsibility, it will tend to be included into the
explanation for the larger box, as the larger box dominates the responsibility
landscape. This situation leads to noise in explanations for larger bounding
boxes, that is, high ranked pixels that actually contribute to a different
label. We handle this problem by treating the bounding boxes as non-
intersecting layers in the saliency landscape, hence ensuring that the
explanations are not mixed up.

###### Observation 1

Explanations found by anon-rex might not be the smallest ones.

It has been observed in the literature that most images contain multiple
explanations for their classification [8, 24]. As anon-rex finds one
explanation per object, which explanations are found depends entirely on the
order in which the algorithm computes the degree of responsibility of each
part, and hence the output might not be the smallest possible explanation. We
note that this issue does not exist in the original rex ranking procedure. It
arises in anon-rex due to the aggressive pruning, which allows us to
significantly reduce the number of queries to the object detector, as
demonstrated in Section 5. An example for such a non-minimal explanation is
shown in Figure 2, an explanation for a “person” being this person’s foot.
While a correct explanation, it seems more likely that a human would choose a
face as an explanation rather than a foot, given an entire picture of a
person.

Figure 2: An explanation produced by anon-rex for a person in Figure 1. The
person’s entire body is present in the original image.

(a) 2 iterations

(b) 4 iterations

(c) 6 iterations

(d) 8 iterations

Figure 3: The development of the pixel ranking of the image in Figure 1 over
multiple iterations, using discrete uniform for partition creation. yolo
recognizes $6$ bounding boxes in this image. We build the ranking for all
detections together, then separate them when extracting explanations.

##  5 Experiments

$0$$10$$20$$30$$0$$2{,}000$$4{,}000$No. ObjectsAverage Compute Costrexanon-rex
(a) with yolo

(PGF Math) Package PGF Math Error: Cannot process negative numbers. (in ’0 0)
(1’)See the PGF Math package documentation for explanation.(PGF Math) Package
PGF Math Error: Cannot process negative numbers. (in ’0 0) (1’)See the PGF
Math package documentation for explanation.$5$$10$$1{,}000$$2{,}000$No.
Objectsrexanon-rex (b) with ssd

$0$$20$$40$$60$$80$$0$$0.5$$1$$\cdot 10^{4}$No. Objectsrexanon-rex (c) with
faster r-cnn

Figure 4: The average number of calls (proxy for performance) produced by rex
(in red) and our tool anon-rex (in blue) using three different object
detectors on the ImageNet-mini validation dataset. The performance of our tool
is largely independent of the number of objects detected in the image. This in
contrast to the performance of rex, which decreases sharply (that is, the
number of calls increases) with the increase in the number of objects.

(a) yolo bounding boxes

(b) eigencam explanation

(c) Some of the anon-rex explanations

Figure 5: The yolo bounding boxes 5(a) and explanations as provided by
eigencam 5(b). Due to the large number of bottles, the resulting explanation
is almost entirely red. While technically correct, this is not very useful.
5(c) show a selection of anon-rex explanations. Figure 5(b) has been resized
as per the tool’s documentation.

In this section we present our experimental results for anon-rex. We compare
anon-rex with rex on the same object detectors, using the number of queries
(number of mutant images) as a proxy for performance. Both tools are evaluated
on yolo, ssd, and faster r-cnn.

| Number of Objects
---|---
| yolo | ssd |  faster
|  |  |  r-cnn
| $10$ | $20$ | $10$ | $13$ | $10$ | $20$
baseline | 2071 | 3864 | 2069 | 2338 | 530 | 4358
anon-rex | 271 | 321 | 483 | 522 | 170 | 751
Table 1: The number of queries (rounded to the nearest integer) produced by
the baseline and our tool anon-rex for explaining object detector output,
using queries as a proxy for performance. We present the results for $10$ and
for $20$ objects, except for ssd, which only recognizes up to $13$ objects.
Smaller numbers are better.

For all experiments we use the pre-trained yolov8n model [15] on the standard
validation dataset ImageNet-mini [2], consisting of $3923$ images. If yolo v8n
cannot perform any detection in an image, we exclude that image from
consideration. In our experiments, this occurred only for $210$ images.

###  5.1 Analysis of eigencam-yolo

A direct comparison with eigencam-yolo is challenging, as eigencam-yolo only
produces heatmaps. Beyond needing post-processing to extract explanations, a
heatmap is a not very useful when there is a very large number of bounding
boxes in an image. Figure 5(a) shows a yolo prediction for an image containing
many bottles. The resulting heatmap produced by eigencam-yolo is in Figure
5(b). Clearly, having almost the whole image marked as “hot” does not provide
a lot of insight into the yolo algorithm.

Our quantitative analysis of eigencam-yolo is based on a non-controversial
assumption that a good explanation should reside inside the corresponding
bounding box, and be at maximum the size of the bounding box. This is because
the bounding box contains all the information necessary for the
classification, in addition to some extraneous pixels due to the rectangular
shape of bounding boxes. In order to measure the efficiency of eigencam-yolo’s
explanations, we count the number of “hot” pixels detected by eigencam-yolo
that reside outside of the bounding boxes determined by yolo, and normalize
the count by the total number of pixels in the image. A normalized score of
$0$ indicates no superfluous pixels, while a normalized score closer to $1$
indicates that almost all of the “hot” pixels fall outside of yolo’s bounding
boxes. An example of these superfluous pixels is shown in Figure 6. This
particular image was one of the worst detected. We used the validation set
from ImageNet-mini for this analysis.

(a) yolo bounding box for a dog.

(b) The greyscale heatmap generated by eigencam-yolo outside of yolo’s
bounding box.

Figure 6: The large proportion of non-black pixels outside the bounding box in
6(b) shows that eigencam-yolo’s explanation takes up the entire image frame,
rather than fitting inside the bounding box described by yolo.

On average, the percentage of “hot” pixels detected in an image by eigencam-
yolo that fall outside of the yolo bounding boxes is $30.5\%$. In other words,
about $30\%$ of the explanation created by eigencam-yolo for a particular
image exceeds the target maximal explanation produced by yolo. On the basis of
this excessive size, we excluded eigencam-yolo from the comparative analysis.

###  5.2 Experimental results

We use rex as a baseline black-box explainability tool to which we compare the
performance of anon-rex. Recall that even black-box tools require a non-
trivial adaptation to the output format of object detectors, and to the best
of our knowledge, no such adaptations exist. In order to use basic rex with
object detectors, we introduced changes to the input format of rex without
modifying the algorithm. Specifically, for each bounding box detected, we
produced a new image in which everything but the bounding box is set to the
masking color. We then executed rex over each independent image (_i.e_., $6$
bounding boxes result in $6$ independent runs of rex). We also added code to
process the results of yolo and ssd, as they are in a different format than
the one expected by rex. Finally, yolo can produce a “no classification”
output, which rex does not expect. If yolo is unable to identify anything in
the image, we ignore it and do not call rex on these outputs.

We use the number of mutants generated by both tools as a proxy for
performance. As the internal algorithm of anon-rex takes a negligible amount
of time, this is a reasonable approximation. In fact, anon-rex is faster than
rex even on the same number of mutants, as it also makes much more aggressive
use of batching than the original rex implementation.

Figure 4 shows the average number of mutants produced as a function of the
number of objects detected in the image for the three object detectors. Beyond
demonstrating the superior performance of anon-rex, it also shows that anon-
rex performance is not affected by the number of objects detected in the
image, in contrast to rex, whose performance rapidly deteriorates with the
increase in the number of objects.

A sample of the experimental results is also presented in Table 1, with the
performance for images with $10$ objects and for $20$ objects (except for ssd,
where the maximal number of objects is $13$), compared to the baseline (rex).
The results show at least an order of magnitude speedup, with anon-rex scaling
to multiple objects with only a minimal increase in the number of queries.

###### Remark 1

The size of produced explanations is often used as a proxy for quality. In our
experiments, explanations produces by anon-rex are $20.7\%$ larger than those
produced by rex for yolo, $22.4\%$ larger for ssd, and $52.9\%$ larger for
faster r-cnn. This is partly due to the aggressive pruning, and partly due to
object detectors not recognizing small fragments of images as well as image
classifiers.

###  5.3 Explaining images with many objects

Recall that Figure 5(a) shows a yolo prediction for an image containing many
bottles. anon-rex explanations are provided in the form of subsets of pixels,
where each subset is sufficient in each bounding box to generate the same
label, though not with the same confidence. For the image of many bottles,
shown in Figure 5(a) with the yolo predictions, Figure 5(c) shows the
explanations generated by anon-rex. Of particular interest is the yellow
bottle explanation on the left. Closer inspection reveals that there are
several bottles in this explanation, not all of them yellow; indeed, on the
other yellow bottles in the image, yolo fails to classify them at all. This
may indicate that yolo is also relying partially on color and not just shape
to distinguish a bottle.

Figure 7: The anon-rex pixel ranking for the cropped version of the bottles in
Figure 5. yolo discovers new bottles in the image, but the saliency landscape
for the yellow bottles is flat, indicating that no subdivision of the bounding
box was detected as a bottle. This in contrast with the colored bottles on the
bottom row, all of which have explanations smaller than their original
bounding boxes.

We cropped the image to include only the section in which yolo did not
identify any objects and re-executed yolo and anon-rex on it. Figure 7 shows
that yolo finds the yellow bottles on the cropped image. However those have a
low confidence, and the saliency landscape for this area is almost completely
flat, indicating that there is no explanation for any of these bottles that is
smaller than the bounding box itself. This contrasts with the partial, dark,
bottles on the bottom row, which have been successfully subdivided at least
once, even though they are only partially visible. This indicates yolo’s
dependency on color which, to a human, is orthogonal to recognizing a bottle.

###  5.4 Shape of explanations

All object detectors we examined produce rectangular bounding boxes. In our
experiments, the explanations provided by anon-rex were rectilinear, even when
the objects do not have straight lines and edges. This may be due to the
requirements of yolo, an insufficient number of iterations from anon-rex, or a
combination of the two. To investigate this issue, we executed anon-rex, with
yolo as the object detector, on a dataset containing images of apples [9].
Apples were chosen as objects without straight lines and ones where the ground
truth is known. LABEL:fig:applepx in the introduction shows an example of some
of its outputs. Inspection of the results indicates that even with $20$
iterations, the shape of anon-rex explanations is strongly influenced by the
bounding box requirements. An explanation must be confirmed by the model, and
yolo and similar tools require enough pixels to draw a rectangle, a
fundamental part of their classification, thus limiting the shape of the final
explanation. On the other hand, anon-rex explanations can be much smaller than
yolo bounding boxes, hence providing insights into the yolo detection process.

(a) yolo classification

(b) anon-rex explanation

Figure 8: 8(b) suggests that yolo misclassifies a fish as a human in 8(a),
because of the part of the fish that is mistaken for red hair.

(a) yolo bird classification

(b) anon-rex shows a misclassification

Figure 9: The image in 9(a) is misclassified as two birds, with the
explanation in 9(b) showing that a branch and white patch in the image are
being interpreted as a bird.

###  5.5 Explaining misclassifications

yolo, like all models, is not $100\%$ accurate. We analyse two examples of
misclassifications and show that their explanations, produced by anon-rex, can
be helpful in identifying the reasons for misclassification. yolo recognizes
$80$ different classes and is capable of returning “no classification” if it
does not recognize anything in the image. Figure 8 shows an image of an object
that yolo does not know: a fish. It still provides a very good bounding box
detection on the image, with a low confidence, $0.27$, that the fish is a
person. The anon-rex explanation (Figure 8(b)) highlights the upper region of
the fish, suggesting that yolo is interpreting the image as containing (red)
hair. Figure 9 shows an object from a class that yolo does understand: a bird.
Again, we see a bounding box with low confidence labelled bird. The
explanation from anon-rex is apparently irreducible, being the same size as
the original bounding box. yolo has found something that is plausibly a bird,
as the explanation shows both a branch at the bottom of the image (for
perching) and a large white shape that swells and contracts much in the
fashion of a bird’s body. In our datasets we were unable to identify a
misclassification which also had a high confidence.

##  6 Conclusions

anon-rex is the first black box XAI tool for explaining the outputs of object
detectors. It significantly outperforms rex, so the impact of anon-rex
explanations on the performance of object detectors is much lower, opening the
possibility for using anon-rex for explainability in real time. We
demonstrated the generality of our method and its efficiency by using anon-rex
on several different object detection tools, both one-pass and two-pass,
namely, yolo, ssd, and faster r-cnn. We show that existing white box
explainability methods, in addition to requiring a significant amount of
coding, do not produce sufficiently precise results in terms of quality of
explanations.

## References

  * [1] Eigencam for YOLO v8 interpretability.  https://github.com/rigvedrs/YOLO-V8-CAM.
  * [2] Imagenet 1000 (mini).  https://www.kaggle.com/datasets/ifigotin/imagenetmini-1000.
  * Armin et al. [2022] Kirchknopf Armin, Djordje Slijepcevic, Ilkay Wunderlich, Michael Breiter, Johannes Traxler, and Matthias Zeppelzauer.  Explaining YOLO: Leveraging Grad-CAM to explain object detections.  2022\.
  * Bach et al. [2015] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PLOS One_ , 10(7), 2015.
  * Chen et al. [2018] Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan.  Learning to explain: An information-theoretic perspective on model interpretation.  In _International Conference on Machine Learning (ICML)_ , pages 882–891. PMLR, 2018.
  * Chockler and Halpern [2004] Hana Chockler and Joseph Y. Halpern.  Responsibility and blame: A structural-model approach.  _J. Artif. Intell. Res._ , 22:93–115, 2004.
  * Chockler et al. [2021] Hana Chockler, Daniel Kroening, and Youcheng Sun.  Explanations for occluded images.  In _IEEE/CVF International Conference on Computer Vision, ICCV_ , pages 1214–1223. IEEE, 2021.
  * Chockler et al. [2023] Hana Chockler, David A. Kelly, and Daniel Kroening.  Multiple different explanations for image classifiers, 2023.
  * [9] Samuel Cortinhas.  Apples or tomatoes – image classification.  https://www.kaggle.com/datasets/samuelcortinhas/apples-or-tomatoes-image-classification.
  * Datta et al. [2016] Anupam Datta, Shayak Sen, and Yair Zick.  Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems.  In _Security and Privacy (S &P)_, pages 598–617. IEEE, 2016.
  * Fong et al. [2019] Ruth Fong, Mandela Patrick, and Andrea Vedaldi.  Understanding deep networks via extremal perturbations and smooth masks.  In _International Conference on Computer Vision (ICCV)_ , pages 2950–2958. IEEE, 2019.
  * Girshick et al. [2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation, 2014.
  * Halpern [2016] J. Y. Halpern.  _Actual Causality_.  MIT Press, Cambridge, MA, 2016.
  * Halpern and Pearl [2005] Joseph Y. Halpern and Judea Pearl.  Causes and explanations: A structural-model approach. Part II: Explanations.  _British Journal for the Philosophy of Science_ , 56(4), 2005.
  * Jocher et al. [2023] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  Ultralytics YOLOv8, 2023.
  * Liu et al. [2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: single shot multibox detector.  In _Proceedings of European Conference in Computer Vision ECCV, Part I_ , pages 21–37. Springer, 2016.
  * Lundberg and Lee [2017] Scott M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 4765–4774, 2017.
  * Muhammad and Yeasin [2020] Mohammed Bany Muhammad and Mohammed Yeasin.  Eigen-CAM: Class activation map using principal components.  _CoRR_ , abs/2008.00299, 2020.
  * Nam et al. [2020] Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, and Seong-Whan Lee.  Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks.  In _AAAI Conference on Artificial Intelligence_ , pages 2501–2508, 2020.
  * Petsiuk et al. [2018] Vitali Petsiuk, Abir Das, and Kate Saenko.  RISE: randomized input sampling for explanation of black-box models.  In _British Machine Vision Conference (BMVC)_. BMVA Press, 2018.
  * Redmon et al. [2016] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of CVPR_ , pages 779–788, 2016.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  “Why should I trust you?” Explaining the predictions of any classifier.  In _Knowledge Discovery and Data Mining (KDD)_ , pages 1135–1144. ACM, 2016.
  * Selvaraju et al. [2017] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual explanations from deep networks via gradient-based localization.  In _International Conference on Computer Vision (ICCV)_ , pages 618–626. IEEE, 2017.
  * Shitole et al. [2021] Vivswan Shitole, Fuxin Li, Minsuk Kahng, Prasad Tadepalli, and Alan Fern.  One explanation is not enough: Structured attention graphs for image classification.  In _Neural Information Processing Systems (NeurIPS)_ , pages 11352–11363, 2021.
  * Shrikumar et al. [2017] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In _International Conference on Machine Learning (ICML)_ , pages 3145–3153. PMLR, 2017.
  * Springenberg et al. [2015] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller.  Striving for simplicity: The all convolutional net.  In _ICLR (Workshop Track)_ , 2015.
  * Sun et al. [2020] Youcheng Sun, Hana Chockler, Xiaowei Huang, and Daniel Kroening.  Explaining image classifiers using statistical fault localization.  In _ECCV, Part XXVIII_ , pages 391–406. Springer, 2020.
  * Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International Conference on Machine Learning_ , pages 3319–3328. PMLR, 2017.