[1]Pranav Singh

1]Center for Data Science, New York University, 60 5th Ave, New York, 10011,
NY, USA

2]Department of Computer Science, Courant Institute of Mathematical Sciences,
New York University, 60 5th Ave, New York, 10012, NY, USA

3]Division of Rheumatology, NYU Grossman School of Medicine, 550 1st Ave, New
York, 10016, NY, USA

4]Colton Center for Autoimmunity, NYU Grossman School of Medicine, 550 1st
Ave, New York, 10016, NY, USA

# Shifting to Machine Supervision: Annotation-Efficient Semi and Self-
Supervised Learning for Automatic Medical Image Segmentation and
Classification

<EMAIL_ADDRESS>Raviteja Chukkapalli<EMAIL_ADDRESS>Shravan Chaudhari
<EMAIL_ADDRESS>Luoyao Chen<EMAIL_ADDRESS>Mei Chen<EMAIL_ADDRESS>Jinqian
Pan<EMAIL_ADDRESS>Craig Smuda<EMAIL_ADDRESS>Jacopo Cirrone
<EMAIL_ADDRESS>[  [  [  [

###### Abstract

Advancements in clinical treatment and research are limited by supervised
learning techniques that rely on large amounts of annotated data, an expensive
task requiring many hours of clinical specialists’ time. In this paper, we
propose using self-supervised and semi-supervised learning. These techniques
perform an auxiliary task that is label-free, scaling up machine-supervision
is easier compared with fully-supervised techniques. This paper proposes S4MI
(Self-Supervision and Semi-Supervision for Medical Imaging), our pipeline to
leverage advances in self and semi-supervision learning. We benchmark them on
three medical imaging datasets to analyze their efficacy for classification
and segmentation. This advancement in self-supervised learning with 10%
annotation performed better than 100% annotation for the classification of
most datasets. The semi-supervised approach yielded favorable outcomes for
segmentation, outperforming the fully-supervised approach by using 50% fewer
labels in all three datasets. We have open-sourced and made our code available
at link333We would open-source our code upon acceptance.

###### keywords:

Medical Image Analysis, Self-supervision, semi-supervised learning

##  1 Introduction

Medical imaging analysis is central to clinical assistance with respect to
diagnosis, treatment, and planning. With the introduction of deep learning
methods, neural networks can now effectively and efficiently support imaging
analysis. The automation of this pipeline is becoming of great value for
decision-making with respect to patient treatments. However, existing deep-
learning techniques rely on the availability of well-structured and annotated
medical image data. Unlike natural imaging, annotations are diffiult to obtain
in the medical domain due to various privacy, cost, and time-related issues
[1]. Additionally, the annotation process cannot be crowd-sourced, as
annotations in medical imaging require trained clinicians and medical experts
to annotate. Consequently, medical imaging datasets are comparatively much
smaller than natural imaging datasets. Training neural networks from scratch
on such a limited amount of data yields sub-par performance. Hence, transfer
learning is employed to transfer the knowledge learned in solving source
domain tasks to solve target domain tasks. Transfer learning involves
initializing the network with weights learned for source domain tasks and
fine-tuning on target domain data. Transfer learning works best when the
source, target dataset image, and output classes are similar [2]; however,
this is not always possible in medical imaging.

In recent times, the field of computer vision has witnessed remarkable
strides, as a result of achievements of learning algorithms such as Deep
Metric Learning [3], self-distillation [4], masked image modeling [5], etc.
These innovative approaches have made learning meaningful visual
representations from unannotated image data possible. Finetuning models from
these learners has shown competitive performance [4] gains. Consequently, a
growing interest exists in exploring and studying the application of such
learning algorithms in medical imaging modalities with limited annotation.
These learning algorithms are typically deployed within the framework of self-
supervised, semi-supervised, or unsupervised settings. They often involve a
two-stage process: 1) a pre-training stage where representations are acquired
from unannotated data and 2) a fine-tuning stage using annotated data where
the model is refined to align with the specific target task. In a self-
supervised setting, pre-training involves an auxiliary task performed without
using ground truth labels to learn strong representations. In the case of a
semi-supervised setting, the pre-training and finetuning stages can be
performed simultaneously. In self-supervised approaches, we focus on DINO
(Distillation with NO labels)[4] and CASS (Cross-Architerual Self-supervision)
[6]. These techniques leverage joint embedding [7] based architecture to learn
useful representations. These approaches essentially learn to map the two
views of an image to the same embedding. DINO [4] relies on augmentation to
generate two views of the same image, whereas CASS [6] relies on two different
neural network architectures for the same image. For the semi-supervised
approach, we focus on the cross-architectural, cross-teaching approach [8].
Shifting reliance from the labels (human annotation) to some fully self-
supervisory signal or in part (semi-supervised) signal could also be
understood as a shift to machine supervision from human supervision. Our study
shows that machine supervision improves performance with modest fractions of
labeled data; the use of these approaches early in the research could save
triage time. Machine supervision methods are also more scalable than
supervised methods as they are label-independent.

In this work, we compare the performance of state-of-the-art machine
supervision approaches against transfer learning on segmentation and
classification, two critical medical image analysis tasks, on three
challenging datasets of different medical modalities - Histopathology Slide
images and skin lesions images. We will also study the effectiveness of
machine supervision by varying the amount of annotated data used for fine-
tuning. Machine supervision approaches studied are [4, 6] for the purpose of
classification, and [8, 9] segmentation.

##  2 Methods

###  2.1 Datasets

To compare transfer learning from Imagenet with machine supervision, we select
three datasets representative of the datasets clinicians have to deal with in
real life, with varying levels of imbalance and sample sizes.

  * •

Dermatomyositis Dataset: We have 198 RGB samples from 7 patients, each of 352
by 469 size. With this dataset, we perform multilabel classification. Our task
here is to classify cells based on their protein staining into TFH-1, TFH-217,
TFH-Like, B cells, and cells that don’t conform to previous cell types as
”others” classes. We use the F1 score as our evaluation metric on the test
set.

  * •

Dermofit Dataset: This dataset contains 1300 image samples taken using an SLR
camera with the following ten classes: Actinic Keratosis (AK), Basal Cell
Carcinoma (BCC), Melanocytic Nevus / Mole (ML), Squamous Cell Carcinoma (SCC),
Seborrhoeic Keratosis (SK), Intraepithelial carcinoma (IEC), Pyogenic
Granuloma (PYO), Haemangioma (VASC), Dermatofibroma (DF) and Melanoma (MEL).
No two images in this dataset are the same size, varying from 205×205 to
1020×1020.

  * •

ISIC-2017 Dataset: For this dataset, we have 2000 JPEG images classified into
three classes - Melanoma, Seborrheic keratosis, and Benign nevi. We use the
Recall score as our evaluation metric on the test set.

###  2.2 Classification

For classification, we compare existing self-supervised techniques with
transfer learning, the de facto norm in medical imaging. As mentioned, due to
a lack of data in medical imaging, classifiers are often trained using
initialization from other large visual datasets like ImageNet. The
architectures are then fine-tuned on the target medical imaging dataset using
these initializations. Alternatively, we can train using self- supervised
approaches to provide better classification priors for the target medical
imaging dataset. In self-supervised learning, an auxiliary task is performed
without labels to learn fine-grained information about the image. This
auxiliary task can be performed in various ways, for example, by corrupting
the image, followed by its reconstruction, creating copies of the same image
(positive pairs), and minimizing the distance between them using two
differently parameterized architectures or redun- dancy reduction. In our
study, we focus on DINO [4] and CASS [6]. DINO learns by creating augmentation
asymmetric copies of the input image, whereas CASS creates architecturally
asymmetric copies followed by similarity maximization between the copies. We
pre-train using these two self-supervised techniques for 100 epochs, fol-
lowed by 50 epochs of fine-tuning with labels. We perform fine-tuning with 10%
and 100% label fractions. For this, we use all of the available labels per
image while using 10% or 100% of the total labels available.

####  2.2.1 Classification pipeline additional details

For the ImageNet supervised classification approach, we used images and their
labels as our inputs. For training with x% labels, we only used x% of the
entire training images and labels while keeping the number of labels per image
the same. For the self-supervised approaches DINO and CASS, for pertaining, we
start with only images. During the fine-tuning process, we initialize the
networks with their pre-trained weights and use corresponding image-label
pairs. Similar to the supervised approach, we also fine-tune these
architectures for two label fractions, 10% and 100%. Further details can be
inferred from our code base open-sourced at link ***We will open-source our
code upon acceptance..

###  2.3 Segmentation

This section will first introduce standard implementation designs mutually
applied to all models, followed by additional implementation details for the
semi-supervised model. Lastly, we will describe the procedure of testing
different cross-entropy weight initialization.

####  2.3.1 Data-Preprocessing

Following Singh and Cirrone [10], we choose the processed image sample size to
be 480 × 480 to avoid empty tiles issues. To ensure uniformity, the image
input size to the model remains consistent across all three datasets, as
illustrated below. For the Dermatomyositis dataset, since it contains images
of uniform sizes 1408$\times$1876 each, we tiled them to a size of
480$\times$480 and then used blank padding at the edges to make them fit in
480$\times$480 sizes. This results in 12 tiled sub-images per sample and then
resized to 224$\times$224\. In contrast, the Dermofit and the ISIC2017
datasets contain images of different sizes. Since the two datasets are about
skin lesions, they have significantly denser and larger mask labels than the
Dermatomyositis dataset. Thus, a different image preprocessing step is applied
to the latter two datasets: bilinear interpolation to 480 × 480 followed by a
resize to 224 $\times$224.

####  2.3.2 Common Implementations:

We implemented all models in Pytorch [11] using a single NVIDIA RTX-8000 GPU
with 64 GB RAM and 3 CPU cores. All models are trained with an Adam optimizer
with an initial learning rate (lr) of 3.6e-4 and a weight decay 1e-5. We also
set a cosine annealing scheduler with a maximum of 50 iterations and a minimum
learning rate of 3.4e-4 to adjust the learning rate based on each epoch. For
splitting our dataset into training, validation, and testing sets, we use a
random train-validation-test split (70%-10%-20%), except in the ISIC2017
dataset, where we adopt the train/val/test split according to [12] for match-
up comparison (57%-8.5%-34%). The batch size is 16, and we use data
augmentation to enrich the training set using random rotation, random flip,
and a further resizing to 224 $\times$ 224 to fit in Swin Transformer’s patch
size requirement[13]. Note that for 3-channel datasets (Dermofit, ISIC), we
add a pre-processing step that normalizes the red channel of the RGB color
model as proposed by Thanh et al. [14]. We repeat all experiments with
different seed values five times and report the mean value in the 95%
confidence interval in all tables. Similar to classification, we fine-tune
with multiple label fractions, with semi-supervised and DEDL. When we mention
that we fine-tune with x% labels, we use all labels per image but only x% of
the total available image-label pairs.

####  2.3.3 Approach specific additional details

##### Semi-Supervised Approach :

The semi-supervised model uses a batch size of 16, and each batch consists of
eight labeled images and eight unlabeled images. In the last column of Table
5, we compare the per- formances between semi- and fully-supervised models in
100% labeled-ratio scenario; for this, we re-adjusted the batch to be 15
labeled images and one unlabeled image to approximate the fully-supervised
setting while still retaining the ”cross teaching” component so that the loss
remains consistent hence comparable with other semi- supervised label ratios.
Additionally, when comparing the performance between fully and semi-supervised
models, we adopt the same practice from Luo et al. [8] and use Swin
Transformer to compare DEDL with Resnet34 backbone as they have a similar
number of parameters

##### Unsupervised Approach :

We implemented PiCIE for Dermofit, Dermatomyositis, and ISIC2017 datasets
similar to the supervised and self/semi-supervised methods. We made slight
changes that suit the datasets and ensure smooth learning, such as using SGD
optimizer instead of Adam optimizer, keeping the learning rate the same as the
original implementation (1e-4), and adding StepLR (Step Learning Rate)
scheduler provided by PyTorch. This is because we observed that the model
sometimes failed to learn and was stuck at predicting all the image pixels as
either foreground (thing) or background (stuff) with the Adam optimizer. We
used batch sizes of 64 and 128 depending on the availability of CUDA RAM and
dataset sizes. We trained the PiCIE unsupervised pipeline for 50 epochs with
ResNet34 backbone as a feature extractor and retained the unsupervised
clustering technique described in [9] to achieve optimal segmentation
performance. The hardware used was Nvidia RTX8000, similar to the previous
methods. For PiCIE, labels were only used for validation and testing, not
during training.

##### Supervised Approach :

We adopted [10]’s approach for the supervised part of our comparison. For
this, we image and mask pairs during training, validation, and testing with a
ResNet-34-based U-net.

##  3 Results

We start by studying the classification pipeline in Section 3.1, followed by
understanding the interpretability aspect of classification in Section 3.1.1.
Since interpretability is ingrained in the segmentation task, we only study
segmentation quantitatively in Section 3.2.

###  3.1 Classification

For classification, we advocate using self-supervision to learn useful
discriminative representations from unlabelled data. Self-supervised
techniques rely on an auxiliary pretext task for pre-training to learn useful
representations. These representations are further improved and aligned with
the downstream task through labeled fine-tuning. We benchmark and study the
performance of transfer learning as well as two self-supervised techniques: 1)
Distillation with No Labels (DINO) [4], a self-supervised approach designed
for natural images, and 2) Cross cross-architectural self-supervision (CASS)
[6] a self-supervision approach for medical imaging. We further expand on our
methodology in Section 2.2.

Figure 1: Applying CASS: In this figure we detail the steps involved in
applying CASS, the best-performing machine supervision approach for
classification [6]. We conducted experiments on three datasets listed on the
left side: the Dermatomyositis, ISIC-2017, and Dermofit datasets. For training
on a dataset, we initialize both the networks with their ImageNet weights and
select one dataset at a time. To train CASS, we start by label-free
pretraining as illustrated in Part (b) of Fig. 1. During pre-training, a CNN
and a Transformer are trained simultaneously. In the case of the
Dermatomyositis dataset, the finetuning is multi-label, while in the case of
the ISIC-2017 and the Dermofit dataset, it is multi-class. This pre-training
in (b) is followed by labeled fine-tuning as shown in Part (c), where image
are fine-tuned one at a time.

We present the results of supervised and self-supervised trained classifiers
in Table 1. We use the F1 score as the comparison metric, similar to its
implementation in previous works [6]. The F1 score is defined as
$F1=\frac{2*Precision*Recall}{Precision+Recall}=\frac{2*TP}{2*TP+FP+FN}$,
where $TP$ represents True Positive, $FN$ is False Negative and $FP$ is False
Positive. We observe that CASS (Figure 1) outperforms DINO and transfer
learning consistently across all the datasets and backbones, with the,
exception for the ISIC 2017 challenge dataset, where DINO outperforms CASS
using the ResNet50 backbone. Interestingly, we also observe that ViT trained
with CASS using just 10% labels, which saves significant annotation time,
performs better than the transfer learning-based supervised approach with 100%
labels. For the same case, DINO performs on par with the supervised approach
while using 90% fewer labels. This shows the impact that machine supervision
can have on improving access to classified medical images. DINO and CASS take
an image to create asymmetry through augmentation (DINO) and architecture
(CASS). These are then passed through different parameterized feature
extractors (DINO) and architectural feature extractors (CASS) to create
embeddings. Since the two embeddings are generated from the same image, they
are expected to be similar; this is then used as the supervisory signal to
maximize the similarity between the two embedding and update encoders. In
CASS, architectural invariance is used to create a positive pair instead of
creating augmentation invariance, as in the case of DINO. This is because CNN
and Transformers learn different representations from the same image [6].

Backbone | Technique |  | Autoimmune Dataset
---
F1 Score
| Dermofit Dataset
---
F1 Score
| ISIC 2017 Dataset
---
Recall Score
10% | 100% | 10% | 100% | 10% | 100%
ResNet-50 | DINO |  | 0.8237
---
±0.001
| 0.84252
---
±0.008
| 0.3749
---
±0.0011
| 0.6775
---
±0.0005
| 0.322
---
±0.009
| 0.6407
---
±0.0363
CASS |  | 0.8158
---
±0.0055
| 0.8650
---
±0.0001
| 0.4367
---
±0.0002
| 0.7132
---
±0.0003
| 0.343
---
±0.002
| 0.599
---
±0.0304
| ImageNet
---
Transfer
| 0.819
---
±0.0216
| 0.83895
---
±0.007
| 0.33
---
±0.0001
| 0.6341
---
±0.0077
| 0.288
---
±0.091
| 0.5774
---
±0.0004
ViT B16 | DINO |  | 0.8445
---
±0.0008
| 0.8639
---
±0.002
| 0.332
---
± 0.0002
| 0.4810
---
±0.0012
| 0.291
---
±0.005
| 0.5571
---
± 0.076
CASS |  | 0.8717
---
±0.005
| 0.8894
---
±0.005
| 0.3896
---
±0.0013
| 0.6667
---
±0.0002
| 0.321
---
±0.007
| 0.588
---
±0.0094
| ImageNet
---
Transfer
| 0.8356
---
±0.007
| 0.8420
---
±0.009
| 0.299
---
±0.002
| 0.456
---
±0.0077
| 0.277
---
±0.007
| 0.5322
---
±0.022
Table 1: Results for DINO, CASS, and Supervised (ImageNet-initialized transfer
learning) methods on the Dermatomyositis, the Dermofit, and ISIC 2017
datasets. In this table, we compare the F1 score on the test sets. In all
cases, self-supervised pre-training (CASS or DINO) improves performance over
the ImageNet Transfer learning benchmark. Additionally, among CASS and DINO,
we observe that CASS, with its focus on architectural invariance, outperforms
DINO for almost all label fractions across three datasets for ResNet as well
as for ViT.

####  3.1.1 Saliency Maps

In segmentation, interpretability is ingrained in the task itself, as we can
easily identify whether model predictions are aligned with the ground truth.
To understand the decision-making process of trained neural classifiers, we
study pixel attribution or saliency maps [15]. To accomplish this, we first
train the neural network and compute the gradient of a class score with
respect to the input pixels. Backpropagating the gradients that lead to a
particular classification onto the input image helps us understand the most
filtered/rewarding extracted features related to classification. We present
the results using DINO, CASS-trained ResNet-50, and ViT Base/16 on the
ISIC-2017 dataset in Figure 2. Although most saliency maps fail to align with
human-interested pathology, CASS-based saliency maps align more than those
from DINO. This could be attributed to the joint training of CASS, where it
trains a CNN that focuses on local information while a Transformer focuses on
global or image-level features. Although both techniques fail to recognize and
focus on the relevant area in all cases, CASS-trained architectures are
consistently better focused on the relevant pathology than DINO-trained
architectures.

Figure 2: Visualization of saliency maps on a random sample from the ISIC-2017
dataset, left (a, b): data (input image), middle (c, d): saliency map from
CASS, and right: saliency map from DINO-trained ViTB/16 at the top and
ResNet-50 at the bottom (f). DINO’s saliency map exhibits notable stochastic-
ity, displaying a lack of strong correlation with the specific pathology under
consideration. Conversely, in the case of CASS, the saliency maps demonstrate
a significantly more aligned with the pathology of interest both for CNN as
well as the Transformer.

###  3.2 Segmentation

Figure 3: In this figure we present the semi-supervised pipeline as described
in 3.2.1. Similar to the classification experiments, we evaluate the
segmentation pipeline on three challenging medical image segmentation datasets
- the Dermatomyositis, ISIC-2017, and the Dermofit dataset. We use one dataset
at a time to train the semi-supervised architecture. Unlike the classification
pipeline, semi-supervised learning involves simultaneous learning from labeled
and unlabeled data. In Part (b) of Fig. 3, we start by training the data in an
unlabeled fashion and during the same iteration labeled data is also passed to
the architecture as shown in part (c) of the figure. Predictions from passing
inputs of the labeled images yield learned predictions as shown in Part (e).
Unsupervised loss is then calculated by comparing the outputs of the CNN and
the Transformer (as shown in Part(d) of this figure) using the
$\mathcal{L}_{\text{Unsupervised}}$ in Section 3.2.1. This unsupervised loss
is then added to the supervised loss denoted by
$\mathcal{L}_{\text{Supervised}}$in Section 3.2.1. The supervised loss is
calculated against the ground truth as shown in part (e) and (f) in this
figure.

Image Segmentation is another important task in medical image analysis. The
area of interest – usually containing the pathology – is segmented from the
rest of the slide image. We study the performance of transfer learning, semi-
supervised, and unsupervised approaches for segmentation. In computer vision,
it is widely acknowledged that image segmentation presents a more intricate
challenge than image classification. This distinction arises from the fact
that image segmentation necessitates the meticulous classification of
individual pixels, whereas image classification solely requires making
predictions at the image level. Moreover, due to the aforementioned rationale,
annotating images for segmentation exhibits a significantly higher complexity
level. Therefore, the present study aims to investigate the effectiveness of
unsupervised, semi-supervised, and fully supervised techniques across four
distinct label fractions instead of the conventional two in classification. In
our case, x% label fractions indicate that we only use x% labels from the
dataset. We include the unsupervised approach [9] to study the performance for
0% labels scenario. We compare the performance of DEDL [16], a transfer
learning-based approach, against [8], a semi-supervised approach using the
Swin transformer-based U-Net model (Figure 3). We choose [16] as it has shown
great promise in histopathology imaging and [8], a state-of-the-art approach
in leveraging unannotated data for MR segmentation. To ensure a fair
comparison, we assess architectures trained with a similar amount of
parameters over three datasets and four-label fractions. From [16], we use
their best-reported, ResNet-34-based U-Net model. We use thr three data sets
as described in Section 2.1. To ensure our results are statistically
significant, we conduct all experiments with five different seed values and
report the mean values in the 95% confidence interval (C.I) over the five
runs. For comparison, we use IoU (Intersection over Union) in line with
previous works in the field [10]. IoU or Jaccard index for two images U and V
is defined as $IoU(U,V)=\frac{|U\cap V|}{|U\cup V|}$. We present these results
in Table 5.

####  3.2.1 Semi-Supervised Approach

Semi-supervision approaches utilize the dataset’s labeled and unlabeled parts
for learning. Techniques generally fall under 1) Adversarial training (e.g.
DAN[17]) 2) Self-Training (e.g. MR image segmentation[18]) 3) Co-Training
(e.g. DCT[19]) and 4) Knowledge Distillation (e.g. UAMT[20]). We study the co-
training-based semi-supervision segmentation technique [8], depicted in Figure
4. The model consists of two segmentation architectures, a U-Net (trained from
scratch) and a pre-trained Swin Transformer, adapted from the Swin Transformer
proposed by Liu et al.[13]. Both architectures are updated simultaneously
using a combined loss over the labeled and unlabeled images.

Figure 4:  In this figure, we depict the Unsupervised Segmentation Approach:
PiCIE pipeline [9]. $View_{1}$ and $View_{2}$ represent two photometrically
transformed views of the input image, whereas $View_{2}^{1}$ represents the
geometric transformation of $View_{2}$. Cross-view training is then used to
train the architecture shared between the two views (parameterized by $\theta$
in the figure); we have expanded further on this in Section 3.2.2.

The U-Net model is trained as follows:
1) The labeled data L is evaluated using the average of dice and cross-entropy
loss

$\mathcal{L}_{\text{Supervised}}=\frac{1}{2}[\text{DICE}(\text{pred}_{\text{U-Net}},\text{L})+\text{CrossEntropy}(\text{pred}_{\text{U-Net}},\text{L})]$

2) For unlabeled data U cross-teaching strategy is used to cross-supervised
between CNN and Transformer: to update U-Net, output from Transformer,
$\text{pred}_{\text{Transformer}}$ (vice versa) is used as ”sudo ground
truth”:

$\text{U}_{\text{sudo}}=\text{argmax}(\text{pred}_{\text{Transformer}})$

3) The unsupervised loss for U-Net is conducted between prediction and pseudo-
ground truth:

$\mathcal{L}_{\text{Unsupervised}}=\text{DICE}(\text{pred}_{\text{U-Net}},\text{U}_{\text{sudo}})$

4) The final loss is the summation of unsupervised and supervised loss.

####  3.2.2 Unsupervised Approach

Unsupervised approaches for learning from unlabelled images are mostly limited
to curated and object-centric images; however, some recently proposed methods
like [9] achieve semantic segmentation with the help of unsupervised
clustering methods. The machine aims to discover and identify the different
foreground objects and separate them from the background. In this paper, the
datasets under study only contain one foreground object, which eases learning;
however, unsupervised segmentation continues to be a challenging task. [9]
proposes a technique PiCIE (Pixel-level feature Clustering using Invariance
and Equivariance) to incorporate geometric consistency to learn invariance to
photometric transitions and equivariance to geometric transitions. The two
main constraints here are 1) to cluster the pixels having similar appearance
together with the same label for all the pixels belonging to a cluster and 2)
to ensure the pixel label predictions follow the invariance and equivariance
constraints mentioned earlier. With these objectives, [9] trains a ConvNet-
based semantic segmentation model in a purely unsupervised manner (without
using any pixel labels). PiCIE uses the alternating strategy between
clustering the feature representations and using the cluster labels as pseudo
labels to train the feature representation proposed by DeepCluster [21].

Figure 4 (b) shows a brief overview and an end-to-end pipeline of the PiCIE
approach. For each input image, two photometric transformations, like
jittering the pixel colors, etc., are randomly sampled to create $View_{1}$
and $View_{2}^{1}$. One of these views ($View_{1}$) is directly fed to the
network, while geometric transformation is applied to the other view before
feeding it to the same network($View_{2}^{1}$). The network then produces the
two feature representations for every pixel corresponding to the two views.
Since we expect equivariance for geometric transformations, they are applied
to feature representations of the original image before clustering the pixels
of the feature representations. The inductive biases of invariance and
equivariance to photometric and geometric transformations are introduced by
aggregating the clustering losses within the same view and across the two
views. Using this cross-view training strategy, the training is guided to
prevent predicted pixel labels from changing with jittering of pixel colors.
Still, at the same time, if the image is warped geometrically, then the
warping would also be reflected in the labeling similarly [9].

####  3.2.3 Comparision

Figure 5 shows the evaluation results of these methods for the three datasets.
The evaluation metric applied is intersection over union (IOU). The
unsupervised approach [9] sets the lower bound on the segmentation performance
and illustrates the difficulty of unsupervised segmentation and its ability to
outperform even the state-of-the-art unsupervised approaches using a small
number of annotations(10%). Semi-supervision continuously outperformed the
transfer learning across all label fractions across all datasets except the
10% case for ISIC, where the performance is close. For all the datasets, semi-
supervision with 50% labels outperforms transfer learning from Imagenet with
100% labels. These results show a significant performance gain by moving to
machine supervision approaches instead of transfer learning from Imagenet.
Figure 1 and Figure 3 show the best-performing approaches for learning from
limited annotation datasets for classifications and segmentation respectively.

Figure 5: In this figure, we compare Full, Semi, and Unsupervised
architectures for segmentation performance over three datasets. Here, Rn34
represents ResNet-34, while ST represents Swin-U-Net. Since [9] is an
unsupervised approach, it does not require any labels for fine-tuning. Hence,
we present results for PiCIE using ResNet-34 backbone under 0% label
fractions. We observe that the semi-supervised approach outperforms the fully-
supervised approach by using 50% fewer per image labels on all three datasets.

##  4 Discussion

We observed that the performance of the trained classifiers increased with the
introduction of pretraining self-supervised objectives. Comparing the two
training regimes, CASS more consistently improved performance over DINO in
almost all cases under study. Studying the pixel attribution of the trained
methods to understand their decision-making, we observed that CASS usually
attributes the relevant pixels more than DINO. Hence, using self-supervision,
i.e., additional training even with reduced human supervision, improved
performance.

We studied three training regimes for segmentation: supervised, semi-
supervised, and unsupervised. While applying the unsupervised approach
requires some adapta- tion and hyperparameter tuning, the dermofit data set
consistently shows that semi-supervised outperforms fully-supervised,
regardless of labeling percentage. However, we con- sistently improve
performance using a semi-supervised approach – wherein we used labeled and
unlabeled data per batch to help the model learn better representation. The
result on the dermatomyositis dataset using no labels is close as compared to
the supervised and semi-supervised with 10% labels. Between 50% and 70%
labeling, the performance of semi-supervised and fully-supervised merges.
However, the result on the ISIC-2017 data set with 10% labels indicates fully-
supervised performs better than the semi-supervised; however as the percentage
of labels increases above 10% semi-supervised outperforms fully-supervised.
Additionally, unlike the unsupervised approach, the supervised and the semi-
supervised approaches don’t treat this problem as pixel clustering and are
less prone to overfitting to the dominant pixel distribution. Since
classification requires image-level identification of classes instead of
pixel-level identification, segmentation is a more difficult objective. Yet,
introducing reduced human supervision with pseudo-labels in the semi-
supervised approach improved performance over the supervised and the
unsupervised paradigms to the where the semi-supervised approach was able to
outperform the fully supervised approach with 50% fewer labels.

##  5 Conclusion

The primary focus of our experimental investigations revolved around two
fundamental medical imaging tasks i.e, classification and segmentation.
Significantly, our research findings indicate that the shift from a transfer-
learning-based solely on supervised methodology to a machine-supervised
framework, which we named S4MI (Self-Supervision and Semi-Supervisio for
Medical Imaging) has generated encouraging outcomes. In classification, our
empirical findings consistently demonstrate that self-supervised training
performs better than the conventional supervised approach across Convolutional
Neural Network (CNN) and Transformer models.

Moreover, in the comparative analysis of the two self-supervised techniques.
CASS demonstrates superiority by exhibiting enhanced performance in nearly all
scenarios compared to DINO. Utilizing reduced supervision in segmentation has
yielded favorable outcomes. Specifically, applying the Semi-supervised
approach, as outlined in [8], has resulted in notable enhancements in
performance across all label fractions for the three datasets under
consideration.

A notable inference emerged from our investigation, wherein the complete
elimination of supervisory signals through the utilization of unsupervised
algorithms resulted in comparable performance solely in the context of the
dermatomyositis dataset when compared to architectures with 10% supervision,
an observation that underscores the inherent capacity of unsupervised
methodologies in distinct contextual situations.

Based on a comprehensive analysis of our experimental findings, adopting
machine-level supervision, such as the S4MI proceses we examined reduces
dependence on human supervision and also yields substantial advantages in time
and accuracy of medical image analysis. The findings of our study offer
significant empirical contributions to medical imaging and limited-supervision
techniques, thereby stimulating future investigations in this area. The
distribution as open-sourced code of our S4MI processes will support other
researchers’ efforts by saving time in labeling and improving image analysis.
This, in turn, will facilitate the advancement of healthcare solutions,
leading to the eventual enhanced outcomes in patient care. We are optimistic
that our study will serve as a catalyst for substantive discourse and
cooperative endeavors within the medical imaging community, thereby propelling
progress in this pivotal domain.

## References

  *   * Matsoukas et al. [2022] Matsoukas, C., Haslum, J.F., Sorkhei, M., Söderberg, M., Smith, K.: What makes transfer learning work for medical images: feature reuse & other factors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9225–9234 (2022)
  * Yang et al. [2020] Yang, X., He, X., Liang, Y., Yang, Y., Zhang, S., Xie, P.: Transfer learning or self-supervised learning? a tale of two pretraining paradigms. arXiv preprint arXiv:2007.04234 (2020)
  * Chen et al. [2020] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607 (2020). PMLR
  * Caron et al. [2021] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)
  * Xie et al. [2022] Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., Hu, H.: Simmim: A simple framework for masked image modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653–9663 (2022)
  * Singh et al. [2022] Singh, P., Sizikova, E., Cirrone, J.: Cass: Cross architectural self-supervision for medical image analysis. arXiv preprint arXiv:2206.04170 (2022)
  * Balestriero et al. [2023] Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al.: A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210 (2023)
  * Luo et al. [2022] Luo, X., Hu, M., Song, T., Wang, G., Zhang, S.: Semi-supervised medical image segmentation via cross teaching between cnn and transformer. In: Medical Imaging with Deep Learning (2022). https://openreview.net/forum?id=KUmlnqHrAbE
  * Cho et al. [2021] Cho, J.H., Mall, U., Bala, K., Hariharan, B.: Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. CoRR abs/2103.17070 (2021) 2103.17070
  * Singh and Cirrone [2022] Singh, P., Cirrone, J.: A Data-Efficient Deep Learning Framework for Segmentation and Classification of Histopathology Images (2022)
  * Paszke et al. [2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc., ??? (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * Codella et al. [2018] Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al.: Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 168–172 (2018). IEEE
  * Liu et al. [2021] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
  * Thanh et al. [2019] Thanh, D.N.H., Erkan, U., Prasath, V.B.S., Kumar, V., Hien, N.N.: A skin lesion segmentation method for dermoscopic images based on adaptive thresholding with normalization of color models. In: 2019 6th International Conference on Electrical and Electronics Engineering (ICEEE), pp. 116–120 (2019). https://doi.org/10.1109/ICEEE2019.2019.00030
  * Selvaraju et al. [2022] Selvaraju, R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. arxiv 2016. arXiv preprint arXiv:1610.02391 (2022)
  * Fu et al. [2023] Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., Isola, P.: Enhancing medical image segmentation: Optimizing cross-entropy weights and post-processing with autoencoders. arXiv preprint arXiv:2306.09344 (2023)
  * Zhang et al. [2017] Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D.P., Chen, D.Z.: Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III, pp. 408–416. Springer, Berlin, Heidelberg (2017). https://doi.org/10.1007/978-3-319-66179-7_47 . https://doi.org/10.1007/978-3-319-66179-7_47
  * Bai et al. [2017] Bai, W., Oktay, O., Sinclair, M., Suzuki, H., Rajchl, M., Tarroni, G., Glocker, B., King, A., Matthews, P.M., Rueckert, D.: Semi-supervised learning for network-based cardiac mr image segmentation. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) Medical Image Computing and Computer-Assisted Intervention - MICCAI 2017, pp. 253–260. Springer, Cham (2017)
  * Peng et al. [2020] Peng, J., Estrada, G., Pedersoli, M., Desrosiers, C.: Deep co-training for semi-supervised image segmentation. Pattern Recognition 107, 107269 (2020)
  * Yu et al. [2019] Yu, L., Wang, S., Li, X., Fu, C.-W., Heng, P.-A.: Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 605–613 (2019). Springer
  * Caron et al. [2018] Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. CoRR abs/1807.05520 (2018) 1807.05520


# Robustness of Explainable Artificial Intelligence in Industrial Process
Modelling

Benedikt Kantz  Clemens Staudinger  Christoph Feilmayr  Johannes Wachlmayr
Alexander Haberl  Stefan Schuster  Franz Pernkopf

###### Abstract

eXplainable Artificial Intelligence (XAI) aims at providing understandable
explanations of black box models. In this paper, we evaluate current XAI
methods by scoring them based on ground truth simulations and sensitivity
analysis. To this end, we used an Electric Arc Furnace (EAF) model to better
understand the limits and robustness characteristics of XAI methods such as
SHapley Additive exPlanations (SHAP), Local Interpretable Model-agnostic
Explanations (LIME), as well as Averaged Local Effects (ALE) or Smooth
Gradients (SG) in a highly topical setting. These XAI methods were applied to
various types of black-box models and then scored based on their correctness
compared to the ground-truth sensitivity of the data-generating processes
using a novel scoring evaluation methodology over a range of simulated
additive noise. The resulting evaluation shows that the capability of the
Machine Learning (ML) models to capture the process accurately is, indeed,
coupled with the correctness of the explainability of the underlying data-
generating process. We furthermore show the differences between XAI methods in
their ability to correctly predict the true sensitivity of the modeled
industrial process.

eXplainable Artificial Intelligence, Robustness, Noise Analysis, Machine
Learning, Industrial Process Modelling, Evaluation Methodology

##  1 Introduction

ML approaches have the power to model complex dependencies in demanding tasks
such as industrial processes. However, the behavior of these industrial
processes that rely on complex, non-linear interactions is often not fully
understood. This results in the need for algorithms to understand and
interpret how these ML models arrive at certain predictions and how they might
react to certain perturbances in the input. In the last years, there has been
an effort to provide explanations to the ML model predictions using XAI
(Lundberg & Lee, 2017; Ribeiro et al., 2018; Alvarez-Melis & Jaakkola, 2018;
Shrikumar et al., 2017).

Most of these works, even if they focus on the robustness and trustworthiness
of the XAI method, have the shortcoming that they can only be evaluated
through surrogate measures (Crabbé & van der Schaar, 2023), and the ground
truth sensitivity of the evaluated datasets cannot be properly calculated
(Alvarez-Melis & Jaakkola, 2018). Some existing approaches rather use data
augmentation (Sun et al., 2020) or create measures estimating the importance
of the features (Yeh et al., 2019); further related work is provided in
Section A.3. None of these systems, to the best of our knowledge, consider the
ground truth sensitivity, or gradient, of the data-generating process that
created the dataset. Modeling the sensitivity to the inputs is, however, key
to understand the underlying process using proxy ML models solely learned on
data.

In this paper, we introduce data-driven evaluation of different XAI methods
using a simulated process of an EAF model and its ground truth sensitivity,
providing insights into the actual limits and robustness properties of state-
of-the-art ML models and interpretability approaches. We propose to use a
specifically generated dataset and perform perturbations to analyze this
robustness empirically. Two central problems, however, arose when scoring
these XAI methods and comparing them to a ground truth sensitivity:

  1. 1.

The feature importance scores over the feature dimensions are not within the
same magnitude and range, requiring scaling (Shrikumar et al., 2017).

  2. 2.

The relative sizes of different XAI feature importance scores are not
necessarily aligned to each other (Lundberg & Lee, 2017; Apley & Zhu, 2019),
requiring normalization.

Therefore, we introduce a novel evaluation methodology, for solving both of
these problems. We essentially analyze how well the XAI methods explain the
sensitivity of the input features, compared to a known ground truth
sensitivity.

##  2 Data-generating processes

This section outlines the two data generation processes: a toy dataset, as
well as the EAF process simulation, including the process variables. These
specifically generated datasets are necessary, as other datasets do not
provide the ground truth effects $\mathbf{w}^{*}_{i}$ of the functions at the
datapoints $\mathbf{x}^{i}$.

###  2.1 Toy dataset

To prove the effectiveness of our evaluation methodology, we first build a
small polynomial data-generating system of the form

$f(x_{1},x_{2})=k_{1}x_{1}^{2}+k_{2}x_{2}^{2}+k_{3}x_{1}x_{2}+k_{4}x_{1}+k_{5}x_{2}+k_{6}$
(1)

with the coefficients $k_{d}$ being drawn from
$k_{d}\sim\mathrm{Uniform}(0,1)$ once. The function effects, or gradients,
were estimated using automatic gradient calculation using PyTorch (Paszke et
al., 2019). This data-generating process was used to generate 1000 samples
$f(x_{1},x_{2})$, where $x_{1},x_{2}\sim\mathrm{Uniform}(-5,5)$, which were
utilized for the evaluation process.

###  2.2 EAF process

The choice of an EAF model as the data-generating process was owed to a wide
range of factors. First, the EAF process itself is relevant in the steel
industry as it promises, given enough clean energy, greenhouse gas emission
reduction compared to traditional steel production in blast furnaces (De Ras
et al., 2019). Furthermore, EAFs have been studied and modeled for a long time
using many different approaches and modeling strategies, from their electrical
characteristics (Billings et al., 1979; Boulet et al., 2003) to their chemical
processes and internal interactions (Zhang & Fruehan, 1995; Basu et al.,
2008). These works helped build an accurate chemical simulation of an EAF,
representing the real-world system quite well while keeping it simulatable and
manageable in parameter space (Binti Ahmad Dzulfakhar et al., 2023). The
functional complexity is sophisticated, providing an interesting process to be
modeled by ML models. Furthermore, the data generating process is non-
independant and identically distrubuted (iid.), as the EAF model has
timesteps, and, when considering these, dependencies between each simulated
tapping of the furnace arise, making modeling even more challenging. The model
itself simulated the individual zones of the EAF reactor as homogeneous zones,
with the same temperature and uniform mixture across the whole zone. These
zones are the gas zone, solid metal zone, liquid slag zone, liquid metal, and
solid slag zone. Additionally, the reactor is modeled as a few discrete parts,
particularly the roof, and walls. The liquid metal and slag zones are the ones
where measurements were simulated during tapping.

The EAF model (Binti Ahmad Dzulfakhar et al., 2023) was converted into a
Python module, where automatic differentiation tools (Paszke et al., 2019)
were used to generate ground truth sensitivities of the input parameters. This
simulated experiment was repeated over sampled combinations of a subset of
different input parameters, which were the oxygen lance rate, oxygen for post-
combustion, power of the arc, carbon injection rate, ferromanganese injection
rate, and the mass addition rate of solids. Furthermore, auxiliary properties
within the simulated tapped material were recorded, more specifically the
ratios of silicon dioxide and iron oxide in the slag, as well as the
temperature of the liquid slag and metal. The observed target variable, the
ratio of carbon in the tapped steel from the furnace, was recorded, too. Each
tapping was considered as one data sample
$\mathbf{x}_{i}\in\left\\{\mathbf{x}_{1},\ldots,\mathbf{x}_{n}\right\\}$,
consisting of the observed variables. The gradient of the input parameters
concerning the output was calculated and accumulated at the timesteps in the
simulation, too. This simulation resulted in a dataset of about $n\approx
10^{4}$ samples after the removal of numerical outliers due to instabilities
after simulating the furnace for about a week. The process was furthermore
restarted after each parameter change.

###  2.3 Perturbing the Dataset

While the datasets at hand provided a perfect ground truth sensitivity for
interpretations, there was still no proper way to assess the robustness of the
ML models in combination with the XAI methods. To this end, the dataset was
artificially perturbed using noise in two ways. The source of the noise was
Gaussian noise added to the feature $j$ of sample $i$, except the target
variable, using

$\tilde{x}_{i,j}=x_{i,j}+n_{j}$ (2)

where
$n_{j}\sim\mathcal{N}(0,l\cdot(\underset{i\in{1,\ldots,n}}{\max}~{}x_{i,j}-\underset{i\in{1,\ldots,n}}{~{}\min}~{}x_{i,j}))$,
$x_{i,j}$ being the $j$-th feature of the observation
$\mathbf{x}_{i}=\\{x_{i,0},x_{i,1},\ldots,x_{i,d}\\}$. The $0$-th feature is
the target $y_{i}\in\mathcal{Y}$. $l\in[0,1)$ is the selected noise level of
the experiment.

##  3 Scoring Methodology for Local Explanations

perturb $\tilde{x}_{i,j}=x_{i,j}+n_{j}$ $n_{j}\sim\mathcal{N}(0,(l\cdot
r)^{2})$ train $f$ interpret $\mathbf{w}_{i}=\bm{\Phi}(f,\mathbf{x}_{i})$
scale $\tilde{\mathbf{w}}_{i}=\mathbf{x}_{i}\odot\mathbf{w}_{i}$ norm to
$\bar{w}_{i,j}\in[0,1]$
$\bar{w}_{i,j}=\frac{\tilde{w}_{i,j}-\underset{j\in{1,\ldots,d}}{~{}\min}~{}\tilde{w}_{i,j}}{\underset{j\in{1,\ldots,d}}{\max}~{}\tilde{w}_{i,j}-\underset{j\in{1,\ldots,d}}{~{}\min}~{}\tilde{w}_{i,j}}$
Toy/EAF Simulation split evaluation/train evaluation set $\mathbf{x_{i}}$
score $s\in[0,1]$
$s=1-\frac{1}{dN}\sum_{i=1}^{N}\sum_{j=1}^{d}\left(\bar{w}_{ij}-\bar{w}_{ij}^{*}\right)^{2}$
score $s\in[0,1]$ evaluation set $\mathbf{x_{i}}$ ground truth interpretation
$\mathbf{w}^{*}_{i}$ scale
$\tilde{\mathbf{w}}^{*}_{i}=\mathbf{x}_{i}\odot\mathbf{w}^{*}_{i}$ norm to
$\bar{w}^{*}_{i,j}\in[0,1]$
$\bar{w}^{*}_{i,j}=\frac{\tilde{w}^{*}_{i,j}-\underset{j\in{1,\ldots,d}}{~{}\min}~{}\tilde{w}^{*}_{i,j}}{\underset{j\in{1,\ldots,d}}{\max}~{}\tilde{w}^{*}_{i,j}-\underset{j\in{1,\ldots,d}}{~{}\min}~{}\tilde{w}^{*}_{i,j}}$
Figure 1: Evaluation methodology, see text for further details and choices of
measures.

We developed an evaluation methodology (see Figure 1) to quantify the effect
of the perturbances on the ML models and the interpretation, as there is no
such common measure to cater to heterogeneous types of feature importance and
interpretability measures, as discussed in Section A.3. Furthermore, the
problems that make comparisons difficult, as mentioned in Section 1, were
addressed using this scoring methodology.

We denote the output of each XAI method as
$\mathbf{w}_{i}={w_{i,1},\ldots,w_{i,d}}$, independent of the underlying
explainer. All ground truth sensitivity values from the data-generating
processes are denoted as $\mathbf{w}_{i}^{*}$. All input features of the data-
generating functions were used in this evaluation system. To solve the first
problem outlined in the introduction, the feature scores $w_{i,j}$ were scaled
using the features values themselves, using

$\tilde{w}_{i,j}=w_{i,j}\cdot x_{i,j}.$ (3)

This helps to adjust the magnitude of the XAI method results to the
appropriate scale. The second problem is approached by min-max normalization
on each data sample to indicate their relative strength within a sample. The
scaling results in an indication of the weights’ approximate strength by
scaling them to a $[0,1]$ range using

$\bar{w}_{i,j}=\frac{\tilde{w}_{i,j}-\underset{j\in{1,\ldots,d}}{~{}\min}~{}\tilde{w}_{i,j}}{\underset{j\in{1,\ldots,d}}{\max}~{}\tilde{w}_{i,j}-\underset{j\in{1,\ldots,d}}{~{}\min}~{}\tilde{w}_{i,j}}.$
(4)

Both scaling operations are performed for the ground truth derivatives
$w^{*}_{i,j}$ of the data-generating process too, leading to the normalized
ground truth $\bar{w}^{*}_{i,j}$. Using both results, the score of the local
interpretation can finally be computed using the Brier Score (Brier, 1950), a
probabilistic scoring method, initially intended for weather forecasts. It is,
in essence, the Mean Squared Error (MSE), and the score $s_{i}$ for one
observation is thus calculated using

$s_{i}=\frac{1}{d}\sum_{j=1}^{d}\left(\bar{w}_{i,j}-\bar{w}^{*}_{i,j}\right)^{2}.$
(5)

The final score $s$ for one set of $n$ observations is computed by averaging
each samples score $s_{i}$ using

$s=1-\frac{1}{n}\sum_{i=1}^{n}s_{i}.$ (6)

This averaging should return a score of $s=1$ if the interpretations
completely align with the ground truth and lower if there are discrepancies.
This scoring methodology, due to the comparison to the ground truth effect,
focuses therefore not on the feature’s importance, but rather on the
correctness of the effects that a feature, at a certain data point, has.

###  3.1 Evaluation Process

The evaluation methodology is performed 50 times over a randomly sampled fold
of 10 percent of the samples from the datasets. However, due to the non-iid
data of the EAF, care is taken during sampling. The simulation runs are
therefore sampled in a way that no data from one run can be taken into both
the evaluation and training set. The sampled training set (90% of the samples)
is then perturbed using a range of different noise levels using the approaches
from Section 2.3. The whole evaluation methodology is illustrated in Figure 1,
showing how the explanations are scaled and used throughout the process. The
blue part of the graph illustrates existing work and systems, while the green
part shows the novel scaling and scoring scheme.

####  3.1.1 Black-box Models

The regressor ML models $f(\mathbf{x}_{i})$ used for this evaluation are a
linear regression model, a neural network with three layers of 32 neurons with
Rectified Linear Unit (ReLU) activation modeled in PyTorch (Paszke et al.,
2019), and the XGBoost regressor (Chen & Guestrin, 2016). Explanations
$\mathbf{w}_{i}=\bm{\Phi}(f,\mathbf{x}_{i})$ are generated using the XAI
methods discussed in the next section. The necessary gradients are calculated
using either the parameters directly from linear regression, automatic
differentiation for the neural network, or finite differences for the XGBoost
tree. These local explanations are only generated on the unperturbed and
complete evaluation set. This allows us to test how well and accurately ML
models can learn feature importances even in noisy settings.

####  3.1.2 Explanation Methods

The introduced measure $s$ and the three ML models $f(\mathbf{x})$ are then
used in five different XAI methods. An overview of the general XAI landscape
is provided in Section A.1, showing the different types of XAI methods and why
we chose local explanations. The selected local XAI methods can generally be
categorized in either Effect-based Methods (EM) and Additive Methods (AM). The
first XAI method describes the effects the input shift has on the output; the
latter how much the feature contributes to the output - usually in a sparse
form. The following XAI methods are used (theoretical details in Section A.2):

  * •

The gradient baseline simply takes the gradient of the input of the ML model -
a very simple approach (EM).

  * •

Improving upon that, the SG method averages the gradient over $k=10$ neighbors
sampled using k-Nearest Neighbors (kNN) (Yeh et al., 2019) (EM).

  * •

Next, the ALE-kNN (Apley & Zhu, 2019) uses a simple Gaussian conditional
distribution with a fixed $\sigma^{2}=0.2$ scaled by the feature range, a
resolution of $n_{samples}=50$ bins over the feature range and $k=10$ samples
for the kNN selection (EM).

  * •

LIME (Ribeiro et al., 2016) simply uses default values, with no modifications
(AM).

  * •

The SHAP (Lundberg & Lee, 2017) methods are used with default parameters.
Different SHAP methods, however, are used, depending on the ML model - Linear
SHAP for the linear regression, Tree SHAP for XGBoost, and Gradient SHAP for
the neural network (AM).

##  4 Results & Discussion

This section presents the evaluation results, beginning with an overview of
how noise affects the different XAI approaches, in combination with different
ML models. Sanity checks with just noise are also performed, providing a lower
empirical bar for $s$ of the evaluated systems.

###  4.1 Robustness Results

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
Level$\displaystyle R^{2}$ scoreLinear RegressionXGBoostNeural Network (a)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (b)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (c)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (d)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (e)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (f)

Figure 2: Score $s$ on toy data with varying levels of noise on the different
combinations of explainers and ML models. The shaded area is the 90th and 10th
percentile over 50 experiments with random sampling. (2(a)) $R^{2}$ score,
(2(b)) Gradient score $s$, (2(c)) SG score $s$, (2(d)) ALE-kNN score $s$,
(2(e)) LIME score $s$, and (2(f)) SHAP score $s$.

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
Level$\displaystyle R^{2}$ scoreLinear RegressionXGBoostNeural Network (a)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (b)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (c)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (d)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (e)

$0$$5\cdot 10^{-2}$$0.1$$0.15$$0.2$$0.25$$0.3$$0$$0.2$$0.4$$0.6$$0.8$$1$Noise
LevelScore $\displaystyle s$ (f)

Figure 3: Score $s$ on EAF data with varying levels of noise on the different
combinations of explainers and ML models. The shaded area is, again, the 90th
and 10th percentile over 50 experiments with random sampling. (3(a)) $R^{2}$
score, (3(b)) Gradient score $s$, (3(c)) SG score $s$, (3(d)) ALE-kNN score
$s$, (3(e)) LIME score $s$, and (3(f)) SHAP score $s$,

The core results of this paper from both datasets are shown in Figure 2 and 3,
where we show the effect of progressively applying more noise to the training
data. Various black-box models are then fitted to this perturbed training set
and XAI methods are evaluated using the scoring method proposed in Section 3.

The first graphs of either dataset, Figures 2(a) and 3(a), show the $R^{2}$
score, a metric to evaluate regression problems. The score can achieve a
maximum of $1$ if the predictions are perfect, $0$ if they predict the mean
and arbitrarily negative if the prediction is worse than the mean (Chicco et
al., 2021). These scores show that the increasing noise has, as expected,
adverse effects on the performance of the evaluation set. Furthermore, the
linear regression model is quite robust in the regime of strong noise for the
EAF, while it fails to capture any of the relations of the toy dataset. The
next graphs of the toy dataset, Figures 2(b) through 2(d) show that the
effect-based XAI methods are strongly dependent on the performance of the
trained ML models. Linear regression fails to capture most relations while
XGBoost and the neural network work quite well, especially in regimes of low
noise using a robust explainer like ALE-kNN. The additive methods, shown in
Figures 2(e) and 2(f), are not able to capture the sensitivity present in the
ground truth $\mathbf{w}^{*}$ even without noise.

Similarly, the EAF results show that the explainer performance recorded is
coupled with the ML model performance, as the effect-based methods fall with
increasing noise. This rising noise plays, again, a key role in the lowering
of the scores of the XGBoost and neural networks for the more robust XAI
methods, SG and ALE-kNN, as seen in Figures 3(c) and 3(d). These two and the
raw gradient of Figure 3(b) show two further, interesting findings: first,
when the linear regression performance is good, the explainer score of the
linear regresssor is quite constant over the range of noise levels. Second,
most ML models needed some initial noise on this dataset to start modeling the
relations well, which is especially pronounced in Figure 3(d), where all ML
models rise slightly when adding a bit of noise. This could be due to a
regularizing effect of the noise on the gradients, effectively creating more
truthful averages of gradients when adding noise. The additive methods are
again not as effective in explaining the ground truth sensitivity as evident
in Figures 3(e) and 3(f)

###  4.2 Random baseline: Empirical lower bounds for Results

We additionally empirically calculate the random lower bounds using noisy data
to make sure that the evaluation measures worked. This check was performed by
training the ML model on the correct, slightly perturbed data using a noise
level of $l=0.05$. The ML model was then given random evaluation data, on
which the scores for the interpreters were calculated. The evaluation dataset
was set to $x_{ij}\sim\mathcal{N}(\mathbf{\mu}_{j},\mathbf{\sigma}_{j})$,
where $\mathbf{\mu}_{j}$ and $\mathbf{\sigma}_{j}$ are the mean and standard
deviation of the training dataset; the used dataset is the toy dataset. The
results for this perturbation can be seen in Table 1, where all XAI methods
approached a score of about $0.5$ to $0.6$, indicating, that this is indeed
similar to the worst results observed above.

Table 1: Results of sanity check with Gaussian noise as evaluation inputs on
the toy dataset ($\pm$ one standard deviation).
Model Name | $R^{2}$ | Grad | SG | ALE kNN | LIME | SHAP
---|---|---|---|---|---|---
Linear Regression |  -0.11$\pm$0.08 |  0.62$\pm$0.05 |  0.62$\pm$0.05 |  0.62$\pm$0.05 |  0.50$\pm$0.05 |  0.50$\pm$0.04
Neural Network |  -1.05$\pm$0.33 |  0.48$\pm$0.06 |  0.56$\pm$0.02 |  0.53$\pm$0.06 |  0.53$\pm$0.05 |  0.50$\pm$0.05
XGBoost |  -0.93$\pm$0.29 |  0.50$\pm$0.06 |  0.55$\pm$0.05 |  0.56$\pm$0.05 |  0.52$\pm$0.05 |  0.51$\pm$0.06
Table 2: Results of sanity check using Gaussian noise as training inputs on
the toy dataset ($\pm$ one standard deviation).
Model Name | $R^{2}$ | Grad | SG | ALE kNN | LIME | SHAP
---|---|---|---|---|---|---
Linear Regression |  -0.02$\pm$0.03 |  0.49$\pm$0.13 |  0.49$\pm$0.13 |  0.49$\pm$0.13 |  0.52$\pm$0.22 |  0.49$\pm$0.22
Neural Network |  -0.03$\pm$0.08 |  0.48$\pm$0.11 |  0.53$\pm$0.01 |  0.47$\pm$0.09 |  0.51$\pm$0.16 |  0.51$\pm$0.15
XGBoost |  -0.52$\pm$0.20 |  0.50$\pm$0.05 |  0.50$\pm$0.12 |  0.48$\pm$0.13 |  0.54$\pm$0.14 |  0.49$\pm$0.08

In the next experiment, we trained the ML model on completely noisy data,
again with $x_{ij}\sim\mathcal{N}(\mathbf{\mu}_{j},\mathbf{\sigma}_{j})$, but
scored on the correct evaluation set. This produced, expectedly, even worse
performance metrics. This indicates that the random baseline for this scoring
method is around $s=0.55$.

###  4.3 Discussion

This investigation of the robustness of XAI methods showed that these
approaches are influenced by the noise and the predictive performance. This is
especially true for gradient-based XAI approaches. The SG as well as the
cohort-based ALE-kNN works well. The properties of the custom cohort approach
foster the correctness of the interpretations, as there is an uplift in
performance, especially in combination with the tree-based XGBoost system.
Both LIME and SHAP are, however, in the category of additive methods. This
makes them unsuitable for interpretations that call for scores reflecting the
input sensitivity on the output, i.e. reflecting the feature _effect_ of the
output.

The low performance of the additive feature importance scores can also be
partially attributed to the metric preferring the effect-based scoring
methods, as the related work by Yeh et al. (2019) notes. Notwithstanding, we
showed that effect-based scoring methods are highly dependent on the
performance of the ML model to accurately reflect the ground truth importance
scores $w^{*}_{ij}$ and that a single gradient of the ML model is often not
enough to estimate them correctly.

##  5 Conclusion & Future Work

We showed how different XAI methods are affected based on the predictive
performance of the ML models. The focus of this work was on model-agnostic
post-hoc explanations for local data samples, as these could be evaluated
using numeric, data-driven approaches. Of these XAI methods, SHAP, LIME, SG,
and a local version of ALE were chosen. These were evaluated using a novel
evaluation process focused on scaling the feature importance scores to a
similar magnitude within one sample, then normalizing them to the same range
as the ground truth effects, and, finally, calculating the distance to the
ground truth effects reference. This ground truth was generated using a
chemical simulation of an EAF model, providing the necessary ground truth
sensitivity for comparison and evaluation. This data-generating distribution
was chosen based on the maturity of the EAF models for these real-world
processes as well as current interest in the technology due to its promise of
cleaner steel production. Additionally, a toy example was initially used to
test the approaches on a limited and known nonlinear dataset.

Noise analysis over a range of perturbances of the initial dataset was
performed using this evaluation methodology. The resulting analysis lends the
conclusion that XGBoost in combination with smooth gradient-based XAI methods
can approximate both the target values as well as the ground truth
interpretations very well, even in noisy environments. LIME and SHAP, however,
were not as successful in correctly finding the ground truth feature
importance scores, probably due to their differing approaches to the
interpretation of the feature importance scores. Some of these XAI methods and
ML models, furthermore, showed a higher variance, indicating that they varied
between sampling runs and were affected by the high noise. Sanity checks on
the validity of the evaluation process were carried out using noise, first, as
evaluation data, and then as training data. Both tests showed that the scores
tend to be around $s=0.55$ since the ML models cannot learn the importance at
all.

###  5.1 Future Work

There is an apparent need to quantify the uncertainty of the XAI methods, as
this high variability of the feature importance scores of ML models with high
uncertainty in noisy environments distorted the feature interpretation
significantly. There are already works investigating such uncertainties for
XAI (Löfström et al., 2024; Zhao et al., 2021; Slack et al., 2021), however,
none of these address the effect-based approaches where the feature importance
score reflects the change of the output with respect to the input.

The empirical evaluation of feature importance scores, especially from LIME
and SHAP could also be further investigated by the comparison of different
metrics on a ground truth dataset. Further improvements on the measures of
infidelity and sensitivity (Yeh et al., 2019), combined with the consideration
of a known ground truth feature importance and deeper analysis of noise could
also lead to further understanding of robustness and failure cases of XAI.

## Acknowledgements

The financial support by the Austrian Federal Ministry of Labour and Economy,
the National Foundation for Research, Technology and Development and the
Christian Doppler Research Association is gratefully acknowledged. We
furthermore thankfully acknowledge the financial support of the project by
voestalpine Stahl GmbH. The authors gratefully acknowledge the funding support
of K1-MET GmbH, whose research program is supported by COMET (Competence
Center for Excellent Technologies), the Austrian program for competence
centers. COMET is funded by the Austrian ministries BMK and BMDW, the Federal
States of Upper Austria, Tyrol, and Styria, and the Styrian Business Promotion
Agency (SFG).

## References

  * Alvarez-Melis & Jaakkola (2018) Alvarez-Melis, D. and Jaakkola, T. S.  On the robustness of interpretability methods.  _CoRR_ , abs/1806.08049, 2018.  URL http://arxiv.org/abs/1806.08049.
  * Apley & Zhu (2019) Apley, D. W. and Zhu, J.  Visualizing the effects of predictor variables in black box supervised learning models, 2019.
  * Basu et al. (2008) Basu, S., Lahiri, A. K., and Seetharaman, S.  Activity of iron oxide in steelmaking slag.  _Metallurgical and Materials Transactions B_ , 39(3):447–456, May 2008.  ISSN 1543-1916.  doi: 10.1007/s11663-008-9148-4.  URL http://dx.doi.org/10.1007/s11663-008-9148-4.
  * Bhatt et al. (2020) Bhatt, U., Weller, A., and Moura, J. M. F.  Evaluating and aggregating feature-based model explanations, 2020.
  * Billings et al. (1979) Billings, S., Boland, F., and Nicholson, H.  Electric arc furnace modelling and control.  _Automatica_ , 15(2):137–148, 1979.  ISSN 0005-1098.  doi: https://doi.org/10.1016/0005-1098(79)90065-7.  URL https://www.sciencedirect.com/science/article/pii/0005109879900657.
  * Binti Ahmad Dzulfakhar et al. (2023) Binti Ahmad Dzulfakhar, A. S., Charaschanya, T., Charos, A., Duru, N., Khan, Fatima Elzahra Lee, S., Sim, E. Y., Tonus, A., Walker, P., and Woolley, R.  Eaf modelling: Process design of key reactor unit, November 2023.  URL https://github.com/SayYoungMan/EAF_modelling.
  * Boulet et al. (2003) Boulet, B., Lalli, G., and Ajersch, M.  Modeling and control of an electric arc furnace.  In _Proceedings of the 2003 American Control Conference, 2003._ , volume 4, pp. 3060–3064 vol.4, 2003.  doi: 10.1109/ACC.2003.1243998.
  * Brier (1950) Brier, G. W.  Verification of forecasts in terms of probabality.  _Monthly Weather Review_ , 78(1):1–3, January 1950.  ISSN 1520-0493.  doi: 10.1175/1520-0493(1950)078¡0001:vofeit¿2.0.co;2.  URL http://dx.doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2.
  * Chen & Guestrin (2016) Chen, T. and Guestrin, C.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16. ACM, August 2016.  doi: 10.1145/2939672.2939785.  URL http://dx.doi.org/10.1145/2939672.2939785.
  * Chicco et al. (2021) Chicco, D., Warrens, M. J., and Jurman, G.  The coefficient of determination r-squared is more informative than smape, mae, mape, mse and rmse in regression analysis evaluation.  _PeerJ Computer Science_ , 7:e623, July 2021.  ISSN 2376-5992.  doi: 10.7717/peerj-cs.623.  URL http://dx.doi.org/10.7717/peerj-cs.623.
  * Crabbé & van der Schaar (2023) Crabbé, J. and van der Schaar, M.  Evaluating the robustness of interpretability methods through explanation invariance and equivariance, 2023.
  * De Ras et al. (2019) De Ras, K., Van de Vijver, R., Galvita, V. V., Marin, G. B., and Van Geem, K. M.  Carbon capture and utilization in the steel industry: challenges and opportunities for chemical engineering.  _Current Opinion in Chemical Engineering_ , 26:81–87, 2019.  ISSN 2211-3398.  doi: https://doi.org/10.1016/j.coche.2019.09.001.  URL https://www.sciencedirect.com/science/article/pii/S221133981930036X.
  * Doshi-Velez & Kim (2017) Doshi-Velez, F. and Kim, B.  Towards a rigorous science of interpretable machine learning, 2017.
  * Huang et al. (2023) Huang, W., Zhao, X., Jin, G., and Huang, X.  Safari: Versatile and efficient evaluations for robustness of interpretability, 2023.
  * Lundberg & Lee (2017) Lundberg, S. M. and Lee, S.-I.  A unified approach to interpreting model predictions.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 30_ , pp. 4765–4774. Curran Associates, Inc., 2017.
  * Löfström et al. (2024) Löfström, H., Löfström, T., Johansson, U., and Sönströd, C.  Calibrated explanations: With uncertainty information and counterfactuals.  _Expert Systems with Applications_ , 246:123154, July 2024.  ISSN 0957-4174.  doi: 10.1016/j.eswa.2024.123154.  URL http://dx.doi.org/10.1016/j.eswa.2024.123154.
  * Munn & Pitman (2022) Munn, M. and Pitman, D.  _Explainable AI for practitioners_.  O’Reilly Media, Sebastopol, CA, November 2022.
  * Murdoch et al. (2019) Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., and Yu, B.  Definitions, methods, and applications in interpretable machine learning.  _Proceedings of the National Academy of Sciences_ , 116(44):22071–22080, 2019.  doi: 10.1073/pnas.1900654116.  URL https://www.pnas.org/doi/abs/10.1073/pnas.1900654116.
  * Naser (2021) Naser, M.  An engineer’s guide to explainable artificial intelligence and interpretable machine learning: Navigating causality, forced goodness, and the false perception of inference.  _Automation in Construction_ , 129:103821, September 2021.  ISSN 0926-5805.  doi: 10.1016/j.autcon.2021.103821.  URL http://dx.doi.org/10.1016/j.autcon.2021.103821.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  _PyTorch: an imperative style, high-performance deep learning library_.  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * Ribeiro et al. (2016) Ribeiro, M. T., Singh, S., and Guestrin, C.  ”why should I trust you?”: Explaining the predictions of any classifier.  _CoRR_ , abs/1602.04938, 2016.  URL http://arxiv.org/abs/1602.04938.
  * Ribeiro et al. (2018) Ribeiro, M. T., Singh, S., and Guestrin, C.  Anchors: High-precision model-agnostic explanations.  In _AAAI Conference on Artificial Intelligence (AAAI)_ , 2018.
  * Shrikumar et al. (2017) Shrikumar, A., Greenside, P., and Kundaje, A.  Learning important features through propagating activation differences.  _CoRR_ , abs/1704.02685, 2017.  URL http://arxiv.org/abs/1704.02685.
  * Slack et al. (2021) Slack, D. Z., Hilgard, S., Singh, S., and Lakkaraju, H.  Reliable post hoc explanations: Modeling uncertainty in explainability.  In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=YUEFlzlG_0c.
  * Sun et al. (2020) Sun, Y., Chockler, H., Huang, X., and Kroening, D.  _Explaining Image Classifiers Using Statistical Fault Localization_ , pp. 391–406.  Springer International Publishing, 2020.  ISBN 9783030586041.  doi: 10.1007/978-3-030-58604-1˙24.  URL http://dx.doi.org/10.1007/978-3-030-58604-1_24.
  * Yeh et al. (2019) Yeh, C.-K., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I., and Ravikumar, P.  On the (in)fidelity and sensitivity for explanations, 2019.
  * Zhang & Fruehan (1995) Zhang, Y. and Fruehan, R. J.  Effect of gas type and pressure on slag foaming.  _Metallurgical and Materials Transactions B_ , 26(5):1088–1091, October 1995.  ISSN 1543-1916.  doi: 10.1007/bf02654112.  URL http://dx.doi.org/10.1007/BF02654112.
  * Zhao et al. (2021) Zhao, X., Huang, W., Huang, X., Robu, V., and Flynn, D.  Baylime: Bayesian local interpretable model-agnostic explanations, 2021.

##  Appendix A Background & Related Work

###  A.1 Interpretable and Explainable ML

Before diving into the XAI methods themselves, the notion of interpretability
and explainability of ML models has to be examined first. These terms span a
wide field of approaches, with the common goal of enabling the end users or
creators of a system to foster their understanding on why, and sometimes how
ML models arrive at certain conclusions. However, the presentation of these
explanations can vary widely, from visual explainers to mathematical formulas.
This paper will focus on the latter, as we will take advantage of that they
are easier to compare and evaluate numerically in a data-driven environment.
Most methods can, despite this variety in modalities, generally be categorized
into either interpretable systems or so-called post-hoc methods (Murdoch et
al., 2019; Doshi-Velez & Kim, 2017).

####  A.1.1 Interpretable Systems

ML models can achieve inherent interpretability through different modalities,
which, however, share the constraint of understandable processes that can be
comprehended by a single person through a reasonable timeframe and effort.
This limitation, coined by Murdoch et al. (2019) as “simulatability”, greatly
inhibits both the choice of system and the function space of the considered
models. This leads, depending on the reduction in expressive power, to a loss
of predictive performance and has to be considered before choosing such a
model. The (numerical) evaluation of these systems is furthermore hindered by
the wide-ranging type of approaches and often depends on human evaluation of
these systems (Murdoch et al., 2019).

One of the two common ways to increase the range of available systems again
can be the introduction of sparsity, where there is an initial, more complex
function and feature space that can later be reduced by fitting the model to
the data and reducing the set of expressions needed for decisionmaking.
Similarly, the model could be broken up into smaller modules, where each is
very well understood on its own - a popular choice for these modularized,
linear components would be ensemble-based approaches with only a few models
(Murdoch et al., 2019).

####  A.1.2 Post-hoc Interpretability

These interpretable approaches, however, do not foster the understanding of
the model if there are no alternatives to the current model due to performance
limitations or other constraining factors limiting choice. In these cases,
post-hoc approaches can offer interpretability for black-box models that would
be too complicated for humans to comprehend easily. These can be further
categorized into global (dataset-level) and local (instance-level) approaches
to interpretability. Global explanations are of interest for an overview of
how the model considers certain features in general or for statistical
analysis. If the interest lies in specific examples, or even failure cases
such as adversarial examples, where a prediction can be maliciously flipped, a
local explanation is often the needed approach (Murdoch et al., 2019).
Furthermore, one can also group or cluster features and analyze these cohorts
with global methods, achieving a similar overview-level interpretation on a
smaller neighborhood (Munn & Pitman, 2022). All levels utilize similar
techniques to arrive at solutions explaining the decision, with the core
concept being called feature scores. These methods measure how influential a
certain feature is, either on the local or global level, and assign it a score
based on the perturbances it might achieve. These methods such as partial
dependence plots (PDPs) and ALE (Apley & Zhu, 2019) (in the 1D case), or their
local counterpart, SG, describe how varying a single feature affects the
output, with the assumption that features of interest are not correlated. We
will call these methods effect-based methods throughout this paper. Feature
interactions, on the other hand, analyze exactly these contributions: how two
or more features influence the output in conjunction (Naser, 2021). Other
methods, such as LIME utilize simple, local representations of models as
surrogates to better explain the behavior of models in certain neighborhoods
(Ribeiro et al., 2016; Munn & Pitman, 2022). Both LIME and SHAP, another
explanation methodology, form a group of systems we will call feature
importance scores, where each feature is assigned a certain contribution to
the output.

###  A.2 Chosen Approaches

Of these methods, the local and cohort-based post-hoc approaches for feature
importance are of interest for the evaluation task at hand, as these allow for
comparison to known feature influences from the simulated data-generating
process. This is why we chose ALE on a small cohort chosen using kNN (ALE-
kNN), SG, LIME and SHAP.

####  A.2.1 ALE-kNN and SG

The intuition behind ALE (Apley & Zhu, 2019) is to simply accumulate the local
effects (derivatives) of a function (model) weighted by a certain prior. This
prior is the main difference to PDPs, as these do not consider the conditional
distribution $p_{2|1}(x_{2}|x_{1})$ of the sample, just the marginal over the
selected variable. This results in a very intuitive and straightforward
formulation of the ALE effect $f_{1,ALE}(x_{1})$ on $x_{1}$ for a two-
dimensional function ($d=2$, $f(x_{1},x_{2})$) as

$f_{1,ALE}(x_{1})=\int_{x_{min,1}}^{x_{1}}\int
p_{2|1}(x_{2}|z_{1})f^{1}(z_{1},x_{2})dx_{2}dz_{1}-\mathrm{constant}$ (7)

where $f^{1}(x_{1},x_{2})$ is the first derivative with respect to the
variable of interest ($\frac{\partial f(x_{1},x_{2})}{\partial x_{1}}$).
$x_{min,1}$ is chosen such that it just touches the lower support of the
conditional distribution, approximating the integral over the whole
distribution. The first integration variable $x_{2}$ integrates over all
samples of the second, unrelated feature (or more in real-world examples),
while $z_{1}$ integrates the effect over the conditional distribution. The
resulting uncentered ALE is retrieved when ignoring the constant, which can be
centered by subtracting the average effect. This method is usually applied on
a dataset level and generates visual plots, showing the influence of variables
over the whole range of the input (Apley & Zhu, 2019).

However, to apply this method to a local data sample, the $k$ nearest
neighbors of the sample are chosen, and the resulting local cohort (Munn &
Pitman, 2022) is used as the basis for the uncentered ALE of this sample. This
local cohort version of the ALE is very closely related to the SG method,
where the gradients or, more general, the explanations are averaged over a
small neighborhood (Yeh et al., 2019). The version using ALEs has, however,
the advantage of the introduction of the conditional distribution, effectively
weighting the effect locally.

####  A.2.2 LIME

LIME (Ribeiro et al., 2016), offers feature importance scores through local
surrogate models, learning a sparse representation of the neighborhood. This
approach combines almost all concepts outlined in the overview:
simulatability, sparsity, local cohorts, and surrogate models. The resulting
method returns feature importance scores. The algorithm itself consists of
three major steps: sample selection, feature importance calculation, and a
final selection process called “submodular pick”. For local explanations, only
the first two steps are necessary.

Sample selection searches a training dataset for similar datapoints using a
configurable distance metric, and evaluates the model for each. This small set
is then used for fitting the surrogate model to the local neighborhood. A
linear regressor is usually used as a local surrogate. The weights of the
resulting model are then directly utilized as the feature importance scores
(Ribeiro et al., 2016).

####  A.2.3 SHAP

The SHAP (Lundberg & Lee, 2017) framework unifies multiple methods for
calculating SHAP values, a measure of feature importance. They are based on
Shapley values, the only set of values satisfying the three desirable
properties for additive feature attribution. This scoring method can be
described as a simple linear model of the form

$g(z^{\prime})=\phi_{0}+\sum_{i=i}^{M}\phi_{i}z_{i}^{\prime},$ (8)

where the model is similar to the function at $g(z^{\prime})\approx
f(h_{x}(z^{\prime}))$ and $z^{\prime}\in\\{0,1\\}^{M}$ being a binary
representation of $x^{\prime}$, a simplified input, using the mapping function
$h_{x}(z^{\prime})=x$. The parameters $\phi_{i}$ describe the actual feature
attribution.

While LIME follows the same structure, it does not necessarily fulfill the
needed theoretical properties. Lundberg & Lee (2017) introduce, based on this
notion, three properties. These properties are

  1. 1.

Local accuracy: the linear model matches the complex model in the local point;

  2. 2.

Missingness: the instances with $x_{i}^{\prime}=0$ have no impact, so
$\phi_{i}=0$; and

  3. 3.

Consistency: a change of the model output w.r.t. one feature resulting in an
output increase should change the attribution $\phi_{i}$ only positively.

The exact computation of Shapley values is quite challenging, as it is NP-hard
- there are, however, heuristics and simplifications for existing models such
as Kernel SHAP, which is model agnostic. Other methods such as Deep SHAP or
Tree SHAP utilize certain model-specific aspects to speed up the computation
and provide more exact estimates. All of these approaches are united in the
SHAP package (Lundberg & Lee, 2017).

###  A.3 Related Work

Most of the mentioned methods prove their value either using human evaluation
(Ribeiro et al., 2016; Lundberg & Lee, 2017; Ribeiro et al., 2018), with the
use of toy examples (Ribeiro et al., 2016; Apley & Zhu, 2019; Shrikumar et
al., 2017; Ribeiro et al., 2018) or improvements in theoretical properties
(Lundberg & Lee, 2017). None of these works consider robustness to
perturbances, unstable outputs, or other chaotic behavior. Some introductions
of newer methods, however, are aware of these problems and decide to forego
human evaluation in favor of data-driven analysis and robustness studies (Sun
et al., 2020). Further works augment this missing robustness consideration in
XAI, studying these problems in more depth (Alvarez-Melis & Jaakkola, 2018;
Yeh et al., 2019; Bhatt et al., 2020; Huang et al., 2023; Crabbé & van der
Schaar, 2023). Alvarez-Melis & Jaakkola (2018) and Yeh et al. (2019) tackle
these problems through the maximal local sensitivity, where they search for
the maximal change of the explanation to the instance within a neighborhood.
This method asses the stability of local explanations and can thus be seen as
a measurement for robustness, especially on noisy data. Other methods try to
reduce the sensitivity through aggregation by combining different metrics into
a more resilient measure (Bhatt et al., 2020). Yeh et al. (2019) introduces,
besides the sensitivity of the explanation, the infidelity of explainable
approaches, where they measure the squared difference from a projected
explanation ($\mathbf{I^{\mathrm{T}}}\Phi(f,\mathbf{x})$) to the difference of
the output when subtracting the projection from the input
($f(\mathbf{x})-f(\mathbf{x}-\mathbf{I})$) using

$\mathrm{INFD}(\Phi,f,\mathbf{x})=\mathbb{E}_{\mathbf{I}\sim\mu_{\mathbf{I}}}\left[\left(\mathbf{I^{\mathrm{T}}}\Phi(f,\mathbf{x})-(f(\mathbf{x})-f(\mathbf{x}-\mathbf{I}))\right)^{2}\right]$
(9)

where $\mathbf{I}\in\mathbb{R}^{d}$ representing the significant perturbances
around $\mathbf{x}\in\mathbb{R}^{d}$ as a random variable, $\mu_{\mathbf{I}}$
the distribution of these perturbances, $f(\mathbf{x})$ the learned model
function and $\Phi(f,\mathbf{x})$ the corresponding explaination. While this
measure is more data-driven and universal, it still suffers from only using an
approximation of the ideal explanation. Furthermore, no studies regarding the
robustness with regard to added noise or known noise levels have been
performed.