4cm3cm3cm3cm [4cm]Figures/Logo/logo_sapienza_new.jpg Sapienza University of
Rome Department of Computer, Control and Management Engineering PhD in
Computer Engineering Thesis For The Degree Of Doctor Of Philosophy Explaining
Deep Neural Networks by Leveraging Intrinsic Methods Candidate []Biagio La
Rosa Academic Year 2023-2024 (XXXVI cycle)

#### Abstract

Deep neural networks have been pivotal in driving AI advancements over the
past decade, revolutionizing domains such as gaming, biology, autonomous
systems, and voice and text assistants. Despite their impact, these networks
are often regarded as black-box models due to their intricate structures and
the absence of explanations for their decisions. This opacity poses a
significant challenge to AI systems’ wider adoption and trustworthiness. This
thesis addresses this issue by contributing to the field of eXplainable AI,
focusing on enhancing the interpretability of deep neural networks.

The core contributions lie in introducing novel techniques aimed at making
these networks more interpretable by leveraging an analysis of their inner
workings. Specifically, the contributions are threefold. Firstly, the thesis
introduces designs for self-explanatory deep neural networks, such as the
integration of external memory for interpretability purposes and the usage of
prototype and constraint-based layers across several domains. These proposed
architectures are specifically designed to preserve most of the black-box
networks, thereby maintaining or improving their performance. Secondly, this
research delves into novel investigations on neurons within trained deep
neural networks, shedding light on overlooked phenomena related to their
activation values. Lastly, the thesis conducts an analysis of the application
of explanatory techniques in the field of visual analytics, exploring the
maturity of their adoption and the potential of these systems to convey
explanations to users effectively.

In summary, this thesis contributes to the growing field of Explainable AI by
proposing intrinsic techniques to enhance the interpretability of deep neural
networks. By mitigating the opacity issue of deep neural networks and applying
them to several different applications, the research aims to foster trust in
AI systems and facilitate their wider adoption across several applications.

#### Acknowledgements

I strongly believe that the goals achieved in life need to be shared with all
the people you interact with. My life has been full of sliding doors, chaos,
dreams, failures, and discoveries, and all of them involved other persons in
the process. Therefore, thank you to all the people who met me in their lives,
no matter if you like or hate me: each of you pushed me to become who I am.

Special thanks for their contribution to my research path must be given to:

  * •

Prof. Roberto Capobianco, for supporting my freedom, my research, and guiding
my studies. We embarked on this mad and risky journey of explainability
together, and I really enjoyed the time under your supervision. Thank you for
pushing me to go outside my comfort zone and for all your precious
suggestions. The calls during the evening to discuss novel ideas and the long
night for the IJCAI paper are examples of memories that I will remember
forever! arigatō Roberto!

  * •

Prof. Leilani Gilpin for welcoming me as a visitor to your lab, for granting
me freedom during the months there, and for being so supportive even in tasks
where you were not supposed to be. You are a rare case in academia: your way
of supervising and the environment that you built around me allowed me to
exploit the best of my skills and reach the goals that I wanted to reach.
Therefore, thank you for everything!

  * •

KRL group for our dinners and casual chats! My first year alone as a student
and under the pandemic was hard, but thanks to your presence, I discovered
what it means to be in a lab group of smart people.

  * •

XAI-KRL group for becoming a house where discussing the field, ideas, and
opinions and grow together.

  * •

the AWARE group (Marco, Graziano, and Prof. Santucci) for introducing me to a
novel field and opening the doors to so many future directions and
collaborations.

  * •

Prof. Daniele Nardi, for the chance to be in your lab, for your support
whenever I needed it, and for being kind and nice to me! I really appreciate
everything that you have done in these last three years!

  * •

Prof. Roberto Navigli, for introducing me to the field of AI and research when
I was an undergraduate without any knowledge of AI. Your suggestions and your
supervision definitely shaped my future, transforming me from a future
software engineer to a future researcher. Your compliments and advice pushed
me to surpass myself at each step of my career until today.

Thank you, Alessio Ragno, for becoming my French partner in crime in XAI. Your
presence and our battles helped me to grow as a supervisor and as a reviewer
2. Because of you, I learned how to choose the best paper title and how to
sell ice to a polar bear. Thank you, Michela Proietti, for being patient and
kind in this last period. I’m sure you will become a great scientist; just
relax and enjoy your journey!

Additionally, I would like to thank all the students that I supervised both
for theses and projects: interacting with you allowed me to experiment and
improve my skills as a supervisor and become a better researcher. Thank you to
all the reviewers2 who rejected my papers and proposals

During this path, I had the chance to collaborate with great colleagues whose
contribution and help to this thesis are valuable in multiple aspects. Other
than my French partners Alessio and Michela, I thank Romain Giot, Graziano
Basili, Sayo M. Makinwa, Enrico Bertini, Marco Angelini, Romain Bourqui, Rino
Ragno, David Auber, and Giuseppe Santucci for contributing to the research
presented in this thesis. I hope to collaborate again with all of you in the
future.

Thank you to all my family for being supportive during my long academic
travel. I hope that today you will be proud of me and you can finally relax.
Thank you also to my friends for pushing me to prove to them that doing
research is “a job”. Finally, thank you to Roberta for being so patient during
the writing process of this thesis and for postponing your drama. Thank you
for encouraging me to make difficult decisions even if these decisions will
impact your life and future. I really appreciate your help, and I hope we will
walk together for a long time.

###### Contents

  1. Acknowledgements
  2. Nomenclature
  3. 1 Introduction
    1. 1.1 Introduction
    2. 1.2 Contributions
    3. 1.3 Thesis Organization
    4. 1.4 Related Publications
  4. I Preliminaries
    1. 2 Background
      1. 2.1 Fundamentals of Deep Neural Networks
        1. 2.1.1 Architectures
      2. 2.2 Explainable AI
        1. 2.2.1 Categorization
        2. 2.2.2 Evaluation of XAI methods
      3. 2.3 Visual Analytics
    2. 3 Related Work
      1. 3.1 Post-Hoc Methods
        1. 3.1.1 Feature Attribution
        2. 3.1.2 Learned Features
        3. 3.1.3 Explanations by examples
        4. 3.1.4 Counterfactuals
      2. 3.2 Self-Explainable Deep Neural Networks
      3. 3.3 Relations with Thesis Contribution
  5. II Self-Explainable Neural Networks
    1. 4 Prototype-based Interpretable Graph Neural Networks
      1. 4.1 Design
        1. 4.1.1 Training Process.
        2. 4.1.2 Explanations
      2. 4.2 Experiments
        1. 4.2.1 Performance
        2. 4.2.2 Explanations
      3. 4.3 Design Choices
      4. 4.4 Contributions
    2. 5 Self-Explainable Memory-Augmented Deep Neural Networks
      1. 5.1 Simplified Differentiable Neural Computers
        1. 5.1.1 Design
        2. 5.1.2 Experiments
        3. 5.1.3 Design Choices
      2. 5.2 Memory Wrap
        1. 5.2.1 Design
        2. 5.2.2 Experiments
        3. 5.2.3 Design choices
      3. 5.3 Contributions
  6. III Explaining Latent Representations
    1. 6 Graph Concept Whitening
      1. 6.1 Design
        1. 6.1.1 Alignment Process
        2. 6.1.2 Explanations
      2. 6.2 Experiments
        1. 6.2.1 Concept Selection
        2. 6.2.2 Performance
        3. 6.2.3 Explanations
      3. 6.3 Design Choices
      4. 6.4 Contributions
    2. 7 Clustered Compositional Explanations
      1. 7.1 Algorithm Design
        1. 7.1.1 Proof of Admissibility of MMESH
      2. 7.2 Metrics
      3. 7.3 Experiments
        1. 7.3.1 Motivation
        2. 7.3.2 Performance
        3. 7.3.3 Explanations
        4. 7.3.4 Metrics
      4. 7.4 Design Choices
      5. 7.5 Contributions
  7. IV Bringing Explanations to the User: Visual Analytics
    1. 8 Explaining Deep Learning in Visual Analytics Systems
      1. 8.1 Overview
        1. 8.1.1 Feature Attribution
        2. 8.1.2 Learned Features
        3. 8.1.3 Explanations by Examples and Counterfactuals
        4. 8.1.4 Model Behavior
      2. 8.2 Analysis
      3. 8.3 Towards a Better Integration
      4. 8.4 Contributions
  8. V Conclusions
    1. 9 Conclusions
      1. 9.1 Summary
      2. 9.2 Limitations
      3. 9.3 Future Research Directions

###### List of Figures

  1. 2.1 An example of a 3-layer neural network. Each node in the hidden layers is connected with all the nodes of the previous layer and all the nodes of the next layer.
  2. 2.2 A simplified version of an artificial neuron.
  3. 2.3 An unfolded view of an LSTM cell.
  4. 4.1 The architecture of PIGNN when the task is graph classification.
  5. 4.2 Avg. Fidelity and standard deviation achieved by the intrinsic method, DeepLIFT, and GNNExplainer on GCN (a) and GAT(b) models across several datasets.
  6. (a)
  7. (b)
  8. 4.3 Avg. Fidelity and standard deviation reached by the intrinsic method, DeepLIFT, and GNNExplainer on GIN models across several datasets.
  9. 4.4 Sub-graphs associated with the prototypes learned by PIGNN based on GCN on the Ba-2Motifs dataset. The most activated node is marked with a red borderline. The blue shade encodes the importance of the node for the subgraph.
  10. 4.5 Visualization of explanation for a prediction. The red border indicates the analyzed node; the Input Activations column reports the similarity between the graph’s nodes and the prototype displayed in the same row in the Prototype Activation column.
  11. 4.6 Accuracy of PIGNN when varying the number of prototypes.
  12. 4.7 Visualization comparison between projecting the similarities towards all the nodes in the graph and only towards the k-hop most important subgraph.
  13. (a)
  14. (b)
  15. 4.8 Avg. Accuracy and standard deviation of PIGNN when jointly trained with the feature extractor (Classic), trained after a pre-training on the same dataset (Pre-training), and fine-tuning a model trained in a different dataset via transfer learning (Transfer)
  16. 5.1 Design of SDNC. The blue boxes represent the read-and-write history. $<C_{i},x_{j}>$ indicates that feature $x_{j}$ is stored in cell $C_{i}$. The ranking sorts the subsequence of the input based on their importance.
  17. 5.2 Example of a 5-step modified T-maze, where the starting position (in white) indicates the relevant cell (in yellow).
  18. 5.3 Cumulative reward and standard deviation of ten agents trained on the T-Maze problem.
  19. 5.4 Matching accuracy score and standard deviation of ten agents trained on the T-Maze problem.
  20. 5.5 Example outputs on the Story Cloze Test. A relevance score is associated with each premise. The blue color indicates the best premise, while the orange indicates the worst one.
  21. 5.6 Output of LIME [197] applied to a couple of samples from the Story Cloze Test dataset.
  22. 5.7 Architecture of a deep neural network augmented with Memory Wrap.
  23. 5.8 Illustration of a snapshot of the memory of Memory Wrap. Each row can be associated with one of the three sets $\bm{M_{0}}$, $\bm{M_{=}}$, and $\bm{M_{\neq}}$ based on its associated content weight $w_{c}$ and prediction.
  24. 5.9 Performance comparison between memory-based modules.
  25. 5.10 Performance comparison between k-NN variants and Memory Wrap variants.
  26. 5.11 Avg. accuracy and standard deviation over 15 runs of the black-box model, the full Memory Wrap, and its ablated version on subsets of CIFAR10.
  27. 5.12 Avg. accuracy and standard deviation over 15 runs of the black-box model, the full Memory Wrap, and its ablated version on subsets of CINIC10.
  28. 5.13 Avg. Input representativeness scores and standard deviation reached by CHP (gray), KNN* (pink), the random baseline (green), and Memory Wrap (blue). Lower is better.
  29. 5.14 Avg. Prediction representativeness scores and standard deviation reached by CHP (gray), KNN* (pink), the random baseline (green), and Memory Wrap (blue). Lower is better.
  30. 5.15 Inputs (first rows) from SVHN (a) and CIFAR10 (b), their associated predictions, and an overview of the samples in the memory set that have an active influence on the decision process (second row).
  31. (a)
  32. (b)
  33. 5.16 An example of a model biased towards the background. The figure shows how the inspection of the memory samples used by the network makes clear the reasons behind the bad performance of the network.
  34. 5.17 Comparison between different encoders trained using Cosine Similarity (solid lines) and $L_{2}$ distance (dashed lines) on a subset SVHN dataset.
  35. 6.1 Separability score of the latent space of a black-box model and its interpretable version. Lower is better.
  36. 6.2 Avg. importance scores assigned to five concepts in the layers of a GAT model trained on BBBP.
  37. 6.3 Feature attribution for the concept #Heteroatoms in the BBBP dataset.
  38. 7.1 Explanations in terms of recognized concepts and sample examples of different activation ranges for unit #1.
  39. 7.2 Visualization of information used by MMESH: the size of the concept mask (white), size of the intersection (yellow), minimum extension (red bounding box), maximum extension (blue bounding box).
  40. 7.3 Percentage of labels’ category associated with the labels returned by CoEx over different activation ranges. Left: ranges from quantile to infinite. Right: range from 0 to quantile. Results are computed over 100 randomly extracted units.
  41. 7.4 Examples of specialization (left) and polysemy (right). Neuron #368 (right) recognizes unrelated concepts among different activation ranges (windmill, closet, amusement park).
  42. 8.1 Example of complex visualization in VA systems of attention flow.
  43. 8.2 Example of abstraction in VA system.
  44. 8.3 Example of interface to steer a model to add, delete, and revise prototypes. All edits are revertable and traceable in the editing history. Image from Ming et al. [157] © 2019 IEEE
  45. 8.4 View to interact with a latent representation. The user can assign values to dimensions of the latent space in the parallel coordinate plot, and the system will adjust the other views accordingly. Image from Wang et al. [258] ©2020 IEEE
  46. 8.5 Distribution of target users in the analyzed VA systems
  47. 8.6 Summary of user support for different XAI categories. The edge width is proportional to the number of systems connecting users and categories. The opacity of XAI categories is proportional to the total number of contributions.
  48. 8.7 Target domain of VA systems. Research refers to research papers targeting general applications.
  49. 8.8 Distribution of interactions supported for the user.
  50. 8.9 Distribution of evaluation procedures used in the analyzed systems.
  51. 8.10  (a) Explanation categories and (b) models coverage over the years.
  52. (a)
  53. (b)

###### List of Tables

  1. 4.1 Avg. Accuracy and standard deviation over 15 runs of GNNs with and without prototype layers on synthetic datasets.
  2. 4.2 Avg. Accuracy and standard deviation over 15 runs of GNNs with and without prototype layers on molecular datasets.
  3. 4.3 Avg. Accuracy and standard deviation over 15 runs of GNNs with and without prototype layers on citation datasets.
  4. 4.4 Explanations performances on big datasets. $Fidelity+^{prob}$ and $Fidelity+^{ROC-AUC}$ (Equation 2.24) are reported for the self-interpretable models and the extrinsic methods on the black-box GCN model.
  5. 4.5 Fidelity comparison between projecting the similarities towards all the nodes in the graph and only towards the k-hop most important subgraph.
  6. 4.6 Performance comparison between the full and the reduced loss applied to PIGNN.
  7. 4.7 Examples of specific cases when the reduced loss can work better.
  8. 4.8 Interpretability scores achieved by PIGNN when using the full or reduced loss.
  9. 5.1 Average test Accuracy reached by the models in the testing Story Cloze Test dataset.
  10. 5.2 Avg. fidelity over ten runs on the Story Cloze Test when feeding the model with only a random, the best and the worst premise.
  11. 5.3 Accuracy and $Fidelity-^{acc}$ when the model is untrained and then labels are shuffled.
  12. 5.4 Avg. fidelity obtained on the development Story Cloze Test dataset using different $topk^{r}$ values.
  13. 5.5 Avg. accuracy and standard deviation of the baselines and Memory Wrap when the training datasets are the whole SVHN, CIFAR10 and CINIC10 datasets.
  14. 5.6 Avg. and standard deviation of the matching accuracy reached over 15 runs by the sample in the memory set with the highest sparse content-based attention weight (Top), the example with the lowest weight but greater than zero (Bottom) and a random sample (Random)
  15. 5.7 Avg. IM1, IM2, and IIM1 scores on the SVHN dataset reached by Memory Wrap and a prototype-based method. Lower is better.
  16. 5.8 Avg. and standard deviation of the Pearson product-moment correlation coefficients between the correctness of the predictions, the number of counterfactuals (number), and the position of the first counterfactual (position).
  17. 5.9 Accuracy reached by the model on SVHN when the sample with the highest weight in the memory set is a counterfactual (Top Counter) and when there are no counterfactuals at all (No Counter).
  18. 5.10 Avg. accuracy and standard deviation over five runs Memory Wrap using different selection mechanisms to select samples in the memory set.
  19. 5.11 Avg. accuracy and standard deviation over five runs of Memory Wrap trained using a variable number of samples in memory, when the training dataset is a subset of SVHN.
  20. 6.1 Avg. Accuracy and ROC-AUC over 15 runs on ClinTox when adding one concept at a time for the GCW layers. We highlight the best results of combined accuracy and ROC-AUC.
  21. 6.2 Avg. ROC-AUC and standard deviation over 15 runs before and after adding GCW to the models.
  22. 6.3 Avg. and standard deviation of the Fidelity+ score of feature attributions computed on both black-boxes and models employing GCW layers.
  23. 6.4 Avg. ROC-AUC and standard deviation reached by models including GCW using different activation modes.
  24. 6.5 Avg. ROC-AUC and standard deviation of models when GCW replaces several types of normalization layers.
  25. 7.1 Avg. and standard deviation of steps required to reach the optimal solution within the beam.
  26. 7.2 Avg. and Std Dev. across metrics reached by NetDissect, CoEx, and Clustered Compositional Explanations using Ade20K as a concept dataset.
  27. 7.3 Avg. and Std Dev. across metrics reached by NetDissect, CoEx, and Clustered Compositional Explanations on ResNet. Concepts are retrieved from the Pascal dataset.
  28. 7.4 Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) applied to ResNet pre-trained on ImageNet. Results are computed for 50 randomly extracted units.
  29. 7.5 Percentage of predictions changed when the activations of a given cluster are masked.
  30. 7.6 Percentage of unspecialized and weakly specialized activation ranges in ResNet18 (ReLU Yes) and DenseNet161(ReLU No).
  31. 7.7 Avg. and Std Dev. of the scores obtained by optimizing the algorithm using Detection Accuracy. Underlined results are those that reach lower scores compared to when using the IoU score. Bold results are those that reach higher scores.
  32. 7.8 Avg. and Std Dev. of the proposed metrics when using a variable number of clusters. Results are computed for 50 randomly extracted units.
  33. 7.9 Avg. and standard deviation of visited states per unit. Results are computed for 100 randomly extracted units.

### Nomenclature

  * $\circ$

Element-wise Multiplication

  * $\lVert\rVert^{n}$

$n$-Norm Distance

  * $\lVert\rVert$

Square Distance

  * $\mathfrak{D}$

Dataset

  * $\mathfrak{L}^{l}$

Set of logical connections of arity $l$

  * $\oplus$

Concatenation

  * $\sigma_{sig}(x)$

Sigmoid Function

  * $\sigma_{soft^{+}}(x)$

Softplus Function

  * $\sigma_{soft}(x)$

Softmax Function

  * $\sigma_{tanh}(x)$

Tanh Function

  * $abs()$

Absolute Value

  * $f_{CL}$

Classifier

  * $f_{FE}$

Feature Extractor

  * a

$n$-Norm Distance

### Acronyms

AI

    Artificial Intelligence
CBM

    Concept Bottleneck Models
CNN

    Convolutional Neural Network
CoEx

    Composotional Explanations
DL

    Deep Learning
DNC

    Differentiable Neural Computer
DNN

    Deep Neural Network
GAT

    Graph Attention Network
GCN

    Graph Convolutional Network
GCW

    Graph Concept Whitening
GIN

    Graph Isomorphism Network
GNN

    Graph Neural Network
MANN

    Memory Augmented Neural Network
ML

    Machine Learning
MLP

    Multi-Layer Perceptron
NetDissect

    Network Dissection
NN

    Neural Network
PIGNN

    Prototype-based Interpretable Graph Neural Network
SDNC

    Simplified Differentiable Neural Computer
VA

    Visual Analytics
XAI

    eXplainable Artificial Intelligence

###  Chapter 1 Introduction

####  1.1 Introduction

Artificial Intelligence (AI) is a field of computer science aimed at
developing machines capable of solving tasks that typically require human
intelligence. The first successful AI approaches were rooted in expert systems
relying on rules and symbolic reasoning. However, despite early optimism,
symbolic AI has revealed limited adaptation capabilities. These systems often
assume perfect knowledge of tasks and disregard uncertainty or ambiguity in
data. Consequently, pure symbolic AI struggles to manage complex tasks for
which humans cannot describe the rules governing the phenomenon. For instance,
tasks like translating text, recognizing images, and exploring unknown
environments were deemed impossible to solve.

Machine Learning (ML) emerged as a paradigm shift in AI research to mitigate
these problems. ML provides algorithms capable of learning from data and
improving performance over time without explicit knowledge of a particular
phenomenon’s rules. Classical ML algorithms use statistical theory for pattern
recognition on a set of data. Examples of this category include decision
trees, logistic regression, and Support Vector Machines [82], which were the
state-of-the-art tools to deal with complex tasks until recently.

In recent years, the internet’s expansion and the availability of cheaper
hardware, open-source platforms, and big data have enabled the collection of
vast raw datasets. However, classical ML struggles to fully exploit these
datasets due to their size, complexity, and lack of explicit semantics
associated with data. Conversely, Deep Neural Networks (DNNs) are explicitly
designed to process raw data and memorize large quantities of information in
the interconnections between network layers. Therefore, the field of Deep
Learning (DL), which leverages DNNs, has become the primary subfield of ML.

A DNN consists of thousands or millions of interconnected neurons. Since the
work of Rosenblatt [199], the design of neurons and DNNs has become
increasingly complex, involving functions such as non-linear operations,
convolution, memorization, attention, and skip connections. The complexity of
these designs enables DNNs to achieve impressive performance in various tasks,
often surpassing human performance. Games [215, 159], vision [233, 196],
robotics [130, 128], and natural language processing [245, 24] have all
undergone a revolution in their respective field. Nowadays, mainstream
applications for speech recognition, machine translation, and text generation
are all powered by DL systems.

However, the gain in performance comes at the cost of transparency. While
symbolic systems are easy to understand in terms of encoded knowledge and
decision process, classical ML tends to be more opaque in both aspects.
Indeed, keeping track of the learning process is challenging, and the
explainability of these systems is often limited to extracting the learned
decision process. But, even in these cases, there is a trade-off between
complexity and transparency. For instance, when using large ML models (e.g., a
wide and deep decision tree), extracting explanations about the behavior that
are easy to understand could be challenging since explanations can be
extremely long. The challenge is exacerbated for DNNs, where there is no
semantic associated with data, and tracking how inputs are transformed into
outputs is a nightmare due to the complexity of the interconnections.
Therefore, DNNs are commonly referred to as black-boxes, where one feeds an
input and receives an output without understanding the motivations behind the
results.

The field of eXplainable Artificial Intelligence (XAI) aims to address the
need for transparency and interpretability in machine learning and AI systems.
XAI aims to provide insights into the inner workings of AI models, enabling
users to understand and interpret their outputs effectively. XAI encompasses a
broad range of techniques, spanning from methods that highlight the most
important parts of the input to those that extract the knowledge learned by
machine learning models.

In classical ML, XAI methods mainly focus on providing concise explanations
summarizing the already known rationale behind decisions. Examples of these
methods are feature importance analysis [292], which identifies the most
relevant features contributing to model predictions, and decision tree
visualization [237], which provides a graphical representation of the
decision-making process. In the context of DL, XAI methods deal with an
unknown decision process and an unknown learned knowledge and aim at
approximating, guessing, or probing the real model behavior.

The first and most popular techniques in the area are the so-called _extrinsic
methods_ , which approximate the behavior and generate explanations for DL
models by exploiting external means. For example, several techniques employ
surrogate models [140, 197], generative models [16], or perturbation-based
analysis [59] to approximate the decision process of the networks around a
given point. Despite the flexibility and high compatibility with existing
models, recent works [4, 200] argue against relying solely on extrinsic
methods. Indeed, extrinsic methods often struggle to capture the complexity of
model behavior of DNNs, are biased by the selection of the external means, and
require too much time to return a reliable approximation due to the complexity
of the tasks managed by DNNs.

To address this issue, researchers are starting to explore _intrinsic methods_
, which aim to enhance the interpretability of deep models by leveraging the
inner workings of the models. This objective can be achieved by modifying the
design of DNNs to make it more explainable, adjusting their training process
to produce explainable representations, or analyzing and connecting the
working mechanisms of their components. These methods include techniques such
as attention mechanisms [13], which highlight relevant input features,
activations analysis [17, 163], and self-explainable DNNs [33, 35]. Intrinsic
methods offer the advantage of directly linking interpretability to the
model’s design. They are generally faster than extrinsic methods and are more
faithful to the model behavior. Nonetheless, they are usually tailored to
specific settings (architectures, training processes, etc.) and, in the case
of self-explainable DNNs, can incur performance trade-offs, thereby limiting
the adoption from the general audience.

This thesis contributes to the ongoing research effort on XAI intrinsic
methods by proposing methods that leverage DNN inner workings for explaining
DL. In this context, the thesis proposes multiple designs of self-explainable
DNNs and a post-training method to investigate the neurons’ recognition
capabilities. The goal of the proposed self-explainable DNNs is to reduce the
performance trade-off and broaden the applicability of such methods. To
achieve the first goal, the proposed layers can be inserted into black-box
models without disrupting their structure and preserving most of their
representation power. The second goal is achieved either by extending
approaches to novel domains [33, 256], treating the preceding layers as black-
boxes (i.e., not exploiting specific shapes or structures), and evaluating the
proposed techniques across several architectures. As a side product, the
thesis also increases the diversity of approaches in self-explainable DNNs
since it introduces a new family of architectures: memory-based self-
explainable DNNs.

Similarly, the proposed post-training method shares the underlying principle
of compatibility and reliance on DNNs inner workings (i.e., activations) and
advances our knowledge of the semantics encoded in neurons. Namely, it enables
the investigation of broader settings than those explored in literature [163,
17], shedding light on novel phenomena related to the neurons’ activation
spectrum.

Finally, the thesis delves into the ongoing discourse on rendering the
explanations usable and beneficial for users. Recently, several interactive
systems have emerged to connect users, DL systems, and explanations by
exploiting interactive interfaces [263, 236] and dialogue systems powered by
large language models [221]. This thesis contributes to this research by
surveying the combination of XAI techniques and Visual Analytics (VA) systems.
By surveying existing methodologies and advocating for the integration of XAI
techniques into VA systems, the thesis aims to increase the awareness of the
XAI and VA communities of each other and foster a new alternative direction
for interactive explanations.

####  1.2 Contributions

The main contributions of this thesis to the field of XAI are the following:

  * •

the introduction of a novel prototype-based layer for Graph Neural Networks
(GNNs), which generalizes prototype-based self-explainable DNNs [33] in terms
of domains, architectures, and assumptions. Differently from the current
research, the layer represents prototypes as node embeddings, allowing its
application to both graph and node classification tasks. Furthermore, our
contribution includes a sparse explanation visualization for graphs that
preserves the faithfulness of explanations while enhancing their
understandability compared to commonly used visualization methods for
prototype-based networks.

  * •

the proposal of novel memory modules designed to enhance the interpretability
of existing neural networks while preserving their performance. The
utilization of memory modules for interpretability purposes represents a
novelty in current literature. By operating within the same training settings
as black-box models, these memory modules address some of the issues
associated with the use of self-explainable DNNs. Additionally, these designs
are flexible and can facilitate the retrieval of various types of
explanations.

  * •

the application and adaptation of concept-whitening [35] to the chemical
domain and drug discovery tasks to ensure that latent representations encode
semantics related to molecular properties. Our contribution includes the
generalization of the method [35] to the graph and chemical domain, thereby
expanding its applicability beyond the vision domain. The generalization
includes the type of data, the covered architectural families, and the type of
normalization layers, an area not explored in the literature.

  * •

the design of a novel algorithm for computing compositional explanations of
neurons behavior. This algorithm enables explanations for a broader spectrum
of neurons behavior than the one investigated in the literature and overcomes
issues related to the computational complexity of previous approaches [163].
Our contributions include the discovery and discussion of novel phenomena
related to neuron activations, as well as the design of novel metrics for
measuring the properties of explanation methods for latent representation.

  * •

a review and analysis of VA systems as a means to connect users and
explanations in an interactive environment without requiring XAI experts
during the usage. By focusing on systems dealing with DNNs and employing XAI
methods, the thesis contributes to identifying strengths and weaknesses of the
proposed VA solutions, laying the foundation for future enhancements in the
integration and collaborations between VA and XAI research.

####  1.3 Thesis Organization

The thesis is organized into five parts: Preliminaries, Self-Explainable
Neural Networks, Explaining Latent Representations, Bringing Explanations to
the User: Visual Analytics, and Conclusions. Each part is organized as
follows:

  * •

Part I Preliminaries provides the necessary background knowledge and
terminology to comprehend the chapters of this thesis and reviews the related
work on explainable AI applied to DNNs.

    * –

Chapter 2 introduces the terminology and key concepts related to DNNs, XAI,
and VA; describes the families of deep architectures examined in this thesis;
introduces the categorization of explanation methods; discusses the evaluation
challenges in XAI; and introduces the metrics used to evaluate the proposed
methods.

    * –

Chapter 3 reviews the literature on post-hoc XAI methods for explaining DL,
focusing on techniques similar to those proposed in this thesis; reviews self-
explainable DNNs; analyzes the limitations of current methods; and discusses
the relationships between existing approaches and the thesis techniques.

  * •

Part II Self-Explainable Neural Networks presents modules that can be added to
existing neural networks to improve their explainability; presents a
prototype-based design for self-explainable DNNs; introduces the novel family
of self-explainable DNNs based on memory;

    * –

Chapter 4 presents the design of a prototype-based layer for enhancing the
interpretability of GNNs; discusses the benefits in terms of interpretability;
evaluates the performance and the explanations over several architectures; and
discusses alternative design choices.

    * –

Chapter 5 introduces a design to improve the interpretability of recurrent
models on sequential data; introduces a memory module to enhance the
interpretability of black-box models on image data; introduces the unified
mechanism of memory tracking to compute different types of explanations;
evaluates the performance and the explanations of the proposed modules;
discusses alternative design choices.

  * •

Part III Explaining Latent Representations presents methods to inspect the
knowledge learned by DNNs; explore a method to enforce semantics in the latent
representation of GNNs; presents a technique to extract semantics from neurons
activations.

    * –

Chapter 6 presents a normalization for the graph neural networks and graph
data domain to enforce latent representation to be aligned to molecular
properties in the chemical domain and drug discovery; discusses the benefits
in terms of interpretability; evaluates the performance and the explanations
over several architectures, datasets, and layers; and discusses alternative
design choices.

    * –

Chapter 7 introduces a heuristic-guided algorithm to compute the semantics
encoded in a wide spectrum of neurons’ activations; proposes a set of metrics
to describe the properties of explanations returned by algorithms that compute
explanations for neurons’ activations; discusses the computational efficiency
of the proposed algorithm; evaluates the explanations computed by the
algorithm; discusses novel phenomena related to neurons activations; discusses
alternative designs.

  * •

Part IV Bringing Explanations to the User: Visual Analytics reviews and
analyzes VA systems that incorporate XAI methods to help users understand
DNNs.

    * –

Chapter 8 collects, reviews, and analyzes, from a XAI perspective, the VA
systems that use XAI methods to improve the explainability of DNNs; analyzes
the potential and benefits of interactive systems; discusses limitations and
future direction of the field.

  * •

Part V Conclusions summarizes the contributions of the thesis and discusses
limitations and future research directions.

    * –

Chapter 9 summarizes the thesis’s contribution; highlights the advantages and
limitations of the proposed approaches; and discusses short and long-term
future research directions that this thesis opened.

####  1.4 Related Publications

Part of the thesis has been previously published in the following journal and
conference articles:

  * •

Towards a fuller understanding of neurons with Clustered Compositional
Explanations.
Biagio La Rosa, Leilani Gilpin, and Roberto Capobianco
In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS
2023)

  * •

Explainable AI in Drug Discovery: Self-interpretable Graph Neural Network for
molecular property prediction using Concept Whitening.
Michela Proietti, Alessio Ragno, Biagio La Rosa, Rino Ragno, and Roberto
Capobianco.
Machine Learning (Journal), pp. 1–32 (2023)

  * •

The State of The Art of Visual Analytics for eXplainable Deep Learning.
Biagio La Rosa, Graziano Blasilli, Romain Bourqui, David Auber, Giuseppe
Santucci, Roberto Capobianco, Enrico Bertini, Romain Giot, and Marco Angelini
Computer Graphic Forum (Journal)
Presented also at 25th EG Conference on Visualization (EuroVIS 2023)

  * •

A self-interpretable module for deep image classification on small data.
Biagio La Rosa, Roberto Capobianco and Daniele Nardi.
Applied Intelligence (Journal) (2023)

  * •

Prototype-based Interpretable Graph Neural Networks.
Alessio Ragno, Biagio La Rosa, and Roberto Capobianco
IEEE Transactions on Artificial Intelligence (Journal) (2022)

  * •

Detection Accuracy for Evaluating Compositional Explanations of Units.
Sayo M. Makinwa, Biagio La Rosa, Roberto Capobianco.
In Proceedings of AIxIA 2021 - Advances in Artificial Intelligence, pages
550–563. Springer International Publishing, (2021)

  * •

A Discussion about Explainable Inference on Sequential Data via Memory-
Tracking.
Biagio La Rosa, Roberto Capobianco and Daniele Nardi.
Discussion Papers of AIxIA 2021

  * •

Explainable Inference on Sequential Data via Memory-Tracking.
Biagio La Rosa, Roberto Capobianco and Daniele Nardi.
In Proceedings of the Twenty-Ninth International Joint Conference on
Artificial Intelligence (IJCAI-20) (2020)

##  Part I Preliminaries

###  Chapter 2 Background

This chapter serves as a foundational introduction necessary for comprehending
the subsequent content of the thesis. It introduces the terminology and
concepts used to describe the methods proposed in this thesis.

The chapter is organized as follows: Section 2.1 introduces terminology and
concepts related to DL and the architectures used in this thesis; Section 2.2
introduces terminology, metrics, evaluation, and categorizations related to
XAI; finally, Section 2.3 introduces terminology, concepts, and
categorizations related to VA.

####  2.1 Fundamentals of Deep Neural Networks

This section describes the terminology used for the building blocks of a
Neural Network (NN) and DL and introduces the families of architectures
examined in this thesis.

We begin the description by highlighting the objective of a NN: learn how to
solve a task based on data observations. A data observation comprises several
elements termed _features_. The collection of observations forms the _training
dataset_ , which is then used to train the NN in a process known as _training
process_. In supervised learning and classification tasks, the primary
paradigms explored in this thesis, each observation is associated with a
_ground truth_ label. The ground truth denotes the desired output and is used
as feedback during the learning process.

$x_{1}$$x_{2}$$...$$x_{n}$ Hidden Layer 1 Hidden Layer 2 Hidden Layer 3 Output
Input Layer Output layer Figure 2.1: An example of a 3-layer neural network.
Each node in the hidden layers is connected with all the nodes of the previous
layer and all the nodes of the next layer.

A NN constitutes a hierarchical mathematical model, composed of multiple
layers of interconnected artificial neurons, which receives an observation and
yields an output referred to as a _prediction_. The layers of a NN can be
categorized based on their position within the architecture: input, hidden,
and output layers (Figure 2.1). The _input layer_ , constituting the lowest
layer in the hierarchy, feeds input features to the subsequent layers. A
_hidden layer_ receives the output of neurons from the previous layers and
outputs to the next layer. The outputs of a hidden layer collectively form the
_latent representation_ of the sample. The manifold encompassing all possible
latent representations of a given hidden layer is termed the _latent space_.
The _output layer_ , positioned at the apex of the hierarchy, accepts the
output of the preceding layers as input and delivers the prediction. A network
featuring at least two hidden layers is termed a DNN.

$x_{1}$ Input Features
$w_{1}$Weights$x_{2}$$w_{2}$$..$$..$$\displaystyle\Sigma$$f$ Activation
Function $\hat{y}$Prediction$x_{n-1}$$w_{n-1}$$x_{n}$$w_{n}$ Bias $b$ Figure
2.2: A simplified version of an artificial neuron.

The neurons across layers are interconnected through edges termed _weights_.
Each weight is associated with a value. In its simplest form (Figure 2.2), a
neuron processes the outputs of neurons from the preceding layer, each
multiplied by the corresponding weight connecting one of the previous neurons
to the current one. Subsequently, an aggregation function (e.g., summation)
combines all inputs. Finally, an activation function is applied to the
aggregated value to compute the neuron output. The weight values are adjusted
throughout the training process to achieve the desired output [199]. The
mechanism for updating the weights, known as the error-correction learning
rule, can be expressed as:

$w_{i,t+1}=w_{i,t}+\eta[y_{j,d}-y_{j,t}]x_{i,t}$ (2.1)

where $w_{i,t}$ denotes the weight $i$ at iteration $t$, $y_{j,d}$ represents
the desired output for the input $j$ and $y_{j,t}$ denotes the current neuron
output. If the prediction is correct, then $y_{d,j}=y_{j,t}$ and the weights
remain unaltered. The $\eta\in[0,1]$ parameter denotes the learning rate
controlling the magnitude of the update: a small value leads to gradual
adjustments, averaging the past inputs; large value facilitates rapid
adaptation, albeit with lesser consideration for past errors. In the case of
neurons in hidden layers, where no pre-defined desired output exists (i.e.,
the ground truth), the backpropagation algorithm is employed to update their
weights. This algorithm computes the error for each neuron $j$ as follows:

$\delta_{j}^{L}=\phi^{\prime}(a_{j})\sum_{k}{w_{jk}}\delta^{L+1}_{k}$ (2.2)

where the superscript indicates the layer of the neuron, $\delta^{L+1}_{k}$
represents the error of neuron $k$ in the layer $L+1$, connected to the
current neuron via weight $w_{jk}$, and $\phi^{\prime}$ denotes the derivative
of the activation function of the node $j$. The error is then multiplied by
the input received by the neuron to obtain the gradient of the error with
respect to the weight $w_{ji}$:

$\frac{\partial E_{n}}{\partial w_{ji}}=\delta_{j}^{L}z_{i}^{L-1}$ (2.3)

The set of the partial derivatives for all the weights in the network forms
the gradient vector $\nabla E_{n}$, which is used to update the weights of the
network:

$w^{t+1}=w^{t}-\eta\nabla E_{n}$ (2.4)

These equations serve as a general framework extended by the optimization
algorithm called _optimizers_ such as Adam [112], RMSProp, or AdaGrad [50].
Each optimizer presents its own learning schedule, differing in how weights
are updated at each step. For instance, Adam and RMSProp employ distinct
learning rates for different parameters and adjust them during the training.
The highly interconnected structure and the training process enable each
neuron to specialize in detecting particular feature correlations, enabling
the entire network to represent any function.

The training process is divided into _epochs_ , where the entire training
dataset is fed to the network to update weights. As the dataset may be too
large to be stored in memory, it is usually divided into _batches_ of $m$
samples progressively fed during the epoch. At each step, the optimizer
computes the _loss_ (i.e., error) associated with the prediction and updates
the weights. Various loss functions can be employed based on the task.

In recent years, numerous pre-trained DNNs have been available publicly. These
models are trained on large corpora to learn fundamental features shared
across several tasks. They can then serve as initial configurations for
training models on smaller downstream tasks. The idea of training pre-trained
models on downstream tasks starts from the observation that most DNNs can be
represented as the composition of two functions, the feature extractor
$f_{FE}$ and the classifier $f_{CL}$:

$\displaystyle f_{FE}\leavevmode\nobreak\ \mathpunct{:}$
$\displaystyle\leavevmode\nobreak\ \bm{x}\rightarrow\bm{h}$ (2.5)
$\displaystyle f_{CL}\leavevmode\nobreak\ \mathpunct{:}$
$\displaystyle\leavevmode\nobreak\ \bm{h}\rightarrow\bm{y}$ (2.6)
$\displaystyle f\leavevmode\nobreak\ \mathpunct{:}$
$\displaystyle\leavevmode\nobreak\ f_{CL}(f_{FE}(\bm{x}))$ (2.7)

The feature extractor $f_{FE}$ transforms input $\bm{x}$ from features
representation to a latent representation $\bm{h}$ capturing feature
relations. The latent representation denotes the output of the last layer of
the feature extractor. Then, this representation is fed to the classifier
$f_{CL}$, which transforms this representation into a prediction. Therefore,
since the common features shared across tasks are likely encoded into the
feature extractor, the idea is to preserve the feature extractor’s learned
knowledge and change the classifier for the downstream task. We can
distinguish between two forms of training for novel tasks: _fine-tuning_ and
_transfer learning_. In the former, the weights of the feature extractor
remain _frozen_ and are not updated during training, with adjustments confined
to the classifier’s weights. Conversely, in transfer learning, all the
parameters are updated during the new training phase and the feature
extractor’s weights are used to initialize the network’s parameters.

At the end of the training process, the network’s capability is assessed by
providing a set of samples unseen during training, referred to as the _testing
dataset_. The quality of the learning process is evaluated using a set of
metrics that summarizes the network’s performance on the testing dataset. This
thesis uses two metrics to evaluate the performance of DNNs: accuracy and ROC-
AUC. _Accuracy_ is the ratio between the number of correctly predicted samples
and the total number of samples in the dataset. It ranges from 0 (there are no
correct predictions) to 1 (all predictions are correct). _ROC-AUC_
additionally takes into account the ratio between true positive (e.g.,
prediction is 1 when the ground truth is 1) and false positive (e.g.,
prediction is 0 when the ground truth is 1) and it is useful when the dataset
is unbalanced in the number of samples per class.

#####  2.1.1 Architectures

As mentioned in the previous section, neurons and layers can be interconnected
in several ways, and multiple activation functions can be employed to compute
the output of neurons. Based on the configuration schema of neurons and
layers, we can distinguish different families of architectures. This section
briefly describes the main layers and architectures used throughout the
thesis: LSTM, attention, memory-augmented neural networks, convolutional
neural networks, and graph neural networks.

###### LSTM

Long-Short Term Memory (LSTM) [86] networks are recurrent neural networks that
deal with sequential data. Sequential data is a type of data where data points
have a temporal dependency on other data points. A dataset of sequential data
is a dataset where points chained by a temporal dependency are collected into
a structure called a _sequence_. Each item in the sequence is a _timestep_.

In an LSTM, each neuron takes one timestep at a time as input and the previous
neuron’s output, yielding a prediction. Therefore, if the sequence includes
$n$ timesteps, the LSTM yields $n$ predictions (Figure 2.3).

$x_{1}$$x_{2}$$x_{3}$..$x_{n}$$y_{1}$$y_{2}$$y_{3}$$y_{n}$$h_{1}$$h_{2}$$h_{n-1}$$w_{x}$$w_{x}$$w_{x}$$w_{x}$
Figure 2.3: An unfolded view of an LSTM cell.

In LSTMs, each neuron is a _memory cell_ with its state $s$. For each
timestep, the neuron takes as input the current element $x_{t}$ in the
sequence, the previous output $h_{t-i}$, and the previous cell state
$s_{t-1}$. The purpose of the cell state is to store, update, and carry
information through the timesteps. The content of the cell state is managed by
two gates: the forget and the input gate. The _forget gate_ $g_{f}$ controls
which information to delete from the state; the _input gate_ $g_{i}$ controls
which information to write into the state. Finally, a tanh function
$\sigma_{tanh}$ computes the information to be stored $\hat{s_{t}}$ as:

$\hat{s_{t}}=\sigma_{tanh}(\bm{x}_{t},h_{t-1})$ (2.8)

The next cell state is computed as the weighted sum between the current state
and the information to be stored, weighted by the gates:

$\bm{s}_{t}=g_{f}\bm{s}_{t-1}+g_{i}\hat{\bm{s}_{t}}$ (2.9)

Finally, the output is obtained by multiplying the _output gate_ $g_{o}$,
which controls the contribution of the cell state, and the candidate output,
which is computed by applying the tanh function to the next cell state:

$\bm{y}_{t}=g_{o}\sigma_{tanh}(\bm{h}_{t-1},\bm{x}_{t})$ (2.10)

The architecture and the memory cell allow LSTM to store information longer
than classical recurrent networks, achieving better performance.

###### Attention Layers

Attention [12] is a mechanism used to model and exploit long-range
correlations between features. This mechanism combines three components: the
queries $\bm{Q}$, the keys $\bm{K}$, and the values $\bm{V}$. These components
can be either distinct vectors or linear projections of the same input [245].
The process begins by comparing each query $\bm{q}$ to the keys and computing
a score value for each pair:

$e_{\bm{q},\bm{k}_{i}}=f_{scoring}(\bm{q},\bm{k}_{i})$ (2.11)

The score value is computed by a scoring function $f_{scoring}$ that changes
based on the type of attention implemented. Then, a softmax function computes
the attention weights based on the scores:

$w_{\bm{q},\bm{k}_{i}}=\sigma_{soft}(e_{\bm{q},\bm{k}_{i}})$ (2.12)

Finally, the layer outputs the weighted sum of the values, weighted by the
attention weights:

$\bm{y}=attention(\bm{q},\bm{K},\bm{V})=\sum_{i}w_{\bm{q},\bm{k}_{i}}\bm{v}_{k_{i}}$
(2.13)

Several attention variants have been proposed by changing the scoring
functions [142, 270], the projections [19], or the applied steps. The
attention layers are the fundamental blocks of the Transformer [245]
architecture, which is the backbone of several state-of-the-art models.

###### Memory Augmented Neural Networks

Memory Augmented Neural Networks (MANNs) are a class of neural networks
designed to overcome the limitation of classical recurrent architectures like
LSTM. Indeed, LSTMs suffer from the vanishing gradient problem. When sequences
are particularly long, gradients are multiplied repeatedly during
backpropagation through the recurrent connections. This multiplication causes
the gradients to get smaller and smaller until they disappear. This problem
leads to a limited capability of memorization and exploitation of information
from the earlier steps of the sequence.

MANNs mitigate this problem by employing an external memory for storing
information for longer periods of time. A MANNs is characterized by five
elements: the controller, the memory, the write heads, the read heads, and the
classifier. The _controller_ is a DNN that takes an input and returns an
output, usually in the form of a latent representation. If the input is a
sequence, then the output of the controller for each step is given by the
following equation:

$\bm{h}_{t}=f(\bm{x}_{t})$ (2.14)

The _memory_ is a matrix $\bm{M}$ of dimensions $N\times M$ that can be
written or read by the network. Writing and reading operations are performed
by functions called _write heads_ and _read heads_. Most of the time, heads
are based on attention mechanisms. Both of them usually depend on the output
of the controller. In this way, the controller can learn how to use the
memory. The output of writing operations is the updated memory:

$\bm{M}_{t+1}=wh(\bm{M}_{t},\bm{h}_{t})$ (2.15)

while the output of reading operations are readings $\bm{r}$ whose shape
depends on the specific design:

$\bm{r}_{t}=rw(\bm{M}_{t},\bm{h}_{t})$ (2.16)

Finally, the _classifier_ yields the prediction by exploiting the readings
from the memory and the controller output.

$\bm{y}_{t}=f_{CL}(\bm{h}_{t},\bm{r}_{t})$ (2.17)

MANN designs can include all or parts of the described elements. Only the
controller and the memory are common to all the MANNs.

###### Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are networks originally designed to deal
with image data characterized by the presence of _convolutional layers_ and
_pooling layers_. The output of each neuron in a convolutional layer depends
on three components: the filter, the receptive field, and the inputs it
receives. The receptive field is the set of neurons connected to the current
neuron. While in most DNNs, each neuron receives information from all the
neurons of the previous layers, in the case of CNNs, each neurons receive the
outputs of only a subset of neurons in the previous layer. The filters are the
weights that connect the neuron to its receptive field. In CNNs, all the
neurons in a layer share the same filters. Convolutional layers are designed
to exploit spatial information. Indeed, neurons are organized in sequence,
grids, or 3d volumes based on the dimension of the input. For 2d data, filters
and receptive fields are grids and each neuron can be identified by the
indices on the grid. In mathematical terms, the output of a neuron placed in
position $[i,j]$ is associated with a $d_{r}\times d_{r}$ receptive field and
a filter $d_{f}\times d_{f}$

$h_{i,j}^{l}=\sigma(\sum_{a=0}^{d_{f}-1}\sum_{b=0}^{d_{f}-1}w_{ab}h_{(i+k)(j+b)^{l-1}})$
(2.18)

where $\sigma$ is a non-linear activation.

Generally, convolutional are interleaved with pooling layers, which reduce the
dimensionality of the input they receive. The reduction consists of clustering
a portion of the input and then combining the elements of each cluster into a
single value. For example, a Max-Pooling layer yields the maximum values of
each cluster as outputs. The clusters are computed using spatial information,
for example, extracting multiple $3\times 3$ squares of adjacent values in the
matrix the layer receives. Several designs of CNN architectures that
interleave pooling and convolutional layers have been proposed in the
literature. Among them, the thesis include experiments on ResNet [80],
EfficientNet [234], MobileNet [207], GoogLeNet [232], DenseNet [92],
ShuffleNet [281], and WideResNet [277].

###### Graph Neural Networks

GNNs are networks built to work with graph data. Their central element is the
message-passing paradigm used in their graph layers, where each node
recursively updates its representations by aggregating the ones of its
neighbors [267]. How representations are updated and aggregated varies based
on the layer’s design. We report the aggregation and update functions used by
the three architectures we use in this thesis: _Graph Convolutional Networks
(GCNs)_ , _Graph Attention Networks (GATs)_ , and _Graph Isomorphism Networks
(GINs)_.

GCNs aggregate latent representation of a node $v$ by averaging the latent
representations of its neighbor nodes:

$\bm{h}^{l}=W\cdot(\\{\bm{h}^{l-1}_{u}:u\in N(v)\\})$ (2.19)

where $W$ is the weight matrix of the convolution, $N(v)$ is the set of
neighbors of the node $v$ and $\bm{h}^{l-1}_{u}$ is the representation of the
neighbor node $u$ in the previous layer.

GATs [246] weight the convolutional aggregation via attention scores,

$\bm{h}^{l}=\sum(\\{\bm{a}(u,v)\cdot W\cdot\bm{h}^{l-1}_{u}:u\in N(v)\\})$
(2.20)

where $\bm{a}(u,v)$ is the attention weight between the nodes $u$ and $v$ and
the other parameters are the same of GCN.

GINs [268] use learned parameters $\epsilon$ to weight the aggregation
performed by a Multi-Layer Perceptron (MLP).

$\bm{h}^{l}=MLP((1+\epsilon){h}^{l-1}_{v}+\sum_{u\in N(v){h}^{l-1}_{u}}$
(2.21)

Node and graph classification are two popular tasks in GNNs. In both cases,
the input is usually a single graph. In _node classification_ , the task is to
predict multiple labels, each associated with a node in the graph (e.g.,
predicting properties of the nodes). In _graph classification_ , the task is
to predict a single output, usually the class or a general property of the
whole graph. In the first case, the output of the layers described above is
directly fed to a classifier. In the latter, the output of the layers is first
fed to a readout layer, such as global pooling layers, and then to the
classifier.

####  2.2 Explainable AI

By analyzing the architectures presented in the previous section, we can note
that the introduced components and advancements between one architecture and
another are designed to exploit novel and more complex relations encoded in
the data, thereby improving the capability of the networks. However, the
combination of these advancements in complexity, the opacity of the training
process, and the usage of raw data make the behavior of these architectures
opaque. To address this issue, the XAI field aims to develop methods that can
improve the explainability of artificial intelligence systems.

As in the case of the definition of intelligence, there is no universally
accepted definition of explainability [155, 200, 66]. In this thesis, we use
explainability as a general term to refer to all the methods that “enable
human users to understand, appropriately trust, and effectively manage the
emerging generation of AI systems” [73]. We use the terms explainability and
interpretability interchangeably throughout the text.

There are several ways to classify XAI methods proposed in the field. Here, we
focus on categories tailored to DNNs and this thesis. A first distinction
separates _post-hoc_ approaches and _self-interpretable DNNs_ [282, 10]:

  * •

post-hoc “methods target models that are not readily interpretable by design
by resorting to diverse means to enhance their interpretability” Arrieta et
al. [10];

  * •

self-explainable DNNs are architectures that include components that can be
used directly to ease the explanation of the results [12, 33, 132].

A second distinction, introduced in the previous chapter, separates between
_intrinsic_ and _extrinsic_ methods:

  * •

intrinsic methods aim to enhance the interpretability of deep models by
looking at and leveraging the inner workings of the models;

  * •

extrinsic methods focus on generating explanations for DL models by exploiting
external means (e.g., gradients, surrogate models).

While often self-explainable DNNs are referred to as intrinsic methods and
post-hoc as extrinsic ones, we prefer to keep the categories separated, given
the proliferation of methods between these categories. For example, methods
that look at attention patterns [2] in Transformers exploit only the
components of the models, and thus, they are intrinsic. However, Transformers
cannot be considered self-explainable architectures. Therefore, a method of
this kind could be just referred to as a post-hoc intrinsic method.

A further distinction [3] can be done by separating _local_ and _global_
methods:

  * •

local methods provide explanations that are valid for a limited set of data
(usually a single prediction), and thus, their explanations do not generalize
to other components;

  * •

global methods explain a model’s whole logic (usually approximating average
outcomes).

Local methods are more faithful and precise in explaining single predictions
but do not offer a guarantee of generalizability. On the other end, global
methods are effective in providing an overview of the main learned relations
but tend to be less precise and are less reliable in explaining individual
components or predictions.

#####  2.2.1 Categorization

XAI methods can also be split based on the type of explanations they return.
While several XAI taxonomies have been proposed in the literature to perform
such a split, this section reports a simplified categorization [121] by
grouping the types of methods discussed in the thesis. Each category is
described in terms of the goals of its methods and explanation objects.

###### Feature attribution

Feature attributions are scores assigned to each input feature or group of
features representing the impact of that item on the network’s decision
process. Methods focusing on feature attributions provide answers to questions
like “Why does the model return this specific output?” and “What are the most
relevant features exploited by the model to recognize objects in this class?”.
Attributions can be either global or local. Global feature attribution
measures the importance of features on average. Local feature attribution
measures the importance of features exclusively for the current prediction.
Scores are usually normalized and visualized as heat maps or numbers.

###### Learned Features

Learned features are features or concepts neurons, groups of neurons, or
layers recognize. In the context of this thesis, a concept is intended as a
set of semantically connected features annotated in a dataset (e.g., the
concept of a wheel in a car). They address the question “What has this
component learned during the training process?” and thus provide global post-
hoc explanations. They can be visualized as samples exhibiting the learned
knowledge or by exposing groups of features connected to such knowledge.

###### Explanations by Example

Explanations by examples are samples similar to the item that the user wants
to explain and associated with the same prediction. Typically, the samples are
extracted from the training dataset and share most features with the object to
explain. Inspecting explanations by example helps users extract patterns
exploited by the models. Explanations by examples can be applied to extract
both global and local information. For example, when applied to the input of
the model, explanations by example provide local information, highlighting
features exploited by the model for the specific input. Conversely, these
explanations provide (global) insights into the learned dynamics when applied
to latent representations (e.g., as shown in Chapter 4).

###### Counterfactuals

Counterfactuals are samples representing alternative configurations of input
features. A counterfactual is a sample as close as possible to the input but
associated with a different prediction. Samples can be either generated or
extracted from a dataset. Methods of this category aim to find the minimum
magnitude of meaningful edits needed on the current input sample to obtain a
different prediction. These explanations are particularly useful for recourse
and, most of the time, are local methods.

#####  2.2.2 Evaluation of XAI methods

One of the open questions of the XAI field regards the evaluation of
explanation methods. Indeed, there is no ground truth as the real inner
workings of the model are unknown. We can distinguish between three categories
of evaluation: user-study, datasets, and proxy metrics.

_User studies_ involve users testing explanations in a real-world task and
providing feedback. This form of evaluation was among the first to be
explored. We identify three common workflows supporting user studies. The
first consists of presenting users with multiple explanations and asking them
to rank their preferences, which are then used to measure the quality of the
tested methods. The second one consists of providing explanations to the users
to assist them while solving a task, followed by collecting their feedback as
a qualitative evaluation. Lastly, the third workflow permits users to interact
with explanations and tasks at the same time. This task can be facilitated
through the usage of interactive interfaces like dashboards, dialogue systems,
and visual analytics systems. At the end of the interactive session, both
quantitative and qualitative metrics and feedback can be collected.

Evaluations based on user studies, and particularly the interactive ones, can
be used to evaluate the usefulness of the explanations, their impact on
specific tasks, and the preference regarding explanation visualization [48].
However, these evaluations are challenging to reproduce, causing problems in
the comparison between methods proposed at different times. Moreover, user
satisfaction is more influenced by their belief and mental models than the
precision of the explanations in capturing the true behavior of the DNN.
Therefore, involving users should occur only after explanation methods have
demonstrated a connection to the real decision process of the network.
Otherwise, explanations can easily cause overtrust and mislead the user [99].

In an effort to establish more reproducible environments and link the
evaluations to the decision process, _Dataset evaluations_ propose to use
datasets with available ground truth explanations [137] to evaluate
explanations. These evaluations draw inspiration from the common evaluation
practices applied in machine learning and discussed in Section 2.1. However,
this form of evaluation is feasible only for a subset of methods and
simplified tasks where the rules governing the task are known and there is
only one unique way to solve the task.

The third choice involves using a _mathematical proxy_ to compare explanation
methods quantitatively. They are not intended as replacements for user
evaluations but as a necessary prerequisite for selecting the appropriate set
of explanation methods. Although several proxy metrics have been proposed to
measure the quality of the explanations quantitatively in recent years, there
is no standard global metric yet, and each explanation type and context has
its own set of metrics.

This thesis mainly uses proxy metrics to evaluate the proposed methods. The
metrics are chosen among each category’s most recent and popular ones. When
available (Chapter 5), we support the evaluation on toy datasets where ground
truth can be retrieved and the models satisfy the appropriate requirements
(e.g., perfect accuracy). We also discuss the interactive systems supporting
user studies in Chapter 8. The following paragraphs list the metrics used to
evaluate the quality of the explanations for each category presented in the
previous section.

###### Feature Attributions.

######  Definition 2.2.1.

The $\bm{fidelity+^{prob}}$ [275] metric measures the difference in
probability of the original prediction and the prediction generated when the
most relevant features are masked.

$Fidelity+^{prob}=\frac{1}{N}{\sum_{i=1}^{N}(f(x_{i})-f(\theta_{1}(x_{i})))}$
(2.22)

where $\theta_{1}(x)$ is a function that mask $x$ by removing the most
important features. Higher is better. This metric is equivalent to the
deletion score proposed in [185].

######  Definition 2.2.2.

The $\bm{fidelity+^{acc}}$ [275] metric measures the difference in the
accuracy of the original prediction and the prediction generated when the most
relevant features are masked.

$Fidelity+^{acc}=\frac{1}{N}{\sum_{i=1}^{N}(\operatorname*{arg\,max}{f(x_{i})}=\operatorname*{arg\,max}{f(\theta_{1}(x_{i})}))}$
(2.23)

where $\theta_{1}(x)$ is a function that mask $x$ by removing the most
important features. Higher is better. Fidelity can be also computed by using
the ROC-AUC instead of the accuracy: in this case, the metric is denoted as
$Fidelity+^{ROC-AUC}$.

######  Definition 2.2.3.

The $\bm{fidelity-^{acc}}$ [275] metric measures the difference in accuracy
between the original prediction and the prediction generated when the least
relevant features are masked.

$Fidelity-^{acc}=\frac{1}{N}{\sum_{i=1}^{N}(\operatorname*{arg\,max}{f(x_{i})}=\operatorname*{arg\,max}{f(\theta_{2}(x_{i})}))}$
(2.24)

where $\theta_{2}(x)$ is a function that mask $x$ by removing the least
important features. Higher is better.

###### Learned Features.

######  Definition 2.2.4.

The intra-concept similarity [35] metric measures the mean pairwise cosine
similarity between samples of the same concept.

$d_{ii}=\frac{1}{n^{2}}\>\left(\sum\limits_{j=1}^{n}\sum\limits_{k=1}^{n}\>\frac{\textrm{{x}}_{ij}\cdot\textrm{{x}}_{ik}}{\lVert\textrm{{x}}_{ij}\rVert\;\lVert\textrm{{x}}_{ik}\rVert}\right)$
(2.25)

where $j$ is the index of the sample, $i$ is the index of the concept,
$\bm{x}_{ij}$ is the latent representation for the sample $j$ of the concept
$i$, and $n$ is the number of samples including the concept $i$.

######  Definition 2.2.5.

The inter-concept similarity [35] metric measures the mean pairwise cosine
similarity between samples of two different concepts.

$d_{pq}=\frac{1}{nm}\>\left(\sum\limits_{j=1}^{n}\sum\limits_{k=1}^{m}\>\frac{\textrm{{x}}_{pj}\cdot\textrm{{x}}_{qk}}{\lVert\textrm{{x}}_{pj}\rVert\;\lVert\textrm{{x}}_{qk}\rVert}\right)$
(2.26)

where $p$ and $q$ are the index of the two concepts, and $m$ and $n$ are the
number of samples for concept $p$ and $q$, respectively.

######  Definition 2.2.6.

The separability [35] metric measures the separability of two concepts in a
latent space. It is expressed as the ratio between intra and inter-concept
similarities.

$Sep_{ij}=\frac{d_{ij}}{\sqrt{d_{ii}d_{jj}}}$ (2.27)

A lower score is considered better.

######  Definition 2.2.7.

The intersection over union (IoU) [17] metric measures the alignment between a
concept annotation and the firing rate of a latent representation.

$IoU(L,\mathfrak{D})=\frac{\sum_{x\in\mathfrak{D}}|M^{k}(x,i)\cap
C(x,c)|}{\sum_{x\in\mathfrak{D}}|M^{k}(x)\cup C(x,c)|}$ (2.28)

where $k$ represents the index of a neuron, $c$ represents a concept,
$M^{k}(x,i)$ is a function that returns the binary mask that indicates the
parts of the input on which the neuron fires, and $C(x,c)$ corresponds to a
function that returns a binary mask that represents which parts of the input
are associated with the concept $c$.

###### Explanations by Examples.

######  Definition 2.2.8.

The input non-representativeness [106] metric measures the L1 distance in the
logits obtained by feeding the current input and the explanation by example to
the model.

$Inr=L^{1}(f(x),f(x_{e}))$ (2.29)

A lower input non-representativeness means the explanation is of better
quality.

######  Definition 2.2.9.

The prediction non-representativeness [188] metric measures the cross entropy
loss between the current predicted class and the class predicted by using the
explanation by example as input.

$Pnr=CrossEntropy(y_{x},y_{x_{e}})$ (2.30)

A lower prediction non-representativeness means the explanation is of better
quality.

###### Counterfactuals.

######  Definition 2.2.10.

Given a predicted class $y_{x}$, a counterfactual class $y_{c}$, an
autoencoder $f_{y_{x}}$ trained on reconstructing samples of the predicted
class, and an autoencoder $f_{y_{c}}$ trained on reconstructing samples of the
counterfactual class, the IM1 [138] metric measures the ratio between errors
of the two autoencoders to reconstruct the counterfactual.

$\frac{\lVert x_{cf}-AE_{y_{c}}(x_{cf})\rVert}{\lVert
x_{cf}-AE_{y_{x}}(x_{cf})\rVert}$ (2.31)

A low IM1 score means the counterfactual is closer to the counterfactual class
data distribution than the input one and it is of a better quality. This is
useful for methods that start from the input and perturb it to generate a
counterfactual.

######  Definition 2.2.11.

Given a predicted class $y_{x}$, a counterfactual class $y_{c}$, an
autoencoder $f_{y_{x}}$ trained on reconstructing samples of the predicted
class, and an autoencoder $f_{y_{c}}$ trained on reconstructing samples of the
counterfactual class, IIM1 score [138] measures the ratio between errors of
the two autoencoders to reconstruct the counterfactual.

$\frac{\lVert x_{cf}-AE_{y_{x}}(x_{cf})\rVert}{\lVert
x_{cf}-AE_{y_{c}}(x_{cf})\rVert}$ (2.32)

IIM1 metric [124] is the inverted version of the IM1 score. A low score means
that the counterfactual is closer to the input’s class data distribution than
the counterfactual’s. This is useful for methods that select counterfactuals
from a pool of samples associated with different predictions.

######  Definition 2.2.12.

Given a predicted class $y_{x}$, an autoencoder $f_{y_{x}}$ trained on
reconstructing samples of the predicted class, and an autoencoder $f_{all}$
trained on reconstructing samples of all the classes in the dataset, the IM2
[138] metric measures the difference in the reconstruction errors of the
counterfactual between the autoencoder trained on the input classes and the
one trained on all the classes.

$\frac{\lVert AE_{y_{x}}(x_{cf})-AE_{all}(x_{cf})\rVert}{\lVert
x_{cf}\rVert_{1}}$ (2.33)

A low IM2 score means the data distribution of the counterfactual class
describes the counterfactuals as good as the distribution over all classes and
the counterfactual can be considered interpretable. Historically, it is a
score proposed to evaluate the usefulness of a generated counterfactual.

####  2.3 Visual Analytics

The XAI field is not the sole field that is actively working on supporting
users to understand DNNs. Indeed, according to Choo and Liu [37], VA systems
are playing “a critical role in enhancing the interpretability of deep
learning models, and it is emerging as a promising research field”. This
section lists the main components used in the Chapter 8 to analyze the VA
systems aiming to help users understand DNNs through XAI methods.

We begin by enunciating the definition of VA: it is the science of analytical
reasoning supported by interactive visual interfaces [238]. A VA system helps
users synthesize and derive insights from data by detecting the expected
relationships, discovering the unexpected ones, and communicating these
findings to the human user for further actions [116].

The VA process [104, 203] can be divided into four components:

  * •

data are the starting point of the system and, in the context of this thesis,
correspond to inputs, outputs, activations, explanations, etc.;

  * •

model applies transformations to the data and, in the context of this thesis,
corresponds to the DNN;

  * •

visualization is the interface between users and the system that allows the
detection and discovery of relationships and insights on the data and model;

  * •

knowledge is the user-driven component and “consists in finding evidence for
existing assumptions or learning new knowledge about the problem domain” La
Rosa et al. [121]

The user interacts with the VA system through interfaces and interactions
[240]. Interactions allow the user to analyze data, generate new
visualizations, and steer the analyzed model [53, 87]. Interactions represent
the means by which the users can achieve the desired goal and help to create a
better mental model of the investigated problem. Given a trained model loaded
in a VA system, we can distinguish between three types of interactions [121]:

  * •

passive observations involve only the navigation across data (samples, layers,
explanations, etc.) in terms of filtering and selection;

  * •

interactive input observations involve the modification or creation of novel
inputs and feeding them to the model to evaluate changes in the models’
response;

  * •

interactive model observations involve the modification of the deep learning
model components to evaluate changes in the models’ response;

Finally, VA systems are tailored to specific target users and tasks. Strobelt
et al. [227] propose the following classification for target users:

  * •

architects are advanced DL experts whose goal is developing new architectures
or modifying the design of the current ones;

  * •

trainers are users (with sufficient background knowledge in DL) whose goal is
to apply existing architectures to new domains, tasks, and datasets;

  * •

end-users have limited or no DL knowledge and use already trained models in
their specific application domain.

Note that end-users cannot use a system built for the other categories, while
all the other categories of users can use systems designed for end-users.

###  Chapter 3 Related Work

This chapter reviews existing methods for explaining DNNs. By utilizing the
categorization introduced in Section 2.2, this chapter discusses post-hoc
methods and self-explainable DNNs. Since the chapter cannot provide an
exhaustive overview of the state-of-the-art methods for the whole XAI field,
the review of post-hoc methods focuses attention on methods that have already
been applied to DNNs, are data agnostic, and related to the approaches
proposed in the next chapters. The chapter is organized as follows: Section
3.1 reviews post-hoc methods for deep learning, Section 3.2 reviews current
self-explainable DNNs, and section 3.3 discusses the relation between the
reviewed approaches and the methods proposed throughout the thesis.

####  3.1 Post-Hoc Methods

This section briefly describes the state of the art of post-hoc methods for
the categories described in Section 2.2. It focuses only on methods tailored
for DNNs or explicitly tested on them. For a deeper description of XAI methods
in machine learning, please refer to the recent surveys on the topic [66, 3,
72].

#####  3.1.1 Feature Attribution

Feature attribution methods assign scores to every input feature based on
their relevance to a set of predictions. It is one of the oldest and most
extensively explored research areas in the XAI field.

Gilpin et al. [66] distinguish between methods based on perturbations,
backpropagations, gradients, and surrogate models. The concept behind
perturbation-based methods is straightforward: modify the value of a feature
(or a set of features), input the perturbed instance into the model, and
collect the results. Scores are computed by iteratively repeating this process
and measuring the deviation between the original prediction and the
predictions of perturbed instances [279, 291, 1]. An optimization process can
be used to guide the generation of meaningful perturbations [59, 273].

Approaches utilizing surrogate models, such as LIME [197] and SHAP [140], also
follow the idea of using perturbations. However, in this case, perturbations
are not directly used to estimate attributions but serve as samples for
training a local interpretable surrogate model. The surrogate model learns to
mimic the black-box model’s decision process in the samples’ neighborhood to
be explained. The weights learned by the surrogate model are then used as
feature attribute scores. However, these approaches are slow, especially with
a large number of features, as in the case of DNNs. Additionally, due to the
nonlinear nature of DNNs, the result is influenced by the number of features
removed altogether at each iteration and the number of permutations.

Gradient-based methods offer a faster alternative to perturbation-based
methods by considering the partial derivative of the target output as scores
for feature attributions [216]. However, vanilla gradients produce noisy
explanations. To address this problem, several works propose enhancements such
as multiplicative terms to the gradients [214, 224, 209, 30], the use of
integrals and baselines [231, 269, 103], or smoothing functions [222, 25].
While popular for their simplicity, these methods tend to be loosely linked to
the decision process [4] and suffer from the gradient-shattering effect.

Backpropagation methods [279, 11, 214] and forward propagation methods [68]
work similarly to gradient-based methods but employ custom rules and
quantities instead of gradients. For instance, LRP [11] computes relevance for
each neuron of the network during input parsing and then backpropagates the
relevance and prediction to compute attribution scores. DeepLIFT [214] extends
this idea by considering baseline inputs as proposed in gradient-based
methods.

In parallel with these approaches, there has been recent interest in computing
scores for sets of semantically connected features (i.e., _concepts_) rather
than individual features. Particularly, TCAV [111] and its derivatives [63,
57] investigate the impact of concepts on the decision process. These methods
collect samples with and without the target concept, build a hyperplane to
separate these samples, and use directional derivatives with respect to this
hyperplane as attribution scores. The sensitivity to the collection of samples
in terms of diversity and number of samples and the computational time
required to probe for several concepts represent the major limitations of
these approaches.

Finally, all the methods described thus far explain individual predictions. To
obtain global explanations, researchers propose various methods that aggregate
attribution scores of individual samples, either by averaging or clustering
them [126, 165, 206].

#####  3.1.2 Learned Features

This category of methods focuses on extracting information regarding what
individual components (e.g., layers, neurons) of neural networks have learned
during the training process. Research in this area has explored two main
directions [27, 204, 66]: feature synthesis and feature probing using external
datasets.

Features synthesis involves generating synthetic explanations through
iterative processes [54, 175] or using external models [169, 171]. These
processes are typically tailored to maximize (or minimize) an objective
function related to the component to be explained. For instance, in the case
of neurons, several approaches aim to generate synthetic inputs that maximize
the activation of a specific neuron [175, 144, 54, 217]. The general process
consists of generating this synthetic input by iteratively altering the
features of a starting random input. However, vanilla maximization of the
activation can yield abstract explanations that are difficult to interpret. To
address this issue, several regularizers have been proposed to increase
variance [144] or impose constraints on the generation process [254, 171, 175,
58, 274].

Nonetheless, these methods encounter several challenges. For example, the
stochastic nature of the process [161] may produce different explanations for
the same neuron. Additionally, while neurons can recognize multiple features
due to superposition [23, 52], these methods tend to converge towards one or a
few concepts, overlooking the multifaceted nature of these components [170].
Furthermore, despite their popularity and the advancements to reduce
abstractness, humans often struggle to comprehend these explanations and
instead prefer feature probing explanations [22, 290].

Feature probing methods employ real samples to represent the learned features.
In this category, the natural counterpart of activation maximization methods
is to select samples from the dataset that maximize the neuron’s activation
[27]. In contrast to the activation maximization method, the selected samples
are not abstract and naturally exhibit diversity. However, the connection
between high activations and the explanation is weaker. Indeed, it is often
unclear if the cause of the high activation is the whole sample or specific
elements within it. While providing more samples can mitigate this issue, an
excessive number of samples can diminish the usefulness of the explanation.

Bau et al. [17] propose addressing this issue using a concept-annotated
dataset. In this case, the algorithm selects the concept that maximally
activates a neuron, removing factors unrelated to the high activation. Similar
works reverse the process by fixing a concept and searching for the neurons
associated with the highest activations when the concept is included in the
samples [41, 51, 83, 9, 164]. Like the generative approaches, associating a
single concept with a neuron is inadequate due to superposition. Therefore,
successive works [163, 76, 150, 26] propose associating logical formulas of
concepts with neurons by finding the formula whose annotations are the most
aligned with the highest activations through search algorithms.

One weakness of feature probing approaches is their dependency on concept-
annotated datasets. Indeed, the same neuron can be associated with two
different explanations simply by changing the annotated dataset [192].
Moreover, the analysis of learned features is constrained by the annotated
concepts and the quality of annotations. Recent works suggest replacing the
concept-annotated dataset with captioning or multimodal models [84, 173] to
mitigate this dependency. However, these models require human annotations
during training and are susceptible to the domain adaptation problem, thus
shifting the dependency from dataset to model aspects.

#####  3.1.3 Explanations by examples

Explanations by examples are samples similar to the current input and
associated with the same prediction. Therefore, one of the first explored
research directions was providing samples that approximate data prediction
distribution as explanations. These approaches [110, 74] rely on clustering
the data, identifying prototypes for each cluster, and using them as
explanations. While effective for small tabular datasets, this approach falls
short in providing descriptive prototypes for clusters in large datasets.

Conversely, post hoc methods for explanations by examples dealing with DNNs
and large datasets typically employ case-based reasoning approaches. These
approaches utilize a proxy model, such as a K-NN classifier [39], to explain
the black-box by learning a mapping between them. Approaches of this category
differ in the information provided to the proxy. The simplest approach is to
directly use the training samples to train the proxy. However, since typically
that not all features are equally important for a given model, several
approaches propose weighting the features to yield more informative
explanations [183, 172]. For example, a popular technique involves weighting
each feature in the sample using feature attribution scores, thereby linking
the explanations to the important part of the input [5, 105, 106]. Lastly,
another viable approach is to use the activations of the last layers of a DNN
as samples to train the proxy [179] or a meaningful subset of them [108].

#####  3.1.4 Counterfactuals

Counterfactuals are samples similar to the current input but associated with a
different prediction. Finding counterfactuals for DNNs dealing with non-
tabular data is challenging. Indeed, DNNs work with raw data, where there are
no formal constraints on the position and values of the features, and the
number of features involved is large. Therefore, it is challenging to generate
plausible counterfactuals. Moreover, the decision process of these networks is
extremely sensitive, and it is often possible to obtain a different prediction
using adversarial attacks that change the value of a few features in a way
that is not discernible to a human. While these modifications align with the
definition of counterfactuals (Section 2.2.1), they are often meaningless and
cannot be considered explanations. Conversely, counterfactual explanation
methods aim to provide counterfactuals that are both plausible and meaningful.

The pioneer work of this category is the method proposed by Wachter et al.
[251]. This method generates counterfactuals by an iterative perturbation
process guided by a loss function that minimizes the difference between the
predictions on the perturbed instance, the desired outcome, and the L1 norm of
the perturbations. Successive works propose alternative losses that take into
account additional factors like the closeness of features [127, 46],
plausibility [271], or the distance from a set of prototypes [138]. One of the
drawbacks of these methods is their latency due to the iterative process. One
possible solution to mitigate such an issue is to use generative models [135,
16, 107] or genetic algorithms [212]. However, since these procedures are
black-boxes themselves, it is difficult to understand why a particular
counterfactual has been selected as a good candidate.

Another possibility is to remove features from the current input until the
prediction is flipped [247, 193] by guiding the process with feature
attribution methods. However, feature attribution methods focus on the feature
important for a specific instance and do not detect discriminative features
that the model uses to discriminate among classes, limiting the applicability
of these approaches. Connected to this line of research, some works aim at
extracting the contrastive features [260, 67, 101], which are highly
discriminant features for a class and uninformative for the others [72]. These
methods can be considered complementary to methods that select counterfactuals
from a dataset based on user-specified properties and classes [189].

####  3.2 Self-Explainable Deep Neural Networks

While post-hoc methods represent the most popular tool for explaining DNNs,
recently, the field has observed a rising interest in the so-called self-
explainable DNNs. These networks return explanations alongside their
predictions or provide designs that can be easily inspected to provide
explanations. The advantage of these methods is that explanations are fast to
compute and are directly linked to the decision process of the networks.
Moreover, they represent the natural next step for the progress in machine
learning. Self-explainable DNNs can be divided into three categories:
prototype-based, constraints-based, and attention-based.

Prototype-based networks have been introduced by Snell et al. [223] to deal
with few-shot classification. In their networks, prototypes are computed as
the average of the learned embedding of a set of data points. However, since
the prototypes are the average of multiple embeddings, they are not
interpretable. The concept of prototypes has been merged with the concept of
self-interpretable neural networks [8] in ProtoPNet [33]. In this case, the
network learns a set of prototypes representing a _part_ of the input (i.e., a
set of features semantically linked). Given an input, the network compares the
input’s latent representation against the learned prototypes and computes the
prediction based on the similarity between the prototypes and the input
representation. While in the networks proposed by Snell et al. [223],
prototypes are not associated with semantic meaning, ProtoPNet enforces
semantics by using losses and a projection phase that projects the prototypes
to real training points. ProtoPNet has been recently extended to enforce
diversity [256], a better prototype organization [77, 201, 202, 166], and
semantic corrispondence [167] of prototypes. Networks inspired by the
ProtoPNet design have also been proposed in reinforcement learning [109, 191],
sequence classification [156], healthcare [160], and graph classification
[283].

The interpretability is ensured by the fact that the prototypes are real
samples and can be visualized. Moreover, the classification layer is easily
interpreted since the predictions are based on a weighted average of the
prototypes’ activations. By comparing the input and the most activated
prototypes, a user can extract insights about important features in the input,
similar to feature attribution. Conversely, by extracting samples close in the
latent space to the prototypes, a user can extract global explanations by
example, both positive and negative [218] (i.e., this does not look like that
prototype).

Constraint-based architectures encourage the network to learn more
interpretable representations in the form of disentangled or concept-aligned
representations [115, 244, 35]. Concept Bottleneck Modelss (CBMs) [115] is an
example of such architecture. Given a latent representation of a sample, CBMs
are trained to predict the presence or absence of a set of pre-defined
concepts in the sample. The probabilities of all the concepts are then
combined to classify the sample. The idea is that a user can look at the
probabilities associated with the set of concepts to understand which ones are
influencing the most the prediction. Moreover, the user can also intervene by
changing the probability of a concept to the desired one. Several works extend
CBMs by improving the concept representations [78], the performance [55, 276],
and the induced bias [174, 272]. Nonetheless, the induced bottleneck limits
the expressiveness of the network, and thus their performance is usually lower
than the black-box counterparts.

An alternative approach is to enforce the latent disentanglement without
introducing the bottleneck [244, 35, 229, 139]. This goal can be achieved
either using additional semantic layers that project the latent representation
in another more aligned subspace [139] or using a normalization layer, as
proposed in Concept Whitening [35]. Despite the progress and improvements in
performance, several open problems are connected to the usage of these
networks, such as concept leakage and dependencies related to the concept
dataset or models.

Attention-based architectures use attention modules to improve the performance
and the interpretability of the decision process [285, 129, 61]. The idea is
that since attention is a weighted sum of the vector representations,
attention weights can be used directly as feature attribution scores as long
as the vector representations are meaningful. Attention has been used to
discover relations such as the coreference and syntax in natural language
processing tasks [235, 38, 249] and modality relations in multimodal models
[32]. For architecture employing multiple attention layers, Abnar and Zuidema
[2] propose a post-hoc intrinsic method to reconstruct the flow of attention
weights along the network and compute feature attribution based on the flow.

Several works in literature explored the conditions under which attention can
be considered as a reliable proxy for explanations [265]. For example,
Wiegreffe and Pinter [264] and Serrano and Smith [211] find altering the
attention weights does not affect the predictions of some configurations of
attention-based architectures, casting doubts on their reliability. In this
case, the problem is caused by input dispersion, a phenomenon connected to the
accumulation of information from different sources in a single vector
representation.

Finally, apart from these general categories, there are several other self-
explainable DNNs tailored for specific tasks or domains [40, 71] that combine
concepts of the previous categories or propose novels one [61, 34, 198, 195],
lacking however in generalizability.

Overall, since the area of self-explainable DNNs is relatively recent and
emerging, several open challenges exist to address. Among them, we can mention
the limited generalizability since most of the approaches are tested only on
specific families of architectures, the limited diversity among the
approaches, the accuracy-explainability trade-off, and the bias induced by
their design.

####  3.3 Relations with Thesis Contribution

This thesis contributes to the research efforts in explaining DNNs outlined in
the previous section by: (i) proposing novel designs for self-explainable
DNNs; (ii) advancing the understanding of features learned by DNNs; and (iii)
discussing the integration of visual analytics and explanation methods to
deliver explanations to users.

The proposed designs of self-explainable DNNs address issues related to
limited generalizability and diversity, to the usage of custom training
recipes, and the accuracy-explainability trade-off. While existing self-
explainable DNNs are primarily applied to vision tasks and convolutional
neural networks (Section 3.2), our contributions extend to novel domains and
architectures. Specifically, our proposed designs based on concepts and
prototypes (Chapter 4 and Chapter 6) relax common assumptions, like the
availability of pre-trained models, and can be applied to various families of
graph neural networks, an area often overlooked by state-of-the-art
approaches. The generalization contribution is directed not only to the type
of data and the families of covered architectures but also to the type of
normalization layers, an unexplored area in the current literature.

While these designs contribute to the literature in terms of generalization,
they reiterate prototypes and concept-based architectures and use custom
training recipes to train the models, following the current literature.
However, using recipes different from those used to train the black-box models
can cause instability in the training process when applied to settings where
the models use non-standard recipes. These issues are addressed by the
proposed designs of self-explainable DNNs based on memory modules (Chapter 5).
Exploiting memory modules for explainability purposes represents a novelty in
the field, increasing the diversity of approaches. These designs can be
directly applied to black-boxes models following their training settings. The
combination of memory modules and attention mechanisms mitigates the problem
of input dispersion, typical of attention-based models, and is more flexible
than current popular designs since it can provide proxies for feature
attribution, explanations by example, and, for the first time for a self-
explainable DNNs, for local counterfactuals.

The designs mentioned above introduce novel architectures replacing the
current black-box models. Conversely, the third and fourth parts of the thesis
contribute to the research about explaining black-box DNNs without modifying
them.

The Part III advances the state-of-the-art feature probing methods by
investigating a wider spectrum of neuron behavior than the one investigated in
past research (i.e., only the highest activations) and proposing metrics to
evaluate such methods. Specifically, the proposed method overcomes the issues
related to the computational complexity of the previous approaches [163] and
provides explanations that better capture the superposition of neuron
activations [52]. Additionally, due to the wider analyzed spectrum, the thesis
contributes to this area by shedding light on novel overlooked activation
phenomena connected to the activations of popular DNNs.

Finally, Part IV contributes to the literature by reviewing visual analytics
systems as a means to provide explanations computed by XAI methods to the
user. This effort targets exclusively VA systems dealing with DNNs and using
XAI methods to enhance their interpretability, and it represents the first
effort in this direction. Indeed, current VA literature [7, 89] targets more
general settings, and XAI literature has limited awareness of these tools.
Therefore, it is unclear whether the current VA solutions are effective and
what the strengths and weaknesses of the current literature are. Both these
contributions, together with increasing the awareness of the XAI community and
paving the steps for increasing the collaborations, are discussed in the Part
IV.

##  Part II Self-Explainable Neural Networks

###  Chapter 4 Prototype-based Interpretable Graph Neural Networks

This chapter represents the entry point of the thesis contribution on self-
explainable designs for DNNs. As mentioned in Section 1.1, these designs aim
to enhance the interpretability of DNNs and help developers to debug and
improve their models. Contextually, these designs aim to reduce the typical
performance trade-off of self-explainable DNNs and broaden the applicability
of such methods. The idea shared by all the techniques proposed in this part
of the thesis is to insert the novel components into the DNN without
disrupting its structure and preserving most of its representation power.
Therefore, the proposed layers are placed in between the feature extractor
$f_{FE}$ and the classifier $f_{CL}$ (Section 2.1).

As a first step, this chapter proposes a self-explainable prototype-based
layer for GNNs inspired by the design of prototype-based architectures [33]
commonly used for image classification. Specifically, the resulting
architecture, called Prototype-based Interpretable Graph Neural Network
(PIGNN), learns a set of prototypes representing nodes and uses the similarity
between the node representations of the current input and the learned
prototypes to perform both graph and node classification. This design enhances
the interpretability of black-box GNNs, supporting developers in understanding
and improving their models. Indeed, this design supports the identification of
pitfalls of the dataset or in the training setup as well as error analysis
through explanations by examples and feature attributions.

The chapter is organized as follows: Section 4.1 describes the proposed
prototype layer and how users can probe its behavior; Section 4.2 evaluates
PIGNN both in terms of performance and explanations quality; Section 4.3
discusses alternative design choices for PIGGN; finally, Section 4.4
summarized the contributions of this chapter.

Figure 4.1: The architecture of PIGNN when the task is graph classification.

####  4.1 Design

As previously mentioned, the PIGNN design introduces a layer after the feature
extractor $f_{FE}$ to enhance interpretability. This layer includes a set of
$m$ learned vectors $\bm{P}\in\mathbb{R}^{m\times emb}=\\{p_{j}\\}^{m}$,
termed _prototypes_. Each prototype is optimized during training to represent
a node embedding of dimension $emb$. Given an input $\bm{x}_{i}$, the feature
extractor returns the latent representation
$\bm{h}=f_{FE}(\bm{x}_{i})\in\mathbb{R}^{n\times emb}$, where $n$ is the
number of nodes in the graph input. The latent representation is then compared
against the full set of prototypes, producing a score that estimates their
similarity:

$\bm{s}=Sim(\bm{h},\bm{P})\in\mathbb{R}^{n\times m}$ (4.1)

Each prototype is associated with $n$ scores, reflecting how strongly the
semantic encoded in the prototype is represented in each node embedding. The
layer employs global Max Pooling (Section 2.1) to select the maximum score for
each prototype:

$\bm{a}_{i}=max_{n\in Np}(\bm{s})\in\mathbb{R}^{n\times m}$ (4.2)

The resulting scores can be regarded as how much the current graph input
activates each prototype. In the case of graph classification, the classifier
takes the activation scores as input to make predictions (Figure 4.1):

$\bm{y}_{i}=f_{CL}(\bm{a}_{i})$ (4.3)

For node classification, the classifier directly utilizes the similarity
scores as inputs, as each node is associated with a prediction:

$\bm{y}_{i}=f_{CL}(\bm{s}_{i})$ (4.4)

#####  4.1.1 Training Process.

PIGNN, like all ProtoPNet [33] variants, requires a custom training recipe.
Each training epoch comprises three phases: weight optimization, prototype
projection, and classifier optimization.

###### Weight Optimization.

This phase adjusts the learned latent space of the features extractor to
cluster relevant subgraphs around the prototypes and to separate prototypes
from each other. To achieve the goal, the feature extractor and the prototype
layer are optimized by minimizing the following loss [256]:

$L=L_{pred}+\lambda_{1}L_{cluster}+\lambda_{2}L_{orth}-\lambda_{3}L_{class\\_separation}-\lambda_{4}L_{separation}$
(4.5)

and $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$ are
hyperparameters that balance the weight of the losses. Here, $L_{pred}$ is the
task loss used to optimize the model (e.g., cross-entropy between prediction
and ground truth):

$L_{pred}=\frac{1}{n}\sum_{i=1}^{n}CrossEntropy(f(x_{i}),y)$ (4.6)

$L_{cluster}$ encourages subgraphs important for similar predictions to
cluster together:

$L_{cluster}=\frac{1}{n}\sum_{i=1}^{n}{min}_{j:\bm{p}_{j}\in
P_{y_{i}}}min_{\bm{h}\in\text{khop\\_subgraph}(x_{i})}d(\bm{h},\bm{p_{j}})$
(4.7)

where $khop\\_subgraph(x\\_i)$ is the set of all the subgraphs of the graph
$x_{i}$ such that they are centered in one of the nodes in $x_{i}$ and include
all the neighbors of that node at a maximum distance equal to $k$ and
$d(\bm{h},\bm{p_{j}})$ is the distance between the latent representation and
the prototype $\bm{p_{j}}$ computed as:

$d(\bm{h},\bm{p_{j}})=\frac{\bm{h}^{T}\bm{p_{j}}}{\lVert\bm{h}\rVert}$ (4.8)

$L_{orth}$ enforces the orthonormality of prototypes, avoiding prototypes
converging toward the same semantics:

$L_{orth}=\sum_{k=1}^{K}\lVert P_{k}P_{k}^{T}-\mathbb{I}\rVert$ (4.9)

$L_{class\\_separation}$ encourages separation between prototypes associated
with different classes:

$L_{class\\_separation}=\frac{1}{\sqrt{2}}\sum_{k_{1}=1}^{K-1}\sum_{k_{2}=k_{1}+1}^{K}\lVert
P_{k_{1}}P_{k_{1}}^{T}-P_{k_{2}}P_{k_{2}}^{T}\rVert$ (4.10)

Finally, $L_{separation}$ promotes differentiation between prototypes
associated with other classes:

$L_{separation}=\frac{1}{n}\sum_{i=1}^{n}{min}_{j:\bm{p}_{j}\notin
P_{y_{i}}}min_{\bm{h}\in\text{khop\\_subgraph}(x_{i})}d(\bm{h},\bm{p_{j}})$
(4.11)

Note that during this phase, the weights of the classifier $f_{CL}$ are
frozen. The total loss is a generalization of the ones proposed in Chen et al.
[33], which can be recovered by setting $L_{orth}$ and
$L_{class\\_separation}$ to zero.

###### Prototype Projection.

Since the first phase can produce out-of-distribution or irrealistic
prototypes, this phase maps each prototype to the closest subgraph in the
training dataset:

$\bm{p}_{j}=\operatorname*{arg\,min}_{\bm{h}\in
FE_{j}}\lVert\bm{h}-\bm{p}_{j}\rVert$ (4.12)

where $FE_{j}$ is the set of node embeddings returned by the feature extractor
for all the samples in the dataset of the same class of the prototype
$\bm{p}_{j}$. This phase is crucial to keep the prototypes interpretable
(i.e., associated with a clear semantic) and visualize them (Section 4.1.2).

###### Classifier Optimization.

This phase optimizes the classifier $f_{CL}$ while keeping the feature
extractor and the prototype layer frozen. It aims to obtain a sparse
connection between prototypes and output neurons, which is a desirable
property for interpretability. To achieve the goal, we use a loss that
penalizes the weights between prototypes associated with a class and the
output neurons corresponding to all the other classes:

$L_{CL}=L_{pred}+\lambda_{5}+\sum_{k=1}^{nc}\sum_{j:\bm{p}_{j}\notin
P_{k}}abs(w^{j,k}_{CL})$ (4.13)

where the weight $w^{j,k}$ connect the the prototype $\bm{p}_{j}$ to the
output neuron corresponding to the class $k$.

#####  4.1.2 Explanations

The decision process of the described PIGNN can be probed in two ways. The
first consists of understanding the semantics of the learned prototypes,
shedding light on what the model deems important during the training process.
The second leverages prototype activation to yield node attributions. To
understand the semantics of the learned prototypes, we can use explanations by
examples, considering prototypes as inputs. As described in the previous
section, during the second training stage, the prototype is projected into one
of the subgraphs included in the dataset, becoming a point in the training
distribution of node embeddings. Consequently, we can extract neighbors of
each prototype using an algorithm like K-NN [39] over the node embeddings of
dataset samples. Then, we can use and visualize the neighbors as explanations
by examples, helping users understand the semantics encoded by the prototypes.

Regarding node attribution, note that, by design, a node embedding encodes
information of the _k-hop subgraph_ originating from the node. This subgraph
includes all the neighbors within the radius $k$, where $k$ denotes the number
of layers before the considered embedding. The k-hop subgraph that excites the
prototype the most is identified by selecting the node embedding with the
highest similarity to the prototype (determined by Equation 4.2). Feature
attribution for each node $i$ in the k-hop subgraph is then computed as the
sum of similarities between its node embedding and each prototype associated
with the prediction. This computation is weighted by the learned weights
connecting the prototypes and the predicted class:

$e(x_{i})=\sum_{p_{j}\in\bm{P}_{c}}w_{jc}s_{ij}$ (4.14)

where $x_{i}$ is one of the nodes in the k-hop subgraph, $c$ is the predicted
class for the input $x$, $w_{jc}\in W_{CL}$ denotes the learned weight of the
classifier connecting the prototype $p_{j}$ with the output neuron
corresponding to the class $c$, and $\bm{P}_{c}$ is the set of prototypes
associated with the class $c$.

This process can be applied to all nodes in the graph. However, given that
graphs are typically compact structures and tasks often involve identifying
specific substructures, we propose considering only the k-hop subgraph of the
current input corresponding to the most activated prototype. This approach
yields explanations that are both easy to understand and sparse, aligning them
with the inherent characteristics of graph structures.

####  4.2 Experiments

#####  4.2.1 Performance

This section evaluates the performance of popular GNNs augmented with the
proposed prototype layer. Specifically, we train GCN [113], GAT [246], and GIN
[268] on various datasets: BA-shapes [273], BA-Community [273], MUTAG [44],
BBBP [149], BACE [230], BA-2Motifs [141], Tree-Grid [273], Cora [151],
Citeseer [64] and Pubmed [210]. BA-shapes, BA-Community, Tree-Grid, and
BA-2Motifs are syntenic datasets built for node classification (BA-shapes, BA-
Community, Tree-Grid) and graph classification (BA-2Motifs). MUTAG, BBBP, and
BACE are molecule datasets for graph classification, where the task is to
classify molecules as active or inactive. Cora, Citeseer, and Pubmed are
datasets for node classification, where nodes represent scientific
publications, and the edges represent citations. Differently from the standard
procedures for prototype-based models in vision, we train these models from
scratch since there are no publicly available pre-trained GNNs on large
corpora. Additionally, we jointly optimize the feature extractor and the
prototype layers during the same training process. We adopt the setup of Ying
et al. [273], training 3-layers GCN, GAT, and GIN with 128 units each. Models
are trained with and without prototype layers using 15 different seeds.
Following [33] and [256], we set the number of prototypes to ten per class,
the value of $\lambda_{1}$ to 0.8, the value of $\lambda_{2}$ to 1e-7, 0.08,
the value of $\lambda_{3}$ to 0.7, the value of $\lambda_{4}$ to 0.8, and the
value of $\lambda_{5}$ to 1e-3.

Table 4.1: Avg. Accuracy and standard deviation over 15 runs of GNNs with and without prototype layers on synthetic datasets. Model | BA-2Motifs | BA-Community | BA-Shapes | Tree-Grid
---|---|---|---|---
GCN | 87.20 $\pm$ 4.19 | 97.34 $\pm$ 0.14 | 91.23 $\pm$ 1.08 | 84.79 $\pm$1.76
\+ Proto | 59.00 $\pm$ 4.12 | 97.31 $\pm$ 0.29 | 96.30 $\pm$ 0.42 | 97.17 $\pm$ 0.59
GAT | 51.20 $\pm$ 2.32 | 86.61 $\pm$ 1.76 | 76.54 $\pm$ 1.18 | 58.49 $\pm$ 0.00
\+ Proto | 50.10 $\pm$ 3.08 | 87.94 $\pm$ 1.66 | 42.86 $\pm$ 0.00 | 58.49 $\pm$ 0.00
GIN | 99.50 $\pm$ 0.67 | 92.34 $\pm$ 1.95 | 89.79 $\pm$ 3.89 | 87.03 $\pm$ 1.93
\+ Prototypes | 99.80 $\pm$ 0.40 | 98.36 $\pm$ 0.34 | 96.96 $\pm$ 0.14 | 98.67 $\pm$ 0.28
Table 4.2: Avg. Accuracy and standard deviation over 15 runs of GNNs with and without prototype layers on molecular datasets. Model | BBBP | MUTAG | BACE
---|---|---|---
GCN | 85.75 $\pm$ 0.61 | 77.00 $\pm$ 6.78 | 76.32 $\pm$ 1.74
\+ Proto | 87.61 $\pm$ 1.18 | 80.50 $\pm$ 5.68 | 79.28 $\pm$ 1.67
GAT | 87.32 $\pm$ 1.20 | 77.50 $\pm$ 8.14 | 78.22 $\pm$ 2.66
\+ Proto | 87.02 $\pm$ 0.73 | 79.00 $\pm$ 6.24 | 77.57 $\pm$ 2.44
GIN | 85.61 $\pm$ 1.56 | 82.50 $\pm$ 7.50 | 75.40 $\pm$ 1.45
\+ Proto | 87.17 $\pm$ 1.05 | 85.50 $\pm$ 3.50 | 79.34 $\pm$ 2.68

We compare the black-box models against the same models augmented with a
prototype layer in synthetic (Table 4.1) and molecular datasets (Table 4.2) to
assess the validity of the augmentation. The results suggest that adding the
layer to the black-box model and jointly training them does not compromise
performance. In most configurations, models with prototype layers exhibit the
same or higher performance than those without prototypes. Exceptions include
GAT on BA-Shapes and GCN on BA-2motif, which can be addressed using a
simplified loss (Section 4.3). We observe that the performance gain depends on
the particular combination of dataset, model, and tasks. The highest gap in
accuracy is reached on node classification tasks and synthetic datasets and by
the GIN model. These results align with the ones reported by Wang et al. [256]
on image data for architectures augmented by prototypes and Xu et al. [268]
for GIN superiority. They also suggest that the better the model, the higher
the gap between the black-box and the augmented model.

Table 4.3: Avg. Accuracy and standard deviation over 15 runs of GNNs with and without prototype layers on citation datasets. | Citeseer | Pubmed | Cora
---|---|---|---
GCN | 73.60 $\pm$ 0.35 | 74.52 $\pm$ 0.54 | 79.09 $\pm$ 0.81
GCN + Proto | 70.72 $\pm$ 1.01 | 76.77 $\pm$ 0.86 | 78.65 $\pm$ 0.80
SEGNN | 74.19 $\pm$ 0.51 | 76.92 $\pm$ 0.30 | 79.66 $\pm$ 0.56
ProtGNN | 53.07 $\pm$ 1.98 | 75.93 $\pm$ 1.14 | 59.73 $\pm$ 1.95

With the approach validated, we compare its performance against an alternative
explainable GNN based on prototypes (ProtGNN [283]), another type of
explainable network (SEGNN [40]), and the black-box models on Cora, Citeseer,
and PubMed (Table 4.3). ProtGNN is a prototype-based model that computes
prototypes as graph embedding and represents the alternative configuration of
prototype-based layers for GNN. SEGNN is an explainable GNN that classifies
nodes based on K-nearest nodes found on the basis of the sub-graphs
similarity.

While SEGNN achieves the best performances, PIGNN closely matches its
performance on two over of three datasets and outperforms the alternative
prototype-based design in all tasks. Lower prototype-based network performance
against SEGNN suggests ten prototypes (Section 4.3) may be insufficient to
capture enough patterns in citation networks. However, in the next section, we
observe that better-quality explanations of these networks somewhat mitigate
this issue.

#####  4.2.2 Explanations

((a))

((b))

Figure 4.2: Avg. Fidelity and standard deviation achieved by the intrinsic
method, DeepLIFT, and GNNExplainer on GCN (a) and GAT(b) models across several
datasets. Figure 4.3: Avg. Fidelity and standard deviation reached by the
intrinsic method, DeepLIFT, and GNNExplainer on GIN models across several
datasets.

This section discusses and evaluates the explanations retrievable using PIGNN.
We begin by evaluating the quality of feature attributions computed using
prototype activations in terms of $Fidelity+^{prob}$ (Equation 2.22). We
compare its scores against DeepLIFT [214], a popular model-agnostic post-hoc
extrinsic method, and GNNExplainer [273], a recent feature attributions method
tailored for GNNs. We compare the performance of both models with and without
prototypes.

Figure 4.2(a), Figure 4.2(b), and Figure 4.3 summarize the explanation quality
of explanation computed on synthetic and molecular datasets over GAT, GCN, and
GIN models. Overall, PIGNN’s explanations exhibit better quality than
extrinsic methods. In only a few cases, the combination of black-box models
and extrinsic methods outperforms PIGNN. When this occurs, the gap is small,
and the explanations require more computation time (up to 20x for
GNNExplainer). The good quality is also confirmed in Table 4.4, where PIGNN
outperforms SEGNN and ProtGNN in terms of both $Fidelity+^{prob}$ and
$Fidelity+^{ROC-AUC}$ when masking the most important subgraph. Notably, SEGNN
is the worst performer in terms of interpretability. We hypothesize that its
low performance is due to its dense decision process, reliant on the
similarity between k-hop subgraphs, and thus, able to recover the prediction
using similarities among other nodes. Conversely, prototype-based networks
rely on the bottleneck induced by the prototypes, making their decision
process more sparse and sensitive to removing crucial subgraphs.

Figure 4.4: Sub-graphs associated with the prototypes learned by PIGNN based on GCN on the Ba-2Motifs dataset. The most activated node is marked with a red borderline. The blue shade encodes the importance of the node for the subgraph. Figure 4.5: Visualization of explanation for a prediction. The red border indicates the analyzed node; the Input Activations column reports the similarity between the graph’s nodes and the prototype displayed in the same row in the Prototype Activation column. Table 4.4: Explanations performances on big datasets. $Fidelity+^{prob}$ and $Fidelity+^{ROC-AUC}$ (Equation 2.24) are reported for the self-interpretable models and the extrinsic methods on the black-box GCN model. |  $Fidelity+^{prob}$ (↑) |  $Fidelity+^{ROC-AUC}$ (↓)
---|---|---
| Citeseer | Pubmed | Cora | Citeseer | Pubmed | Cora
PIGNN | 0.29 $\pm$ 0.03 | 0.24 $\pm$ 0.02 | 0.50 $\pm$ 0.06 | 0.81 $\pm$ 0.01 | 0.87 $\pm$ 0.01 | 0.88 $\pm$ 0.02
SEGNN | 0.03 $\pm$ 0.00 | 0.02 $\pm$ 0.01 | 0.10 $\pm$ 0.01 | 0.86 $\pm$ 0.02 | 0.90 $\pm$ 0.03 | 0.96 $\pm$ 0.00
ProtGNN | 0.02 $\pm$ 0.02 | 0.06 $\pm$ 0.02 | 0.07 $\pm$ 0.07 | 0.81 $\pm$ 0.03 | 0.89 $\pm$ 0.02 | 0.83 $\pm$ 0.01

Now, we describe how developers can exploit the design and characteristics of
PIGNN to understand the model’s behavior. The examples presented here are
drawn from a PIGNN model based on GIN and trained on the Tree-Grid dataset.
Nevertheless, the same approach is applicable across different scenarios.

Figure 4.4 shows the globally most influential k-hop graphs used by the model
for making predictions. We can note that the first two rows of the image
display a diverse array of patterns associated with class 0, including various
tree structures. Conversely, the lowest rows show duplications in the
prototypes for class 1. By inspecting the dataset, a developer could verify
that there is low variability in the patterns associated with class 1, most of
which are associated with 3x3 grid patterns. These dataset’s characteristics
cause the convergence of PIGNN towards the recognition of class 1 associated
with the recognition of 3x3 grid patterns, captured by all the prototypes.
Once that the problem has been identified, the developer can mitigate the
issue, for example, by limiting the number of prototypes or increasing
diversity in the dataset.

The visualization of prototypes, together with the similarity associated with
them, can also be used to understand individual predictions and perform error
analysis, as shown in Figure 4.5. In this particular case, the model
incorrectly predicts the class of a node as 1. This misclassification is
indicated by a high similarity score of 0.7 with the fourth prototype, which
represents class 1. The analysis of the prototype’s meaning (Prototype
Activations column) reveals that the misclassification arises from the model
detecting a grid pattern within a portion of the input graph, highlighted in
the fourth row of the Input Activation column.

####  4.3 Design Choices

This section discusses alternative design choices for PIGNN and experiments
varying the values of hyperparameters.

###### Number of Prototypes.

Figure 4.6: Accuracy of PIGNN when varying the number of prototypes.

We begin the exploration of alternative design choices for PIGNN by examining
the impact of varying the number of prototypes in the model. As observed in
the previous section, an excess of prototypes may lead to duplication despite
incorporating the orthonormality loss. Figure 4.6 depicts the performance of
PIGNN with 5, 10, and 15 prototypes across the BBBP, BACE, BA-Community, and
Tree-Grid datasets. We observe that there are no clear winners: while
employing a larger number of prototypes generally seems beneficial, this trend
does not hold true for BBBP, where using either 5 or 15 prototypes results in
equivalent performance deterioration. Therefore, the number of prototypes is a
hyperparameter to be tuned before training. Both domain-specific insights and
prototype inspection, as shown in the previous section, can be used to guide
the selection of the right number of prototypes.

###### Similarity Projection.

Table 4.5: Fidelity comparison between projecting the similarities towards all the nodes in the graph and only towards the k-hop most important subgraph. | MUTAG | BACE | BBBP | Ba-2Motifs
---|---|---|---|---
k-hop | 0.35 $\pm$ 0.10 | 0.25 $\pm$ 0.06 | 0.18 $\pm$ 0.07 | 0.16 $\pm$ 0.15
Full Nodes | 0.36 $\pm$ 0.13 | 0.24 $\pm$ 0.06 | 0.18 $\pm$ 0.05 | 0.16 $\pm$ 0.15

((a))

((b))

Figure 4.7: Visualization comparison between projecting the similarities
towards all the nodes in the graph and only towards the k-hop most important
subgraph.

A design choice about the explanations is whether to project the prototypes’
similarities onto all the graph nodes or solely onto the k-hop most important
subgraph, as proposed in Section 4.1.2. Table 4.5 compares the fidelity of
both approaches, revealing no discernible difference when considering only a
subset of nodes. This observation is motivated by the nature of graph-related
tasks, where often, identifying a single pattern suffices for graph or node
classification. While interpretability scores remain unaffected, the
visualization of the two approaches, as depicted in Figure 4.7, differs
significantly, with a sparse visualization offering enhanced comprehension and
immediacy.

###### Loss Function.

Table 4.6: Performance comparison between the full and the reduced loss applied to PIGNN. Dataset | Full Loss | Reduced Loss
---|---|---
Citeseer | 70.72 $\pm$ 1.01 | 70.12 $\pm$ 0.90
Pubmed | 76.77 $\pm$ 0.86 | 76.38 $\pm$ 1.07
Cora | 78.65 $\pm$ 0.80 | 78.62 $\pm$ 1.57
BBBP | 87.17 $\pm$ 1.05 | 85.66 $\pm$ 1.74
MUTAG | 85.50 $\pm$ 3.50 | 83.50 $\pm$ 5.50
BACE | 79.34 $\pm$ 2.68 | 80.86 $\pm$ 3.12
BA-2Motifs | 99.80 $\pm$ 0.40 | 98.90 $\pm$ 1.14
BA-Community | 98.36 $\pm$ 0.34 | 96.96 $\pm$ 0.14
BA-Shapes | 96.65 $\pm$ 0.74 | 85.11 $\pm$ 1.41
Tree-Grid | 98.67 $\pm$ 0.28 | 82.42 $\pm$ 6.82
Table 4.7: Examples of specific cases when the reduced loss can work better. Dataset | Model | Full Loss | Reduced Loss
---|---|---|---
BA-2Motifs | GCN | 59.00 $\pm$ 4.12 | 99.40 $\pm$ 0.80
BA-2Motifs | GAT | 50.10 $\pm$ 3.08 | 83.30 $\pm$ 5.92
Table 4.8: Interpretability scores achieved by PIGNN when using the full or reduced loss. |  $Fidelity+^{prob}$ (↑) |  $Fidelity+^{ROC-AUC}$ (↓)
---|---|---
Dataset | Full Loss | Reduced Loss | Full Loss | Reduced Loss
Citeseer | 0.29  $\pm$ 0.03 | 0.29  $\pm$ 0.02 | 0.81  $\pm$ 0.01 | 0.86  $\pm$ 0.01
Pubmed | 0.24  $\pm$ 0.02 | 0.20  $\pm$ 0.01 | 0.87  $\pm$ 0.01 | 0.90  $\pm$ 0.01
Cora | 0.50  $\pm$ 0.06 | 0.33  $\pm$ 0.06 | 0.88  $\pm$ 0.02 | 0.94  $\pm$ 0.00

Another design choice is the one related to the selection of the loss function
to be used during the first training phase. As mentioned earlier, an
alternative loss function to the one outlined in Section 4.1 is proposed by
Chen et al. [33], which omits the orthogonal and class separation losses.
Table 4.6 illustrates that the full loss remains the optimal choice, as the
reduced loss achieves comparable performance on only half of the datasets.
However, specific scenarios, detailed in Table 4.7, demonstrate instances
where the reduced loss outperforms the full one by a significant margin. In
such cases, inherent characteristics of datasets like BA-2Motifs, such as
dependency on a single motif, coupled with additional constraints imposed by
the added losses, may cause the model to be stuck on a local minimum. A search
over hyperparameters tailored to specific cases could mitigate this issue.
Furthermore, as evidenced in Table 4.8, in terms of interpretability, the full
loss consistently demonstrates superiority in both $Fidelity+^{prob}$ and
$Fidelity+^{ROC-AUC}$, thus suggesting that the added losses help the network
to better separate the prototypes and thus improving their usefulness for
interpretability purposes. Therefore, the full loss represents the best
choice.

###### Transfer Learning

Figure 4.8: Avg. Accuracy and standard deviation of PIGNN when jointly trained
with the feature extractor (Classic), trained after a pre-training on the same
dataset (Pre-training), and fine-tuning a model trained in a different dataset
via transfer learning (Transfer)

Finally, unlike other prototype-based architectures [33, 256], PIGNN does not
use pre-trained models and jointly trains both the feature extractor and the
prototype layer simultaneously. Here, we explore the alternative scenario of
transfer learning. It’s worth noting that no publicly available pre-trained
models on large graph datasets exist due to the disparate meanings of edges
and nodes across different tasks. Therefore, we test transfer learning by pre-
training the feature extractor in a dataset and then fine-tuning the whole
model using the target dataset. Note that we select the first dataset among
the ones sharing the same data type (i.e., molecular). We also consider the
case where we pre-train the feature extractor and then fine-tune the same
model augmented with the proposed layer on the same dataset.

Figure 4.8 demonstrates that simultaneously training from scratch the feature
encoder and the prototype layer yields superior results, outperforming the
alternative in all considered tasks. The lower performance of transfer
learning was expected, given that molecules from different datasets often
belong to separate chemical spaces, thereby leading to the problem of the
domain of applicability and lowering the general performance. On the other
hand, the better performance and the large gap between simultaneous training
and a pre-training phase are surprising and underscores the potential for
further exploration of this phenomenon in different domains.

####  4.4 Contributions

This chapter contributed to the research on self-explainable neural networks
in the emerging field of GNNs. Specifically, we investigated the application
of prototype layers to graph data. We analyzed the differences and specific