# A Survey on Explainable AI for Sixth Generation (6G) Open Radio Access
Network (O-RAN): Architecture, Use Cases, Challenges and Research Directions

Bouziane Brik, , Hatim Chergui, , Lanfranco Zanzi, , Francesco Devoti, , Adlen
Ksentini, , Muhammad Shuaib Siddiqui, Xavier Costa-Pérez, , and Christos
Verikoukis B. Brik is with the University of Bourgogne, France (e-mail:
bouziane.brik@u-bourgogne.fr).H. Chergui and M.-S. Siddiqui are with i2CAT
Foundation (e-mail<EMAIL_ADDRESS>shuaib.siddiqui@i2cat.net).L. Zanzi and
F. Devoti are with NEC Laboratories Europe, 69115 Heidelberg, Germany
(e-mails<EMAIL_ADDRESS>francesco.devoti@neclab.eu).A. Ksentini
is with EURECOM, France (e-mail: adlen.ksentini@eurecom.fr).X. Costa-Pérez is
with NEC Laboratories Europe, i2CAT Foundation, and ICREA, 08034 Barcelona,
Spain (e-mail: xavier.costa@neclab.eu).C. Verikoukis is with the University of
Patras, ATHENA/ISI, and IQUADRAT Informatica, Barcelona 08006, Spain (e-mail:
cveri@ceid.upatras.gr).

###### Abstract

The recent O-RAN specifications promote the evolution of RAN architecture by
function disaggregation, adoption of open interfaces, and instantiation of a
hierarchical closed-loop control architecture managed by RAN Intelligent
Controllers (RICs) entities. This paves the road to novel data-driven network
management approaches based on programmable logic. Aided by Artificial
Intelligence (AI) and Machine Learning (ML), novel solutions targeting
traditionally unsolved RAN management issues can be devised. Nevertheless, the
adoption of such smart and autonomous systems is limited by the current
inability of human operators to understand the decision process of such AI/ML
solutions, affecting their trust in such novel tools. eXplainable AI (XAI)
aims at solving this issue, enabling human users to better understand and
effectively manage the emerging generation of artificially intelligent
schemes, reducing the _human-to-machine_ barrier. In this survey, we provide a
summary of the XAI methods and metrics before studying their deployment over
the O-RAN Alliance RAN architecture along with its main building blocks. We
then present various use-cases and discuss the automation of XAI pipelines for
O-RAN as well as the underlying security aspects. We also review some
projects/standards that tackle this area. Finally, we identify different
challenges and research directions that may arise from the heavy adoption of
AI/ML decision entities in this context, focusing on how XAI can help to
interpret, understand, and improve trust in O-RAN operational networks.

###### Index Terms:

6G, AI, ML, O-RAN, Survey, Trust, XAI

##  I Introduction

###  I-A Context and Motivation

6G wireless networks are growing to revolutionize the way we connect,
communicate, and share information, catalyzing smart services and innovative
applications [1, 2, 3, 4]. 6G is expected to transform mobile communication
networks from the Internet of Things (IoT) to "connected intelligence", by
leveraging Artificial Intelligence (AI) techniques and connecting billions of
devices and people [5, 6, 7, 8]. The promise of immense connected devices,
ultra-low latency, low energy footprint, and extremely high data rates is
expected to enhance the sustainability, connectivity, and trustworthiness of
the next-generation mobile network, and support the development of innovative
applications, such as truly immersive eXtended Reality (XR), smart grid 2.0,
high-fidelity mobile hologram, and Industry 5.0 [9, 10, 11, 12].

The co-existence of such a variety of applications, along with their specific
requirements, demands a versatile mobile network capable of accommodating and
guaranteeing the expected performances by means of accurate and smart
management of network components and resources [13, 14, 15] across different
technological domains, i.e., Radio Access Network (RAN), core network, cloud,
and edge. To this end, both industry and academia are leveraging Network
Slicing (NS), Software Defined Network (SDN), and Network Function
Virtualization (NFV) paradigms to transform the mobile ecosystem into a more
intelligent, energy-efficient, virtual, and software-focused ecosystem [16,
17, 18, 19].

In this context, a global initiative was formed, consisting of over $200$
companies from the telecommunication industry, who collaborated under the
umbrella of the Open Radio Access Network (O-RAN) alliance to introduce a
novel RAN architectural design for the forthcoming generation of mobile
networks (B5G and 6G) [20][21]. The core concept of O-RAN revolves around the
disaggregation of traditional RAN system functionalities and their conversion
into software components, known as Virtual Network Functions Virtual Network
Function (VNF), which are interconnected through standardized and open
interfaces. Additionally, O-RAN introduces a novel hierarchical RAN
Intelligent Controller (RIC) architecture [22], which includes two main
building blocks namely Non Real-Time RAN Intelligent Controller (Non RT RIC)
[23] and Near Real-Time RAN Intelligent Controller (Near RT RIC) [24],
designed to enhance the capabilities and flexibility of the RAN ecosystem. The
Non RT RIC is responsible for executing non-time-critical functions and tasks,
such as policy management,

List of Acronyms 3GPP | 3rd Generation Partnership Project
---|---
4G | Fourth Generation
5G | Fifth Generation
6G | Sixth Generation
A2C | Advantage Actor Critic
AI | Artificial Intelligence
ANN | Artificial Neutral Networks
API | Application Programming Interface
B5G | Beyond Fifth-Generation
BBU | Baseband Unit
BS | Base Station
CAPEX | CAPital EXpenditures
CI/CD | Continuous Integration and Delivery
CNN | Convolutional Neural Network
CP | Control Plane
CT | Continuous Training
CU | Central Unit
DAG | Directed Acyclic Graph
DARPA | Defense Advanced Research Projects Agency
DevOps | DEVelopment and IT Operations
DeepLIFT | Deep Learning Important FeaTures
DL | Deep Learning
DNN | Deep Neural Network
DQN | Deep Q-Network
DRL | Deep Reinforcement Learning
DU | Distributed Unit
eMBB | enhanced Mobile Broadband
eNB | eNodeB
ENI | Experiential Networked Intelligence
ETSI | European Telecommunications Standards Institute
FL | Federated Learning
GAN | Generative Adversarial Network
gNB | gNodeB
GNN | Graph Neural Network
HE | Horizon Europe
IEEE | Institute of Electrical and Electronics Engineers
IG | Integrated Gradients
IoT | Internet of Things
ISG | Industry Specification Group
KL | Kullback-Leibler
LIME | Local Interpretable Model-Agnostic Explanations
LLM | Large Language Model
LO | Log-Odds
LSTM | Long Short Term Memory
MAC | Medium Access Control
MDP | Markov Decision Process
MEC | Multi-access Edge Computing
MLOps | ML system operations
ML | Machine Learning
mMTC | massive Machine Type Communications
MM | Mobility Management
MNO | Mobile Network Operator
MR | Machine Reasoning
MVNO | Mobile Virtual Network Operator
Near RT RIC | Near Real-Time RAN Intelligent Controller
NFV | Network Function Virtualization
NG RAN | New Generation RAN
Non RT RIC | Non Real-Time RAN Intelligent Controller
NR-MAC | New Radio Medium Access Control
NS | Network Slicing
O-Cloud | Open Cloud
O-CU-CP | Open RAN Central Unit Control Plane
O-CU-UP | Open RAN Central Unit User Plane
O-CU | Open RAN Central Unit
O-DU | Open RAN Distributed Unit
O-RAN | Open Radio Access Network
O-RU | Open RAN Radio Unit
OPEX | OPerational EXpenditures
OSC | Open RAN Software Community
PCA | Principal Component Analysis
---|---
PDCCH | Physical Downlink Control Channel
PDCP | Packet Data Control Protocol
PDSCH | Physical Downlink Shared Channel
PHY | Physical
PNF | Physical Network Function
PUCCH | Physical Uplink Control Channel
PUSCH | Physical Uplink Shared Channel
QoE | Quality of Experience
QoS | Quality of Service
R2 | R-squared
RAN | Radio Access Network
rApp | Non RT RIC Applicaiton
RAT | Radio Access Technologies
ReCo | Relative Consistency
RIC | RAN Intelligent Controller
RL | Reinforcement Learning
RM | Resource Management
RNN | Recurrent Neural Network
RSS | Received Signal Strenght
RT | Real Time
RU | Radio Unit
SCM | Structural Causal Models
SDN | Software Defined Network
SHAP | SHapley Additive exPlanations
SLA | Service Level Agreement
SMO | Service Management and Orchestration
SM | Spectrum Management
SVM | Support Vector Machines
TTI | Transmission Time Interval
UE | User Equipment
UP | User Plane
URLLC | Ultra Reliable Low Latency Communication
vBBU | Virtual BBU
VNF | Virtual Network Function
vO-CU | virtual Open RAN Central Unit
vO-DU | virtual Open RAN Distributed Unit
vRAN | Virtual RAN
WG | Working Group
XAI | eXplainable AI
xApp | Near RT RIC Applicaiton
XR | eXtended Reality

network optimization, and long-term analytics, while the Near RT RIC focuses
on time-critical operations and tasks that require low latency and quick
decision-making.

It is easy to claim that AI will play a critical role in the development and
implementation of future network management operations, pursuing better
network performance, cost savings, and enhanced customer experience
[25][26][27]. In this context, O-RAN envisions RIC entities to support
programmable-based functions and logics, featured by the heavy usage of AI
techniques, in particular, Machine Learning (ML) and Deep Learning (DL), to
ease the development of intelligent and flexible RAN applications and reduce
operational complexity [28]. Among others, the AI-based RICs aim to tackle
traditionally hard-to-solve aspects of the RAN domain, such as spectrum
management, mobility, radio resource assignment and scheduling, admission
control, link management, and power allocation [29, 30]. This is particularly
beneficial in the 6G landscape when considering various vertical industries
and their corresponding networking requirements.

Despite a promising future in radio management scenarios, the widespread
adoption of AI techniques in O-RAN opens security threads and requires a deep
understanding of such technologies before their deployment in real systems,
including a characterization of how these techniques perform decisions and
behave based on the knowledge they acquire from the input data [31]. Mainly
due to a lack of trust, transparency, and explainability caused by the opaque
nature of black-box AI models [32], network operators are reluctant to deploy
AI-based applications at the RAN level. Moreover, an erroneous adoption of
AI/ML-based actuators could bring infrastructure providers to face Service
Level Agreement (SLA) violations [33], or in the worst case, lead to potential
failures.

Therefore, it urges us to clearly identify the operational boundaries of AI/ML
models, characterize and understand their behaviour, and prioritize faithful
and trustworthy decision-making processes to enable automated network service
management, while leaving the quality of service unaffected. On that account,
new approaches are required to provide explainable and understandable
decisions [34]. eXplainable AI (XAI) is an emerging paradigm that aims to shed
light on the decision process that is performed by closed (black box) AI
models. The main objective of XAI is to create a transparent and human-
understandable model (white box) that clarifies the internal processes of AI
models, e.g., by determining the contribution of each input feature to an AI
decision or prediction [35]. XAI is crucial to demonstrate the accuracy,
fairness, and transparency of AI models that drive decisions and operations in
the network, thereby instilling trust and confidence in the deployment of AI-
powered components in the O-RAN ecosystem by businesses and organizations [36,
37].

Table I: Existing surveys on O-RAN, XAI, XAI for B5G. H: High, M: Medium, and L: Low. Works | AI | XAI |  B5G/6G support |  O-RAN Architecture |  O-RAN use cases |  Future research directions (B5G, O-RAN, or XAI) | Contribution
---|---|---|---|---|---|---|---
[38] | L | L | H | H | L | H |  | A tutorial on O-RAN framework, by describing recent specifications in terms of
---
architecture, design, and open interfaces.
[39] | L | L | H | H | L | H |  | A short survey on O-RAN’s architecture, benefits, shortcomings, and future
---
directions.
[40] | M | L | H | H | H | H |  | A concise paper on O-RAN architecture. It designed a DL-based resource allocation scheme
---
and discussed future directions.
[41] | M | L | H | H | H | H |  | A short survey on O-RAN’s architecture, benefits, and future directions. It showed
---
the deployment of DL-based scenarios in O-RAN.
|  [42] [43]
---
L | L | H | H | L | L |  | Short review papers that discussed the evolution of RAN architectures towards O-RAN
---
in terms of functionality and implementation.
[44] | L | L | H | M | L | L |  | A concise paper discussed the integration of emergent B5G concepts with O-RAN,
---
such as network slicing and Multi-access Edge Computing (MEC).
[45] | H | L | H | H | H | H |  | A survey on DL/ML-based solutions for RAN/O-RAN. It includes O-RAN architecture
---
description along with its use cases as well as future directions and open
challenges.
[46][47] | H | H | M | L | L | H |  | A review of XAI approaches, in terms of their algorithmic aspects, classifications,
---
application domains, and future research directions.
[37] | H | H | M | L | L | H |  | A review of the main principles and practice of XAI. In particular, the specific pattern
---
recognition models of machine learning are targeted.
[48] | H | H | M | L | L | H |  | A review on a set of key measurement metrics, which can help to measure and evaluate
---
any explainable AI system.
[35] | H | H | H | L | L | M |  | A survey on the use of XAI for B5G/6G networks. It addresses how to design XAI
---
systems for B5G use cases, as well as future research directions in such
context.
[49] | H | H | H | L | L | L |  | A review of DL-based solutions in PHY and MAC layers and their performance
---
vs XAI trade-off.
[50] | H | H | H | L | L | L |  | A review of existing XAI techniques and their applicability to deal with
---
the 6G network challenges.
[51] | H | H | H | L | L | M |  | A survey on the application of XAI to the security aspects of B5G networks as well
---
as future research directions.
This survey | H | H | H | H | H | H |  |  A comprehensive survey on the use of XAI to design transparent and trustworthy
---
O-RAN architecture, covering architectural aspects, use cases, projects,
standardization approaches, and future research directions.

###  I-B Review of Existing Related Surveys

Several studies already addressed the novel O-RAN architecture, highlighting
its novel approach and investigating potential benefits and drawbacks. In
[39], the authors provided a short review of both advantages and limitations
of O-RAN, focusing on the O-RAN architecture and its main modules. The authors
conducted a community survey on the benefits of O-RAN among $95$ researchers
from all around the world. Most of them agreed on the fact that O-RAN will be
the foundation of next-generation networks. Finally, the authors discussed the
benefits, current shortcomings, and future research directions of O-RAN.
Similarly, [40] described the O-RAN architecture and its key concepts. In
addition, the authors present a novel DL-based scheme for radio resource
assignment, validating their performance using data collected from real mobile
network deployments. The authors conclude their work by discussing open
challenges and future research opportunities. Another review study is provided
by [41]. The authors showcase how a DL-based scenario can be deployed on top
of the O-RAN architecture, highlighting the main advantages and shortcomings
of O-RAN.

The evolution of RAN architectures towards the O-RAN proposal both in terms of
functionality and implementation is discussed in [42] [43]. In the same
context, the support of B5G key concepts, such as network slicing and MEC, by
the O-RAN architecture is elaborated by [44][52][53][54].
In our previous work [45], we proposed a survey study on the O-RAN
architecture, discussing the evolution of RAN architectures, and comparing
different studies based on various perspectives. We focused our review on
existing AI-based schemes dealing with the RAN challenges, and show how these
schemes can be supported by O-RAN by considering the deployment of two
realistic DL-based case studies. Similarly, in [38], the authors provided a
tutorial on the O-RAN framework describing recent specifications in terms of
architecture, design, and open interfaces. They also discuss the main open
research challenges and the new innovation possibilities in the O-RAN
architecture, focusing on AI and deep learning.

Besides, the XAI topic is attracting interest from research and industry
domains. Currently, XAI is one of the main programs of the Defense Advanced
Research Projects Agency (DARPA), expected to design efficiently the "third-
wave AI systems" [55]. In [46][47], the authors reviewed and analyzed several
XAI approaches focusing on algorithmic aspects, classifications, and
application domains, identifying several still open challenges and key future
research directions. The main principles and practices of XAI are summarized
in [37]. In particular, the authors target the specific pattern recognition
models of machine learning in order to enhance the understanding of such
models for industry practitioners (data scientists). In [48], the authors
discussed a set of key measurement metrics that can help evaluate explainable
AI systems. In 6G networks context, the authors of [35] discussed the use of
XAI, targeting different 6G use cases (e.g. Industry $5.0$). Similarly, in
[50] the authors highlight existing tools in addition to their use to deal
with 6G network challenges, discussing how to integrate XAI into 6G networks
architecture through a real mobile traffic prediction use-case, and validating
their findings on realistic traffic data. Conversely, the authors of [49]
focused on XAI methods in low protocol layers of mobile networks , e.g.,
Physical (PHY) and Medium Access Control (MAC). In the same context, the
authors of [51] describe the application of XAI related to security aspects,
discussing how XAI can improve the interpretation of AI-based models for a
wide range of security use-cases related to B5G/6G networks.

Table I summarizes the main topics discussed along the above works, and
compares their contributions with respect to our work, in order to provide an
easy understanding of the differentiation features with respect to the state-
of-the-art. Despite the presence of several survey papers discussing XAI and
O-RAN, there is a lack of comprehensive surveys joinlty investigating XAI and
O-RAN aspects able to effectively explore the potential of XAI for developing
responsible, trustworthy, and transparent AI-powered O-RAN architecture. In
addition, although the integration of XAI with B5G networks has been addressed
e.g., in [35][49], such studies do not focus either on the RAN part or
consider the novel O-RAN architecture. Therefore, a comprehensive survey of
XAI and its potential in designing the future O-RAN is greatly needed to guide
the practitioners as well as researchers.

Figure 1: The taxonomy of the article.

###  I-C Main Contributions

The contributions of this paper can be summarized as follows:

  * •

Bridging the gap between O-RAN and XAI: Existing surveys on O-RAN focused on
its enabling technologies, such as hierarchical RAN Intelligent Controller,
open interfaces, and programmable functions. To the best of our knowledge,
there is no survey addressing the potential of human and O-RAN interactions,
through XAI systems. Similarly, existing surveys on XAI targeted different XAI
approaches and their taxonomies, and more recently their applications to
B5G/6G networks. However, discussions on the potential of XAI for O-RAN are
still missing. Therefore, this survey paper aims to bridge this gap by jointly
exploring the key benefits of the introduction of XAI to O-RAN.

  * •

A comprehensive survey of XAI deployment on top of O-RAN: Existing works
studied both O-RAN and XAI separately, i.e. no work has combined both
paradigms in its study. Hence, in this survey paper, we study the promising
deployment of XAI on top of the AI-enabled O-RAN. This includes O-RAN
architecture as well as O-RAN use cases. Furthermore, We study the mapping of
existing O-RAN research solutions to XAI-supported solutions.

  * •

A depth analysis of XAI automation for O-RAN: We provide an exhaustive
analysis of how to automate the whole XAI pipeline on top of O-RAN, in order
to ensure stable performance of deployed XAI techniques. To the best of our
knowledge, no existing work has discussed the automation of XAI process for
O-RAN. We also design new architectures showing the automated deployment of
XAI, for different levels of automation.

  * •

O-RAN Security aspects and XAI: We present the key findings of the official
security risk assessments conducted on the O-RAN environment. We explore the
potential of XAI to significantly improve the security layer of O-RAN, and how
it could be used to build interpretable security threat detection mechanisms.
Additionally, we discuss how XAI can help establish trust among stakeholders.

  * •

Identifying New XAI-related Issues and Promising Research Directions:
Integrating XAI with O-RAN will rise new issues, which should also be
considered in future research studies. Thus, we exhaustively discuss new open
challenges, along with future research directions.

###  I-D Paper Organization

As shown in Fig. 1, the survey is organized as follows. Sec. II provides
background information related to the topics considered in the survey. Sec.
III describes how XAI methods and models can be deployed on top of the O-RAN
architecture, considering three realistic deployment scenarios, and
communication interfaces. Sec. IV gives a literature review of existing
related works in the field, focusing on AI techniques targeting RAN
optimization and highlighting how XAI could enhance their performances. Sec. V
provides an overview of O-RAN use cases taken from the literature and standard
documentation, highlighting how XAI could bring benefits to the considered
scenarios. Sec. VI details an automation pipeline for XAI model training and
deployment in the O-RAN context, involving multiple architectural components
and communication interfaces. Sec. VII provides an overview of security issues
related to the O-RAN architecture, focusing on XAI-related aspects. Sec. VIII
presents the main ongoing projects and standards that are working to promote
the adoption of AI/ML techniques in O-RAN, and show how they can be enhanced
by XAI. Sec. IX highlights and discusses still open challenges along with
their future research directions to deal with them. Finally, Sec. X concludes
this paper. Note that the used acronyms in this paper are described in the
_List of Acronyms_ , in alphabetical order, for ease of reference.

##  II Background

This section provides background information on AI, XAI, and O-RAN topics
which are required to fully understand the potential of XAI techniques in the
O-RAN domain. Firstly, we introduce the two main branches of AI, namely
machine and deep learning, and their characteristics. Secondly, we describe
the main concepts, techniques, and emergent applications of XAI. Finally, we
present the O-RAN architecture along with its main modules as designed by the
O-RAN alliance.

###  II-A Artificial Intelligence (AI)

AI aims to reproduce human intelligence in machines through rules and
algorithms. It combines various different fields including, planning,
reasoning, communicating, learning, interaction, perception, etc. In our
study, we focus on ML/DL as one of the AI branches that require XAI for more
explanations and interpretations.

####  II-A1 Machine Learning (ML)Classes

Machine learning is a sub-field of AI that is based on mathematical and
statistical models to process and build inferences from data patterns.
Usually, machine learning algorithms are categorized into three main classes:

  * •

Supervised Machine Learning: It uses labeled data to build a mapping function
from input variable(s) to output data. It can deal with two different
problems: classification to predict the class a particular data observation
may belong to, and regression to predict a real (continuous) value. In this
machine learning class, a wide range of algorithms have been designed: linear
and logistic regression to learn the relationship between inputs and output
data through a linear line and curve, respectively. Noting that logistic
regression is only used for binary classification since its outputs are in the
range $[0,1]$ [56]. Random Forests or random decision forests is a set of
decision trees that are built for classification and regression. Random
Forests belongs to a class of learning techniques named Ensemble learning
techniques, which also comprises other techniques such as AdaBoost and
Gradient Boosting Machines [57]. Support Vector Machines (SVM) is another
supervised learning algorithm for classification and regression, that builds
learning models from a statistical point of view. SVM is mainly based on
Vapnik-Chervonenkis theory to determine the best hyperplane, separating the
data instances [58]. Artificial Neutral Networks (ANN) emulate the human brain
to connect a set of neurons with each other, through edges with associated
weights. ANN aims to adjust edges’ weights, improving the learning accuracy
and minimizing the loss function. Noting that deep learning is also based on
advanced ANN, with more hidden layers [59].

  * •

Unsupervised Machine Learning: It processes unlabeled data to deduce standard
patterns/information. Unsupervised learning includes clustering, association,
and dimensionality reduction. K-means is a clustering algorithm that groups
data into $k$ different clusters, based on the distance to the centroid of
each group. The distance may be computed using Euclidean, Dynamic time
wrapper, and Manhattan [60]. Association rules enable us to determine the
associations among data observations, which will then help to understand the
relationship between data, for instance, establishing associations between
shoppers according to their purchasing histories or browsing [61]. Principal
Component Analysis (PCA) is a dimensionality reduction algorithm used to
reduce the dataset dimension while keeping the main information. PCA is based
on geometrical projections to project data into new components, named
Principal Components [60].

  * •

Reinforcement Learning (RL): It enables an agent (or set of agents) to
interpret and perceive its environment, take actions, and learn via error and
trial. In particular, the agent gets positive or negative rewards based on its
actions. Thus, the agent aims to devise the optimal policy that is maximizing
the cumulative reward. The most popular reinforcement algorithms are
Q-learning, Deep Q-learning, and advantage actor-critic, where "Q" stands from
Quality [62]. Q-learning consists of finding the optimal action for each state
based on stochastic transitions. Q-learning builds a Q-table containing the
Q-values that correspond to each transition from one state to another. Deep
Q-learning replaces the Q-table with an ANN. Advantage Actor Critic (A2C) is
another reinforcement learning that builds two different networks, called
actor and critic. The actor chooses optimal actions, while the critic network
evaluates their qualities (actions) [63].

####  II-A2 Deep Learning Algorithms

are considered as advanced machine learning algorithms and thus a sub-class of
the machine learning field. Deep learning first enables automated data pre-
processing without human intervention. It also consists of extracting unknown
knowledge from input data through multiple levels. Some well-used deep
learning algorithms are as follows:

  * •

Convolutional Neural Network (CNN): It is a deep learning algorithm mostly
used for image/object classification and recognition. CNN is based on
multilayer neural networks comprising an input layer, a set of hidden layers,
and an output layer. The intermediate (hidden) layers can be composed of
different types of layers including convolutional, pooling, normalization, and
full-connected layers [64].

  * •

Sequence Algorithms: These algorithms are used to deal with problems related
to sequential data and time series, e.g., language translation and speech
recognition. Recurrent Neural Network (RNN) and Long Short Term Memory (LSTM)
are widely spread deep learning algorithms belonging to this class. Their
basic idea is to leverage internal memory units known as cell state, which is
the long-term memory of the system, and the output of the previous point in
time, known as the hidden state, which is the short-term memory. The output
depends on the cell state, the hidden state, and the current input. At the
same time, the long-/short-term memory depends on past LSTM experience and
greatly influences its performance [65]. For example, if the phrase
"Artificial Neural Network" is part of an LSTM model’s training data, the
model would probably predict the word "Network" after seeing the input
"Artificial Neural".

  * •

Generative Adversarial Network (GAN): It is based on CNN to enable generative
modelling. In other words, Generative Adversarial Network (GAN) is applied in
unsupervised learning to learn new knowledge and common proprieties in data,
based on previous information. GAN comprises two modules: a generator module
to create new data, and a discriminator module to evaluate the quality of the
newly generated data.

  * •

Auto-encoder: It is another unsupervised deep learning algorithm applied to
the task of data representation. It aims to learn an efficient coding model of
unlabeled data. Auto-encoder is composed of two main branches: an encoder to
map input data to a, typically compressed, encoded representation of it, and a
decoder to reconstruct the inputs from its encoded representation. It can be
used in different applications including data compression, noise reduction,
and anomaly detection.

  * •

Transformers: They are advanced models that use self-attention mechanisms to
capture dependencies between sequence elements, surpassing traditional
recurrent neural network-based models. By weighing element importance based on
relevance to others, transformers achieve superior contextual understanding.
They excel in tasks like machine translation, text generation, sentiment
analysis, and question answering. Transformers [66, 67] capture long-range
dependencies, generating coherent and contextually rich text which makes them
a strong component in building Large Language Models (LLMs) [68].

###  II-B eXplainable AI (XAI)

In this subsection, we provide the background on XAI and its main concepts,
applications, and ongoing studies.

####  II-B1 Definitions and Key Concepts

XAI is a set of methods and tools that help human users to interpret,
understand, and trust results made by AI-based models [46][37]. XAI is used to
describe AI models, e.g. ML/DL models, their possible biases as well as
expected impacts. In other words, XAI aims to build a white-box model, which
gives information on the inner working of the underlying ML/AI black-box
model. Thus, it helps characterize model fairness, accuracy, and transparency
in AI-enabled decisions. XAI is then vital for businesses and organizations in
providing confidence and trust when deploying AI models [37].

More specifically, the XAI model exposes information about the inner working
of the AI models leveraging on the concepts of explainability and
interoperability. Explainability involves explaining the behavior of complex
models using model-agnostic methods, and it is considered an active
characteristic of a model, referring to any function or action taken by a
model, in order to describe or clarify its internal functions. Whereas,
interpretability, also known as transparency, involves observing a model’s
inner mechanics to understand its behavior. In other words, interpretability
is defined as a passive characteristic of a model, that reflects at which
level a particular model makes sense from the point of view of a human
observer. XAI models incorporate the so-called explanation user interface to
generate a user-understandable explanation and/or interpretation of the
rationale behind decisions taken by the model. Most AI models can be
translated into an equivalent XAI counterpart, at the expense of integrating
additional layers supporting the explanation user interface on top of the
deployed model. Based on the design of the explanation user interface, the XAI
model can provide both explainability and interpretability or only one,
depending on the target human user [46].

Table II: XAI Taxonomy Model Type |  Explainability Basis |  Technique |  Algorithm |  Pros and Cons | Reference
---|---|---|---|---|---
Black-Box Models |  Attributions |  Gradient |  Saliency Maps |  Pros: Simplicity, visual interpretability, widely applicable. Cons: Lack of context, sensitivity to input perturbations, limited to input gradients. | [69, 70]
Gradient x Input |  Pros: Simplicity, direct relevance, feature importance ranking. Cons: Input scaling sensitivity, limited to linear relationships, potential for misleading interpretations. | [71, 72]
Integrated Gradients |  Pros: Baseline comparison, path-based attribution, completeness and sensitivity. Cons: Computationally intensive, baseline selection challenge, linearity assumption. | [73, 74]
Smooth Gradient |  Pros: Noise reduction, robustness to adversarial examples, gradient visualization. Cons: Interpretation challenges, hyperparameter sensitivity, computational overhead. | [75, 76]
|  epsilon-LRP |  Pros: Deep model interpretability, conceptual clarity, attribution preservation. Cons: Complexity, parameter tuning, vulnerability to network architecture. | [77, 78]
Perturbation |  SHAP |  Pros: Theoretical grounding based on game theory, global and local interpretability, consistency. Cons: Computational complexity, high-dimensional data challenge, model approximation dependency. | [79, 80]
|  DeepLIFT |  Pros: Model-agnostic, captures interactions, relevance conservation. Cons: Computational overhead, baseline selection challenge, interpretation complexity. | [81]
|  |  Occlusion |  Pros: Intuitive visual interpretation, robustness to model architecture, spatial localization. Cons: Computational expense, coarseness of occlusion, interpretation subjectivity. | [82, 83]
|  Importance Weights |  GNNExplainer |  Pros: Graph-specific interpretability, node and edge importance, feature relevance analysis. Cons: Complexity, model-specific, interpretation scalability. | [84]
Surrogates |  Local Techniques |  LIME |  Pros: Model-agnostic, local interpretability, simplicity. Cons: Interpretability limitation, instability, assumes linearity. | [85, 86]
Global Techniques |  TREPAN |  Pros: Decision tree interpretability, human-readable explanations, transparent model behavior. Cons: Limited to decision tree models, model-specific, interpretation scalability. | [87]
|  |  Rule-Based |  RuleFit |  Pros: combines decision trees and linear regression to provide interpretable insights into the model’s decision-making process. Cons: may struggle to model highly intricate or complex nonlinear patterns in the data. | [88, 89]
|  RL Reward |  Rule-based |  Reward Shaping |  Pros: Provides explicit guidance to the RL/DRL agent, allowing it to focus on desired behaviors. Cons: Can introduce biases if the reward shaping is not carefully designed. | [90, 91]
|  RL State |  Model-based |  Attention Mechanisms |  Pros: Offers transparency by showing which parts of the input state the RL/DRL agent attends to. Cons: Attention mechanisms do not explicitly explain the agent’s internal reasoning or decision-making process. | [92, 93]
|  Symbolic | Machine Reasoning |  Pros: Provides human-interpretable explanations for model decision and shows explicit reasoning behind decisions, enhancing transparency. Cons: Requires expertise, computationally expensive and may struggle with uncertain or probabilistic information. | [94, 95]
|  Transformers’ Attention Head | Attention Flow Analysis |  Pros: Provides interpretability, enables fine-grained analysis, helps improve models, offers domain-specific insights. Cons: Complexity, lack of unique interpretations, limited context, challenges in generalization. | [96]
|  Visual | SCM |  Pros: Causal understanding, intuitive visualization, identifying confounding variables. Cons: Limited to causal modeling, simplified representation, expert knowledge required. | [97, 98]
|  Text | Caption Generation |  Pros: Contextual understanding, language comprehension, multimodal interpretation. Cons: Subjectivity and ambiguity, lack of fine-grained control, reliance on training data. | [99, 100]
|  Graph | Knowledge Graphs |  Pros: Structured representation, relationship understanding, integration and interoperability. Cons: Knowledge acquisition and maintenance, incompleteness and accuracy, limited context and ambiguity. | [101, 102]
Transparent Models | Logistic / Linear Regression |  Pros: Explainability, trust and accountability, debugging and error analysis. Cons: Performance limitations, vulnerability to adversarial attacks. | [103]
Decision Trees | [104]
K-Nearest Neighbors | [105]
Rule-Based Learners | [106]
Generative Additive Models | [107]
Bayesian Models | [108]
Self-Explainable Neural Networks | [109]

####  II-B2 Taxonomy of XAI Techniques, Applications, and Stakeholders

There are several existing taxonomies in the XAI realm, which can complement
and/or overlap each other. Table II describes an XAI taxonomy that is mainly
inspired by [46, 37], and is based on the following three main criteria:

  * •

Model Transparency: XAI models can be classified based on the target ML
models’ transparency. In this regard, models are classified as interpretable
or complex. Interpretable models are by themselves understandable for human
users. In other words, such models are able to provide the rationale behind
their decisions in an interpretable way to users [46]. Several proposed works
succeeded in interpreting some relatively low-complex ML models, including
logistic/linear regression, decision trees, K-Nearest neighbors, rule-based
learners, etc. [46]. On the other hand, more complex models such as deep
neural networks, in order to be interpretable, have to be approximated by
generating simpler surrogate models that ease the explanation task by means of
a technique known as post-hoc explainability [110]. The model complexity is a
widely considered aspect in the literature related to XAI and is generally
adopted to classify XAI approaches [46].

  * •

Model Agnosticity: This criterion targets complex ML/DL models, where XAI
models can be categorized based on the nature of their target explanations
[46][37]. Indeed, XAI models may consider the internal structure and
functioning of the learning models, such as model weights, to build their
explanations. In such cases, they target a specific ML/DL model. Besides, XAI
models can give their interpretations without considering the internal
functioning and structure of the studied learning models, but only based on
their predictions (ML models). Therefore, such XAI models can be applied to
any black-box ML/DL-based models.

  * •

Explainability Methods: when ML/DL models are considered complex models, some
techniques should be devised and used to interpret such models. Thus, XAI
models rely on several explanation types, to describe how these ML/DL models
output their predictions for any input data.

    * –

Explanations by simplification refer to the techniques that simplify a complex
model and approximate it to an interpretable model, which is easier to explain
[85].

    * –

Feature relevance explanations study and quantify the impact of each input
data, to explain a given ML model’s prediction [111].

    * –

Local explanations focus on a single or particular prediction (output) of ML
models to generate explanations [85].

    * –

Visual explanations aim to generate explanations in a visual way, describing
the inner functioning of ML/DL models [97]. For instance, they could reveal
which set of pixels is the most relevant to recognize content in image
classification tasks. Visual explanations rely on several tools, e.g, graphs,
heatmaps, scatter plots, etc.

    * –

Text explanations generate symbol interpretations of learning models using for
example natural language text, to explain their results [99]. For instance,
they could be used to highlight which words (or forms) are leveraged in
automatic email spam filtering.

Table III: XAI Users XAI Users |  Needs |  Key Application Areas |  Reference
---|---|---|---
Data Scientists and Machine Learning Researchers |  They require XAI techniques to understand and debug complex models, identify biases, and improve model performance |  Model development, debugging, and optimization across various domains such as telecommunications, healthcare, finance, natural language processing, and computer vision |  [85, 112]
End Users and Consumers |  They need explanations to trust and understand AI systems in applications like recommender systems, personalized marketing, and decision support tools |  E-commerce and personalized recommendation systems, healthcare decision support tools, financial advice platforms, and autonomous vehicles |  [113, 114]
Managers and Decision Makers |  They require transparent and interpretable AI models to make informed decisions, assess risks, and gain insights into the AI system’s behavior |  Business intelligence and analytics, risk assessment and management, fraud detection, and regulatory compliance across industries such as finance, healthcare, and manufacturing |  [115]
Developers and Engineers |  They need tools and methods to troubleshoot networks faults/SLA violations, build explainable AI systems, ensure reliability, and meet regulatory requirements |  Building interpretable machine learning models, developing explainable AI frameworks and libraries, ensuring model reliability and security in domains like cybersecurity, telecommunications, and autonomous systems |  [116, 34]
Auditors and Compliance Officers |  They require XAI to assess the fairness, accountability, and compliance of AI systems and to identify potential biases or risks |  Assessing the fairness and legality of AI systems in finance, hiring practices, loan approvals, credit scoring, and regulatory compliance in sectors such as finance and human resources |  [117]
Legal Professionals and Judges |  They need explanations to understand AI decisions, assess legal implications, and ensure transparency and fairness in legal proceedings |  Interpreting AI-driven legal decisions, evaluating algorithmic fairness, ensuring transparency and accountability in legal proceedings, and addressing ethical concerns in areas like criminal justice and civil rights |  [118]
Regulators and Policy Makers |  They require XAI to establish guidelines, standards, and regulations around AI ethics, transparency, and accountability |  Establishing guidelines, standards, and regulations for trustworthy AI in sectors including healthcare, finance, autonomous systems, and data privacy to protect public interests and ensure ethical AI deployment |  [119]

Based on the above taxonomy criteria, several XAI approaches have been
proposed in the literature. In what follows, we present the most popular ones,
highlighting their main features:

  * •

SHapley Additive exPlanations (SHAP): This approach relies on feature
relevance explanation to interpret a particular prediction of supervised ML/DL
models [120]. It computes an additive feature importance score with respect to
a set of required properties (e.h., accuracy, consistency, and missingness).
Hence, SHapley Additive exPlanations (SHAP) determines feature influence by
applying the Shapley values method, which enables estimating the marginal
contribution of one feature over the final reward function. In addition,
combining several predictions can also be considered to build a global
explanation. Several variants of SHAP have been proposed in the literature in
order to optimize its computational complexity, such as DeepSHAP [120] and
TreeSHAP [121].

  * •

Deep Learning Important FeaTures (DeepLIFT) [81]: The purpose of Deep Learning
Important FeaTures (DeepLIFT) is to clarify the output of a neural network by
calculating the significance of each input feature to the output. This is
accomplished by comparing the activation of each neuron in the network for a
particular input to the activation that would have been obtained if a
reference input had been used. The difference in the activations between the
input and the reference is measured by DeepLIFT to compute the contribution of
each input feature to the output. The contribution score obtained can be
utilized to comprehend how the network reached its conclusion and to identify
the most relevant input features. DeepLIFT has been effective in explaining
the behavior of different neural network models, such as convolutional neural
networks and recurrent neural networks, and has been applied to various
fields, including drug discovery, image classification, and speech
recognition.

  * •

Local Interpretable Model-Agnostic Explanations (LIME): It is one of the most
known solutions, that relies on local and simplification explanations, to
explain supervised ML/DL models [85]. LIME is a model-agnostic approach
targeting different types of data, e.g., tabular, text, graphs, and images.
Local Interpretable Model-Agnostic Explanations (LIME) aims to approximate the
learning models by developing locally linear models, which replace the black-
box models to explain their individual predictions.

  * •

Integrated Gradients (IG): also known as Path-Integrated Gradients or
Axiomatic Attribution for Deep Networks. Integrated Gradients (IG) is an XAI
technique that gives an importance value to each feature of the input using
the gradients of the model output [122]. Specifically, it is a local method
that consists of accumulating the gradients by sampling points at a uniform
spacing along a straight line between the input and the baseline. This
procedure avoids getting null gradients when e.g., the deep learning model is
flat in the proximity of the input feature. This method yields the specific
positive or negative attributions of the input features.

  * •

Graph Neural Network (GNN)Explainer: It is a technique that explains the
predictions of Graph Neural Networks (GNNs) for graph-structured data. It
identifies the most important nodes and edges contributing to the output by
generating explanation vectors using an additional neural network. This
generates an attention map that shows the relative importance of each node and
edge. GNN Explainer can be applied to various GNN architectures and input
graphs, without requiring changes to the model or training data. It is useful
for understanding how GNNs make predictions and identifying potential issues
[84].

  * •

Reward Shaping: It entails altering the reward function of the agent to offer
supplementary feedback or incentives. This adjustment assists in steering the
agent’s learning process by molding the reward signal [90].

  * •

Attention Mechanism: It enhances interpretability by identifying and
highlighting the crucial elements in the input that significantly impact the
decision-making process of the agent. They shed light on the specific features
that capture the agent’s attention and influence its decision [92].

  * •

Machine Reasoning (MR): It utilizes logical reasoning and inference techniques
to offer insights into the decision-making process of AI models, thereby
improving transparency and trust. It generates explanations that are easily
comprehensible to humans, fostering a deeper understanding and acceptance of
AI systems. Nevertheless, applying machine reasoning in XAI necessitates
expertise in logic and reasoning, and it may encounter difficulties when
dealing with uncertain or probabilistic information. Nonetheless, the
incorporation of machine reasoning in XAI contributes to the advancement of
interpretable and accountable AI systems [94].

  * •

Attention Flow Analysis: It assesses the individual contribution of attention
heads in the encoder to the overall performance of the transformer’s model.
Specifically, it examines the roles played by these attention heads, with a
particular focus on the most important and confident ones. These heads often
exhibit consistent and linguistically interpretable roles, providing valuable
insights into the model’s decision-making process [96].

  * •

Structural Causal Models (SCM): It is another method that targets
reinforcement learning models, aiming to show the causal link between the data
variables. In [98], the authors leverage Structural Causal Models (SCM) method
to explain the behavior of the reinforcement learning model. They are based on
visual explanations through a Directed Acyclic Graph (DAG), where the nodes
and edges reflect the model states and actions, respectively. By exploring the
DAG, it can be extracted which actions take to move from one state to another.
Once DAG is created, regression models are built to approximate the
relationships using the minimum number of variables. Then, analyzing the DAG’s
variables will help in generating the explanations, in order to answer the
question: "Why action X and not Y ?".

  * •

Caption generation: It is a class of methods that aims to generate text
interpretations to explain the outputs of DL models. In [100], the authors
combined a Convolutional Neural Network (CNN) model and a bi-directional LSTM
encoder/decoder model. The LSTM encoder helps to extract video features, which
are then used by the LSTM decoder to generate textual video captions.

  * •

Knowledge Graphs: To produce human-understandable explanations, it is
necessary to represent ideas in terms of concepts rather than numeric values.
Concepts and the connection between them make what is called _knowledge
graph_. It is a powerful way of representing data because Knowledge Graphs can
be built automatically and can then be explored to reveal new insights about
the domain, especially to find inferred concepts that were not asserted, along
with being able to trace back all the steps, making it fully explainable
[102].

Table IV: Taxonomy of XAI metrics. XAI Method | Basis | Metric | Type of Problem | Reference
---|---|---|---|---
Attributions-based | Features Mutation/Masking | Confidence/Faithfulness | Classification and Regression | [123]
Log-odds | Classification | [124]
Comprehensiveness | Classification | [125]
Sufficiency | Classification | [125]
Raw features | Interpretability | Regression | [126]
|  | Uncertainty | Decision/Action | [127]
Surrogates-based | Perturbation | Robustness/Sensitivity | Regression and Decision/Action | [109]
(in)fidelity | Regression | [128, 129]
LIME Explainer R2 Score | Classification and Regression | [85]
Relative Consistency | Regression | [130]
White-box baseline | Explainer Recall | Classification | [131]
Explainer Precision | Classification | [131]

As anticipated before, the selection of suitable explainability methods
depends both on the complexity of the targeted model to be explained and on
the target audience. Indeed, the type of explanation exposed and their level
of detail depend mainly on the people who are getting such information. In
this context, different user profiles may be targeted by XAI models, and XAI
models’ explanations should differ from one user to another [46]. Table. III
shows the different objectives of XAI explainability, expected by different
user profiles. For instance, users of the models look at trusting as well as
understanding how the model works, while users affected by models’ decisions
aim to understand their decisions and the main reasons for conducting such
decisions. Besides, developers and data scientists expect explanations related
to the AI models’ performance, in order to optimize them over time. However,
both regulatory and manager users aim to get more details related to the
compliance of AI models with the legislation in force to check and assess
them.

###  II-C XAI Metrics

XAI metrics are used to measure the quality of the original AI model and its
explanation. We summarize relevant XAI metrics in Table IV, and compile a list
of them in the following.

  * •

Confidence/Faithfulness: A common approach to measuring the confidence of the
explanation relies on the notion of feature relevance. Specifically, observing
the effect of muting, i.e., replacing a feature with a baseline
value—generally zero—helps to measure the effect on the prediction in both
classification and regression tasks [123]. For instance, for a probabilistic
classification model, we can obscure or remove features according to a policy
defined as follows

$\hat{x}_{i,k}=x_{i,k}\times(1-p),\,\,p\sim\mathcal{B}(1,\pi_{i,k})$ (1)

where $\pi_{i,k}$ is a probability distribution of the features that can be
computed as

$\pi_{i,k}=\frac{\exp\Bigl{\\{}\left|a_{i,k}/x_{i,k}\right|\Bigr{\\}}}{\sum_{l=1}^{N}{\exp\Bigl{\\{}\left|a_{l,k}/x_{l,k}\right|\Bigr{\\}}}},\,i=1,\ldots,N,$
(2)

where $a_{i,k}$ is the attribution of feature $i$ in a sample of class $k$. It
is obtained using any attribution-based XAI method, such as IG or SHAP. The
confidence score in this case is

$c_{k}=\frac{\Delta_{k}^{(c)}}{\Delta_{k}},$ (3)

where $\Delta_{k}^{(c)}$ is the number of samples that conserve their class
$k$ after the mutation of the dataset and $\Delta_{k}$ stands for the original
count of samples with class label $k$. For regression tasks, however, the
classes are replaced with the notion of groups which are defined by comparing
the continuous prediction output with one or several thresholds.

  * •

Log-Odds (LO): Similarly to the confidence, this score is defined as the
average difference of the negative logarithmic probabilities on the predicted
class before and after masking the top $p\%$ features with zero padding [124].
Given the attribution scores generated by an explanation algorithm, we select
the top $p\%$ features based on their attributions and replace them with zero
padding. More concretely, for a dataset with $L$ samples, it is defined as:

$\mathrm{log}\text{-}\mathrm{odds}(p)=-\frac{1}{L}\sum_{i=1}^{L}\log\frac{\Pr\left(\hat{y}|\mathbf{x}_{i}^{(p)}\right)}{\Pr\left(\hat{y}|\mathbf{x}_{i}\right)}$
(4)

where $\hat{y}$ is the predicted class, $\mathbf{x}_{i}$ is the $i$th sample,
and $\mathbf{x}_{i}^{(p)}$ is the modified samples with top $p\%$ features
replaced with zero padding. Lower scores are better.

  * •

Comprehensiveness: is the average difference of the change in predicted class
probability before and after removing the top $p\%$ features. Similar to Log-
odds, this measures the influence of the top-attributed words on the model’s
prediction. It is defined as [125]:

$\mathrm{Comp}(p)=\frac{1}{L}\sum_{i=1}^{L}\left[\Pr\left(\hat{y}|\mathbf{x}_{i}^{(p)}\right)-\Pr\left(\hat{y}|\mathbf{x}_{i}\right)\right]$
(5)

Here $\mathbf{x}_{i}^{(p)}$ denotes the modified dataset with top $p\%$
samples deleted. Higher scores are better.

  * •

Sufficiency: is defined as the average difference of the change in predicted
class probability before and after keeping only the top $p\%$ features. This
measures the adequacy of the top $p\%$ attributions for the model’s
prediction. It is defined in a similar fashion as comprehensiveness, except
the $x_{i}^{(p)}$ is defined as the samples containing only the top $p\%$
features. Lower scores are better [125].

  * •

Robustness/Sensitivity: A crucial property that interpretability methods
should satisfy to generate meaningful explanations is that of robustness with
respect to local perturbations of the input. This is not the case for popular
interpretability methods; even adding minimal white noise to the input
introduces visible changes in the explanations [109]. To formally quantify the
stability of an explanation generation model, one can estimate the Lipschitz
constant $\lambda$ for a given input $x_{i}$ and a neighborhood $B_{\epsilon}$
of size $\epsilon$ as,

$\lambda(x_{i})=\operatorname*{arg\,max}_{{x_{j}\in
B_{\epsilon}(x_{i})}}\frac{\lVert\Phi(x_{i})-\Phi(x_{j})\rVert_{2}}{\lVert
x_{i}-x_{j}\rVert_{2}},$ (6)

where the evaluation of the explaining function $\Phi$ for methods like LIME
and SHAP is expensive as it involves model estimation for each query. In
contrast, gradient-based attribution methods present a lower complexity. On
the other hand, computing (6) for post-hoc explanation frameworks is much more
challenging, since they are not end-to-end differentiable. Thus, one needs to
rely on black-box optimization instead of gradient ascent. This continuous
notion of local stability in (6) might be inadequate for discrete inputs or
settings where adversarial perturbations are overly restrictive. In such
cases, one can instead define a (weaker) sample-based notion of stability. For
any x in a finite sample $X=\\{x_{i}\\}_{i=1}^{n}$ one replace
$B_{\epsilon}(x_{i})$ with an $\epsilon$-neighborhood within $X$, i.e.,

$\mathcal{N}_{\epsilon}(x)=\\{x^{\prime}\in X|\,\,\lVert
x-x^{\prime}\rVert\leq\epsilon\\}.$ (7)

  * •

Uncertainty: The explanation is certainly linking the input features to the
output prediction/decision when high attributions (in absolute value) are more
concentrated in some features compared to a less certain uniform distribution.
Indeed, let $N$ denote the number of features. If we map the attributions to a
probability space (using e.g., Eq. (2)), the resulting entropy,

$\mathcal{H}_{k}=-\sum_{i=1}^{N}\pi_{i,k}\log(\pi_{i,k}),$ (8)

measures the uncertainty of the output (prediction or decision) with respect
to the input (features or states) [127]. On the other hand, when the number of
features is very high, one can characterize the uncertainty by comparing the
distributions of both the attributions and a reference uniform probability
density function. This can be done by invoking the discrete Kullback-Leibler
(KL) divergence. The larger the KL divergence, the higher the certainty yield
by the XAI method.

  * •

Infidelity: In XAI surrogate methods, i.e., the schemes that consist of
approximating the original model with a low-complexity more interpretable
surrogate such as LIME, the fidelity of the surrogate to the original model
can be quantified. Indeed, given a black-box function $f$, explanation
functional $\Phi$, a random variable $\mathbf{I}\in\mathbb{R}^{n}$ with
probability measure $\mu_{\mathbf{I}}$, which represents meaningful
perturbations of interest, the explanation infidelity can be defined as [128]

$\mathcal{I}(\Phi,f,\mathbf{x})=\mathbf{E}_{\mathbf{I}\sim\mu_{\mathbf{I}}}\left[\mathbf{I}^{T}\Phi(f,\mathbf{x})-(f(\mathbf{x})-f(\mathbf{x}-\mathbf{I}))^{2}\right]$
(9)

where $\mathbf{I}$ represents significant perturbations around $\mathbf{x}$
and can be specified in various ways, such as the difference to a baseline
$\mathbf{I}=\mathbf{x}-\mathbf{x}_{0}$.

  * •

Fidelity and Soundness: Two metrics can be applied to evaluate fidelity.
Firstly, [85] used recall ($\mathcal{R}$) as a measure of fidelity for this
method, which is defined as follows,

$\mathcal{R}=\frac{|\mathcal{T}\cap\mathcal{E}|}{|\mathcal{T}|}$ (10)

where the term True Features $\mathcal{T}$ represents the relevant features as
extracted directly from the white box model and Explanation Features
$\mathcal{E}$ represents the features characterized as most relevant by the
explanation [129]. This measure indicates how well the explanation captures
the most relevant features from the predictive model, i.e., as a measure of
the completeness of the explanation. Additionally, to understand how well the
explanation excludes irrelevant features (soundness of the explanation),
precision ($\mathcal{P}$) can be measured,

$\mathcal{P}=\frac{|\mathcal{T}\cap\mathcal{E}|}{|\mathcal{E}|}$ (11)

  * •

R-squared (R2)Score: Behind the workings of LIME lies the assumption that
every complex model is linear on a local scale. LIME tries to fit a simple
model around a single observation that will mimic how the global model behaves
at that locality. The simple model can then be used to explain the predictions
of the more complex model locally. In this respect, R-squared (R2) score is
used to measure the performance of the surrogate local model.

  * •

Relative Consistency: Let $f_{i}$ denote a predictor trained over dataset
$\mathcal{D}_{i}$. Explanations arising from different predictors are said to
be consistent if they are close when the predictions agree with one another,
i.e., given the sets

$\begin{split}\mathcal{S}^{\prime}=\\{\delta_{i,j}(x)|f_{i}(x)=y\cup
f_{j}(x)=y\\}\\\
\mathcal{S}^{\prime\prime}=\\{\delta_{i,j}(x)|f_{i}(x)=y\oplus
f_{j}(x)=y\\},\end{split}$ (12)

where $\delta_{i,j}(x)$ is a similarity measure of the explanations $f_{i}(x)$
and $f_{j}(x)$, and $\gamma$ is a fixed threshold. We aim at making the gap
between the set of consistent explanations $\mathcal{S}^{\prime}$ and
inconsistent ones $\mathcal{S}^{\prime\prime}$ visible. In this respect, we
invoke the true positive rate,

$\mathrm{TPR}(\gamma)=\frac{|\\{\delta\in\mathcal{S}^{\prime}:\delta\leq\gamma\\}|}{|\\{\delta\in\mathcal{S}:\delta\leq\gamma\\}|},$
(13)

where $\mathcal{S}=\mathcal{S}^{\prime}\cup\mathcal{S}^{\prime\prime}$. In
addition, we also consider the true negative rate,

$\mathrm{TNR}(\gamma)=\frac{|\\{\delta\in\mathcal{S}^{\prime\prime}:\delta>\gamma\\}|}{|\\{\delta\in\mathcal{S}:\delta>\gamma\\}|}.$
(14)

The quality of these explanations can be assessed independently of the
accuracy of the predictor via the _Relative Consistency (ReCo)_ metric [130]:

$\mathrm{ReCo}=\max_{\gamma}\mathrm{TPR}(\gamma)+\mathrm{TNR}(\gamma)-1,$ (15)

with a score of $1$ indicating perfect consistency of the predictors’
explanations, and a score of $0$ indicating complete inconsistency.

###  II-D O-RAN Alliance Specifications

The O-RAN Alliance aims to lead the telecom Industry toward designing an
intelligent and open RAN [20][21] leveraging and extending 3rd Generation
Partnership Project (3GPP) reference RAN architecture towards greater
flexibility in network deployment and enable scalability for new services. By
disaggregating software from hardware and establishing open interoperable
interfaces, the O-RAN Alliance aims to foster a more modular and flexible RAN
ecosystem. This approach allows for greater compatibility and
interchangeability among different vendors’ equipment, enabling network
operators to avoid vendor lock-in and embrace a wider range of technology
solutions.

As shown in Fig. 2, the O-RAN Alliance creates $11$ Working Groups (WGs) and
three focus groups, for RAN cloudification, automation, and disaggregation. A
technical steering committee coordinates workgroups, where each one focuses on
a part of the O-RAN architecture. For instance, WG 1 specifies the Service
Management and Orchestration (SMO) part, while WG 2 and WG 3 are specifying
Non RT RIC and Near RT RIC, respectively.

Figure 2: O-RAN Alliance Technical Steering Committee structure.

The new O-RAN architecture leverages NFV and SDN technologies to define new
open interfaces and disaggregate the RAN functional blocks, to allow the
deployment of new services and applications. O-RAN divides the Baseband Unit
(BBU) of RAN into three functional blocks, Central Unit (CU), Distributed Unit
(DU), and Radio Unit (RU). To support control user plane separation, the CU
block is also divided into control plane CU-Control Plane (CP) and user plane
CU-User Plane (UP) sub-blocks. Fig. 3 shows the reference architecture of the
O-RAN alliance. The radio frequency signals are received, transmitted,
amplified, and digitized at RU, which is located near the antenna, while CU
and DU represent the base station’s computation parts and are in charge of
transmitting the digitalized radio signal to the network. Noting that the DU
block may be deployed near or at the RU block, while the CU block may be
deployed near the core network part. It is also worth noting that 3GPP has
defined different RAN deployment scenarios and functional split options, which
are described in [45] [132].

The two main components introduced by the O-RAN architecture are summarized
below:

  * •

Non Real-Time RAN Intelligent Controller (Non RT RIC): it supports non-RT
functions (i.e., with a time granularity greater than 1s) such as policy-based
guidance. The Non RT RIC is located at the SMO and comprises two sub-
functions: Non RT RIC Applicaitons (rApps) and Non RT RIC framework. The
latter is in charge of providing all required services to rApps via the R1
interface, whether from Non RT RIC framework or SMO, while rApps leverage the
functionality provided by the Non RT RIC framework, such as data monitoring
via O1 interface (stored in a database), to perform intelligent RAN
optimization functions at non-RT scale. Such functionality enables rApps to
get information and trigger actions, e.g. re-configuration and policies.
Hence, Non RT RIC enables exposing an intelligent RAN policy to Near RT RIC,
through A1 interface, based mainly on data analytics and ML/DL inference.

  * •

Near Real-Time RAN Intelligent Controller (Near RT RIC): it is in charge of
controlling and optimizing the O-RAN nodes (CU and DU) and their resources
through fine-grained data monitoring and actions over E2 interface, at a near
RT scale (i.e., from $10$ms to $100$ms). It hosts several Near RT RIC
Applicaitons (xApps), which may collect near RT information (e.g., at a User
Equipment (UE) or Cell basis) through E2 interface, and provide value-added
services, with respect to the Non RT RIC’s policies received via the A1
interface. xApps include Mobility Management (MM), Resource Management (RM),
Spectrum Management (SM), etc.

Figure 3:  O-RAN Architecture.

##  III XAI Deployment on O-RAN

In this section, we describe how XAI methods can be deployed in the O-RAN
framework and architecture by means of three realistic reference scenarios.

###  III-A Introduction and Motivation

As described in Sec. II-D, the basic idea of O-RAN is not only to disaggregate
RAN functions exploiting the flexibility brought by virtualization techniques,
but also to design RICs that locally host specific RAN applications (e.g.,
rApps and xApps), addressing several and heterogeneous control tasks such as
handover management, energy management, fault detection, and radio resource
allocation. The O-RAN framework has been devised to natively support a heavy
usage of machine/deep learning (ML/DL) techniques to enhance the development
and operations of intelligent RAN applications to pave the road for future B5G
network services. For instance, as shown in [29], enabling cooperation among
several xApps can help to optimize network performance, both in terms of data
throughput and packet delivery ratio. However, one of the main challenges of
AI-based O-RAN management is the lack of transparency on the decision-making
processes that govern AI algorithms, which makes it difficult for network
operators and engineers to diagnose problems, and further optimize the network
behavior.

Therefore, there is a pressing need to integrate XAI into the O-RAN management
operations, as to gain more detailed information about the decision-making
processes of ML and DL algorithms. Specifically, XAI techniques should be
incorporated into the running AI-based rApps/xApps to provide transparent
explanations of their outputs. This would not only improve the accuracy and
transparency of the decisions made by these systems but also increase the
trust of network operators and engineers in the performance of the network.

###  III-B Local Interpretable AI Deployment

The availability of open interfaces and the distributed nature of RAN
deployments allows for the design and implementation of advanced federated and
distributed schemes that aim to can overcome traditional RAN management
scalability issues.

As mentioned before, in the O-RAN context, rApps and xApps deployed at the
Near RT RIC and Non RT RIC, respectively, oversee the RAN management
operations supporting heterogeneous network services. To achieve this,
supervised learning has been widely leveraged, either in a centralized way or
distributed (federated) way [133][134]. Indeed, Federated Learning (FL) aims
to generate learning models in a collaborative way while preserving the
privacy of involved learners [135][136][137]. FL is a promising technique also
for the O-RAN context, especially for rApps/xApps that belong to different
network operators (or vendors), or that are spatially distributed, in order to
jointly preserve data privacy and confidentiality, and gain knowledge from
heterogeneous local scenarios as to collect a global view.
Fig. 4 depicts an example scenario that considers the possibility to train and
deploy both AI and XAI models in a federated way.

Figure 4: Deployment of Federated AI and XAI in O-RAN.

To avoid communication overhead and incurring significant operational costs,
as well as to accommodate specific latency requirements, it is always
beneficial to process data locally, where they are made available from
dedicated monitoring functions. Therefore, raw control plane information
generated by end-users at a given cell (or multiple cells) can be processed
locally, in the Near RT RIC, and used to train AI/ML-based models and their
corresponding local XAI models (Step 1). In order to gain from the distributed
nature of RAN deployments, such local information can be transferred to the
Non RT RIC, exploiting the O1 interface, (Step 2). By combining multiple local
models trained over a particular portion of the input space, the Non RT RIC
aims to derive more generalized and advanced models to the Near RT RIC and the
corresponding xApp. This information can be provided as feedback (Step 3) via
the A1 interface. Hence, leveraging collected data from distributed nodes via
the O1 interfaces, predictions along with their corresponding explanations can
be performed in real-time. Favored by a continuous learning process, both AI
and XAI’s outputs should be considered to perform management decisions and
improve network performance. For instance, such outputs can help to update
users’ scheduling policy or that radio resource assignment. To do so, such
decisions are communicated from the Near RT RIC to the corresponding O-RAN
modules, i.e., Open RAN Central Unit (O-CU) or Open RAN Distributed Unit
(O-DU), via the E2 interface (Step 4). In this context, different XAI
techniques can be leveraged. For instance, RuleFit is one of the most used XAI
techniques [88][89]. Its fundamental idea is to capture interactions between
the original dataset features in order to create new features in the form of
decision rules. Then, RuleFit learns a new transparent learning model using
the original features, and also a number of new features that are decision
rules.

Figure 5: Deployment of explanation-guided deep reinforcement learning in
O-RAN. Adapted from [127].

Furthermore, the XAI explainability (outputs) may target different user
profiles (Step 5). For instance, users of the models may want to trust and
understand how the model works, while explanations related to the AI models’
performance are sent to developers and data scientists to optimize their
behavior over time. In addition, more details about AI models’ compliance with
the legislation in force should be communicated to both regulatory and manager
users in order to check and assess them.

###  III-C Explanation-Guided Deep Reinforcement Learning Deployment

Undoubtedly, Reinforcement Learning (RL) will play a significant role in
enabling smart and automated O-RAN operations. In the context of explanation-
guided learning [138, 139], explanation-guided deep reinforcement learning is
a branch of artificial intelligence that combines deep learning and
reinforcement learning with human-interpretable explanations. The goal of this
approach is to enable humans to better understand the decision-making process
of RL agents. In this method, the RL agent learns from its environment while
considering human knowledgeable inputs providing explanations for the agent’s
behavior. The explanations can be in the form of natural language,
visualizations, or other means. By providing these contextual and external
information, a human expert in the field of application can guide the agent
toward better decision-making and improve its overall performance.

As shown in Fig. 5, a Deep Reinforcement Learning (DRL) agent at the Near RT
RIC performs resource allocation under latency constraints and interacts with
the O-CU environment through the E2 interface. The agent temporarily stores
its experiences and observations in a replay buffer, which is continually
updated. Then, the XAI xApp co-located with the Near RT RIC derives the SHAP
importance values from a batch state-action dataset extracted from the buffer.
To quantify the uncertainty of an action given a specific input state, the
obtained SHAP values are afterward converted to a probability distribution via
_softmax_ and used to calculate the entropy that measures the uncertainty, as
formulated in Eq. (8). The multiplicative inverse of the maximum entropy value
is used as an XAI reward (to minimize uncertainty). Combining the SLA reward
(e.g., the multiplicative inverse of the latency) with the XAI reward results
in a composite reward that reduces the uncertainty of state-action pairs and
guides the agent to select the best and most explainable actions for specific
network state values.

Figure 6: Deployment of explanation-aided confident federated learning in
O-RAN.

###  III-D Explanation-Aided Confident Federated Learning Deployment

Explanation-aided confident FL is a type of machine learning that combines
federated learning with human-interpretable explanations. In FL, data is
collected and processed locally on individual devices, and only the necessary
information is shared with a central server for model training [140][141]. The
goal of explanation-aided confident FL is to enable individuals and
organizations to collaborate on training models while maintaining privacy and
security. To achieve a confident FL-based resource allocation/prediction, the
local learning is performed iteratively with a run-time explanation as
detailed in [142]. The overall working principle of the scheme is manifested
in Fig. 6. For each local epoch, the dataset collected through the E2
interface is used to train a local resource allocation model via constrained
optimization, which yields the features and the corresponding predictions to
the XAI xApp where an _explainer_ generates the features attributions using
one of the feature attribution XAI methods (e.g., SHAP, Integrated Gradient,
etc.). The _confidence mapper_ then converts these attributions to a soft
probability distribution and translates it afterward into a confidence metric
according to Eq. (3), and feeds it back to the optimizer to include it as an
additional constraint in the local optimization. Moreover, the confidence
metric is sent via the NG-c interface to the peer O-CUs. In this respect, each
O-CU uses the gathered set of confidence scores to assess its priority, where
only the $K$ O-CUs with the largest confidence scores out of the available $N$
O-CUs take part in the FL training to guarantee better confidence. Upon the
termination of the local optimization, the model weights are reported to the
federation layer—located at the Non RT RIC—to perform model aggregation and
broadcast it via the A1-P interface.

Table V: Mapping of Existing AI-based O-RAN works to XAI-enabled Solutions. |  |  |  XAI Deployment at RIC as xApps
---|---|---|---
Works |  Addressed RAN Function | AI Technique |  | XAI
---
Technique
Metrics |  | O-RAN
---
Module
| Functional
---
Blocks
Interfaces
| Yang et al.
---
[143]
[144]
User access control |  | Federated deep
---
reinforcement
learning
| Confidence | O-CU-CP |  |  UE and gNB
---
procedure
management
| Hoejoo et al.
---
[145]
Total cell throughput |  | Deep
---
reinforcement
learning
| State-action certainty | O-DU |  | Resource
---
assignment
(NR-MAC)
| Shahram et al.
---
[146]
| Resource allocation
---
and function placement
| Actor-Critic
---
learning
| State-action certainty | O-DU |  | Resource
---
assignment
(NR-MAC)
E2 (Realization). A1 (Data Analytics), O1 (Monitoring),
| Rivera et al.
---
[147]
[148]
Han et al.
[149]
| Resource allocation
---
and power control
| Federated deep
---
reinforcement
learning
Reactive/Proactive Explanations | Confidence, Log-odds | O-DU |  | Resource
---
assignment
(NR-MAC) and
PDSCH
(High-PHY)
| Han et al.
---
[150]
Resource allocation | Q-learning |  | State-action certainty | O-DU |  | Resource
---
assignment
(NR-MAC)
| Farhad et al.
---
[151]
Resource allocation |  | Distributed deep
---
reinforcement
learning
| Robustness | O-DU |  | Resource
---
assignment
(NR-MAC)
| Wang et al.
---
[152]
Resource allocation |  | Deep
---
reinforcement
learning
| State-action certainty |  | O-DU
---
O-RU
| Resource
---
assignment
(NR-MAC)
| Abouaomar et al.
---
[153]
Resource allocation |  | Federated Deep
---
reinforcement
learning
| Confidence, Log-odds |  | O-DU
---
| Resource
---
assignment
(NR-MAC)

##  IV XAI for Literature Solutions Targeting O-RAN

In this section, we first give a literature review of existing works, which
leverage AI (ML/DL) techniques on top of the O-RAN architecture, in order to
optimize RAN functions. We then discuss how these works can be mapped to XAI
methods.

###  IV-A Existing AI-driven O-RAN Works

  * •

User Access Control: The user access control or user association challenge is
addressed in [143][144], in order to ensure load balancing among Base Stations
(BSs) and avoid frequent handovers. The authors designed a federated deep
reinforcement learning. The UEs collaboratively trained their local models and
then aggregated them at the RIC level. The designed model succeeded to
maximize the overall UEs’ throughput and reducing frequent handovers.

  * •

Total Cell Throughput: An online training environment of a reinforcement
learning model is deployed at the RIC level in [145]. The developed model
controlled function parameters in DU, in order to maximize total cell
throughput. Thanks to deployed learning model, the total cell throughput
increased by $19.4\%$.

  * •

Function Placement: The O-RAN architecture leverages virtualization and
disaggregation of RAN functionalities among three key units (RU, DU, and CU).
The authors of [146] studied the placement of resource allocation function
based on service requirements, by dynamically selecting CU-DU units. Thus,
they generated two reinforcement learning models based on Actor-Critic. The
first one is used to assign resource blocks to UEs according to traffic types,
delay budget, and UEs priorities, while the second one is leveraged to
optimize function placement and hence the decisions of resource allocation.
The authors showed that through this dynamic placement, both latency and
throughput are highly improved.

  * •

Resource allocation: In [147] [148] [149], the authors studied the multi-agent
team learning deployment on top of the O-RAN architecture by deciding about
each agent placement and the required AI feedback. As a case study, the
authors addressed the challenge of how to coordinate several running and
independent xApps in O-RAN. They designed two xApps, called resource
allocation xApp and power control xApp, and then used federated deep
reinforcement learning to enhance learning efficiency as well as network
performance in terms of throughput and latency. Similarly, in [150], the
authors aimed to deal with the conflicts that may occur between running xApps
when these xApps are deployed by different vendors. Leveraging Q-learning,
they proposed a team learning algorithm for resource allocation, to increase
cooperation between xApps and hence optimize the performance of the network.
Another distributed RL model was generated in [151], in order to manage RAN
slice resource orchestration on top of the O-RAN architecture. The distributed
architecture of distributed RL is composed of multiple intelligent agents, one
for each network slice, that performed local radio allocation decisions.
Similarly, in [153], the authors leveraged federated distributed RL to manage
the radio resource allocation among multiple Mobile Virtual Network Operators
(MVNOs) for two different network slices (Ultra Reliable Low Latency
Communication (URLLC) and enhanced Mobile Broadband (eMBB)). In [152], the
challenge of how to optimally assign DU resources for various RUs is studied.
A deep reinforcement learning model is built to achieve efficient management
of RUs-DU resources. Experimental results showed that the proposed scheme
improves highly resource usage efficiency.

###  IV-B How XAI can Help

Integrating ML/DL-based algorithms with RAN functionalities has been found to
address many challenging tasks: power control, user access control, handover,
resources management, etc., which accordingly helps to optimize the
performance of the RAN part. This was highly motivated in O-RAN, especially
with the introduction of RIC modules. Indeed, the RAN functions are usually
formulated as Markov Decision Process (MDP) [154], which explains the wide
application of reinforcement learning, e.g. Q-learning, deep Deep Q-Network
(DQN), and Actor-Critic, either in a centralized or a federated way, in order
to derive the optimal policy about the corresponding RAN function. In
addition, team learning is also an emerging paradigm to optimize the
coordination and control of the running xApps at the O-RAN’s RICs.
It is worth noting that resource management is the most studied RAN function
using feature engineering approaches, such as feature extraction and feature
selection, in addition to reinforcement learning algorithms (DQN and
Q-learning). With this strategy of RAN functions analysis, it is possible to
determine the contribution of every feature, e.g., higher-order cumulants,
related to the RAN performances. This helps to adjust the features’ values in
order to optimize the ML/DL predictions. In fact, humans/users understanding
of Q-learning models limits to small scenarios involving a small number of
states and actions. However, these models may become complex, especially with
a high number of features, states, and actions, making them less interpretable
by humans. The challenge here is the accuracy-interpretability trade-off,
which means that the greater the accuracy, the less likely the model is
interpretable and vice versa. For example, the Q-learning model can improve
the overall performance of radio resource management by exploiting more
descriptive frequency and time features, but its complexity increases when
considering more features including network density and network/users power,
service requirements, and the trust and security of the wireless
communications, and thus this will introduce more states and actions in the
system. Besides, despite being a more advanced algorithm as compared to
Q-learning, DQN gives black-box models that output a lack of explainability.
For instance, radio resource allocation using a DQN can introduce many
ambiguous points, which should be explained, such as which layers/neurons in
the DQN architecture can help to improve the accuracy, and why some UEs get
the same number of radio block than others, even with different service
requirements (URLLC, eMBB, massive Machine Type Communications (mMTC)). In
this context, XAI is highly recommended since it provides profound insights
into automated decisions and predictions. These details can help different
users, as well as the network operators, to deal with unexpected problems,
either related to the ML/DL models or to the corresponding xApps of the
O-RAN’s RICs. Therefore, the performance of the different RAN functions can be
highly enhanced.

###  IV-C Mapping to XAI-enabled Works

In Table V, we compare the existing AI-driven O-RAN works according to several
criteria, including the addressed RAN function and the leveraged AI
techniques. In addition, we illustrate how XAI can be deployed, as xApps, on
top of these works, in order to explain their AI-based decisions. We clearly
observe that most of the existing works are based on reinforcement learning to
manage RAN functions, especially power and resource allocation, user access
control, function placement, and total cell throughput. According to [155],
two main groups of XAI techniques can be applied to reinforcement learning
strategies, in order to give both local and global explanations.

  * •

Reactive XAI techniques imply that explanations are given in an immediate time
horizon. It includes three main approaches: $i$) Policy simplification, which
aims to simplify the policy of transition function into a form that is
interpretable/understandable by itself, by using for instance decision trees
and fuzzy rule-sets [156]. $ii$) Reward decomposition into understandable
components, which aims to better understand the reasons for certain reward
values [157]. $iii$) Feature contribution and visual methods, to determine
features’ contribution to the decision-making process, and then generate
explanations based on that contribution. Examples of such techniques are both
LIME [85] and SHAP [120].

  * •

Proactive XAI techniques focus on a longer time horizon to provide the
required explanations. These techniques can be classified into four main
classes: $i$) Structural causal model, that aims to learn the relationships
between variables/features. This technique generates human explanations since
it considers the world as a causal lens [98][158]. $ii$) Explanation in terms
of consequences, which enables agents in reinforcement learning to answer a
question about their policy in terms of consequences [159]. In other words, it
enables to determine what each agent can get from a state, and which outputs
it expects from a visited state or corresponding action. $iii$) Hierarchical
policy by decomposing a given task into multiple sub-tasks, located at
different abstraction levels [160]. Then, a prerequisite for executing a
subsequent task is given as an interpretation of a particular action. $iv$)
Relational reinforcement learning, which is based on a set of rules to provide
background knowledge to the decision agent [161]. In this way, actions,
states, and policies are represented by relation language, which helps to
understand the reinforcement learning model’s outputs. Moreover, the deployed
XAI techniques can be evaluated according to several metrics, as shown in
Table V. Specifically, in DRL-based resource management use-cases, the state-
action mapping _certainty_ can be measured via the entropy, which is computed
from the attributions of input state features. Moreover, the _confidence and
Log-odds_ metrics serve to quantify the trust in AI predictions/decisions by
using the input XAI attributions as a basis to mask the impactful features in
either offline or DRL-based on-the-fly datasets, and measure the corresponding
deviation of the output, which is fed back to the optimizer/agent for
accountability.

It is noteworthy that different data types can be monitored at the Non RT RIC
from Open RAN Radio Unit (O-RU) modules, via the O1 interfaces, in order to
build both AI and XAI models. These models are then deployed at the Near RT
RIC as xApps through the A1 interface. In addition, the made decisions with
respect to both AI and XAI outputs are sent and executed at different levels
(O-DU and O-CU), via the E2 interface.

Figure 7: Use case: User-Centric QoE Optimization. Adapted from [162]

##  V XAI for O-RAN Use-cases

In the following, we collect a list of use-cases in the context of O-RAN and
network slicing, highlighting how they would benefit from the introduction of
XAI methods.

###  V-A Quality of Experience (QoE) Optimization

Modern applications in the Fifth Generation (5G) ecosystem demand large
bandwidth and low-latency communication to provide an adequate level of QoE,
which can hardly be achieved by current semi-static Quality of Service (QoS)
frameworks devoted to concurrently supporting heterogeneous broadband
applications like in the Fourth Generation (4G) era. Radio fluctuations impair
radio transmission capabilities, especially when adopting higher carrier
frequencies like millimeter waves, leading to variable application
requirements even within the same communication session. In order to improve
QoE, estimation and prediction tasks performed at the application level can
help in dealing with such a dynamic environment, favoring both user experience
and efficient use of RAN resources [162].

The open interfaces introduced by the O-RAN architecture significantly ease
the per-user flow modification and configuration by means of proactive closed-
loop network optimization. Fig. 7 depicts a possible deployment addressing
this use-case. It involves Non RT RIC, Near RT RIC, E2 Nodes, and external
applications running on the UE. The open interface allows external
applications to interface with the O-RAN domain, which, empowered by ad-hoc
optimization logic, would be capable of dynamically re-configuring the
networking settings in response to real-time triggering events. In this case,
XAI methods would help in understanding the reasons for specific
reconfiguration decisions both at Non RT RIC and Near RT RIC.

###  V-B Traffic Steering

Imbalances in the traffic load across cells of different access technologies
may lead to performance degradation. 5G systems aim to seamlessly support
different combinations of access technologies, e.g., LTE, 5GNR, Wi-Fi, etc.,
with the 5G Core development converging towards a unified domain able to
support a wide variety of applications and requirements, combining different
Radio Access Technologies (RAT) and enhancing the mobile network offloading
capabilities [162]. In this context, O-RAN A1 interface would allow enforcing
desired optimization policies and utilizing the appropriate performance
criteria to proactively manage user traffic across different access
technologies. The Non RT RIC monitors the user experience by UE level
performance measurements on a cell level and may decide to relocate one or
more users to other cells based on global optimization objectives, e.g.,
fairness in bandwidth sharing, QoE maximization, load-balancing. In all these
scenarios, XAI methods can be applied to guide the definition of such
optimization criteria.

Figure 8: Use case: Explainable slice SLA assurance. Adapted from [163]

###  V-C RAN Slice Service Level Agreement (SLA)Assurance

The 5G infrastructure has been designed to cope with highly diverse
performance requirements coming from heterogeneous services and vertical
applications. In this context, network slicing arises as a key technology to
efficiently support tailored end-to-end connectivity satisfying specific
business requirements. In general, the business parties and the infrastructure
provider define the set of networking capabilities required to successfully
run the service in a SLA, e.g., in terms of data rate, latency, and resource
availability [164]. Perhaps not surprisingly, this introduced the need for ad-
hoc mechanisms able to efficiently measure and expose such information to
$3^{rd}$ party entities traditionally alien to the telecommunication market.
In this context, O-RAN’s open interfaces and AI/ML-based architecture will
enable such mechanisms, enabling the operators to take full advantage of the
business opportunities brought by the network slicing concept.

More in detail, specific slice configuration settings derived from the SLA
requirements can be easily enforced by initiating the procedure from the SMO
layer, and finely adjusted over time thanks to measurements feedback and zero-
touch XAI-based mechanisms applied at the different layers of the
architecture, especially at the Non RT RIC and Near RT RIC. Fig. 8 summarizes
the main workflow to achieve the solution.

Figure 9: Use case: Multi-vendor deployment with explainable coordination.
Adapted from [162].

###  V-D Multi-vendor Slices

The coexistence of different network functions provided by different vendors
to instantiate operators’ services is one of the key enablers for flexible and
efficient use of radio resources and CAPital EXpenditures (CAPEX)/OPerational
EXpenditures (OPEX) optimization. To this extent, the O-RAN architecture
enables the deployment of multiple slices comprising functions provided by
different vendors offering a variety of virtual Open RAN Distributed Unit (vO-
DU) and virtual Open RAN Central Unit (vO-CU) options, specifically optimized
to meet the requirements of a certain service. This brings several advantages
such as a more flexible and time-to-market slice deployment, where operators
can select from the available options the most suitable vO-DU and vO-CU to
deploy their services and huge business opportunities for the vendors.

(a) Resource allocation over Non RT RIC.

(b) Resource allocation over Near RT RIC.

Figure 10: Use case: Explainable RAN slice resource allocation decisions taken
at Non-Real Time and Near-Real Time RICs.

To deploy multi-vendor slices, vO-DUs and vO-CUs must coordinate to coexist
and share the radio environment efficiently and avoid conflicts among the
deployed services [163]. Fig. 9 depicts three possible ways of coordination:
$i$) loose coordination where there is no direct coordination between deployed
services, and the radio resource is fully controlled by the RICs through the
O1, A1, and E1 interfaces. $ii$) moderate coordination where different network
functions are allowed to communicate with each other through the X2 and the F1
interfaces to negotiate radio resources without directly involving the RICs.
In this case, the negotiation must cope with the time frame allowed by the X2
interface communication exchange, which is in the order of seconds; $iii$) the
WG 1 and WG 4 of O-RAN alliance envision also a so-called tight coordination
allowing faster radio resource negotiation among slices, which would require a
new interface, dubbed as New IF in Fig. 9, for direct communication between
vO-DUs. In this context, distributed AI/ML models are particularly suitable to
smartly perform the negotiation task. Indeed, an XAI-enabled component can be
deployed to take control of the coordination and negotiation of resources
between different vendors. We report in the figure an example of a deployment
of this component, suitable for both the moderate and the tight coordination
case.

###  V-E Resource Allocation Optimization

The need to concurrently support multiple heterogeneous slices characterized
by service-tailored networking requirements exacerbates the setup of efficient
and dynamic resource allocation mechanisms able to cope with highly different
spatio-temporal traffic distributions.

For example, end-user mobility towards public events causes spatially
localized peaks of traffic in eMBB kind of slices, or IoT smart sensors
sporadically generating data volumes from highly distributed deployments in
mMTC settings. Compared to traditional RAN deployments characterized by
monolithic architecture and private management interfaces, the O-RAN’s
paradigm would allow for easier and more flexible control of the radio
resources. In addition, the possibility to devise a data-driven ML-based
optimization algorithm would help to automatize the process, exploiting the
closed-loop management framework and previous historical information to
perform the best allocation decisions. Additionally, AI/ML model can be used
to perform proactive management of the radio resources, predicting recurrent
traffic demand patterns of 5G networks in different time epochs and spatial
locations, and for each network slice, therefore anticipating the slice’s
networking needs favoring a better end-users QoE, and limiting the overall
energy consumption. All these methods are traditionally based on RL algorithms
and agents interacting with the environment and learning by trial and error.
More advanced solutions adopt federated learning techniques to improve the
performances of the agents, gaining global knowledge of the system from the
collection of multiple locally-trained models [151]. Such enriched information
is then sent back to the single agents, improving their training, and speed,
and allowing more general management policy definitions.
In both these scenarios, XAI methods can further extend the potential of the
RL management solutions. On the one side, they will allow to better control
the learning procedure, and guide the agent towards the definition of a safe
decision process by adding confidence and trust in the devised management
policies. On the other side, they may help in limiting the information
exchange required by the federated learning approach [165][166]. In fact, only
being able to map the context-decision space uniquely, would allow sharing to
the federation layer only local models actually carrying insightful
information while filtering out erroneous or redundant items. Fig. 10 depicts
two possible deployments options, one assuming the main optimization and
computing effort running within theNon RT RIC entity, and the other
envisioning such task running within the Near RT RIC. The final deployment
choice would depend on multiple factors, including the type of use-case and
machine-learning model to be run and its timescale and complexity, also
considering the different computing capabilities of the RICs.

###  V-F User Access Control

The O-RAN vision aims at evolving the current RAN architecture providing open
interfaces and ML-based optimization to attract new business entities and ease
overall management and reduce operational costs. Current RAN deployments are
composed of thousands of nodes [167]. In such a complicated deployment, it is
expected from the network assigns each UE to a serving BS, maximizing the
overall throughput and the single end-user QoE. This problem is also known as
_user access control_. Traditional user access control schemes imply that user
associations are based on networking metrics such as the Received Signal
Strenght (RSS), which guides the UE towards the base station providing the
best channel. Handover ping-pong effect and load balancing have been
identified as two main issues brought by RSS-based schemes [168].

Figure 11: XAI-driven Automated Continuous Integration and Delivery Pipeline.

##  VI Automation of AI/XAI Pipeline for O-RAN

XAI aims to create equivalent learning models to approximate ML/DL decision
process, hence replacing closed approaches from black-box models into human-
understandable contents [46][37]. In addition, as discussed in Subsec. III-A,
XAI models will be built and deployed at the same level as the ML/DL models on
top of the O-RAN architecture, in order to interpret their decisions and
target different user profiles. In such context, the critical challenge is how
to train/deploy both AI and XAI models, while providing stable life-cycle
performances. In fact, data profiles evolving may cause performance
degradation of the AI/XAI learning models [169]. Thus, both models’
performance degradation and new data profiles should be considered and
studied, to ensure good performance of the intelligent RAN functions over
time.
Hence, it is required to not only perform continuous monitoring of both model
and data profiles, but also automate the whole AI/XAI learning models
development, including data collection/extraction, model training, validation,
and deployment [170].
DevOps paradigm includes a set of practices that combines software development
(Dev) and IT operations (Ops). DevOps aims not only to reduce the systems’
development life cycle but also to provide continuous software delivery with
high quality, by leveraging paradigms and concepts like Continuous Integration
and Delivery (CI/CD). When dealing with machine learning operations, and
automation of the learning process, the paradigm can also be called ML system
operations (MLOps) [170].
It is worth noting that O-RAN specification [171] introduces three control
loops that facilitate the deployment of AI/ML (Artificial Intelligence/Machine
Learning) functionalities within the O-RAN framework. These control loops are
designed to operate at different time scales, enabling efficient integration
and utilization of AI/ML capabilities in the network. Loop $1$ is deployed at
O-DU level to deal with per Transmission Time Interval (TTI) scheduling, and
operates at a timescale of the TTI or above. Loop $2$ deployed at the Near RT
RIC to operate within the range of $10-500$ msec and above. Loop $3$ at the
Non RT RIC at greater than $500$ msec (ML/DL training, orchestration, etc.).
In what follows, we focus more on both loops $3$ and $2$ for XAI models
training, inference, and performance monitoring. Indeed, three main levels of
automation have been categorized [172]: Manual (no MLOps), training pipeline
automation, and CI/CD pipeline automation. A typical architecture integrating
XAI with MLOps pipeline is introduced in [127].

###  VI-A Manual Pipeline

It corresponds to the basic level of maturity, where all the ML steps,
including data collection and preparation, model training, and validation, are
performed manually (cf. Fig. 11). Hence, it is called no MLOps. At this level,
data scientists usually use a rapid application development tool to build
learning models, such as Jupyter Notebooks. In this case, the different steps
of ML are released at the Non RT RIC module (ML), while the trained models are
deployed at the Near RT RIC through the A1 interface, in order to provide
prediction services (Ops). Note that the transitions from one step to another
are also performed manually, and driven by a source code, developed
interactively, till an executable model is created.
In practice, this pipeline corresponds to the learning models which are rarely
updated and often break when they are deployed (models) in the real world. In
addition, the performance of learning models at the RAN environment may
degrade, due mainly either to the dynamic evolving of data profiles describing
the environment or to the very dynamic changes that may occur in the radio
access environment. Hence, automating the whole learning process becomes
primordial.

###  VI-B Training Pipeline Automation

This level introduces a continuous training of the models and thus consists to
perform the model training steps automatically. In particular, when new data
profiles are monitored, the model retraining process is triggered. This
process also includes data and model validation phases to achieve continuous
delivery of the learning models. This level introduces two new components,
named feature store as a centralized repository to store features and enable
access to new features for training serving, and machine learning metadata to
store information about the execution of ML pipeline (cf. Fig. 11).

###  VI-C Continuous Integration and Delivery Pipeline Automation

At this level, a complete CI/CD system is introduced, to enable reliable and
fast learning model deployments in production. Thus, this level achieves the
highest level of automation in ML Ops, by enabling data scientists and
developers to efficiently explore new ideas about feature engineering, model
hyperparameters, and architecture. The main difference with the last level is
that this level enables to build, validate, and deploy automatically the data,
learning models, and model training pipeline components.
Fig. 11 shows the automation of the ML pipeline using CI/CD in O-RAN context,
which mainly features automated both ML pipelines and CI/CD routines.

##  VII O-RAN Security Aspects and XAI

Due to the central role of the 5G network in providing communication to
backbone society infrastructures, security, and security risk awareness play a
key role in network deployment. The O-RAN Alliance has recently created a
Security Working Group (WG 11). This group has identified a list of
stakeholders responsible for ensuring the security of the RAN. This goes
beyond the parties involved in traditional 4G and 5G networks, such as
vendors, operators, and system integrators. In fact, operators will play a
greater role in securing the infrastructure, given the platform’s openness and
the use of multi-vendor components, which allows them to customize and secure
the infrastructure. This also enables them to evaluate and verify the security
of the open components that are introduced in the network, which may not be
possible in fully vendor-driven closed architectures. In addition, according
to [173], network functions and virtualization platform vendors, as well as
third-party xApp and rApp developers, Open Cloud (O-Cloud) providers, and
administrator profiles, that manage virtualized and disaggregated components,
are all new stakeholders. Furthermore, the orchestrator, which manages the
SMO, also has a responsibility to ensure that the network’s operations are
secure. However, due to the plethora of heterogeneous components forming the
O-RAN ecosystem, and the high exploitation of AI-driven network management and
third parties components running services, securing the O-RAN infrastructure
is still a challenge. In this regard, XAI can strongly enhance security in
O-RAN deployments by providing insights, explanations, and transparency into
the decision-making process of AI models. Moreover, XAI helps with threat
detection, model transparency, accountability, analysis of training data, and
human-in-the-loop security, leading to improved threat detection, increased
trust, and compliance with security regulations. However, it could also be the
target of cyber attacks that could vanish its benefits.

###  VII-A Distributed Architecture

The open architecture defined in the O-RAN specifications has been identified
as a possible security issue due to its distributed nature, which expands the
attack surface to malicious entities. The WG 11 has recently identified the
possible vulnerabilities coming from the openness of the platform and
classified them into different threat categories [174]. Such categories
include threats against the O-RAN system which are related to new
architectural elements and interfaces that can be compromised through
different attacks. These attacks could compromise the availability, data,
infrastructure integrity, and data confidentiality of the infrastructure.
Threats against the O-Cloud, which encompasses virtualized environments. These
attacks could compromise virtual network functions, misuse containers or
virtual machines, or spoof underlying networking or auxiliary services.
Threats against open-source code, which could contain backdoors intentionally
compromised by trusted developers or upstream libraries. Physical threats that
are against wireless functionalities. Threats against the protocol stack and
threats against AI/ML. The latter include poisoning attacks that exploit
unregulated access to the data stored in the O-RAN system to inject altered
and misleading data. It is worth pointing out that AI/ML models could
themselves be deployed by malicious entities [175, 176]. In order to
counteract such threads, security principles have been defined in [174]. Such
principles are intended to be developed into security requirements,
recommendations, and potential countermeasures, which will define what is
required, recommended, or optional. They include mutual authentication (i.e.,
the _zero-trust_ paradigm), access control, secure cryptography, trusted
communication, secure storage, secure boot and self-configuration, secure
update, recoverability and backup, and security management of risks in open
source components. At the time of writing, there is no explicit mention of the
exploitation of XAI in enhancing security in O-RAN. Nonetheless, XAI can
provide transparency and insights into the decision-making process of AI
models, helping stakeholders better understand how these security principles
are being applied in practice. This can lead to improved validation,
compliance, and trust in the security measures implemented in O-RAN
deployments, ultimately enhancing the overall security posture of the network.

Figure 12: O-RAN architecture with additional XAI-based security components.
Adapted from [175].

###  VII-B Risk Assessments

The WG 11 has assessed security analysis for the Non RT RIC, the O-Cloud, and
the Near RT RIC frameworks [177, 178, 179]. Such assessments addressed the
likelihood of attack and impacting protection goals that can be summarized
with $i$) Confidentiality, which refers to the ability of unauthorized
entities to access sensitive information. $ii$) Integrity, which refers to the
risk of manipulability of the data by unauthorized entities and ensures that
data is not corrupted, and not outdated. $iii$) Availability, which refers to
the availability of data information and services to authorized entities.
Specifically, the Non RT RIC Security Technical Report identifies $26$ threats
to the Non RT RIC Framework, rApps, R1 interface, and A1 interface, with
corresponding recommended security controls [177]. The technical report on the
security analysis of O-Cloud covers critical services, cloud service and
deployment models, stakeholder roles and responsibilities, threat models, and
best practices for countering threats [178]. The Near RT RIC and xApps
Security Technical Report addresses $11$ key issues and provides $13$
solutions, including changes to existing documents and specifications
maintained by WG 3, with a mapping table for which solutions correspond to
which key issues [179].

As the interest in O-RAN deployment increases, security risk assessments have
been carried out by third parties and government entities. In [180] the
aforementioned integrity, availability, and confidentiality are evaluated
alongside two additional protection goals namely Accountability, which refers
to the possibility of attributing an action to a given entity, and privacy,
which refers to the protection of sensitive data, i.e., anonymity,
unlinkability, and unobservability. From the analysis it emerged that O-RAN
specifications lack a "security/privacy by design/default" approach, resulting
in a system with multiple security risks and highlighting the importance of
revising the O-RAN specifications with a stronger security focus before the
first productive applications are implemented. In [181] the risks due to the
potential for multiple suppliers, new network functions, and additional
interfaces that increase the attack surface are highlighted. Moreover, in
[181], particular attention is given to the risks coming from the
implementation of network functions with AI and ML, which could negatively
impact the network. Additionally, using cloud platforms to run base station
software in O-RAN could increase dependency on cloud service providers and
lead to vulnerabilities, especially if different Mobile Network Operators
(MNOs) use the same cloud provider.

At the time of writing, and to the best of our knowledge, there are no risk
assessments specifically targeting security threads introduced by the use of
XAI, nor suggesting XAI-based solutions/recommendations to enhance security in
open networks. Nonetheless, we will discuss them shortly in the upcoming
subsections..

###  VII-C XAI to Improve O-RAN Security

The utilization of XAI in the security domain of O-RAN could help enhance the
transparency and comprehensibility of the operations and decision-making
processes of third-party deployed components. This could be particularly
helpful to enable stakeholders to fully understand the decision process of
such elements, helping to catch malicious behaviors, thus ensuring
accountability and reducing the risk of errors or malicious actions. This is
particularly relevant when considering the high number of AI\- and ML-driven
components that will be deployed in the network, which due to their black-box
nature pose a significant challenge to reveal malicious behavior and security
threats. As a result, ensuring that AI remains accountable and trustworthy is
of utmost importance [175]. By implementing XAI techniques, the complex
algorithms used in ML-/AI-based systems can be made more interpretable,
allowing stakeholders to better understand the factors influencing the
automated decision-making processes. This enhances transparency and enables
stakeholders to identify potential biases or shortcomings in the system,
allowing for continuous improvement and optimization.

In the O-RAN architecture, AI/ML models are mainly deployed in the RIC as
xApps/rApps [171], as depicted in 12. Such elements bring in the autonomous
operation of several vital network functions including mobility management,
resource allocation, etc. Hence, the deployment of malicious AI/ML models or
the manipulation of benign ones by attackers could disrupt RAN node
functionalities, resulting in severe network failures [38]. In order to
overcome this issue, XAI-enabled security engines can be in the Non RT RIC and
Near RT RIC as additional functional blocks performing interpretable
monitoring of the xApp/rApp operation and malicious components detection
[175]. Nonetheless, the employment of XAI techniques O-RAN will require
additional effort to build pipelines to generate and communicate explanations
in the components hosting the XAI models. Additionally, it will call for more
computation power and resources to run both ML and XAI models. However, these
added costs are justifiable by the enhancement of O-RAN security and
management capabilities [175].

###  VII-D Security threats related to XAI

While XAI methods can be employed to improve security in O-RAN deployments, as
XAI becomes an emerging trend, cyberattacks targeting specifically the XAI
models are rising [182]. In [183] two different attacks to the XAI layer are
proposed: the first wherein the underlying ML model and XAI interpreted are
simultaneously corrupted. The attack aims to construct adversarial samples
that are sparse, interpretable, and close to the model’s training data
distribution by using a manifold approximation algorithm on a small set of
test data to find data and explanation distributions and inducing minimal
distortions on the input distribution to move the explanation distribution
towards the target (distorted) explanation.

Similar attacks can vanish the security advantage of an overlaying XAI layer
[184]. For example, Adversarial and Data poisoning attacks involve
intentionally modifying input data to mislead an AI system’s predictions and
explanation, aiming of biasing the outcome of the model and leading to
intentional misinterpretation of the model’s behavior [183]. Evasion attacks,
involve revealing sensitive information through the explanations or
interpretations provided by an AI system. This can be achieved by using the
explanations to infer sensitive information about individuals, such as their
health status, financial situation, or other private information, even if the
AI system was not explicitly trained on such data [184]. Finally, social
engineering attacks, involve manipulating users or human interpreters of the
AI system’s explanations to make incorrect or biased interpretations. This can
be achieved by providing misleading or persuasive explanations that influence
the human interpreter’s decision-making or perception of the AI system’s
behavior [185]. However, the research in the area of security attacks
specifically targeting XAI models is still in its infancy.

##  VIII Projects/Standards on XAI for O-RAN

XAI is increasingly becoming critical for the adoption of ML/DL in O-RAN. To
achieve trustworthiness and transparency in ML/DL models in O-RAN, there are
some ongoing standardization activities and research projects targeting XAI
and O-RAN aspects. Some of them include:

  * •

O-RAN Alliance: As we describe in Subsec. II-D, the O-RAN Alliance is a global
organization that is working to promote an intelligent and open RAN for mobile
cellular networks. As shown in Fig. 2, the O-RAN Alliance comprises $11$ WGs
and three focus groups, for RAN cloudification, automation, and
disaggregation. In particular, WG 2 in [171] describes lifecycle management of
AI/ML models on O-RAN including learning model design, composition, training,
runtime, and deployment solutions. it also highlights the main criteria for
determining multiple ML training and inference host deployment options. In
this context, the focus of WG 2 can be extended to implementing XAI in O-RAN.
To promote XAI adoption in O-RAN, the WG 2 can work on various initiatives,
including the creation of XAI platforms and tools, the development of
interfaces and standards for XAI, and the promotion of XAI best practices.

  * •

IEEE P2894 and P2976: these standards aim to deliver specifications on XAI in
order to facilitate its adoption in real-world scenarios. The IEEE P2894
standard aims to design an architectural framework and define application
guidelines for XAI, including the definition and description of XAI, the main
classes of XAI techniques, the main application scenarios of XAI techniques,
and performance evaluations of XAI in real systems such as telecommunication
networks [186]. Besides, the IEEE P2976 standard is working to achieve
interoperability and clarity of AI systems design through leveraging XAI
techniques [187]. Specifically, IEEE P2976 defines optional and mandatory
constraints and requirements that should be satisfied for an AI algorithm,
method, or system to be considered explainable. In this, context, these
specifications can be leveraged by O-RAN standards such as O-RAN Alliance in
order to develop and advance the adoption of XAI in the O-RAN ecosystem.

  * •

ETSI Experiential Networked Intelligence (ENI): The ETSI Industry
Specification Group (ISG)is working on defining a cognitive network management
architecture, based on context-aware policies and leveraging AI techniques to
adjust provided services in 5G networks and beyond, based on changes in
business goals, environmental conditions, and user requirements. Thus, it aims
to provide automated service operation, provision, and assurance, as well as
efficient resource management and orchestration. Besides, recently, ETSI
releases its first specifications on O-RAN called "O-RAN Fronthaul Control,
User and Synchronization Plane Specification v7.02" [188]. This specification
focuses on Open Fronthaul, as one of the interfaces in the O-RAN Architecture.
It specifies the synchronization plane protocols, user plane, and control
plane used over the fronthaul interface to link the O-RU and O-RU components.
Noting that this specification has been submitted to ETSI as a publicly
available specification (PAS) produced by the O-RAN WG 4 and approved by the
ETSI Technical Committee. Therefore, considering this first specification of
ETSI about O-RAN, the ETSI Experiential Networked Intelligence (ENI) Industry
Specification Group (ISG) can also focus on adopting XAI on top of the
designed cognitive network architecture in order to create an AI framework
that is explainable, transparent, and thus can be used to ensure the
accountability of AI-enabled systems in O-RAN.

  * •

6G-Bricks: is a Horizon Europe project that explores novel unified control
paradigms based on Explainable AI and Machine Reasoning, which will be
delivered in the form of a reusable component with open Application
Programming Interfaces (APIs), termed "bricks" [189]. Initial integration with
O-RAN will be performed, aiming for the future-proofing and interoperability
of 6G-BRICKS outcomes.

  * •

NANCY: it is the acronym of _An Artificial Intelligent Aided Unified Network
for Secure Beyond 5G Long Term Evolution_; a Horizon Europe project which
partly investigates the design of an XAI engine, to provide transparency and
trustworthiness [190]. It also aims to identify the key factors that affect
the system’s local and overall performance.

  * •

Nokia Project: Nokia has opened a testing center in the United States. and
initiated new O-RAN collaboration to support the development of partnerships
among O-RAN vendors, that will help with the introduction, verification, and
launch of O-RAN compliant solutions to market [191]. Specifically, vendors
will be able to perform end-to-end and Interoperability Tests for O-DU/O-RU
Open Fronthaul, as well as xApps testing for Nokia’s Near RT RIC. The project
is the latest in Nokia’s continued commitment to Virtual RAN (vRAN) and O-RAN
innovation. This project can also be leveraged to implement, deploy, and
validate XAI-based xApps on top of the O-RAN Near RT RIC.

Overall, these standards and projects are working to promote the adoption of
AI techniques, in particular machine learning and deep learning, in O-RAN,
while ensuring that these technologies are interpretable, accountable, and
transparent. By doing so, they can help build trust in AI systems deployed in
O-RAN, and thus encourage competition and innovation in the telecommunications
industry.

##  IX Open Challenges and Future Research Directions

In this section, we analyze different open questions and research topics for
an efficient deployment of XAI in future-proof 6G O-RAN architecture, while
pointing out various challenges in this area.

Figure 13: Confidence vs. FL rounds

###  IX-A Explainability-Performance Trade-Off

The flip side of the highly performing yet complex AI models, such as DNNs and
Transformers, is being ill-disposed to direct interpretation. They consist of
numerous layers and billions of parameters, making it difficult to explain how
specific inputs are transformed into outputs. In contrast, simpler models like
decision trees or linear regressors are more interpretable, as their decision-
making processes are more transparent. Therefore, it turns out that there is a
trade-off between performance and explainability, especially when the type of
training data justifies the use of complex models [46, 192]. Such observation
is also valid when it comes to model optimization. To exemplify this, we plot
the confidence metric described in Subsec. II-C in two scenarios, i) In each
round, an FL model is optimized via a vanilla loss function such as mean
square error and then a post-hoc explanation in the form of attributions is
generated and used to calculate the confidence metric and ii) In each round,
an FL model is trained through a constrained optimization approach, where the
confidence metric is evaluated in run time and jointly enforced as a
constraint along with the original loss during training. According to Fig. 13,
the model confidence degrades as it gradually converges in the post-hoc
scenario which highlights the abovementioned trade-off. Interestingly, the
_in-hoc_ strategy succeeds in maintaining the model confidence across the
training rounds as thoroughly studied in [142]. Striking a balance between
performance and interpretability is therefore an open research direction,
especially to guarantee a successful deployment of AI in critical 6G use-cases
under O-RAN architectures.

###  IX-B LLMs in O-RAN: An Explainability Perspective

As anticipated in [193], the potential of LLMs to transform the Telecom domain
lies in their ability to harness generative capabilities and leverage the
multimodal nature of wireless network data, thereby enhancing contextual,
situational, and temporal awareness. This advancement holds the promise of
significantly improving wireless networks operations, including localization,
beamforming, power allocation, handover, and spectrum management, while also
eliminating the requirement for task-specific AI models. Although this is
still a far cry from achievement, future-proof 6G O-RAN might incorporate LLMs
into the design of the different radio functions. In such a scenario, the
complex architecture of LLMs that is mainly based on transformers with
billions of parameters raises new challenges with respect to their
explainability and opens a new research line to tackle it.

###  IX-C Lack of Standardization

One of the still open challenges related to the adoption of XAI to O-RAN is
the lack of standardization efforts across different components and
interfaces, except the O-RAN alliance and some ongoing projects initiated by
network operators [20][21][191] that are more focusing on the main O-RAN’s
components. This makes it difficult to design XAI tools that can be deployed
across different O-RAN components. Indeed, standardization is critical for
enabling interoperability among various components of O-RAN and facilitating
the development and deployment of XAI tools, on top of O-RAN. In this context,
the research and industry communities should work towards developing common
standards for open interfaces, data formats, APIs, etc.

###  IX-D Privacy Concerns of Distributed XAI Models for the Complex Multi-
Vendor O-RAN

As mentioned before, one of the main features of O-RAN is to disaggregate the
RAN functions and manage them through the running xApps at the Near RT RIC, to
fulfill the QoS requirements of the envisioned B5G network services. In
addition, the different O-RAN components are supplied and supported by various
isolated vendors/operators. However, XAI tools typically need large amounts of
data to train and test their models for O-RAN systems, and the availability of
data may be limited or difficult to access, due to security and privacy
concerns in a multi-vendor context. Therefore, these vendors/operators should
collaborate with each other, in order to not only ensure stable RAN
performance/cost but also deal with the limited available data.
In this context, distributed/collaborative deep learning is expected to be
widely leveraged. For instance, FL is one of the promising collaborative
learning techniques that consists of generating learning models in a
collaborative way while preserving the privacy of involved learners such as
vendors/operators [135]. However, generating learning models (XAI or AI) in a
federated way is still challenging since it still presents privacy concerns.
Indeed, even FL avoids sharing learners’ private data, however, it was
demonstrated that FL is still vulnerable to some attacks, such as poisoning
and Byzantine attacks, and sharing model updates during the training process
can reveal private information [194]. Privacy-preserving XAI techniques can be
used to provide explanations without revealing sensitive information. These
techniques include methods such as differential privacy, blockchain, and