# How well can large language models explain business processes?111This
project has received funding from the European Union’s Horizon research and
innovation programme under grant agreements no 101092639 (FAME), 101094905
(AI4GOV), and 101092021 (AutoTwin).

Dirk Fahland<EMAIL_ADDRESS>Fabiana Fournier<EMAIL_ADDRESS>Lior
Limonad<EMAIL_ADDRESS>Inna Skarbovsky<EMAIL_ADDRESS>Ava J.E. Swevels
<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) are trained on a vast amount of text to interpret
and generate human-like textual content. They are becoming a vital vehicle in
realizing the vision of the autonomous enterprise, with organizations today
actively adopting LLMs to automate many aspects of their operations. LLMs are
likely to play a prominent role in future AI-augmented business process
management systems (ABPMSs) catering functionalities across all system
lifecycle stages. One such system’s functionality is Situation-Aware
eXplainability (SAX), which relates to generating causally sound and yet
human-interpretable explanations that take into account the process context in
which the explained condition occurred.

In this paper, we present the SAX4BPM framework developed to generate SAX
explanations. The SAX4BPM suite consists of a set of services and a central
knowledge repository. The functionality of these services is to elicit the
various knowledge ingredients that underlie SAX explanations. A key innovative
component among these ingredients is the causal process execution view. In
this work, we integrate the framework with an LLM to leverage its power to
synthesize the various input ingredients for the sake of improved SAX
explanations.

Since the use of LLMs for SAX is also accompanied by a certain degree of doubt
related to its capacity to adequately fulfill SAX along with its tendency for
hallucination and lack of inherent capacity to reason, we pursued a
methodological evaluation of the quality of the generated explanations. To
this aim, we developed a designated scale and conducted a rigorous user study.
Our findings show that the input presented to the LLMs aided with the guard-
railing of its performance, yielding SAX explanations having better-perceived
fidelity. This improvement is moderated by the perception of trust and
curiosity. More so, this improvement comes at the cost of the perceived
interpretability of the explanation.

###### keywords:

Business Process , Methodologies and Tools , AI , Explainability , Large
Language Models

††journal: Data and Knowledge Engineering

[o1]organization=Eindhoven University of Technology,city=Eindhoven,
country=Netherlands [o2]organization=IBM Research,city=Haifa, country=Israel

##  1 Introduction and motivation

Explanations are the foundation for the adoption and trust of humans in
business processes augmented by AI (ABPMSs) [1]. Via explanations, process
users can understand and act upon the various situations that evolve during
process executions. Thus, explainability is deemed as an inherent
functionality of the ABPMS.

Explanations in the context of business (BPs) processes are expected to take
advantage of the knowledge of the business process definitions and full
runtime process traces. Furthermore, they are expected to embed the ability to
go beyond a local reasoning context, handle a large variety of situations, and
facilitate the (automatic or by humans) tracking of execution consistency for
a better understanding of process flows and process outcomes. Adequate
explanations of business processes are key to driving ongoing process
improvements.

According to Forrester’s Q3 2022 survey results, process optimization and
automation are now mainstream skills that directly align with strategic
business
objectives222https://www.forrester.com/report/forresters-q3-2022-digital-
process-automation-survey-results-organizations-respond-to-the-automation-
imperative/RES178220. This progress of automation and AI also presents an
opportunity for the automation of explainability in the area of BPM. The
global business process management (BPM) industry generated $15.4 billion in
2022 and is anticipated to generate $65.8 billion by
2032333https://www.marketwatch.com/press-release/business-process-management-
bpm-market-to-reach-65-8-billion-globally-by-2032-at-15-8-cagr-allied-market-
research-25380466. As with the evolution of technologies for explainability
that accompany recent advances in ML, explainability in BP has a great
potential to untangle the evolving complexity of organizational processes and
promote trust in and adoption of the automation technology (e.g., [2]).

One way to automate explainability is to exploit Large Language Models (LLMs)
capabilities. Concretely, as supported by recent Gartner’s report [3], LLMs
are being adopted across all industries and business functions, driving a
variety of use cases such as text summarization, question-answering, document
translation, and alike. LLMs can also be augmented by additional capabilities
to create more powerful systems, and feature a growing ecosystem of tools.
Among these, we foresee the benefit of leveraging LLMs as a means for the
automation of explanations in business processes.

However, the use of LLMs as a vehicle for explanation generation comes with a
certain caveat. While LLMs present a remarkable performance in various NLP
tasks without the need for supervised training [4], they are not good causal
reasoners and exhibit a critical tendency to produce hallucinations, resulting
in content that is inconsistent with real-world facts or user inputs. This
phenomenon poses substantial challenges to their practical deployment and
raises concerns over the reliability of LLMs in real-world scenarios, which
attracts increasing attention to detect and mitigate these hallucinations [5].

Our main objective is to guardrail the functionality of LLMs such that they
can be leveraged to give explanations that adequately point to the
circumstances that bring about the unfolding of business processes towards
certain conditions. We hypothesize in this work that the perceived quality of
LLM-generated explanations of this sort could be altered and even improved via
the methodological injection of different knowledge articulations that are
given to it as an input leveraging LLM prompt engineering.

Furthermore, we focus on the incorporation of a recently developed view that
highlights the causal execution dependencies among the activities in the BP
[6]. We add it as an additional input to the LLM. To our knowledge, this is
the first attempt to prompt an LLM with a causal component for the sake of BP
explainability. Correspondingly, our goal is to automate explanations for BPs
using LLMs. For this, our research approach is to blend a variety of business
process-related views e.g., process, causal, and XAI. The combination of these
views forms the fuel for the automatic generation of narratives that serve as
input for prompt engineering of LLMs to achieve better process outcome
explanations as illustrated in Figure 1.

Figure 1: General approach

The paper’s contribution is in providing tools for integration with LLMs to
automate such explanations automation in business processes, examining how
such explanations may be shaped by different inputs to the LLM, and a
methodological evaluation of the quality of such explanations.

Our paper is structured as follows: We begin by reviewing some fundamental
concepts in Section 2. This is followed by a presentation of the developed
tooling in Section 3, and an illustrative example demonstrating the usage of
the framework in Section 4. Subsequently, we evaluate the tool, addressing our
research question in Section 5, describe our method of evaluation in Section
6, detail a corresponding experimental design in Section 7, and discuss the
results in Section 8. We conclude the paper with a review of related work in
Section 9, and our conclusions in Section 10.

##  2 Background

We briefly explain in this section the main concepts related to our work.

Large language models (LLMs) are deep-learning (DL) models trained on vast
amounts of text data to perform various natural language processing tasks. One
of their strengths is their ability to perform few-shot and zero-shot learning
with prompt-based learning [7].

With the emergence of LLMs, the practice of ‘prompt engineering’ has been
explored recently as a new field of research that focuses on designing,
refining, and implementing different instructions to guide the output of LLMs
in various tasks to achieve a more concise and effective utilization of the
LLMs, adapting it to the corresponding skills of the users and the context of
the different tasks such as analysis or reasoning [8]. This may include
different techniques such as one-shot or few-shot samples, the use of quotes
to annotate different parts of a prompt, employing methodological evolution of
the interaction via advanced methodologies such as “Chain of Thought” (CoT),
and configuration of LLM-specific settings like temperature and top-p sampling
to control the variability of the output.

In this work, we evaluate the use of LLMs as a means to synthesize a
combination of business process-related knowledge ingredients presented in
textual form as input to the LLM. Subsequently, we employ the LLM to
articulate explanations that correspond to user inquiries about particular
situations or ‘conditions’ that may occur during business process executions.

Business Process Management (BPM) is a managerial effort devoted to ensuring
the constant, efficient, and effective movement of a business entity towards
its constantly changing goals. This entails adopting a process view of the
business and includes planning, analysis, design, implementation, execution,
and control of business processes. A business process (BP) is a collection of
tasks that execute in a specific sequence to achieve some business goal [9].
The digital footprint that depicts a single execution of a process as a
concrete sequence of activities or events is termed a ‘trace’ [10]. A multi-
set of traces is usually referred to as a trace-log or event-log.

Process discovery (PD) aims at gaining insights into the business processes of
organizations by analyzing event data recorded in their information systems
for the sake of business improvement [10]. PD summarizes an event log $L$ into
a graph model $M$ that represents activities and control-flow dependencies
[11]. Typical process mining techniques are inherently associational,
approaching process discovery from a time precedence perspective, i.e., they
discover ordering constraints among the process’ activities. That is, most PD
algorithms construct edges in $M$ that indicate to which subsequent activities
process control “flows to”.

Associative relationships are not enough to develop the causal understanding
necessary to inform intervention recommendations as aforementioned. Most
statistical learning procedures reflect correlation structures between
features instead of their true, inherent, causal structure which is the true
goal of explanations [2]. In order to understand which intervention should be
made to improve a process outcome, we first must understand the causal chains
that tie the unfolding of the processes generating our data. Unraveling the
causal relationships among the execution of process activities is a crucial
element in predicting the consequences of process interventions and making
informed decisions regarding process improvements. Our previous work [6]
demonstrated that relying only on time precedence between activities in a
business process does not necessarily reflect the full cause-effect
dependencies among the tasks and a more fundamental analysis of causal
relationships among the tasks is required.

Causal discovery (CD) infers causal graphs from data by exploring
relationships like $A\xrightarrow{c}B$ where changes in $A$ entail changes in
$B$. Causal relationships describe the connection between a cause and its
effect, where the cause is an event that contributes to the production of
another event, the effect [12]. Causal discovery is responsible for creating
models (causal graphs) that illustrate the causal relationships inherent in
the data [13, 12, 14]. Identification of causal relationships is key to the
ability to reason about the consequences of interventions. One of the
fundamental goals of causal analysis is not only to understand exactly what
causes a specific effect but rather to be able to conclude if certain
interventions account for the formation of certain outcomes, thus, being able
to answer questions of the form: Does the execution of a certain activity in a
loan approval process entail a delay in the handling of the application? or if
the same activity is skipped, may the process duration be shortened? In our
work, we focus on causal discovery and adapt for its employment over process
execution times, as inherently recorded in process event logs. More
specifically, we leverage the Linear Non-Gaussian Acyclic Model (LiNGAM) [15]
for CD as in [6] to uncover the causal execution dependencies among the events
in event logs. Inspired by[16], which highlights LLMs’ ability to provide
interpretable explanations, we aim to demonstrate that CD can enhance
explanations of process execution outcomes when used as input for LLMs.

The penetration of AI into BPM has vast implications across the various
dimensions of BPM [1], featuring new capabilities to better predict process
executions, understand and foresee the implications of changes, and gradually
automate the different practices associated with process management. However,
the efficiency of AI-based solutions is proportionally related to their
complexity. As the accuracy and efficiency of machine learning (ML) solutions
increase, their complexity increases while reducing human interpretability
requiring external explanation frameworks [17, 18, 19, 20].

eXplainable AI (XAI) is the ability to understand and interpret how AI systems
make decisions or arrive at conclusions. Such frameworks are predominately
developed for post-hoc interpretations of ML models [18, 19]. Context-wise,
they can be divided into global, local, and hybrid explanations [18, 21, 22].
Global explanations attempt to explain the ML model’s internal logic, local
explanations try to explain the ML model’s prediction for a single input
instance, and hybrid approaches vary (e.g., explaining the ML model’s internal
logic for a subspace of the input space).

In contemporary XAI techniques (e.g., LIME [23] or SHAP [24]), the ML model
serves as a surrogate model typically trained using historical process
execution logs of the BP. The predicted value for a single instance (process
outcome) serves as input for the XAI explainer to produce an explanation. Our
work adds to a series of recent efforts [25, 26] that focus on exploiting XAI
frameworks that are compatible with tabular data for the interpretation of BP
execution results. We use process logs as the main data input and train
surrogate ML models with this data to represent real-world business processes.

However, contemporary techniques are not adequate to produce explanations
faithfully and correctly when applied to BPs as they generally fail to

  * •

Express the business process model constraints (i.e., the semantics of the
process model),

  * •

Include the richness of contextual situations that affect process outcomes
(additional information that affects the outcome but usually not modeled),

  * •

Reflect the true causal execution dependencies among the activities in the
business process, or

  * •

Make sense and be interpretable to process users (explanations are usually not
given in a human-interpretable form that can ease the understanding by
humans).

Our goal is to combine PD, CD, and XAI to generate narratives for improved
process outcome explanations using LLMs.

To this end, we introduce Situation-aware eXplainability (SAX) as a framework
for generating explanations that address the aforementioned shortcomings.

More specifically, a “SAX explanation” is a causally-sound explanation that
takes into account the process context in which some condition occurred (the
“explanandum”). Causally-sound means that the explanation given provides an
account of why the condition occurred in a faithful and logical entailment
that reflects the genuine chain of business process executions yielding the
condition. Context includes knowledge elements that were originally implicit
or part of the surrounding system, yet affected the choices that have been
made during process execution.

The SAX framework is realized through a set of services in the SAX4BPM library
(Section 3) that aids with the automatic derivation of SAX explanations
leveraging existing LLMs.

##  3 SAX tooling (towards a plugin for LLM)

The SAX4BPM library includes a central knowledge repository and a set of
services and capabilities to support the generation of SAX explanations. In
this work, we specifically focus on the services that are related to the
integration with the LLM. The library is implemented using Python 3.9
programming language. A boilerplate application demonstrating its use is
accessible at: https://github.com/IBM/SAX4bpm/notebook/demo.ipynb.

###  3.1 Knowledge graphs

All data persistence for the SAX4BPM library is realized in the form of a
knowledge graph (KG) [27].

We use a _Labeled Property Graph_ (LPG)[28] as a data model, as it allows to
store process-related data more compactly than RDF-based knowledge graphs
[29]. An LPG $G$ is a directed multi-graph where each node (modeling an entity
or concept) is typed by one or more _labels_ , and each directed edge
(modeling a relationship between entities/concepts) is typed by exactly one
label. Each node and relationship has _properties_ in the form of attribute-
value pairs.

We use a multi-layered LPG stored in a Neo4j database as infrastructure for
our KG, and use Neo4j’s query language Cypher for _knowledge inference_. We
use the schema of [29] as the “base layer” for our KG as shown in the graph
schema in Fig. 2. Each _Event_ node has a _timestamp_ , and is correlated to
one _case_. _Inference rules_ over events and cases infer the _directly-
follows_ relation that describes the temporal order of all events correlated
to the same case. These concepts allow modeling any event log in a KG [29].

Figure 2: Knowledge graph schema

The graph schema can be extended with additional “layers” of nodes and
relations for _integrating_ a variety of views on the process in the KG. For
example, the schema can be extended with an _Entity_ node to enable a multi-
dimensional or “object-centric” view of the process [30]. In our case, we
extended the schema with the _activity_ and _feature_ nodes (the former
holding a _set-of_ the latter), and the _flows-to_ and _causes_ relations for
two activity-level inferred views:

  * •

_Process view_ \- we infer _activities_ and _directly-follows_ ordering of
activities by aggregation from events [29]. Accordingly, we infer the _flows-
to_ relation through process discovery, e.g., [31].

  * •

_Causal view_ \- we infer the _causes_ relationship as a _causal-execution-
dependency_ as in [6]. Accordingly, we infer the _causes_ relation among the
activities as annotated.

  * •

_XAI view_ \- we capture the set of features and their corresponding
importance values in the context of each activity.

###  3.2 Elicitation of process knowledge ingredients

The library includes a set of functionalities or services for the elicitation
of the process knowledge ingredients applied in our study, namely: process,
causal, and XAI. Figure 3 depicts the flow of the SAX4BPM services yielding an
explanation.

Mining4Process – The goal of this service is to generate the process model
knowledge ingredient. This service fetches the process event log data from the
KG and forwards it for the invocation of an external process discovery
algorithm (e.g., heuristic miner [32]). The result process model (a graph) is
stored back in the KG and also exported in JavaScript Object Notation (JSON)
format as input to the LLM.

Causal4Process – The goal of this service is to generate the causal process
model knowledge ingredient. This service fetches the process event log data
and the process model from the KG and employs a novel algorithmic approach [6]
to elicit the causal process model. This algorithm leverages an existing
causal discovery algorithm (i.e., LiNGAM), and complements it with the needed
adjustments for process execution times. The result graph model is stored back
in the KG and exported in JSON format as input to the LLM.

ContextEnrichment – The goal of this service is to enrich the event log with
any contextual information associated with the process and the environment in
which it executes. As an example of such enrichment, our experimentation with
complex event processing [33] shows that temporal contextual information can
be leveraged to improve the adequacy of explanations given for process
execution instances. The effect of the enrichment manifests itself not just in
properly adjusting the importance of features that correctly correspond to the
outcome, but also in promoting the accuracy of the surrogate ML model. This
service fetches the process event log from the KG and applies a set of
situational rules to enrich the event log. The result is an enriched event log
which is stored back in the KG.

X4Process – The goal of this service is twofold. First, to generate and
determine importance ranking over the set of features that are used in the
process to predict the condition of interest while adhering to the constraints
of the process model. Second, it uses the ContextEnrichment service to enrich
the event log with broader, context-relevant attributes, about the same
condition. This service invokes the ContextEnrichment service from the library
and subsequently fetches the enriched event log and the process model
(including constraints and flow control logic) from the KG, both used to
constrain the search space of an invoked XAI service (e.g., SHAP or LIME) to
ensure conformance to process [25]. The result is a feature importance vector,
segmented according to the activities in the process, that is stored back in
the KG and exported in JSON format as input to the LLM.

###  3.3 Synthesize of knowledge elements for LLM input

We consider different possible combinations of knowledge ingredients as input
for an LLM to facilitate the derivation of explanations about concrete process
conditions.

NLP4X – The goal of this service is to interweave the various knowledge
ingredients and act as a facade for the interaction with the user, prompting
this composition to an LLM. This service receives as input the query (the
condition the user seeks an explanation for) and the selection of knowledge
ingredients. The core functionality of the service is the synthesis of the
input ingredients by invoking the corresponding SAX4BPM services and their
textual streamlining via the LLM prompt toward the eventual elicitation of the
explanation narrative. While the current implementation is hard-coded with
predetermined leading phrases to facilitate the interaction, future work aims
to automate this step by applying dynamic templating. Our library currently
applies ChatGPT444https://chat.openai.com from OpenAI as the LLM.

Figure 3: SAX4BPM services invocation

##  4 Illustrative example: Parking fines

To illustrate the application of the SAX4BPM services, we generated data on a
business process of parking fines.

In this process (see Figure 4), a parking ticket is given when a vehicle is
parked in a prohibited lot and does not possess a disabled permit. In this
case, two types of fines can be given depending on whether the parking place
is hazardous (e.g., the vehicle is parked on a sidewalk or a crosswalk) or
not. In the case of a hazardous place, an extended fine is issued, and a tow
truck is called. Note that the arrival time of the tow truck is always later
than the time taking to submit the extended fine.

We generated the dataset for this example using BIMP open-source log
simulation tool555https://bimp.cs.ut.ee/simulator/ and stored it in the KG.

Figure 4: Parking fines scenario

We demonstrate the capabilities developed in the library through the
application of the different services.

The condition for which an explanation is sought: Why does the processing of
fines for cars that are parked within hazardous locations take so long?

###  4.1 Mining4Process Invocation

The next step is invoking the Mining4Process service to discover a process
model from the event log stored in the KG. Figure 5 shows the process model
discovered in our scenario of parking fines. In the context of this concrete
example, we further zoomed in on the process model on the branch corresponding
to the issuing of fines for cars parked at hazardous locations (with 249
cases). The generated JSON that serves as an input for the LLM is as follows:

(’EVENT 1 START’, ’verify handicap parking permit’): 1000, (’verify handicap
parking permit’, ’check if hazardous parking’): 893, (’check if hazardous
parking’, ’Submit fine’): 644, (’Submit fine’, ’EVENT 3 END’): 644, (’check if
hazardous parking’, ’Submit extended fine’): 249, (’Submit extended fine’,
’call a tow truck’): 249, (’call a tow truck’, ’EVENT 3 END’): 249, (’verify
handicap parking permit’, ’EVENT 3 END’): 107

Figure 5: Process model discovery for the parking fines scenario

###  4.2 Causal4Process Invocation

The next step is invoking the Causal4Process service to discover the causal
model. Note that the two consecutive tasks - “submit extended fine” and “call
a tow truck” - in the discovered process model are not causally dependent in
the causal model and can be carried out concurrently. Figure 6 shows the
causal model discovered in our scenario of parking fines. The generated JSON
that serves as an input for the LLM is as follows:

‘Cause’:‘EVENT 1 START’, ‘Effect’:‘verify handicap parking permit’,
‘Coefficient‘:‘1.00’ ‘Cause’:‘verify handicap parking permit’, ‘Effect’:‘check
if hazardous parking’, ‘Coefficient’:‘1.41432045’, ‘Cause’:‘check if hazardous
parking’, ‘Effect’:‘call a tow truck’, ‘Coefficient’:‘0.95186106’,
‘Cause’:‘check if hazardous parking’, ‘Effect’:‘Submit extended fine’,
‘Coefficient’:‘1.00753386’, ‘Cause’:‘call a tow truck’, ‘Effect’:‘EVENT 3
END’, ‘Coefficient’:‘1.00’

Figure 6: Causal graph discovery for the parking fines scenario

###  4.3 X4Process Invocation

The next step is invoking the X4Process service to elicit the feature
importance vector. The context enrichment service enhanced the event log with
the following features: “driver’s credit”, “filling out hazardous
circumstances”, “region in the city”, and “choice of towing company”. Figure 7
shows the importance of the features discovered in our scenario of parking
fines. As can be seen, the “reason for hazardous location” is the most
important factor causing delays in the process. The generated JSON that serves
as an input for the LLM is as follows:

”check if hazardous parking”: “region in city: 0.1, ”Submit extended fine”:
”filling out hazardous circumstances”: 0.9, ”driver’s credits”: 0.2, ”call a
tow truck”: ”choice of towing company”: 0.6

Figure 7: XAI feature importance for the parking fines scenario

###  4.4 NLP4X Invocation

The last step is invoking the NLP4X service to drive the interaction with the
LLM for explanation generation. As per the user’s selection of the combination
of knowledge ingredients, all three are selected in this example as depicted
in Figure 8, and the query, the service is invoked. In our example, this
invocation yields the answer as shown in the bottom part of the figure.

Figure 8: SAX4BPM explanation for the parking fines scenario

##  5 Hypotheses

With our intention to employ LLMs as instrumentation for the generation of
process-related explanations, we have set up our goal to tackle the following
research question:

Research Question: How does having informed knowledge of business processes
influence the perceived quality of explanations generated by LLMs regarding
business process conditions?

Where a condition in a business process relates to any possible state or
outcome that may be brought about during the execution of the business
process. In an effort to empirically address this question, we designed an
experiment that concretely incorporates the various elements of this research
question. To formalize it further, we first reformulated the question by
operationalizing the ‘perceived quality of LLM-generated explanations’ into
its two main constructs (i.e., dependent variables), as identified in prior
literature: namely, ‘fidelity’ and ‘interoperability’. Each of the two
constructs was further instantiated by the formation of the underlying
measurement dimensions (i.e., as formative constructs [34]) as further
elaborated in the scale development section 6.1. From this, we derived two
high-level hypotheses as follows.

######  Hypothesis 1 (H1).

Explanations generated by LLMs informed by knowledge about business processes
will be perceived as having higher fidelity compared to explanations generated
by uninformed LLMs.

######  Hypothesis 2 (H2).

Explanations generated by LLMs informed by knowledge about business processes
will be perceived as having higher interpretability compared to explanations
generated by uninformed LLMs.

##  6 Method

We conducted a controlled experiment to assess our hypotheses. The experiment
involved between-group manipulations of various knowledge components related
to different business processes. These components were presented as input to
an LLM, which was then prompted to generate an explanation — a textual
narrative — articulating the reasons for a specific condition occurring during
the execution of the business process. The different explanation variants
generated under various manipulations were then incorporated into
corresponding versions of a questionnaire. This questionnaire included a set
of rating scales developed to measure the perceived quality of the
explanations. After collecting user ratings, we tested how the manipulations
influenced the perceptions across different groups corresponding to the
various explanation variants.

We targeted the population of the global Information Technology community that
might benefit from recent advances in LLMs to leverage such technology to
automate the generation of explanations about situations in their
corresponding area of interest. Pragmatically, we exploited an online
questionnaire that was disseminated via three channels for the acquisition of
responses: graduate students in the computer science faculty at TU/e,
employees at IBM research, and the Reddit r/SampleSize online community.
Participation was voluntary and anonymous without collecting any personal
information. Before their participation, users gave their consent. Ethical
approval was given by the Tu/e Ethical Review Board (#ERB2023MCS42).

The survey included a total of 50 participants. One respondent did not give
consent, resulting in 49 completed survey responses. Regarding gender, 28 were
men (57.1%), 19 were women (38.8%), and 2 chose not to specify their gender
(4.1%). With respect to age, the distribution was as follows: 23 participants
(46.9%) were above the age of 45, 15 participants (30.6%) were aged between 25
and 35 years, 6 participants (12.2%) were below 25 years of age, and 5
participants (10.2%) were between 36 and 45 years old.

###  6.1 Scale Development

To assess the quality of the explanations generated by the LLM, we first
developed scale metrics based on different measurement dimensions for an
explanation’s quality. For various reasons, to date, there is no agreed
framework for the evaluation of explanations. There is no agreed definition of
explainability, furthermore, there is no agreement on what an explanation is,
and which are the salient properties that should be considered to make it
effective and understandable for end users ( [20] and [35]). There is a
current lack of consensus regarding a common set of properties or agreed
metrics that every explainable method should be evaluated against ( [20],
[36], [37], [38], [2], and [39]). Lastly, there is no standardized method for
evaluation of explanation’s quality.

We aim to assess the quality of SAX explanations given by an LLM, therefore
the dimensions selected should reflect desiderata applied in the XAI
literature tailored to our specific needs. It is noteworthy to stress that we
are actually not assessing the quality of explanations given by an XAI method
but rather the explanations generated by an LLM when synthesizing inputs from
different knowledge ingredients about the process. Our focus was mostly on the
intrinsic qualities of explanations and their content rather than features
related to the hedonic and exogenous qualities of the explanations related to
the user’s experience and interaction with the system. We did embed some
attitude aspects in the scale, but we considered these more as background
factors that could moderate the other perceptions about the quality of the
explanations.

We opted to carry out a user study following the common practice (e.g., [37],
[38]¸ [40]) with a scale developed to our needs.

We henceforth detail some of the common dimensions applied in the literature
and assess their level of applicability to our case. Whenever it is not
explicitly indicated that the specific property or measurement is not taken
into account, it is also reflected in our scale.

In [20], the authors relate to the following characteristics of explanations:

  * •

Robustness, meaning that an explanation can withstand small perturbations of
the input that do not change the output prediction. Consequently, robustness
expresses a low sensitivity of the XAI method to changes in inputs. As will be
shown below, we consider robustness in its broader sense of explanation
generalizability, covered by the completeness dimension.

  * •

Fidelity, meaning that the XAI method should preserve the internal concepts
and original behavior of the black box ML model whenever there is a need to
mimic that model.

  * •

Causality, meaning that the XAI method should maintain causal relationships
between inputs and outputs. An ML model is perceived as being more human-like
whenever it provides such causal explanations. Therefore, causality is
fundamental to achieving a human understanding of the ML model.

  * •

Trust, meaning that the outcomes of an XAI method enable gaining confidence
that the ML model acts as intended. For our purposes, we adapted trust to
gaining confidence in the explanations generated by the LLM.

  * •

Fairness, meaning that an explanation must enable humans to ensure unbiased
decisions of the employed ML models. We assume that the inputs generated by
the knowledge ingredients possess fairness, therefore if the resulting
explanation by the LLM lacks fairness it is related to the soundness of the
explanation, which is considered in our scale as described later.

Trust is the main dimension in [41]. The claim is that a trustable technology
may be a critical factor in the success of automation and computer technology
and that “The operator will tend to trust the automation if its algorithms can
be understood and seem capable of achieving the operator’s goals in the
current situation”. Furthermore, [41] states that trust can be achieved by
making the algorithms of the automation simpler or by revealing their
operation more clearly. In our case, this means trust in the tool that
generates the explanations, i.e., the LLM.

In [36] the authors articulate 11 properties of explanations that are
important from an explainee’s point of view. Many of these are grounded in
social science research.

  * •

Soundness - This property measures how truthful an explanation is with respect
to the underlying predictive model. Its goal is to quantify local or global
adherence of the explanation to the underlying model.

  * •

Completeness - For an explanation to be trusted, it also needs to generalize
well beyond the particular case in which it was produced.

  * •

Contextfullness - If there are known issues with an explanation’s
completeness, a user may not trust it. To overcome this and help the user
better understand how an explanation can be generalized, it can be framed in a
context, thereby allowing the user to assess its soundness and completeness.
For example, the user will better understand the limitations of an explanation
if it is accompanied by all the necessary conditions for it to hold, critiques
(i.e., explanation oddities) and its similarities to other cases. As shown in
section 7, the specific context for each domain use case is given, therefore
this dimension is not required. In addition, this metric is also accommodated
in comprehensibility as shown in our definitions below.

  * •

Interactiveness - To improve the overall user experience the explanation
process should be controllable and interactive. In our empirical
experimentation, the interaction was blocked to enable the control for the
manipulations of the inputs. Therefore, this property is not taken into
consideration in our final scale.

  * •

Actionable - When an explanation is provided to help users understand a reason
behind a decision, then the users prefer explanations that they can treat as
guidelines towards the desired outcome. In our scale, this property is
reflected in the comprehensibility property as shown below.

  * •

Chronology - Some aspects of an explanation may have inherent time ordering,
for example, loans taken by a borrower. In such cases, if one of the reasons
given in an explanation has a timeline associated with it, users prefer
explanations that account for more recent events as their cause, i.e.,
proximal causes. This property is inherently represented in the causability
dimension in our scale.

  * •

Coherence - Many users of explainable systems can have prior background
knowledge and beliefs about the matter that is being predicted and explained,
that is, their mental model of the domain. This dimension is tightly related
to and included in comprehensibility measurement in our scale.

  * •

Novelty - Providing users with a mundane or expected explanation should be
avoided. Explanations should contain surprising or abnormal characteristics.
We considered this property as strongly coupled with the perceived curiosity
in our scale.

  * •

Complexity - Given a wide spectrum of skills and background knowledge of
explainees, the complexity of explanations should be tuned to the recipient.
Instead of complexity, we refer to the clarity of an explanation.

  * •

Personalisation - Adjusting an explanation to users requires the
explainability technique to model their background knowledge and mental model.
This property is tightly connected to the comprehensibility property in our
scale.

  * •

Parsimony - Explanations should be selective and succinct enough to avoid
overwhelming the explainee with unnecessary information. Parsimony is
equivalent to compactness in or scale.

Vilone and Longo [35] address the following dimensions:

  * •

Causability - A system that provides a causal explanation on its inferential
process is perceived as more human-like by end users as a consequence of the
innate tendency of human psychology to anthropomorphism. Explanations must
make the causal relationships between the inputs and the model’s predictions
explicit especially when these relationships are not evident to the end users.

  * •

Interpretability - The capacity to provide or bring out the meaning of an
abstract concept.

  * •

Understandability - The capacity to make the model understandable by end
users. We name this dimension comprehensibility.

In [37], the prime metrics used to quantify the interpretability of the
explanations were the response time for understanding, the accuracy of
understanding, and the subjective satisfaction of the users. Both response
time and user satisfaction metrics are not essential to the core quality of
explanations and therefore were excluded from our scale. Accuracy is covered
by comprehensibility and soundness. In [38] and [42] the authors relate to
explainability as the overarching concept, consisting of two main properties:
interpretability and fidelity, both necessary to reach explainability. The
interpretability of an explanation captures how understandable an explanation
is for humans. The fidelity of an explanation expresses how accurately an
explanation describes model behavior, i.e., how faithful an explanation is to
the model. Furthermore, they consider clarity and parsimony as part of
interpretability, and completeness and soundness as part of fidelity.

  * •

Clarity - The explanation is unambiguous.

  * •

Parsimony - The explanation is not too complex, i.e., presented in a compact
form. We name this property compactness.

  * •

Completeness - The explanation describes the entire dynamic of the ML model.

  * •

Soundness - The explanation is correct, i.e., truthful to the model.

Soundness and completeness are also two dimensions of fidelity separately
considered in the qualitative study detailed in [40]. Here the aim was to
investigate how intelligent agents should explain themselves to their users.
The survey paper [2] presents a comprehensive overview of methods proposed in
the current literature for the evaluation of ML explanations. One of the main
concepts is causability which refers to the measurable extent to which an
explanation to a human achieves a specified level of causal understanding,
based on causality, which refers to the relationship between cause and effect
[12]. They also refer to interpretability as having the following properties:
clarity and parsimony or compactness as presented before but also broadness.
By broadness, the authors mean how generally applicable the explanation is. We
use completeness which embeds broadness and therefore supersedes it. Curiosity
is another dimension worthwhile tackling according to [43]. The idea is that
the seeking of an explanation can be driven by curiosity. The assessment of
users’ feelings of curiosity might be informative in the evaluation of XAI
systems due to the following:

  * •

An explanation might overwhelm people with details,

  * •

The XAI system might not allow questions or might make it difficult for the
user to pose questions,

  * •

Explanations might make people feel reticent because of their lack of
knowledge,

  * •

Explanations may include too many open variables and loose ends, and curiosity
decreases when confusion and complexity increase.

Curiosity in our case is translated to the general desire to acquire knowledge
about the reason for a certain condition.

Hoffman et al. [43] state that in the XAI context, a mental model is a user’s
understanding of the AI system and its context. XAI system development
requires a method for eliciting, representing, and analyzing users’ mental
models. The authors distinguish between two separable things: (1) the
intrinsic goodness of explanations and (2) the user’s satisfaction with the
explanations. Explanation satisfaction is defined here as the degree to which
users feel that they sufficiently understand the AI system or process being
explained to them. Mental model goodness includes correctness (soundness in
our set of dimensions), comprehensiveness (completeness in our set of
dimensions), coherence (soundness in our set of dimensions), and usefulness
(excluded as this is not an intrinsic property of an explanation) while
several key attributes of explanation satisfaction include: understandability
(comprehensibility in our set of dimensions), feeling of satisfaction
(excluded as this is not an intrinsic property of an explanation), sufficiency
of detail (included as per the definition of completeness), completeness,
usefulness (not relevant), accuracy (included in soundness), and
trustworthiness.

Holzinger et al. [44] introduce their System Causability Scale to measure the
quality of explanations. Here causability is defined as the extent to which an
explanation of a statement to a user achieves a specified level of causal
understanding with effectiveness, efficiency, and satisfaction in a specified
context of use.

Following our survey literature, we adopted fidelity and interpretability as
two main perceived latent constructs that subsume some of the measurable
dimensions adapted to our user study. As a result, under fidelity, we include
completeness, soundness, and causability dimensions. Interpretability includes
clarity, compactness, and comprehensibility. We relate causability to fidelity
as it refers to the correctness of the underlying model whereas
comprehensibility relates to the level of understanding of the explanation and
therefore is connected to interpretability. Figure 9 depicts the six
dimensions grouped by the constructs of fidelity and interpretability along
with the final questions posed in the survey (explained below). In addition,
for the sake of our study, we also included perceived curiosity (in the
problem) and trust (in the LLM) as potential background factors or moderators.
The idea behind this selection was that these two dimensions cater more to the
interaction formed a-priori to it, rather than reflect an intrinsic quality
about the explanation or the content it embeds. Given the nature of our
controlled experiment with voluntary participation, we were concerned that the
primary effect might be obscured by the participants’ interest in the
explanations provided to them. Therefore, we also measured and later
incorporated ‘trust’ and personal ‘curiosity’ as potential covariates in our
analysis. Figure 10. shows these additional two dimensions.

Overall, our literature survey concluded having two latent constructs, each
operationalized by 3 underlying measurement dimensions (i.e., 6 in total), and
two additional background factors. Table 1 shows our adapted definitions for
all dimensions.

| Dependent
---
Variables
| Measurement
---
Dimensions
Definition
Fidelity |  | How faithful an explanation is to the condition explained.
| Completeness | The extent to which an explanation describes all the information relevant to the condition explained.
| Soundness | How truthful the explanation is with respect to the condition explained.
| Causability | The explanation provides the reasons for the occurrence of the condition explained.
Interpretability |  | How understandable an explanation is for humans.
| Clarity | The extent the explanation is unambiguous.
| Compactness | The extent the explanation is presented in a simple and compact form.
| Comprehensibility | The explanation aligns with my understanding of the problem and how to react to it.
| Background
---
Factors
|
Curiosity |  | The general desire to acquire knowledge about the reason for a certain condition.
Trust |  | An attitude toward the LLM that affects reliance on its explanations.

Table 1: Definitions for all experimental constructs, measurement dimensions,
and background factors

We next populated a preliminary set of questions based on existing scales for
each of the 8 measurement dimensions. Before its dissemination, we pursued a
card sorting procedure ( [45, 46, 47]) in order to ensure the validity and
reliability of our scale. This was subsequently complemented by a reliability
analysis of the responses that we conducted after the survey.

Card sorting is an iterative scale development technique in which a panel of
judges is asked in several rounds to sort a set of scale items into separate
categories, based on similarities and differences among them. This technique
was previously used in ( [45, 46]) to assess the coverage of an intended
domain of constructs. Later it was further refined [47] to generally account
for scale reliability (i.e., content and construct validity).

Figure 9: (Perceived) explanation quality – high-level (latent) constructs –
as a composition of perceived characteristics Figure 10: [Perceived]
moderating factors (covariates)

Prior to conducting the first card sorting round, we populated five items for
each category or measurement dimension. We used
KardSort666https://kardsort.com/ as the tool for card sorting.

The set of all 40 items (5 items for each of the 8 dimensions or categories)
was jointly considered as the input pool for the first card sorting round. To
eliminate ambiguous items, we conducted three separate card sorting rounds
with five independent judges in each round. As noted in [47], this was done to
increase dimension validity and confidence in the developed scale. The judges
at each round were presented with all the items and were asked to relate each
item to the best-fitted category.

We used two indicators to monitor validity and the convergence of the sorting
procedure: (1) Inter-rater agreement between the judges for each dimension in
each round, and (2) Hit ratio (‘convergent’ validity) - item-wise portion of
true item classifications within its ‘target’ dimension (see Table 2).

After each round, we inspected each item’s inter-rater and hit ratio values.
The inter-rater indicator reflected the cumulative (so far in the rounds)
agreement among the judges on a specific item. Items with a hit ratio less
than the majority of card assignments were rephrased to better reflect their
category. As aforementioned, we performed this adjustment after each round.
After the third time, we selected the top 3 items that best fit their
category, concluding with a total of 24 items. Table 2 summarizes the overall
procedure whereas Table 3 presents the hit-ratio loading by the end of the
procedure. For all items in all dimensions except one, we achieved a hit ratio
above 50%. We acknowledge a convergence drift in one of the items between
comprehensibility and clarity. However, the wording of this item had a perfect
alignment with the definition of comprehensibility, so we decided to keep it.
Later analysis of the loading scores reinforced this decision.

The resulting three items for each dimension grouped by construct are shown in
Figure 9 and Figure 10.

Measurement Dimension |  | Round 1
---
(5 judges)
| Round 2 cumulative
---
(+5 judges)
| Round 3 cumulative
---
(+5 judges)
| Final
---
(15 judges)
| #
---
items
| inter-
---
rater
agree
| hit
---
ratio
| #
---
items
| #
---
reworded
| inter-
---
rater
agree
| hit
---
ratio
| #
---
items
| #
---
reworded
| inter-
---
rater
agree
| hit
---
ratio
| #
---
items
| #
---
reworded
| inter-
---
rater
agree
| hit
---
ratio
Completeness | 5 | 0.58 | 0.72 | 5 | 2 | 0.59 | 0.72 | 5 | 1 | 0.45 | 0.61 | 3 | 1 | 0.57 | 0.76
Soundness | 5 | 0.46 | 0.6 | 5 | 3 | 0.59 | 0.7 | 5 | - | 0.42 | 0.59 | 3 | - | 0.48 | 0.68
Causability | 5 | 0.28 | 0.44 | 5 | 3 | 0.38 | 0.6 | 5 | 1 | 0.49 | 0.64 | 3 | 1 | 0.68 | 0.81
Clarity | 5 | 0.4 | 0.44 | 5 | 3 | 0.43 | 0.64 | 5 | 1 | 0.41 | 0.62 | 3 | 1 | 0.49 | 0.69
Compactness | 5 | 0.42 | 0.68 | 5 | - | 0.4 | 0.6 | 5 | 2 | 0.55 | 0.69 | 3 | 1 | 0.70 | 0.82
Comprehensibility | 5 | 0.16 | 0.32 | 5 | 4 | 0.4 | 0.38 | 5 | 2 | 0.38 | 0.40 | 3 | - | 0.30 | 0.47
Curiosity | 5 | 0.78 | 0.88 | 5 | - | 0.76 | 0.86 | 5 | - | 0.79 | 0.88 | 3 | - | 0.91 | 0.96
Trust | 5 | 0.72 | 0.84 | 5 | - | 0.76 | 0.86 | 5 | - | 0.76 | 0.85 | 3 | - | 0.91 | 0.96

Table 2: Card sorting results: cumulative inter-rater agreement and hit ratio

| Causality | Clarity | Compactness | Completeness | Comprehensibility | Curiosity | Soundness | Trust
---|---|---|---|---|---|---|---|---
COM-Q1 | 20% |  |  | 80% |  |  |  |
COM-Q2 |  |  |  | 80% | 13% |  |  | 7%
COM-Q3 |  |  | 7% | 67% | 20% |  |  | 7%
SND-Q1 |  | 7% |  |  | 7% |  | 73% | 13%
SND-Q2 |  | 7% |  |  |  |  | 60% | 33%
SND-Q3 |  | 10% |  | 10% | 10% |  | 70% |
CAS-Q1 | 70% |  |  | 10% |  |  | 20% |
CAS-Q2 | 100% |  |  |  |  |  |  |
CAS-Q3 | 73% | 20% |  |  |  |  | 7% |
CLR-Q1 |  | 67% | 7% |  | 20% |  | 7% |
CLR-Q2 | 20% | 60% |  |  |  |  | 20% |
CLR-Q3 |  | 80% | 10% |  | 10% |  |  |
CMP-Q1 |  |  | 100% |  |  |  |  |
CMP-Q2 |  | 7% | 67% | 7% |  |  | 20% |
CMP-Q3 |  |  | 80% | 13% |  |  | 7% |
CPR-Q1 | 10% | 10% |  |  | 50% |  | 10% | 20%
CPR-Q2 |  | 40% | 30% |  | 30% |  |  |
CPR-Q3 |  | 20% |  |  | 60% |  | 13% | 7%
CUR-Q1 |  |  |  |  |  | 100% |  |
CUR-Q2 |  |  |  |  |  | 100% |  |
CUR-Q3 |  | 7% |  | 7% |  | 87% |  |
TRU-Q1 |  |  |  |  |  |  |  | 100%
TRU-Q2 |  | 7% |  | 7% |  |  |  | 87%
TRU-Q3 |  |  |  |  |  |  |  | 100%

Table 3: Item hit-ratio after the sorting procedure

###  6.2 User Study

In our experiment, we primarily focused on investigating how different types
of explanatory knowledge about business processes could be incorporated into
the LLM prompt preceding the interaction with the LLMs when seeking
explanations about process execution outcomes. As an independent variable in
our experimental design, we employed three between-group manipulations based
on distinct knowledge types: process, XAI (feature importance), and causal.
All input combinations are given in Table 4.

We consistently used business process knowledge as a baseline in all
manipulations. The first manipulation involved augmenting this baseline with
XAI knowledge related to the factors influencing the specific process
condition being explained. This was compared to using only XAI knowledge as
input. In the second manipulation, instead of XAI knowledge, we explored the
impact of adding causal knowledge about the execution dependencies among
process activities that could lead to the condition being explained. This was
compared with using only the baseline process knowledge as an input. Lastly,
the third manipulation compared the effect of combining process and XAI
knowledge with the effect of integrating all three knowledge types as input
for the LLM.

Problem Domain | Group 1 | Group 2
---|---|---
Pizza delivery | Process and Feature-Importance (XAI) | Feature-Importance (XAI)
Parking fines | Process and Feature-Importance (XAI) | Process, Feature-Importance (XAI), and Causal
Loan approval | Process and Causal | Process

Table 4: Three manipulations of input knowledge type

Respective to the various manipulations depicted in Table 4, each of the above
hypotheses was further instantiated for more concrete testing as follows:

######  Subhypothesis 1 (H1.1).

Explanations generated by LLMs informed by knowledge about business processes
and feature importance (XAI) will be perceived as having higher fidelity
compared to explanations informed only by feature importance knowledge (XAI).

######  Subhypothesis 2 (H1.2).

Explanations generated by LLMs informed by knowledge about business processes,
feature importance, and causal execution dependencies will be perceived as
having higher fidelity compared to explanations informed only by business
processes and feature importance knowledge.

######  Subhypothesis 3 (H1.3).

Explanations generated by LLMs informed by knowledge about business processes
and causal execution dependencies will be perceived as having higher fidelity
compared to explanations informed only by business process knowledge.

######  Subhypothesis 4 (H2.1).

Explanations generated by LLMs informed by knowledge about business processes
and feature importance (XAI) will be perceived as having higher
interpretability compared to explanations informed only by feature importance
knowledge (XAI).

######  Subhypothesis 5 (H2.2).

Explanations generated by LLMs informed by knowledge about business processes,
feature importance, and causal execution dependencies will be perceived as
having higher interpretability compared to explanations informed only by
business processes and feature importance knowledge.

######  Subhypothesis 6 (H2.3).

Explanations generated by LLMs informed by knowledge about business processes
and causal execution dependencies will be perceived as having higher
interpretability compared to explanations informed only by business process
knowledge.

###  6.3 Domains

Each manipulation was implemented within a distinct problem domain to derive
explanations for specific conditions. In particular, the first manipulation,
hypothesized in H1.1 and H2.1, was developed within the ‘pizza delivery’
domain, focusing on creating explanations for delays in pizza delivery. The
second manipulation, outlined in H1.2 and H2.2, is related to the ‘parking
fines’ domain, targeting the development of explanations for the lateness in
fine processing. Finally, the third manipulation, as hypothesized in H1.3 and
H2.3, was situated in the ‘loan approval’ domain, with a focus on elucidating
the potential reduction in loan approval times.

Corresponding to each domain, we generated the knowledge ingredients about the
process model, the causal model, and the XAI feature importance using the
SAX4BPM library services. This was applied to two synthesized process data in
the pizza delivery and parking fines domains, and to the loan approval domain
using an open
dataset777https://data.4tu.nl/articles/dataset/BPI_Challenge_2017/12696884.

##  7 Experimental Design

We conducted a 3X2 (knowledge type, perceived explanation quality) mixed
experimental design to evaluate our hypotheses. This design was applied across
three problem domains: pizza delivery, parking fines, and loan approval. In
the experimental setup, participants were randomly assigned to one of two
distinct groups. They completed an online questionnaire that included a
consent form, basic demographics, and a series of 3 sets of [1-7 Likert]
rating scales relevant to the aforementioned problem domains. The Likert
method [48] is widely used as a standard psychometric scale to measure human
responses. Each set comprised 24 items (see Figures 9 and 10), phrased based
on the methodological scale development process described in section 6.1. This
included 9 items each for fidelity and interpretability (3 for each
dimension), plus 6 additional items (3 each for curiosity and trust).
Preceding each set of rating scales was a description of a specific business
process in one domain, a question addressing a particular process condition
requiring an explanation, a “ground truth” explanation provided by the
researchers, and an explanation generated by the LLM (in our case ChatGPT 4)
in response to the same question. The ground truth serves as a baseline to
which we can make the comparisons relative. Participants were instructed to
rate the LLM-generated explanation, assessing its quality against the ground
truth. Each LLM was pre-prompted descriptive text blending the manipulated
knowledge elements, as detailed in Figure 11. The two questionnaires can be
accessed at: https://github.com/IBM/SAX/tree/main/KDE-SI-2024/survey.

In the pizza-delivery case, the explanation concerned possible reasons for
pizza delivery lateness, and the explanation generation by the LLM varied
between prompt inputs that included both process and XAI knowledge (group 1)
and prompt inputs that included only XAI knowledge (group 2). In the parking-
fines case, the explanation addressed possible reasons for the extensive
processing time of parking fines. Here, the LLM-generated explanation differed
between prompt inputs that included process, causal, and XAI knowledge (group
1) and prompt inputs that included process and XAI knowledge (group 2).
Lastly, in the loan-approval case, the explanation focused on reasons for
delayed loan offers. Accordingly, the explanation generated by the LLM was
modified between prompt inputs that included both process and causal knowledge
(group 1) and prompt inputs that included only process knowledge (group 2).

Figure 11: Experiment model

##  8 Results

###  8.1 Data Preparation

Prior to running the statistical analysis, we explored the data using BOXPLOTs
for the two target constructs across all three domains in order to determine
extreme responses. There were only 2 responses out of the 49 (1 for fidelity
in the parking-fines domain, and 1 for interoperability in the loan-approval)
that were mild outliers, scoring more than 1.5 times the inter-quartile
range888The inter-quartile range (IQR) contains the middle 50 percent of the
distribution. For normally distributed data, a range that is 1.5 times the IQR
covers $\sim 99.3\%$ of the distribution population.. These two responses were
not consistent across the three domains, hence we did not remove them from the
analysis. Running a similar BOXPLOT analysis for general homogeneity of
variance across all individual respondents yielded two respondents who were
identified as having mildly low variance in their responses overall. However,
we executed our subsequent analysis for the main effects twice, with and
without the outliers, concluding that outlier elimination is negligible and
has no effect on the significance of our results. Hence, the final results
reporting utilized 47 responses across all three domains.

Measurement dimension | Cronbach’s Alpha | Construct | Cronbach’s alpha
---|---|---|---
Completeness | 0.91 |  |
Soundness | 0.63 |  |
Causability | 0.69 | Fidelity | 0.81
Clarity | 0.81 |  |
Compactness | 0.49 (0.516) |  |
Comprehensibility | 0.64 | Interpretebility | 0.80
Curiosity | 0.92 |  |
Trust | 0.90 |  |

Table 5: Reliability scores

###  8.2 Scale Reliability

After exploring outliers, we assessed the scale reliability for all
measurement dimensions. The reliability scores, item-wise (three per
dimension), for each measurement dimension and the latent constructs, are
presented in Table 5. Cronbach’s alpha values ranging from 0.6 to 0.8 are
conventionally acceptable [49]. As shown in the table, most measurement
dimensions had Cronbach’s alpha scores above 0.6 (ranging from 0.63 to 0.92),
indicating an overall internal consistency of 0.81 for fidelity and 0.80 for
interoperability. This excludes the ‘compactness’ dimension, which scored
below the recommended threshold with an alpha of 0.49. Eliminating the third
item from this dimension slightly improved its internal consistency to 0.516.
To further validate these findings, we conducted an exploratory factor
analysis for all measurement dimensions, revealing a single component for all
dimensions with factor loading scores as shown in Table 6. The results were
consistent with our reliability scores, with most loading scores surpassing
the recommended loading threshold (¿ 0.4, [50]) and explained variance
exceeding 50%. Notably, the loadings for the compactness dimension,
particularly its third item, were marginally above the threshold (0.584).
Therefore, we repeated the analyses for the main effects both with and without
this item. Its removal did not significantly alter the results, eventually
leading to our decision to retain it.

Measurement dimension | Item 1 | Item 2 | Item 3 | Variance explained
---|---|---|---|---
Completeness | 0.919 | 0.924 | 0.914 | 84.42%
Soundness | 0.837 | 0.606 | 0.821 | 58.05%
Causability | 0.829 | 0.804 | 0.718 | 61.61%
Clarity | 0.846 | 0.863 | 0.841 | 72.24%
Compactness | 0.714 | 0.804 | 0.584 | 49.88%
Comprehensibility | 0.803 | 0.667 | 0.835 | 59.53%
Curiosity | 0.930 | 0.955 | 0.894 | 85.88%
Trust | 0.934 | 0.869 | 0.928 | 82.94%

Table 6: Factor loadings

Problem domain | Construct | Measurement dimension | Group 1 | Group 2
---|---|---|---|---
M | SD | n | M | SD | n
Pizza delivery | Fidelity | Completeness | 4.48 | 1.58 | 24 | 3.73 | 1.92 | 23
Soundness | 4.31 | 0.98 | 24 | 3.59 | 1.13 | 23
Causability | 4.16 | 1.39 | 24 | 3.65 | 1.08 | 23
Interpretability | Clarity | 4.37 | 1.20 | 24 | 4.69 | 1.42 | 23
Compactness | 3.50 | 1.13 | 24 | 4.10 | 1.19 | 23
Comprehensibility | 4.08 | 1.29 | 24 | 3.71 | 1.26 | 23
Parking fines | Fidelity | Completeness | 2.79 | 1.56 | 24 | 3.65 | 1.66 | 23
Soundness | 3.18 | 0.96 | 24 | 3.62 | 1.10 | 23
Causability | 3.72 | 1.38 | 24 | 3.88 | 1.12 | 23
Interpretability | Clarity | 4.25 | 1.68 | 24 | 2.73 | 1.32 | 23
Compactness | 3.23 | 1.12 | 24 | 2.42 | 0.95 | 23
Comprehensibility | 3.41 | 1.20 | 24 | 2.53 | 0.91 | 23
Loan approval | Fidelity | Completeness | 3.66 | 1.54 | 24 | 3.63 | 1.18 | 23
Soundness | 4.31 | 0.98 | 24 | 3.59 | 1.13 | 23
Causability | 4.16 | 1.39 | 24 | 3.65 | 1.08 | 23
Interpretability | Clarity | 3.31 | 1.62 | 24 | 3.11 | 1.26 | 23
Compactness | 3.18 | 1.02 | 24 | 3.31 | 1.02 | 23
Comprehensibility | 3.48 | 1.28 | 24 | 3.31 | 1.14 | 23

Table 7: Descriptives for all measurement dimensions across groups

###  8.3 Effect on perceived explanation quality

Descriptive statistics for all underlying measurement dimensions are detailed
in Table 7. Domain-wise analysis of the effects was carried out.

####  8.3.1 Pizza-delivery domain

Results for the pizza-delivery domain are detailed in Table 8. An Analysis of
Variance (ANOVA) was conducted first to examine the effects of knowledge
manipulation (between-group differences) on two dependent variables: fidelity
and interpretability. The analysis revealed that for the dependent variable of
fidelity, there was no significant effect, F(1,46) = 2.42, p = .12, $\eta^{2}$
= .051. This suggests that knowledge type alone does not have a substantial
effect on fidelity. Also for the dependent variable of interpretability, the
effect was not statistically significant, F(1, 46) = 0.42, p = .51, $\eta^{2}$
= .009, suggesting that the type of knowledge manipulation that was conducted
has no significant effect on interpretability when not controlling for any
covariate.

An Analysis of Covariance (ANCOVA) was conducted next to evaluate the effects
of knowledge type manipulation on the two dependent variables: fidelity and
interpretability while controlling for the covariates trust and curiosity. The
analysis revealed that for the dependent variable fidelity, the adjusted model
was statistically significant, F(3, 44) = 7.60, p ¡ .001, $\eta^{2}$ = .347,
indicating a substantial effect of the controlled manipulation. Similarly,
regarding the dependent variable interpretability, the adjusted model was also
significant, F(3, 44) = 5.08, p = .004, $\eta^{2}$ = .262. Thus, when
incorporating trust and curiosity as covariates, the manipulation of knowledge
type as was realized in the pizza-delivery domain, significantly affected the
perceptions of fidelity and interoperability.

Running the ANCOVA analysis also revealed that trust has a significant direct
effect on both dependent variables: on fidelity, F(1,46) = 17.79, p ¡ .001,
$\eta^{2}$ = .29, and on interpretability, F(1,46) = 14.67, p ¡ .001,
$\eta^{2}$ = .25.

Therefore, in relation to the corresponding hypotheses (H1.1; H2.1), the
results support the expectation that adding process-related knowledge to XAI
knowledge as an input to the LLM can potentially yield a significant effect on
both perceptions of fidelity and interoperability. However, this effect is
moderated by the degree to which the user is curious about the condition
explored and trusts the LLM tool. Stretched beyond our original hypotheses,
the results also identified trust as a background factor. This factor not only
moderates the effect of the manipulated input to the LLM but could also
directly mask this effect.

Construct | ANOVA | ANCOVA
---|---|---
Group 1 | Group 2 | df | F | Sig. | $\eta^{2}$ | Group 1 | Group 2 | df | F | Sig. | $\eta^{2}$
M | SE | M | SE | M | SE | M | SE
Fidelity | 4.44 | 0.26 | 3.86 | 0.26 | (1,46) | 2.42 | 0.12 | 0.051 | 4.46 | 0.22 | 3.84 | 0.22 | (3,44) | 7.60 | <.001* | 0.347
Interpretability | 3.98 | 0.19 | 4.16 | 0.20 | (1,46) | 0.42 | 0.51 | 0.009 | 4.01 | 0.17 | 4.13 | 0.17 | (3,44) | 5.08 | .004* | 0.262

Table 8: ANOVA and ANCOVA (i.e., including curiosity and trust as covariates)
results for the pizza delivery case. When controlling for curiosity and trust,
both effects on fidelity and interoperability were significant.

####  8.3.2 Parking fines domain

Results for the parking-fines domain are detailed in Table 9. For the parking-
fine case, an Analysis of Variance (ANOVA) to examine the effects of knowledge
manipulation on two dependent variables fidelity and interoperability,
revealed that for the dependent variable of fidelity, there was no significant
effect, F(1,46) = 2.71, p = .107, $\eta^{2}$ = .057. However, for the
dependent variable of interpretability, the effect was statistically
significant, F(1, 46) = 15.30, p = ¡.001, $\eta^{2}$ = .254, suggesting that
the particular type of knowledge manipulation that was conducted in this case
has a significant effect on interpretability, even when not controlling for
any covariate.

An Analysis of Covariance (ANCOVA) was conducted next to evaluate the effects
of knowledge manipulation on the two dependent variables while controlling for
the covariates trust and curiosity. The analysis revealed that for the
dependent variable fidelity, the adjusted model was statistically significant,
F(3, 44) = 3.67, p = .019, $\eta^{2}$ = .204, indicating a substantial effect
of the manipulation. Similarly, regarding the dependent variable
interpretability, the adjusted model was also significant, F(3, 44) = 9.84, p
= ¡.001, $\eta^{2}$ = .407. Thus, when incorporating trust and curiosity as
covariates, the manipulation of knowledge type as was realized in the parking-
fines domain, significantly affected the perceptions of fidelity and
interoperability while controlling for trust and curiosity. For
interpretability, the effect size is 15% stronger than when not controlling
for the covariates.

Running the ANCOVA analysis also revealed that curiosity has a significant
direct effect on the perception of interpretability, F(1,46) = 6.43, p =
0.015, $\eta^{2}$ = 0.13, and as in the pizza domain, trust has a significant
effect on both fidelity, F(1,46) = 5.71, p = .021, $\eta^{2}$ = .11, and on
interpretability, F(1,46) = 7.00, p = .011, $\eta^{2}$ = .14.

Concerning the corresponding hypotheses (H1.2; H2.2), the results support the
expectation that adding causal-related knowledge to process and XAI knowledge
as an input to the LLM can potentially yield a significant effect on both
perceptions of fidelity and interoperability. In the case of interpretability,
the effect is present regardless of any particular moderation, whereas in the
case of fidelity, the effect is moderated by the user’s curiosity about the
explored condition and trust in the LLM tool. Beyond our original hypotheses,
it is also observed that curiosity has a direct effect on interpretability,
and trust has a direct effect on both perceptions, interpretability, and
fidelity.

Construct | ANOVA | ANCOVA
---|---|---
Group 1 | Group 2 | df | F | Sig. | $\eta^{2}$ | Group 1 | Group 2 | df | F | Sig. | $\eta^{2}$
M | SE | M | SE | M | SE | M | SE
Fidelity | 3.23 | 0.20 | 3.71 | 0.21 | (1,46) | 2.71 | 0.107 | 0.057 | 3.31 | 0.19 | 3.62 | 0.20 | (3,44) | 3.67 | 0.019* | 0.204
Interpretability | 3.63 | 0l19 | 2.56 | 0.19 | (1,46) | 15.30 | <.001* | 0.254 | 3.73 | 0.17 | 2.45 | 0.18 | (3,44) | 9.84 | <.001* | 0.407

Table 9: ANOVA and ANCOVA (i.e., including curiosity and trust as covariates)
results for the parking fines case. When controlling for curiosity and trust,
both effects on fidelity and interoperability were significant. The effect on
interpretability was also significant regardless of the covariates.

####  8.3.3 Loan approval domain

Results for the loan-approval domain are detailed in Table 10. For the loan-
approval case, an Analysis of Variance (ANOVA) revealed that for the dependent
variable of fidelity, there was no significant effect, F(1,46) = 1.97, p =
.16, $\eta^{2}$ = .0542. Likewise, for the dependent variable of
interpretability, the effect was statistically insignificant, F(1, 46) =
0.058, p = .81, $\eta^{2}$ = .001, suggesting that the particular type of
knowledge manipulation that was conducted in this case has no significant
effect on either fidelity or interpretability when not controlling for any
covariate.

An Analysis of Covariance (ANCOVA) was conducted next to evaluate the effects
of knowledge manipulation on the two dependent variables while controlling for
the covariates trust and curiosity. The analysis revealed that for the
dependent variable fidelity, the adjusted model was statistically significant,
F(3, 44) = 19.16, p ¡ .001, $\eta^{2}$ = .572, indicating a substantial effect
of the manipulation. Similarly, regarding the dependent variable
interpretability, the adjusted model was also significant, F(3, 44) = 10.56, p
= ¡.001, $\eta^{2}$ = .424, also showing a substantial effect. Thus, when
incorporating trust and curiosity as covariates, the manipulation of knowledge
type as was realized in the loan-approval domain, significantly affected the
perceptions of fidelity and interoperability while controlling for trust and
curiosity.

Running the ANCOVA analysis also revealed that curiosity has a significant
direct effect on the perception of fidelity, F(1,46) = 6.67, p = 0.013,
$\eta^{2}$ = 0.13, and as in the two other domains, trust has a significant
effect on both fidelity, F(1,46) = 51.40, p ¡ .001, $\eta^{2}$ = .54, and on
interpretability, F(1,46) = 31.58, p ¡ .001, $\eta^{2}$ = .42.

Revisiting the related hypotheses (H1.3; H2.3), the results support the
expectation that adding causal-related knowledge to process knowledge as an
input to the LLM can potentially yield a significant effect on both
perceptions of fidelity and interoperability. The effect on both is moderated
by the user’s curiosity about the explored condition and trust in the LLM
tool. Beyond our original hypotheses, it is also observed that curiosity has a
direct effect on fidelity, and that trust has a direct effect on both
perceptions, interpretability and fidelity.

Construct | ANOVA | ANCOVA
---|---|---
Group 1 | Group 2 | df | F | Sig. | $\eta^{2}$ | Group 1 | Group 2 | df | F | Sig. | $\eta^{2}$
M | SE | M | SE | M | SE | M | SE
Fidelity | 4.05 | 0.21 | 3.62 | 0.21 | (1,46) | 1.97 | 0.16 | 0.042 | 4.02 | 0.14 | 3.65 | 0.14 | (3,44) | 19.16 | <.001* | 0.572
Interpretability | 3.31 | 0.22 | 3.25 | 0.23 | (1,46) | 0.058 | 0.81 | 0.001 | 3.27 | 0.17 | 3.31 | 0.18 | (3,44) | 10.56 | <.001* | 0.424

Table 10: ANOVA and ANCOVA (i.e., including curiosity and trust as covariates)
results for the loan approval case. When controlling for curiosity and trust,
both effects on fidelity and interoperability were significant.

###  8.4 Direction of the effect

Careful observation of the study results revealed that, regardless of the
significance of the effects, an important outcome that stands out across all
three domains — one which we did not fully anticipate in our hypotheses — is
related to the direction of the effects on fidelity and interoperability.
While our original thought was that informing the LLM prompt with additional
knowledge, whether process-related (as in the pizza domain) or causal-related
(as in the two other domains), would lead to a consistent effect direction on
both dependent constructs, potentially improving both. However, this was, in
fact, not the result. The actual effect directions, as illustrated in Figure
12, not only demonstrate an interaction between the two constructs (supported
by prior literature [38]) but also show that the addition of either type of
knowledge ingredient (process or causal) works towards improving the perceived
fidelity of the explanation while compensating for the perceived
interpretability of the explanation. Concerning the size of the effect, in the
pizza delivery and loan approval domains, the improvement in fidelity was
larger than the loss in interpretability. However, in the parking fines
domain, the opposite was true. Hence, we highlight this particular caveat to
be watched for in specific problem domains where one might attempt a similar
type of intervention to ‘improve’ the perceived quality of generated
explanations.

Figure 12: Constructs interaction

##  9 Related Work

This study builds on our previously published extended abstract [51] where the
original idea of synthesizing different knowledge ingredients for improved
business process explanations given by LLMs was conceived. In this work, we
further expand from these roots in various directions. First, the core idea is
formalized in a set of hypotheses. Second, a library of services has been
developed to automate the concept and integration with LLMs. Third, to
evaluate the hypotheses, we rigorously developed a corresponding scale to
assess the effectiveness of the altered inputs and employed it to test the
effectiveness of using the services with ChatGPT, and the reliability of the
scale itself. This effort is followed by a quantitative analysis of user
responses and derived conclusions.

In the following, we discuss how our study relates to and contributes to the
literature on the intersection between LLMs and business processes for the
sake of business process explainability.

Overall, the use of LLMs for business processes has been researched only very
recently, enabled only by the availability and accessibility of the GPT-based
foundation models. In a broader perspective, business process characteristics
present a unique opportunity for the development of a new class of foundation
models that utilize the timely sequencing of processes for various process-
related tasks such as activity prediction, process optimization, and decision
making (e.g., [52, 53]). Vigdof et al. [16] explore the potential and broader
application for integrating LLMs along all stages of the BPM life-cycle and
propose further broad research directions for the use of LLMs in BPM. While
covering process mining, optimization, and decision-making, the paper does not
conduct experiments. Grohs et al. [54] demonstrate in several experiments that
advanced LLMs such as GPT-4 are capable of transforming natural language
descriptions of smaller processes into declarative and imperative process
models and can aid in detecting automate-able tasks from natural language
process descriptions. Berti et al. [55] conducted several experiments to
demonstrate the ability of GPT-4 to generate SQL queries over event data for
answering analysis questions stated in natural language. None of the above
papers delves into the detailed use of LLMs for explainability in business
processes nor examines the consequences of such employment, which is the focus
of our study.

Prompt engineering for LLMs that are employed towards various tasks in BPM is
also recently surveyed in [53]. While zero-short or few-short learning is
reported to improve the performance of the LLMs, other techniques such as
modifying its structure, and even subtle reordering of the various input
elements, may have destructive effects on the performance of the LLM. These
experiences have recently also resulted in a public repository of prompts
[56]. To mitigate such destructive consequences, Jessen et al. [57] propose a
fully automated, structured process for generating prompts for translating
natural-language analysis questions into SQL queries over event data. Their
approach relies on (iteratively) enriching the prompt with additional
information from different perspectives, which then results in a more stable
quality of answers (correct SQL queries). While our approach for prompt
engineering bears similarities as we also integrate information from multiple
perspectives, our focus is on altering the content of the input (semantics) to
the LLM rather than reshuffling of its structure (syntax). We pursue this to
address business process explainability and leverage information in a
knowledge graph to automatically generate the prompt.

A general assumption made by existing studies on LLMs for BPM is that the
textual output generated by an LLM is inherently understandable and suitable
for the user. While some prior studies do analyze fidelity [57], this study is
the first to also investigate interpretability of LLM output by users and
other background factors that influence these. Notably, we establish that
users perceive a trade-off between fidelity and interoperability.

Several works studied the application of explainability techniques for process
prediction and prescription. For example, Shapley Values [58] have been used
to explain which features influence the prediction made by a ML prediction
model learned from an event log [59]. Shapley values are also used as a
feature for generating explanations for the recommendations produced by a
prescriptive model [60]. Several other works also explore (combinations of)
other “explainability” techniques for process prediction and recommendation,
e.g., [61, 62, 63, 64]. Generally, this line of research focuses on explaining
a predictive or prescriptive model of a process, i.e., it aims at explaining
how an approximation of the observed event log relates input to output
features. We extend the conventional applicability of XAI techniques to
business process explainability to also include other perspectives such as
causal and process views which are inferred directly from the process event
log.

Further, all studies have in common that they generate explanations in the
form of charts that visualize the influence of input on output features. A
user study confirmed that the generated visualizations “are generally
comprehensible to correctly carry out analysis’ tasks” for prediction [65].
The study also observed that accuracy in understanding influences on outcomes
and the overall quality of the approach (esp. on difficult tasks) could
improve when analysts are better aided in understanding how different
influencing factors integrate towards the final effect. Another user study
[66] showed that also experts undergo a learning curve when interpreting the
plots and require additional explanations for answering more difficult tasks.
Our study contributes to integrating various features for explanations and
complements these prior studies by analyzing the factors that influence the
understandability of explanations in natural language.

A novel component in this work is the incorporation of the causal execution
view as a basis for generating high-fidelity explanations. With regards to the
applicability of causal inference and discovery to business processes,
recently, Dasht Bozorgi et al. demonstrated how to apply statistical causality
analysis techniques to identify cause-effect relations in business processes
from event logs [67]. These causality analysis techniques can be used to
improve control policies using reinforcement learning [68] or to improve
prescriptive process monitoring by estimating the causal effect of an
intervention on a performance metric [69]. While Dasht Bozorgi et al. focus on
identifying individual rules of cause-effect of an action utilizing the
Conditional Average Treatment Effect (CATE) technique [70], our work employs a
novel approach that utilizes the timing of the activities for the discovery of
causal execution interdependency between process activities, and use this view
as an additional input to the LLM.

##  10 Conclusions and Future Work

In regards to the hypotheses, our results clearly indicate that providing the
LLM with additional knowledge about causal execution dependencies, or even
just the temporal sequencing embedded in the process knowledge, can enhance
the perceived fidelity. However, this effect may diminish entirely when the
user has low trust in the LLM. It may also lessen when the user has limited
curiosity about the problem for which the explanation is sought. One should
also exercise caution in overly emphasizing the potential benefit of increased
fidelity, as it may come at the expense of reduced perceived interpretability.
Further examination of the trade-off between fidelity and interoperability,
and the extent to which an explanation remains “sufficient” may be explored in
the future.

For pragmatic reasons, we limited the number of manipulations. Therefore, we
focused on those closely aligned with the novelty of incorporating the causal
knowledge ingredient with the process one for explainability. We also explored
how the more conventional approach of using XAI in itself is influenced when
integrated with the process perspective. Additionally, we investigated whether
incorporating the causal view enhances the perceived quality when combined
with the process component. We acknowledge that exploring the effects of
augmenting process knowledge with XAI, as opposed to using only process
knowledge, may still be an interesting direction to pursue. In general, other
process-related perspectives could also be incorporated as additional
knowledge ingredients.

We merged the manipulation of knowledge components with that of the domain. We
have made all experimental instruments available to facilitate future work in
replicating our methods and extending the examination to other problem domains
and communities.

The generation of knowledge-informed explanations could be fully automated
with the use of the tooling framework developed in this paper. Additional
‘tweaking’ of the prompt could be employed to mitigate for the potential
drawback with interpretability, e.g., by embedding prompt instructions to the
LLM to also try to summarise the result as much as possible without
compromising fidelity. Statements such as “make the explanation not longer
than 2-3 sentences without losing on its core detail, correctness, and
accuracy.” may be attempted. We already have the result of such a feature
illustrated in figure 8. However, we did not include it in the experiment to
keep the output of the LLM conform to its default hyper-parameter
configuration, keeping the configuration consistent across all manipulations
(e.g., temperature, top p, max length).

Future work might include the templating of the prompts as a means to fully
automate the blending of contemporary techniques for prompt engineering (e.g.,
sequencing and restructuring) and our content manipulation to combine all as
an input to the LLM. Our use of the LLM in this work relied on a one-off, out-
of-the-box execution of ChatGPT-4 with no further model tuning (i.e., few-shot
learning) to train the language model to interpret the input narratives, with
a focus on explanations that arise from causal, process, and XAI knowledge
synthesize. Next steps could leverage on our developed scale to methodological
gather scoring for process explanations and subsequently use these scores to
prioritize among a “repository” of explanations to fine-tune the LLM model.

## References

  * [1] M. Dumas, F. Fournier, L. Limonad, A. Marrella, M. Montali, J.-R. Rehse, R. Accorsi, D. Calvanese, G. De Giacomo, D. Fahland, A. Gal, M. La Rosa, H. Völzer, I. Weber, AI-augmented Business Process Management Systems: A Research Manifesto, ACM Transactions on Management Information Systems 14 (1) (2023).  doi:10.1145/3576047.
  * [2] J. Zhou, A. H. Gandomi, F. Chen, A. Holzinger, Evaluating the quality of machine learning explanations: A survey on methods and metrics (2021).  doi:10.3390/electronics10050593.
  * [3] A. Chandrasekaran, L. Ramos, Hype Cycle for Generative AI, Tech. rep., Gartner (9 2023).
  * [4] J. Gao, X. Ding, B. Qin, T. Liu, Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation, in: H. Bouamor, J. Pino, K. Bali (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore, 2023, pp. 11111–11126.  doi:10.18653/v1/2023.findings-emnlp.743.
URL https://aclanthology.org/2023.findings-emnlp.743

  * [5] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, T. Liu, A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, ArXiv (11 2023).
  * [6] F. Fournier, L. Limonad, I. Skarbovsky, Y. David, The WHY in Business Processes: Discovery of Causal Execution Dependencies (10 2023).  doi:10.48550/arXiv.2310.14975.
URL https://arxiv.org/abs/2310.14975v1

  * [7] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, G. Neubig, Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, ACM Computing Surveys 55 (9) (2023).  doi:10.1145/3560815.
  * [8] B. Meskó, Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial, Journal of Medical Internet Research 25 (2023) e50638.  doi:10.2196/50638.
  * [9] M. Weske, Business Process Management Architectures, in: Business Process Management, Springer Berlin Heidelberg, Berlin, Heidelberg, 2019, pp. 351–384.  doi:10.1007/978-3-662-59432-2{\\_}8.
URL http://link.springer.com/10.1007/978-3-662-59432-2_8

  * [10] W. van der Aalst, Process Mining, Springer Berlin Heidelberg, Berlin, Heidelberg, 2016.  doi:10.1007/978-3-662-49851-4.
URL http://link.springer.com/10.1007/978-3-662-49851-4

  * [11] S. J. J. Leemans, Automated Process Discovery, in: S. Sakr, A. Y. Zomaya (Eds.), Encyclopedia of Big Data Technologies, Springer International Publishing, Cham, 2019, pp. 121–130.  doi:10.1007/978-3-319-77525-8{\\_}88.
URL https://doi.org/10.1007/978-3-319-77525-8_88

  * [12] J. Pearl, Causality: Models, reasoning, and inference, second edition, Causality: Models, Reasoning, and Inference, Second Edition (2011) 1–464doi:10.1017/CBO9780511803161.
URL
https://www.cambridge.org/core/books/causality/B0046844FAE10CBF274D4ACBDAEB5F5B

  * [13] P. Spirtes, C. Glymour, R. Scheines, Causation, Prediction, and Search, The MIT Press, 2001.  doi:10.7551/mitpress/1754.001.0001.
URL https://direct.mit.edu/books/book/2057/causation-prediction-and-search

  * [14] M. S. Qafari, W. van der Aalst, Root Cause Analysis in Process Mining Using Structural Equation Models, in: Lecture Notes in Business Information Processing, Vol. 397, LNBIP, 2020, pp. 155–167.  doi:10.1007/978-3-030-66498-5{\\_}12.
URL https://link.springer.com/10.1007/978-3-030-66498-5_12

  * [15] S. Shimizu, Statistical Causal Discovery: LiNGAM Approach, SpringerBriefs in Statistics, Springer Japan, Tokyo, 2022.  doi:10.1007/978-4-431-55784-5.
URL https://link.springer.com/10.1007/978-4-431-55784-5

  * [16] M. Vidgof, S. Bachhofner, J. Mendling, Large Language Models for Business Process Management: Opportunities and Challenges, in: Lecture Notes in Business Information Processing, Vol. 490, LNBIP, 2023, pp. 107–123.  doi:10.1007/978-3-031-41623-1{\\_}7.
URL https://link.springer.com/10.1007/978-3-031-41623-1_7

  * [17] S. Verma, A. Lahiri, J. P. Dickerson, S.-I. Lee, Pitfalls of Explainable ML: An Industry Perspective (6 2021).
URL http://arxiv.org/abs/2106.07758

  * [18] A. Adadi, M. Berrada, Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI), IEEE Access 6 (2018).  doi:10.1109/ACCESS.2018.2870052.
  * [19] C. Meske, E. Bunde, J. Schneider, M. Gersch, Explainable Artificial Intelligence: Objectives, Stakeholders, and Future Research Opportunities, Information Systems Management 39 (1) (2022).  doi:10.1080/10580530.2020.1849465.
  * [20] G. Elkhawaga, O. Elzeki, M. Abuelkheir, M. Reichert, Evaluating Explainable Artificial Intelligence Methods Based on Feature Elimination: A Functionality-Grounded Approach, Electronics (Switzerland) 12 (7) (2023).  doi:10.3390/electronics12071670.
  * [21] J. R. Rehse, N. Mehdiyev, P. Fettke, Towards Explainable Process Predictions for Industry 4.0 in the DFKI-Smart-Lego-Factory, KI - Kunstliche Intelligenz 33 (2) (2019).  doi:10.1007/s13218-019-00586-1.
  * [22] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi, A survey of methods for explaining black box models, ACM Computing Surveys 51 (5) (2018).  doi:10.1145/3236009.
  * [23] M. T. Ribeiro, S. Singh, C. Guestrin, ”Why should i trust you?” Explaining the predictions of any classifier, in: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Vol. 13-17-August-2016, 2016, pp. 1135–1144.  doi:10.1145/2939672.2939778.
  * [24] S. M. Lundberg, S. I. Lee, A unified approach to interpreting model predictions, in: Advances in Neural Information Processing Systems, Vol. 2017-December, 2017, pp. 4768–4777.
  * [25] G. Amit, F. Fournier, S. Gur, L. Limonad, Model-informed LIME Extension for Business Process Explainability, in: CEUR Workshop Proceedings, Vol. 3310, 2022, pp. 1–12.
  * [26] S. Upadhyay, V. Isahagian, V. Muthusamy, Y. Rizk, Extending LIME for Business Process Automation (8 2021).
URL http://arxiv.org/abs/2108.04371

  * [27] L. Ehrlinger, W. Wöß, Towards a definition of knowledge graphs, in: CEUR Workshop Proceedings, Vol. 1695, 2016, pp. 1–4.
  * [28] A. Bonifati, G. Fletcher, H. Voigt, N. Yakovets, Querying Graphs, Synthesis Lectures on Data Management 10 (3) (2018).  doi:10.2200/s00873ed1v01y201808dtm051.
  * [29] S. Esser, D. Fahland, Multi-Dimensional Event Data in Graph Databases, Journal on Data Semantics 10 (1-2) (2021).  doi:10.1007/s13740-021-00122-1.
  * [30] D. Fahland, Process Mining over Multiple Behavioral Dimensions with Event Knowledge Graphs, in: Lecture Notes in Business Information Processing, Vol. 448, LNBIP, 2022, pp. 274–319.  doi:10.1007/978-3-031-08848-3{\\_}9.
URL https://link.springer.com/10.1007/978-3-031-08848-3_9

  * [31] A. Augusto, R. Conforti, M. Dumas, M. La Rosa, A. Polyvyanyy, Split miner: automated discovery of accurate and simple business process models from event logs, Knowledge and Information Systems 59 (2) (2019).  doi:10.1007/s10115-018-1214-x.
  * [32] F. Mannhardt, M. De Leoni, H. A. Reijers, Heuristic mining revamped: An interactive, data-Aware, and conformance-Aware miner, in: CEUR Workshop Proceedings, Vol. 1920, 2017, pp. 1–5.
  * [33] G. Amit, F. Fournier, L. Limonad, I. Skarbovsky, Situation-Aware eXplainability for Business Processes Enabled by Complex Events, in: Business Process Management Workshops, Vol. 460, Springer, Cham, 2023, pp. 45–57.  doi:10.1007/978-3-031-25383-6{\\_}5.
  * [34] R. D. Freeze, R. L. Raschke, An assessment of formative and reflective constructs in is research, in: Proceedings of the 15th European Conference on Information Systems, ECIS 2007, 2007, pp. 1481–1492.
  * [35] G. Vilone, L. Longo, Notions of explainability and evaluation approaches for explainable artificial intelligence, Information Fusion 76 (2021).  doi:10.1016/j.inffus.2021.05.009.
  * [36] K. Sokol, P. Flach, Explainability fact sheets: A framework for systematic assessment of explainable approaches, in: FAT* 2020 - Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 2020, pp. 56–67.  doi:10.1145/3351095.3372870.
  * [37] I. Lage, E. Chen, J. He, M. Narayanan, S. Gershman, B. Kim, F. Doshi-Velez, An Evaluation of the Human-Interpretability of Explanation, in: Conference on Neural Information Processing Systems (NeurIPS) Workshop on Correcting and Critiquing Trends in Machine Learning, 2018, pp. 1–7.
  * [38] A. F. Markus, J. A. Kors, P. R. Rijnbeek, The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies (2021).  doi:10.1016/j.jbi.2020.103655.
  * [39] D. V. Carvalho, E. M. Pereira, J. S. Cardoso, Machine learning interpretability: A survey on methods and metrics (2019).  doi:10.3390/electronics8080832.
  * [40] T. Kulesza, S. Stumpf, M. Burnett, S. Yang, I. Kwan, W. K. Wong, Too much, too little, or just right? Ways explanations impact end users’ mental models, in: Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC, 2013, pp. 3–10.  doi:10.1109/VLHCC.2013.6645235.
  * [41] J. D. Lee, K. A. See, Trust in automation: Designing for appropriate reliance (2004).  doi:10.1518/hfes.46.1.50{\\_}30392.
  * [42] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, L. Kagal, Explaining Explanations: An Overview of Interpretability of Machine Learning, in: 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), IEEE, 2018, pp. 80–89.  doi:10.1109/DSAA.2018.00018.
  * [43] R. R. Hoffman, S. T. Mueller, G. Klein, J. Litman, Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance (2023).  doi:10.3389/fcomp.2023.1096257.
  * [44] A. Holzinger, A. Carrington, H. Müller, Measuring the Quality of Explanations: The System Causability Scale (SCS), KI - Künstliche Intelligenz 34 (2) (2020).  doi:10.1007/s13218-020-00636-z.
  * [45] F. D. Davis, A technology acceptance model for empirically testing new end-user information systems: Theory and results, Management Ph.D. (1985).  doi:oclc/56932490.
  * [46] F. D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly: Management Information Systems 13 (3) (1989).  doi:10.2307/249008.
  * [47] G. C. Moore, I. Benbasat, Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation, Information Systems Research 2 (3) (1991) 192–222.  doi:10.1287/isre.2.3.192.
  * [48] R. Likert, A technique for the measurement of attitudes, Archives of Psychology 140 (1932).
  * [49] J. Shi, X. Mo, Z. Sun, [Content validity index in scale development]., Zhong nan da xue xue bao. Yi xue ban = Journal of Central South University. Medical sciences 37 (2) (2012) 152–5.  doi:10.3969/j.issn.1672-7347.2012.02.007.
  * [50] D. L. Streiner, Figuring out factors: the use and misuse of factor analysis., Canadian journal of psychiatry. Revue canadienne de psychiatrie 39 (3) (1994) 135–40.  doi:10.1177/070674379403900303.
  * [51] D. Fahland, F. Fournier, L. Limonad, I. Skarbovsky, A. J. Swevels, Why are my Pizzas late?, in: CEUR-WS Vol-3569, 2023, pp. 25–28.
  * [52] A. Beheshti, J. Yang, Q. Z. Sheng, B. Benatallah, F. Casati, S. Dustdar, H. R. M. Nezhad, X. Zhang, S. Xue, ProcessGPT: Transforming Business Process Management with Generative Artificial Intelligence, in: Proceedings - 2023 IEEE International Conference on Web Services, ICWS 2023, 2023, pp. 731–739.  doi:10.1109/ICWS60048.2023.00099.
  * [53] Y. Rizk, P. Venkateswaran, V. Isahagian, A. Narcomey, V. Muthusamy, A Case for Business Process-Specific Foundation Models, in: Business Process Management Workshops. BPM 2023. Lecture Notes in Business Information Processing, Vol. 492, Springer, Cham, 2024, pp. 44–56.  doi:10.1007/978-3-031-50974-2{\\_}4.
  * [54] M. Grohs, L. Abb, N. Elsayed, J.-R. Rehse, Large Language Models Can Accomplish Business Process Management Tasks, in: Business Process Management Workshops. BPM 2023. Lecture Notes in Business Information Processing, Vol. 492, Springer, Cham, 2024, pp. 453–465.  doi:10.1007/978-3-031-50974-2{\\_}34.
  * [55] A. Berti, D. Schuster, W. M. P. van der Aalst, Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: A Case Study, Business Process Management Workshops. BPM 2023. Lecture Notes in Business Information Processing 492 (1 2024).  doi:https://doi.org/10.1007/978-3-031-50974-2{\\_}32.
  * [56] S. H. Bach, V. Sanh, Z. X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Fevry, Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-David, C. Xu, G. Chhablani, H. Wang, J. A. Fries, M. S. Al-Shaibani, S. Sharma, U. Thakker, K. Almubarak, X. Tang, D. Radev, M. T. J. Jiang, A. M. Rush, PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2022, pp. 93–104.  doi:10.18653/v1/2022.acl-demo.9.
  * [57] U. Jessen, M. Sroka, D. Fahland, Chit-Chat or Deep Talk: Prompt Engineering for Process Mining, ArXiv (7 2023).
  * [58] L. S. Shapley, A Value for n-Person Games, in: H. W. Kuhn, A. W. Tucker (Eds.), Contributions to the Theory of Games II, Princeton University Press, Princeton, 1953, pp. 307–317.
  * [59] R. Galanti, B. Coma-Puig, M. de Leoni, J. Carmona, N. Navarin, Explainable Predictive Process Monitoring, in: B. F. van Dongen, M. Montali, M. T. Wynn (Eds.), 2nd International Conference on Process Mining, ICPM 2020, Padua, Italy, October 4-9, IEEE, 2020, pp. 1–8.  doi:10.1109/ICPM49681.2020.00012.
URL https://doi.org/10.1109/ICPM49681.2020.00012

  * [60] A. Padella, M. de Leoni, O. Dogan, R. Galanti, Explainable Process Prescriptive Analytics, in: A. Burattin, A. Polyvyanyy, B. Weber (Eds.), 4th International Conference on Process Mining, ICPM 2022, Bolzano, Italy, October 23-28, IEEE, 2022, pp. 16–23.  doi:10.1109/ICPM57379.2022.9980535.
URL https://doi.org/10.1109/ICPM57379.2022.9980535

  * [61] G. El-Khawaga, M. Abu-Elkheir, M. Reichert, XAI in the Context of Predictive Process Monitoring: An Empirical Analysis Framework, Algorithms 15 (6) (2022) 199.  doi:10.3390/A15060199.
URL https://doi.org/10.3390/a15060199

  * [62] A. Stevens, J. D. Smedt, J. Peeperkorn, Quantifying Explainability in Outcome-Oriented Predictive Process Monitoring, in: J. Munoz-Gama, X. Lu (Eds.), Process Mining Workshops - ICPM 2021 International Workshops, Eindhoven,The Netherlands, October 31 - November 4, Revised Selected Papers, Vol. 433 of Lecture Notes in Business Information Processing, Springer, 2021, pp. 194–206.  doi:10.1007/978-3-030-98581-3{\\_}15.
URL https://doi.org/10.1007/978-3-030-98581-3_15

  * [63] M. Velmurugan, C. Ouyang, C. Moreira, R. Sindhgatta, Evaluating Stability of Post-hoc Explanations for Business Process Predictions, in: H. Hacid, O. Kao, M. Mecella, N. Moha, H.-y. Paik (Eds.), Service-Oriented Computing - 19th International Conference, ICSOC2021, Virtual Event, November 22-25, Proceedings, Vol. 13121 of Lecture Notes in Computer Science, Springer, 2021, pp. 49–64.  doi:10.1007/978-3-030-91431-8{\\_}4.
URL https://doi.org/10.1007/978-3-030-91431-8_4

  * [64] B. Wickramanayake, C. Ouyang, Y. Xu, C. Moreira, Generating multi-level explanations for process outcome predictions, Eng. Appl. Artif. Intell. 125 (2023) 106678.  doi:10.1016/J.ENGAPPAI.2023.106678.
URL https://doi.org/10.1016/j.engappai.2023.106678

  * [65] R. Galanti, M. de Leoni, M. Monaro, N. Navarin, A. Marazzi, B. D. Stasi, S. Maldera, An explainable decision support system for predictive process analytics, Eng. Appl. Artif. Intell. 120 (2023) 105904.  doi:10.1016/J.ENGAPPAI.2023.105904.
URL https://doi.org/10.1016/j.engappai.2023.105904

  * [66] W. Rizzi, M. Comuzzi, C. D. Francescomarino, C. Ghidini, S. Lee, F. M. Maggi, A. Nolte, Explainable Predictive Process Monitoring: A User Evaluation (2022).
  * [67] Z. D. Bozorgi, I. Teinemaa, M. Dumas, M. La Rosa, A. Polyvyanyy, Process Mining Meets Causal Machine Learning: Discovering Causal Rules from Event Logs, in: 2020 2nd International Conference on Process Mining (ICPM), IEEE, 2020, pp. 129–136.  doi:10.1109/ICPM49681.2020.00028.
  * [68] Z. D. Bozorgi, I. Teinemaa, M. Dumas, M. L. Rosa, A. Polyvyanyy, Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction, in: 2021 3rd International Conference on Process Mining (ICPM), IEEE, 2021, pp. 96–103.  doi:10.1109/ICPM53251.2021.9576853.
  * [69] Z. Dasht Bozorgi, I. Teinemaa, M. Dumas, M. La Rosa, A. Polyvyanyy, Prescriptive process monitoring based on causal effect estimation, Information Systems 116 (2023) 102198.  doi:10.1016/j.is.2023.102198.
  * [70] S. R. Künzel, J. S. Sekhon, P. J. Bickel, B. Yu, Metalearners for estimating heterogeneous treatment effects using machine learning, Proceedings of the National Academy of Sciences of the United States of America 116 (10) (2019).  doi:10.1073/pnas.1804597116.