# Are Graph Representation Learning Methods Robust to Graph Sparsity and
Asymmetric Node Information?

Marine Neyret∗1 and Pierre Sevestre∗2
1DataLab, Institut Louis Bachelier, Paris, France
2DataLab, Inspection Générale et Audit, Société Générale, Paris, France
∗Both authors contributed equally.

###### Abstract

The growing popularity of Graph Representation Learning (GRL) methods has
resulted in the development of a large number of models applied to a
miscellany of domains. Behind this diversity of domains, there is a strong
heterogeneity of graphs, making it difficult to estimate the expected
performance of a model on a new graph, especially when the graph has
distinctive characteristics that have not been encountered in the benchmark
yet. To address this, we have developed an experimental pipeline, to assess
the impact of a given property on the models performances. In this paper, we
use this pipeline to study the effect of two specificities encountered on
banks transactional graphs resulting from the partial view a bank has on all
the individuals and transactions carried out on the market. These specific
features are graph sparsity and asymmetric node information. This study
demonstrates the robustness of GRL methods to these distinctive
characteristics. We believe that this work can ease the evaluation of GRL
methods to specific characteristics and foster the development of such methods
on transactional graphs.

##  1 INTRODUCTION

Graph Representation Learning (GRL) has gained increasing interest in the past
few years, due to the ubiquity of its applications and the rapid development
of new learning methods. First, graphs are a powerful way to represent
relational data, with applications in numerous domains such as social media
[Monti et al., 2019], biology ([Zitnik et al., 2018], [Jumper et al., 2021]),
knowledge graphs ([Bordes et al., 2013], [Wang et al., 2020]) or recommender
systems [McAuley et al., 2015]. Second, the rising interest led to a rich and
evolving literature about graph learning ([Goyal and Ferrara, 2017], [Zhang et
al., 2020]), propelled by the recent development of Graph Neural Network
(GNN), becoming the de facto method for machine learning on graph, with
notable contributions of Graph Convolutional Networks (GCN) [Kipf and Welling,
2017] and Graph Attention Networks (GAT) [Veličković et al., 2018].

The diversity of application domains and the associated graph heteromorphism
also present some drawbacks: it is difficult to foresee, from the
characteristics of a graph, the expected performances of a model. In
particular, a model performing well on a given graph benchmark could fail on
others. Furthermore, the graph benchmark has historically been built around
three citation graphs (Citeseer [Giles et al., 1998], Cora [McCallum et al.,
2000] and Pubmed [Sen et al., 2008]), whose limitations, both in terms of
complexity and scale, started to be highlighted ([Dwivedi et al., 2020],
[Salha et al., 2019]). Recent initiatives have been undertaken to provide more
diverse and complex open-source datasets (OGB111https://ogb.stanford.edu/ and
TUDataset222https://chrsmrrs.github.io/datasets/), and to improve the quality
of the benchmark with a strong emphasis on comprehensiveness and consistency
([Dwivedi et al., 2020], [Lv et al., 2021], [Errica et al., 2022]). The final
objective being to accurately evaluate of models performances. Our work is in
line with this philosophy, with the objective to accelerate and ease the
evaluation of model performances, and more specifically to further understand
how specific characteristics of a graph affect model performances.

This can be particularly useful in the banking industry, where most of its
core activities involve transactions that can be intuitively modelled using
graphs. A transaction usually corresponds to a money transfer between two
parties. The nodes represent the parties involved in the transactions and the
edges encode the properties of the transaction between the two nodes. However,
work on transactional graphs has been sporadic, mainly centered around fraud
detection for financial security ([Weber et al., 2019], [Weber et al., 2018],
[Bruss et al., 2019], [Li et al., 2020]). The very specific nature of this
application does not allow to constitute a comprehensive evaluation benchmark
on transactional graphs. Furthermore, the transactional graphs obtained by the
banks present distinctive characteristics which differ from common benchmark
graphs. Indeed, the banks only access the transactions where their clients are
involved without having access to transactions between non-clients. The
resulting transactional subgraph is sparser than the complete graph of all the
transactions being performed on the market. It has deteriorated local
structures and a modified node distribution compared to common graphs found in
the literature as shown in Appendix. A question arising is the ability of
Graph Representation Learning methods to learn representations on these sparse
subgraphs. Moreover, the banks hold more information about their clients than
they have about the non-client counterparts. Thus, a larger range of node
features are available for clients. An efficient model should be able to take
advantage of this asymmetric node information.

We propose an experimental pipeline to better understand how the two intrinsic
characteristics (sparsity and asymmetric node information) of transactional
graphs encountered within the bank affect the performance of GRL models,
compared to the performance obtained on the full graphs. The objective is to
analyze if node-level embeddings produced from transactional graphs encompass
enough structural and semantic information about the graph to perform
downstream tasks with sufficient performances. This study is performed on
public benchmark graphs commonly used in the literature that are preprocessed
to match the banks transactional graphs distinctive characteristics. This
choice was made, in the absence of open-source datasets with the specific
characteristics of transactional graphs. This allows to benefit from the
knowledge acquired, especially in terms of model selection, to favor models
comparison, and to isolate a specific graph property for the experiments.

The contributions of this paper are as follows:

  1. 1.

Challenge some Graph Representation Learning methods on transactional graph
distinctive characteristics. We study the effect of the two banks
transactional graphs distinctive characteristics that are graph sparsity and
asymmetric node information on the performance of state-of-the-art models,
through the transformation of commonly used graphs in the literature to match
these characteristics.

  2. 2.

Design of a robust experimental pipeline. We develop a standardized pipeline
to easily experiment various models and graph configurations.

The rest of the paper is organized as follows. Section 2 introduces
definitions about graph theory. Section 3 defines the preprocessors designed
to transform commonly used graphs and gives some theory about the surveyed
Graph Representation Learning methods. Our experimental results are detailed
in Section 4.

##  2 DEFINITIONS AND PRELIMINARIES

In this paper, we consider a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$,
with $\mathcal{V}=\\{v_{1},...,v_{n}\\}$ its set of $n$ vertices (or nodes),
and $\mathcal{E}\subset\mathcal{V}\times\mathcal{V}$, its set of $m$ edges.
When node attributes are available, the graph can be associated with a node
features matrix $X\in\mathbb{R}^{n\times f}$, with $f$ being the node features
dimension. We focus in the following on undirected and unweighted graphs,
containing neither self-loops nor multiple edges.

We denote by

$A=(a_{ij})_{i,j=1,...,n}$ (1)

the adjacency matrix representing the graph structure, with $a_{ij}=1$ if an
edge exists between nodes $v_{i}$ and $v_{j}$, and $a_{ij}=0$ otherwise.

$\tilde{A}=A+I_{n}$ (2)

corresponds to the adjacency matrix of the graph with all self-loops included.

We further define $d_{i}$, the degree of node $v_{i}\in\mathcal{V}$, as the
sum of neighboring weights, i.e. the number of neighbors in the case of an
unweighted graph, $d_{i}=\sum_{j=1}^{n}a_{ij}$. The degree matrix is then
defined as the diagonal matrix containing all node degrees,

$D=diag(d_{1},...,d_{n}).$ (3)

The degree matrix of $\tilde{A}$ is denoted by $\tilde{D}$.

From the two matrices introduced above, we derive the unnormalized graph
Laplacian matrix, defined as:

$L=D-A.$ (4)

Table 1 summarizes the main notations used throughout the paper.

Table 1: Summary of commonly used notations.
Notation | Description
---|---
$\mathcal{G}$, $\mathcal{V}$, $\mathcal{E}$ | Graph, set of nodes, set of edges
$n$, $m$ | Number of nodes and edges
$v_{i}$, $\mathcal{N}_{i}$ | Node $i$ and its first-order neighborhood
$A$, $D$, $L$ | Adjacency matrix, degree matrix, Laplacian
$d_{i},d$ | Degree of node $v_{i}$, average degree
$X$, $f$ | Node features matrix, node features dimension
$\tilde{A}$, $\tilde{D}$ | Adjacency and degree matrices of the graph with all self-loops
a, $\mathbf{a}$, A | Scalar, vector, matrix
$\mathbb{R}^{d}$ | Euclidean space, d-dimensional
$A^{\top}$ | Transposed of $A$
$\bigparallel$ | Concatenation operation
$H^{l}$, $H^{l}_{i}$ | Hidden representation matrix and hidden representation of node $v_{i}$ at layer $l$
$f_{l}$ | Hidden representation dimension at layer $l$

##  3 PROPOSED BENCHMARK METHOD

The proposed experimental pipeline consists in two parts. The first part,
which we refer to as the preprocessor, corresponds to the transformations
applied to the graphs to match certain distinctive characteristics. Further
details can be found in Section 3.1. Then follows the model training and
evaluation part. Assessing the quality of the learned node embeddings is
usually reduced to the ability to perform a given downstream task. In this
study, we chose to focus on the node classification and link prediction tasks,
as described in Section 3.2.

###  3.1 Preprocessors

As mentioned previously, this study intends to evaluate the impact of the
banks transactional graph singularities, that are partial graph information,
or sparsity, and asymmetric node information. To achieve this, two graph
preprocessors are designed to transform original graphs from the literature to
match the banks transactional graphs distinctive features:

  * –

The first processor is targeting solely the sparsity aspect. The procedure,
described on Figure 1, consists in creating a subgraph of the original graph,
by first sampling nodes that will act as client nodes and then removing any
edge not connected to this subset of client nodes. We will refer to this
processor as the sampler, and to the corresponding portion of client nodes as
$r_{sampling}$.

  * –

The second processor further targets the asymmetric node information. Here,
the sampler is first applied. Node features of sampled clients nodes are left
unchanged and we deteriorate a portion of the node features of the remaining
set of nodes. In the reported results, the deterioration method simply
consists in the deletion of a portion of node features. This processor will be
referred to as features_sampler, and we will use $r_{sampling}$ to invoke the
portion of sampled client nodes and $r_{nf\\_sampling}$ for the portion of
deteriorated node features.

Figure 1: Illustration of the sampling procedure. Client nodes are first
sampled according to $r_{sampling}$, then, all edges not connected to these
nodes are removed.

In the experiments, we will use the term baseline when no processor is
applied, i.e. when the entire graph is provided to the model. A decreasing
$r_{sampling}$ is used with values in ($0.5$, $0.1$, $0.01$, $5e-3$, $1e-3$),
provided that the number of remaining nodes from the original graph is
sufficient. $r_{nf\\_sampling}$ is chosen among ($0.1$, $0.5$, $0.75$), in
order to monitor the impact of the two distinctive characteristics.

###  3.2 Models Surveyed

Robustness to these graph distinctive characteristics of both unsupervised,
random-walk based and supervised graph neural networks based methods are
assessed. To each method corresponds a dedicated evaluation pipeline based on
downstream tasks detailed below. The evaluation of the models focuses on
nodes-centric tasks, more precisely node classification and link prediction.
Node classification stands for supervised learning of node labels. Link
prediction refers to the task of predicting the existence of an edge between
source and destination nodes.

####  3.2.1 Unsupervised Models

DeepWalk operates on the graph structure only, without taking nodes features
into account. Starting from the observation that the distribution of nodes in
a corpus of small random walks follows the same power-law distribution as the
one observed for words in a text corpus, [Perozzi et al., 2014] introduced
DeepWalk and used the progress in word representation [Mikolov et al., 2013]
to create node representation. The algorithm first transforms the graph
sequences of nodes using truncated, unbiased random walks (corresponding to
the text corpus for a word representation). Then, this set of walks is fed to
Skipgram [Mikolov et al., 2013], that learns to predict context nodes in
random walks for each target node. The resulting representation creates close
embedding for nodes co-occuring in random walks, preserving high-order
proximity.

node2vec [Grover and Leskovec, 2016] is derived from DeepWalk. The difference
lies on the random walks generation, through the use of biased random walks to
give more flexibility to the exploration. Specifically, two parameters allow
the exploration to favor either Breadth-first or Depth-first strategy. These
walks create a new definition of neighborhood, and the resulting embedding no
longer depends only on the community, but on the local structure around the
node as well.

Training and evaluation processes. Once embeddings are computed from these
unsupervised models, the pipelines are split according to the evaluation task.
The unsupervised embeddings are used for two distinct downstream tasks: node
classification and link prediction. For node classification, a logistic
regression is trained using node embeddings as input, to predict some node
labels. For link prediction, nodes embeddings of two nodes are concatenated
and a logistic regression is trained to predict the likelihood of these two
nodes being connected. Existing and non-existing edges are sampled evenly.

####  3.2.2 Supervised Models

Graph Convolutional Networks (GCN) iteratively apply spatial filter on one
node’s direct neighbors information to produce its embedding. This model takes
into account both graph structure and node features, and it assumes that graph
edges encode nodes similarity.

The propagation rule from [Kipf and Welling, 2017] focuses on first order
approximation and is defined by:

$H^{l+1}=\sigma\left(\widetilde{D}^{-\frac{1}{2}}\widetilde{A}\widetilde{D}^{-\frac{1}{2}}H^{l}\Theta^{l}\right),$
(5)

where $\sigma(.)$ is a non-linear transformation, $\Theta^{l}$ a trainable
filter matrix, of shape $f_{l}\times f_{l+1}$ and $H^{l}\in\mathbb{R}^{n\times
f_{l}}$ the hidden state at layer l ($f_{l}$ being the embedding dimension at
layer l and $H^{0}$ being initialized to X).

At each layer, the embedding of one node corresponds to a combination of
surrounding nodes embeddings with pre-determined weights depending on nodes
degrees.

GraphSage [Hamilton et al., 2017] learns a representation of one node based on
an aggregation of a fixed $K$-hop neighborhood information. For each node, the
embedding is also initialized to the node features matrix and, then, the model
iteratively creates an aggregated embedding using $k<K+1$ depth neighbors
embeddings that is concatenated with the embeddings computed from smaller
neighborhoods. All the neighboring nodes share the same weight in the
embedding.

For a node $v_{i}$, using the same notations as for GCN, the embedding at
layer $l+1$ is given by:

$\displaystyle a^{l+1}_{\mathcal{N}_{i}}$
$\displaystyle=\texttt{agg}\left(\\{H^{l}_{u},\forall
u\in\mathcal{N}_{i}\\}\right),$ (6) $\displaystyle H^{l+1}_{i}$
$\displaystyle=\sigma\left(\Theta^{l}*\left(H^{l}_{i}\bigparallel
a^{l}_{\mathcal{N}_{i}}\right)\right).$ (7)

The authors of the paper proposed to use either a mean, a LSTM or a pooling
aggregator (agg in previous equation). We focused on the mean aggregation
function only.

Graph Attention Networks (GAT) [Veličković et al., 2018] integrates attention
related strategy in their layers to evaluate connectivity strength between
nodes. The idea behind GAT is to employ a self-attention mechanism to compute
the representation of a node by attending over all its neighbors, learning to
pick the most relevant ones.

The output at layer $l+1$ is a weighted linear combination of the previous
hidden state representations:

$H^{l+1}_{i}=\sigma\left(\sum_{j\in\mathcal{N}_{i}}\alpha^{l}_{ij}\Theta^{l}H^{l}_{i}\right),$
(8)

where $\alpha^{l}_{ij}$ is the normalized attention coefficient between node
$v_{i}$ and $v_{j}$ at layer $l$:

$\displaystyle\alpha^{l}_{ij}$
$\displaystyle=\texttt{softmax}_{\mathcal{N}_{i}}\left(e^{l}_{ij}\right),$ (9)
$\displaystyle e^{l}_{ij}$
$\displaystyle=\texttt{LeakyReLU}\left(\mathbf{a}^{\top}\left[\Theta^{l}H_{i}\bigparallel\Theta^{l}H_{j}\right]\right).$
(10)

Training and evaluation processes. For node classification, no stage is added
as the models are directly trained to classify the nodes according to their
labels. For link prediction, a 3-layer perceptron is stacked to the models and
trained using contrastive loss, with negative graphs generated at every epoch
and the models learning to discriminate existing edges from the graph and non-
existing edges from the negative graph.

###  3.3 Datasets

For this study, GRL robustness assessment is performed on the four datasets
described below. This choice was motivated by the need to use graphs large
enough to be able to sample nodes in substantial proportion preserving decent
scale and complexity for the resulting graph. The selected datasets are:

  * –

Pubmed [Sen et al., 2008], an undirected citation network of diabetes-related
publications, with nodes corresponding to research papers authors and edges to
citation relationship. The 3 available labels are derived from the type of
diabetes addressed in the publication, and node features are sparse bag-of-
words of dimension 500.

  * –

CoAuthor CS, a citation network within the field of computer science, created
using Microsoft Academic Graph from the KDD Cup 2016. Nodes are also authors,
and edges exist if authors co-authored a paper. The corresponding node label
is assigned based on the most active field of research of the author for a
total of 15 classes. Each node is associated with a 6,805 dimensional features
vector representing papers keywords.

  * –

Amazon Computer [McAuley et al., 2015], a recommendation network extracted
from Amazon co-purchase graph. Nodes are products, and edges indicate whether
these products are frequently bought together. The 5 classes are derived from
the product categories, and node features from bag-of-words encoded product
reviews of dimension 767.

  * –

Reddit [Hamilton et al., 2017], a social network created from posts made in
September 2014 on the online service. Nodes correspond to posts, each
belonging to a community, or subreddit, assigned as label. The network has 50
labels. Edges between posts are created if a user of the platform commented
both posts. Nodes are described by 602 features containing a concatenation of
an average embedding of dimension 300 of the post title, an average embedding
of dimension 300 of the post’s comments, the post’s score and the number of
post’s comments.

##  4 EXPERIMENT PIPELINE RESULTS

(a) $r_{sampling}$Accuracy

(b) $r_{sampling}$AUC

Figure 2: (a) Node classification accuracy of surveyed models using the
sampler preprocessor on Amazon Computer. (b) Link prediction AUC of surveyed
models using the sampler preprocessor on Pubmed. Both for (a) and (b), results
are averaged over 100 independent runs (10 trainings on 10 generated graphs)
and reported for the test set. Line thickness accounts for the standard
deviation.

###  4.1 Experimental Setting

We implemented our own version of DeepWalk. Default parameters defined in the
original paper are applied to report results: 10 walks of length 80 are
generated from each node and the resulting node embedding is of dimension 128.

The nodevectors333https://github.com/VHRanger/nodevectors fast and scalable
implementation of node2vec is used. Walks length, walks number and embedding
dimension are set to the same values as for DeepWalk. According to
hyperparameters sensitivity analysis presented in [Goyal and Ferrara, 2017],
we set $(p,q)=(1,2)$ for link prediction and $(p,q)=(1,0.5)$ for node
classification.

The three graph neural network models under review are implemented using Deep
Graph Library (DGL) [Wang et al., 2019]. For each dataset, the models
parameters are gridsearched on the baseline graph and then used on all
sampling experiments. Although it would have been more accurate to search for
optimal parameters on each graph configuration, the associated computation
cost was high, motivating the use of this approach.

Experiments are conducted on NVIDIA Tesla V100-PCIE-16GB.

###  4.2 Impact of Graph Sparsity on Models Performances

Node classification. Although models are built upon various approaches, the
sampler preprocessor has limited, consistent effect over all models and
datasets. If we ignore a few exceptions encountered for Pubmed that has a
small number of nodes, the drop in performance when sampling stays below
$11\%$ compared to the unsampled baseline for supervised graph neural
networks. This value drops to $4\%$ when restricting $r_{sampling}$ to values
bigger than $0.01$. The change is even less significant on CoAuthor CS and
Amazon Computer. Random walks based unsupervised methods tend to be more
impacted by the sampling preprocessing than graph neural networks. The first
models cannot use node features to compensate for the loss of information from
the sampling, nor benefit from the supervised signal of the remaining models.
This can be seen in Figure 2(a).

Link prediction. Experiments on link prediction provide less consistent
results across models and datasets: the performance when sampling can either
stay stable, increase or decrease, with higher variability than on node
classification. Convergences issues are faced for some experiments with GAT
contributing to results discrepancy (see Figure 2(b)). The unbalanced effect
between random walk based models and supervised graph neural networks no
longer stands out for link prediction.

Setting aside the evaluation task, when the number of nodes in the whole graph
is limited, as for Pubmed, the models behavior becomes unstable when sampling
ratio is decreasing: the performance can increase for small ratios as seen in
Figure 2(b), and the drop in performance can soar (up to$80\%$ below baseline)
in node classification.

###  4.3 Impact of Asymmetric Node Information on Models Performances

As DeepWalk and node2vec do not introduce node features into embeddings
computation. Only the robustness evaluation to asymmetric node information of
supervised graph neural networks is performed.

Table 2: Results of surveyed models in terms of node classification accuracies using the features_sampler preprocessor on Amazon Computer. Results are averaged over 100 independent runs (10 trainings on 10 generated graphs) and standard deviations are reported. Amazon Computer | GCN | GAT | GraphSage
---|---|---|---
$r_{nf\\_sampling}$ | $r_{sampling}$
baseline | 0.869 $\pm 0.015$ | 0.908 $\pm 0.003$ | 0.857 $\pm 0.030$
0.1 | 0.5 | 0.882 $\pm 0.013$ |  0.910 $\pm 0.004$ |  0.871 $\pm 0.026$
0.1 | 0.886 $\pm 0.018$ | 0.895 $\pm 0.005$ | 0.861 $\pm 0.022$
0.01 | 0.875 $\pm 0.016$ | 0.874 $\pm 0.004$ | 0.848 $\pm 0.030$
0.5 | 0.5 |  0.896 $\pm 0.004$ | 0.900 $\pm 0.004$ | 0.852 $\pm 0.023$
0.1 | 0.884 $\pm 0.004$ | 0.889 $\pm 0.008$ | 0.855 $\pm 0.017$
0.01 | 0.879 $\pm 0.009$ | 0.883 $\pm 0.006$ | 0.833 $\pm 0.020$
0.75 | 0.5 | 0.878 $\pm 0.002$ | 0.888 $\pm 0.009$ | 0.814 $\pm 0.022$
0.1 | 0.857 $\pm 0.008$ | 0.866 $\pm 0.006$ | 0.792 $\pm 0.032$
0.01 | 0.850 $\pm 0.004$ | 0.841 $\pm 0.009$ | 0.774 $\pm 0.025$
Table 3: Results of surveyed models in terms of link prediction AUC using the features_sampler preprocessor on CoAuthor CS. Results are averaged over 100 independent runs (10 trainings on 10 generated graphs) and standard deviations are reported. CoAuthor CS | GCN | GAT | GraphSage
---|---|---|---
$r_{nf\\_sampling}$ | $r_{sampling}$
baseline |  0.792 $\pm 0.018$ |  0.851 $\pm 0.074$ |  0.933 $\pm 0.009$
0.1 | 0.5 | 0.713 $\pm 0.029$ | 0.758 $\pm 0.119$ | 0.922 $\pm 0.008$
0.1 | 0.731 $\pm 0.037$ | 0.738 $\pm 0.106$ | 0.907 $\pm 0.008$
0.01 | 0.699 $\pm 0.022$ | 0.778 $\pm 0.078$ | 0.891 $\pm 0.009$
0.5 | 0.5 | 0.734 $\pm 0.041$ | 0.670 $\pm 0.144$ | 0.921 $\pm 0.010$
0.1 | 0.709 $\pm 0.022$ | 0.672 $\pm 0.138$ | 0.896 $\pm 0.006$
0.01 | 0.699 $\pm 0.023$ | 0.610 $\pm 0.116$ | 0.883 $\pm 0.011$
0.75 | 0.5 | 0.723 $\pm 0.024$ | 0.635 $\pm 0.138$ | 0.901 $\pm 0.011$
0.1 | 0.696 $\pm 0.020$ | 0.650 $\pm 0.128$ | 0.877 $\pm 0.016$
0.01 | 0.684 $\pm 0.007$ | 0.644 $\pm 0.099$ | 0.860 $\pm 0.018$

Node classification. Similar conclusions to these obtained for the sampler
preprocessor can be drawn about the impact of the features_sampler (see Table
2). Out of most experiments, with constant $r_{nf\\_sampling}$ (resp. constant
$r_{sampling}$), accuracies decrease as $r_{sampling}$ decreases (resp.
$r_{nf\\_sampling}$ increases) and the performance drop lies into the $0-13\%$
range. The least variable results are obtained on CoAuthor CS dataset as the
change is $1.5\%$ at most. Both preprocessors have similar and weak effect on
models performances. Regarding Reddit and Amazon Computer, the
features_sampler preprocessor has almost always 1.5 times more impact than the
sampler. Finally, GCN is less influenced by nodes features deletion than GAT
and GraphSage. Results on Pubmed also suffer from inconsistencies with low
values of $r_{sampling}$.

Link prediction. As for the sampler preprocessor, no matter what
$r_{nf\\_sampling}$ is used, no global trend is identified as $r_{sampling}$
decreases. Results are summarized in Table 3. However, with constant
$r_{sampling}$, the impact of $r_{nf\\_sampling}$ is limited over all datasets
and models. Setting aside experiments with convergence issues, the drop in
performance stays below $9\%$ compared to the sampled experiment with the
corresponding $r_{sampling}$ and $15\%$ compared to the unsampled baseline.

##  5 CONCLUSION

We developed an experimental pipeline to ease the evaluation of Graph
Representation methods against distinctive properties, and used this pipeline
for the particular case of transactional graphs encountered in the banking
industry. This leads to the experimentation around the two properties that are
graph sparsity and asymmetric node information. We demonstrated that Graph
Representation Learning methods are robust to graph sparsity and asymmetric
node information when performing node classification. For link prediction,
although our study does not allow to draw a unified conclusion due to unstable
training encountered in many graph configurations, we did not observe
significant performance decrease even in harsh preprocessing configurations.

## ACKNOWLEDGEMENTS

This research was conducted within the "Data Science for Banking Audit"
research program, under the aegis of the Europlace Institute of Finance, a
joint program with the General Inspection at Société Générale.

## References

  * Bordes et al., 2013 Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., and Yakhnenko, O. (2013).  Translating Embeddings for Modeling Multi-relational Data.  In Burges, C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K., editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
  * Bruss et al., 2019 Bruss, C., Khazane, A., Rider, J., Serpe, R., Gogoglou, A., and Hines, K. (2019).  DeepTrax: Embedding Graphs of Financial Transactions.
  * Dwivedi et al., 2020 Dwivedi, V., Joshi, C., Laurent, T., Bengio, Y., and Bresson, X. (2020).  Benchmarking Graph Neural Networks.
  * Errica et al., 2022 Errica, F., Podda, M., Bacciu, D., and Micheli, A. (2022).  A Fair Comparison of Graph Neural Networks for Graph Classification.
  * Giles et al., 1998 Giles, C., Bollacker, K., and Lawrence, S. (1998).  CiteSeer: An Automatic Citation Indexing System.  In Proceedings of the Third ACM Conference on Digital Libraries, DL ’98, page 89–98, New York, NY, USA. Association for Computing Machinery.
  * Goyal and Ferrara, 2017 Goyal, P. and Ferrara, E. (2017).  Graph Embedding Techniques, Applications, and Performance: A Survey.
  * Grover and Leskovec, 2016 Grover, A. and Leskovec, J. (2016).  node2vec: Scalable Feature Learning for Networks.
  * Hamilton et al., 2017 Hamilton, W., Ying, R., and Leskovec, J. (2017).  Inductive Representation Learning on Large Graphs.
  * Jumper et al., 2021 Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, E., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A., Kavukcuoglu, K., Kohli, P., and Hassabis, D. (2021).  Highly accurate protein structure prediction with AlphaFold.  Nature, 596(7873):583–589.
  * Kipf and Welling, 2017 Kipf, T. and Welling, M. (2017).  Semi-Supervised Classification with Graph Convolutional Networks.
  * Kunegis and Preusse, 2012 Kunegis, J. and Preusse, J. (2012).  Fairness on the Web: Alternatives to the Power Law.  In Proceedings of the 4th Annual ACM Web Science Conference, WebSci ’12, page 175–184, New York, NY, USA. Association for Computing Machinery.
  * Li et al., 2020 Li, X., Liu, S., Li, Z., Han, X., Shi, C., Hooi, B., Huang, H., and Cheng, X. (2020).  FlowScope: Spotting Money Laundering Based on Graphs.  34:4731–4738.
  * Lv et al., 2021 Lv, Q., Ding, M., Liu, Q., Chen, Y., Feng, W., He, S., Zhou, C., Jiang, J., Dong, Y., and Tang, J. (2021).  Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks.
  * McAuley et al., 2015 McAuley, J., Targett, C., Shi, Q., and van den Hengel, A. (2015).  Image-based Recommendations on Styles and Substitutes.
  * McCallum et al., 2000 McCallum, A., Nigam, K., Rennie, J., and Seymore, K. (2000).  Automating the Construction of Internet Portals with Machine Learning.  Information Retrieval, 3(2):127–163.
  * Mikolov et al., 2013 Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013).  Efficient estimation of word representations in vector space.
  * Monti et al., 2019 Monti, F., Frasca, F., Eynard, D., Mannion, D., and Bronstein, M. (2019).  Fake News Detection on Social Media using Geometric Deep Learning.
  * Newman, 2005 Newman, M. (2005).  Power laws, Pareto distributions and Zipf’s law.  Contemporary Physics, 46(5):323–351.
  * Perozzi et al., 2014 Perozzi, B., Al-Rfou, R., and Skiena, S. (2014).  DeepWalk: Online Learning of Social Representations.
  * Salha et al., 2019 Salha, G., Hennequin, R., and Vazirgiannis, M. (2019).  Keep It Simple: Graph Autoencoders Without Graph Convolutional Networks.
  * Sen et al., 2008 Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., and Eliassi-Rad, T. (2008).  Collective Classification in Network Data.  AI Magazine, 29(3):93.
  * Veličković et al., 2018 Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (2018).  Graph Attention Networks.
  * Wang et al., 2019 Wang, M., Zheng, D., Ye, Z., Gan, Q., Li, M., Song, X., Zhou, J., Ma, C., Yu, L., Gai, Y., Xiao, T., He, T., Karypis, G., Li, J., and Zhang, Z. (2019).  Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks.
  * Wang et al., 2020 Wang, R., Li, B., Hu, S., Du, W., and Zhang, M. (2020).  Knowledge Graph Embedding via Graph Attenuated Attention Networks.  IEEE Access, 8:5212–5224.
  * Weber et al., 2018 Weber, M., Chen, J., Suzumura, T., Pareja, A., Ma, T., Kanezashi, H., Kaler, T., Leiserson, C., and Schardl, T. (2018).  Scalable Graph Learning for Anti-Money Laundering: A First Look.
  * Weber et al., 2019 Weber, M., Domeniconi, G., Chen, J., Weidele, D., Bellei, C., Robinson, T., and Leiserson, C. (2019).  Anti-Money Laundering in Bitcoin: Experimenting with Graph Convolutional Networks for Financial Forensics.
  * Zhang et al., 2020 Zhang, Z., Cui, P., and Zhu, W. (2020).  Deep Learning on Graphs: A Survey.
  * Zitnik et al., 2018 Zitnik, M., Agrawal, M., and Leskovec, J. (2018).  Modeling polypharmacy side effects with graph convolutional networks.  Bioinformatics, 34(13):i457–i466.

## APPENDIX

Table 4: Detailed datasets statistics on full graphs and sampled versions. Real transactional graph provided as example. Name | $r_{sampling}$ | $n$ | $m$ | $d$ | $p$ | $\overline{t}$ | $\gamma$ | $H_{er}$
---|---|---|---|---|---|---|---|---
Transactional graph |  | 140,674 | 190,371 | 2.71 | 1.9e-5 | 0.24 | 5.29 | 0.70
Pubmed | 1 | 19,717 | 44,327 | 4.50 | 2.3e-4 | 1.90 | 2.18 | 0.93
0.5 | 16,884 | 33,166 | 3.93 | 2.3e-4 | 1.07 | 2.28 | 0.93
0.1 | 7,216 | 8,840 | 2.45 | 3.4e-4 | 0.15 | 3.17 | 0.93
0.01 | 1,083 | 988 | 1.82 | 1.7e-3 | 0.05 | 5.34 | 0.91
CS | 1 | 18,333 | 81,894 | 8.93 | 4.9e-4 | 14.04 | 1.55 | 0.96
0.5 | 17,638 | 61,828 | 7.01 | 4.0e-4 | 7.41 | 1.65 | 0.96
0.1 | 10,269 | 16,154 | 3.15 | 3.1e-4 | 0.77 | 2.48 | 0.94
0.01 | 1,656 | 1,604 | 1.94 | 1.2e-3 | 0.06 | 5.03 | 0.91
Amazon | 1 | 13,471 | 245,861 | 36.50 | 2.7e-3 | 340.17 | 1.33 | 0.93
0.5 | 13,288 | 182,199 | 27.42 | 2.1e-3 | 164.72 | 1.38 | 0.93
0.1 | 10,856 | 4,685 | 8.63 | 8.0e-4 | 10.64 | 1.75 | 0.89
0.01 | 2,927 | 3,982 | 2.72 | 9.3e-4 | 0.20 | 4.00 | 0.84
Reddit | 1 | 232,965 | 57,307,946 | 491.99 | 2.1e-3 | 1.08e5 | 1.19 | 0.94
0.5 | 232,036 | 43,064,864 | 371.19 | 1.6e-3 | 5.42e4 | 1.20 | 0.94
0.1 | 222,650 | 10,960,320 | 98.45 | 4.4e-4 | 3.24e3 | 1.30 | 0.90
0.01 | 172,492 | 1,174,440 | 13.62 | 7.9e-5 | 48.39 | 1.70 | 0.82

In Table 4, detailed statistics about all datasets under study are provided,
along with their sampled version according to the sampling procedure defined
in Section 3.1.

$n$, $m$ and $d$ are defined in Table 1.

The fill, referred to as $p$, denotes the probability that an edge is present
between two randomly chosen nodes. For an undirected graph without loops, we
have:

$p=\frac{2m}{n(n-1)}.$ (11)

$\overline{t}$ corresponds to the number of triangles (see Figure 3), or
3-cycle, normalized by the number of nodes, $n$.

Figure 3: Illustration of a triangle pattern.

The two other statistics are derived solely from the distribution of degree
values $(d_{i})_{i=1,...,n}$, without considering the graph structure. It can
be shown that most often, the distribution of degree of any graph follows a
power law distribution, meaning that the number of nodes with degree $n$ is
proportional to $n^{-\gamma}$, with $\gamma>1$. The power law exponent gamma
is the most notable statistics. The closer the value is to one, the more
skewed towards higher degree the distribution is, thus the more nodes with
high degree the graph possesses. On the contrary, higher values indicates that
the number of neighbors falls rapidly in the graph.
This $\gamma$ value is often estimated using the following robust estimator
[Newman, 2005]:

$\gamma=1+n\left(\sum_{v_{i}\in\mathcal{V}}\ln{\frac{d_{i}}{d_{min}}}\right)^{-1},$
(12)

where $d_{min}=\underset{v_{i}\in\mathcal{V}}{\min d_{i}}$

Another metric derived from the degree distribution is the relative edge
distribution entropy, measuring the equality of the degree distribution. This
statistics approaches zero when all edges are attached to a single node, and
equals one when all nodes have equal degree. It is defined as [Kunegis and
Preusse, 2012]:

$H_{er}=\frac{1}{\ln n}\sum_{v_{i}\in\mathcal{V}}-\frac{d_{i}}{2m}\
\ln{\frac{d_{i}}{2m}}.$ (13)


#  Large Language Models and Explainable Law: a Hybrid Methodology ††thanks:
The work has been supported by the “CompuLaw” project, funded by the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation programme (Grant Agreement No. 833647).

Marco Billi0000-0002-6807-073X  Alessandro Parenti0000-0002-9855-7792
Giuseppe Pisano0000-0003-0230-8212  Marco
Sanchihttps://orcid.org/0009-0005-3264-2168 Corresponding Author: Marco
Sanchi<EMAIL_ADDRESS>University of Bologna

###### Abstract

The paper advocates for LLMs to enhance the accessibility, usage and
explainability of rule-based legal systems, contributing to a democratic and
stakeholder-oriented view of legal technology. A methodology is developed to
explore the potential use of LLMs for translating the explanations produced by
rule-based systems, from high-level programming languages to natural language,
allowing all users a fast, clear, and accessible interaction with such
technologies. The study continues by building upon these explanations to
empower laypeople with the ability to execute complex juridical tasks on their
own, using a Chain of Prompts for the autonomous legal comparison of different
rule-based inferences, applied to the same factual case.

###### keywords:

Large Language Models GPT-4 CrossJustice Explainable AI Accessibility Chain of
Thought

, and

##  1 Introduction

Rule-based systems are one of the most common type of knowledge-based system
used for automated legal reasoning, and they have found application in many
different socio-juridical domains, such as credit approval, insurance policy
determination, and public organisational structures (healthcare, welfare,
pensions, etc.)[1, 2, 3, 4]. These systems contain a knowledge base made up of
rules, often traceable to if-then statements, paired with an inference engine
which applies them to factual data related to specific cases, in a transparent
and explainable way.

In the domain of law, rule-based systems historically hold a well-known series
of problems and limitations [5, 6], including that of communicating the output
resulting from their expert legal reasoning to laypeople. Such an issue is
directly related to how the knowledge base is created: to represent the domain
rules in a computable way, a human expert must use specific, high-level
programming languages, which ties into how the system then presents its
answers to users. The difference between natural language and these
programming languages is stark, and as such it requires a way to communicate
the output to stakeholders with no computer science background. This creates a
critical problem of accessibility. As the aforecited programming languages are
not comprehensible by everyday users, they may not appreciate the output of
the rule-based model, nor understand the justification of its legal reasoning.

The problem of communicating the complex syntax and specialised terms used in
legal provisions to laypeople is not a novelty, and has been the focus of
discussion ever since the sixties [7, 8], under the plain language movement.
As a matter of fact, improving access to justice has been achieved by working
on the lexical aspect of the juridical language, analyzing layperson
ontologies [9, 10], and applying natural language processing (NLP) to simplify
legal documents [11, 12].

A different approach towards the solution of the same problem focuses instead
on developing understandable programming languages, such as Logical English
[13]. As a Controlled Natural Language, Logical English resembles natural
language in wording, thus increasing the intelligibility of the system to the
user and the programmer alike. However, this solution rarely takes into
account the possibility for users to directly interact with the system, as
these methods appear as static, and do not provide more meaningful information
to different users.

In the present paper we tackle this set of issues by developing a methodology
focused on employing LLMs for reprocessing the outcomes of rule-based systems
in a form that is accessible to laypeople. Large Language Models (LLMs) are a
kind of generative artificial intelligence system that leverages deep learning
methodologies trained on Big Data, to achieve the processing and creation of
human-like text. These models not only hold the ability to successfully
generate and manipulate natural language, but also create and model
programming languages, including coding script, and as such are being
implemented in various fields, including the legal domain. One of the most
known and used LLMs at the state of the art is GPT-4, which “exhibits human-
level performance on various professional and academic benchmarks” according
to testing and research conducted by Open
AI111https://openai.com/research/gpt-4.

We argue that LLMs, by relying on the legal reasoning provided by rule-based
systems, can carry out different legally relevant tasks ad present them in a
form that is more accessible to end-users, compared to the one produced by the
expert system. Our goal is to aid everyday users, lacking both juridical and
programming skills, in appreciating the output of rule-based systems and,
through the same means, in making more complex legal activities available to
them. To test these hypothesis, we provide a case study where we apply the
GPT-4 model to the legal reasoning of an established rule-based system,
developed using the Prolog language: CrossJustice[14]. In particular, the
experiments will focus on providing an accessible natural language explanation
of the output of the expert system and operating a comparison between the
applications of two different legal sources to the same specific case.

##  2 Methodology

In spite of their capabilities as natural language processors, LLMs have
proven to struggle when they are employed for reasoning about legal norms and
their applications. In these cases, such systems have shown to be plagued by
hallucinations, lack of coherence and misinterpretation of the norms and the
specific context [15, 16]. For this reason, we distance ourselves from these
approaches and, instead, bring about a a new way to explore the relationship
between Large Language Models and rule-based systems that can enhance the
access of everyday users to law.

We therefore introduce a hybrid approach where the rule-based system is
employed as the legal reasoner, automatically applying the relevant norms to
the specific case provided. The rule-based system’s output, presented in the
form of a logic programming language, is then fed into GPT-4, for it to be
reprocessed and used to carry out different operations.

###  2.1 Objectives

In developing this system, our main goal is to allow the LLM to output a
result in a form that is as accessible as possible to end users, as well as
providing all the necessary legal elements to enable a clear understanding of
the situation at hand.

To achieve this, we focus on one hand on giving instructions on the structure
and lexicon it should be used in the answer, and on the other on ensuring that
the direct relation between the specific case provided and the final outcome
of the rule-based system is clearly stated in the explanation. In other words,
we want the LLM to report and explain which specific conditions of the norm
were applied, and exactly which facts triggered this application.

With regards to the more complex legal tasks, we aim at performing a legal
analysis on different texts, in order to support both citizens and legal
professionals. Our main focus is to ensure that the contrasts between legal
sources were clearly highlighted and that the system could explain how these
variations may be relevant for the user. We believe all these information to
be crucial for allowing the user to have a clear understanding of their legal
position.

Finally, A fundamental step in our methodology is represented by repeating the
experiment several times, using the same prompts on the same specific case.
Given the intrinsic non-deterministic nature of LLMs, such a procedure allows
us to verify whether the approach can provide correct results in a stable and
consistent way. To decrease inconsistencies, we set the model’s temperature to
its minimum, limiting the creativity and inventiveness of GPT-4, thus forcing
it to focus on the extraction of legal inferences made by the expert system
and the identification of relevant, case-based facts, to be presented in
natural language.

###  2.2 Evaluation Criteria

The results of the approach described above are evaluated according to
criteria capable of validating the accessibility and legal soundness of the
output. The criteria appear as follows:

  * •

Correctness: accuracy in grasping key points, legal issues and essential
information by the LLM. This criteria is used to exclude all output which does
not match the meaning and legal argumentation of the source provided,
overcoming any misinterpretations and misapplications of juridical norms,
under a lenses of juridical validation.

  * •

Form: coherence, readability and simplification of legal vocabulary
(legalese), to maximise the accessibility of everyday users to the output.
Moreover, it verifies the correspondence between input and output in terms of
structure and presentation, under the lenses of formal validation.

  * •

Completeness: inclusion of all the elements requested by the prompt, with
particular emphasis on those necessary to evaluate the success of operation.
This criteria is used to exacerbate output that did not consider key facts
about the overall process, under a lenses of substantial validation.

##  3 Case Study

The CrossJustice platform is a rule-based system capable of automatic legal
reasoning in the domain of criminal law, which provides its users with
meaningful information about their rights and freedoms as suspects, or
accused, of criminal conduct. CrossJustice was identified as suitable as it
holds all the main characteristics we ought to look for in order to ground our
approach: it explains its inferences, it uses the high-level Prolog
programming language to do so, it reasons about extremely different categories
of rights, and finally it applies EU law as well as Member State law to
justify its inferences.

To introduce the factual scenario provided in Listing 1, let us imagine that a
person, named Mario, is involved in criminal proceedings taking place in
Poland (line 10), and does not speak the polish language (line 13). Also, he
has been presented with a document charging him of his crime (line 12).
According to Article 3, paragraph 2, such a document is to be considered
essential (line 11). Thus, according to Article 3, paragraph 1, of the
European Directive 2010/64, Mario has the right to receive a translation of
this document, which is essential to ensure that he is able to exercise his
right of defence and to safeguard the fairness of the proceedings. This is the
main right, presented in the CrossJustice platform in lines 8-9, followed by a
recap of the specific conditions needed for that right to be granted.

In the CrossJustice system we also express the relation between a primary
right and the further rights that expand the meaning or the implementation of
the main.

Auxiliary rights do no directly regulate the legal sphere of the defendant,
but are depending upon, or reasonably linked to a primary right. This
connection can either be a temporal one, where the right exists only after the
primary one has been applied, or a subjective one, which implies that the
defendant has particular needs. In this case the defendant, Mario, has the
right to have the costs of the translator covered by the State under Article 4
(lines 15-23).

The property norms are used to explicitly define certain characteristics, or
details, of a right. The difference between auxiliary rights and properties
consists in the fact that the latter exists irrespective of the presence of
the defendant, because it attains directly to the main right. In this case,
according to Article 3, paragraph 7, an oral translation of essential
documents may be provided instead of a written translation on condition that
such oral translation does not prejudice the fairness of the proceedings
(lines 25-34).

Listing 1: Right to Translation - EU Directive

⬇

1directive_2010_64 - art3_1

2

3Article 3

4Option: essentialDocument

5

6Explanation:

7

8has_right(right_to_translation, dir, art3_1, mario, essentialDocument)

9 has_right(art3_1, mario, right_to_translation, essentialDocument)

10 proceeding_language(mario, polish) [FACT]

11 essential_document(art3_2, mario, documents)

12 person_document(mario, charge) [FACT]

13 not(person_understands(mario, polish))

14

15Auxiliaries:

16

17art4 - cost - state

18

19Article 4

20Explanation:

21

22auxiliary_right(art4, art3_1, mario, cost, state)

23 auxiliary_right(art4, mario, cost, state)

24

25Properties:

26

27art3_7 - form - oral

28

29Article 3.7

30Explanation:

31

32right_property(art3_7, art3_1, mario, form, oral)

33 right_property(art3_7, mario, form, oral)

34 not(proceeding_event(mario, prejudice_fairness))

We will now explore the outcome of the same factual scenario, as applied in
the Polish legal system (Listing 2). Starting from the same basic facts -
Mario is involved in criminal proceedings taking place in Poland (line 10),
and does not speak the polish language (line 11) - the polish legislator
states that, according to Article 204, paragraph 2, of the Polish Code of
Criminal Procedure, Mario has the right to translation (lines 8-9) if there is
a need to translate a document drawn up in a foreign language (line 12). A
document presenting a charge is one such document (line 13).

Furthermore, according to Article 618, paragraph 1, part 7, Mario has the
right to have the costs of the translator covered by the State (lines 15-23).

There is no applicable article regarding the form of the translation.

Listing 2: Right to Translation - Polish Law

⬇

1directive_2010_64_pl - article204_2

2

3Article 204.2 code of criminal procedure

4Option: documents

5

6Explanation:

7

8has_right(right_to_translation, pl, article204_2, mario, documents)

9 has_right(article204_2, mario, right_to_translation, documents)

10 proceeding_language(mario, polish) [FACT]

11 not(person_understands(mario, polish))

12 person_document(mario, translation_needed)

13 person_document(mario, charge) [FACT]

14

15Auxiliaries:

16

17article618_7 - cost - state

18

19Article 618.1.7 code of criminal procedure

20Explanation:

21

22auxiliary_right(article618_7, article204_2, mario, cost, state)

23 auxiliary_right(article618_7, mario, cost, state)

Having established our platform of choice and case study, we are now going to
focus on prompt engineering for the achievement of two different tasks.

##  4 Natural Language Translation

For the first task, our goal was for GPT-4 to be capable of extracting from
the Prolog trace the following pieces of information:

  * •

A simplified summary of the norms relevant to the inference;

  * •

The list of rights granted to the user according to the input facts
representing the case;

  * •

A description of the inference process that led the system to its solution.

With these objectives in mind, we directed our efforts towards achieving the
most flexible and versatile prompt, able to be applied to any of the
inferences from the Crossjustice system.

There were however several problems we encountered. An initial challenge was
to obtain a fixed structure in the output, which was often presented in
different formatting (e.g. bullet lists, numbered lists, free-form text). We
found that it was necessary to give a fixed structure for the LLM to follow,
in order to decrease the degree of variance between answers and maximise
repeatability and reliability.

Second, we required the LLM to take into consideration all legal terms and
facts from the scenario, and to apply those to build the explanation, as it
had a propensity to miss key facts if not well prompted.

Finally, the most challenging aspect was to overcome the tendency of the model
to ascribe meaning to legal text where not explicitly mentioned nor provided.
In particular, the meaning was often guessed based on the most general
significance, disregarding the specificity of juridical norms and legal
lexicon.

Through trial and error, we achieved the following prompt:

⬇

1You have been provided a Prolog inference tree using a legal norm in a
specific case (Prolog Tree).

2Provide the following info according to the given structure:

3

4Summary: simplified text of the legal norm. Use everyday langaguage with a
serious register. Add any information you have on the source of law;

5What Rights do You Have: what rights do you have according to the Prolog
explanation;

6Why do You Have Them: inference steps and reasoning that led to the the
rights. Use all the Prolog terms in the explanation explicitly referencing the
original Prolog when needed.

7

8Use enumerations in the ’What Rights do You Have’ and ’Why do You Have Them’
sections if needed.

First, we introduced the object and the form of the input data (line 1).
Second, we provided a fixed structure to follow (Summary - What Rights do You
Have - Why do You Have Them). This is aimed at guaranteeing that the model
follows the given formal criteria, in order to maximise repeatability and
reliability.

Furthermore, we specifically requested the system to use all the Prolog terms
in the explanation, with explicit references to the original (line 6). This is
aimed at providing an easier way to read and evaluate the answer, both from a
substantial and juridical point of view.

Finally, an example of the answers we obtained using the prompt follows:

⬇

1Summary: According to Directive 2010/64, Article 3.1, if you are involved in
a legal proceeding and do not understand the language of the proceeding, you
have the right to translation of essential documents. The cost of the
translation will be covered by the state, as stated in Article 4.
Additionally, under Article 3.7, you may have the right to oral translation if
it does not prejudice the fairness of the proceeding.

2

3What Rights do You Have:

41. Right to translation of essential documents.

52. State covering the cost of translation.

63. Right to oral translation, if it does not prejudice the fairness of the proceeding.

7

8Why do You Have Them:

91. You have the right to translation of essential documents because:

10 - Your proceeding language is Polish (proceeding_language(mario, polish)).

11 - You have essential documents related to the charge
(person_document(mario, charge)).

12 - You do not understand Polish (not(person_understands(mario, polish))).

13 - This right is granted under Directive 2010/64, Article 3.1
(has_right(right_to_translation, dir, art3_1, mario, essentialDocument)).

142. The state will cover the cost of translation because:

15 - This auxiliary right is granted under Directive 2010/64, Article 4
(auxiliary_right(art4, art3_1, mario, cost, state)).

163. You have the right to oral translation if it does not prejudice the fairness of the proceeding because:

17 - This right is granted under Directive 2010/64, Article 3.7
(right_property(art3_7, art3_1, mario, form, oral)).

18 - The fairness of the proceeding is not prejudiced
(not(proceeding_event(mario, prejudice_fairness))).

It appears clear how GPT-4 was able to present its output in an accessible and
readable way, upholding the instructions given about structure and formatting,
even after much repetition (formal validation). However, we cannot say that
the next two criteria have been fully satisfied. GPT-4 did not include all the
relevant Prolog facts used to apply juridical reasoning to the specific case
(substantial validation), as it failed to grasp and correctly represent one of
the legal inferences of CrossJustice, based on the application of the the sub-
rule (juridical validation).

We noticed how GPT-4 struggles in giving the exact meaning to Prolog terms
when these may be open to different interpretations. In particular, in line
11, GPT reports the fact that Mario has an essential document related to the
charge as a condition of the Prolog rule. The predicate has been
misinterpreted, and a better version would have been: ”a document containing a
charge”, or ”a document which states that the person has been charged”.
Furthermore, this has lead the LLM to mix up two facts, of which one is a
condition for the application of the other. In this case, the correct solution
would have been to identify that a document would be considered essential if
that document was a charge.

The same prompt was applied to the Polish legal source:

⬇

1Summary: According to Article 204.2 of the Code of Criminal Procedure, if you
are involved in a legal proceeding and do not understand the language of the
proceeding (Polish in this case), you have the right to translation of
necessary documents. Additionally, Article 618.1.7 states that the cost of
this translation will be covered by the state.

2

3What Rights do You Have:

41. Right to translation of necessary documents.

52. The cost of translation will be covered by the state.

6

7Why do You Have Them:

81. You have the right to translation because:

9 a. The proceeding language is Polish (proceeding_language(mario, polish)).

10 b. You do not understand Polish (not(person_understands(mario, polish))).

11 c. You have a document that needs translation (person_document(mario,
translation_needed)).

12 d. The document in question is a charge (person_document(mario, charge)).

13 e. Therefore, you have the right to translation according to Article 204.2
(has_right(right_to_translation, pl, article204_2, mario, documents)).

14

152. You have the right to have the cost of translation covered by the state because:

16 a. The auxiliary right in Article 618.1.7 is connected to the right in
Article 204.2 (auxiliary_right(article618_7, article204_2, mario, cost,
state)).

17 b. Therefore, you have the right to have the cost of translation covered by
the state (auxiliary_right(article618_7, mario, cost, state)).

Here, all 3 criteria have been fully satisfied. The performance of the LLM
visibly improved, as both the sub-condition that the person Mario is presented
with a document containing a charge (line 12), and that because of this Mario
has a document that needs translation (line 11), have been correctly
identified. This highlights the contrast between the Natural Language
translation of the same Prolog fact (”person_document(mario, charge”)) applied
to the two corresponding legal sources, which has been interpreted
differently, for no apparent reason. These small but substantial mistakes
might be a consequence of the limited context provided to the model. However,
even when experimenting by providing the LLM the full text of the relevant
legal norms, we found that it did not cause a substantial improvement in
performances, nor in the language and terminology used.

Overall, we could not find a way to reliably prompt the system to correctly
identify and present all sub-rules and conditions, although mistakes were
significantly lowered throughout the experimenting process.

##  5 Comparison of legal sources

Building upon the results of Task 1, we followed by instructing GPT-4 to enact
legal comparison between two sources. To reach a successful result, we
experimented with several prompts. We also tested beforehand the capacity of
the LLM to produce legal comparison directly on the text of the norm; however,
the results were extremely poor.

We noticed that, especially for more complex tasks, employing a Chain of
Thought[17, 18] approach decreases the probabilities of mistakes in the final
answer. Chain of Thought prompting consists of having an LLM generate a series
of intermediate reasoning steps necessary to get to the final answer.

To implement this method, we first tried to have a single prompt describing
multiple logical steps, ranging from extracting the information to the
analysis of the differences. However, we found inconsistency in the answers
provided by the LLM, possibly because of the length of the step by step
process and the high number of actions required. We therefore decided to
divide the prompt into two sub-prompts, each related to a specific task. The
first one – described in Section 4 – requiring an explanation of both the
legal sources provided, and the second one requiring their comparison based on
the output of the previous step. Doing so, we created a Chain of Prompts
capable of enabling laypeople not only to better access and understand legal
reasoning, but also to execute complex juridical operations such as comparison
in full autonomy.

Our goal was for the model to identify the following:

  * •

Differences and contrasts in the legal reasoning and application of the norms
to the case between the sources.

  * •

Diverging interpretations of the rights present in the legal sources and how
those may lead to different applications of the law to the specific case.

This subsequent operation resulted in the application of the following prompt
to the output obtained from the first step:

⬇

1You have received two legal sources. Compare them according to the following
steps:

21 - COMPARE the differences in the reasonings. USE all the inference steps
and describe how they influence the final outcome;

32 - analyse potential consequences of the differences from step 1 only based
on the data provided.

Here we followed the same principles as per the initial prompt. First, we
introduced the object of the text given as input, although now in natural
language, so no clarification regarding the form was needed. Second, we fixed
the structure to be followed when providing an answer. Again, we required the
system to use all the inference steps in order to minimise the risk of
misinterpretation and hallucinations.

An example of the final results gathered using the Chain of Prompts in
discussion follows:

⬇

11. Comparison of differences in reasonings:

2

3Directive 2010/64:

4- The right is granted by Article 3.1 of Directive 2010/64.

5- The right is specifically for the translation of "essential" documents.

6- The state covers the costs of translation as per Article 4.

7- There is an additional right to oral translation if it does not prejudice
the fairness of the proceeding, as per Article 3.7.

8

9Code of Criminal Procedure:

10- The right is granted by Article 204.2 of the Code of Criminal Procedure.

11- The right is specifically for the translation of "necessary" documents.

12- The state covers the costs of translation as per Article 618.1.7.

13- There is no mention of an additional right to oral translation.

14

15The differences in the reasonings lie in the terminology used for the
documents that require translation (essential vs. necessary) and the presence
of an additional right to oral translation in Directive 2010/64. The sources
of the rights also differ, with one being a European directive and the other
being a national code of criminal procedure.

16

172. Potential consequences of the differences:

18

19- The difference in terminology (essential vs. necessary) might lead to
different interpretations of which documents qualify for translation. This
could potentially affect the scope of the right to translation and the number
of documents that are translated for the individual.

20- The presence of an additional right to oral translation in Directive
2010/64 could provide more comprehensive language assistance to the individual
during the legal proceeding, ensuring better understanding and participation
in the process. This right is not mentioned in the Code of Criminal Procedure,
which might limit the language assistance provided to the individual.

21- The difference in the sources of the rights (European directive vs.
national code) could lead to potential conflicts or discrepancies in the
application of these rights, depending on the jurisdiction and the specific
case. This might affect the individual’s access to these rights and the
overall fairness of the legal proceeding.’

With regards to the formal validation criteria, GPT-4 was able to present its
output in an accessible and readable way, upholding the instructions given
about structure and formatting, even after much repetition.

Before going into the details of the next two criteria, a clarification is
necessary. One of the issues we faced when using a single prompt lied in the
LLM’s understanding of the words ”documents” and ”needed”. We observed a
tendency to ignore the context of the two words when together, and replace
them with the expression ”all documents”, regardless of our insistence on the
use of the correct terms as extracted by the Prolog trace. This created
multiple issues from both a substantial and a juridical point of view, as it
missed inference steps in its comparison, as well as misrepresented legal
terms and inferences. After changing our approach, we saw a steep increase in
the quality of the outcome, as can be seen from the example above.

Currently, one of the few remaining limitations is due to the Prolog
representation of the norm. In this case, GPT-4 cannot reliably infer that the
Code of Criminal Procedure belongs to Poland. This is achieved in the
CrossJustice platform through the use of a visual interface, while in the
Prolog representation this is done through the use of the suffix _pl.
Furthermore, the LLM still has trouble highlighting the fact that the same
document has two different interpretations according to the two applicable
legal sources.

To conclude with the juridical validation criteria, the LLM has correctly
grasped the relevant terms and compared them without changing their meaning.
However, it does not include all the relevant inference steps used to apply
juridical reasoning to the specific case (substantial validation).

##  6 Conclusions

This paper explores the opportunities and limitations regarding the use of
LLMs for the autonomous generation of accessible natural language
explanations, within the context of rule-based systems in the legal domain.
Moreover, it tackles the possibility of building upon these explanations to
empower stakeholders with the ability of enacting autonomous legal tasks, such
as comparing the application of different norms to the same specific case.

To reach our first objective, we provided a methodology for the engineering of
flexible prompts, able to process juridical inferences in a stable, repeatable
and simplified way. We followed by successfully applying our hybrid approach
to the CrossJustice platform - a system based in Prolog language for the
domain of criminal law - showing our method to be effective in making the
rule-based reasoning accessible, while preserving its substantial, juridical
and formal validity.

After establishing such sound foundation, we moved to our second objective by
creating a chain of prompts able to process different rule-based outputs and
their explanations. Our goal was to produce legal comparison by identifying
the relevant juridical and factual differences present amongst inferences
relating to the same specific case. The methodology proved to be once again
successful, showing the potential of a hybrid approach based on the expert
reasoning of rule-based systems, paired with the versatility of LLMs, which
opens the door to various legal operations, as shown in the case study.

On this note, future works would expand our methodology by further developing
the Chain of Prompts used in our trials, exploring the potential modularity of
such an approach. By this, we mean the creation of a versatile and flexible
initial prompt for the natural language translation and explanation of rule-
based inferences, followed by the engineering of multiple different and
subsequent prompts, each dedicated to a different legal operation. Those would
be applied accordingly to the output of the first, creating chains going
beyond normative comparison, enabling more complex and differentiated
operations.

Finally, our methodology could easily embrace the multilingual nature of
European Law given the capabilities of state-of-the-art LLMs, thus
contributing to overcome language barriers in the fruition of legal
technology, as well as to bolster access to European and Member State law.

## References

  * [1] Lymer A, Richards K.  A Hybrid-Based Expert System for Personal Pension Planning in the UK.  Intelligent Systems in Accounting, Finance and Management. 1995;4(1):71-88.
  * [2] Michaelsen RH.  An expert system for federal tax planning.  Expert Systems. 1984;1(2):149-67.
  * [3] Masri N, Sultan YA, Akkila AN, Almasri A, Ahmed A, Mahmoud AY, et al.  Survey of rule-based systems.  International Journal of Academic Information Systems Research (IJAISR). 2019;3(7):1-23.
  * [4] van Melle W.  MYCIN: a knowledge-based consultation program for infectious disease diagnosis.  International Journal of Man-Machine Studies. 1978;10(3):313-22.  Available from: https://www.sciencedirect.com/science/article/pii/S0020737378800492.
  * [5] Susskind R.  Expert systems in law.  Oxford University Press, Inc.; 1987.
  * [6] Schauer F.  Playing by the rules: A philosophical examination of rule-based decision-making in law and in life.  Clarendon Press; 1991.
  * [7] Mellinkoff D.  The Language of the Law.  Little, Brown & Co; 1963.
  * [8] Charrow RP, Charrow VR.  Making legal language understandable: A psycholinguistic study of jury instructions.  Columbia law review. 1979;79(7):1306-74.
  * [9] Uijttenbroek EM, Lodder AR, Klein MC, Wildeboer GR, Van Steenbergen W, Sie RL, et al.  Retrieval of case law to provide layman with information about liability: Preliminary results of the best-project.  Springer; 2008.
  * [10] Fernández-Barrera M, Casanovas P.  From user needs to expert knowledge: mapping laymen queries with ontologies in the domain of consumer mediation.  In: AI Approaches to the Complexity of Legal Systems. Models and Ethical Challenges for Legal Systems, Legal Language and Legal Ontologies, Argumentation and Software Agents: International Workshop AICOL-III, Held as Part of the 25th IVR Congress, Frankfurt am Main, Germany, August 15-16, 2011. Revised Selected Papers 3. Springer; 2012. p. 286-308.
  * [11] Paquin LC, Blanchard F, Thomasset C.  Loge–expert: from a legal expert system to an information system for non-lawyers.  In: Proceedings of the 3rd international conference on Artificial intelligence and law; 1991. p. 254-9.
  * [12] Garimella A, Sancheti A, Aggarwal V, Ganesh A, Chhaya N, Kambhatla N.  Text Simplification for Legal Domain:$\\{$I$\\}$ nsights and Challenges.  In: Proceedings of the Natural Legal Language Processing Workshop 2022; 2022. p. 296-304.
  * [13] Kowalski R.  Logical english.  Proceedings of Logic and Practice of Programming (LPOP). 2020.
  * [14] Billi M, Calegari R, Contissa G, Pisano G, Sartor G, Sartor G.  Explainability Through Argumentation in Logic Programming.  In: Proceedings of the International Conference on Logic Programming 2021 Workshops co-located with the 37th International Conference on Logic Programming (ICLP 2021), Porto, Portugal (virtual), September 20th-21st, 2021. vol. 2970 of CEUR Workshop Proceedings. CEUR-WS.org; 2021. Available from: https://ceur-ws.org/Vol-2970/causalpaper2.pdf.
  * [15] Savelka J, Ashley KD, Gray MA, Westermann H, Xu H.  Explaining Legal Concepts with Augmented Large Language Models (GPT-4).  arXiv preprint arXiv:230609525. 2023.
  * [16] Westermann H, Meeùs S, Godet M, Troussel AC, Tan J, Savelka J, et al.  Bridging the Gap: Mapping Layperson Narratives to Legal Issues with Language Models.  In: Proceedings of the 6th Workshop on Automated Semantic Analysis of Information in Legal Text co-located with the 19th International Conference on Artificial Intelligence and Law (ICAIL 2023), Braga, Portugal, 23rd September, 2023. vol. 3441 of CEUR Workshop Proceedings. CEUR-WS.org; 2023. p. 37-48.  Available from: https://ceur-ws.org/Vol-3441/paper5.pdf.
  * [17] Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, et al.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  In: NeurIPS; 2022. Available from: http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.
  * [18] Kojima T, Gu SS, Reid M, Matsuo Y, Iwasawa Y.  Large language models are zero-shot reasoners.  Advances in neural information processing systems. 2022;35:22199-213.