# Causal Inference Using LLM-Guided
Discovery

Aniket Vashishtha
Microsoft Research, India
<EMAIL_ADDRESS>&Abbavaram Gowtham Reddy
IIT Hyderabad, India
<EMAIL_ADDRESS>&Abhinav Kumar
Massachusetts Institute of Technology, USA
<EMAIL_ADDRESS>&Saketh Bachu
IIT Hyderabad, India
<EMAIL_ADDRESS>&Vineeth N Balasubramanian
IIT Hyderabad, India
<EMAIL_ADDRESS>&Amit Sharma
Microsoft Research, India
<EMAIL_ADDRESS>
###### Abstract

At the core of causal inference lies the challenge of determining reliable
causal graphs solely based on observational data. Since the well-known
backdoor criterion depends on the graph, any errors in the graph can propagate
downstream to effect inference. In this work, we initially show that complete
graph information is not necessary for causal effect inference; the
topological order over graph variables (causal order) alone suffices. Further,
given a node pair, causal order is easier to elicit from domain experts
compared to graph edges since determining the existence of an edge can depend
extensively on other variables. Interestingly, we find that the same principle
holds for Large Language Models (LLMs) such as GPT-3.5-turbo and GPT-4,
motivating an automated method to obtain causal order (and hence causal
effect) with LLMs acting as virtual domain experts. To this end, we employ
different prompting strategies and contextual cues to propose a robust
technique of obtaining causal order from LLMs. Acknowledging LLMs’
limitations, we also study possible techniques to integrate LLMs with
established causal discovery algorithms, including constraint-based and score-
based methods, to enhance their performance. Extensive experiments demonstrate
that our approach significantly improves causal ordering accuracy as compared
to discovery algorithms, highlighting the potential of LLMs to enhance causal
inference across diverse fields.

##  1 Introduction

Causal inference plays a pivotal role across scientific disciplines, aiding
researchers in uncovering fundamental causal relationships and how they affect
observed phenomena. For example, causal inference is used to discern the
causes of diseases and design effective interventions for diagnosis and
treatment in epidemiology (Mahmood et al., 2014), to evaluate policy impact
based on observational studies in economics (Imbens & Rubin, 2015), and to
understand the effects of pollution on ecosystems in environmental science
(Boslaugh, 2023). A key technical question for these studies is estimating the
causal effect of variables on a specific outcome variable.

Inferring causal effect from observational data, however, is a challenging
task because the effect estimate depends critically on the causal graph
considered in the analysis. While there has been progress in graph discovery
algorithms, especially for specific parametric settings (Shimizu et al., 2006;
Hoyer et al., 2008b; Hyvärinen et al., 2010; Rolland et al., 2022), studies on
real-world datasets such as from atmospheric science (Huang et al., 2021) and
healthcare (Tu et al., 2019) show that inferring the causal graph from data
remains a challenging problem in practice (Reisach et al., 2021). As a result,
causal inference studies often rely on a human expert to provide the causal
graph.

In this paper, we show that obtaining the full causal graph is not necessary.
Instead, a simpler structure, the topological ordering over graph variables
(causal order) is sufficient for causal effect inference. Moreover, providing
the order between variables is the right question to ask experts because order
depends only on the variables under question, unlike existence of a graph edge
that depends on which other variables are present (to account for direct and
indirect effects). For example, consider the data-generating process, _lung
cancer_ $\rightarrow$ _doctor visit_ $\rightarrow$ _positive Xray_. If an
expert is asked whether there should be a causal edge from _lung cancer_ to
_positive Xray_ , they would answer “Yes” (indeed, such an edge exists in the
BNLearn Cancer dataset (Scutari & Denis, 2014)). However, if they are told
that the set of observed variables additionally includes doctor visit, then
the correct answer would be to not create a direct edge between lung cancer
and positive Xray, but rather create edges mediated through _doctor visit_.
However, note that the causal order, lung cancer $\prec$ positive Xray remains
the same in both settings ($a\prec b$ indicates that $a$ occurs before $b$ in
a casual process).

Figure 1: The LLM-augmented causal inference process based on inferring causal
order. We propose a triplet-based prompting technique to infer all three-
variable subgraphs and aggregate them using majority voting to produce a
causal order. The causal order can then be used to identify a valid backdoor
adjustment set. Ties in causal order are broken using another LLM (e.g.,
GPT-4). For robustness, LLM-generated causal order may be used in conjunction
with discovery algorithms like PC or CaMML.

Our second contribution is to show that large language models (LLMs) like
GPT-3.5 (Hagendorff et al., 2022) and GPT-4 (Peng et al., 2023) can be used to
approximate experts’ capability to provide causal order, thereby automating
the process of causal inference. Extending results using LLMs for pairwise
causal discovery (Kıcıman et al., 2023), we find that LLMs can provide
accurate causal order for a diverse set of benchmark graphs. To do so, we
propose a novel triplet-based prompting strategy that asks LLM to consider
three variables at once, compared to the pairwise prompts employed in past
work (Kıcıman et al., 2023; Willig et al., 2022; Long et al., 2023; 2022).
Causal order outputted using the triplet-based prompt outperforms pairwise
prompts across all benchmark datasets we considered. Importantly, it avoids
cycles in the predicted order whereas LLM outputs from pairwise prompts often
yield cycles.

Still, LLMs can exhibit unknown failure modes. Therefore, a more principled
way is to adapt existing graph discovery algorithms to utilize LLM output. To
this end, we present two algorithms based on constraint-based and score-based
discovery algorithms respectively. The first uses causal order from an LLM to
orient the undirected edges outputted by a constraint-based algorithm such as
PC (Spirtes et al., 2000). The second algorithm utilizes the LLM causal order
as a prior to a score-based algorithm like CaMML (Wallace et al., 1996).
Results show that LLM-augmented algorithms outperform the base causal
discovery algorithms in determining the causal order. The overall methodology
is depicted in Figure 1. Our contributions can be summarized as follows.

  * •

Based on the backdoor criterion, we prove that causal graph is not required
for effect estimation; only the causal order suffices. Empirically, we show
that causal order correlates strongly with the accuracy of effect inference.

  * •

We provide a novel prompting strategy based on triplets and show that LLMs
like GPT-3.5 can be used to obtain causal order for a diverse range of
datasets.

  * •

We propose two algorithms combining causal discovery algorithms with LLM
output and show that the final causal order is substantially more accurate
than the discovery algorithms alone.

##  2 Related Work

Combining graph discovery and causal inference. Historically, causal discovery
and causal effect inference have been studied separately. Graph discovery
algorithms can broadly be divided into (i) algorithms using conditional
independence tests (constraint-based) (Glymour et al., 2019); (ii) algorithms
using a score function to evaluate predicted graph (score-based) (Glymour et
al., 2019); (iii) algorithms that determine a causal order and then infer
edges (order-based) (Rolland et al., 2022; Teyssier & Koller, 2005); and (iv)
deep learning-based methods that formulate an optimization problem based on
acyclicity and sparsity constraints (Zheng et al., 2018; Lachapelle et al.,
2020). Causal discovery methods are evaluated on error wrt. the true graph,
e.g., using the structural hamming distance (SHD) (Acid & de Campos, 2003;
Tsamardinos et al., 2006). In contrast, causal inference methods focus on the
estimation of causal effect given a causal graph (Pearl, 2009); the graphs are
assumed to be known. A natural way to combine these approaches to use the
graph outputted by discovery algorithms in inference methods, as in (Hoyer et
al., 2008a; Mooij et al., 2016; Maathuis et al., 2010; Gupta et al., 2022). In
this paper, we show that there exists a simpler way to combine the two
approaches: only a causal order is needed instead of the full graph.

Knowledge-driven Causal Discovery: Prior knowledge has been used in causal
discovery literature (Hasan & Gani, 2022; Constantinou et al., 2023; Heckerman
& Geiger, 2013; Teshima & Sugiyama, 2021; O’Donnell et al., 2006; Wallace et
al., 1996). These methods rely on prior knowledge such as domain expert
opinions and documented knowledge from randomized controlled trials (RCT).
Various priors have been studies in literature, including the priors of the
form edge existence, forbidden edge, ancestral constraints (Constantinou et
al., 2023; Ban et al., 2023). Prior knowledge significantly reduces the search
space over all possible causal graphs.

Recent advancements in LLMs has led to more attention towards knowledge-driven
causal discovery (Kıcıman et al., 2023; Ban et al., 2023; Long et al., 2023;
Willig et al., 2022). Unlike causal discovery algorithms that use statistical
patterns in the data, LLM-based algorithms use metadata such as variable
names. Most of these methods use only LLMs to predict the causal relationships
among a set of variables (Kıcıman et al., 2023; Long et al., 2023; Willig et
al., 2022; Long et al., 2022). Recent work also shows how LLMs can be used as
priors for existing score-based causal discovery methods (Ban et al., 2023).
However, the focus of these works has been on minimizing graph error metrics
such as SHD. Instead, we focus on the downstream causal inference task and
choose causal order as the metric since it directly correlates with accuracy
in effect estimation whereas SHD does not.

LLM Prompting Strategies for Causal Discovery: Existing LLM-based algorithms
for graph discovery (Kıcıman et al., 2023; Long et al., 2022; Ban et al.,
2023) use a pairwise prompt, essentially asking “does A cause B” with varying
levels of prompt complexity. Extending this line of work, we propose a
triplet-based prompt that provides more accurate answers and avoids cycles
when querying relationships between variables. As a result, our triplet-based
prompt may be of independent interest to improve LLM-based graph discovery. We
also explore the chain-of-thought prompting strategy (Wei et al., 2022) in our
experiments.

##  3 Background and Problem Formulation

Let $\mathcal{G}(\textbf{X},\textbf{E})$ be a causal directed acyclic graph
(DAG) consisting of a set of variables $\textbf{X}=\\{X_{1},\dots,X_{n}\\}$
and a set of directed edges E among the variables in $\mathbf{X}$. A directed
edge $X_{i}\rightarrow X_{j}\in\mathbf{E}$ denotes the direct causal influence
of the variable $X_{i}$ on the variable $X_{j}$. Let
$pa(X_{i})=\\{X_{k}|X_{k}\rightarrow X_{i}\\}$,
$de(X_{i})=\\{X_{k}|X_{k}\leftarrow\dots\leftarrow X_{i}\\}$ denote the set of
parents and descendants of $X_{i}$ respectively. We focus on a downstream
application of causal graph discovery called causal effect inference, defined
as follows.

######  Definition 3.1.

(Average Causal Effect (Pearl, 2009)) The average causal effect (ACE) of a
variable $X_{i}$ on a variable $X_{j}$ is defined as

$ACE_{X_{i}}^{X_{j}}=\mathbb{E}[X_{j}|do(X_{i}=x_{i})]-\mathbb{E}[X_{j}|do(X_{i}=x_{i}^{*})]$
(1)

In Defn 3.1, $X_{i}$ is called the treatment variable and $X_{j}$ is called
the target variable. $do(X_{i}=x_{i})$ denotes an external intervention to the
variable $X_{i}$ with the value $x_{i}$. The interventional quantity
$\mathbb{E}[X_{j}|do(X_{i}=x_{i})]$ is different from conditional
$\mathbb{E}[X_{j}|X_{i}=x_{i}]$ since it involves setting the value of $X_{i}$
rather than conditioning on it. To estimate the quantity
$\mathbb{E}[X_{j}|do(X_{i}=x_{i})]$ from observational data, the backdoor
adjustment formula is used.

######  Definition 3.2.

(Back-door Adjustment (Pearl, 2009)) Given a DAG $\mathcal{G}$, a set of
variables $\mathbf{Z}$ satisfies back-door criterion relative to a pair of
treatment and target variables ($X_{i},X_{j}$) if

  * (i)

no variable in $\mathbf{Z}$ is a descendant of $X_{i}$; and

  * (ii)

$\mathbf{Z}$ blocks every path between $X_{i}$ and $X_{j}$ that contains an
arrow into $X_{i}$.

where a path in a causal DAG is a sequence of unique vertices
$X_{i},X_{i+1},\dots,X_{j}$ with a directed edge between each consecutive
vertices $X_{k}$ and $X_{k+1}$ (either $X_{k}\rightarrow X_{k+1}$ or
$X_{k+1}\rightarrow X_{k}$). If a set of variables $\mathbf{Z}$ satisfies the
back-door criterion relative to $(X_{i},X_{j})$,
$\mathbb{E}[X_{j}|do(X_{i}=x_{i})]$ can be computed using the formula:
$\mathbb{E}[X_{j}|do(X_{i}=x_{i})]=\mathbb{E}_{\mathbf{z}\sim\mathbf{Z}}\mathbb{E}[X_{j}|X_{i}=x_{i},\mathbf{Z}=\mathbf{z}]$
(Thm. 3.3.2 of (Pearl, 2009)). To ensure that all variables in $\mathbf{Z}$
are observed, we make the following assumption.

######  Assumption 3.3.

(No-latent confounding.) There are no unobserved variables causing both
treatment and target variables.

##  4 Causal Order is Sufficient for Effect Estimation

Although backdoor adjustment is defined wrt a DAG $\mathcal{G}$, we now show
that the causal order is sufficient to find a valid backdoor set. We also
discuss why providing the causal order is a better task for experts than
providing the graph.

###  4.1 Causal (Topological) Order Yields a Valid Backdoor Set

######  Definition 4.1.

(Topological Order.) Given a causal graph
$\mathcal{G}(\mathbf{X},\mathbf{E})$, a sequence $\pi$ of variables
$\mathbf{X}$ is said to be a topological order iff for each edge
$X_{i}\rightarrow X_{j}\in\mathbf{E}$, $\pi_{i}<\pi_{j}$.

######  Proposition 4.2.

Under assumption 3.3, given an pair of treatment and target variables
$(X_{i},X_{j})$ in $\mathcal{G}$, $\mathbf{Z}=\\{X_{k}|\pi_{k}<\pi_{i}\\}$ is
a valid adjustment set relative to $(X_{i},X_{j})$ for any topological
ordering $\pi$ of $\mathcal{G}$.

Proof of all Propositions are in Appendix § A. Propn 4.2 states that all the
variables that precede the treatment variable in a topological order $\pi$ of
$\mathcal{G}$ constitute a valid adjustment set. Note that the set
$\mathbf{Z}$ may contain variables that are not necessary to adjust for, e.g.,
ancestors of only treatment or only target variables. For statistical
estimation, ancestors of target variable are beneficial for precision whereas
ancestors of treatment can be harmful (Cinelli et al., 2022). On balance
though, causal effect practitioners tend to include all confounders that do
not violate the backdoor criterion; we are following the same principle.

In practice, however, we may not know the true order. To evaluate the goodness
of a given causal order, we use the topological divergence metric from
(Rolland et al., 2022) (for an example, see Figure 3). The topological
divergence of an estimated topological order $\hat{\pi}$ with ground truth
adjacency matrix $A$, denoted by $D_{top}(\hat{\pi},A)$, is defined as
$D_{top}(\hat{\pi},A)=\displaystyle\sum_{i=1}^{n}\sum_{j:\hat{\pi}_{i}>\hat{\pi}_{j}}A_{ij}$.
Where $A_{ij}=1$ if there is a directed arrow from node $i$ to $j$ else
$A_{ij}=0$. $D_{top}(\hat{\pi},A)$ counts the number of edges that cannot be
recovered due to estimated topological order $\hat{\pi}$.

###  4.2 Topological divergence is the correct metric for effect estimation

Below we show that $D_{top}$ is the correct metric to optimize for effect
estimation: $D_{top}=0$ for a topological order is equivalent to obtaining the
correct backdoor adjustment set using Proposition 4.2.

######  Proposition 4.3.

For an estimated topological order $\hat{\pi}$ and a true topological order
$\pi$ of a causal DAG $\mathcal{G}$ with the corresponding adjacency matrix
$A$, $D_{top}(\hat{\pi},A)=0$ iff
$\mathbf{Z}=\\{X_{k}|\hat{\pi}_{k}<\hat{\pi}_{i}\\}$ is a valid adjustment set
relative to $(X_{i},X_{j}),\ \forall i,j$.

We now compare $D_{top}$ to structural hamming distance (SHD), a common metric
used to evaluate graph discovery algorithms. Given a true causal DAG
$\mathcal{G}$ and an estimated causal DAG $\hat{\mathcal{G}}$, SHD counts the
number of missing, falsely detected, and falsely directed edges in
$\hat{\mathcal{G}}$: $SHD=E_{m}+E_{f}+E_{d}$ where $E_{m}=\\{X_{i}\rightarrow
X_{j}\not\in\hat{\mathcal{G}}|X_{i}\rightarrow X_{j}\in\mathcal{G}\\}$,
$E_{f}=\\{X_{i}\rightarrow X_{j}\in\hat{\mathcal{G}}|X_{i}\rightarrow
X_{j}\not\in\mathcal{G}\\}$, $E_{d}=\\{X_{i}\rightarrow
X_{j}\in\hat{\mathcal{G}}|X_{i}\leftarrow X_{j}\in\mathcal{G}\\}$. Formally,
$D_{top}$ acts as a lower-bound on the structural hamming distance (SHD)
(Rolland et al., 2022). However, as we show below, SHD is not a good metric
for evaluating downstream effect estimation accuracy. Specifically, we show
that SHD can be very high even when $D_{top}=0$ and a valid backdoor set can
be inferred. This result is of significance since most estimated graphs
(included those that are LLM-generated (Ban et al., 2023; Long et al., 2023))
are evaluated on SHD.

######  Definition 4.4.

(Level Order.) Given a causal graph $\mathcal{G}(\mathbf{X},\mathbf{E})$, the
level order refers to a systematic assignment of levels to variables. This
assignment begins with the set of variables
$\\{X_{i}|\text{pa}(X_{i})=\emptyset\\}$ at level 0. Subsequently, each of the
remaining variables is assigned a level $i$ such that all nodes within a given
level $i$ has a directed path of length $i$ from one/more nodes in level 0.

######  Proposition 4.5.

In a causal DAG $\mathcal{G}$ with $N$ levels in the level-ordering of
variables where the level $i$ contains $n_{i}$ variables,$\exists\
\hat{\mathcal{G}}$ s.t.
$SHD(\hat{\mathcal{G}},\mathcal{G})\geq\sum\limits_{i=1}^{N-1}(n_{i}\times\sum\limits_{j=i+1}^{N}n_{j})-|\mathbf{E}|$
and $D_{top}(\hat{\pi},A)=0\ \forall\hat{\pi}\text{ of }\hat{\mathcal{G}}$.

Figure 2: Variability of SHD for various graph sizes with consistent
$D_{top}=0$ within each graph.

Figure 2 shows the unsuitability of SHD for our work empirically. Given a
fixed number of nodes, we sample a graph at random as the “ground-truth” and
then consider all graph orientations of the same size (number of nodes) such
that $D_{top}=0$ w.r.t to ground-truth graph. For these set of graphs, we
compute the SHD w.r.t the ground-truth graph. Notice that SHD exhibits high
variance. For graphs with six nodes, SHD can vary from 0 to 14 even as
$D_{top}=0$ and backdoor set validity stays the same. Figure 3 shows this
phenomenon on a real-world BNLearn dataset, _Cancer_. The candidate graph
(right panel) has $D_{top}=0$ wrt. the true graph (left) and yields valid
backdoor identification sets. However, its SHD is high (6), showing the
disconnect between SHD and causal effect identification.

###  4.3 Causal order is more suitable to elicit from experts than graph edges

PollutionSmokingCancerX-rayDyspnoeaPollutionSmokingCancerX-rayDyspnoea

Figure 3: Left: Causal graph of Cancer dataset. Right: GPT-3.5’s estimated
causal graph of Cancer dataset. GPT-3.5 gets causal order correct at the cost
of higher SHD score, which is not a relevant metric for causal inference. Here
$D_{top}=0$ and $SHD=6$, showing the importance of $D_{top}$ in our study as
compared to SHD.

In addition to the favorable theoretical properties, causal order is easier to
elicit from experts and can be objectively evaluated. This is because given
two variables, their relative causal order does not depend on other variables
whereas existence of an edge between them depends on which other variables are
considered. To see this, let us continue the example from Figure 3 (left)
where pollution causes dyspnoea (breathing difficulty) through the
intermediary node cancer. Whether an edge exists between pollution and
dyspnoea depends on whether _Cancer_ variable is part of the study. In case an
expert is only provided _pollution_ and dyspnoea, they may add an edge between
the two, but if _Cancer_ node is also provided, they may not. Apriori, it is
difficult to know which nodes may be relevant for a pair; hence experts’
answers may not be consistent for questions about edges, but will always be
consistent on causal order.

##  5 Large Language Models as Virtual Experts for Causal Order

While causal order is a simpler construct than the graph, it still needs to be
provided by a domain expert. We now study whether LLMs can used to obtain
causal order, thereby making automating the process. We propose two kinds of
prompting strategies; 1) Pairwise; and 2) Triplet-based. All these techniques
use variable names or additional metadata but do not use the data associated
with variables.

###  5.1 Prompt techniques based on a pair of variables

A natural way to elicit causal order from LLMs is to ask about each pair of
variables and aggregate the result. A similar pairwise strategy has been
adopted by past work on inferring graph edges (Kıcıman et al., 2023; Ban et
al., 2023; Long et al., 2022). Here we augment such strategies with additional
contextual information. Our hypothesis is that adding context relevant to the
pair of variables may help increase the accuracy of the LLM answers.

We propose four types of pairwise prompts (see Appendix § D for full prompts).

Input: (‘HypDistrib’, ‘LowerBodyO2’)
---
Answer: Low oxygen areas equally distributed around the body can affect the
level of oxygen in the lower body by reducing the amount of oxygen available
for circulation. Therefore, the answer is <Answer>A</Answer>.
Input: (‘Disease’, ‘LungFlow’)
Answer: Infant methemoglobinemia is a condition where the blood is unable to
carry enough oxygen, which can lead to low blood flow in the lungs. Therefore,
the answer is <Answer>A</Answer>.
Input: (‘CardiacMixing’, ‘HypDistrib’)
Answer: Mixing of oxygenated and deoxygenated blood is a common cause of
congenital heart disease in babies. This mixing can occur due to structural
defects in the heart, such as a hole between the heart chambers or abnormal
connections between the blood vessels. This mixing can lead to low oxygen
areas equally distributed around the body. Therefore, the answer is
<Answer>B</Answer>.
Table 1: Example LLM (GPT-3.5-turbo) outputs for estimating the causal
direction between a given pair of nodes (with their descriptions) using the
CoT Prompt (refer Table A4 in Appendix for the prompt). Given a pair (X,Y),
Option $A$ refers to X causing Y, option $B$ refers to Y causing X, and option
$C$ refers to no causal relationship between the variables. The pairs shown
are from the Child graph from BNLearn repository (Scutari & Denis, 2014), that
is used for modeling Congenital Heart Disease in infants.

  * •

Basic prompt. This is the simplest technique. We directly ask LLM to find the
causal direction between a given pair of variables.

  * •

Iterative Context. Here we provide the previously oriented pairs as context in
the prompt. Since the LLM has access to its previous decisions, we expect that
it may avoid creating cycles through its predictions.

  * •

Markov Blanket Context.  Providing previously oriented pairs may become
prohibitive for large graphs. Using the fact that a variable is independent of
all other nodes given the Markov Blanket (Pearl, 2009), here we provide the
Markov Blanket of the given node pairs as additional context in the prompt.

  * •

Chain-of-Thought (+In-context learning). Based on encouraging results of
providing in-context examples in the prompt for various tasks (Brown et al.,
2020), here we include 3 examples of the ordering task that we expect the LLM
to perform. Effectively, we provide example node pairs and their correct
causal ordering before asking the question about the given nodes. Each example
answer also contains an explanation of the answer, generated using Bing GPT-4.
Adding the explanation encourages LLM to employ chain-of-thought reasoning
(Wei et al., 2022) when deciding the causal order. To avoid overfitting, we
select node pairs from graphs that are not evaluated in our study. Node pairs
with and without direct edges were equally chosen. Examples of LLM’s answers
(and their explanations) using the CoT prompt can be found in Table 1 and
Table A12 in Appendix.

Identify the causal relationships between the given variables and create a
directed acyclic graph to {context}. Make sure to give a reasoning for your
answer and then output the directed graph in the form of a list of tuples,
where each tuple is a directed edge. The desired output should be in the
following form: [(‘A’,‘B’), (‘B’,‘C’)] where first tuple represents a directed
edge from Node ‘A’ to Node ‘B’, second tuple represents a directed edge from
Node ‘B’ to Node ‘C’and so on.
---
If a node should not form any causal relationship with other nodes, then you
can add it as an isolated node of the graph by adding it seperately. For
example, if ‘C’ should be an isolated node in a graph with nodes ‘A’, ‘B’,
‘C’, then the final DAG representation should be like [(‘A’,‘B’), (‘C’)].
Use the description about the node provided with the nodes in brackets to form
a better decision about the causal direction orientation between the nodes.
It is very important that you output the final Causal graph within the tags
<Answer></Answer>otherwise your answer will not be processed.
Example:
Input: Nodes: [‘A’, ‘B’, ‘C’, ‘D’];
Description of Nodes: [(description of Node A), (description of Node B),
(description of Node C), (description of Node D)]
Output: <Answer>[(‘A’,‘B’),(‘C’,‘D’)]</Answer>
Question:
Input: Nodes: {Triplet Nodes Input}
Description of Nodes: {Description of Each Node from the Triplet}
Output:
Table 2: The triplet prompt template, which includes a concise context of the
graph, the input triplet nodes and their respective descriptions. As an
example, for the Child graph, the context is "to model congenital heart
disease in babies", the three nodes may be (‘HypoxiaInO2’, ‘Grunting’,
‘GruntingReport’); and their node descriptions are ["hypoxia when breathing
oxygen", "grunting in infants", "report of infant grunting"] respectively.

###  5.2 Prompt technique based on triplets

As we shall see, while pairwise prompts are conceptually simple, they are
prone to yielding cycles in the graph since they decide about each edge
separately. Taking inspiration from the PC algorithm that employs constraints
over three variables , we now describe a prompting technique based on
iterating over all possible triplets given a set of nodes. Once the LLM has
provided subgraphs for each triplet, we determine causal order between a pair
by aggregating over all triplet LLM answers where the pair was included. To
resolve ties, we use another LLM. The algorithm has the following steps.

  * •

From a given set of nodes in the graph, we generate all possible triplets,
each triplet treated as independently from the others.

  * •

We prompt the LLM to orient nodes of each triplet group to form a Directed
Acyclic Graph representing the causal relationship between the nodes of the
triplet. This will result in multiple mini graphs representing causal
relationship for each triplet group.

  * •

LLMs will be prompted to directly orient the three nodes for each triplet,
hence identifying causal relationship based on the third node, similar to how
PC functions.

  * •

Once we have DAGs representing each triplet, we focus on merging them. Our
Merging process can be broken down into two major steps:

    * –

We iterate over all node pairs, and for each combination we check what is the
majority orientation between them over all the triplets containing the node
pair.

    * –

In case there occurs a clash (same number of votes) between any of the two or
all three possible edge orientation (A $\rightarrow$ B; B $\rightarrow$ A; No
connection between A and B nodes), we resort to GPT-4 to resolve the clash by
taking the final decision via CoT prompt.

In addition, the triplet prompt uses the techniques of in-context examples and
chain-of-thought from the pairwise setup. An example prompt is shown in Table
2.

##  6 Algorithms for LLM-Guided Causal Discovery

LLMs using the above prompts may yield accurate causal order predictions, but
may also exhibit some unknown failure modes (Kıcıman et al., 2023). To
increase robustness of the final result, we now provide algorithms for
combining LLM-outputted causal order with existing causal discovery paradigms:
(i) constraint-based and (ii) score-based methods.

###  6.1 Constraint-based methods using post-hoc edge orientation by LLMs

Constraint-based algorithms return a graph where some edges may not be
oriented. Given a graph from constraint-based algorithm like PC, we use the
causal order $\hat{\pi}$ from LLM to orient the undirected edges. Iterating
over the undirected edges, we first check if the nodes of that edge are
occurring in $\hat{\pi}$. If yes, we orient the edge according to the causal
order. Since there is a possibility that LLM’s final graph might have some
isolated nodes which won’t be in $\hat{\pi}$, therefore if either (or both)
nodes of the undirected edge are not included in $\hat{\pi}$, we query GPT-4
using pairwise CoT prompt (from Section 5.1) to finalise a direction between
the pair.

Algorithm 1 Combining constraint based methods and experts to get $\hat{\pi}$
for a given set of variables.

1: Input: LLM topological ordering $\hat{\pi}$, Expert $\mathcal{E}_{GPT4}$,
PC-CPDAG $\hat{\mathcal{G}}$

2: Output: Estimated topological order $\hat{\pi}_{\text{final}}$ of
$\\{X_{1},\dots,X_{n}\\}$.

3: for $(i-j)\in\text{undirected-edges}(\hat{\mathcal{G}})$ do

4: If both the node $i$ and $j$ are in $\hat{\pi}$ and if
$\hat{\pi}_{i}<\hat{\pi}_{j}$, orient $(i-j)$ as $(i\rightarrow j)$ in
$\hat{\mathcal{G}}$.

5: Otherwise, use the expert $\mathcal{E}_{GPT4}$ with CoT prompt to orient
the edge $(i-j)$.

6: end for

7: $\hat{\pi}_{\text{final}}=$ topological ordering of $\hat{\mathcal{G}}$

8: return $\hat{\pi}$

###  6.2 Score-based methods using expert priors

We utilize the output of LLM as a prior in the score-based algorithms. We
provide the level order of the causal graph returned by LLM as a prior for a
score-based algorithm. Unlike a similar LLM-prior approach by Ban et al.
(2023), where they combine the output of LLM and a score based method using an
ancestral constraint as a prior, ours is a sequential approach where a score
based algorithm starts with the order based constraint, aligning with our goal
of recovering causal order among variables. Optionally, we can provide prior
probability to control the influence of prior on the algorithm. Algorithm 2
outlines the steps to combine score based method and expert knowledge in terms
of level order of variables.

Algorithm 2 Combining score based methods and experts to get $\hat{\pi}$ for a
given set of variables.

1: Input: $\mathcal{D}$, variables $\\{X_{1},\dots,X_{n}\\}$, Expert
$\mathcal{E}$, Score based method $\mathcal{S}$, Prior probability $p$.

2: Output: Estimated topological order $\hat{\pi}$ of
$\\{X_{1},\dots,X_{n}\\}$.

3: Step (I) $\hat{\mathcal{G}}=\mathcal{E}(X_{1},\dots,X_{n})$

4: Step (II) Prior = level order traversal of $\hat{\mathcal{G}}$.

5: Step (II.I) If $\hat{\mathcal{G}}$ is cyclic, keep all the variables in a
cycle at the same level in Prior.

6: Step (III)
$\hat{\mathcal{G}}=\mathcal{S}(\mathcal{D},\textit{Prior},\textit{Prior
probability}=p)$

7: Step (IV) $\hat{\pi}=$ topological ordering of $\hat{\mathcal{G}}$

8: return $\hat{\pi}$

##  7 Experiments and Results

To evaluate the accuracy of LLM-based algorithms on inferring causal order, we
perform experiments on the benchmark datasets from Bayesian network repository
(Scutari & Denis, 2014): Earthquake, Cancer, Survey, Asia, Asia modified
(Asia-M), and Child. Asia-M is derived from Asia by removing the node either
since it is not a node with a semantic meaning (see Appendix§ C for details).
We also used a medium sized subset graph (refer Figure A6 in Appendix) from
the Neuropathic dataset (Tu et al., 2019) used for pain diagnosis. Except
Child (with 20 nodes) and Neuropathic subgraph (with 22 nodes), all other
graphs are small-scale graphs with <10 nodes.

###  7.1 $D_{top}$ correlates with effect estimation error

Cancer
---
$SHD$ vs. $\epsilon_{ACE}\mid D_{top}=0$ |  $D_{top}$ vs. $\epsilon_{ACE}\mid SHD=2$
$SHD$ | $\epsilon_{ACE}$ | $D_{top}$ | $\epsilon_{ACE}$
0 | 0.00 | 0 | 0.00
2 | 0.00 | 1 | 0.25
4 | 0.00 | 2 | 0.50
Asia
$SHD$ vs. $\epsilon_{ACE}\mid D_{top}=0$ |  $D_{top}$ vs. $\epsilon_{ACE}\mid SHD=3$
$SHD$ | $\epsilon_{ACE}$ | $D_{top}$ | $\epsilon_{ACE}$
0 | 0.00 | 1 | 0.14
6 | 0.00 | 2 | 0.22
10 | 0.00 | 3 | 0.57
Survey
$SHD$ vs. $\epsilon_{ACE}\mid D_{top}=0$ |  $D_{top}$ vs. $\epsilon_{ACE}\mid SHD=2$
$SHD$ | $\epsilon_{ACE}$ | $D_{top}$ | $\epsilon_{ACE}$
0 | 0.00 | 0 | 0.00
2 | 0.00 | 1 | 0.25
4 | 0.03 | 2 | 0.50

Table 3: $\epsilon_{ACE}$ vs. $SHD$ ($D_{top}$) given $D_{top}$ ($SHD$)

Before comparing methods on the $D_{top}$ metric, we first show that $D_{top}$
has a strong correlation with effect estimation error and hence is the correct
metric for effect inference. Specifically, we study how the error in causal
effect, $\epsilon_{ACE}$, changes as values of the metrics $SHD,D_{top}$
change. For the datasets Cancer, Asia and Survey, we consider $dyspnoea$,
$dyspnoea$, and $Travel$ respectively as the target variables. In each graph,
we evaluate causal effects of each variable on the target variable. We iterate
through estimated causal graphs with different values of SHD and $D_{top}$ and
report the mean absolute difference between estimated and true causal effects.
As Table 3 shows, when $D_{top}$ is zero, effect error $\epsilon_{ACE}$ is
also zero. And as $D_{top}$ increases (right panel), effect error increases.
In contrast, SHD has no correlation with the $\epsilon_{ACE}$.

Dataset | $\mathbf{D_{top}}$ | SHD | IN/TN | Cycles
---|---|---|---|---
Base Prompt
Earthquake | 0 | 7 | 0/5 | 0
Cancer | 0 | 6 | 0/5 | 0
Survey | 3 | 12 | 0/6 | 0
Asia | - | 21 | 0/8 | 7
Asia-M | - | 15 | 0/7 | 6
Child | - | 177 | 0/20 | 20
Neuropathic | - | 212 | 0/22 | 22
All Directed Edges
Earthquake | 1 | 9 | 0/5 | 0
Cancer | 1 | 7 | 0/5 | 0
Survey | 2 | 11 | 0/6 | 0
Asia | - | 21 | 0/8 | 8
Asia-M | 0 | 13 | 0/7 | 0
Child | - | 139 | 0/20 | 18
Neuropathic | - | 194 | 0/22 | 20
Markov Blanket
Earthquake | 0 | 8 | 0/5 | 0
Cancer | 0 | 6 | 0/5 | 0
Survey | 3 | 12 | 0/6 | 0
Asia | - | 21 | 0/8 | 5
Asia-M | 0 | 14 | 0/7 | 0
Child | - | 167 | 0/20 | 20
Neuropathic | - | 204 | 0/22 | 21

tableComparison of various prompting strategies for only LLM based setups,
providing different contextual cues in each setup about the graph. IN:
Isolated Nodes, TN:Total Nodes.

Dataset | $\mathbf{D_{top}}$ | SHD | IN/TN | Cycles
---|---|---|---|---
Chain of Thought
Earthquake | 1 | 4 | 0/5 | 0
Survey | 1 | 6 | 2/6 | 0
Asia | 1 | 17 | 0/8 | 0
Asia-M | 1 | 11 | 0/7 | 0
Child | - | 91 | 0/20 | 13
Neuropathic | - | 64 | 0/22 | 8
Triplet Prompt
Earthquake | 0 | 4 | 0/5 | 0
Cancer | 0 | 4 | 1/5 | 0
Survey | 0 | 6 | 0/6 | 0
Asia | 1 | 7 | 1/8 | 0
Asia-M | 0 | 3 | 2/7 | 0
Child | 0 | 29 | 11/20 | 0
Neuropathic | 2 | 23 | 16/22 | 0

tableTriplet Prompt output using variable names with their descriptions. It
prompts the LLM to orient all nodes of a triplet subgroup together to get a
graph. It then uses CoT prompt (base prompt for Cancer to avoid overfitting
since it contains examples of node pairs from cancer graph) with GPT-4 for
resolving ties during edge orientations for a given node pair based on
majority votes across all triplet subgraphs. IN: Isolated Nodes, TN:Total
Nodes.

###  7.2 Triplet prompting technique is most accurate for causal order

Tables 7.1 and 7.1 compare the different prompting techniques. As the graph
size increases, we observe limitations with pairwise prompts. In many cases,
pairwise prompts yield cycles in many cases due to which $D_{top}$ cannot be
computed. In particular, for Child dataset with 20 nodes, pairwise prompts
yield anywhere from 13-79 cycles. LLM output tends to connect more edges than
needed, which explains why SHD is high. Overall, among the pairwise prompts,
the chain of thought prompt performs the best: it has the lowest $D_{top}$ on
the four small graphs and lowest number of cycles for Child and Neuropathic
datasets. This indicates that in-context examples and chain-of-thought
reasoning helps to increase accuracy of causal order output, but other
contextual cues do not matter.

Finally, the triplet prompt provides the most accurate causal order. Even for
medium-size graphs like Child and Neuropathic, the LLM output includes no
cycles and SHD is fairly low betwen 4-29. Moreover ,$D_{top}$ is zero for all
datasets, except for Asia and Neuropathic where it is 1 and 2 respectively.
That said, we do see that isolated nodes in the output increase compared to
the pairwise prompts (all graphs are connected, so outputting an isolated node
is an error). Considering LLMs as virtual experts, this indicates that there
are some nodes on which the LLM expert cannot determine the causal order. This
is still a better tradeoff than outputting the wrong causal order, which can
confuse downstream algorithms. Overall, therefore, we conclude that the
triplet prompt provides the most robust causal order predictions.

| Dataset | PC | SCORE | ICA | Direct | NOTEARS | CaMML | Ours | Ours
---|---|---|---|---|---|---|---|---|---
|  |  |  | LiNGAM | LiNGAM |  |  | (PC+LLM) | (CaMML+LLM)
| Earthquake | 0.30$\pm$0.44 | 4.00$\pm$0.00 | 3.20$\pm$0.39 | 3.00$\pm$0.00 | 1.80$\pm$0.74 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 4.00$\pm$0.00 | 3.60$\pm$0.48 | 2.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 0.50$\pm$0.00 | 3.00$\pm$0.00 | 6.00$\pm$0.00 | 6.00$\pm$0.00 | 3.20$\pm$0.39 | 3.33$\pm$0.94 | 0.00$\pm$0.00 | 3.33$\pm$0.94
| Asia | 2.33$\pm$0.59 | 5.00$\pm$0.00 | 6.20$\pm$0.74 | 7.00$\pm$0.00 | 4.00$\pm$0.00 | 1.85$\pm$0.58 | 0.00$\pm$0.00 | 0.97$\pm$0.62
| Asia-M | 2.00$\pm$0.00 | 5.00$\pm$0.00 | 7.60$\pm$0.48 | 6.20$\pm$1.16 | 3.40$\pm$0.48 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 1.71$\pm$0.45
$N=250$ | Child | 8.16$\pm$1.58 | 8.80$\pm$2.70 | 12.8$\pm$0.97 | 13.0$\pm$0.63 | 15.0$\pm$1.09 | 3.00$\pm$0.00 | 4.00$\pm$0.00 | 3.53$\pm$0.45
| Neuropathic | 3.25$\pm$0.00 | 6.00$\pm$0.00 | 13.0$\pm$6.16 | 10.0$\pm$0.00 | 9.00$\pm$0.00 | 10.4$\pm$1.95 | 1.00$\pm$0.00 | 5.00$\pm$0.00
| Earthquake | 0.85$\pm$0.65 | 4.00$\pm$0.00 | 3.20$\pm$0.39 | 3.40$\pm$0.48 | 1.20$\pm$0.40 | 0.00$\pm$0.00 | 0.4$\pm$0.89 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 3.40$\pm$0.48 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 1.00$\pm$0.00
| Survey | 1.75$\pm$0.00 | 4.00$\pm$0.00 | 6.00$\pm$0.0 | 6.00$\pm$0.00 | 3.40$\pm$0.48 | 3.39$\pm$0.08 | 1.00$\pm$0.00 | 3.33$\pm$0.94
| Asia | 3.00$\pm$0.00 | 5.00$\pm$0.00 | 5.60$\pm$0.48 | 7.00$\pm$0.00 | 3.20$\pm$0.39 | 3.81$\pm$0.39 | 1.00$\pm$0.00 | 0.97$\pm$0.62
| Asia-M | 2.00$\pm$0.00 | 6.00$\pm$0.00 | 7.60$\pm$0.48 | 5.00$\pm$0.00 | 3.80$\pm$0.39 | 2.00$\pm$0.00 | 1.00$\pm$0.00 | 0.17$\pm$0.45
$N=500$ | Child | 9.79$\pm$1.17 | 6.20$\pm$1.32 | 12.2$\pm$0.74 | 10.6$\pm$1.35 | 15.4$\pm$0.48 | 2.00$\pm$0.00 | 4.6$\pm$1.34 | 2.00$\pm$0.00
| Neuropathic | 7.50$\pm$0.00 | 6.00$\pm$0.00 | 9.00$\pm$1.41 | 13.0$\pm$0.00 | 11.0$\pm$0.00 | 5.32$\pm$0.57 | 8.00$\pm$0.00 | 7.49$\pm$0.64
| Earthquake | 0.80$\pm$0.57 | 4.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 0.20$\pm$0.44 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 1.60$\pm$0.48 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 1.00$\pm$0.00 | 4.00$\pm$0.00 | 5.80$\pm$0.39 | 5.40$\pm$0.48 | 3.20$\pm$0.39 | 2.71$\pm$0.27 | 1.00$\pm$0.00 | 2.83$\pm$0.00
| Asia | 3.09$\pm$1.05 | 4.00$\pm$0.00 | 6.20$\pm$0.74 | 6.60$\pm$0.48 | 3.40$\pm$0.48 | 1.75$\pm$0.43 | 1.75$\pm$0.95 | 0.97$\pm$0.62
| Asia-M | 2.50$\pm$0.00 | 4.00$\pm$0.00 | 8.00$\pm$0.00 | 5.20$\pm$0.39 | 3.40$\pm$0.48 | 2.04$\pm$0.51 | 2.00$\pm$0.00 | 0.65$\pm$0.47
$N=1000$ | Child | 9.61$\pm$1.07 | 3.80$\pm$0.74 | 12.2$\pm$1.72 | 11.8$\pm$0.74 | 15.2$\pm$0.97 | 2.00$\pm$0.00 | 8.0$\pm$0.00 | 2.00$\pm$0.40
| Neuropathic | - | 6.00$\pm$0.00 | 4.00$\pm$0.81 | 12.0$\pm$0.00 | 12.0$\pm$0.00 | 5.54$\pm$0.75 | - | 10.1$\pm$2.12
| Earthquake | 0.30$\pm$0.67 | 4.00$\pm$0.00 | 2.80$\pm$0.39 | 3.00$\pm$0.00 | 1.00$\pm$0.00 | 0.80$\pm$0.97 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 2.00$\pm$0.00 | 4.00$\pm$0.00 | 5.00$\pm$0.00 | 5.00$\pm$0.00 | 3.00$\pm$0.00 | 3.33$\pm$0.69 | 2.00$\pm$0.00 | 2.60$\pm$0.00
| Asia | 3.05$\pm$0.94 | 4.00$\pm$0.00 | 6.60$\pm$0.79 | 4.40$\pm$1.35 | 3.40$\pm$0.48 | 1.75$\pm$0.43 | 2.80$\pm$1.30 | 0.97$\pm$0.62
| Asia-M | 1.00$\pm$0.00 | 4.00$\pm$0.00 | 7.60$\pm$0.48 | 4.60$\pm$0.48 | 3.20$\pm$0.39 | 1.68$\pm$0.46 | 0.20$\pm$0.44 | 0.00$\pm$0.00
$N=5000$ | Child | 8.42$\pm$0.75 | 3.00$\pm$0.00 | 12.6$\pm$0.79 | 10.8$\pm$1.72 | 14.2$\pm$0.40 | 3.00$\pm$0.00 | 7.00$\pm$0.00 | 3.00$\pm$0.00
| Neuropathic | 9.00$\pm$0.00 | 6.00$\pm$0.00 | 9.33$\pm$0.94 | 10.0$\pm$0.00 | 10.0$\pm$0.00 | 4.20$\pm$0.96 | 9.00$\pm$0.00 | 1.23$\pm$0.42
| Earthquake | 0.00$\pm$0.00 | 4.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 1.00$\pm$0.00 | 0.40$\pm$0.48 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 0.60$\pm$0.80 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 2.00$\pm$0.00 | 4.00$\pm$0.00 | 5.00$\pm$0.00 | 5.00$\pm$0.00 | 3.00$\pm$0.00 | 3.60$\pm$1.35 | 2.00$\pm$0.00 | 1.83$\pm$0.00
| Asia | 1.95$\pm$0.41 | 4.00$\pm$0.00 | 6.00$\pm$0.00 | 4.40$\pm$1.35 | 3.00$\pm$0.00 | 1.40$\pm$0.48 | 1.20$\pm$0.83 | 0.34$\pm$0.47
| Asia-M | 1.75$\pm$0.00 | 4.00$\pm$0.00 | 8.00$\pm$0.00 | 4.80$\pm$0.39 | 3.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
$N=10000$ | Child | 7.67$\pm$0.65 | 3.00$\pm$0.00 | 12.2$\pm$1.46 | 11.6$\pm$0.48 | 14.4$\pm$0.48 | 2.80$\pm$0.84 | 7.00$\pm$0.00 | 1.00$\pm$0.00
| Neuropathic | 10.00$\pm$0.00 | 6.00$\pm$0.00 | 1.00$\pm$0.00 | 10.0$\pm$0.00 | 10.0$\pm$0.00 | 3.00$\pm$0.00 | 10.00$\pm$0.00 | 1.00$\pm$0.00

Table 4: Comparison with existing discovery methods. Mean and std dev of
$D_{top}$ over 5 runs. (For Neuropathic subgraph (1k samples), PC Algorithm
returns cyclic graphs in the MEC)

###  7.3 LLMs improve causal order accuracy of existing discovery algorithms

We now study whether LLM output can be used to increase accuracy of discovery
algorithms in inferring causal order. We compare with popular causal discovery
methods: PC (Spirtes et al., 2000), SCORE (Rolland et al., 2022), ICA-LiNGAM
(Shimizu et al., 2006), Direct-LiNGAM (Shimizu et al., 2011), NOTEARS (Zheng
et al., 2018), and Causal discovery via minimum message length (CaMML)
(Wallace et al., 1996); across five different sample sizes:
$250,500,1000,5000,10000$. For LLM, we use the triplet prompt. Table 4 shows
the $D_{top}$ metric for different algorithms and compares it to the $D_{top}$
of our combined algorithms: PC+LLM and CaMML+LLM. Among the discovery
algorithms, we find that PC and CaMML perform the best, with the lowest
$D_{top}$ across the five datasets. For Neuropathic dataset, ICA LiNGAM is
also competitive.

For the BNLearn datasets, adding LLM output decreases the $D_{top}$ of both
algorithms further. Specifically, PC+LLM leads to a significant reduction in
$D_{top}$ and the gains are larger at lower sample sizes. This indicates that
obtaining causal order from LLMs may matter more in limited sample settings.
At sample size of 500, $D_{top}$ of PC is nearly double that of PC+LLM for
most datasets. Going from CaMML to CaMML+LLM, we also see significant
reductions in $D_{top}$. Interestingly, CaMML+LLM yields benefits even at
higher sample sizes. At sample size of 10000, for example, $D_{top}$ of CaMML
for Child and Asia is three and five times more than that of CaMML+LLM
respectively.

For the Neuropathic dataset, we see a similar pattern: adding LLM to existing
algorithms improves $D_{top}$ or keeps it constant, except at sample sizes 500
and 1000 where CaMML+LLM yields a worse $D_{top}$ than CaMML alone. However,
as sample size increases to 5000 and 10000, we do see that CaMML+LLM improves
the $D_{top}$ substantially compared to CaMML. Overall, these results show
that LLM output can signficantly improve the accuracy of existing causal
discovery algorithms.

##  8 Limitations and Conclusions

We presented causal order as a suitable metric for evaluating quality of
causal graphs for downstream effect inference tasks. Using a novel formulation
of LLM prompts based on triplets, we showed that LLMs can be useful in the
generating accurate causal order, both individually and in combination with
existing discovery algorithms. Our results point to techniques that can
automate the causal inference process.

That said, our work has limitations. We studied LLMs utility on popular
benchmarks which may have been partially memorized. It will be interesting to
extend our experiments to diverse and bigger datasets. In addition, we studied
only one downstream task (effect inference). Identifying the necessary graph
metrics for tasks such as causal prediction and counterfactual inference will
be useful future work.

## References

  * Acid & de Campos (2003) Silvia Acid and Luis M de Campos.  Searching for bayesian network structures in the space of restricted acyclic partially directed graphs.  _Journal of Artificial Intelligence Research_ , 18:445–490, 2003.
  * Ban et al. (2023) Taiyu Ban, Lyvzhou Chen, Xiangyu Wang, and Huanhuan Chen.  From query tools to causal architects: Harnessing large language models for advanced causal discovery from data.  _arXiv preprint arXiv:2306.16902_ , 2023.
  * Boslaugh (2023) Sarah E. Boslaugh.  Silent spring.  2023\.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165, 2020.  URL https://arxiv.org/abs/2005.14165.
  * Cinelli et al. (2022) Carlos Cinelli, Andrew Forney, and Judea Pearl.  A crash course in good and bad controls.  _Sociological Methods & Research_, pp. 00491241221099552, 2022\.
  * Constantinou et al. (2023) Anthony C. Constantinou, Zhigao Guo, and Neville K. Kitson.  The impact of prior knowledge on causal structure learning.  _Knowledge and Information Systems_ , 65(8):3385–3434, 2023.
  * Glymour et al. (2019) Clark Glymour, Kun Zhang, and Peter Spirtes.  Review of causal discovery methods based on graphical models.  _Frontiers in genetics_ , 10:524, 2019.
  * Gupta et al. (2022) Shantanu Gupta, David Childers, and Zachary Chase Lipton.  Local causal discovery for estimating causal effects.  In _NeurIPS 2022 Workshop on Causality for Real-world Impact_ , 2022\.
  * Hagendorff et al. (2022) Thilo Hagendorff, Sarah Fabi, and Michal Kosinski.  Machine intuition: Uncovering human-like intuitive decision-making in gpt-3.5, 12 2022.
  * Hasan & Gani (2022) Uzma Hasan and Md Osman Gani.  Kcrl: A prior knowledge based causal discovery framework with reinforcement learning.  In _Proceedings of the 7th Machine Learning for Healthcare Conference_ , 2022.
  * Heckerman & Geiger (2013) David Heckerman and Dan Geiger.  Learning bayesian networks: a unification for discrete and gaussian domains.  _arXiv preprint arXiv:1302.4957_ , 2013.
  * Hoyer et al. (2008a) Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf.  Nonlinear causal discovery with additive noise models.  In _Advances in Neural Information Processing Systems_ , 2008a.
  * Hoyer et al. (2008b) Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf.  Nonlinear causal discovery with additive noise models.  In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), _Advances in Neural Information Processing Systems_ , 2008b.
  * Huang et al. (2021) Yiyi Huang, Matthäus Kleindessner, Alexey Munishkin, Debvrat Varshney, Pei Guo, and Jianwu Wang.  Benchmarking of data-driven causality discovery approaches in the interactions of arctic sea ice and atmosphere.  _Frontiers in big Data_ , 4:642182, 2021.
  * Hyvärinen et al. (2010) Aapo Hyvärinen, Kun Zhang, Shohei Shimizu, and Patrik O. Hoyer.  Estimation of a structural vector autoregression model using non-gaussianity.  _Journal of Machine Learning Research_ , 11(56):1709–1731, 2010.
  * Imbens & Rubin (2015) Guido W. Imbens and Donald B. Rubin.  _Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction_.  Cambridge University Press, 2015.  doi: 10.1017/CBO9781139025751.
  * Kıcıman et al. (2023) Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan.  Causal reasoning and large language models: Opening a new frontier for causality.  _arXiv preprint arXiv:2305.00050_ , 2023.
  * Lachapelle et al. (2020) Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien.  Gradient-based neural dag learning.  In _ICLR_ , 2020.
  * Long et al. (2022) Stephanie Long, Tibor Schuster, and Alexandre Piché.  Can large language models build causal graphs?  In _NeurIPS 2022 Workshop on Causality for Real-world Impact_ , 2022\.
  * Long et al. (2023) Stephanie Long, Alexandre Piché, Valentina Zantedeschi, Tibor Schuster, and Alexandre Drouin.  Causal discovery with language models as imperfect experts.  In _ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling_, 2023.
  * Maathuis et al. (2010) Marloes H Maathuis, Diego Colombo, Markus Kalisch, and Peter Bühlmann.  Predicting causal effects in large-scale systems from observational data.  _Nature methods_ , 7(4):247–248, 2010.
  * Mahmood et al. (2014) Syed S Mahmood, Daniel Levy, Ramachandran S Vasan, and Thomas J Wang.  The framingham heart study and the epidemiology of cardiovascular disease: a historical perspective.  _The lancet_ , 383(9921):999–1008, 2014.
  * Mooij et al. (2016) Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf.  Distinguishing cause from effect using observational data: methods and benchmarks.  _The Journal of Machine Learning Research_ , 17(1):1103–1204, 2016.
  * O’Donnell et al. (2006) Rodney T O’Donnell, Ann E Nicholson, Bin Han, Kevin B Korb, Md Jahangir Alam, and Lucas R Hope.  Causal discovery with prior information.  In _AI 2006: Advances in Artificial Intelligence: 19th Australian Joint Conference on Artificial Intelligence, Hobart, Australia, December 4-8, 2006. Proceedings 19_ , pp. 1162–1167. Springer, 2006.
  * Pearl (2009) Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4, 2023.
  * Peters & Bühlmann (2015) Jonas Peters and Peter Bühlmann.  Structural intervention distance for evaluating causal graphs.  _Neural computation_ , 27(3):771–799, 2015.
  * Reisach et al. (2021) Alexander Reisach, Christof Seiler, and Sebastian Weichwald.  Beware of the simulated dag! causal discovery benchmarks may be easy to game.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), _NeurIPS_ , pp. 27772–27784, 2021.
  * Rolland et al. (2022) Paul Rolland, Volkan Cevher, Matthäus Kleindessner, Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello.  Score matching enables causal discovery of nonlinear additive noise models.  In _ICML_ , 2022.
  * Scutari & Denis (2014) M. Scutari and J.B. Denis.  _Bayesian Networks: With Examples in R_.  Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2014\.
  * Shimizu et al. (2006) Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan.  A linear non-gaussian acyclic model for causal discovery.  _JMLR_ , 7(10), 2006.
  * Shimizu et al. (2011) Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, Kenneth Bollen, and Patrik Hoyer.  Directlingam: A direct method for learning a linear non-gaussian structural equation model.  _JMLR_ , 12(Apr):1225–1248, 2011.
  * Spirtes et al. (2000) Peter Spirtes, Clark N Glymour, and Richard Scheines.  _Causation, prediction, and search_.  MIT press, 2000.
  * Teshima & Sugiyama (2021) Takeshi Teshima and Masashi Sugiyama.  Incorporating causal graphical prior knowledge into predictive modeling via simple data augmentation.  In _UAI_ , pp. 86–96. PMLR, 2021.
  * Teyssier & Koller (2005) Marc Teyssier and Daphne Koller.  Ordering-based search: A simple and effective algorithm for learning bayesian networks.  In _UAI_ , UAI’05, pp. 584–590, 2005.
  * Tsamardinos et al. (2006) Ioannis Tsamardinos, Laura E Brown, and Constantin F Aliferis.  The max-min hill-climbing bayesian network structure learning algorithm.  _Machine learning_ , 65:31–78, 2006.
  * Tu et al. (2019) Ruibo Tu, Kun Zhang, Bo Bertilson, Hedvig Kjellstrom, and Cheng Zhang.  Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Wallace et al. (1996) Chris Wallace, Kevin B Korb, and Honghua Dai.  Causal discovery via mml.  In _ICML_ , volume 96, pp. 516–524, 1996.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Willig et al. (2022) Moritz Willig, Matej Zečević, Devendra Singh Dhami, and Kristian Kersting.  Probing for correlations of causal facts: Large language models and causality.  2022\.
  * Zheng et al. (2018) Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing.  Dags with no tears: Continuous optimization for structure learning.  _NeurIPS_ , 31, 2018.

## Appendix

##  Appendix A Proofs of Propositions

See 4.2

###### Proof.

We need to show that the set $\mathbf{Z}=\\{X_{k}|\pi_{k}<\pi_{i}\\}$
satisfies the conditions (i) and (ii) in Defn 3.2. For any variable $X_{k}$
such that $\pi_{k}<\pi_{i}$, we have $X_{k}\not\in de(X_{i})$ and hence the
condition (i) is satisfied. Additionally, for each $X_{k}\in pa(X_{i})$ we
have $\pi_{k}<\pi_{i}$ and hence $pa(X_{i})\subseteq\mathbf{Z}$. Since
$pa(X_{i})$ blocks all paths from $X_{i}$ to $X_{j}$ that contains an arrow
into $X_{i}$ (Peters & Bühlmann, 2015), $\mathbf{Z}$ satisfies condition (ii).
∎

See 4.3

###### Proof.

The statement of proposition is of the form $A\iff B$ with $A$ being
“$D_{top}(\hat{\pi},A)=0$” and $B$ being
“$\mathbf{Z}=\\{X_{k}|\hat{\pi}_{k}<\hat{\pi}_{i}\\}$ is a valid adjustment
set relative to $(X_{i},X_{j}),\ \forall i,j$”. We prove $A\iff B$ by proving
(i) $A\implies B$ and (ii) $B\implies A$.

(i) Proof of $A\implies B$: If $D_{top}(\hat{\pi},A)=0$, for all pairs of
nodes $(X_{i},X_{j})$, we have $\hat{\pi}_{i}<\hat{\pi}_{j}$ whenever
$\pi_{i}<\pi_{j}$. That is, causal order in estimated graph is same that of
the causal order in true graph. Hence, from Propn 4.2,
$\mathbf{Z}=\\{X_{k}|\hat{\pi}_{k}<\hat{\pi}_{i}\\}$ is a valid adjustment set
relative to $(X_{i},X_{j}),\ \forall i,j$.

(ii) Proof of $B\implies A$: we prove the logical equivalent form of
$B\implies A$ i.e., $\neg A\implies\neg B$, the contrapositive of $B\implies
A$. To this end, assume $D_{top}(\hat{\pi},A)\neq 0$, then there will be at
least one edge $X_{i}\rightarrow X_{j}$ that cannot be oriented correctly due
to the estimated topological order $\hat{\pi}$. i.e.,
$\hat{\pi}_{j}<\hat{\pi}_{i}$ but $\pi_{j}>\pi_{i}$. Hence, to find the causal
effect of $X_{i}$ on $X_{l};\ l\neq j$, $X_{j}$ is included in the back-door
adjustment set $\mathbf{Z}$ relative to $(X_{i},X_{l})$. Adding $X_{j}$ to
$\mathbf{Z}$ renders $\mathbf{Z}$ an invalid adjustment set because it
violates the condition (i) of Defn 3.2. ∎

See 4.5

###### Proof.

Recall that SHD counts the number of missing, falsely detected, and falsely
directed edges in the estimated causal graph as compared to the ground truth
graph. Since we want $D_{top}(\hat{\pi},A)=0;\ \forall\hat{\pi}\text{ of
}\hat{\mathcal{G}}$, there cannot be an edge $X_{i}\rightarrow X_{j}$ in
$\hat{\mathcal{G}}$ such that $X_{i}\leftarrow X_{j}$ is in $\mathcal{G}$.
This constraint avoids the possibility of having falsely directed edges in
$\hat{\mathcal{G}}$. Consider a $\hat{\mathcal{G}}$ with all the edges in
$\mathcal{G}$ and in addition, each variable in level $i$ having a directed
edge to each variable in all levels below level $i$. All such edges contribute
to the SHD score while still obeying the causal ordering in $\mathcal{G}$.
This number will be equal to
$\sum\limits_{i=1}^{N-1}(n_{i}\times\sum\limits_{j=i+1}^{N}n_{j})-|\mathbf{E}|$.
The quantity
$\sum\limits_{i=1}^{N-1}(n_{i}\times\sum\limits_{j=i+1}^{N}n_{j})$ is the
number of edges possible from each node to the every other node in the levels
below it. We need to subtract the number of existing edges in $\mathbf{E}$ to
count the newly added edges that contribute to the SHD score. Now, we can
remove some of the edges $X_{i}\rightarrow X_{j}$ from $\hat{\mathcal{G}}$
such that $X_{i}\rightarrow X_{j}$ is in $\mathcal{G}$ while still leading to
same causal ordering of variables. This leads to increased SHD score due to
missing edges in $\hat{\mathcal{G}}$. Since it will only increase the SHD
score, we ignore such corner cases. ∎

##  Appendix B Additional Results

Table A1 shows the results of various prompt strategies and their improvements
over no-prior methods.

|  |  |  | Ours | Ours | Ours | Ours
---|---|---|---|---|---|---|---
| Dataset | PC | CaMML | (CoT) | (CoT) | (Triplet Pairwise) | (Triplet Pairwise)
|  |  |  | (PC+LLM) | (CaMML+LLM) | (PC+LLM) | (CaMML+LLM)
| Earthquake | 0.30$\pm$0.44 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 2.75$\pm$0.43 | 0.2$\pm$0.44 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 0.50$\pm$0.00 | 3.33$\pm$0.94 | 0.00$\pm$0.00 | 3.33$\pm$0.94 | 0.00$\pm$0.00 | 2.66$\pm$0.94
| Asia | 2.33$\pm$0.59 | 1.85$\pm$0.58 | 0.00$\pm$0.00 | 0.97$\pm$0.62 | 1.00$\pm$0.00 | 3.39$\pm$0.62
| Asia-M | 2.00$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 1.71$\pm$0.45 | 0.00$\pm$0.00 | 1.71$\pm$0.45
$N=250$ | Child | 8.16$\pm$ 1.58 | 3.00$\pm$0.00 | 4.70$\pm$1.34 | 3.53$\pm$0.45 | 8.00$\pm$3.17 | 4.00$\pm$0.00
| Neuropathic | 3.25$\pm$0.00 | 10.4$\pm$1.95 | 1.00$\pm$0.00 | 8.00$\pm$0.00 | 0.00$\pm$0.00 | 12.63$\pm$1.92
| Earthquake | 0.85$\pm$0.65 | 0.00$\pm$0.00 | 0.4$\pm$0.89 | 2.60$\pm$0.48 | 1.00$\pm$0.00 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 3.00$\pm$0.00 | 0.00$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 1.75$\pm$0.00 | 3.39$\pm$0.08 | 1.00$\pm$0.00 | 3.33$\pm$0.94 | 0.60$\pm$0.00 | 2.66$\pm$0.94
| Asia | 3.00$\pm$0.00 | 3.81$\pm$0.39 | 1.00$\pm$0.00 | 0.97$\pm$0.62 | 0.00$\pm$0.00 | 3.28$\pm$0.64
| Asia-M | 2.00$\pm$0.00 | 2.00$\pm$0.00 | 1.00$\pm$0.00 | 0.17$\pm$0.45 | 1.00$\pm$0.00 | 1.04$\pm$0.20
$N=500$ | Child | 9.79$\pm$1.17 | 2.00$\pm$0.00 | 4.60$\pm$1.34 | 2.00$\pm$0.00 | 9.20$\pm$2.16 | 3.00$\pm$0.00
| Neuropathic | 7.50$\pm$0.00 | 5.32$\pm$0.57 | 9.00$\pm$0.00 | 8.90$\pm$0.00 | 9.00$\pm$0.00 | 12.1$\pm$1.56
| Earthquake | 0.80$\pm$0.57 | 0.00$\pm$0.00 | 0.20$\pm$0.44 | 2.00$\pm$0.00 | 1.40$\pm$1.31 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 1.00$\pm$0.00 | 2.71$\pm$0.27 | 1.00$\pm$0.00 | 2.83$\pm$0.00 | 1.00$\pm$0.00 | 2.16$\pm$0.74
| Asia | 3.09$\pm$1.05 | 1.75$\pm$0.43 | 1.75$\pm$0.95 | 0.97$\pm$0.62 | 3.00$\pm$0.00 | 1.78$\pm$0.41
| Asia-M | 2.50$\pm$0.00 | 2.04$\pm$0.51 | 2.00$\pm$0.00 | 0.65$\pm$0.47 | 1.00$\pm$0.00 | 0.65$\pm$0.47
$N=1000$ | Child | 9.61$\pm$1.07 | 2.00$\pm$0.00 | 8.00$\pm$0.00 | 2.00$\pm$0.40 | 6.6$\pm$1.14 | 2.83$\pm$0.00
| Neuropathic | - | 5.54$\pm$0.75 | - | 6.00$\pm$0.00 | - | 4.00$\pm$0.00
| Earthquake | 0.30$\pm$0.67 | 0.80$\pm$0.97 | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 2.00$\pm$0.00 | 3.33$\pm$0.69 | 2.00$\pm$0.00 | 2.60$\pm$0.00 | 2.00$\pm$0.00 | 1.80$\pm$0.83
| Asia | 3.05$\pm$0.94 | 1.75$\pm$0.43 | 2.80$\pm$1.30 | 0.97$\pm$0.62 | 0.00$\pm$0.00 | 1.78$\pm$0.41
| Asia-M | 1.00$\pm$0.00 | 1.68$\pm$0.46 | 0.20$\pm$0.44 | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 1.00$\pm$0.00
$N=5000$ | Child | 8.42$\pm$0.75 | 3.00$\pm$0.00 | 7.00$\pm$0.00 | 3.00$\pm$0.00 | 7.4$\pm$1.51 | 1.00$\pm$0.00
| Neuropathic | 9.00$\pm$0.00 | 4.20$\pm$0.96 | 9.00$\pm$0.00 | 3.00$\pm$0.00 | 9.00$\pm$0.00 | 3.31$\pm$0.00
| Earthquake | 0.00$\pm$0.00 | 0.40$\pm$0.48 | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Cancer | 0.00$\pm$0.00 | 2.00$\pm$0.00 | 0.60$\pm$0.80 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00
| Survey | 2.00$\pm$0.00 | 3.60$\pm$1.35 | 2.00$\pm$0.00 | 1.83$\pm$0.00 | 2.00$\pm$0.00 | 1.08$\pm$0.64
| Asia | 1.95$\pm$0.41 | 1.40$\pm$0.48 | 1.20$\pm$0.83 | 0.34$\pm$0.47 | 0.00$\pm$0.00 | 1.78$\pm$0.41
| Asia-M | 1.75$\pm$0.00 | 2.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 1.00$\pm$0.00
$N=10000$ | Child | 7.67$\pm$0.65 | 2.80$\pm$0.84 | 4.66$\pm$3.05 | 1.00$\pm$0.00 | 5.80$\pm$1.48 | 1.00$\pm$0.00
| Neuropathic | 10.00$\pm$0.00 | 3.00$\pm$0.00 | - | 2.39$\pm$0.48 | 10.00$\pm$0.00 | 1.00$\pm$0.00

Table A1: $D_{top}$ metric results. Comparison with various prompting
strategies. Neuropathic subgraph for 1k samples return cyclic graphs in the
MEC. Using LLM CoT prior with Neuropathic 10k samples, orients the undirected
edges to create cyclic graphs

###  B.1 LLMs used in post processing for graph discovery

We conducted some experiments where we utilised discovery algorithms like PC
for creating skeletons of the graph and employed LLMs for orienting the
undirected edges. The idea was to utilise LLMs ability to correctly estimate
the causal direction while leveraging PC algorithm’s ability to give a
skeleton which could be oriented in a post processing setup. We saw that LLM
ended up giving improved results as compared to PC alone.

1000 samples
---
Context | Base prompt | Past iteration | Markov Blanket | PC
|  | orientations |  | (Avg. over MEC)
$D_{top}$ | $8.0$ | $5.3$ | $6.6$ | $9.61$
SHD | $14.33$ | $12.66$ | $14.0$ | $17.0$
10000 samples
$D_{top}$ | $6.33$ | $9.66$ | $6.0$ | $7.67$
SHD | $9.0$ | $13.33$ | $8.33$ | $12.0$
Table A2: PC + LLM results where LLM is used to orient the undirected edges of the skeleton PC returns over different data sample sizes. We show how LLMs can be used in a post processing setup for edge orientation besides having the capability of acting as a strong prior for different discovery algorithms. Dataset | Number of | Number of | Description
---|---|---|---
| Nodes | Edges | (used as a context)
Asia | 8 | 8 | Model the possible respiratory problems
|  |  | someone can have who has recently visited
|  |  | Asia and is experiencing shortness of breath
Cancer | 5 | 4 | Model the relation between various variables
|  |  | responsible for causing Cancer and its possible
|  |  | outcomes
Earthquake | 5 | 5 | Model factors influencing the probability of a burglary
Survey | 6 | 6 | Model a hypothetical survey whose aim is to investigate
|  |  | the usage patterns of different means of transport
Child | 20 | 25 | Model congenital heart disease in babies
Neuropathic Pain |  |  |
Diagnosis (subgraph) | 22 | 25 | For neuropathic pain diagnosis
Table A3: Overview of the datasets used.

##  Appendix C Causal Graphs used in Experiments

Figures A1-A5 show the causal graphs and details we considered from BNLearn
repository (Scutari & Denis, 2014).

Figure A1: Earthquake Bayesian network. Abbreviations/Descriptions: Burglary:
burglar entering, Earthquake: earthquake hitting, Alarm: home alarm going off
in a house, JohnCalls: first neighbor to call to inform the alarm sound,
Marycalls: second neighbor to call to inform the alarm sound. Figure A2:
Cancer Bayesian network. Abbreviations/Descriptions: Pollution: exposure to
pollutants, Smoker: smoking habit, Cancer: Cancer, Dyspnoea: Dyspnoea, Xray:
getting positive xray result. Figure A3: Survey Bayesian network.
Abbreviations: A=Age/Age of people using transport, S=Sex/male or female,
E=Education/up to high school or university degree, O=Occupation/employee or
self-employed, R=Residence/the size of the city the individual lives in,
recorded as either small or big, T=Travel/the means of transport favoured by
the individual. Figure A4: Asia Bayesian network. Abbreviations/Descriptions:
asia=visit to Asia/visiting Asian countries with high exposure to pollutants,
smoke=smoking habit, tub=tuberculosis, lung=lung cancer, either=either
tuberculosis or lung cancer, bronc=bronchitis, dysp=dyspnoea, xray=getting
positve xray result. Figure A5: Child Bayesian network. Abbreviations:
BirthAsphyxia: Lack of oxygen to the blood during the infant’s birth,
HypDistrib: Low oxygen areas equally distributed around the body, HypoxiaInO2:
Hypoxia when breathing oxygen, CO2: Level of carbon dioxide in the body,
ChestXray: Having a chest x-ray, Grunting: Grunting in infants, LVHreport:
Report of having left ventricular hypertrophy, LowerBodyO2: Level of oxygen in
the lower body, RUQO2: Level of oxygen in the right upper quadricep muscle,
CO2Report: A document reporting high levels of CO2 levels in blood,
XrayReport: Report of having a chest x-ray, Disease: Presence of an illness,
GruntingReport: Report of infant grunting, Age: Age of infant at disease
presentation, LVH: Thickening of the left ventricle, DuctFlow: Blood flow
across the ductus arteriosus, CardiacMixing: Mixing of oxygenated and
deoxygenated blood, LungParench: The state of the blood vessels in the lungs,
LungFlow: Low blood flow in the lungs, Sick: Presence of an illness Figure A6:
For Neuropathic dataset, we consider a sub-graph induced by one of the root
nodes, containing the following 22 nodes and corresponding edges taken from
https://observablehq.com/@turuibo/the-complete-causal-graph-of-neuropathic-
pain-diagnosis: ‘right C7’, ‘right elbow trouble’, ‘left shoulder trouble’,
‘left bend of arm problem’, ’right shoulder trouble’, ‘right hand problem’,
‘left medival elbow problems’, ‘right finger trouble’, ‘left neck problems’,
‘left wrist problems’, ’left shoulder problem’, ‘right neck’, ‘right wrist
problem’, ‘right shoulder problem’, ‘discoligment injury C6 C7’, ‘left hand
problem’, ‘left C7’, ‘right arm band’, ‘left lower arm disorders’, ‘neck
pain’, ‘left finger trouble’, ‘left arm’. We did not use descriptions for the
nodes of Neuropathic graph.

##  Appendix D Various Prompt Strategies Studied

Question: For a causal graph used to model relationship of various factors and
outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’,
’Smoker’, ’Xray’, ’Dyspnoea’],
---
Which cause-and-effect relationship is more likely between nodes ’smoker’ and
’cancer’?
A. changing the state of node ’smoker’ causally effects a change in another
node ’cancer’.
B. changing the state of node ’cancer’ causally effects a change in another
node ’smoker’.
C. There is no causal relation between the nodes ’cancer’ and ’smoker’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: The causal effect of "smoker" directing to "cancer" is based on the
strong evidence from epidemiological studies linking smoking to an increased
risk of developing cancer. Smoking introduces harmful substances into the
respiratory system, leading to cellular damage and mutation, which
significantly raises the likelihood of cancer development in the lungs or
respiratory tract, subsequently impacting the occurrence of respiratory
problems like shortness of breath. Therefore answer is <Answer>A</Answer>
Question: For a causal graph used to model relationship of various factors and
outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’,
’Smoker’, ’Xray’, ’Dyspnoea’], Which cause-and-effect relationship is more
likely between nodes ’xray’ and ’dyspnoea’?
A. changing the state of node ’xray’ causally effects a change in another node
’dyspnoea’.
B. changing the state of node ’dyspnoea’ causally effects a change in another
node ’xray’.
C. There is no causal relation between the nodes ’xray’ and ’dyspnoea’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: Reasoning behind the lack of causal relation between X-ray and
dyspnoea is that X-ray and dyspnoea are both effects of having cancer, but
they do not directly cause or affect each other. X-ray is a diagnostic test
that can help detect cancer in the lungs or other organs, while dyspnoea is a
symptom of cancer that involves feeling short of breath. Therefore, X-ray and
dyspnoea are not causally related, but they are both associated with cancer.
Therefore answer is <Answer>C</Answer>
Question: For a causal graph used to model relationship of various factors and
outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’,
’Smoker’, ’Xray’, ’Dyspnoea’], Which cause-and-effect relationship is more
likely between nodes ’xray’ and ’cancer’?
A. changing the state of node ’xray’ causally effects a change in another node
’cancer’.
B. changing the state of node ’cancer’ causally effects a change in another
node ’xray’.
C. There is no causal relation between the nodes ’xray’ and ’cancer’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Table A4: Chain of Thought Prompt Answer: The causal effect of cancer on X-ray
is that X-rays are often used to diagnose or detect cancer in different parts
of the body, such as the bones, lungs, breasts, or kidneys123. X-rays use low
doses of radiation to create pictures of the inside of the body and show the
presence, size, and location of tumors. X-rays can also help monitor the
response to treatment or check for signs of recurrence. Therefore, having
cancer may increase the likelihood of getting an X-ray as part of the
diagnostic process or follow-up care. Therefore answer is <Answer>B</Answer>
---
Question: For a causal graph used to model relationship of various factors and
outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’,
’Smoker’, ’Xray’, ’Dyspnoea’], Which cause-and-effect relationship is more
likely between nodes ’pollution’ and ’cancer’?
A. changing the state of node ’pollution’ causally effects a change in another
node ’cancer’.
B. changing the state of node ’cancer’ causally effects a change in another
node ’pollution’.
C. There is no causal relation between the nodes ’pollution’ and ’cancer’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: The causal effect of pollution on cancer is that air pollution
contains carcinogens (cancer-causing substances) that may be absorbed into the
body when inhaled and damage the DNA of cells. Another possible reasoning is
that particulate matter (tiny dust-like particles) in air pollution may cause
physical damage to the cells in the lungs, leading to inflammation and
oxidative stress and eventually cell mutations. A third possible reasoning is
that air pollution may create an inflamed environment in the lungs that
encourages the proliferation of cells with existing cancer-driving mutations.
These are some of the hypotheses that researchers have proposed to explain how
air pollution may cause cancer, but more studies are needed to confirm them.
Therefore answer is <Answer>A</Answer>
Question: For a causal graph used to model relationship of various factors and
outcomes related to cancer with the following nodes: [’Pollution’, ’Cancer’,
’Smoker’, ’Xray’, ’Dyspnoea’], Which cause-and-effect relationship is more
likely between nodes ’pollution’ and ’smoker’?
A. changing the state of node ’pollution’ causally effects a change in another
node ’smoker’.
B. changing the state of node ’smoker’ causally effects a change in another
node ’pollution’.
C. There is no causal relation between the nodes ’pollution’ and ’smoker’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: Reason behind the lack of causal relation between pollution and smoker
is that pollution and smoking are both independent risk factors for
respiratory problems, but they do not directly cause or affect each other.
Pollution and smoking both contribute to air pollution, which can harm the
health of people and the environment. However, pollution is mainly caused by
human activities such as burning fossil fuels, deforestation, or industrial
processes, while smoking is a personal choice that involves inhaling tobacco
smoke. Therefore, pollution and smoker are not causally related, but they are
both associated with respiratory problems. Therefore answer is
<Answer>C</Answer>.
Table A5: Chain of Thought Prompt (continued..) Question: For a causal graph
used for modeling factors causing Coronary Heart Diseases with the following
nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart
Disease’, ’Headache’], Which cause-and-effect relationship is more likely
between nodes ’Family Disease’ and ’Gene’?
---
A. changing the state of node ’Family Disease’ causally effects a change in
another node ’Gene’.
B. changing the state of node ’Gene’ causally effects a change in another node
’Family Disease’.
C. There is no causal relation between the nodes ’Family Disease’ and ’Gene’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: Reason behind the causal effect of family disease on gene is that
family disease is a term that refers to diseases or health conditions that run
in the family, meaning that they are influenced by genetic factors. Gene is a
term that refers to the basic unit of heredity that carries information for a
specific trait or function. Family disease can affect gene by altering the
type or frequency of genes that are inherited by the offspring from their
parents. For example, some family diseases are caused by deterministic genes,
which are genes that guarantee the development of a disease if they are
present in a person’s genome. Other family diseases are influenced by risk
genes, which are genes that increase the likelihood of developing a disease
but do not guarantee it. Therefore, family disease can causally direct to gene
by changing the genetic makeup of a person or a population. Therefore answer
is <Answer>A</Answer>.
Question: For a causal graph used for modeling factors causing Coronary Heart
Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’,
’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’], Which cause-and-
effect relationship is more likely between nodes ’Coronary Heart Disease’ and
’Gene’?
A. changing the state of node ’Coronary Heart Disease’ causally effects a
change in another node ’Gene’.
B. changing the state of node ’Gene’ causally effects a change in another node
’Coronary Heart Disease’.
C. There is no causal relation between the nodes ’Coronary Heart Disease’ and
’Gene’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: Possible reasoning behind the causal effect of gene on coronary heart
disease is that gene is a term that refers to the basic unit of heredity that
carries information for a specific trait or function. Gene can affect coronary
heart disease by influencing the structure and function of the blood vessels,
the metabolism and transport of lipids (fats) in the blood, the inflammation
and clotting processes, or the response to environmental factors such as
smoking or diet. For example, some genes code for proteins that regulate the
cell cycle and growth of the cells that line the arteries, which can affect
their susceptibility to damage or plaque formation. Other genes code for
proteins that control the synthesis and clearance of cholesterol or other
lipids, which can affect their levels and deposition in the arteries.
Therefore, gene can causally direct to coronary heart disease by modifying the
biological pathways that contribute to the development or progression of the
disease. Therefore answer is <Answer>B</Answer>
Table A6: Chain of Thought Prompt (continued..) Question: For a causal graph
used for modeling factors causing Coronary Heart Diseases with the following
nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart
Disease’, ’Headache’], Which cause-and-effect relationship is more likely
between nodes ’Blood Pressure’ and ’Smoking’?
---
A. changing the state of node ’Blood Pressure’ causally effects a change in
another node ’Smoking’.
B. changing the state of node ’Smoking’ causally effects a change in another
node ’Blood Pressure’.
C. There is no causal relation between the nodes ’Blood Pressure’ and
’Smoking’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: Possible reasoning behind the causal effect of smoking on blood
pressure is that smoking is a habit that involves inhaling tobacco smoke,
which contains nicotine and other harmful chemicals. Smoking can affect blood
pressure by activating the sympathetic nervous system (SNS), which is the part
of the nervous system that controls the body’s response to stress or danger.
When the SNS is activated, it releases hormones such as adrenaline and
noradrenaline, which cause the heart to beat faster and harder, and the blood
vessels to constrict. This results in a temporary increase in blood pressure,
which can last for 15 to 20 minutes after each cigarette. Therefore, smoking
can causally direct to blood pressure by stimulating the SNS and increasing
the cardiac output and vascular resistance. Therefore answer is
<Answer>B</Answer>.
Question: For a causal graph used for modeling factors causing Coronary Heart
Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’,
’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’], Which cause-and-
effect relationship is more likely between nodes ’Headache’ and ’Smoking’?
A. changing the state of node ’Headache’ causally effects a change in another
node ’Smoking’.
B. changing the state of node ’Smoking’ causally effects a change in another
node ’Headache’.
C. There is no causal relation between the nodes ’Headache’ and ’Smoking’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: One possible reasoning behind the lack of causal relation between
headache and smoking is that headache and smoking are both associated with
various health conditions, but they do not directly cause or affect each
other12. Headache is a term that refers to pain or discomfort in the head,
scalp, or neck, which can have many possible causes, such as stress,
dehydration, infection, injury, or medication. Smoking is a habit that
involves inhaling tobacco smoke, which contains nicotine and other harmful
chemicals, which can increase the risk of diseases such as cancer, heart
disease, stroke, and lung disease. Therefore, headache and smoking are not
causally related, but they are both linked to different health problems.
Therefore the answer is <Answer>C</Answer>
Table A7: Chain of Thought Prompt (continued..) Question: For a causal graph
used for modeling factors causing Coronary Heart Diseases with the following
nodes: [’Family Disease’, ’Gene’, ’Smoking’, ’Blood Pressure’, ’Coronary Heart
Disease’, ’Headache’], Which cause-and-effect relationship is more likely
between nodes ’Headache’ and ’Smoking’?
---
A. changing the state of node ’Headache’ causally effects a change in another
node ’Smoking’.
B. changing the state of node ’Smoking’ causally effects a change in another
node ’Headache’.
C. There is no causal relation between the nodes ’Headache’ and ’Smoking’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: One possible reasoning behind the lack of causal relation between
headache and smoking is that headache and smoking are both associated with
various health conditions, but they do not directly cause or affect each
other. Headache is a term that refers to pain or discomfort in the head,
scalp, or neck, which can have many possible causes, such as stress,
dehydration, infection, injury, or medication. Smoking is a habit that
involves inhaling tobacco smoke, which contains nicotine and other harmful
chemicals, which can increase the risk of diseases such as cancer, heart
disease, stroke, and lung disease. Therefore, headache and smoking are not
causally related, but they are both linked to different health problems.
Therefore the answer is <Answer>C</Answer>
Question: For a causal graph used for modeling factors causing Coronary Heart
Diseases with the following nodes: [’Family Disease’, ’Gene’, ’Smoking’,
’Blood Pressure’, ’Coronary Heart Disease’, ’Headache’], Which cause-and-
effect relationship is more likely between nodes ’Coronary Heart Disease’ and
’Smoking’?
A. changing the state of node ’Smoking’ causally effects a change in another
node ’Coronary Heart Disease’.
B. changing the state of node ’Coronary Heart Disease’ causally effects a
change in another node ’Smoking’.
C. There is no causal relation between the nodes ’Coronary Heart Disease’ and
’Smoking’.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Answer: Possible reasoning behind the causal effect of smoking on coronary
heart disease is smoking damages the heart and blood vessels by raising
triglycerides, lowering HDL, increasing blood clotting, and impairing blood
flow to the heart. This can lead to plaque buildup, heart attacks, and death.
Therefore answer is <Answer>A</Answer>.
Question: For a causal graph used for context with the following nodes: nodes,
Which cause-and-effect relationship is more likely between nodes X and Y?
A. changing the state of node X causally effects a change in another node Y.
B. changing the state of node Y causally effects a change in another node X.
C. There is no causal relation between the nodes X and Y.
Make sure to first provide a grounded reasoning for your answer and then
provide the answer in the following format: <Answer>A/B/C</Answer>. It is very
important that you output the final Causal graph within the tags like
<Answer>A/B/C</Answer> otherwise your answer will not be processed.
Table A8: Chain of Thought Prompt (continued..) Which cause-and-effect
relationship is more likely?
---
A. changing the state of node which says X causally effects a change in
another node which says Y.
B. changing the state of node which says Y causally effects a change in
another node which says X.
C. There is no causal relationship between node X and Y.
Make sure to first output a factually grounded reasoning for your answer. X
and Y are nodes of a Causal Graph. The causal graph is sparse and acyclic in
nature. So option C could be chosen if there is some uncertainity about causal
relationship between X and Y.
First give your reasoning and after that please make sure to provide your
final answer within the tags <Answer>A/B/C</Answer>.
It is very important that you output your final answer between the tags like
<Answer>A/B/C</Answer> otherwise your response will not be processed.
Table A9: Base prompt For the nodes X and Y which form an edge in a Causal
Graph, you have to identify which cause-and-effect relationship is more likely
between the nodes of the edge. This will be used to rearrange the nodes in the
edge to create a directed edge which accounts for causal relation from one
node to another in the edge.
---
A. changing the state of node X causally affects a change in another node Y.
B. changing the state of node Y causally affects a change in another node X.
C. There is no causal relation between the nodes X and Y.
You can also take the edges from the skeleton which have been rearranged to
create a directed edge to account for causal relationship between the nodes:
directed_edges.
Make sure to first output a factually grounded reasoning for your answer.
First give your reasoning and after that please make sure to provide your
final answer within the tags <Answer>A/B/C</Answer>.
It is very important that you output your final answer between the tags like
<Answer>A/B/C</Answer> otherwise your response will not be processed.
Table A10: Iterative orientation prompt For the following undirected edge in a
Causal Graph made of nodes X and Y, you have to identify which cause-and-
effect relationship is more likely between the nodes of the edge. This will be
used to rearrange the nodes in the edge to create a directed edge which
accounts for causal relation from one node to another in the edge.
---
A. changing the state of node X causally effects a change in another node Y.
B. changing the state of node Y causally effects a change in another node X.
C. There is no causal relation between the nodes X and Y.
You can also take the other directed edges of nodes X: X_edges and Y: Y_edges
of the Causal graph as context to redirect the edge to account for causal
effect.
Make sure to first output a factually grounded reasoning for your answer.
First give your reasoning and after that please make sure to provide your
final answer within the tags <Answer>A/B/C</Answer>.
It is very important that you output your final answer between the tags like
<Answer>A/B/C</Answer> otherwise your response will not be processed.
Table A11: Markov Blanket prompt Input: (‘Right C7’,‘Discoligment injury
C6-C7’)
---
Answer: Discoligment injury C6-C7 can cause compression of the nerve roots
that exit the spinal cord at the C7 level, which can lead to symptoms such as
pain, numbness, and weakness in the right C7 dermatome. Therefore, the answer
is <Answer>B</Answer>.
Input: (‘Right C7’, ‘Left C7’)
Answer: Right C7 and left C7 are both parts of the cervical spine and are not
known to directly influence each other. Therefore, the answer is
<Answer>C</Answer>.
Input: (‘Right elbow trouble’, ‘Left shoulder trouble’)
Answer: There is no direct causal relationship between right elbow trouble and
left shoulder trouble. They may both be symptoms of a larger underlying
condition, but they do not directly cause or affect each other. Therefore the
answer is <Answer>C</Answer>.
Table A12: Example LLM (GPT-3.5-turbo) reasoning outputs for estimating causal
directionality between different pairs of nodes using CoT Prompt (refer Table
A4 for the prompt) for Neuropathic subgraph (used for pain diagnosis).