Counterfactual Image Editing

Yushu Panyyy
Elias Bareinboimyyy

yyyDepartment of Computer Science, Columbia University, New York, USA

Yushu<EMAIL_ADDRESS>Elias<EMAIL_ADDRESS>
Machine Learning, ICML

Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different.
The current literature on the topic focuses primarily on changing individual features while remaining silent about the causal relationships between these features, as present in the real world.

In this paper, we formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images through a special type of model called augmented structural causal models (ASCMs).

Second, we show two fundamental impossibility results: (1) counterfactual editing is impossible from i.i.d. image samples and their corresponding labels alone; (2) even when the causal relationships between the latent generative factors and images are available, no guarantees regarding the output of the model can be provided.
Third, we propose a relaxation for this challenging problem by approximating non-identifiable counterfactual distributions with a new family of counterfactual-consistent estimators.
This family exhibits the desirable property of preserving features that the user cares about across both factual and counterfactual worlds.
Finally, we develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.

(Left) A causal graph depicting the causal relationships among features. (Right) Image editing results are displayed, with the first row showing edits incorporating causal relations, and the second row without them.
Each column represents a unique counterfactual query, altering the age, gender, and gray hair of the individuals.
These instances provide preliminary evidence that the causal approach introduced in this paper ensures the preservation of the relevant causal invariances for the query across both factual and counterfactual images.

§ INTRODUCTION

Counterfactual reasoning is a critical component of our cognitive system. It is essential for solving various tasks, including assigning credit, determining blame and responsibility, understanding why events occurred in a particular way and articulating explanations, and generalizing across changing conditions and environments [Pearl & Mackenzie, 2018, Bareinboim et al., 2022, Correa et al., 2021].
More recently, there has been a growing interest in counterfactual questions regarding image generation and editing.
For instance, one might ask “how would the image change had the dog been a cat?” or “What would the image look like had the person been smiling?”.
Addressing these prototypical counterfactual questions is challenging and requires the understanding of the causal relationships between the features, with practical applications in various downstream tasks, including data augmentation, fairness analysis, generalizability, and transportability  [Bareinboim et al., 2015, Schölkopf et al., 2021, Lee et al., 2020, Mao et al., 2022].

Some initial methods for counterfactual image editing tasks typically involve searching for adversarial samples  [Goyal et al., 2019, Wang & Vasconcelos, 2020, Dhurandhar et al., 2018]. For example,  [Dhurandhar et al., 2018] proposed a minimum-edit counterfactual method that aims to identify the minimum and most effective perturbations needed to change the classifier's prediction. With the ability to generate high-quality synthetic images from a latent space through GANs  [Brock et al., 2019, Karras et al., 2019], VAEs  [Child, 2021, Vahdat & Kautz, 2020], and Diffusion Models  [Ho et al., 2020, Song et al., 2021], recent approaches edit images by manipulating vectors in the latent space  [Shen et al., 2020, Härkönen et al., 2020, Khorram & Fuxin, 2022, Chai et al., 2021].

More recently, text information has also been leveraged in image editing tasks. The image description in text is beneficial to the encoding process and guiding manipulations in the latent space [Radford et al., 2021, Avrahami et al., 2022, Crowson et al., 2022, Gal et al., 2022, Patashnik et al., 2021] and the natural editing instruction text can be directly used to prompt the transition from the original tothe counterfactual images [Brooks et al., 2023].
However, such approaches focus primarily on changing a single categorical label of a given image, and more fundamentally, do not take the causal relationships among the underlying generative factors into account. The next example illustrates the challenge when multiple features are involved in the generation.

Consider an image dataset of human faces.
Based on our understanding of human anatomy and facial expressions, we know that both  $Gender$ and $Age$ do not causally affect each other while age does affect hair color. Meanwhile, the dataset collected has older males and younger females, i.e., there exists a strong correlation between age and gender.
Formally, the causal relationships between these three generative factors are shown in Fig. <ref>.

Existing methods focus on the editing of a single concept while the effects of the intervened concepts on others are not taken into account.
Suppose we are evaluating the counterfactual query: "Given a certain image, what would the face look like had she been older?".
If the age of the person is changed naively, gender and hair color may also change due to the correlation between these features found in the data.
For example, when making an image of a woman older, it may inadvertently also change her gender to male; see the yellow row in <ref>. However,
it would be expected that changes in age should not affect gender when performing causal editing, as shown in the figure's first row (in blue).

More importantly, existing methods are unable to answer to what extent hair color should change after an intervention on age.
Even though some recent methods may be able to enforce consistency in terms of gender, the causal effect from the age to the hair color may not be reflected correctly in the counterfactual images.
For instance, gray hair may never appear after the editing by non-causal approaches.
In contrast, causal image editing ensures the effects of target interventions on other features are carried over properly from factual to the proper counterfactual world.
To illustrate, edits in Fig. <ref> (blue) are more closely aligned with the reality in which these causal invariances are presented.

To capture the causal relationships among generative factors, we build on a class of generative models known as Structural Causal Models (SCMs) [Pearl, 2000]. A fully instantiated SCM induces what is known as the Pearl Causal Hierarchy (PCH; also called ladder of causation) [Pearl & Mackenzie, 2018, Bareinboim et al., 2022].
The PCH consists of families of distributions in increasing levels of refinement: layer 1  ($\cL_1$) corresponds to passive observations and typical correlations, layer 2 ($\cL_1$) to interventions  (e.g., changing a variable to see the effect), and layer 3 ($\cL_3$) to counterfactuals (e.g., considering what would happen under hypothetical scenarios).
A result known as the causal hierarchy theorem states that higher-layer distributions cannot be answered only from the lower-layer ones [Bareinboim et al., 2022].

Recently, researchers have connected SCMs with deep generative models by implicitly finding surrogate models of the true generative model relating images and its generative factors.
Despite the progress made so far, many of these works have limitations in different dimensions important in our context.
First, they assume Markovianity, which implies the absence of unobserved confounding among generative factors. While this assumption may hold in specific settings, the same is certainly strong and does not hold in many others [Kocaoglu et al., 2018, Pawlowski et al., 2020, Sanchez & Tsaftaris, 2022, Sauer & Geiger, 2021].

Second, many of these works estimate counterfactual queries for images and generate samples without considering whether the target query is identifiable. In particular, samples are generated even though the query is non-identifiable, which implies that no guarantee can be provided in terms of the quality and causal consistency of the image. In particular, it is unclear whether the causal invariances present in the real systems are preserved across the original and generated images.

Third, other works focus on parametric SCMs over generative factors, such as linear mechanisms, while we study a more general class of non-parametric models [Yang et al., 2021, Shen et al., 2022].
Recently, a new class of generative models has been developed, the Neural Causal Model (NCM), which encodes causal constraints into deep generative models. These models are capable of both identifying and then estimating counterfactual quantities in non-parametric settings [Xia et al., 2021, Xia et al., 2022].

Despite the soundness of this approach to handling general, non-parametric variables in theory, it remains challenging to estimate counterfactual images, as the structure between generative factors and images is not taken into account and it's hard to scale these models to higher dimensions. Further discussion on related works is provided in <ref>.

In this paper, we study the principles underpinning counterfactual image editing tasks and develop a practical, causally-grounded framework for these critical generative capabilities for high-dimensional settings.
To achieve this goal, we formalize counterfactual image tasks according to augmented SCMs (ASCMs), a special class of SCMs taking the image generation step into account.
This formulation allows for the formal encoding of causal relationships between generative factors and the image. It also enables modeling of the image editing tasks as querying counterfactual distributions induced by the true yet unknown ASCMs.
More specifically, our contributions are as follows:

1. We formally show that image counterfactual distributions are almost never identifiable from only observational i.i.d image samples. Further, even when the causal relationships between generative factors and images are given, the target counterfactual distribution is still non-identifiable (Sec. <ref>).

2. We relax these settings and develop a new family of counterfactual (Ctf-) consistent estimators to approximate non-identifiable distributions.
This provides the first procedure with formal guarantees of causal consistency w.r.t. the true generative model.
With a sufficient condition to obtain Ctf-consistent estimators,
we then develop an efficient algorithm (ANCMs) to sample counterfactual images in practice (Sec. <ref>).
Extensive experiments are conducted to demonstrate the effectiveness of ANCMs (Sec. <ref>).

§.§ Preliminary

In this section, we provide the necessary background to understand this work. An uppercase letter $X$ indicates a random variable and a lowercase letter $x$ indicates
its corresponding value;
bold uppercase $\*X$ denotes a set of random variables, and lowercase letter $\*x$ is its corresponding values.
We use $\mathcal{X}_{X}$ to denote the domain of $X$ and $\mathcal{X}_{\*X} = \mathcal{X}_{{X}_1} \times \cdots \times \mathcal{X}_{{X}_d}$ for $\*X = \{X_1, \dots, X_d\}$. We denote $P(\*X)$ as a probability distribution over a set of random variables $\*{X}$ and $P(\*X=\*x)$ as the probability of $\*X$ being equal to the value of $\*x$ under the distribution $P(\*X)$.

Our work relies on the basic semantical framework structural causal models (SCMs) <cit.>; we follow the presentation in [Bareinboim et al., 2022].

A Structure Causal Model (for short, SCM) is a 4-tuple  $<\*{U}, \*{V}, \mathcal{F}, P({\*{U}})>$, where

(1) $\*{U}$ is a set of background variables, also called exogenous variables, that are determined by factors outside the model;

(2) $\*{V} = \{V_1, V_2, \dots, V_d\}$ is the set of endogenous variables that are determined by other variables in the model;

(3) $\mathcal{F}$ is the set of functions $\{f_{V_1}, f_{V_2}\dots, f_{V_d}\} $ mapping $\*U_{V_j} \cup \Pai{V_j}$ to $V_j$, where $\*U_{V_j} \subseteq \*{U}$ and $\*{Pa}_{V_j} \subseteq \*V \backslash V_j$;

(4) $P(\*{U})$ is a probability function over the domain of $\*{U}$. $\blacksquare$

Each SCM $\cM$ induces a causal diagram $\cG$, which is a directed acyclic graph where every $V_j$ is a vertex. There is a directed arrow from $V_j$ to $V_k$ if $V_j \in \Pai{V_k}$. And there is a bidirected arrow between $V_j$ and $V_k$ if $\*{U}_{V_j}$ and $\*{U}_{V_k}$ are not independent with each other <cit.>.

An intervention on a subset of $\*X \subseteq \*V$, denoted by $do(\*x)$, is an operation where $\*X$ takes value $\*x$, regardless how $\*X$ are originally defined. For an SCM $\cM$, let $\cM_{\*x}$ be the submodel of $\cM$ induced by $do(\*x)$. For any subset $\*Y \subseteq \*V$, the potential outcome $\*Y_{\*x}(\*u)$ is defined as the solution of $\*Y$ after feeding $\*U = \*u$ into the submodel $\cM_{\*x}$. Then $\*Y_{\*x}$ is called a counterfactual variable induced by $\cM$. Specifically, the event $\*Y_{\*x} = \*y$ represent "$\*Y$ would be $\*y$ had $\*X$ been $\*x$". The counterfactual quantities induced by an SCM $\cM$ are defined as <cit.>:

\begin{align}
\label{eq:def:l3-semantics-nested}
P^{\cM}(\*y_{\*x}, \dots, \*z_{\*w}) =
\int_{\cX_{\mathbf{U}}} \mathbbm{1}_{\*Y_{{\*x}}(\*u)=\*y,\dots, \*Z_{{\*w}}(\*u) = \*z} dP(\*u),
\end{align}

where $\*Y, \dots, \*Z, \*X, \dots, \*W \subseteq \*V$. Specifically, $P(\*Y_{\*x})$ reduces to an observational distribution $P(\*Y) $ taking $\*X$ as an empty set.

Given the observed distribution $P(\*V)$ and causal diagram $\cG$, the optimal counterfactual bounds are closed intervals based on the following optimization problem [Zhang et al., 2022].

For a causal diagram $\cG$ and observed distributions $P(\*V)$, the optimal bound $[l, r]$ over a counterfactual probability $P^{\cM}(\*y_{\*x}, \dots, \*z_{\*w})$ is
defined as, respectively, the minimum and maximum of the following optimization problem:

\begin{align}
\label{eq:def:ctf-optimal-bound}
\underset{\cM \in \Omega(\cG)}{\max / \min} \ \  P^{\cM}(\*y_{\*x}, \dots, \*z_{\*w}) \ \  \mathrm{s.t. } P^{\cM}(\*V) = P(\*V)
\end{align}

where $\Omega(\cG)$ is the space of all SCMs that agree with the diagram $\cG$, i.e., $\Omega(\cG) = \{\forall \cM|\cG_{\cM} = \cG\}$. $\blacksquare$

By the formulation of <ref>, all possible values of counterfactual query induced by SCMs that agree with the observational distributions and causal diagram are contained in the closed interval $[l, r]$.

We use neural causal models (NCMs) for estimating counterfactual distributions, which are defined as follows [Xia et al., 2021]:

Given a causal diagram $\cG$, a $\cG$-constrained Neural Causal Model (for short, $\cG$-NCM) $\widehat{M}(\bm{\theta})$ over variables $\*V$ with parameters $\bm{\theta} = \{\theta_{V_i} : V_i \in \*V\}$ is an SCM $\langle \widehat{\*U}, \*V, \widehat{\cF}, \widehat{P}(\widehat{\*U}) \rangle$ such that $\widehat{\*U} = \{\widehat{U}_{\*C} : \*C \subseteq \*V\}$, where

(1) each $\widehat{U}$ is associated with some subset of variables $\*C \subseteq \*V$, and $\cD_{\widehat{U}} = [0, 1]$ for all $\widehat{U} \in \widehat{\*U}$;

(2) $\widehat{\cF} = \{\hat{f}_{V_i} : V_i \in \*V\}$, where each $\hat{f}_{V_i}$ is a feed forward neural network parameterized by $\theta_{V_i} \in \bm{\theta}$ mapping values of $\Ui{V_i} \cup \Pai{V_i}$ to values of $V_i$ for $\Ui{V_i} = \{\widehat{U}_{\*C} : \widehat{U}_{\*C} \in \widehat{\*U} \text{ s.t. } V_i \in \*C\}$ and $\Pai{V_i} = \Parents_{\cG}(V_i)$;

(3) $\widehat{P}(\widehat{\*U})$ is defined s.t. $\widehat{U} \sim \unif(0, 1)$ for each $\widehat{U} \in \widehat{\*U}$.     $\blacksquare$

§ AUGMENTED SCMS AND IMAGE COUNTERFACTUAL DISTRIBUTIONS

In this section, we model the image counterfactual editing problems in causal language. We first define a special type of SCM to model the generation process of an image variable $\*I$, which is called the Augmented SCM (ASCM).

An Augmented Structure Causal Model (for short, ASCM) over a generative level SCM $\cM_{0} = \langle \{\*U_0, \*V_0, \cF_0, P^0(\*U_{0})\} \rangle$ is a tuple $\cM = \langle \*U, \{\*V, \*I\}, \cF, P(\*U) \rangle$ such that

(1) exogenous variables $\*U = \{\*U_{0}, \*U_{\*I}\}$;

(2) $\*V  = \*V_0$ are labeled observed endogenous variables; $\*I$ is an $m$ dimensional image variable;

(3) $\cF = \{\cF_{0}, f_{\*I}\}$,

where $f_{\*I}$ maps from (the respective domains of) $\*V \cup {\*U}_{\*I}$ to $\*I$, which is an invertible function regarding $\*V$. Namely, there exists a function $h$ such that $\*V = h(\*I)$.

(4) $P(\*U_0) = P^0(\*U_0)$. $\blacksquare$

$F$     $Y$     $H$     $P(F, Y, H)$
0         0         0        0.216
0         0         1        0.144
0         1         0        0.128
0         1         1        0.032
1         0         0        0.144
1         0         1        0.096
1         1         0        0.192
1         1         1        0.048

$P(\*V)$ induced by the ASCM in Example. <ref>.

The ASCM $\cM$ is in fact a "larger" SCM describing a two-stage generative process, where first the low-dimensional generative factors are produced and second these generative factors are mapped to a high-dimensional image. More specifically, the
$\*U_{\*I}$ interact with labeled $\*V$ to produce other unlabeled features $\tilde{\*U}$ through part of $f_{\*I}$ in the first stage.
In the second stage, the remaining part of $f_{\*I}$ mixes the observed $\*V$ and unobserved generative factors $\tilde{\*U}$ to create the image's set of pixels. Notice that $\tilde{U}$ is not a part of $\*U_{\*I}$ since $\tilde{U}$ can be generated from $\*V$ plus $\*U_{\*I}$.
Throughout this paper, we assume that domains of observed generative factors $\*V$ are discrete and finite.
An important aspect of $f_{\*I}$ is that it is invertible regarding $\*V$ since these generative factors $\*V$ are present directly in a given image $\*i$.
This assumption is commonly used in non-linear ICA and representation learning literature [Locatello et al., 2019, Lachapelle et al., 2021, Hyvärinen & Pajunen, 1999, Khemakhem et al., 2020].
The inverse $h$ represents a labeling process that assigns the correct labels of $\*V$ to $\*i$. Then, for any $\*W \subseteq \*V$:

\begin{equation}
\label{eq:bij}
P(\*w \mid \*i) = \left\{
\begin{aligned}
&1  & \*w=h_{\*W}(\*i)\\
&0  &otherwise,\\
\end{aligned}
\right.
\end{equation}

where $h_{\*W}(\cdot)$ is the subfunction of $h$ mapping from $\*I$ to $\*W$.

The next example illustrates the modeling of face images discussed earlier.

(<ref> continued). Now we consider the augmented generative process, ASCM
$\cM^* = \langle \*U=\{U_F, U_{Y}, U_{H_1}, U_{H_2}, \*U_{\*I}\}, \{\{F, H, Y\}, \*I\},
\cF^*, P^*(\*U) \rangle$, where the mechanisms

\begin{equation}
{\cF^*} = \left\{
\begin{aligned}
F & \leftarrow U_F \oplus U_Y \\
Y & \leftarrow U_Y \\
H & \leftarrow (\neg Y \land U_{H_1}) \oplus (Y \land U_{H_2})  \\
\*I & \leftarrow {f}_{\*I}^{\mathrm{face}}(F, Y, H, \*U_{\*I})
\end{aligned}
\right.
\end{equation}

and the exogenous variables $U_F, U_Y, U_{H_1}, U_{H_2}$ are independent binary variables, and $P(U_F = 1) = 0.4, P(U_{Y} = 1) = 0.4, P(U_{H_1} = 1) = 0.4, P(U_{H_2} = 1) = 0.2$. $\*U_{\*I}$ can be correlated with $U_F, U_Y, U_{H_1}, U_{H_2}$.

The variable $F$ represents gender (male $F=0$; female $F=1$), $Y$ represents age (young $Y=0$; old $Y=1$), and $H$ represents whether the person has gray hair (gray $H=1$; non-gray $H=0$). The observational distribution $P(F, Y, H)$ induced by $\cM^*$ is shown in <ref>.
From the distribution, we can see that $Y=1$ and $H=1$ are positively correlated ($P(Y=1, F=1) > P(Y=1, F=0), P(Y=0, F=1)< P(Y=0, F=0)$), and older people are more likely to have gray hair $P(H=1 \mid Y=0) > P(H=1 \mid Y=1)$.

(a) The proxy generator $\widehat{\cM}$ is compatible with the same observational distributions with the unobserved true model but is not guaranteed to induce the same image counterfactual distributions. (b) Two different image counterfactual distributions in <ref>.
Each sample from a $P(\*I, \*I_{Y=0})$ has an initial image $\*i$ and a counterfactual image $\*i'_{Y=0}$. Sampling from the red part of distributions, counterfactual images do not contain gray hair. Sampling from the blue part of distributions, counterfactual images have gray hair.

Before the image is taken, $\*U_{\*I}$ and $\{F, Y, H\}$ produce other unobserved generative factors $\tilde{\*U}$, such as wrinkles, smiling, and narrow eyes at the generative level. Among them, some factors (such as wrinkles) can be produced by both $\*V$ and $\*U_{\*I}$, and some other factors (such as smiling) can be only produced by $\*U_{\*I}$. Then, $f_{\*I}$ maps all generative factors (including unobserved and observed ones) to image pixels $\*I$ at the second stage. Looking at the image, $\{F, Y, H\}$ are deterministic and one can in principle label them through function $h$, the inverse of ${f}_{\*I}^{\mathrm{face}}$ $\{F, Y, H\}$.

Equipped with ASCMs, we now formalize the counterfactual image generation tasks through formal causal semantics. Suppose the true underlying ASCM is given by $\cM^*$, which is unobserved. The goal is to query a specific type of counterfactual distribution induced by $\cM^*$ given the input distribution $P(\*V, \*I)$, i.e., $P^{\cM^*}(\*I, \*I_{{\*x'}})$, where $\*X \subseteq \*V$. Factorizing this joint probability distribution, we can write:

\begin{equation}
\label{eq:img-ctf-query-decomp}
\begin{split}
& P^{\cM^*}(\*I = \*i, \*I_{{\*x'}} = \*i') \\
& = P^{\cM^*}(\*I = \*i)P^{\cM^*}(\*I_{{\*x'}} = \*i' \mid \*I = \*i ),
\end{split}
\end{equation}

this $\cL_3$-quantity can be explained as follows. The initial image $\*i$ is sampled from $P^{\cM^*}(\*I)$ and the goal is to edit $\*i$ to a counterfactual version $\*i'$ with modified features $\*X = \*x'$, where $\*i'$ is sampled from $P^{\cM^*}(\*I_{{\*x'}} \mid \*I = \*i )$ [$P^{\cM^*}( \*I_{{\*x'}} \mid \*I = \*i)$ serves for editing real images when the initial image $\*i$ is a real one given by an user.]. For example, the distribution $P^{\cM^*}(\*I, \*I_{Y=0})$ (induced by the ASCM introduced in <ref>) can answer the query "generate an image describing people's face and edit the face to make the person look older".

Throughout this paper, we call this type of $\cL_3$-distributions as Image Counterfactual Distributions. A particular instantiation of the image variable, such as $P(\*I = \*i, \*I_{\*x'} = \*i')$, is called on Image Counterfactual Query. The explanation of image counterfactual distributions at the generative level is that given all generative factors in the initial images, what would they be had $\*X$ taken value $\*x'$. For instance, $P^{\cM^*}(\*I, \*I_{Y=0})$ is asking what would observed factors (gender, hair color) and unobserved factors (wrinkles, smiling, narrow eyes, ...) be had the person been older.

§ NON-IDENTIFIABILITY OF IMAGE COUNTERFACTUAL DISTRIBUTIONS

In classic counterfactual image generation tasks, a generator $\widehat{\cM}$ is trained to match the distribution $P(\*V, \*I)$ (e.g., through a GAN), and then the pair of an initial image and its counterfactual can be sampled from $P^{\widehat{\cM}}(\*I, \*I_{{\*x'}})$ induced by the generator (see <ref> bottom). For concreteness, after sampling an initial image $\*i$, one can get the counterfactual image $\*i_{\*x'}$ by manipulating the latent space or conditional signal of $\widehat{\cM}$.
However, as alluded to earlier, the Causal Hierarchy Theorem <cit.> states that counterfactual distributions cannot be computed merely from correlations. In particular, we show next the (non-)identifiability of any image counterfactual query from pure observational data:
[Image Causal Hierarchy Theorem]corollaryimagecht

Any image counterfactual distribution is almost never uniquely computable from the observational distribution (or its samples).

In other words, Corol. <ref> states that $P^{\widehat{\cM}}(\*I, \*I_{{\*x'}})$ induced by the proxy generator may not be consistent with the true $P^{{\cM}^*}(\*I, \*I_{{\*x'}})$ even when the proxy generator fits the observed distributions perfectly (i.e., $P^{\widehat{\cM}}(\*V, \*I) = P^{{\cM}^*}(\*V, \*I)$). This inconsistency implies the effect of intervention $\*X=\*x'$ on other generative factors (features) may differ from the true model and the proxy generator.

To illustrate this issue, suppose the target query is $P^{\cM^*}(\*I, \*I_{Y=0})$, where $\cM^*$ is defined in <ref>. As shown in <ref> and the next example, even when $\widehat{\cM}$ matches the observational distribution of the true underlying model $\cM^*$, $\widehat{\cM}$ can be less likely to generate counterfactual images with gray hair than $\cM^*$ after the intervention $Y=0$.

Consider an ASCM $\cM'$ that is exactly the same as $\cM^*$ introduced in <ref> except that

\begin{equation}
f'_{H} = (\neg U_Y \land U_{H_1}) \oplus (U_Y \land U_{H_2})
\end{equation}

$\cM'$ implies the same $P(\*V)$ as shown in <ref>.
Then, it's immediately verifiable that $P^{\cM^*}(\*V, \*I) = P^{\cM'}(\*V, \*I)$.

Now consider the counterfactual query $P(\*i , \*i_{Y=0})$, where $\*i$ is a young male without gray hair (with generated features $F=0, Y=1, H=0$) and $\*i'$ is an old male with gray hair (with features $F=0, Y=0, H=1$). In $\cM^*$, $H=0$ will change to $H=1$ and $F$ will remains invariant after the intervention $do(Y=1)$ with probability $0.4$, i.e.:

\begin{equation}
\label{eq:face-doage-hair}
P^{\cM^*}(F_{Y=0}=0, H_{Y=0}=1 \mid F=0, Y=1, H=0) = 0.4.
\end{equation}

However, $H$ will never change after the same intervention in $\cM'$ since the input of $f'_{H}$ does not involve $Y$, i.e.,

\begin{equation}
P^{\cM'}(F_{Y=0}=0, H_{Y=0}=1 \mid F=0, Y=1, H=0) = 0.
\end{equation}

Consequently, the true model suggests that the hair color is likely to change after making a young male look older with probability $0.4$ (see blue part of the distribution in the upper part of <ref>) while the counterfactual image generated by the proxy model would have gray hair with zero probability (blue part of upper distribution in <ref>). This is an instance of the aforementioned non-identifiability result (Corol. <ref>). $\blacksquare$

The main issue is that various generative models are capable of producing the same observational image distribution, yet they can yield qualitatively distinct counterfactual images. Broadly speaking, there is nothing in the observational distribution that indicates how an image would change under a hypothetical interventional scenario, so the counterfactual distribution remains undetermined by the observational one.

§.§ Identification of Image Counterfactual Distributions with Causal Diagrams

One of the realizations from the broader causal inference literature is that further assumptions are needed in order to perform counterfactual reasoning.
In this section, we will leverage the causal diagram of the true underlying ASCM to discuss whether an image counterfactual distribution is uniquely computable from a combination of these assumptions and these observational distributions.

[fill=white, draw=gray,dashed] (-0.5,-0.6) rectangle (3.5,0.8);

(X) at (0,0) [label=above:$F$, point];
(Y) at (3.0,0) [label=above:$H$, point];
(Z) at (1.5,0) [label=above:$Y$,point];
(I) at (1.5,-1.5) [label=below:$\*I$,point];

(Z) edge (Y);
(X) edge (I);
(Y) edge (I);
(Z) edge (I);
[bd] (X) edge [bend left=40] (Z);
[bd] (X) edge [bend right=40] (I);
[bd] (Z) edge [bend left=40] (I);
[bd] (Y) edge [bend left=40] (I);

The causal diagram of the in $\cM^*$ in <ref>

A causal diagram encodes constraints over counterfactual distributions compatible with the true and unobserved ASCM, narrowing down the hypothesis space of the proxy generator <cit.>. It can be obtained from prior information about concepts in images. For instance, the qualitative understanding that getting older likely leads to gray hair suggests that there should be a direct edge from $Y$ to $H$ in <ref>.
Causal diagrams can be regarded as a causal inductive bias based on human knowledge.
The complete causal diagram induced by $\cM^*$ is shown in <ref>; the diagram induced by $\cM^*_0$, at the generative level, is in the dashed box.
To illustrate, direct edges from $\{F, Y, H\}$ mean that these generative factors construct the image $\*I$. The bidirected edge between $F$ and $Y$ encodes that gender and age in the dataset collected are confounded. Bidirected edges between one of the generative factors $\{F, Y, H\}$ and the images imply that some unobserved generative factors can directly affect or be confounded with $\{F, Y, H\}$.

Once qualitative knowledge about the generative process is encoded in the causal model, our new goal is to infer a target image counterfactual query $P^{\cM^*}(\*I, \*I_{\*x'})$ given a causal diagram $\cG$ over $\{\*V, \*I\}$ and observational distributions $P(\*V, \*L)$.

We next define the notation of identifiability in the context of ASCMs.

Consider the true underlying ASCM $\cM^*$ defined over $\{\*V, \*I\}$ and the corresponding causal diagram $\cG$ and observational distribution $P(\*V, \*I)$. An image counterfactual query $P(\*i, \*i'_{\*x'})$ is said to be identifiable from the input $\langle P(\*V, \*I), \cG \rangle$ if $P^{\cM^{(1)}}(\*i, \*i'_{\*x'}) = P^{\cM^{(2)}}(\*i, \*i'_{\*x'})$ for every pair of ASCMs $\cM^{(1)}, \cM^{(2)} \in \Omega_{\*I}(\cG)$ s.t. $P^{\cM^{(1)}}(\*V, \*I) = P^{\cM^{(2)}}(\*V, \*I)$, where $\Omega^{\*I}$ is the space of ASCMs. The distribution $P(\*I, \*I_{\*x'})$ is said to be identifiable if $P(\*i, \*i'_{\*x'})$ is identifiable for every $\*i, \*i' \in \cX_{\*I}$.  $\blacksquare$

Compared to the previous definition of identifiability used in causal inference (e.g., <cit.>), Def. <ref> restricts
the space of SCMs to the space of ASCMs and considers image counterfactual queries.
The identifiability of $P(\*I, \*I_{\*x'})$ is equivalent to saying that $P(\*I, \*I_{\*x'})$ is uniquely computable given the observational distribution and the graphical constraints encoded in $\cG$. If satisfied, any proxy model $\widehat{\cM}$ that is compatible with $P(\*V, \*I)$ and $\cG$ could be used to evaluate $P^{\cM^*}(\*I, \*I_{\*x'})$ when the query is identifiable. However, the following proposition implies that even with prior causal information about $\*V$ as encoded in $\cG$, $P(\*i, \*i'_{\*x'})$ is still not identifiable.

The image counterfactual distribution $P(\*I, \*I'_{\*x'})$ is not identifiable from any combination of $\langle P(\*V, \*I), \cG \rangle$.

This non-identifiability challenge comes from two perspectives. First, it is unknown how $U_{\*I}$ interacts with $\*V$ to produce unobserved factors $\tilde{\*U}$ while these interactions have implications for determining how the counterfactual image should look like. The next example illustrates this point.

We split $\*U_{\*I}$ in Example <ref> into $\{U_S, \*U^-_{\*I}\}$, where $U_S$ controls the smiling generative factor and $\*U^-_{\*I}$ contributes to all other unobserved generative factors. Consider two ASCMs $\cM^{(1)}$ and $\cM^{(2)}$ with the same $\cF \backslash \{f_{\*I}\}$ but different $f_{\*I}$ from $\cM^*$ defined in <ref>:

\begin{equation}
\begin{split}
f_{\*I}^{(1)}(F, Y, H, U_S, \*U^-_{\*I}) &= {f}_{\*I}^{\mathrm{s}}(F, Y, H, U_S, \*U^-_{\*I}), \\
f_{\*I}^{(2)}(F, Y, H, U_S, \*U^-_{\*I}) &= {f}_{\*I}^{\mathrm{s}}(F, Y, H, U_S \oplus Y, \*U^-_{\*I}),
\end{split}
\end{equation}

where $U_S$ is a fair coin and is independent with $\*U \backslash \{U_S\}$.
$\*U^-_{\*I}$ can be correlated with $U_Y, U_F, U_H$. ${f}_{\*I}^{\mathrm{s}}$ is the same in both $\cM^{(1)}$ and $\cM^{(2)}$  mapping from $\{F, Y, H, S, \*U^-_{\*I}\}$ to $\*I$, where $S = U_S$ in $\cM^{(1)}$ and $S = U_S \oplus Y$ in $\cM^{(2)}$. $f_{\*I}^{\mathrm{s}}$ produces a smiling person image if and only if $S=1$. $\cM^{(1)}$ and $\cM^{(2)}$ are compatible with graphical constraints encoded in the causal diagram shown in Fig. <ref>, and it is verifiable that $\cM^{(1)}$ and $\cM^{(2)}$ induce the same observational distributions.

Consider the counterfactual image query $P(\*i, \*i_{Y=0})$, where $\*i$ is a non-smiling young male and $\*i'$ is a smiling old male with gray hair. In $\cM^{(1)}$, changing $Y$ to 0 will not affect the value of $S$ while changing $Y$ to 0 will always flip the value of $S$ in $\cM^{(2)}$. This implies that $P^{\cM^{(1)}}(\*i , \*i'_{Y=0})$ is the same as $P^{\cM^{(2)}}(\*i , \*i'_{Y=0})$. $\blacksquare$

Second, the other perspective follows that given the observed values of a generative factor $X$ and its child $Y$, $P(y'_{x'} \mid y, x)$ is never point identifiable from the observational distribution. The next example illustrates this point.

Consider an ASCM $\cM^{(3)}$ that is exactly the same as $\cM^*$ defined in <ref> except for

\begin{equation}
f^{(3)}_{H} \leftarrow ((\neg Y) \land U_{H_1}) \lor U_{H_2}
\end{equation}

and $P(U_{H_1}=1) = 0.25$. Then, it is verifiable that $P^{\cM^*}(\*V, \*I) = P^{\cM^{(3)}}(\*V, \*I)$ and $\cM^{(3)}$ is compatible with the graphical constraints induced by the model in <ref>.

Consider the same counterfactual image query $P(\*i , \*i_{Y=0})$, and note that $H=0$ will change to $H=1$,

\begin{equation}
P^{\cM^{(3)}}(F_{Y=0}=0, H_{Y=0}=1 \mid F=0, Y=1, H=0) = 0.25,
\end{equation}

after the intervention $do(Y=0)$ with probability $0.25$ in $\cM^{(3)}$, which is different from the same quantity induced by $\cM^*$ (<ref>). This implies $P^{\cM^{(3)}}(\*i , \*i'_{Y=0})$ is not equal to $P^{\cM^*}(\*i , \*i'_{Y=0})$. $\blacksquare$

§ COUNTERFACTUALLY CONSISTENT ESTIMATION OF IMAGE COUNTERFACTUAL DISTRIBUTIONS

We have seen so far that no image counterfactual distribution is identifiable given the causal diagram and the observational distribution alone. A question naturally arises considering this situation:
can these non-identifiable distributions be estimated in any reasonable way? In other words, when the proxy generator (${\widehat{\cM}}$) does not induce the exact same image counterfactual distributions, what tolerance could be acceptable between $P^{\widehat{\cM}}(\*I, \*I_{\*x'})$ and the true $P^{\cM^*}(\*I, \*I_{\*x'})$? In addition, we need an estimator to guarantee the approximation of $P^{\cM^*}(\*I, \*I_{\*x'})$ be within the tolerance no matter what causal relationships among generative factors are.
To achieve this, we propose the following two directions to relax the exact estimation of query $P^{\cM^*}(\*i, \*i_{\*x'})$ while retaining causal principles and reasonable results.

(1) Care set $\*W$. As illustrated in <ref>, $P^{\cM^*}(\*i, \*i_{\*x'})$ takes into account how all generative factors ($\{\*V, \tilde{\*U}\}$) in an image would change after the intervention $do(\*X=\*x)$ takes place. Still, in some practical situations, one may only be concerned about how some specific features behave after the intervention but not the whole image.
In <ref>, all facial features should change causally after making the person older. To illustrate, the intervention on age should preserve the gender and smiling status, and change the hair color with probability $0.4$ (<ref>). However, in practice, one may only care about the gender and age (i.e., $\*W = \{F, Y\}$) after the intervention, but not whether the hair color, smiling status, and background in the image are presented the same way or not. If so, the counterfactual image can have gray hair features and smiling features with arbitrary probability. We introduce the following definition to describe the counterfactual distributions among these selected features regarding an image counterfactual query.

Denote $\*W$ as a set of features one cares about and $\phi$ as a function mapping from $\*I$ to $\*W$ ($\*W = \phi(\*I)$). The feature counterfactual query regarding to $P(\*i, \*i_{\*x'})$ is defined as:

\begin{equation}
\label{eq:fea-ctf}
\hspace{-8pt} \int_{\*i^{(1)}, \*i^{(2)} \in \cX_{\*I}} \hspace{-10pt} \mathbf{1}\left [{\phi(\*i^{(1)}) = \*w, \phi(\*i^{(2)}) = \*w'}\right ]dP(\*i^{(1)}, \*i^{(2)}_{\*x'})
\end{equation}

where $\*w = \phi(\*i)$, and $\*w' = \phi(\*i')$.
We denote the feature counterfactual query  as ${\phi}(P(\*i, \*i'_{x'}))$.

In other words, the feature counterfactual query is a push-forward measure from $P(\*i, \*i'_{x'})$ through $\phi$.
The quantity in Eq. <ref> integrates over all $P(\*i^{(1)}, \*i^{(2)}_{\*x'})$ such that $\{\*i^{(1)}, \*i^{(2)}\}$ has the same cared features $\{\*w, \*w'\}$ with $\{\*i, \*i'\}$ in the target query.

For concreteness, consider the counterfactual image query $P(\*i , \*i_{Y=0})$, where $\*i$ is a smiling young male without gray hair and $\*i'$ is a smiling old male with gray hair. Suppose the care set $\*W$ contains the features gender ($F$) and age ($Y$). The feature counterfactual query $\phi(P(\*i, \*i'_{x'}))$ calculates the probability that the original image describes a young male and the counterfactual image describes an old male after editing. Following <ref>, $\phi(P(\*i, \*i'_{x'}))$ sums over $P(\*i^{(1)}, \*i^{(2)}_{\*x'})$, where $\*i^{(1)}$ describes a young male, $\*i^{(2)}$ describes an old male. In addition, $\*i^{(1)}$ and $\*i^{(2)}$ can have arbitrary hair and smiling features since those are not part of $\*W$. Then,
the feature counterfactual query induced by a proxy ASCM can be simplified using the following result.

Consider the true underlying ASCM $\cM^*$ over $\{\*V, \*I\}$, and a feature set with mapping function $\phi = h^*_{\*W}$, where $h^*_{\*W}$ is the inverse function of $f^*_{\*I}$ w.r.t. $\*W$, and a  a proxy ASCM $\widehat{\cM}$ over $\{\*V, \*I\}$. if $P^{\widehat{\cM}}(\*V, \*I) = P^{\cM^*}(\*V, \*I)$, then

\begin{equation}
h^*_{\*W}(P^{\widehat{\cM}}(\*i, \*i'_{\*x'})) = P^{\widehat{\cM}}(\*w, \*w'_{\*x'}),
\end{equation}

where $\*w = h_{\*W}(\*i)$, and $\*w' = h_{\*W}(\*i')$.

This result suggests that if $\widehat{\cM}$ agrees on the observational distribution of $\cM^*$ and the care set $\*W$ is a subset of observed generative factors, the feature counterfactual query is equivalent to a counterfactual query $P^{\widehat{\cM}}(\*w, \*w'_{\*x'})$ over $\*W$ induced by $\widehat{\cM}_0$ at the generative level.
We normalize $P^{\widehat{\cM}}(\*w, \*w'_{\*x'})$ as $P^{\widehat{\cM}}(\*w'_{\*x'} \mid \*w)$ following:

\begin{equation}
P^{\widehat{\cM}}(\*w'_{\*x'} \mid \*w) = P^{\widehat{\cM}}(\*w, \*w'_{\*x'}) / P^{\widehat{\cM}}(\*w)
\end{equation}

We will focus on the conditional feature counterfactual query $P^{\widehat{\cM}}(\*w'_{\*x'} \mid \*w)$ when the proxy model satisfies $P^{\widehat{\cM}}(\*V, \*I) = P^{\cM^*}(\*V, \*I)$, which implies $P^{\widehat{\cM}}(\*w) = P^{\cM^*}(\*w)$. The following example illustrates this point.

Consider the counterfactual image query $P(\*i , \*i_{Y=0})$, where $\*i$ is a young male without gray hair ($F=0, Y=1, H=0$) and $\*i'$ describes an old male with gray hair ($F=0, Y=0, H=1$). Suppose the care set $\*W$ contains the feature gender ($F$) and age ($Y$) as in <ref>. Lem. <ref> suggests the feature counterfactual query is

\begin{equation}
P^{\widehat{\cM}}(F_{Y=0}=0, F = 0, Y = 1)
\end{equation}

whenever $\widehat{\cM}$ is compatible with $\cM^*$ w.r.t. the observational distribution. The normalized conditional feature counterfactual query is

\begin{equation}
P^{\widehat{\cM}}(F_{Y=0}=0 \mid F = 0, Y = 1),
\end{equation}

which illustrates the probability that the gender was still male had a young male gotten older induced by the proxy model. $\blacksquare$

(2) Optimal Bounds.
A complementary relaxation arises from the observation that even when a query is not point identifiable, it is still possible to compute informative bounds over the target distribution from a combination of the observational data and the causal diagram [Manski, 1990, Balke & Pearl, 1994, Zhang et al., 2022].
These bounds serve as a natural measure of distance, or tolerance, between what is empirically obtainable from the data and the true, yet unobserved, counterfactual distribution.
This occurs because numerous ASCMs, compatible with the observed data, can generate counterfactual distributions encompassing the bound.

Any value within the optimal bound $[l, r]$ (Def. <ref>) falls within the range of some possible ground truth, contingent on the given assumptions. As assumptions are strengthened, the bounds naturally narrow.
Based on the above discussion, we formally define a class of counterfactual consistent estimators of the target $P(\*I, \*I_{\*x'})$.

Based on the above discussion, we formally define the Ctf-consistent estimation of an image counterfactual query.

Consider a feature set $\*W \subseteq \*V$ and its mapping function $\phi = h^*_{\*W}$, where $h^*_{\*W}$ is the inverse function of $f^*_{\*I}$ regarding $\*W$. ${P}^{\widehat{\cM}}(\*i, \*i'_{\*x'})$ is said to be a Ctf-consistent estimator of  ${P}^{{\cM}^*}(\*i, \*i'_{\*x'})$ w.r.t. $\*W$ if

(1) the observational distributions induced by $\widehat{\cM}$ and  $\cM^*$ are the same, namely, $P^{\widehat{\cM}}(\*V, \*I) = P^{\cM^*}(\*V, \*I)$ and

(2) the feature counterfactual query $\phi({P}^{\widehat{\cM}}(\*w, \*w'_{\*x'}))$ is within the optimal bound of ${P}(\*w, \*w'_{\*x'})$ derived by $P(\*V)$ and $\cG$, where $\*w = h_{\*W}^*(\*i)$ and $\*w' = h_{\*W}^*(\*i')$;

The proxy quantity $P^{\widehat{\cM}}(\*I, \*I_{\*x'})$ is said to be a Ctf-consistent estimator of the true $P^{{\cM}^*}(\*I, \*I_{\*x'})$ w.r.t. $\*W$ if $P^{\widehat{\cM}}(\*i, \*i_{\*x'})$ is Ctf-consistent for every $\*i, \*i' \in \cX_{\*I}$.

Notice that the feature counterfactual query $\phi({P}^{\widehat{\cM}}(\*i, \*i'_{\*x'}))$ is equivalent to $P^{\widehat{\cM}}(\*w, \*w'_{\*x'})$ here according to Lem. <ref>.
Def. <ref> states that if
(1) the observational distribution induced by the proxy model is the same as the true model, and
(2) the feature counterfactual query induced by the proxy model is within the optimal bound of ${P}(\*w, \*w'_{\*x'})$, then the corresponding image counterfactual query can be regarded as a Ctf-consistent estimation of the true image counterfactual query.
Def. <ref> does not require that the proxy model $\widehat{\cM}$ induces the same counterfactual image distribution $P(\*I, \*I_{\*x'})$ but expect $\widehat{\cM}$ to be Ctf-consistent with $\cM^*$ regarding the care set $\*W$ while ignoring other observed generative factors $\*V \backslash \*W$ and $\tilde{\*U}$.
Specifically, the feature counterfactual distribution $P^{\widehat{\cM}}(\*w, \*w'_{\*x'})$ should be within the optimal bound but no restriction is imposed over the features for $\*V \backslash \*W$ and $\tilde{\*U}$.
The next example illustrates this idea.

(<ref> continued).

Def. <ref> suggests the conditional feature counterfactual query $Q = P^{\widehat{\cM}}(F_{Y=0}=0 \mid F = 0, Y = 1)$ induced by the proxy model $\widehat{\cM}$ should be in the optimal bound $[r, l]$, where

\begin{equation}
r = l = P^{\widehat{\cM}}(F=0 \mid F = 0, Y = 1) = 1
\end{equation}

since the intervention $do(Y=0)$ has no effect on $F$ in the causal diagram (<ref>).This implies that the gender must remain the same after the editing. In the meantime, it does not matter whether the hair is gray ($\*V \backslash \*W$) or not and whether the person is smiling ($\tilde{\*U}$) since these features are not in the care set.

Now suppose the user cares about gender, age, and hair color, namely, $\*W = \{F, Y, H\}$ (instead of $\{F, Y\}$). Based on Def. <ref> and Lemma <ref>, the corresponding conditional feature counterfactual query is

\begin{equation}
\label{n-feature-ctf}
Q = P(F_{Y=0}=0, H_{Y = 0}=1 \mid F = 0, Y = 1, H = 0),
\end{equation}

and $Q$ illustrates the probability that the individual is still a male and has gray hair after getting older. This optimal bound analytically can be derived as (see <cit.>):

\begin{equation}
\begin{split}
&\hspace{-10pt}l = \max \{ 0, 1 - \frac{P(H = 0 \mid F=0, Y = 0)}{P(H=0 \mid F = 0,  Y = 1)}\} =  0.25\\
&\hspace{-10pt}r = \min\{1, \frac{P(H=1 \mid F=0, Y=0)}{P(H=0 \mid F=0, Y=1)} \} = 0.5
\end{split}
\end{equation}

Any $P^{\widehat{\cM}}(\*i , \*i_{Y=0})$ induced by $\widehat{\cM}$ such that $P^{\widehat{\cM}}(\*V, \*I) = P^{\cM^*}(\*V, \*I)$ and $Q^{\widehat{\cM}} \in [0.25, 0.5]$ is a Ctf-consistent estimator of $P^{\cM^*}(\*i , \*i_{Y=0})$.
Even if $Q^{\widehat{\cM}}$ is not equal to the true feature counterfactual query $Q^{\cM^*} = 0.4$, the error is acceptable compared to the non-causal method currently used in practice.
One may change $Y$ from $1$ to $0$ and keep the other features as close as possible. With such methods, the counterfactual image will never have gray hair, thus the estimation $Q = P(F_{Y=0}=0, H_{Y = 0}=1 \mid F = 0, Y = 1, H = 0) = 0$. The causal effect of the intervention $Y=0$ on $H$ is not reflected.

The ANCM network structure for <ref>.

[fill=white, draw=gray,dashed] (-1.8,-1.5) rectangle (1.8, 0.5);
(D) at (-1.5, 0) [label=below:$D$,point];
(C) at (0,-1) [label=above:$C$,point];
(B) at (1.5,-1) [label=above:$B$, point];
(I) at (0,-2) [label=below:$\*I$,point];

[neg]  (D) edge (B);
[neg] (C) edge (B);
[pos-bd] (D) edge [bend right=10] (C);
(D) edge [bend right=5.5] (I);
(C) edge (I);
(B) edge (I);

The causal diagram $\cG^{\mathbf{B}}$ and samples for "Backdoor" setting. There are more red larger digits and green smaller digits; larger digits are less likely to have a bar on top; red digits are less likely to have a bar on top.

From now on, our goal is to obtain a Ctf-consistent estimator of the non-identifiable target $P(\*I, \*I_{\*x'})$ w.r.t. the care set $\*W$.
[Counterfactually Consistent Estimation]theorempartest

$P^{\widehat{\cM}}(\*I, \*I'_{\*x'})$ is a Ctf-consistent estimator with respect to $\*W \subseteq \*V$ of $P^{\cM^*}(\*I, \*I'_{\*x'})$ if $\widehat{\cM} \in \Omega_{\*I}(\cG)$ and $P^{\widehat{\cM}}(\*V, \*I) = P(\*V, \*I)$.

The above result says that any proxy ASCM that is compatible with the diagram $\cG$ and $P(\*V, \*I)$ guarantees the estimation of the target distribution being Ctf-consistent with the true one. Specifically, in order to construct Ctf-consistent estimators, apart from fitting the generator $\widehat{\cM}$ with the given observation $P(\*V, \*I)$, it is sufficient to enforce the graphical constraints into $\widehat{\cM}$.

§.§ Estimating and Sampling with NCMs

We learned in the previous section that one could generate Ctf-consistent samples by fitting observational distributions to an SCM $\widehat{\cM}$ that is compatible with the given diagram (Thm. <ref>).
In this section, we develop a practical method for training $\cG$-Constrained causal deep generative models ($\cG$-NCMs) with two primary objectives: (a) to fit the observational distribution $P(\*V, \*I)$; (b) to sample images ($\*i$) and their counterfactual counterparts ($\*i'$) from them.

Towards realizing these goals, we first acknowledge that $P^{\cM^*}(\*V, \*I)$ is typically not directly accessible in most settings, but rather its empirical counterpart $\widehat{P}^{\cM^*}(\*V, \*I) = \{\*v_k, \*i_k\}_{k=1}^n$ derived from finite datasets.
Subsequently, we will train $\widehat{\cM}$ to match this empirical distribution $\widehat{P}^{\cM^*}(\*V, \*I)$.
Given the substantial difference in the dimensions of variables $\*V$ (feature labels) and $\*I$ (images), we prefer to fit $P(\*I)$ and $P(\*V \mid \*I)$ separately.
Initially, $P(\*I)$ will be learned by minimizing the data negative log-likelihood through VAEs [Kingma & Welling, 2013].
In this context, the proxy $\cG$-NCM $\widehat{\cM}$ serves as the decoder to approximate $P(\*I \mid \widehat{\*U})$ with the prior $P(\widehat{\*U})$.
Furthermore, a separate deep neural network $Q_{\bm{\omega}}(\widehat{\*U} \mid \*I)$ is utilized to approximate the posterior $P(\widehat{\*U} \mid \*I)$, acting as the encoder, with $\bm{\omega}$ denoting the network's parameters.
The network structure corresponding to <ref> is illustrated in <ref>. The optimization objective, $L_1$, is then defined as the following evidence lower bound (ELBO) to minimize the data negative log-likelihood:

\begin{equation}
\begin{split}
&\quad \ \ L_1(\bm{\theta}, \bm{\omega}, \widehat{P}^{\cM^*}(\*V, \*I)) \\
\mathbb{E}_{\widehat{P}^{\cM^*}(\*V, \*I)}\left [\mathbb{E}_{Q_{\bm{\omega}}(\widehat{\*U} \mid \*I)}[\log P^{\widehat{\cM}}(\*i \mid \widehat{\*u})] \right]\\
&\quad - {D}_{KL}[q_{\bm{\omega}}(\widehat{\*U} \mid \*I) || P(\widehat{\*U})],
\end{split}
\end{equation}

where $\bm{\theta}$ are parameters of $\widehat{\cM}$ (see Def. <ref>) and ${D}_{KL}[\cdot \| \cdot]$ denotes KL divergence. To match $P(\*V \mid \*I)$, we minimize the cross-entropy loss $L_2$ of the true labels of an image sample and its predicted labels, which can be inferred through $Q_{\bm{\omega}}(\widehat{\*U} \mid \*I)$ and $\cM$ like [Locatello et al., 2020, Shen et al., 2022]. Namely,

\begin{equation}
\begin{split}
&\quad \ \ L_2(\bm{\theta}, \bm{\omega}, \widehat{P}^{\cM^*}(\*V, \*I)) \\
\mathbb{E}_{\widehat{P}^{\cM^*}(\*V, \*I)}\left [ D_{CE}(\*V^{\widehat{\cM}}(r(\*i)), \*v)] \right]\\
\end{split}
\end{equation}

where $r(\*i)$ corresponds to the mean (vector) of $Q_{\bm{\omega}}(\widehat{\*U} \mid \*I)$ and $D_{CE}(\cdot)$ is the cross-entropy loss.
Formally, the objective for training an NCM $\widehat{\cM}$ can be written as

\begin{equation}
\label{eq:ncm-loss}
\hspace{-5pt} L = L_1(\bm{\theta}, \bm{\omega}, \widehat{P}^{\cM^*}(\*V, \*I)) + \lambda L_2(\bm{\theta}, \bm{\omega}, \widehat{P}^{\cM^*}(\*V, \*I))
\end{equation}

where $\lambda$ is a parameter trying to balance the likelihood $P(\*V)$ and $P(\*I | \*V)$. Specifically, a larger $\lambda$ prioritizes the fit of $P(\*V \mid \*I)$ and a smaller one prioritizes the fit of $P(\*I)$ during the training stage.
Alg. <ref> implements more specifically the training procedure of an NCM. To illustrate, in line 1, the decoder is constructed based on the given causal diagram through Def. <ref>. In lines 2, all training parameters are initialized. And then in lines 3 to 6, the encoder and decoder are trained iteratively based on <ref>.
We refer to this approach as ANCM. More details about network architecture and hyperparameters used throughout this work can be found in Appendix <ref>.

Input: Data $\{\hP^{\cM^*}(\*V, \*I) = \{\*v_{k}, \*l_{k}\}_{k=1}^{n}\}$, causal diagram $\cG$, temperature $\lambda$, learning rate $\eta$, training epochs $T$.
[1]

$\widehat{\cM} \gets$ $\mathrm{NCM}(\*V, \cG)$ from Def. <ref>
Initialize parameters $\bm{\theta}$ for $\widehat{\cM}$ and $\bm{\omega}$ for the inference network $Q_{\bm{\omega}}(\widehat{\*U} \mid \*I)$
$t \gets 1$ to $T$
$L$$ \gets$$    L_1(\bm{\theta}, \bm{\omega}, \widehat{P}^{\cM^*}(\*V, \*I))$$ +$$ \lambda L_2(\bm{\theta}, \bm{\omega}, \widehat{P}^{\cM^*}(\*V, \*I))$
$\bm{\theta} \gets \bm{\theta}$$ - $$\eta \nabla L$
$\bm{\omega} \gets \bm{\omega}$$ - $$\eta \nabla L$

(a) The optimal bound of feature counterfactual queries when editing a red "3" with a bar to "6". (b) The counterfactual image generation results when editing a red "3" with a bar to "6" in the backdoor model. (c) The selected (blue circle) feature counterfactual query estimated by ANCMs and baselines.

After training the ANCM,  we first sample $\widehat{\*u}$ from $P(\widehat{\*U})$ to generate samples of the target $P(\*I, \*I_{\*x'})$.
The initial image sample $\widehat{\*i}$ could be derived from $\*I^{\widehat{\cM_{\*x'}}}(\widehat{\*u})$, where $\*I^{\widehat{\cM_{\*x'}}}$ is the network mapping from $\widehat{\*u}$ to $\*i$ in the decoder $\widehat{\cM}$. To edit the concept $\*X = \*x'$, the counterfactual image sample $\widehat{\*i}_{\*x'}$ could be derived through $\*I^{\widehat{\cM_{\*x'}}}(\widehat{\*u})$, where $\*I^{\widehat{\cM_{\*x'}}}$ is the network but evaluated through submodel $\widehat{\cM}_{\*x'}$ of the trained NCM.

§ EXPERIMENTS

In this section, we conduct an empirical evaluation of the methods proposed in the paper, beginning with a modified Colored MNIST dataset (based on <ref>) and then moving on to CelebA-HQ dataset [Karras et al., 2018] (which describes peoples' faces) (<ref>).
Further details of the model architectures are provided in Appendix <ref>.

§.§ Colored MNIST with Bars

We first conduct experiments on the modified handwritten MNIST dataset [Deng, 2012], featuring colored digits and a horizontal blue bar in images.
[A bar in an image refers to complete rows of blue pixels.]
The observed generative factors include $\{D, C, B\}$, where $D$ denotes the digits from 0 to 9; $C$ indicates the digit color (green for $C=0$; red for $C=1$); $B$ determines whether the top of the image features a blue bar ($B=1$) or not ($B=0$). We explore two settings, named "Backdoor" (<ref>) and "Frontdoor" (<ref>), each defined by unique causal relationships among the generative factors.
This is an attractive dataset since we have full control over the generative process (SCM) and the ground truth is well-defined.
For each task, we illustrate the concept of counterfactual editing and demonstrate that our method is capable of achieving great success in counterfactual editing tasks when compared with baselines.

(a) The optimal bound of feature counterfactual queries when removing the bar for a green "1". (b) The counterfactual image generation results when removing the bar for a green "1" in the backdoor model. (c) The selected (blue circle) feature counterfactual query estimated by ANCMs and baselines.

§.§.§ MNIST Backdoor Model

In the Backdoor setting, the digit ($B$) and the color ($C$) are confounded with a positive correlation, but they do not directly affect each other. There are more red/larger ($\geq 5$) digits and green/smaller ($< 5$) digits in the dataset. The digit ($D$) has a negative effect on the existence of the bar ($B$). Larger digits are less likely to have a bar on the top. The color ($C$)  also has a negative effect on the existence of the bar ($B$). Red digits are less likely to have a bar on top. The true and unknown ASCM $\cM^{\mathbf{B}}$ is given by:

\begin{equation}
\left \{
\begin{aligned}
D &\leftarrow U_D \\
C &\leftarrow \bern(0.95 - 0.1U_D) \\
B &\leftarrow (\mathbf{1}\left [ U_D \geq 5\right ] \oplus U_1) \lor (C \oplus U_2) \land U_3 \\
\*I &\leftarrow f^\mathbf{B}_{\*I}(D, C, B, \*U_{\*I}),\\
\end{aligned}
\right .
\end{equation}

where the exogenous variables' distributions are:

\begin{equation}
\begin{split}
U_D &\sim \mathrm{Uniform}[0, 9] \\
U_1 &\sim \bern(0.8)\\
U_2 &\sim \bern(0.9) \\
U_3 &\sim \bern(0.75) \\
\end{split}
\end{equation}

The mechanism $f^\mathbf{B}$ maps the observed generative factors $\{D, C, B\}$ and unobserved generative factors (such as the position and thickness of the digit) produced by $\*U_{\*I}$ to the image $\*I$. <ref> shows the causal diagram $\cG^{\mathbf{B}}$ induced by $\cM^{\mathbf{B}}$.
The green edge indicates a positive effect and the red one represents a negative effect. We randomly sample 40 images from the collected samples in this setting and show them in <ref>.

Task 1: Counterfactually Editing the Digits

The first case we consider is to counterfactually edit the digit $D$. We let the cared features be the digit, color, and whether the image has a bar, namely, $\*W = \{B, C, D\}$. This implies that we do not care about how other generative factors (i.e., position, thickness) change in the counterfactual world.
For counterfactual editing, changing $D$ should not affect $C$ while it might possibly change $B$, since $D$ is confounded with $C$ but has a direct effect on $B$ (<ref>).

For instance, suppose we are editing a red "3" with a bar (an image with $\{D=3, C=1, B=1\}$) and wonder what would happen had the digit "3" been a "6". In this case, the optimal bounds of conditional feature counterfactual distribution $P(D_{D=6}, C_{D=6}, B_{D=6} \mid D=3, C = 1, B=1)$ derived from the observational distribution $P(D, C, B, \*I)$ and the causal diagram $\cG^{\mathbf{B}}$ are shown in <ref>. Specifically, the probability that the counterfactual image has features $\{D=6, C=1, B=1\}$ is

\begin{equation}
\begin{split}
P(C_{D=6}=1, B_{D=6} = 1 \mid D=6, C=1, B = 1) \in [0, 0.34].
\end{split}
\end{equation}

Further, the probability that the counterfactual image has features $\{D=6, C=1, B=0\}$ is

\begin{equation}
P(C_{D=6}=1, B_{D=6} = 0 \mid D=6, C=1, B = 1) \in [0.66, 1].
\end{equation}

The probability that the counterfactual image has other features (such as green "6", red "7") is zero. In other words, the counterfactual image by causal generative models can only be a red "6" with a bar ($\{D=6, C=1, B=0\}$) or a red "6" without a bar ($\{D=6, C=1, B=0\}$). Furthermore, the probability of the latter scenario where the bar disappears should be no less than $0.66$ due to the effect from $D$ to $B$. To achieve Ctf-consistency, we expect the generation process to follow these theoretical bounds.

[fill=white, draw=gray,dashed] (-1.8,-1.5) rectangle (1.8, 0.5);
(D) at (-1.5, 0) [label=below:$D$,point];
(C) at (0,-1) [label=above:$C$,point];
(B) at (1.5,-1) [label=below:$B$, point];
(I) at (0,-2) [label=below:$\*I$,point];

[neg] (D) edge (C);
[neg] (C) edge (B);
[pos-bd] (D) edge [bend left=30] (B);
(D) edge (I);
(C) edge (I);
(B) edge (I);

The causal diagram $\cG^{\mathbf{F}}$ and samples for "Frontdoor" setting. Bigger digits are likely to be green; red digits are less likely to have a bar on top; there are bigger digits with bars and smaller digits without bars.

The full counterfactual image editing results are shown in <ref>.

After changing the digit, CVAE is likely to change the color $C$ as it uses the correlation between $D$ and $C$, while $D$ and $C$ are spuriously correlated, as discussed earlier.
Also, the CVAE fails to capture the causal effect from $D$ to $B$ since the bar hardly disappears after the intervention $do(D=6)$.
CGN preserves the values of both $C$ and $B$ after the intervention $D$ since it learns independent mechanisms from the generative factors to images and all generative factors are independent given the label (see more details in the Appendix <ref>). The results suggest that the causal effect from $D$ to $B$ is not reflected in this estimation.
DEAR follows a Markovian graph and ignores bi-direct edges in the causal diagram. Thus, DEAR fails to fit the observational distribution according to Prop. <ref> and cannot preserve the color after the intervention.
ANCM preserves the original colors in counterfactual images and is likely to remove the bar, reflecting the bound value discussed above.

These results provide a qualitative suggestion that ANCM generates more realistic images that preserve the causal features found in the true generation model. Still, we would like to quantitatively understand these results, so we re-run each method 4 times and calculate the empirical probability that counterfactual images describe a red "6" without a bar after editing a red "3" with a bar to digit "6", namely $P(C_{D=6}=1, B_{D=6} = 0 \mid D=6, C=1, B = 1)$. The corresponding results are shown in <ref>.
To illustrate, the gray dashed line denotes the value given by the above ASCM $\cM^{\mathbf{B}}$, which is unknown by any of the methods. The red line represents the upper bound of the feature counterfactual query and the green line represents the lower bound. We can see that queries generated by all baseline methods are not within the optimal bound.
On the other hand, the queries generated by ANCMs are all within the optimal bound, thus they are also not far from the unknown value and are the best that can be obtained without further assumptions (over the ASCM). Both the visualization, numerical results, and theoretical results state the ANCMs are able to capture the causal effects among $\{D, C, B\}$ and produce Ctf-consistent estimators while the baselines do not.

Task 2: Counterfactually Editing the Bars

We now consider editing the bar $B$. We also let the care set to be $\*W = \{B, C, D\}$.
Based on the causal diagram $\cG^{\mathbf{B}}$, we can see that changing $B$ should affect neither $D$ nor $C$. This is because $B$ is a descendant of $D$ and $C$ and not the other way around.

For concreteness, suppose we are editing an image describing a green "1" with a bar (an image with $\{D=1, C=0, B=1\}$) and wonder what would happen had the bar been removed.
In this case, the optimal bounds of conditional feature counterfactual distribution $P(D_{B=0}, C_{B=0} \mid D=1, C = 0, B=1)$ are shown in <ref>. Specifically, the optimal bound of the probability that the counterfactual image has features $\{D=1, C=0, B=0\}$ is

\begin{equation}
P(D_{B=0}=1, C_{B=0}=0 \mid D=1, C=0, B=0) \in [1, 1]
\end{equation}

Further, the probability that the counterfactual image has other features (such as red "1" and green "5") is zero.
In other words, the counterfactual images must be an image describing a green "1" without a bar.

The counterfactual image editing results of ANCM and baselines are shown in <ref>. All methods remove bars in counterfactual images.
Since $B$ is spuriously correlated with $D$ and $C$, CVAE and DEAR are also likely to change the color to red and to the larger digit when changing $B$ to zero.
CGN successfully learns the independent mechanisms and changes the bars without affecting the original color and digit. We can see this branch of work (changing the intervened features but preserving others) does work for some causal relationships, but there are situations where these methods cannot work, for instance, Task 1 above (see more discussion about this in Appendix <ref>).
Our method ANCM also preserves these two features in counterfactual images.

These results provide a qualitative suggestion that ANCM and CGN is able to generate realistic results images that preserve the causal features found in the original model in this setting.
In order to quantitatively understand these results, we re-run each method 4 times and calculate the empirical probability that counterfactual images describe a green "1" without bar after $do(B=0)$, namely $P(D_{B=0}=1, C_{B=0}=0 \mid D=1, C=0, B=0)$ (<ref>).
The upper bound, lower bound, and the true value collapse to one line, which implies $P(D_{B=0}=1, C_{B=0}=0 \mid D=1, C=0, B=0)=1$. We can see that queries generated by CVAEs and DEAR are much smaller than the true value 1 while the queries generated by CGNs and ANCMs coincide with the true value.  Both the visualization and the numerical results suggest that CGNs and ANCMs provide Ctf-consistent estimators.

(a) The optimal bound of feature counterfactual queries when editing a green "7" with a bar to "2". (b) The counterfactual image generation results when editing a green "7" with a bar to "2" in the frontdoor model. (c) The selected (blue circle) feature counterfactual query estimated by CVAEs and ANCMs.

§.§.§ MNIST Frontdoor Model

In the Frontdoor setting, the digit ($D$) has a negative effect on the color ($C$). Larger ($\geq 5$) digits are more likely to be green. The color $C$ has a negative effect on the existence of the bar ($B$). Red digits are less likely to have a bar on top. The digit ($D$) is confounded with the existence of the bar ($B$) with a positive correlation, but do not directly affect each other. There are larger ($\geq 5$) digits with bars and smaller ($<5$) digits without bars in the dataset. The true and unknown ASCM $\cM^{\mathbf{F}}$ is given by:

\begin{equation}
\left \{
\begin{aligned}
D &\leftarrow U_D \\
C &\leftarrow \bern(0.05 + 0.1U_D) \\
B &\leftarrow (\mathbf{1}\left [ D < 5\right ] \oplus U_2) \lor (C \oplus U_1) \land U_3 \\
\*I &\leftarrow f^\mathbf{F}_{\*I}(D, C, B, \*U_{\*I}),\\
\end{aligned}
\right .
\end{equation}

where the exogenous variable distributions are:

\begin{equation}
\begin{split}
U_D &\sim \mathrm{Uniform}[0, 9] \\
U_1 &\sim \bern(0.8)\\
U_2 &\sim \bern(0.9) \\
U_3 &\sim \bern(0.7)
\end{split}
\end{equation}

The mechanism $f^\mathbf{F}$ maps the observed generative factors $\{D, C, B\}$ and unobserved generative factors (such as the position and thickness of digits) produced by $\*U_{\*I}$ to the image $\*I$.
<ref> shows the causal diagram $\cG^{\mathbf{F}}$ induced by $\cM^{\mathbf{F}}$.
The green edge indicates a positive effect and the red one represents a negative effect.
We randomly sample 40 images from the collected samples in this setting and show them in <ref>.

(a) The optimal bound of feature counterfactual queries when editing a red "4" without a bar to green. (b) The counterfactual image generation results when editing a red "4" with a bar to green in the frontdoor model. (c) The selected (blue circle) feature counterfactual query estimated by ANCMs and baselines.

Task 3: Counterfactually Editing the Digits

We first consider editing the digit $D$.
We let the care set be $\*W = \{B, C, D\}$ similar to previous cases.
Based on the causal diagram $\cG^{\mathbf{F}}$, changing $D$ should directly affect $C$ and is possible to change indirectly through $B$, since $D$ is the direct parent of $C$ but not a parent of $B$ (even though it's an ancestor).
For instance, suppose we are editing a green "7" with a bar (an image with $\{D=7, C=0, B=1\}$) and wonder what would happen had the "7" been "2".
In this case, the optimal bounds of conditional feature counterfactual distribution $P(C_{D=2}, B_{D=2} \mid D=7, C = 0, B=1)$ derived from the observational distribution $P(D, C, B, \*I)$ and the causal diagram $\cG^{\mathbf{F}}$ are shown in <ref>.
Specifically, the probability that the counterfactual image has features $\{D=2, C=0, B=1\}$ is

\begin{equation}
P(C_{D=2}=0, B_{D=2} = 1 \mid D=7, C=0, B = 1) \in [0, 0.33].
\end{equation}

And the probability that the counterfactual image has features $\{D=2, C=0, B=0\}$ is

\begin{equation}
P(C_{D=2}=0, B_{D=2} =0 \mid D=7, C=0, B = 1) \in [0, 0].
\end{equation}

Further, the probability that the counterfactual image has features $\{D=2, C=1\}$ is

\begin{equation}
P(D_{D=2}=2, C_{D=2}=1 \mid D=6, C=1, B = 1) \in [0.67, 1].
\end{equation}

And the probability that the counterfactual image has other features (such as green "7" and red "7") is zero.
In other words, the counterfactual image by causal generative models can only be a green "2" with a bar ($\{D=6, C=1, B=0\}$) or a red "2" ($\{D=6, C=1, B=0\}, \{D=6, C=1, B=1\}$). Since $D$ can only indirectly affect $B$ through $C$, changing the digit will not influence the bar if the color remains the same. Thus, we expect that the counterfactual image generated by estimation methods would not be a green "2" without a bar.

The counterfactual image editing results of ANCM and baselines are shown in <ref>.

ANCM and CVAE methods generate red digits with bars and red digits without bars, which implies they capture the effect from $D$ to $C$ and $B$.
However, CVAE generates counterfactual images describing green "2" without bars since $D$ and $B$ are correlated. ANCM does not change the existence of the bar when $C$ remains the same. This implies that ANCM captures the indirect effect from $D$ to $B$ as discussed above.
CGN fails to capture the effect from $D$ to $C$ and $B$ and simply keep $C$ and $B$ the same. DEAR also fails to capture the correct causal relationships since the graph encoded in the network is incorrect and the data distribution cannot be fit.

Editing results of the CelebaHQ Experiment.

To quantitatively understand these results, we re-run each method 4 times and calculate the empirical probability that counterfactual images describing a green "2" without bars after editing a green "7" with a bar to digit "2", namely $P(C_{D=2}=0, B_{D=2} = 0 \mid D=7, C=0, B = 1)$. The results are shown in <ref>.
The upper bound, lower bound, and the feature counterfactual query generated by the true model collapse to one line, which implies $P(C_{D=2}=0, B_{D=2} = 0 \mid D=7, C=0, B = 1) = 0$. We can see that the query estimated by CVAE and DEAR are greater than 0.
The query estimated by CGN is 0 but this is because CGNs fail to capture any (indirect or direct) effect from the digits to bars.
In contrast, the queries generated by ANCMs are exactly 0. Both the visualization and the numerical results state that ANCMs capture the full causal invariances among $\{D, C, B\}$ and produce Ctf-consistent estimators while the other methods do not.

Task 4: Counterfactually Editing the Colors

We then consider editing the color $\*C$ of digits. Similarly, we let $\*W = \{B, C, D\}$.
Based on the causal diagram $\cG^{\mathbf{F}}$, we can see that changing $C$ should not affect $D$ and is possible to change $B$ since $C$ is confounded with $D$ and has a direct effect on $B$ in <ref>.
For example, suppose we are editing a red "4" without a bar (an image with $\{D=4, C=1, B=0\}$) and wonder what would happen had the color been green. In this case, the optimal bounds of conditional feature counterfactual distribution $P(D_{C=0}, B_{C=0} \mid D=4, C = 1, B=0)$ derived from the observational distribution $P(D, C, B, \*I)$ and the causal diagram $\cG^{\mathbf{F}}$ are shown in <ref>.
Specifically, the probability that the counterfactual image has features $\{D=4, C=0, B=1\}$ is

\begin{equation}
P(D_{C=0}=4, B_{C=0} = 1 \mid D=4, C=1, B = 0) \in [0.12, 1].
\end{equation}

Futher the probability that the counterfactual image has features $\{D=4, C=1, B=0\}$ is

\begin{equation}
P(D_{C=0}=4, B_{C=0} = 0 \mid D=4, C=1, B = 0) \in [0, 0.88].
\end{equation}

And the probability that the counterfactual image has other features (such as green "9" and red "4") is zero. In other words, the counterfactual image by causal generative models can only be a green "4" and the bar will appear with a probability at least $0.12$ after changing the color to green.

The counterfactual image editing results of ANCM and the baselines are shown in <ref>.

CVAE and DEAR are also likely to change the digit since $D$ is spuriously correlated with each other and DEAR fails to fit the given distribution.
CGN fails to capture the direct causal effect from the color to the bar and preserves the digits and the bars as the same.
ANCM preserves the original digit in counterfactual images after editing and the change of $B$ shows up.

These results provide a qualitative suggestion that ANCM generates more realistic results images that preserve the causal features found in the original model.
To quantitatively understand these results, we re-run each method 4 times and calculate the empirical probability that counterfactual images describe a green "4" with a bar after editing a red "4" without a bar to green, namely $P(D_{C=0}=4, B_{C=0} = 1 \mid D=4, C=1, B = 0)$. The results are shown in <ref>.
We can see that ANCMs present Ctf-consistent estimators while queries generated by CVAEs and CGN are nearly 0, which is not within the bound. DEARs roughly present Ctf-consistent estimators for $P(D_{C=0}=4, B_{C=0} = 1 \mid D=4, C=1, B = 0)$, but the former visualization results demonstrate DEAR cannot preserve the digit, which means it cannot provide Ctf-consistent estimators for the whole distribution.
Both the visualization and the numerical results state that ANCMs capture the causal invariances among $\{D, C, B\}$ while the baseline methods do not.

§.§ Celeba-HQ

In CelebA-HQ experiment, we consider two causal diagrams as shown in Fig. <ref>. In the first experiment, we consider generative factors $Smile$ $(S)$ and $Open\ Mouth$ $(O)$, and in the second experiment, we consider $Female$ $(F)$, $Young$ $(Y)$ and $Grayhair$ $(H)$.
The first target counterfactual queries are "What would the image be had the person opened the mouth?", and the second is "What would the image be had the person been older?". The feature sets are $\*W = \{S, O\}$ and $\*W = \{F, Y, H\}$ in these two settings, respectively.
We also compare ANCM (ours) against the CVAE and DEAR baselines. CGN is not compared here since the variables of CGN are restricted to $Shape, Texture$ and $Background$. Meanwhile, DiffuseVAE [Pandey et al., 2022] is leveraged for ANCM and CVAE here to refine samples to high quality since VAEs often produce blurry images that lack high-frequency information [Dosovitskiy & Brox, 2016].

The empirical results are shown in Fig. <ref>. In the first setting, the feature set $\*W = \{S, O\}$ implies the counterfactual query is $P(S, O, S_{O=1})$, namely, "Would the person smile (or not) had the person opened the mouth?".
The constraints induced by the ground truth model imply that changing the mouth should not affect smiling since $O$ is the direct child of $S$ and not the other way around.
As shown <ref>, the smiling features are preserved after the editing by ANCM and DEAR. However, CVAE only captures the correlation between these factors, thus the non-smiling person changes to smiling after editing of mouth. On the other hand, ANCM produces higher-quality images compared to DEAR.

The second causal diagram indicates the correlations between gender and age in Example <ref>. The dataset has more face images of young females and old males.
More specifically,  71% of the young people are female and 66% of the old people are male.
The features set $\*W = \{F, Y, H\}$ implies the counterfactual distribution is "What would the gender and the hair color of the person be had the person been older?". The causal constraints suggest that the gender of the person should be preserved and the likelihood of gray hair should increase.
ANCM matches these causal relationships while baselines may change the original gender as shown in <ref>, which is of course undesirable.

§ CONCLUSIONS

We study the problem of counterfactual image generation and editing through formal causal language. Our goal in this paper is to provide guarantees that when a particular feature is edited within an image, the resulting changes in other generative factors are faithfully reflected in the counterfactual images produced.
We formally showed that image counterfactual distributions are not identifiable from a combination of observational data and prior causal knowledge about the generating model represented as a causal diagram.
In such non-identifiable cases, we propose new estimators (Ctf-consistent) that come accompanied with guarantees that the generated counterfactual images remain causally consistent with the true image counterfactual distribution for any causal relationship between generative factors.
We developed an efficient algorithm to train neural causal models and sample counterfactual images.
Finally, we demonstrate our methods are able to generate high-quality counterfactual images for synthetic images.

Building on the machinery developed in this paper, and the understanding gained from it about image counterfactual generation, we identify some future challenges to extend these results in a broader range of practical settings.
First, the set of features of interests, referred to as the “care set” in the paper  ($\*W$), guarantees the preservation of invariances and causal relations across counterfactual conditions. Although we consider settings where the set $\*W$ is labeled,  the challenge of handling unlabeled data in many practical situations remains significant.
Second, another important area for future research is enhancing the efficiency and scalability of the inferences made in this paper.

Impact statemnet.
This paper presents work whose goal is to advance the field of machine learning. There are many societal implications of our work and we hope to be beneficial, as elaborated next.  Reflecting on the broader literature, we propose the first method capable of providing formal guarantees over counterfactual image generation and editing.
The main advancement of our work lies in its emphasis on preserving the causal relationships among features, enabling sound, robust, and more realistic counterfactual generation.
This approach differs significantly from the existing literature, which primarily focuses on reflecting the intervened features in the image.
The critical distinction centers on what happens with the other features that were not intervened upon, and determining which features are shared or not between factual and counterfactual worlds.
Although almost never formally articulated, there are two prevalent approaches to this problem in the prior literature.
Some works remain silent regarding the counterfactual status of the non-intervened features. This means that the neural network might leverage the correlation between features found in the factual world, leading to the various spurious results discussed earlier.
For instance, instructing a generative AI to change a specific feature of an individual might result in a completely different person with other features, such as a different gender or race, despite they are not being causally related.
This occurs because the neural model tends to leverage the correlation between factors found in the observational data, which is oblivious to their causal relationship.
Other works attempt to ensure that the non-intervened features are preserved across factual and counterfactual worlds.
However, this approach is also inadequate in settings where some of the features exert causal influence on others, and the generative AI should accordingly ascertain these relations.
For instance, making a person older should logically lead to changes in hair color (or its amount) in both factual and counterfactual images.

After all, we believe the results stemming from this work have broad implications for the development of the next generation of generative AI.
First, we note that the training datasets used for large generative models are almost never balanced (see, for example, [Buolamwini & Gebru, 2018]), which implies spurious correlations across features and the generated images.
In practice, this often leads to more frequent, unexpected inaccuracies and biases in these models (e.g., refer to [Plecko & Bareinboim, 2022]. )
Understanding and accounting for the causal relationships among generative factors is fundamental for the accuracy and fairness of these models.
Second, the lack of proper treatment of the causal invariances required for sound counterfactual reasoning translates into the impossibility of providing any sort of guarantees over what these models generate as output and their plausibility, a certainly undesirable state of affairs.

§ ACKNOLWEDGEMENTS

This research was supported in part by the NSF, ONR, AFOSR, DARPA, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation.
We thank Kevin Xia for the feedback provided in early versions of this manuscript.

[Augustin et al., 2022]
Augustin, M., Boreiko, V., Croce, F., and Hein, M.
Diffusion visual counterfactual explanations.
Advances in Neural Information Processing Systems, 35:0 364–377, 2022.

[Avin et al., 2005]
Avin, C., Shpitser, I., and Pearl, J.
Identifiability of Path-Specific Effects.
In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, pp.  357–363. Morgan-Kaufmann Publishers, 2005.

[Avrahami et al., 2022]
Avrahami, O., Lischinski, D., and Fried, O.
Blended diffusion for text-driven editing of natural images.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18208–18218, 2022.

[Ba et al., 2016]
Ba, J. L., Kiros, J. R., and Hinton, G. E.
Layer normalization.
arXiv preprint arXiv:1607.06450, 2016.

[Balke & Pearl, 1994]
Balke, A. and Pearl, J.
Counterfactual Probabilities: Computational Methods, Bounds, and Applications.
In de Mantaras, R. L. and D.Poole (eds.), Uncertainty in Artificial Intelligence 10, pp.  46–54. Morgan Kaufmann, San Mateo, CA, 1994.

[Bareinboim et al., 2015]
Bareinboim, E., Forney, A., and Pearl, J.
Bandits with unobserved confounders: A causal approach.
In Advances in Neural Information Processing Systems, pp.  1342–1350, 2015.

[Bareinboim et al., 2022]
Bareinboim, E., Correa, J. D., Ibeling, D., and Icard, T.
On pearl’s hierarchy and the foundations of causal inference.
In Probabilistic and Causal Inference: The Works of Judea Pearl, pp.  507–556. Association for Computing Machinery, New York, NY, USA, 2022.

[Bengio et al., 2013]
Bengio, Y., Courville, A., and Vincent, P.
Representation learning: A review and new perspectives.
IEEE transactions on pattern analysis and machine intelligence, 350 (8):0 1798–1828, 2013.

[Brehmer et al., 2022]
Brehmer, J., De Haan, P., Lippe, P., and Cohen, T. S.
Weakly supervised causal representation learning.
Advances in Neural Information Processing Systems, 35:0 38319–38331, 2022.

[Brock et al., 2019]
Brock, A., Donahue, J., and Simonyan, K.
Large scale GAN training for high fidelity natural image synthesis.
In International Conference on Learning Representations, 2019.
URL <https://openreview.net/forum?id=B1xsqj09Fm>.

[Brooks et al., 2023]
Brooks, T., Holynski, A., and Efros, A. A.
Instructpix2pix: Learning to follow image editing instructions.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18392–18402, 2023.

[Buolamwini & Gebru, 2018]
Buolamwini, J. and Gebru, T.
Gender shades: Intersectional accuracy disparities in commercial gender classification.
In Conference on fairness, accountability and transparency, pp.  77–91. PMLR, 2018.

[Chai et al., 2021]
Chai, L., Wulff, J., and Isola, P.
Using latent space regression to analyze and leverage compositionality in {gan}s.
In International Conference on Learning Representations, 2021.

[Chen et al., 2018]
Chen, T. Q., Li, X., Grosse, R. B., and Duvenaud, D. K.
Isolating sources of disentanglement in variational autoencoders.
In NeurIPS, 2018.

[Child, 2021]
Child, R.
Very deep {vae}s generalize autoregressive models and can outperform them on images.
In International Conference on Learning Representations, 2021.

[Correa et al., 2021]
Correa, J., Lee, S., and Bareinboim, E.
Nested counterfactual identification from arbitrary surrogate experiments.
Advances in Neural Information Processing Systems, 34:0 6856–6867, 2021a.

[Correa et al., 2021]
Correa, J., Lee, S., and Bareinboim, E.
Nested counterfactual identification from arbitrary surrogate experiments.
In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  6856–6867. Curran Associates, Inc., 2021b.

[Crowson et al., 2022]
Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., and Raff, E.
Vqgan-clip: Open domain image generation and editing with natural language guidance.
In European Conference on Computer Vision, pp.  88–105. Springer, 2022.

[Deng, 2012]
Deng, L.
The mnist database of handwritten digit images for machine learning research [best of the web].
IEEE signal processing magazine, 290 (6):0 141–142, 2012.

[Dhurandhar et al., 2018]
Dhurandhar, A., Chen, P.-Y., Luss, R., Tu, C.-C., Ting, P., Shanmugam, K., and Das, P.
Explanations based on the missing: Towards contrastive explanations with pertinent negatives.
Advances in neural information processing systems, 31, 2018.

[Donahue et al., 2017]
Donahue, J., Krähenbühl, P., and Darrell, T.
Adversarial feature learning.
In International Conference on Learning Representations, 2017.

[Dosovitskiy & Brox, 2016]
Dosovitskiy, A. and Brox, T.
Generating images with perceptual similarity metrics based on deep networks.
Advances in neural information processing systems, 29, 2016.

[Dumoulin et al., 2017]
Dumoulin, V., Belghazi, I., Poole, B., Lamb, A., Arjovsky, M., Mastropietro, O., and Courville, A.
Adversarially learned inference.
In International Conference on Learning Representations, 2017.

[Falcon & Cho, 2020]
Falcon, W. and Cho, K.
A framework for contrastive self-supervised learning and designing a new approach.
arXiv preprint arXiv:2009.00104, 2020.

[Gal et al., 2022]
Gal, R., Patashnik, O., Maron, H., Bermano, A. H., Chechik, G., and Cohen-Or, D.
Stylegan-nada: Clip-guided domain adaptation of image generators.
ACM Transactions on Graphics (TOG), 410 (4):0 1–13, 2022.

[Goetschalckx et al., 2019]
Goetschalckx, L., Andonian, A., Oliva, A., and Isola, P.
Ganalyze: Toward visual definitions of cognitive image properties.
In Proceedings of the ieee/cvf international conference on computer vision, pp.  5744–5753, 2019.

[Goodfellow et al., 2020]
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.
Generative adversarial networks.
Communications of the ACM, 630 (11):0 139–144, 2020.

[Goyal et al., 2019]
Goyal, Y., Feder, A., Shalit, U., and Kim, B.
Explaining classifiers with causal concept effect (cace).
arXiv preprint arXiv:1907.07165, 2019a.

[Goyal et al., 2019]
Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., and Lee, S.
Counterfactual visual explanations.
In International Conference on Machine Learning, pp.  2376–2384. PMLR, 2019b.

[Härkönen et al., 2020]
Härkönen, E., Hertzmann, A., Lehtinen, J., and Paris, S.
Ganspace: Discovering interpretable gan controls.
Advances in Neural Information Processing Systems, 33:0 9841–9850, 2020.

[He et al., 2016]
He, K., Zhang, X., Ren, S., and Sun, J.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.

[Heckman, 1992]
Heckman, J. J.
Randomization and Social Policy Evaluation.
In Manski, C. and Garfinkle, I. (eds.), Evaluations: Welfare and Training Programs, pp.  201–230. Harvard University Press, Cambridge, MA, 1992.

[Higgins et al., 2017]
Higgins, I., Matthey, L., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A.
beta-vae: Learning basic visual concepts with a constrained variational framework.
In ICLR, 2017.

[Ho et al., 2020]
Ho, J., Jain, A., and Abbeel, P.
Denoising diffusion probabilistic models.
Advances in neural information processing systems, 33:0 6840–6851, 2020.

[Ho et al., 2022]
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T.
Cascaded diffusion models for high fidelity image generation.
The Journal of Machine Learning Research, 230 (1):0 2249–2281, 2022.

[Hyvärinen & Pajunen, 1999]
Hyvärinen, A. and Pajunen, P.
Nonlinear independent component analysis: Existence and uniqueness results.
Neural networks, 120 (3):0 429–439, 1999.

[Jaber et al., 2018]
Jaber, A., Zhang, J., and Bareinboim, E.
Causal identification under Markov equivalence.
In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, pp.  978–987. AUAI Press, Aug 2018.

[Jaber et al., 2019]
Jaber, A., Zhang, J., and Bareinboim, E.
Identification of conditional causal effects under Markov equivalence.
In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32, pp.  11512–11520. Curran Associates, Inc., 2019.

[Jaber et al., 2020]
Jaber, A., Kocaoglu, M., Shanmugam, K., and Bareinboim, E.
Causal discovery from soft interventions with unknown targets: Characterization and learning.
In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  9551–9561. Curran Associates, Inc., 2020.

[Jaber et al., 2022]
Jaber, A., Ribeiro, A., Zhang, J., and Bareinboim, E.
Causal identification under markov equivalence: Calculus, algorithm, and completeness.
In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.  3679–3690. Curran Associates, Inc., 2022.

[Jahanian* et al., 2020]
Jahanian*, A., Chai*, L., and Isola, P.
On the "steerability" of generative adversarial networks.
In International Conference on Learning Representations, 2020.
URL <https://openreview.net/forum?id=HylsTT4FvB>.

[Joshi et al., 2019]
Joshi, S., Koyejo, O., Vijitbenjaronk, W., Kim, B., and Ghosh, J.
Towards realistic individual recourse and actionable explanations in black-box decision making systems.
arXiv preprint arXiv:1907.09615, 2019.

[Karras et al., 2018]
Karras, T., Aila, T., Laine, S., and Lehtinen, J.
Progressive growing of GANs for improved quality, stability, and variation.
In International Conference on Learning Representations, 2018.

[Karras et al., 2019]
Karras, T., Laine, S., and Aila, T.
A style-based generator architecture for generative adversarial networks.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4401–4410, 2019.

[Khemakhem et al., 2020]
Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A.
Variational autoencoders and nonlinear ica: A unifying framework.
In International Conference on Artificial Intelligence and Statistics, pp.  2207–2217. PMLR, 2020.

[Khorram & Fuxin, 2022]
Khorram, S. and Fuxin, L.
Cycle-consistent counterfactuals by latent transformations.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10203–10212, 2022.

[Kim & Mnih, 2018]
Kim, H. and Mnih, A.
Disentangling by factorising.
In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.  2649–2658. PMLR, 10–15 Jul 2018.
URL <https://proceedings.mlr.press/v80/kim18b.html>.

[Kingma & Ba, 2014]
Kingma, D. P. and Ba, J.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.

[Kingma & Welling, 2013]
Kingma, D. P. and Welling, M.
Auto-encoding variational bayes.
arXiv preprint arXiv:1312.6114, 2013.

[Kocaoglu et al., 2017]
Kocaoglu, M., Shanmugam, K., and Bareinboim, E.
Experimental design for learning causal graphs with latent variables.
In Advances in Neural Information Processing Systems 30, pp.  7018–7028. Curran Associates, Inc., 2017a.

[Kocaoglu et al., 2017]
Kocaoglu, M., Snyder, C., Dimakis, A. G., and Vishwanath, S.
Causalgan: Learning causal implicit generative models with adversarial training.
arXiv preprint arXiv:1709.02023, 2017b.

[Kocaoglu et al., 2018]
Kocaoglu, M., Snyder, C., Dimakis, A. G., and Vishwanath, S.
CausalGAN: Learning causal implicit generative models with adversarial training.
In International Conference on Learning Representations, 2018.

[Kocaoglu et al., 2019]
Kocaoglu, M., Jaber, A., Shanmugam, K., and Bareinboim, E.
Characterization and learning of causal graphs with latent variables from soft interventions.
In Advances in Neural Information Processing Systems 32, pp.  14346–14356, Vancouver, Canada, 2019. Curran Associates, Inc.

[Kwon & Ye, 2022]
Kwon, G. and Ye, J. C.
Clipstyler: Image style transfer with a single text condition.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18062–18071, 2022.

[Lachapelle et al., 2021]
Lachapelle, S., Rodriguez, P., Sharma, Y., Everett, K. E., Le Priol, R., Lacoste, A., and Lacoste-Julien, S.
Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica.
In First Conference on Causal Learning and Reasoning, 2021.

[Lee et al., 2020]
Lee, S., Correa, J., and Bareinboim, E.
Generalized transportability: Synthesis of experiments from heterogeneous domains.
In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY, 2020. AAAI Press.

[Li et al., 2023]
Li, A., Jaber, A., and Bareinboim, E.
Causal discovery from observational and interventional data across multiple environments.
In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

[Locatello et al., 2019]
Locatello, F., Bauer, S., Lucic, M., Gelly, S., Schölkopf, B., and Bachem, O.
Challenging common assumptions in the unsupervised learning of disentangled representations.
ArXiv, abs/1811.12359, 2019a.

[Locatello et al., 2019]
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O.
Challenging common assumptions in the unsupervised learning of disentangled representations.
In international conference on machine learning, pp.  4114–4124. PMLR, 2019b.

[Locatello et al., 2020]
Locatello, F., Poole, B., Raetsch, G., Schölkopf, B., Bachem, O., and Tschannen, M.
Weakly-supervised disentanglement without compromises.
In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.  6348–6359. PMLR, 13–18 Jul 2020a.
URL <https://proceedings.mlr.press/v119/locatello20a.html>.

[Locatello et al., 2020]
Locatello, F., Tschannen, M., Bauer, S., Rätsch, G., Schölkopf, B., and Bachem, O.
Disentangling factors of variations using few labels.
In International Conference on Learning Representations, 2020b.
URL <https://openreview.net/forum?id=SygagpEKwB>.

[Manski, 1990]
Manski, C. F.
Nonparametric bounds on treatment effects.
American Economic Review, Papers and Proceedings, 80:0 319–323, 1990.

[Mao et al., 2022]
Mao, C., Xia, K., Wang, J., Wang, H., Yang, J., Bareinboim, E., and Vondrick, C.
Causal transportability for visual recognition.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  7521–7531, 2022.

[Mooij et al., 2020]
Mooij, J. M., Magliacane, S., and Claassen, T.
Joint causal inference from multiple contexts.
The Journal of Machine Learning Research, 210 (1):0 3919–4026, 2020.

[Moraffah et al., 2020]
Moraffah, R., Karami, M., Guo, R., Raglin, A., and Liu, H.
Causal interpretability for machine learning-problems, methods and evaluation.
ACM SIGKDD Explorations Newsletter, 220 (1):0 18–33, 2020.

[Nasr-Esfahany & Kiciman, 2023]
Nasr-Esfahany, A. and Kiciman, E.
Counterfactual (non-) identifiability of learned structural causal models.
arXiv preprint arXiv:2301.09031, 2023.

[Pandey et al., 2022]
Pandey, K., Mukherjee, A., Rai, P., and Kumar, A.
DiffuseVAE: Efficient, controllable and high-fidelity generation from low-dimensional latents.
Transactions on Machine Learning Research, 2022.
ISSN 2835-8856.

[Paszke et al., 2017]
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A.
Automatic differentiation in pytorch.

[Patashnik et al., 2021]
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D.
Styleclip: Text-driven manipulation of stylegan imagery.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  2085–2094, 2021.

[Pawlowski et al., 2020]
Pawlowski, N., Coelho de Castro, D., and Glocker, B.
Deep structural causal models for tractable counterfactual inference.
Advances in Neural Information Processing Systems, 33:0 857–869, 2020.

[Pearl, 2000]
Pearl, J.
Causality: Models, Reasoning, and Inference.
Cambridge University Press, New York, NY, USA, 2nd edition, 2000.

[Pearl, 2001]
Pearl, J.
Direct and indirect effects.
In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI'01, pp.  411–420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc.
ISBN 1558608001.

[Pearl, 2009]
Pearl, J.
Causality: Models, Reasoning, and Inference.
Cambridge University Press, New York, 2nd edition, 2009.

[Pearl & Mackenzie, 2018]
Pearl, J. and Mackenzie, D.
The Book of Why.
Basic Books, New York, 2018.

[Peters et al., 2017]
Peters, J., Janzing, D., and Schlkopf, B.
Elements of Causal Inference: Foundations and Learning Algorithms.
The MIT Press, 2017.

[Plecko & Bareinboim, 2022]
Plecko, D. and Bareinboim, E.
Causal fairness analysis.
Technical Report R-90, Causal Artificial Intelligence Lab, Columbia University, Jul 2022.

[Radford et al., 2021]
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.
Learning transferable visual models from natural language supervision.
In International conference on machine learning, pp.  8748–8763. PMLR, 2021.

[Rodríguez et al., 2021]
Rodríguez, P., Caccia, M., Lacoste, A., Zamparo, L., Laradji, I., Charlin, L., and Vazquez, D.
Beyond trivial counterfactual explanations with diverse valuable explanations.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1056–1065, 2021.

[Saharia et al., 2022]
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M.
Image super-resolution via iterative refinement.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 450 (4):0 4713–4726, 2022.

[Samangouei et al., 2018]
Samangouei, P., Saeedi, A., Nakagawa, L., and Silberman, N.
Explaingan: Model explanation via decision boundary crossing transformations.
In Proceedings of the European Conference on Computer Vision (ECCV), pp.  666–681, 2018.

[Sanchez & Tsaftaris, 2022]
Sanchez, P. and Tsaftaris, S. A.
Diffusion causal models for counterfactual estimation.
In Schölkopf, B., Uhler, C., and Zhang, K. (eds.), Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pp.  647–668. PMLR, 11–13 Apr 2022.

[Sauer & Geiger, 2021]
Sauer, A. and Geiger, A.
Counterfactual generative networks.
In International Conference on Learning Representations, 2021.
URL <https://openreview.net/forum?id=BXewfAYMmJw>.

[Schölkopf et al., 2021]
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y.
Toward causal representation learning.
Proceedings of the IEEE, 1090 (5):0 612–634, 2021.

[Shen et al., 2022]
Shen, X., Liu, F., Dong, H., Lian, Q., Chen, Z., and Zhang, T.
Weakly supervised disentangled generative causal representation learning.
Journal of Machine Learning Research, 23:0 1–55, 2022.

[Shen et al., 2020]
Shen, Y., Gu, J., Tang, X., and Zhou, B.
Interpreting the latent space of gans for semantic face editing.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9243–9252, 2020.

[Shpitser & Pearl, 2009]
Shpitser, I. and Pearl, J.
Effects of Treatment on the Treated: Identification and Generalization.
In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, Quebec, 2009. AUAI Press.

[Shpitser & Sherman, 2018]
Shpitser, I. and Sherman, E.
Identification of Personalized Effects Associated With Causal Pathways.
In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, pp.  530–539, 2018.

[Sohn et al., 2015]
Sohn, K., Lee, H., and Yan, X.
Learning structured output representation using deep conditional generative models.
Advances in neural information processing systems, 28, 2015.

[Song et al., 2021]
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B.
Score-based generative modeling through stochastic differential equations.
In International Conference on Learning Representations, 2021.

[Spirtes et al., 2000]
Spirtes, P., Glymour, C. N., and Scheines, R.
Causation, Prediction, and Search.
MIT Press, Cambridge, MA, 2nd edition, 2000.

[Squires et al., 2020]
Squires, C., Wang, Y., and Uhler, C.
Permutation-based causal structure learning with unknown intervention targets.
In Conference on Uncertainty in Artificial Intelligence, pp.  1039–1048. PMLR, 2020.

[Vahdat & Kautz, 2020]
Vahdat, A. and Kautz, J.
Nvae: A deep hierarchical variational autoencoder.
Advances in neural information processing systems, 33:0 19667–19679, 2020.

[Van Looveren & Klaise, 2021]
Van Looveren, A. and Klaise, J.
Interpretable counterfactual explanations guided by prototypes.
In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part II 21, pp.  650–665. Springer, 2021.

[Vaswani et al., 2017]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Verma et al., 2020]
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., and Shah, C.
Counterfactual explanations and algorithmic recourses for machine learning: a review.
arXiv preprint arXiv:2010.10596, 2020.

[Von Kügelgen et al., 2021]
Von Kügelgen, J., Sharma, Y., Gresele, L., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F.
Self-supervised learning with data augmentations provably isolates content from style.
Advances in neural information processing systems, 34:0 16451–16467, 2021.

[von Kügelgen et al., 2023]
von Kügelgen, J., Besserve, M., Wendong, L., Gresele, L., Kekić, A., Bareinboim, E., Blei, D. M., and Schölkopf, B.
Nonparametric identifiability of causal representations from unknown interventions.
In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

[Wang & Vasconcelos, 2020]
Wang, P. and Vasconcelos, N.
Scout: Self-aware discriminant counterfactual explanations.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  8981–8990, 2020.

[Xia et al., 2021]
Xia, K., Lee, K.-Z., Bengio, Y., and Bareinboim, E.
The causal-neural connection: Expressiveness, learnability, and inference.
In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  10823–10836. Curran Associates, Inc., 2021.
URL <https://causalai.net/r80.pdf>.

[Xia et al., 2022]
Xia, K., Pan, Y., and Bareinboim, E.
Neural causal models for counterfactual identification and estimation.
In International Conference on Learning Representations, 2022.

[Yang et al., 2021]
Yang, M., Liu, F., Chen, Z., Shen, X., Hao, J., and Wang, J.
Causalvae: Disentangled representation learning via neural structural causal models.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9593–9602, 2021.

[Zhang et al., 2019]
Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A.
Self-attention generative adversarial networks.
In International conference on machine learning, pp.  7354–7363. PMLR, 2019.

[Zhang & Bareinboim, 2018]
Zhang, J. and Bareinboim, E.
Fairness in decision-making–the causal explanation formula.
In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp.  2037–2045, 2018.

[Zhang et al., 2022]
Zhang, J., Jin, T., and Bareinboim, E.
Partial counterfactual identification from observational and experimental data.
In Proceedings of the 39th International Conference on Machine Learning (ICML-22), 2022.

[Zhang et al., 2023]
Zhang, J., Squires, C., Greenewald, K., Srivastava, A., Shanmugam, K., and Uhler, C.
Identifiability guarantees for causal disentanglement from soft interventions.
arXiv preprint arXiv:2307.06250, 2023.

§ EXPERIMENTAL DETAILS

This section provides details about our experiments and models. Our models are primarily written in PyTorch [Paszke et al., 2017], and trained with PyTorch Lightning [Falcon & Cho, 2020].

§.§ Colored MNIST and Bar

We first provide more details of the architectures of the proposed ANCM and the other three baselines: CVAE, CGN, and DEAR.

ANCM (ours). As illustrated in Alg. <ref> and <ref>, VAE-ANCM use the VAE architecture to maximize $P(\*I)$ and $P(\*V \mid \*I)$ separately, where the decoder is a $\mathcal{G}$-constrained NCM $\widehat{\cM}$ (Def. <ref>).
Each function $\widehat{f}_{V} \ \widehat{\cF}$ in $\widehat{\cM}$ is a feedforward neural network with 2 hidden layers of width 64 with layer normalization applied [Ba et al., 2016]. Each exogenous variable $\widehat{U} \in \widehat{\*U}\backslash \widehat{\*U_{\*I}}$ is a 4-dimensional standard normal distribution. The dimension of $\widehat{\*U_{\*I}}$ is set as 512. The architectures of encoder $Q_{\*w}(\tilde{\*U} \mid \*I)$ and $\widehat{f}_{\*I}$ designed based on ResNet [He et al., 2016] and are shown in Tab. <ref>.

CVAE [Sohn et al., 2015]. With a latent vector $\*Z$ and data samples $\*i$, the CVAE is trained to maximize the evidence lower bound [Kingma & Welling, 2013] log-likelihood of the conditional image distributions, namely, $P(\*I \mid \*X)$. Specifically, the optimization objective is as follows:

\begin{equation}
\begin{split}
ELBO(\bm{\theta}, \bm{\omega}) = &\mathbb{E}_{Q_{\bm{\omega}}(\*Z \mid \*I, \*X)}[\log P_{\bm{\theta}}(\*I \mid \*Z, \*X)] \\
& - {D}_{KL}[Q_{\bm{\omega}}(\*Z \mid \*I, \*X) \Vert P(\*Z)]
\end{split}
\end{equation}

where $\*X$ is the intervened set ($D$ in this setting), $p(\*Z)$ is the guassian prior distribution of the latent vector $\*Z$, the encoder $Q_{\omega}(\*Z \mid \*I, \*X)$ is modeled as a neural network mapping from $\{\*I, \*X\}$ to $\*Z$ parametrized by $\bm{\omega}$, and the decoder $p_{\bm{\theta}}(\*I \mid \*Z, \*X)$ is modeled as a neural network mapping from $\{\*Z, \*X\}$ to $\*I$. Trained with the re-parameterization trick [Kingma & Welling, 2013], the decoder is capable of generating samples from $P(\*I \mid \*X)$ ideally.
We choose the dimension of $\*Z$ as 512, and the decoder and encoder are chosen the same as ANCM.

Encoder      Decoder $\widehat{f}_{\*I}$
$3\times3$ 32 conv, $28^2 \uparrow 32^2$, $3ch \rightarrow 32ch$     Concat $\{\*v, \*l, \*u_{\*I}\}$, 1*1*512 fully-connected

7 ResBlock, $32^2 \downarrow 16^2$, $32ch \rightarrow 64ch$     1 ResBlock, $1^2 \uparrow 4^2$, $512ch \rightarrow 256ch$

7 ResBlock, $16^2 \downarrow 8^2$, $64ch \rightarrow 128ch$     2 ResBlock, $4^2 \downarrow 8^2$, $256ch \rightarrow 128ch$

3 ResBlock, $8^2 \downarrow 4^2$, $128ch \rightarrow 256ch$     3 ResBlock, $8^2 \downarrow 16^2$, $128ch \rightarrow 64ch$

3 ResBlock, $4^2 \downarrow 1^2$, $256ch \rightarrow 512ch$     7 ResBlock, $16^2 \downarrow 32^2$, $64ch \rightarrow 32ch$

2 ResBlock, 1*1*512 fully-connected      8 ResBlock, $3\times 3$ 3 conv. $32^2 \uparrow 28^2$, $64ch \uparrow 3ch$

Architectures of the networks for MNIST and Bar experiments. $3 \times 3$ represents a $3 \times 3$ convolutional kernel is adopted. $a^2 \uparrow b^2$ denotes $a \times a$ resolution is upsampled to $b \times b$. $b^2 \downarrow a^2$ denotes $b \times b$ resolution is downsampled to $a \times a$. $i\ ch \rightarrow j\ ch$ denotes the channel changes.

CGN [Sauer & Geiger, 2021]. CGN proposes to encode an SCM over variables $Shape, Texture, Background$, and $Label$ into the proxy generative model. Given the label of the image, $Shape, Texture, Background$ are independent. Formally, the mechanism of this SCM is designed as follows:

\begin{equation}
\left \{
\begin{aligned}
&Label \leftarrow f_{l}(U_{l})\\
&Shape \leftarrow \widehat{f}_s(Label, U_d) \\
&Texture \leftarrow \widehat{f}_t(Label, U_s) \\
&Background \leftarrow \widehat{f}_b(Label, U_b) \\
&\*I \leftarrow \widehat{f}_{\*I}(Shape, Texture, Background),\\
\end{aligned}
\right .
\end{equation}

where  mechanism ${f}_s, {f}_t, {f}_b$ is designed to learn the conditional distribution $P(V \mid Label)$ with prior knowledge, where $V \in \{Shape, Texture, Background\}$. The composition mechanism $\widehat{f}_{\*I}$ is not learned but defined analytically. After fitting the given observational distribution $P(Label, \*I)$, the intervention can be performed by changing the $Label$.
In Colored MNIST and Bar experiments, the digit is regarded as $Shape$; the color is regarded as $background$ and the color is regarded as $Background$. We use the same VAE structure as ANCM to learn mechanism ${f}_s, {f}_t, {f}_b$ and the composition mechanism is designed as:

\begin{equation}
\*I = \mathrm{Concat}(Shape \odot Texture, Background)
\end{equation}

where $\odot$ represents an elementwise multiplication operation. Theoretically, CGN learns the independent mechanism from $Shape, Texture, Background$ to the image. After performing interventions on one variable, others should be preserved in the image.
This work can represent a branch of works that tries to change some specific features but remains others. However, they do not work on general causal relationships among generative factors (see more discussion in Appendix <ref>).

DEAR [Shen et al., 2022]. DEAR is designed to learn causal representations under the supervision of annotations and Markovian causal diagrams. It encodes the given graph to the latent space by a mask adjacency matrix. DEAR also fits the $P(\*I)$ and $P(\*V \mid \*I)$ separately similar to ANCM. $P(\*I)$ is fitted with BiGAN that is capable of learning representation and generating data simultaneously [Donahue et al., 2017, Dumoulin et al., 2017]. And $P(\*V \mid \*I)$ is fitted through a regularizer that predicts annotations from the representations. The intervention can be performed by ancestral sampling in the latent space.
In this work, we do not aim for learning the representations but only use their way to encode graphs into generative networks for comparison. When implementing DEAR, we simply ignore the bidirected edges in the given graph and encode the graph with directed edges. Theoretically, since DEAR relies on the Markovian assumption, it cannot fit the given observational distribution perfectly. This work can represent a branch of works with the Markovian assumption (see Appendix <ref>).
In theory, these methods that rely on the Markovian assumption can fail to provide Ctf-consistent estimator. See the following example:

Suppose one tries to use a Markovian model $\widehat{M}$ to fit observational distributions shown in Fig. <ref> and provide counterfactual image samples for $\cM^*$ in Example <ref>. The relationships between $F$ and $Y$ in $\widehat{\cM}$ should be (1) $F$ and $Y$ are independent; (2) $F \leftarrow Y$; (3) $F \rightarrow Y$;

First, case (1) fails to fit the given observational distribution since they are strongly correlated from the data. Notice that the optimal bound of $P(f_{y'} \mid f, y)$ is $[1, 1]$, which implies $F$ is guaranteed not to change after the intervention on $Y$.
$\widehat{\cM}$ with case (2) fails to provide such estimation.
To illustrate, the optimal bound of $P(y_{f'} \mid f, y)$ is $[1, 1]$, which implies $Y$ is guaranteed not to change after the intervention on $F$.
However, $Y$ has a direct effect on $F$ when $\widehat{\cM}$ has case (2) property, which means $Y$ is likely to change after the intervention.
Case (3) fails to provide such estimations for the same reason. $\widehat{\cM}$ cannot provide in-bound estimation of $P(y_{f'} \mid f, y)$.

We choose DEAR since it leverages self-attention [Vaswani et al., 2017] and SAGAN [Zhang et al., 2019] architectures for the discriminator and generator and then can generate high-quality images, which is important for the CelebA-HQ experiments. We choose the same network architecture and hyperparameters in the original paper for pendulum experiments.

Encoders and decoders for ANCM, CVE, and CGN are trained with a learning rate of $10^{-4}$, and they are optimized with Adam optimizer [Kingma & Ba, 2014]. All training processes are performed with a batch size of 100. We choose the temperature $\lambda$ as $100$ initially and gradually decrease it during training.

§.§ CelebA-HQ

Encoder      Decoder $\widehat{f}_{\*I}$
$3\times3$ 64 conv, $128^2 \uparrow 128^2$, $3ch \rightarrow 64ch$     Concat $\{\*v, \*l, \*u_{\*I}\}$, fully 1*1*1024 fully-connected

1 ResBlock, $128^2 \downarrow 64^2$, $64ch \rightarrow 64ch$     1 ResBlock, $1^2 \uparrow 4^2$, $1024ch \rightarrow 512ch$

3 ResBlock, $64^2 \downarrow 32^2$, $64ch \rightarrow 128ch$     2 ResBlock, $4^2 \downarrow 8^2$, $512ch \rightarrow 256ch$

3 ResBlock, $32^2 \downarrow 16^2$, $128ch \rightarrow 128ch$     2 ResBlock, $8^2 \downarrow 16^2$, $256ch \rightarrow 128ch$

7 ResBlock, $16^2 \downarrow 8^2$, $128ch \rightarrow 256ch$     7 ResBlock, $16^2 \downarrow 32^2$, $128ch \rightarrow 128ch$

3 ResBlock, $8^2 \downarrow 4^2$, $256ch \rightarrow 512ch$      2 ResBlock, $32^2 \downarrow 64^2$, $128ch \rightarrow 64ch$

3 ResBlock, $4^2 \downarrow 1^2$, $512ch \rightarrow 1024ch$     2 ResBlock, $64^2 \downarrow 128^2$, $64ch \rightarrow 64ch$

2 ResBlock, 1*1*1024 fully-connected      1 ResBlock, $3\times 3$ 3 conv. $128^2 \uparrow 128^2$, $64ch \rightarrow 3ch$

Architectures of the networks for CelebA-HQ experiments. $3 \times 3$ represents a $3 \times 3$ convolutional kernel is adopted. $a^2 \uparrow b^2$ denotes $a \times a$ resolution is upsampled to $b \times b$. $b^2 \downarrow a^2$ denotes $b \times b$ resolution is downsampled to $a \times a$. $i\ ch \rightarrow j\ ch$ denotes the channel changes.

The CelebA-HQ experiment is conducted on the CelebA-HQ [Karras et al., 2018] dataset describing human faces. The causal diagram of the ground truth is not given, however, causal diagrams shown in Fig <ref> and <ref> are used as inductive bias based on prior knowledge about human faces. To illustrate, in Smiling setting (Fig <ref>), Smiling ($S$) has a positive effect on Open_Mouth ($O$). But $S$ and $O$ can be confounded with $\*I$ by unknown generative factors $\*U_{\*I}$.
In Age setting (Fig <ref>), Young $Y$ are confounded by Female $F$ like Example <ref> and $Y$ has a negative effect on gray hair color. Similarly, these three generative factors can be confounded with the image variable.

(Z) at (-1.5,-1.5) [label=above:$S$,point];
(W) at (1.5,-1.5) [label=above:$O$,point];
(I) at (0,-3) [label=below:$I$,point];

(Z) edge (I);
(Z) edge (W);
(W) edge (I);
[bd] (Z) edge [bend right=40] (I);
[bd] (W) edge [bend left=40] (I);

(Y) at (0,-1.5) [label=above:$Y$,point];
(Z) at (-1.5,-1.5) [label=above:$F$,point];
(W) at (1.5,-1.5) [label=above:$H$,point];
(I) at (0,-3) [label=below:$I$,point];

(Y) edge (W);
(Y) edge (I);
(Z) edge (I);
(W) edge (I);
[bd] (Y) edge [bend right=40] (Z);
[bd] (Y) edge [bend left=40] (I);
[bd] (Z) edge [bend right=40] (I);
[bd] (W) edge [bend left=40] (I);

Causal diagrams used in CelebA-HQ experiments.

Internal image samples of the CelebaHQ Experiment in the Smiling setting.

In the Smiling setting, we are given the observed distribution $P(S, O, \*I)$ (image with labels of $S$ and $O$). We are asking the counterfactual image distribution $P(\*I, \*I_{O=1})$ (what would the image be had the person opened the Mouth?).
We set the care set as $\{S, O\}$. In other words, the counterfactual question we ask is "Given an image, would this person smile had the person opened the mouth?". Based on the graphical constraints induced by Fig <ref> and the observed distributions,
the optimal bound of $P(S=s, O=o, S_{O=o}=s)$ is $[P(S=s, O=o), P(S=s, O=o)]$;
the $P(S=s, O=o, S_{O=o}=s')$ are $[0, 0]$, where $o \neq o'$ and $s \neq s'$. In other words, the smiling feature should be preserved no matter how $O$ changes. In the Age setting, we are given the observed distribution $P(F, Y, H, \*I)$ (image with labels of $F$, $Y$ and $H$) and we are asking the counterfactual image distribution $P(\*I, \*I_{Y=0})$. We set the care set as $\{F, Y, H\}$. In other words, the counterfactual question we ask is "Given an image, would this person have gray hair and change gender had the person become old?". Based on the graphical constraints induced by Fig <ref> and the observed distributions, changing $Y$ should not change $F$. And $P(F=f, Y=1, H=h, F_{Y=0}=f, H_{Y=0}=h')$ is bounded by $[r, l]$, where $r = \max(0, P(F=f, Y=1, H=h) - P(F=f, Y=1)P(H = h \mid Y = 0))$ and $l = \min(P(F=f, Y=1, H=h), P(F=f, Y=1)P(H = h' \mid Y = 0))$. Thus $P(F_{Y=0}=0, H_{Y=0}=1 \mid F=0, Y=1, H=0)$ is bounded by $[0.179, 0.1808]$ from the observational data. To illustrate, given a young male image, the probability he has gray hair is at least 0.179 but at most 0.181.

Similarly to the last section, our causal method ANCM is compared with CVAE and DEAR in this section. We do not implement CGN since the generative variables for CGN are limited in $Shape, Texture, Background$, In this experiment, the composition function from face generative factors to images cannot be designed by hand.

We leverage DiffuseVAE [Sanchez & Tsaftaris, 2022], a two-stage method incorporating VAE and the Diffusion Probabilistic Models (DDPM) techniques, to get high-quality images. At the first stage, CVAEs are trained with association information and have the ability to edit target generative factors. ANCMs are trained with causal information (causal diagram) and have the ability to intervene on generative factors causally. However, both of them produce blurred images, $\hat{\*i}$, due to the naive VAE structure. In the second stage, these blurred image samples are refined by DDPM to high quality $\*i$. More details about the training procedure and architectures are provided in the next section. <ref> illustrates the internal two-stage image samples for DiffuseVAE in the Smiling setting. In the VAE stages, ANCMs and CVAE generate the initial images. After performing intervention $O=1$ on each image by corresponding models, the counterfactual images are produced. Notice that in these VAE stages, image samples exhibit blurred properties. At the DDPM stage, both initial images and counterfactual images are refined to the final results.

The experimental results are shown in Fig. <ref>.
Other additional results are provided in Fig. <ref> (Smiling setting) and Fig. <ref> (Young setting).

In the Smiling setting, all methods open the person's mouth. ANCM preserves $S$ after changing $O$.
DEAR also successfully changes $O$ without changing $S$ since the causal diagram among $S$ and $O$ is Markovian.
However, CVAE changes the smiling features at the same time. Other features, such as background are not guaranteed to be the same since they might be confounded with $S$ and $O$.

In the Age setting, all methods make the person's appearance older. ANCMs preserve $F$ after changing $Y$.
However, the other two baselines change the gender features at the same time. DEAR does not work in this setting since the causal diagram is non-Markovian.
Gray hair is possible to appear after changing the Age for all three methods.

§.§.§ Models and Hyperparameters

As we discussed above, the training process has two stages: (1) the VAE stage; (2) the DDPM stage. The training procedure at the VAE stage is the same as the VAE training introduced in the Colored MNIST and Bar experiments. We choose the different architecture for encoders and $\widehat{f}_{\*I}$ as shown in Tab. <ref>. We set the dimension of the latent vector $\*z$ as 1024 for CVAEs. For ANCMs, exogenous variables of $\*I$ are $1024 / \max{(c, 1)}$-dimensional standard normal distributions, where $c$ is the number of bidirected arrows to $\*I$. Other exogenous variables are 10-dimensional standard normal distributions. For DEAR, the network architecture and hyperparameters are chosen as the same in [Shen et al., 2022].

Two stages of DiffuseVAE.

In the second stage, the DDPM procedure is adopted to refine the decoder output at the VAE stage. DDPMs [Sohn et al., 2015, Ho et al., 2020] are deep generative models that consist of a forward process and reverse process with $T$ time-steps.
The forward process is modeled as a Markov chain that gradually perturbs $\*i^{t-1}$ (the image at step $t-1$) with gaussian noise to $\*i^{t}$ (the image at step $t$), where $\*i^0$ (image at step $0$) is the original image sample. Denoting the image variable at time step $t$ as $\*I^t$ ($\*I^0 = \*I$), the forward model can be expressed as

\begin{equation}
q(\*I^{1:T} \mid \*I_0) = \prod_{t=1}^T q(\*I^t \mid \*I^{t-1})
\end{equation}

The reverse process is also modeled as a Markov chain that gradually denoise $\*i^{t-1}$ to $\*i_{t}$ at each time step and finally recovers the original $\*i$. Formally,

\begin{equation}
p_{\varphi}(\*I^{0:T}) = p(\*I^T) \prod_{t=1}^T p_{\varphi}(\*I^{t-1} \mid \*I^t)
\end{equation}

where  $p_{\varphi}$ implies the density is modeled by a neural network parameterized by $\varphi$ and $p(\*I^T)$ is often chosen to be an isotropic Gaussian distribution. To maximize the log-likelihood of $P(\*I)$, the following objective is minimized suggested by [Ho et al., 2020]:

\begin{equation}
\begin{split}
&\mathbb{E}_{q}[D_{KL}(q(\*I^{T} \mid \*I^{0}) \Vert p(\*I^{T}) ) + \\
&\sum_{t>1}D_{KL}(q(\*I^{T-1} \mid \*I^{T}, \*I^{0}) \Vert p_{\varphi}(\*I^{T}))  \\
&-\log p_{\varphi}(\*I^{0} \mid \*I^{1})]
\end{split}
\end{equation}

When the image $\*I$ has an associated conditioning signal $\*C$, for example, a low-resolution image [Ho et al., 2022, Saharia et al., 2022] or a classification label, one can maximize the log-likelihood of the conditional distribution $P(\*I \mid \*C)$ with

\begin{equation}
\label{eq:ddpm-con-obj}
\begin{split}
&\mathbb{E}_{q}[D_{KL}(q(\*I^{T} \mid \*I^{0}) \Vert p(\*I^{T}) ) \\
&+ \sum_{t>1}D_{KL}(q(\*I^{T-1} \mid \*I^{T}, \*I^{0}, \*C) \Vert p_{\omega}(\*I^{T}))    \\
&-\log p_{\omega}(\*I^{0} \mid \*I^{1}, \*C)]
\end{split}
\end{equation}

In our experiments, we leverage the reconstructions of the VAEs as the conditional signal at the second stage as suggested by DiffuseVAE. For concreteness, we freeze the encoder and decoder of the VAE trained at the first stage and train $p_{\omega}$ individually with objective Eq. <ref>, where $\*C$ is the reconstructions of VAEs. The whole training procedure is illustrated in Fig. <ref>.

All encoders and decoders are trained with a learning rate of $10^{-4}$ and are optimized with Adam optimizer [Kingma & Ba, 2014]. All training processes are performed with a batch size of 64. We choose the temperature $\lambda$ as $100$ and gradually decrease it. At the DDPM stage, we choose the same hyperparameters and architecture as [Pandey et al., 2022].

§ CONNECTION TO RELATED WORKS

This paper systematically establishes the counterfactual image editing tasks within the framework of causal language.
In this section, we explicitly discuss some subtleties of extant research about image editing and counterfactual data generation.
By integrating the causal framework proposed in this paper, we (1) interpret the empirical findings and current evaluation metrics already present in the literature; (2) show other methods may not be as general as the method discussed in this paper due to their different assumptions and problem settings. Before that, we first present the following propositions for discussion.

Let the care set $\*W = \*X$. Any proxy model $\widehat{\cM}$ provides Ctf-consistent estimator regarding $\*W$ such that $P^{\widehat{\cM}}(\*V, \*I) = P^{{\cM}^*}(\*V, \*I)$

For any ASCM, the normalized feature counterfactual distribution
$P(\*W'_{\*x'} = \*w' \mid \*W = \*w ) = P(\*X'_{\*x'} = \*w' \mid \*X = \*w) = 1$ if the intervened value is $\*x'$ consistent with $\*w'$. Otherwise, this quantity is 0. This implies any ASCM provides Ctf-consistent estimators when $\*W = \*X$ from Def. <ref>.

There exist settings in which the lower bound of normalized feature counterfactual query $P^{\cM^*}(\*w'_{\*x'} \mid \*w)$ is strictly bigger than 0 and smaller than 1 and $v \neq v'$, where $V \in \*W \backslash \*X$.

See Example. <ref> and <ref>. To illustrate, the optimal bound of the feature counterfactual query $Q$ (Eq. <ref>) is 0.25 and the value of $H$ differs in observation $\*w$ and counterfactual $\*w'$.

Consider the true underlying ASCM $\cM^*$ with a semi-Markovian graph $\cG$.
There exist settings where any model $\widehat{\cM}$ in which the Markovianity assumption (also known as causal sufficiency) is enforced cannot satisfy (1) $P^{\widehat{\cM}}(\*V, \*I)$ =  $P^{\cM^*}(\*V, \*I)$ and (2) any feature counterfactual query is in the optimal bounds.

See the counterexample provided in Example <ref>.

§.§ Counterfactual Visual Explanation

Counterfactual visual explanation aims for the following question:
"Given an image $\*i$ for which a vision system predicts class $C=c$, how $\*i$ could change such that the system would output a different specified class $C=c'$?". Earlier methods construct adversarial sample $\*i'$ as the counterfactual image of $\*i$. Namely, the goal is to query a permutation $\delta$ such that $\*i + \delta$ is labeled as $C=c'$ by the given predictor.
[Wang & Vasconcelos, 2020] search the region contributes most to the prediction in pixel levels. Other approaches map the given image to the feature space and find minimal and sufficient input features to flip the prediction [Dhurandhar et al., 2018, Goyal et al., 2019, Van Looveren & Klaise, 2021]. However, the optimization process of search adversarial samples can push $\*i'$ far away from the data manifold of images. To enhance the realisticness of counterfactual images, VAEs [Goyal et al., 2019, Joshi et al., 2019, Rodríguez et al., 2021], GANs [Samangouei et al., 2018, Khorram & Fuxin, 2022] and Diffusions [Augustin et al., 2022] are incorporated to push counterfactual images close to the training distributions.
In the course of these studies, additional properties (as shown below) have been suggested to assess the generated counterfactual image, aside from realism [Verma et al., 2020, Moraffah et al., 2020]. It is noteworthy that these properties do not necessarily have to be satisfied simultaneously.

Validity. The counterfactual image should be labeled as the target $C=c'$ by the given predictor. Suppose one wants to edit an image $\*i$ describing a young person to an older look in Example <ref>. Validity states that the pixels of counterfactual image $\*i'$ should truly constitute old features.

Sparsity. The permutation $\delta$ should affect a minimal number of features. Following the previous example, sparsity encourages that counterfactual image $\*i'$ preserves features as much as possible.

Proximity. The counterfactual image $\*i'$ should stay as close to the initial pixel-level image. Proximity also motivates the counterfactual image not to change too much. Nevertheless, compared to sparsity, proximity focuses more on the change of each pixel. For example, the 180-degree rotation of an initial image satisfies sparsity (preserves almost all features) but does not satisfy proximity since each individual pixel varies.

Diversity. The counterfactual image $\*i'$ should be as diverse as possible. To illustrate, $\*i'$ should not be unique but are motivated to be different from each other. Continue the previous example. The diversity encourages that the old version images of the original young one scan have different old extent and various other features.

Our causal framework and estimation method provide explanations for these metrics.
First, The invertibility of $f_{\*I}$ in ASCMs supports the validity theoretically. To illustrate, the generative factor $Old=1$ causes image pixels containing old features through $f_{\*I}$, and images are predicted correctly as old class through the invertible function $h$.
Second, the sparsity and proximity are challenged. The achievement of sparsity and proximity implies that only the intervened generative factors $\*X$ change while other generative factors keep the same. This implies that $\*X$ does not have a causal effect on other generative factors, which only happens in the causal diagram that all other generative factors are non-descendants of $\*X$. Formally, based on Prop. <ref>, sparsity and proximity should not always be satisfied. Third, the non-identifiability results (Thm. <ref>) support the diversity. Thm. <ref> states that ASCMs that are compatible with the same observational distribution and the same causal diagram can induce different image counterfactual distributions. Diversity can be obtained by sampling different counterfactual images from  $P(\*I_{\*x'} \mid \*I = \*i)$.