# Diffusion-based Image Translation using Disentangled Style and Content
Representation

Gihyun Kwon1, Jong Chul Ye2,1
Department of Bio and Brain Engineering1, Kim Jaechul Graduate School of AI2,
KAIST
<EMAIL_ADDRESS>

###### Abstract

Diffusion-based image translation guided by semantic texts or a single target
image has enabled flexible style transfer which is not limited to the specific
domains. Unfortunately, due to the stochastic nature of diffusion models, it
is often difficult to maintain the original content of the image during the
reverse diffusion. To address this, here we present a novel diffusion-based
unsupervised image translation method using disentangled style and content
representation. Specifically, inspired by the slicing Vision Transformer
(Tumanyan et al., 2022), we extract intermediate keys of multihead self
attention layer from ViT model and used them as the content preservation loss.
Then, an image guided style transfer is performed by matching the [CLS]
classification token from the denoised samples and target image, whereas
additional CLIP loss is used for the text-driven style transfer. To further
accelerate the semantic change during the reverse diffusion, we also propose a
novel semantic divergence loss and resampling strategy. Our experimental
results show that the proposed method outperforms state-of-the-art baseline
models in both text-guided and image-guided translation tasks.

Figure 1: Image translation results. Our model can generate high-quality
translation outputs using both text and image conditions. More results can be
found in the experiment section.

##  1 Introduction

Image translation is a task in which the model receives an input image and
converts it into a target domain. Early image translation approaches (Zhu et
al., 2017; Park et al., 2020; Isola et al., 2017) were mainly designed for
single domain translation, but soon extended to multi-domain translation (Choi
et al., 2018; Lee et al., 2019). As these methods demand large training set
for each domain, image translation approaches using only a single image pairs
have been studied, which include the one-to-one image translation using
multiscale training (Lin et al., 2020), or patch matching strategy (Granot et
al., 2022; Kolkin et al., 2019). Most recently, Splicing ViT (Tumanyan et al.,
2022) exploits a pre-trained DINO ViT (Caron et al., 2021) to convert the
semantic appearance of a given image into a target domain while maintaining
the structure of input image.

On the other hand, by employing the recent text-to-image embedding model such
as CLIP (Radford et al., 2021), several approaches have attempted to generate
images conditioned on text prompts (Patashnik et al., 2021; Gal et al., 2021;
Crowson et al., 2022; Couairon et al., 2022). As these methods rely on
Generative Adversarial Networks (GAN) as a backbone generative model, the
semantic changes are not often properly controlled when applied to an out-of-
data (OOD) image generation.

Recently, score-based generative models (Ho et al., 2020; Song et al., 2020b;
Nichol & Dhariwal, 2021) have demonstrated state-of-the-art performance in
text-conditioned image generation (Ramesh et al., 2022; Saharia et al., 2022b;
Crowson, 2022; Avrahami et al., 2022). However, when it comes to the image
translation scenario in which multiple conditions (e.g. input image, text
condition) are given to the score based model, disentangling and separately
controlling the components still remains as an open problem.

In fact, one of the most important open questions in image translation by
diffusion models is to transform only the semantic information (or style)
while maintaining the structure information (or content) of the input image.
Although this could not be an issue with the conditional diffusion models
trained with matched input and target domain images (Saharia et al., 2022a),
such training is impractical in many image translation tasks (e.g. summer-to-
winter, horse-to-zebra translation). On the other hand, existing methods using
unconditional diffusion models often fail to preserve content information due
to the entanglement problems in which semantic and content change at the same
time (Avrahami et al., 2022; Crowson, 2022). DiffusionCLIP (Kim et al., 2022)
tried to address this problem using denoising diffusion implicit models (DDIM)
(Song et al., 2020a) and pixel-wise loss, but the score function needs to be
fine-tuned for a novel target domain, which is computationally expensive.

In order to control the diffusion process in such a way that it produces the
output that simultaneously retain the content of the input image and follow
the semantics of the target text or image, here we introduce a loss function
using a pre-trained Vision Transformer (ViT) (Dosovitskiy et al., 2020).
Specifically, inspired by the recent idea (Tumanyan et al., 2022), we extract
intermediate keys of multihead self attention layer and [CLS] classification
tokens of the last layer from the DINO ViT model and used them as our content
and style regularization, respectively. More specifically, to preserve the
structural information, we use the similarity and contrastive loss between
intermediate keys of the input and denoised image during the sampling. Then,
an image guided style transfer is performed by matching the [CLS] token
between the denoised sample and the target domain, whereas additional CLIP
loss is used for the text-driven style transfer. To further improve the
sampling speed, we propose a novel semantic divergence loss and resampling
strategy.

Extensive experimental results including Fig. 1 confirmed that our method
provide state-of-the-art performance in both text- and image- guided style
transfer tasks quantitatively and qualitatively. To our best knowledge, this
is the first unconditional diffusion model-based image translation method that
allows both text- and image- guided style transfer without altering input
image content.

##  2 Related Work

Text-guided image synthesis. Thanks to the outstanding performance of text-to-
image alignment in the feature space, CLIP has been widely used in various
text-related computer vision tasks including object generation (Liu et al.,
2021; Wang et al., 2022a), style transfer (Kwon & Ye, 2021; Fu et al., 2021),
object segmentation (Lüddecke & Ecker, 2022; Wang et al., 2022b), etc. Several
recent approaches also demonstrated state-of-the-art performance in text-
guided image manipulation task by combining the CLIP with image generation
models. Previous approaches leverage pre-trained StyleGAN (Karras et al.,
2020) for image manipulation with a text condition (Patashnik et al., 2021;
Gal et al., 2021; Wei et al., 2022). However, StyleGAN-based methods cannot be
used in arbitrary natural images since it is restricted to the pre-trained
data domain. Pre-trained VQGAN (Esser et al., 2021) was proposed for better
generalization capability in the image manipulation, but it often suffers from
poor image quality due to limited power of the backbone model.

With the advance of score-based generative models such as Denoising Diffusion
Probabilistic Model (DDPM) (Ho et al., 2020), several methods (Ramesh et al.,
2022; Saharia et al., 2022b) tried to generate photo-realistic image samples
with given text conditions. However, these approaches are not adequate for
image translation framework as the text condition and input image are not
usually disentangled. Although DiffusionCLIP (Kim et al., 2022) partially
solves the problem using DDIM sampling and pixelwise regularization during the
reverse diffusion, it has major disadvantage in that it requires fine-tuning
process of score models.

Single-shot Image Translation. In image translation using single target image,
early models mainly focused on image style transfer (Gatys et al., 2016; Huang
& Belongie, 2017; Park & Lee, 2019; Yoo et al., 2019). Afterwards, methods
using StyleGAN adaptation (Ojha et al., 2021; Zhu et al., 2021; Kwon & Ye,
2022; Chong & Forsyth, 2021) showed great performance, but there are
limitations as the models are domain-specific (e.g. human faces). In order to
overcome this, methods for converting unseen image into a semantic of target
(Lin et al., 2020; Kolkin et al., 2019; Granot et al., 2022) have been
proposed, but these methods often suffer from degraded image quality.
Recently, Splicing ViT (Tumanyan et al., 2022) successfully exploited pre-
trained DINO ViT(Caron et al., 2021) to convert the semantic appearance of
given image into target domain while preserving the structure of input.

##  3 Proposed Method

###  3.1 DDPM Sampling with Manifold Constraint

In DDPMs (Ho et al., 2020), starting from a clean image ${\bm{x}}_{0}\sim
q({\bm{x}}_{0})$, a forward diffusion process $q({\bm{x}}_{t}|{\bm{x}}_{t-1})$
is described as a Markov chain that gradually adds Gaussian noise at every
time steps $t$:

$\displaystyle
q({\bm{x}}_{T}|{\bm{x}}_{0}):=\prod_{t=1}^{T}q({\bm{x}}_{t}|{\bm{x}}_{t-1}),\quad\mbox{where}\quad
q({\bm{x}}_{t}|{\bm{x}}_{t-1}):={\mathcal{N}}({\bm{x}}_{t};\sqrt{1-\beta_{t}}{\bm{x}}_{t-1},\beta_{t}{\bm{I}}),$
(1)

where $\\{\beta\\}_{t=0}^{T}$ is a variance schedule. By denoting
$\alpha_{t}:=1-\beta_{t}$ and $\bar{\alpha_{t}}:=\prod_{s=1}^{t}\alpha_{s}$,
the forward diffused sample at $t$, i.e. $\bm{x}_{t}$, can be sampled in one
step as:

$\displaystyle{\bm{x}}_{t}=\sqrt{\bar{\alpha}_{t}}{\bm{x}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},\quad\mbox{where}\quad\bm{\epsilon}\sim{\mathcal{N}}(\textbf{0},{\bm{I}}).$
(2)

As the reverse of the forward step $q({\bm{x}}_{t-1}|{\bm{x}}_{t})$ is
intractable, DDPM learns to maximize the variational lowerbound through a
parameterized Gaussian transitions $p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t})$
with the parameter $\theta$. Accordingly, the reverse process is approximated
as Markov chain with learned mean and fixed variance, starting from
$p({\bm{x}}_{T})={\mathcal{N}}({\bm{x}}_{T};\textbf{0},\bm{I})$:

$\displaystyle
p_{\theta}({\bm{x}}_{0:T}):=p_{\theta}({\bm{x}}_{T})\prod_{t=1}^{T}p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t}),\quad\mbox{where}\quad
p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t}):={\mathcal{N}}({\bm{x}}_{t-1};{\bm{\mu}}_{\theta}({\bm{x}}_{t},t),\sigma_{t}^{2}{\bm{I}}).$
(3)

where

$\displaystyle{\bm{\mu}}_{\theta}({\bm{x}}_{t},t):=\frac{1}{\sqrt{\alpha}_{t}}\Big{(}{\bm{x}}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)\Big{)},$
(4)

Here, $\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)$ is the diffusion model trained
by optimizing the objective:

$\displaystyle\min_{\theta}L(\theta),\quad\mbox{where}\quad
L(\theta):=\mathbb{E}_{t,{\bm{x}}_{0},\bm{\epsilon}}\Big{[}\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\sqrt{\bar{\alpha}_{t}}{\bm{x}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},t)\|^{2}\Big{]}.$
(5)

After the optimization, by plugging learned score function into the generative
(or reverse) diffusion process, one can simply sample from
$p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t})$ by

$\displaystyle{\bm{x}}_{t-1}={\bm{\mu}}_{\theta}({\bm{x}}_{t},t)+\sigma_{t}\bm{\epsilon}=\frac{1}{\sqrt{\alpha_{t}}}\Big{(}{\bm{x}}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)\Big{)}+\sigma_{t}\bm{\epsilon}$
(6)

In image translation using conditional diffusion models (Saharia et al.,
2022a; Sasaki et al., 2021), the diffusion model $\bm{\epsilon}_{\theta}$ in
(5) and (6) should be replaced with
$\bm{\epsilon}_{\theta}({\bm{y}},\sqrt{\bar{\alpha}_{t}}{\bm{x}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},t)$
where ${\bm{y}}$ denotes the matched target image. Accordingly, the sample
generation is tightly controlled by the matched target in a supervised manner,
so that the image content change rarely happen. Unfortunately, the requirement
of the matched targets for the training makes this approach impractical.

To address this, Dhariwal & Nichol (2021) proposed classifier-guided image
translation using the unconditional diffusion model training as in (5) and a
pre-trained classifier $p_{\phi}({\bm{y}}|{\bm{x}}_{t})$. Specifically,
${\bm{\mu}}_{\theta}({\bm{x}}_{t},t)$ in (4) and (6) are supplemented with the
gradient of the classifier, i.e.
${\bm{\hat{}}\mu}_{\theta}({\bm{x}}_{t},t):={\bm{\mu}}_{\theta}({\bm{x}}_{t},t)+\sigma_{t}\nabla_{{\bm{x}}_{t}}\log
p_{\phi}({\bm{y}}|{\bm{x}}_{t})$. However, most of the classifiers, which
should be separately trained, are not usually sufficient to control the
content of the samples from the reverse diffusion process.

Inspired by the recent manifold constrained gradient (MCG) for inverse
problems (Chung et al., 2022a), here we formulate our content and style
guidance problem as an inverse problem, which can be solved by minimizing the
following total cost function with respect to the sample ${\bm{x}}$:

$\displaystyle\ell_{total}({\bm{x}};{\bm{x}}_{trg},{\bm{x}}_{src}),\quad\mbox{or}\quad\ell_{total}({\bm{x}};{\bm{d}}_{trg},{\bm{x}}_{src},{\bm{d}}_{src})$
(7)

where ${\bm{x}}_{src}$ and ${\bm{x}}_{trg}$ refer to the source and target
images, respectively; and ${\bm{d}}_{src}$ and ${\bm{d}}_{trg}$ refer to the
source and target text, respectively. In our paper, the first form of the
total loss in (7) is used for image-guided translation, where the second form
is for the text-guided translation. Then, the sampling from the reverse
diffusion with MCG is given by

$\displaystyle{\bm{x}}^{\prime}_{t-1}$
$\displaystyle=\frac{1}{\sqrt{\alpha_{t}}}\Big{(}{\bm{x}}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)\Big{)}+\sigma_{t}\bm{\epsilon}$
(8) $\displaystyle{\bm{x}}_{t-1}$
$\displaystyle={\bm{x}}^{\prime}_{t-1}-\nabla_{{\bm{x}}_{t}}\ell_{total}(\hat{\bm{x}}_{0}({\bm{x}}_{t}))$
(9)

where $\hat{\bm{x}}_{0}({\bm{x}}_{t})$ refers to the estimated clean image
from the sample ${\bm{x}}_{t}$ using the Tweedie’s formula (Kim & Ye, 2021):

$\displaystyle\hat{{\bm{x}}}_{0}({\bm{x}}_{t}):=\frac{{\bm{x}}_{t}}{\sqrt{\bar{\alpha}_{t}}}-\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}{\bm{\epsilon}}_{\theta}({\bm{x}}_{t},t).$
(10)

In the following, we describe how the total loss $\ell_{total}$ is defined.
For brevity, we notate $\hat{{\bm{x}}}_{0}({\bm{x}}_{t})$ as ${\bm{x}}$ in the
following sections.

Figure 2: Given the input image ${\bm{x}}_{src}$, we guide the reverse
diffusion process $\\{{\bm{x}}_{t}\\}_{t=T}^{0}$ using various losses. (a)
$\ell_{cont}$: the structural similarity loss between input and outputs in
terms of contrastive loss between extracted keys from ViT. (b) $\ell_{CLIP}$:
relative distance to the target text ${\bm{d}}_{trg}$ in CLIP space in terms
of ${\bm{x}}_{src}$ and ${\bm{d}}_{src}$. (c) $\ell_{sty}$: the [CLS] token
distances between the outputs and target ${\bm{x}}_{trg}$. (d) $\ell_{sem}$:
dissimilarity between the [CLS] token from the present and past denoised
samples.

###  3.2 Structure Loss

As previously mentioned, the main objective of image translation is
maintaining the content structure between output and the input image, while
guiding the output to follow semantic of target condition. Existing methods
(Couairon et al., 2022; Kim et al., 2022) use pixel-wise loss or the
perceptual loss for the content preservation. However, the pixel space does
not explicitly discriminate content and semantic components: too strong pixel
loss hinders the semantic change of output, whereas weak pixel loss alters the
structural component along with semantic changes. To address the problem, we
need to separately process the semantic and structure information of the
image.

Recently, Tumanyan et al. (2022) demonstrated successful disentanglement of
both components using a pre-trained DINO ViT (Caron et al., 2021). They showed
that in ViT, the keys $k^{l}$ of multi-head self attention (MSA) layer contain
structure information, and [CLS] token of last layer contains the semantic
information. With above features, they proposed a loss for maintaining
structure between input and network output with matching the self similarity
matrix $S^{l}$ of the keys, which can be represented in the following form for
our problem:

$\displaystyle\ell_{ssim}({\bm{x}}_{src},{\bm{x}})=\|S^{l}({\bm{x}}_{src})-S^{l}({\bm{x}})\|_{F},\quad\mbox{where}\quad\left[S^{l}({\bm{x}})\right]_{i,j}=\cos(k_{i}^{l}({\bm{x}}),k_{j}^{l}({\bm{x}})),$
(11)

where $k_{i}^{l}({\bm{x}})$ and $k_{j}^{l}({\bm{x}})$ indicate $i,j$th key in
the $l$-th MSA layer extracted from ViT with image ${\bm{x}}$. The self-
similarity loss can maintain the content information between input and output,
but we found that only using this loss results in a weak regularization in our
DDPM framework. Since the key $k_{i}$ contains the spatial information
corresponding the $i$-th patch location, we therefore use additional
regularization with contrastive learning as shown in Fig. 2(a). Specifically,
leveraging the idea of patch-wise contrastive loss (Park et al. (2020)), we
define the infoNCE loss using the DINO ViT keys:

$\displaystyle\ell_{cont}({\bm{x}}_{src},{\bm{x}})=-\sum_{i}\log\left(\frac{\exp(\mathrm{sim}({k}^{l}_{i}({\bm{x}}),{k}^{l}_{i}({\bm{x}}_{src}))/\tau)}{\exp(\mathrm{sim}({k}^{l}_{i}({\bm{x}}),{k}^{l}_{i}({\bm{x}}_{src}))/\tau+\sum_{j\neq
i}\exp(\mathrm{sim}({k}^{l}_{i}({\bm{x}}),{k}^{l}_{j}({\bm{x}}_{src}))/\tau)}\right),$
(12)

where $\tau$ is temperature, and $\mathrm{sim}(\cdot,\cdot)$ represents the
normalized cosine similarity. With this loss, we regularize the key of same
positions to have closer distance, while maximizing the distances between the
keys at different positions.

###  3.3 Style Loss

#### CLIP Loss for Text-guided Image Translation

Based on the previous work of Dhariwal & Nichol (2021), CLIP-guided diffusion
(Crowson (2022)) proposed to guide the reverse diffusion using pre-trained
CLIP model using the following loss function:

$\displaystyle\ell_{CLIP}({\bm{d}}_{trg},{\bm{x}}):=-\mathrm{sim}\left(E_{T}({\bm{d}}_{trg}),E_{I}({\bm{x}})\right),$
(13)

where ${\bm{d}}_{trg}$ is the target text prompt, and $E_{I},E_{T}$ refer to
the image and text encoder of CLIP, respectively. Although this loss can give
text-guidance to diffusion model, the results often suffer from poor image
quality.

Instead, we propose to use input-aware directional CLIP loss (Gal et al.
(2021)) which matches the CLIP embedding of the output image to the target
vector in terms of ${\bm{d}}_{trg}$, ${\bm{d}}_{src}$, and ${\bm{x}}_{src}$.
More specifically, our CLIP-based semantic loss is described as (see also Fig.
2(b)):

$\displaystyle\ell_{CLIP}({\bm{x}};{\bm{d}}_{trg},{\bm{x}}_{src},{\bm{d}}_{src}):=-\mathrm{sim}({\bm{v}}_{trg},{\bm{v}}_{src})$
(14)

where

$\displaystyle{\bm{v}}_{trg}:=E_{T}({\bm{d}}_{trg})+\lambda_{i}E_{I}({\bm{x}}_{src})-\lambda_{s}E_{T}({\bm{d}}_{src}),$
$\displaystyle\quad{\bm{v}}_{src}:=E_{I}(\mathrm{aug}({\bm{x}}))$ (15)

where $\mathrm{aug(\cdot)}$ denotes the augmentation for preventing
adversarial artifacts from CLIP. Here, we simultaneously remove the source
domain information $-\lambda_{s}E_{T}({\bm{d}}_{src})$ and reflect the source
image information to output $+\lambda_{i}E_{I}({\bm{x}}_{src})$ according to
the values of $\lambda_{s}$ and $\lambda_{i}$. Therefore it is possible to
obtain stable outputs compared to using the conventional loss.

Furthermore, in contrast to the existing methods using only single pre-trained
CLIP model (e.g. ViT/B-32), we improve the text-image embedding performance by
using the recently proposed CLIP model ensemble method (Couairon et al.
(2022)). Specifically, instead of using a single embedding, we concatenate the
multiple embedding vectors from multiple pre-trained CLIP models and used the
it as our final embedding.

#### Semantic Style Loss for Image-guided Image Translation

In the case of image-guide translation, we propose to use [CLS] token of ViT
as our style guidance. As explained in the previous part 3.2, the [CLS] token
contains the semantic style information of the image. Therefore, we can guide
the diffusion process to match the semantic of the samples to that of target
image by minimizing the [CLS] token distances as shown in Fig. 2(c). Also, we
found that using only [CLS] tokens often results in misaligned color values.
To prevent this, we guide the output to follow the overall color statistic of
target image with weak MSE loss between the images. Therefore, our loss
function is described as follows:

$\displaystyle\ell_{sty}({\bm{x}}_{trg},{\bm{x}})=||{\bm{e}}_{[CLS]}^{L}({\bm{x}}_{trg})-{\bm{e}}_{[CLS]}^{L}({{\bm{x}}})||_{2}+\lambda_{mse}||{\bm{x}}_{trg}-{{\bm{x}}}||_{2}.$
(16)

where ${\bm{e}}_{[CLS]}^{L}$ denotes the last layer [CLS] token.

###  3.4 Acceleration Strategy

#### Semantic Divergence Loss

With the proposed loss functions, we can achieve text- or image-guided image
translation outputs. However, we empirically observed that the generation
process requires large steps to reach the the desired output. To solve the
problem, we propose a simple approach to accelerate the diffusion process. As
explained before, the [CLS] token of ViT contains the overall semantic
information of the image. Since our purpose is to make the semantic
information as different from the original as possible while maintaining the
structure, we conjecture that we can achieve our desired purpose by maximizing
the distance between the [CLS] tokens of the previous step and the current
output during the generation process as described in Fig. 2(d). Therefore, our
loss function at time $t$ is given by

$\displaystyle\ell_{sem}({\bm{x}}_{t};{\bm{x}}_{t+1})=-||{\bm{e}}_{[CLS]}^{L}(\hat{{\bm{x}}}_{0}({\bm{x}}_{t}))-{\bm{e}}_{[CLS]}^{L}(\hat{{\bm{x}}}_{0}({\bm{x}}_{t+1}))||_{2},$
(17)

Specifically, we maximize the distance between the denoised output of the
present time and the previous time, so that next step sample has different
semantic from the previous step. One could think of alternatives to maximize
pixel-wise or perceptual distance, but we have experimentally found that in
these cases, the content structure is greatly harmed. In contrast, our
proposed loss has advantages in terms of image quality because it can control
only the semantic appearance.

#### Resampling Strategy

As shown in CCDF acceleration strategy (Chung et al., 2022b), a better
initialization leads to an accelerated reverse diffusion for inverse problem.
Empirically, in our image translation problem we also find that finding the
good starting point at time step $T$ for the reverse diffusion affects the
overall image quality. Specifically, in order to guide the initial estimate
${\bm{x}}_{T}$ to be sufficiently good, we perform $N$ repetition of one
reverse sampling ${\bm{x}}_{T-1}$ followed by one forward step
${\bm{x}}_{T}=\sqrt{1-\beta_{T-1}}{\bm{x}}_{T-1}+\beta_{T-1}{\bm{\epsilon}}$
to find the ${\bm{x}}_{T}$ whose gradient for the next step is easily affected
by the loss. With this initial resampling strategy, we can empirically found
the initial ${\bm{x}}_{T}$ that can reduce the number of reverse steps. The
overall process is in our algorithm in Appendix.

###  3.5 Total Loss

Putting all together, the final loss in (7) for the text-guided reverse
diffusion is given by

$\displaystyle\ell_{total}=\lambda_{1}\ell_{cont}+\lambda_{2}\ell_{ssim}+\lambda_{3}\ell_{CLIP}+\lambda_{4}\ell_{sem}+\lambda_{5}\ell_{rng},$
(18)

where $\ell_{rng}$ is a regularization loss to prevent the irregular step of
reverse diffusion process suggested in (Crowson (2022)). If the target style
image ${\bm{x}}_{trg}$ is given instead of text conditions ${\bm{d}}_{src}$
and ${\bm{d}}_{trg}$, then $\ell_{CLIP}$ is simply substituted for
$\ell_{sty}$.

Figure 3: Qualitative comparison of text-guided translation on Animals
dataset. Our model generates realistic samples that reflects the text
condition, with better perceptual quality than the baselines.

##  4 Experiment

###  4.1 Experimental Details

For implementation, we refer to the official source code of blended diffusion
(Avrahami et al. (2022)). All experiments were performed using unconditional
score model pre-trained with Imagenet 256$\times$256 resolution datasets
(Dhariwal & Nichol (2021)). In all the experiments, we used diffusion step of
$T=60$ and the resampling repetition of $N=10$; therefore, the total of 70
diffusion reverse steps are used. The generation process takes 40 seconds per
image in single RTX 3090 unit. In $\ell_{CLIP}$, we used the ensemble of 5
pre-trained CLIP models (RN50, RN50x4, ViT-B/32, RN50x16, ViT-B/16) for the
text-guidance, following the setup of Couairon et al. (2022). Our detailed
experimental settings are elaborated in Appendix.

###  4.2 Text-guided Semantic Image Translation

To evaluate the performance of our text-guided image translation, we conducted
comparisons with state-of-the-art baseline models. For baseline methods, we
selected the recently proposed models which use pre-trained CLIP for text-
guided image manipulation: VQGAN-CLIP (Crowson et al. (2022)), CLIP-guided
diffusion (CGD) (Crowson (2022)), DiffusionCLIP (Kim et al. (2022)), and
FlexIT (Couairon et al. (2022)). For all baseline methods, we referenced the
official source codes.

Since our framework can be applied to arbitrary text semantics, we tried
quantitative and qualitative evaluation on various kinds of natural image
datasets. We tested our translation performance using two different datasets:
animal faces (Si & Zhu (2012)) and landscapes (Chen et al. (2018)). The animal
face dataset contains 14 classes of animal face images, and the landscapes
dataset consists of 7 classes of various natural landscape images.

Figure 4: Qualitative comparison of text-guided image translation on Landscape
dataset. Our model generates outputs with better perceptual quality than the
baselines.

Method | Animals | Landscapes | User Study
---|---|---|---
SFID$\downarrow$ | CSFID$\downarrow$ | LPIPS$\downarrow$ | SFID$\downarrow$ | CSFID$\downarrow$ | LPIPS$\downarrow$ | Text $\uparrow$ | Realism $\uparrow$ | Content $\uparrow$
VQGAN-CLIP | 30.01 | 65.51 | 0.462 | 33.31 | 82.92 | 0.571 | 2.78 | 2.05 | 2.16
CLIP-GD | 12.50 | 53.05 | 0.468 | 18.13 | 62.19 | 0.458 | 2.61 | 2.24 | 2.28
DiffusionCLIP | 25.09 | 66.50 | 0.379 | 29.85 | 76.29 | 0.568 | 2.50 | 2.54 | 3.06
FlexIT | 32.71 | 57.87 | 0.215 | 18.04 | 60.04 | 0.243 | 2.22 | 3.15 | 3.89
Ours | 9.98 | 41.07 | 0.372 | 16.86 | 54.48 | 0.417 | 3.68 | 4.28 | 4.11

Table 1:  Quantitative comparison in the text-guided image translation. Our
model outperforms baselines in overall scores for both of Animals and
Landscapes datasets as well as user study.

To measure the performance of the generated images, we measured the FID score
(Heusel et al. (2017)). However, when using the basic FID score measurement,
the output value is not stable because our number of generated images is not
large. To compensate for this, we measure the performance using a simplified
FID (Kim et al. (2020)) that does not consider the diagonal term of the
feature distributions. Also, we additionally showed a class-wise SFID score
that measures the SFID for each class of the converted output because it is
necessary to measure whether the converted output accurately reflects the
semantic information of the target class. Finally, we used the averaged LPIPS
score between input and output to verify the content preservation performance
of our method. Further experimental settings can be found in our Appendix.

In Table 1, we show the quantitative comparison results. In image quality
measurement using SFID and CSFID, our model showed the best performance among
all baseline methods. Especially for Animals dataset, our SFID value
outperformed others in large gain. In the content preservation by LPIPS score,
our method scored the second best. In case of FlexIT, it showed the best score
in LPIPS since the model is directly trained with LPIPS loss. However, too low
value of LPIPS is undesired as it means that the model failed in proper
semantic change. This can be also seen in qualitative result of Figs. 3 and 4,
where our results have proper semantic features of target texts with content
preservation, whereas the results from FlexIT failed in semantic change as it
is too strongly confined to the source images. In other baseline methods, most
of the methods failed in proper content preservation.

To further evaluate the perceptual quality of generated samples, we conducted
user study. In order to measure the detailed opinions, we used custom-made
opinion scoring system. We asked the users in three different parts: 1) Are
the output have correct semantic of target text? (Text-match), 2) are the
generated images realistic? (Realism), 3) do the outputs contains the content
information of source images? (Content). Detailed user-study settings are in
our Appendix. In Table 1, our model showed the best performance, which further
shows the superiority of our method.

###  4.3 Image-guided Semantic Image Translation

Method | Style $\uparrow$ | Realism $\uparrow$ | Content $\uparrow$
---|---|---|---
SANet | 2.75 | 4.08 | 4.37
WCT2 | 2.59 | 4.64 | 4.90
STROTSS | 3.92 | 2.91 | 3.17
SplicingViT | 3.50 | 2.08 | 2.15
Ours | 4.23 | 4.25 | 4.51

Table 2: User study comparison of image-guided translation tasks. Our model
outperforms baseline methods in overall perceptual quality.

Since our methods can be easily adapted to the image translation guided by
target images, we further evaluate the performance of out model with
comparison experiments. We compare our model with appearance transfer models
of Splicing ViT (Tumanyan et al. (2022)), STROTSS (Kolkin et al. (2019)), and
style transfer methods WCT2 (Yoo et al. (2019)) and SANet (Park & Lee (2019)).

Fig. 5 is a qualitative comparison result of image guided translation task.
Our model successfully generated outputs that follow the semantic styles of
the target images while maintaining the content of the source images. In the
case of other models, we can see that the content was severely deformed or the
semantic style was not properly reflected. We also measured the overall
perceptual quality through a user study. As with text-guided translation, we
investigated user opinion through three different questions. In Table 2, our
model obtained the best score in style matching score and the second best in
realism and content preservation scores. Baseline WCT2 showed the best in
realism and content scores, but it shows the worst score in style matching
because the outputs are hardly changed from the inputs except for overall
colors. The opinions scores confirm that our model outperforms the baselines.
More details are in our Appendix.

Figure 5: Qualitative comparison of image-guided image translation. Our
results have better perceptual quality than the baseline outputs.

###  4.4 Ablation Study

Figure 6: Qualitative comparison on ablation study. Our full setting shows the
best results.

To verify the proposed components in our framework, we compare the generation
performance with different settings. In Fig. 6, we show that (a) the outputs
from our best setting have the correct semantic of target text, with
preserving the content of the source; (b) by removing $\ell_{sem}$, the
results still have the appearance of source images, suggesting that images are
not fully converted to the target domain; (c) without $\ell_{cont}$, the
output images totally failed to capture the content of source images; (d) by
using LPIPS perceptual loss instead of proposed $\ell_{cont}$, the results can
only capture the approximate content of source images; (e) using pixel-wise
$l_{2}$ maximization loss instead of proposed $\ell_{sem}$, the outputs suffer
from irregular artifacts; (f) without using our proposed resampling trick, the
results cannot fully reflect the semantic information of target texts.
Overall, we can obtain the best generation outputs by using all of our
proposed components. For further evaluation, we will show the quantitative
results of ablation study in our Appendix.

##  5 Conclusion

In conclusion, we proposed a novel loss function which utilizes a pre-trained
ViT model to guide the generation process of DDPM models in terms of content
preservation and semantic changes. We further propose a novel strategy of
resampling technique for better initialization of diffusion process. For
evaluation, our extensive experimental results show that our proposed
framework has superior performance compared to baselines in both of text- and
image-guided semantic image translation tasks.

## References

  * Anoosheh et al. (2018) Asha Anoosheh, Eirikur Agustsson, Radu Timofte, and Luc Van Gool.  Combogan: Unrestrained scalability for image domain translation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 783–790, 2018.
  * Avrahami et al. (2022) Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18208–18218, 2022.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9650–9660, 2021.
  * Chen et al. (2018) Yang Chen, Yu-Kun Lai, and Yong-Jin Liu.  Cartoongan: Generative adversarial networks for photo cartoonization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 9465–9474, 2018.
  * Choi et al. (2018) Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8789–8797, 2018.
  * Chong & Forsyth (2021) Min Jin Chong and David Forsyth.  Jojogan: One shot face stylization.  _arXiv preprint arXiv:2112.11641_ , 2021.
  * Chung et al. (2022a) Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye.  Improving diffusion models for inverse problems using manifold constraints.  _arXiv preprint arXiv:2206.00941_ , 2022a.
  * Chung et al. (2022b) Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye.  Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12413–12422, 2022b.
  * Couairon et al. (2022) Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Flexit: Towards flexible semantic image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18270–18279, 2022.
  * Crowson (2022) Katherine Crowson.  Clip-guided diffusion.  2022\.  URL https://github.com/afiaka87/clip-guided-diffusion.
  * Crowson et al. (2022) Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  _arXiv preprint arXiv:2204.08583_ , 2022.
  * Deng et al. (2019) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 4690–4699, 2019.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 12873–12883, 2021.
  * Fu et al. (2021) Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang.  Language-driven image style transfer.  2021\.
  * Gal et al. (2021) Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  _arXiv preprint arXiv:2108.00946_ , 2021.
  * Gatys et al. (2016) Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2414–2423, 2016.
  * Granot et al. (2022) Niv Granot, Ben Feinstein, Assaf Shocher, Shai Bagon, and Michal Irani.  Drop the gan: In defense of patches nearest neighbors as single image generative models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 13460–13469, 2022.
  * Hahne & Aggoun (2021) Christopher Hahne and Amar Aggoun.  Plenopticam v1.0: A light-field imaging framework.  _IEEE Transactions on Image Processing_ , 30:6757–6771, 2021.  doi: 10.1109/TIP.2021.3095671.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  pp. 6626–6637, 2017.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Huang & Belongie (2017) Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  pp. 1501–1510, 2017.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1125–1134, 2017.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8110–8119, 2020.
  * Kim et al. (2020) Chung-Il Kim, Meejoung Kim, Seungwon Jung, and Eenjun Hwang.  Simplified fréchet distance for generative adversarial nets.  _Sensors_ , 20(6), 2020.  ISSN 1424-8220.  doi: 10.3390/s20061548.  URL https://www.mdpi.com/1424-8220/20/6/1548.
  * Kim et al. (2022) Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2426–2435, 2022.
  * Kim & Ye (2021) Kwanyoung Kim and Jong Chul Ye.  Noise2score: tweedie’s approach to self-supervised image denoising without clean images.  _Advances in Neural Information Processing Systems_ , 34:864–874, 2021.
  * Kolkin et al. (2019) Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich.  Style transfer by relaxed optimal transport and self-similarity.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10051–10060, 2019.
  * Kwon & Ye (2021) Gihyun Kwon and Jong Chul Ye.  Clipstyler: Image style transfer with a single text condition.  _arXiv preprint arXiv:2112.00374_ , 2021.
  * Kwon & Ye (2022) Gihyun Kwon and Jong Chul Ye.  One-shot adaptation of gan in just one clip.  _arXiv preprint arXiv:2203.09301_ , 2022.
  * Lee et al. (2019) Dongwook Lee, Junyoung Kim, Won-Jin Moon, and Jong Chul Ye.  Collagan: Collaborative gan for missing image data imputation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2487–2496, 2019.
  * Lin et al. (2020) Jianxin Lin, Yingxue Pang, Yingce Xia, Zhibo Chen, and Jiebo Luo.  Tuigan: Learning versatile image-to-image translation with two unpaired images.  In _European Conference on Computer Vision_ , pp. 18–35. Springer, 2020.
  * Liu et al. (2021) Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu.  Fusedream: Training-free text-to-image generation with improved clip+ gan space optimization.  _arXiv preprint arXiv:2112.01573_ , 2021.
  * Lüddecke & Ecker (2022) Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7086–7096, 2022.
  * Nichol & Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pp. 8162–8171. PMLR, 2021.
  * Ojha et al. (2021) Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10743–10752, 2021.
  * Park & Lee (2019) Dae Young Park and Kwang Hee Lee.  Arbitrary style transfer with style-attentional networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5880–5888, 2019.
  * Park et al. (2020) Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu.  Contrastive learning for unpaired image-to-image translation.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), _Computer Vision – ECCV 2020_ , pp. 319–345, Cham, 2020. Springer International Publishing.  ISBN 978-3-030-58545-7.
  * Patashnik et al. (2021) Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 2085–2094, 2021.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_ , 2021.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Saharia et al. (2022a) Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pp. 1–10, 2022a.
  * Saharia et al. (2022b) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_ , 2022b.
  * Sasaki et al. (2021) Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon.  Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models.  _arXiv preprint arXiv:2104.05358_ , 2021.
  * Si & Zhu (2012) Zhangzhang Si and Song-Chun Zhu.  Learning hybrid image templates (hit) by information projection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 34(7):1354–1367, 2012.  doi: 10.1109/TPAMI.2011.227.
  * Song et al. (2020a) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2020a.
  * Song et al. (2020b) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Tumanyan et al. (2022) Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Splicing vit features for semantic appearance transfer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10748–10757, 2022.
  * Wang et al. (2022a) Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  Clip-nerf: Text-and-image driven manipulation of neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3835–3844, 2022a.
  * Wang et al. (2022b) Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu.  Cris: Clip-driven referring image segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11686–11695, 2022b.
  * Wei et al. (2022) Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Lu Yuan, Weiming Zhang, and Nenghai Yu.  Hairclip: Design your hair by text and reference image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18072–18081, 2022.
  * Yoo et al. (2019) Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Ha.  Photorealistic style transfer via wavelet transforms.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9036–9045, 2019.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * Zhu et al. (2021) Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka.  Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks.  _arXiv preprint arXiv:2110.08398_ , 2021.

##  Appendix A Experimental Details

###  A.1 Implementation Details

For implementation, in case of using text-guided image manipulation, our
initial sampling numbers are set as $T=100$, but we skipped the initial 40
steps to maintain the abstract content of input image. Therefore, the total
number of sampling steps is $T=60$. With resampling step of $N=10$, we use
total of 70 iterations for single image output. We found that using more
resampling steps does not show meaningful performance improvement. In image-
guided manipulation, we set initial sampling number $T=200$, and skipped the
initial 80 steps. We used resampling step $N=10$. Therefore, we use total of
130 iterations. Although we used more iterations than text-guided translation,
it takes about 40 seconds.

For hyperparameters, we use $\lambda_{1}=200$, $\lambda_{2}=100$,
$\lambda_{3}=2000$, $\lambda_{4}=1000$, $\lambda_{5}=200$. For image-guided
translation, we set $\lambda_{mse}=1.5$. For our CLIP loss, we set
$\lambda_{s}=0.4$ and $\lambda_{i}=0.2$. For our ViT backbone model, we used
pre-trained DINO ViT that follows the baseline of Splicing ViT (Tumanyan et
al., 2022). For extracting keys of intermediate layer, we use layer of $l=11$,
and for [CLS] token, we used last layer output. Since ViT and CLIP model only
take 224$\times$224 resolution images, we resized all images before
calculating the losses with ViT and CLIP.

To further improve the sample quality of our qualitative results, we used
restarting trick in which we check the $\ell_{reg}$ loss calculated at initial
time step $T$, and restart the whole process if the loss value is too high. If
the initial loss $\ell_{reg}>0.01$, we restarted the process. For quantitative
result, we did not use the restart trick for fair comparison.

For augmentation, we use the same geometrical augmentations proposed in
FlexIT(Couairon et al., 2022). Also, following the setting from CLIP-guided
diffusion(Crowson, 2022), we included noise augmentation in which we mix the
noisy image to $\hat{x}_{0}(x_{t})$ as it further removes the artifacts.

In our image-guided image translation on natural landscape images, we matched
the color distribution of output image to that of target image with (Hahne &
Aggoun, 2021), as it showed better perceptual quality. Our detailed
implementation can be found in our official GitHub
repository.111https://github.com/anon294384/DiffuseIT

For baseline experiments, we followed the official source codes in all of the
models222https://github.com/afiaka87/clip-guided-
diffusion333https://github.com/nerdyrodent/VQGAN-
CLIP444https://github.com/gwang-
kim/DiffusionCLIP555https://github.com/facebookresearch/SemanticImageTranslation.
For diffusion-based models (DiffusionCLIP, CLIP-guided diffusion), we used
unconditional score model pre-trained on 256$\times$256 resolutions. In
DiffusionCLIP, we fine-tuned the score model longer than suggested training
iteration, as it showed better quality. In CLIP-guided diffusion, we set the
CLIP-guided loss as 2000, and also set initial sampling number as $T=100$ with
skipping initial 40 steps. For VQGAN-based models (FlexIT, VQGAN-CLIP), we
used VQGAN trained on imagenet 256$\times$256 resolutions datasets. In VQGAN-
CLIP, as using longer iteration results in extremely degraded images,
therefore we optimized only 30 iterations, which is smaller than suggested
iterations ( $\geq$80). In the experiments of FlexIT, we followed the exactly
same settings suggested in the original paper.

For baselines of image-guided style transfer tasks, we also referenced the
original source
codes666https://github.com/omerbt/Splice777https://github.com/nkolkin13/STROTSS888https://github.com/clovaai/WCT2999https://github.com/GlebSBrykin/SANET.
In all of the experiments, we followed the suggested settings from the
original papers.

###  A.2 Dataset Details

For our quantitative results using text-guided image translation, we used two
different datasets Animals and Landscapes. In Animals dataset, the original
dataset contains 21 different classes, but we filtered out the images from 14
classes (bear, cat, cow, deer, dog, lion, monkey, mouse, panda, pig, rabbit,
sheep, tiger, wolf) which can be classified as mammals. Remaining classes
(e.g. human, chicken, etc.) are removed since they have far different
semantics from the mammal faces.Therefore we reported quantitative scores only
with filtered datasets for fair comparison. The dataset contains 100-300
images per each class, and we selected 4 testing images from each class in
order to use them as our content source images. With selected samples, we
calculated the metrics using the outputs of translating the 4 images from a
source class into all the remaining classes. Therefore, in our animal face
dataset, total of 676 generated images are used for evaluation.

In Landscapes dataset, we manually classified the images into 7 different
classes (beach, desert, forest, grass field, mountain, sea, snow). Each class
has 300 different images except for desert class which have 100 different
images. Since some classes have not enough number of images, we borrowed
images from seasons (Anoosheh et al., 2018) dataset. For metric calculation,
we selected 8 testing images from each class, and used them as our content
source images. Again, we translated the 8 images from source class into all
the remaining classes. Therefore, a total of 336 generated images are used for
our quantitative evaluation.

For single image guided translation, we selected random images from AFHQ
dataset for animal face translation; and for natural image generation, we
selected random images from our Landscapes datasets.

###  A.3 User Study Details

For our user study in text-guided image translation task, we generated 130
different images using 13 different text conditions with our proposed and
baseline models. Then we randomly selected 65 images and made 6 different
questions. More specifically, we asked the participants question about three
different parts: 1) Are the outputs have correct semantic of target text?
(Text-match), 2) Are the generated images realistic? (Realism), 3) Do the
outputs contain the content information of source images (Content). We
randomly recruited a total of 30 users, and provided them the questions using
Google Form. The 30 different users come from age group 20s and 50s. We set
the minimum score as 1, and the maximum score is 5. The users can score among
5 different options : 1-Very Unlikely, 2-Unlikely, 3-Normal, 4-Likely, 5-Very
Likely.

For the user study on image-guided translation task, we generated 40 different
images using 8 different images conditions. Then we followed the same protocol
to user study on text-guided image translation tasks, except for the content
of questions. We asked the users in three different parts: 1) Are the outputs
have correct semantic of target style image? (Style-match), 2) Are the
generated images realistic? (Realism), 3) Do the outputs contain the content
information of source images (Content).

###  A.4 Algorithm

For detailed explanation, we include Algorithm of our proposed image
translation mathods in Algorithm 1.

Algorithm 1 Semantic image translation: given a diffusion score model
$\epsilon_{\theta}({\bm{x}}_{t},t)$, CLIP model, and VIT model

Input: source image ${\bm{x}}_{src}$, diffusion steps $T$, resampling steps
$N$, target text ${\bm{d}}_{trg}$, source text ${\bm{d}}_{src}$ or target
image ${\bm{x}}_{trg}$
Output: translated image $\hat{{\bm{x}}}$ which has semantic of
${\bm{d}}_{trg}$ (or ${\bm{x}}_{trg}$) and content of ${\bm{x}}_{src}$
${\bm{x}}_{T}\sim\mathcal{N}(\sqrt{\bar{\alpha}_{t}}{\bm{x}}_{src},(1-\bar{\alpha}_{t})\mathbf{I})$,
index for resampling $n=0$

1:for all $t$ from $T$ to 0 do

2: $\epsilon\leftarrow\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)$

3:
$\hat{{\bm{x}}}_{0}({\bm{x}}_{t})\leftarrow\frac{{\bm{x}}_{t}}{\sqrt{\bar{\alpha}_{t}}}-\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\epsilon$

4: if text-guided then

5:
$\nabla_{total}\leftarrow\nabla_{{\bm{x}}_{t}}\ell_{total}(\hat{{\bm{x}}}_{0}({\bm{x}}_{t});{\bm{d}}_{trg},{\bm{x}}_{src},{\bm{d}}_{src})$

6: else if image-guided then

7:
$\nabla_{total}\leftarrow\nabla_{{\bm{x}}_{t}}\ell_{total}(\hat{{\bm{x}}}_{0}({\bm{x}}_{t});{\bm{x}}_{trg},{\bm{x}}_{src})$

8: end if

9: ${\bm{z}}\sim\mathcal{N}(0,\mathbf{I})$

10:
${\bm{x}}^{\prime}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\Big{(}{\bm{x}}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}\Big{)}+\sigma_{t}{\bm{z}}$

11: ${\bm{x}}_{t-1}={\bm{x}}^{\prime}_{t-1}-\nabla_{total}$

12: if $t$ = $T$ and $n<N$ then

13:
${\bm{x}}_{t}\leftarrow\mathcal{N}(\sqrt{1-\beta_{t-1}}{\bm{x}}_{t-1},\beta_{t-1}\mathbf{I})$

14: $n\leftarrow n+1$

15: go to 2

16: end if

17:end for

18:return ${\bm{x}}_{-1}$

##  Appendix B Quantitative Ablation Study

For more thorough evaluation of our proposed components, we report ablation
study on quantitative metrics. In this experiment, we only used Animals
dataset due to the time limit. In Table 3, we show the quantitative results on
various settings. When we remove one of our acceleration strategies, in
setting (b) and (f), we can see that the fid score is degraded as the outputs
are not properly changed from the original source images. (e) When we use L2
maximization instead of our proposed $\ell_{sem}$, FID scores are improved
from setting (b), but still the performance is not on par with our best
settings. (d) When we use weak content regularization using LPIPS, we can see
that the overall scores are degraded. When we remove our proposed
$\ell_{cont}$, we can observe that SFID and CSFID scores are lower than other
settings. However, we can see that LPIPS score is severely high as the model
hardly reflect the content information of original source images. Overall,
when using our best setting, we can obtain the best output considering all of
the scores.

Settings | Animals
---|---
SFID$\downarrow$ | CSFID$\downarrow$ | LPIPS$\downarrow$
No resampling (f) | 11.88 | 59.09 | 0.316
L2 Max instead of $\ell_{sem}$ (e) | 13.18 | 49.47 | 0.324
LPIPs instead of $\ell_{cont}$ (d) | 11.15 | 58.67 | 0.400
No $\ell_{cont}$ (c) | 9.90 | 33.07 | 0.477
No $\ell_{sem}$ (b) | 15.00 | 53.43 | 0.347
Ours (a) | 9.98 | 41.07 | 0.372
Table 3: Quantitative comparison of ablation studies.

##  Appendix C Artistic Style Transfer

With our framework, we can easily adapt our method to artistic style transfer.
With simply changing the text conditions, or using artistic paintings as our
image conditions, we can obtain the artistic style transfer results as shown
in Fig. 7.

Figure 7: Various outputs of artistic style transfer. We can translation
natural images into artistic style paintigns with both of text or image
conditions.

##  Appendix D Face image translation

Figure 8: Outputs from face image translation models. The outputs from our
model successfully translated the human face images with proper target domain
semantic information.

Instead of using score mode pre-trained on Imagenet dataset, we can use pre-
trained score model on FFHQ human face dataset. In order to keep the face
identity between source and output images, we include $\lambda_{id}\ell_{id}$
which leverage pre-trained face identification model ArcFace(Deng et al.,
2019). We calculate identity loss between ${\bm{x}}_{src}$ and denoised image
$\hat{{\bm{x}}}_{0}({\bm{x}}_{t})$. We use $\lambda_{id}=100$.

In Fig. 8, we show that our method also can be used in face image translation
tasks. For comparison, we included baseline models of face editing method
StyleCLIP (Patashnik et al., 2021), and one-shot face stylization model of
JojoGAN (Chong & Forsyth, 2021). The results show that our method can
translate the source faces into target domain with proper semantic change. In
baseline models, although some images show high quality outputs, in most cases
the image failed in translating the images. Also, since the baseline models
rely on pre-trained StyleGAN, they require additional GAN inversion process to
translate the source image. Therefore, the content information is not
perfectly matched to the source image due to the limitation of GAN inversion
methods.

##  Appendix E Limitation and Future work

Although our method has shown successful performance in image conversion, it
still has limitations to solve. First, if the semantic distance between the
source image and the target domain is too far (e.g building $\rightarrow$
Tiger), the output is not translated properly. We conjecture that this occurs
when the text-image embedding space in the CLIP model is not accurately
aligned, therefore it can be solved by using the advanced text-to-image
embedding model. Second, our method has limitation that the image generation
quality heavily relies on the performance of the pre-trained score model. This
can also be solved if we use a diffusion model backbone with better
performance. In future work, we plan to improve our proposed method in these
two directions.

##  Appendix F Additional Results

For additional results , in Fig. 9 we show the image translation outputs using
text conditions. In Fig. 10, we additionally show the results from our image-
guided image translation. We can successfully change the semantic of various
natural images with text and image conditions.

Figure 9: Qualitative results of text-guided image translation.

Figure 10: Qualitative results of image-guided image translation.


* Rao, (1970) Rao, C. R. (1970).  Estimation of heteroscedastic variances in linear models.  Journal of the American Statistical Association, 65:161–172.
  * Reichenbach, (1957) Reichenbach, H. (1957).  The Direction of Time.  University of California Press.
  * Rigdon and Hudgens, (2015) Rigdon, J. and Hudgens, M. G. (2015).  Randomization inference for treatment effects on a binary outcome.  Statistics in Medicine, 34:924–935.
  * Robins et al., (2007) Robins, J., Sued, M., Lei-Gomez, Q., and Rotnitzky, A. (2007).  Comment: Performance of double-robust estimators when inverse probability weights are highly variable.  Statistical Science, 22:544–559.
  * Robins, (1999) Robins, J. M. (1999).  Association, causation, and marginal structural models.  Synthese, 121:151–179.
  * Robins and Greenland, (1992) Robins, J. M. and Greenland, S. (1992).  Identifiability and exchangeability for direct and indirect effects.  Epidemiology, 3:143–155.
  * Robins et al., (2000) Robins, J. M., Hernan, M. A., and Brumback, B. (2000).  Marginal structural models and causal inference in epidemiology.  Epidemiology, 11:550–560.
  * Robins et al., (1992) Robins, J. M., Mark, S. D., and Newey, W. K. (1992).  Estimating exposure effects by modelling the expectation of exposure conditional on confounders.  Biometrics, 48:479–495.
  * Robins and Wasserman, (1997) Robins, J. M. and Wasserman, L. A. (1997).  Estimation of effects of sequential treatments by reparameterizing directed acyclic graphs.  In Proceedings of the Thirteenth conference on Uncertainty in artificial intelligence, volume 409–420.
  * Rosenbaum, (1984) Rosenbaum, P. R. (1984).  The consequences of adjustment for a concomitant variable that has been affected by the treatment.  Journal of the Royal Statistical Society. Series A, 147:656–666.
  * (244) Rosenbaum, P. R. (1987a).  Model-based direct adjustment.  Journal of the American Statistical Association, 82:387–394.
  * (245) Rosenbaum, P. R. (1987b).  Sensitivity analysis for certain permutation inferences in matched observational studies.  Biometrika, 74:13–26.
  * Rosenbaum, (1989) Rosenbaum, P. R. (1989).  The role of known effects in observational studies.  Biometrics, 45:557–569.
  * (247) Rosenbaum, P. R. (2002a).  Covariance adjustment in randomized experiments and observational studies (with discussion).  Statistical Science, 17:286–327.
  * (248) Rosenbaum, P. R. (2002b).  Observational Studies.  Springer, 2nd edition.
  * Rosenbaum, (2015) Rosenbaum, P. R. (2015).  Two R packages for sensitivity analysis in observational studies.  Observational Studies, 1:1–17.
  * Rosenbaum, (2018) Rosenbaum, P. R. (2018).  Sensitivity analysis for stratified comparisons in an observational study of the effect of smoking on homocysteine levels.  Annals of Applied Statistics, 12:2312–2334.
  * Rosenbaum, (2020) Rosenbaum, P. R. (2020).  Modern algorithms for matching in observational studies.  Annual Review of Statistics and Its Application, 7:143–176.
  * (252) Rosenbaum, P. R. and Rubin, D. B. (1983a).  Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome.  Journal of the Royal Statistical Society - Series B (Statistical Methodology), 45:212–218.
  * (253) Rosenbaum, P. R. and Rubin, D. B. (1983b).  The central role of the propensity score in observational studies for causal effects.  Biometrika, 70:41–55.
  * Rosenbaum and Rubin, (1984) Rosenbaum, P. R. and Rubin, D. B. (1984).  Reducing bias in observational studies using subclassification on the propensity score.  Journal of the American statistical Association, 79:516–524.
  * Rosenbaum and Rubin, (2023) Rosenbaum, P. R. and Rubin, D. B. (2023).  Propensity scores in the design of observational studies for causal effects.  Biometrika, 110:1–13.
  * Rothman et al., (2008) Rothman, K. J., Greenland, S., Lash, T. L., et al. (2008).  Modern epidemiology, volume 3.  Wolters Kluwer Health/Lippincott Williams & Wilkins Philadelphia.
  * Rubin, (1974) Rubin, D. B. (1974).  Estimating causal effects of treatments in randomized and nonrandomized studies.  Journal of Educational Psychology, 66:688–701.
  * Rubin, (1975) Rubin, D. B. (1975).  Bayesian inference for causality: The importance of randomization.  In The Proceedings of the social statistics section of the American Statistical Association, volume 233, page 239. American Statistical Association Alexandria, VA.
  * Rubin, (1978) Rubin, D. B. (1978).  Bayesian inference for causal effects: The role of randomization.  Annals of Statistics, 6:34–58.
  * Rubin, (1980) Rubin, D. B. (1980).  Comment on “Randomization analysis of experimental data: the Fisher randomization test” by D. Basu.  Journal of American Statistical Association, 75:591–593.
  * Rubin, (2005) Rubin, D. B. (2005).  Causal inference using potential outcomes: Degisn, modeling, decisions.  Journal of American Statistical Association, 100:322–331.
  * (262) Rubin, D. B. (2006a).  Causal inference through potential outcomes and principal stratification: application to studies with “censoring” due to death (with discussion).  Statistical Science, 21:299–309.
  * (263) Rubin, D. B. (2006b).  Matched Sampling for Causal Effects.  Cambridge: Cambridge University Press.
  * Rubin, (2007) Rubin, D. B. (2007).  The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials.  Statistics in Medicine, 26:20–36.
  * Rubin, (2008) Rubin, D. B. (2008).  For objective causal inference, design trumps analysis.  Annals of Applied Statistics, 2:808–840.
  * Rudolph et al., (2018) Rudolph, K. E., Goin, D. E., Paksarian, D., Crowder, R., Merikangas, K. R., and Stuart, E. A. (2018).  Causal mediation analysis with observational data: considerations and illustration examining mechanisms linking neighborhood poverty to adolescent substance use.  American Journal of Epidemiology, 188:598–608.
  * Sabbaghi and Rubin, (2014) Sabbaghi, A. and Rubin, D. B. (2014).  Comments on the Neyman–Fisher controversy and its consequences.  Statistical Science, 29:267–284.
  * Salsburg, (2001) Salsburg, D. (2001).  The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century.  Henry Holt and Company.
  * Sanders and Karim, (2021) Sanders, E. Gustafson, P. and Karim, M. E. (2021).  Incorporating partial adherence into the principal stratification analysis framework.  Statistics in Medicine, 40:3625–3644.
  * Sanderson et al., (2017) Sanderson, E., Macdonald-Wallis, C., and Davey Smith, G. (2017).  Negative control exposure studies in the presence of measurement error: implications for attempted effect estimate calibration.  International Journal of Epidemiology, 47:587–596.
  * Scharfstein et al., (1999) Scharfstein, D. O., Rotnitzky, A., and Robins, J. M. (1999).  Adjusting for nonignorable drop-out using semiparametric nonresponse models.  Journal of the American Statistical Association, 94:1096–1120.
  * Schlesselman, (1978) Schlesselman, J. J. (1978).  Assessing effects of confounding variables.  American Journal of Epidemiology, 108:3–8.
  * Schochet et al., (2008) Schochet, P. Z., Burghardt, J., and McConnell, S. (2008).  Does job corps work? impact findings from the national job corps study.  American Economic Review, 98:1864–1886.
  * Sekhon, (2009) Sekhon, J. S. (2009).  Opiates for the matches: Matching methods for causal inference.  Annual Review of Political Science, 12:487–508.
  * Sekhon, (2011) Sekhon, J. S. (2011).  Multivariate and propensity score matching software with automated balance optimization: The matching package for R.  Journal of Statistical Software, 47:1–52.
  * Sekhon and Titiunik, (2017) Sekhon, J. S. and Titiunik, R. (2017).  On interpreting the regression discontinuity design as a local experiment.  In Regression Discontinuity Designs, volume 38. Emerald Publishing Limited.
  * Shinozaki and Matsuyama, (2015) Shinozaki, T. and Matsuyama, Y. (2015).  Doubly robust estimation of standardized risk difference and ratio in the exposed population.  Epidemiology, 26:873–877.
  * Sobel, (1982) Sobel, M. E. (1982).  Asymptotic confidence intervals for indirect effects in structural equation models.  Sociological Methodology, 13:290–312.
  * Sobel, (1986) Sobel, M. E. (1986).  Some new results on indirect effects and their standard errors in covariance structure models.  Sociological Methodology, 16:159–186.
  * Sommer and Zeger, (1991) Sommer, A. and Zeger, S. L. (1991).  On estimating efficacy from clinical trials.  Statistics in Medicine, 10:45–52.
  * Stuart, (2010) Stuart, E. A. (2010).  Matching methods for causal inference: A review and a look forward.  Statistical Science, 25:1–21.
  * Stuart and Jo, (2015) Stuart, E. A. and Jo, B. (2015).  Assessing the sensitivity of methods for estimating principal causal effects.  Statistical Methods in Medical Research, 24:657–674.
  * Tao and Fu, (2019) Tao, Y. and Fu, H. (2019).  Doubly robust estimation of the weighted average treatment effect for a target population.  Statistics in Medicine, 38:315–325.
  * Theil, (1953) Theil, H. (1953).  Estimation and simultaneous correlation in complete equation systems. central planning bureau.  Technical report, Mimeo, The Hague.
  * Thistlethwaite and Campbell, (1960) Thistlethwaite, D. L. and Campbell, D. T. (1960).  Regression-discontinuity analysis: An alternative to the ex post facto experiment.  Journal of Educational Psychology, 51:309.
  * Thistlewaite and Campbell, (2016) Thistlewaite, D. L. and Campbell, D. T. (2016).  Regression-discontinuity analysis: An alternative to the ex-post facto experiment (with discussion).  Observational Studies, 2:119–209.
  * Tibshirani, (1996) Tibshirani, R. (1996).  Regression shrinkage and selection via the lasso.  Journal of the Royal Statistical Society: Series B (Methodological), 58:267–288.
  * Titterington, (2013) Titterington, D. (2013).  Biometrika highlights from volume 28 onwards.  Biometrika, 100:17–73.
  * Valeri and Vanderweele, (2014) Valeri, L. and Vanderweele, T. J. (2014).  The estimation of direct and indirect causal effects in the presence of misclassified binary mediator.  Biostatistics, 15:498–512.
  * Van der Laan and Rose, (2011) Van der Laan, M. J. and Rose, S. (2011).  Targeted Learning: Causal Inference for Observational and Experimental Data.  New York: Springer.
  * Van der Vaart, (2000) Van der Vaart, A. W. (2000).  Asymptotic Statistics.  Cambridge: Cambridge University Press.
  * Van Elteren, (1960) Van Elteren, P. (1960).  On the combination of independent two-sample tests of wilcoxon.  Bulletin of the Institute of International Statistics, 37:351–361.
  * VanderWeele, (2008) VanderWeele, T. J. (2008).  Simple relations between principal stratification and direct and indirect effects.  Statistics and Probability Letters, 78:2957–2962.
  * VanderWeele, (2015) VanderWeele, T. J. (2015).  Explanation in Causal Inference: Methods for Mediation and Interaction.  Oxford: Oxford University Press.
  * VanderWeele et al., (2012) VanderWeele, T. J., Asomaning, K., and Tchetgen Tchetgen, E. J. (2012).  Genetic variants on 15q25.1, smoking, and lung cancer: An assessment of mediation and interaction.  American Journal of Epidemiology, 175:1013–1020.
  * VanderWeele and Ding, (2017) VanderWeele, T. J. and Ding, P. (2017).  Sensitivity analysis in observational research: introducing the E-value.  Annals of Internal Medicine, 167:268–274.
  * VanderWeele and Shpitser, (2011) VanderWeele, T. J. and Shpitser, I. (2011).  A new criterion for confounder selection.  Biometrics, 67:1406–1413.
  * VanderWeele and Tchetgen Tchetgen, (2017) VanderWeele, T. J. and Tchetgen Tchetgen, E. J. (2017).  Mediation analysis with time varying exposures and mediators.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79:917–938.
  * VanderWeele et al., (2014) VanderWeele, T. J., Tchetgen Tchetgen, E. J., Cornelis, M., and Kraft, P. (2014).  Methodological challenges in Mendelian randomization.  Epidemiology, 25:427.
  * Vansteelandt and Daniel, (2014) Vansteelandt, S. and Daniel, R. M. (2014).  On regression adjustment for the propensity score.  Statistics in Medicine, 33:4053–4072.
  * Vansteelandt and Dukes, (2022) Vansteelandt, S. and Dukes, O. (2022).  Assumption-lean inference for generalised linear model parameters (with discussion).  Journal of the Royal Statistical Society, Series B (Statistical Methodology), 84:657–685.
  * Vansteelandt and Joffe, (2014) Vansteelandt, S. and Joffe, M. (2014).  Structural nested models and G-estimation: the partially realized promise.  Statistical Science, 29:707–731.
  * Vermeulen and Vansteelandt, (2015) Vermeulen, K. and Vansteelandt, S. (2015).  Bias-reduced doubly robust estimation.  Journal of the American Statistical Association, 110:1024–1036.
  * Voight et al., (2012) Voight, B. F., Peloso, G. M., Orho-Melander, M., Frikke-Schmidt, R., Barbalic, M., Jensen, M. K., Hindy, G., Hólm, H., Ding, E. L., and Johnson, T. (2012).  Plasma HDL cholesterol and risk of myocardial infarction: a Mendelian randomisation study.  The Lancet, 380:572–580.
  * Wager and Athey, (2018) Wager, S. and Athey, S. (2018).  Estimation and inference of heterogeneous treatment effects using random forests.  Journal of the American Statistical Association, 113:1228–1242.
  * Wager et al., (2016) Wager, S., Du, W., Taylor, J., and Tibshirani, R. J. (2016).  High-dimensional regression adjustments in randomized experiments.  Proceedings of the National Academy of Sciences of the United States of America, 113:12673–12678.
  * Wald, (1940) Wald, A. (1940).  The fitting of straight lines if both variables are subject to error.  Annals of Mathematical Statistics, 11:284–300.
  * Wang et al., (2020) Wang, L., Zhang, Y., Richardson, T. S., and Zhou, X.-H. (2020).  Robust estimation of propensity score weights via subclassification.  arXiv preprint arXiv:1602.06366.
  * White, (1980) White, H. (1980).  A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity.  Econometrica, 48:817–838.
  * Wooldridge, (2016) Wooldridge, J. (2016).  Should instrumental variables be used as matching variables?  Research in Economics, 70:232–237.
  * Wooldridge, (2015) Wooldridge, J. M. (2015).  Control function methods in applied econometrics.  Journal of Human Resources, 50:420–445.
  * Wu and Ding, (2021) Wu, J. and Ding, P. (2021).  Randomization tests for weak null hypotheses in randomized experiments.  Journal of the American Statistical Association, 116:1898–1913.
  * Yang and Small, (2016) Yang, F. and Small, D. S. (2016).  Using post-outcome measurement information in censoring-by-death problems.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78:299–318.
  * Yang and Ding, (2018) Yang, S. and Ding, P. (2018).  Asymptotic causal inference with observational studies trimmed by the estimated propensity scores.  Biometrika, 105:487–493.
  * Zelen, (1979) Zelen, M. (1979).  A new design for randomized clinical trials.  New England Journal of Medicine, 300:1242–1245.
  * Zhang and Rubin, (2003) Zhang, J. L. and Rubin, D. B. (2003).  Estimation of causal effects via principal stratification when some outcomes are truncated by “death”.  Journal of Educational and Behavioral Statistics, 28:353–368.
  * Zhang et al., (2009) Zhang, J. L., Rubin, D. B., and Mealli, F. (2009).  Likelihood-based analysis of causal effects of job-training programs using principal stratification.  Journal of the American Statistical Association, 104:166–176.
  * Zhang and Ding, (2022) Zhang, M. and Ding, P. (2022).  Interpretable sensitivity analysis for the baron-kenny approach to mediation with unmeasured confounding.  arXiv preprint arXiv:2205.08030.
  * (319) Zhao, A. and Ding, P. (2021a).  Covariate-adjusted Fisher randomization tests for the average treatment effect.  Journal of Econometrics, 225:278–294.
  * (320) Zhao, A. and Ding, P. (2021b).  No star is good news: A unified look at rerandomization based on $p$-values from covariate balance tests.  arXiv preprint arXiv:2112.10545.
  * Zhao et al., (2020) Zhao, Q., Wang, J., Hemani, G., Bowden, J., and Small, D. (2020).  Statistical inference in two-sample summary-data Mendelian randomization using robust adjusted profile score.  Annals of Statistics, 48:1742–1769.

### Index