Stony Brook University
The Graduate School

Xuewen Yang

We the dissertation committee for the above candidate for the
Doctor of Philosophy degree,
hereby recommend acceptance of this dissertation.
Dr. Xin Wang - Advisor of Dissertation
Associate Professor, Department of Electrical and Computer Engineering

Dr. Murali Subbarao, - Chairperson of Defense
Professor, Department of Electrical and Computer Engineering

Dr. Peter Milder - Defense Committee Member
Associate Professor, Department of Electrical and Computer Engineering

Dr. Niranjan Balasubramanian - Defense Committee Member
Assistant Professor, Department of Computer Science

This dissertation is accepted by the Graduate School

Eric Wertheimer
Dean of the Graduate School
.

# Exploring External Knowledge for Accurate modeling of Visual and Language
Problems

Xuewen Yang

##  Chapter 1 Introduction and Background

###  1.1 Artificial Intelligence

We humans have the nature or intelligence to acquire new knowledge and skills.
This intelligence has helped us to complete our most important achievements,
from hunting and farming in the wild to modern medicine and spacecraft, by
creating handful tools like bows and arrows, machines, microscopes, robots,
etc. However, we humans also have the innate limitations, such as we are prone
to be tired. It’s getting more and more difficult for us to keep updated with
the massive data everyday. The question for now is “what if we apply our
intelligence to create new tools that can learn from the increasing amount of
data in more efficient and effective ways?” This question is central to the
field of artificial intelligence. In the rapidly developing subfield of
machine learning, specifically, we set out to acquire new knowledge and skills
to build machines that themselves can acquire new knowledge and skills. The
aim of this work is further advancement of the field of artificial
intelligence, and through this means, to increase the probability of a future
that is bright.

###  1.2 Deep Generative Models

As Richard Feynman said “What I cannot create, I do not understand”, for a
model to understand the visual and language world, it has to be capable to
create new things based on what it has. Generative models are the models that
are able to achieve this goal. The term “generative model”, in this work,
refer to any model that takes training data, consisting of samples drawn from
a distribution $p_{data}$, and learns to represent an estimate of that
distribution somehow. The result is a probability distribution $p_{model}$.

One might legitimately wonder why generative models are worth studying and why
they are important to AI, especially generative models that are only capable
of generating data rather than providing an estimate of the density function.
After all, when applied to images, such models seem to merely generate more
images, and the world has no shortage of images. There are several reasons to
study generative models. In this work, the author lists three of them,
including:

  * •

Training and sampling from generative models is an excellent test of our
ability to represent and manipulate high-dimensional probability
distributions, which are important objects in a wide variety of applied math
and engineering domains.

  * •

Generative models can be trained with missing data and can provide predictions
on inputs that are missing data. One particularly interesting case of missing
data is semi-supervised learning, in which the labels for many or even most
training examples are missing. Generative models are able to perform semi-
supervised learning reasonably well.

  * •

Many tasks intrinsically require realistic generation of samples from some
distribution, including image super-resolution and image-to-image translation
applications, etc.

All of these and other applications of generative models provide compelling
reasons to invest time and resources into improving generative models. In this
work, we target on image-to-image translation and how to use domain labels as
the external knowledge to solve the multi-domain image-to-image translation
problem.

###  1.3 Sequence to Sequence Models

Despite their flexibility and power, Convolutional Neural Networks (CNNs) [1,
2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] can only be applied to
problems whose inputs and targets can be sensibly encoded with vectors of
fixed dimensionality. It is a significant limitation, since many important
problems are best expressed with sequences whose lengths are not known
a-priori. For example, speech recognition and machine translation are
sequential problems. Likewise, question answering can also be seen as mapping
a sequence of words representing the question to a sequence of words
representing the answer. It is therefore clear that a domain-independent
method that learns to map sequences to sequences would be useful.

A straightforward application of the Recurrent Neural Networks (RNNs),
including the following Long Short-Term Memory (LSTM) architectures and
Transformers can solve general sequence to sequence problems. The idea is to
use one RNN to read the input sequence, one time step at a time, to obtain
large fixed dimensional vector representation, and then to use another RNN to
extract the output sequence from that vector. The second RNN is thus
conditioned on the input sequence. In the current applications, LSTMs and
Transformers are more widely used than the vanilla RNNs because of their
ability to successfully learn on data with long range temporal dependencies,
which makes them a natural choice for this sequence to sequence application
due to the considerable time lag between the inputs and their corresponding
outputs.

##  Chapter 2 Image-to-Image Translation

###  2.1 Introduction

In this work, we define multi-domain as multiple datasets or several subsets
of one dataset that are applied to complete the same task, but these datasets
(or subsets) have different statistical biases. As some examples, images taken
at Alps in the summer and in the winter are considered as two different
domains, while faces with hair and faces with eyeglasses form another two
different domains. Under this domain definition, for faces with black hair and
faces with yellow hair, the black hair and yellow hair are two different
attributes of the same domain. In multi-domain learning, each sample $\bm{x}$
is drawn from a domain $d$ specific distribution $\bm{x}\sim p_{d}(\bm{x})$
and has a label $y\in\\{0,1\\}$, with $y=1$ signifying $\bm{x}$ from domain
$d$, $y=0$ signifying $\bm{x}$ not from domain $d$.

Image-to-image translation is the task of learning to map images from one
domain to another, e.g., mapping grayscale images to color images [16],
mapping images of low resolution to images of high resolution [17], changing
the seasons of scenery images [18], and reconstructing photos from edge maps
[19]. The most significant improvement in this research field came with the
application of Generative Adversarial Networks (GANs) [20, 21].

The image-to-image translation can be performed in supervised [19] or
unsupervised way [18], with the unsupervised one becoming more popular since
it does not need to collect ground-truth pairs of samples. Despite the quick
progress of research on image-to-image translation, state-of-the-art results
for unsupervised translation are still not satisfactory. In addition, existing
research generally focuses on image-to-image translation between two domains,
which is limited by two drawbacks. First, the translation task is specific to
two domains, and the model has to be retrained when there is a need to perform
image translation between another pair of similar domains. Second, it can not
benefit from the features of multiple domains to improve the training quality.
We take the most representative work in this research field CycleGAN [18] as
an example to illustrate the first limitation. The translation between two
image domains $X$ and $Y$ is achieved with two generators, $G_{X\rightarrow
Y}$ and $G_{Y\rightarrow X}$. However, this model is inefficient in completing
the task of multi-domain image translation. To derive mappings across all $n$
domains, it has to train $n(n-1)$ generators, as shown in Fig. 2.1(a).

(a)

(b)

Figure 2.1: Image-to-image translation of 4 domains. (a) CycleGAN needs
$4\times 3$ generators. (b) Our model only needs 2 encoder-generator pairs. In
every iteration, we randomly pick two domains, and sample two batches of
training data from the domains to train the model. The two encoders first
encode domain information into a latent code $\bm{z}$ using two encoders
$E_{X}$ and $E_{Y}$ and then generate two samples of the two domains using the
generators $G_{X}$ and $G_{Y}$.

To enable more efficient multi-domain image translation with unsupervised
learning where image pairing across domains is not predefined, we propose
Crossing-Domain GAN (CD-GAN), which is a multi-domain encoding generative
adversarial network that consists of a pair of encoders and a pair of
generative adversarial networks (GANs). We would like the encoders to
efficiently encode the information of all domains to form a high-level feature
space with an encoding process, then images of different domains will be
translated by decoding the high-level features with a decoding process. CD-GAN
achieves this goal with the integrated use of three techniques. First, the two
encoders are constrained by a weight sharing scheme, where the two encoders
(or the two generators) share the same weights at both the highest-level
layers and the lowest-level layers. This ensures that the two encoders can
encode common high-level semantics as well as low-level details to obtain the
feature space, based on which generators can decode the high-level semantics
and low-level details correctly to generate images of different domains.
Second, we use a selected or existing label to guide the generator to generate
images of a corresponding domain from the high-level features learnt. Third,
we propose an efficient training algorithm that jointly train the model across
domains by randomly selecting two domains to train at each iteration.

Different from [21] where only weights at high-level layers of generators are
shared, in CD-GAN, we propose the concurrent sharing of the lowest-level and
the highest-level layers at both the encoders and the generators to improve
the quality of image translation between any two domains. The sharing of
highest layers between two encoders helps to enable more flexible cross-domain
image translation, while the sharing of the lowest layers across domains helps
improve the training quality by taking advantage of the transferring learning
across domains.

The contributions of our work are as follows:

  * •

We propose CD-GAN that learns mappings across multiple domains using only two
encoder-generator pairs.

  * •

We propose the concurrent use of weight-sharing at highest-level and lowest-
level layers of both encoders and generators to ensure that CD-GAN generates
images with sufficient useful high-level semantics and low-level details
across all domains.

  * •

We leverage domain labels to make a conditional GAN training that greatly
improves the performance of the model.

  * •

We introduce a cross-domain training algorithm that efficiently and
sufficiently trains the model by randomly taking samples from two of domains
at a time. CD-GAN can fully exploit data from all domains to improve the
training quality for each individual domain.

Our experiment results demonstrate that when trained on more than two domains,
our method achieves the same quality of image translation between any two
domains as compared to directly training for translation between the pair.
However, our model is established with much less training time and can
generate better quality images for a given amount of time. We also show how
CD-GAN can be successfully applied to a variety of unsupervised multi-domain
image-to-image translation problems.

The remainder of this chapter is organized as follows. Section 2.2 reviews the
relevant research for image-to-image translation problems. Section 2.4
describes our model and training method in details. Section 2.5 presents our
evaluation metrics, experimental methodology, and the evaluation results of
the model’s accuracy and efficiency on different datasets. Finally, we discuss
some limitations of our work and conclude our work in Section 2.6.

###  2.2 Related Work

####  2.2.1 Generative Adversarial Networks (GANs)

GANs [20] were introduced to model a data distribution using independent
latent variables. Let $\bm{x}\sim p(\bm{x})$ be a random variable representing
the observed data and $\bm{z}\sim p(\bm{z})$ be a latent variable. The
observed variable is assumed to be generated by the latent variable, i.e.,
$\bm{x}\sim p_{\bm{\theta}}(\bm{x}|\bm{z})$, where
$p_{\bm{\theta}}(\bm{x}|\bm{z})$ can be explicitly represented by a generator
in GANs. GANs are built on top of neural networks, and can be trained with
gradient descent based algorithms.

The GAN model is composed of a discriminator $D_{\bm{\phi}}$, along with the
generator $G_{\bm{\theta}}$. The training involves a min-max game between the
two networks. The discriminator $D_{\bm{\phi}}$ is trained to differentiate
‘fake’ samples generated from the generator $G_{\bm{\theta}}$ from the ‘real’
samples from the true data distribution $p(\bm{x})$. The generator is trained
to synthesize samples that can fool the discriminator by mistaking the
generated samples for genuine ones. They both can be implemented using neural
networks.

At the training phase, the discriminator parameters $\bm{\phi}$ are firstly
updated, followed by the update of the generator parameters $\bm{\theta}$. The
objective function is given by:

$\displaystyle\min_{\bm{\theta}}\max_{\bm{\phi}}V(D,G)=$
$\displaystyle\mathbb{E}_{x\sim p(x)}[\log D_{\bm{\phi}}(x)]$ (2.1)
$\displaystyle+\mathbb{E}_{\bm{z}\sim
p(\bm{z})}[\log(1-D_{\bm{\phi}}(G_{\bm{\theta}}(\bm{z})))]$

The samples can be generated by sampling $\bm{z}\sim p(\bm{z})$, then
$\hat{\bm{x}}=G_{\bm{\theta}}(\bm{z})$, where $p(\bm{z})$ is a prior
distribution, for example, a multivariate Gaussian.

####  2.2.2 Image-To-Image Translation

Image-to-image translation problem is a kind of image generation task that
given an input image $\bm{x}$ of domain X, the model maps it into a
corresponding output image $\bm{y}$ of another domain Y. It learns a mapping
between two domains given sufficient training data [19]. Early works on image-
to-image translation mainly focused on tasks where the training data of domain
$X$ are similar to the data of domain $Y$ [22, 23], and the results were often
unrealistic and not diverse.

In recent years, deep generative models have shown increasing capability of
synthesizing diverse, realistic images that capture both fine-grained details
and global coherence of natural images [24, 25, 26]. With Generative
Adversarial Networks (GANs) [19, 18, 27], recent studies have already taken
significant steps in image-to-image translation. In [19], the authors use a
conditional GAN on different image-to-image translation tasks, such as
synthesizing photos from label maps and reconstructing objects from edge maps.
However, this method requires input-output image pairs for training, which is
in general not available in image-to-image translation problems. For
situations where such training pairs are not given, in [18], the authors
proposed CycleGAN to tackle unsupervised image-to-image translation. With a
pair of Generators $G$ and $F$, the model not only learns a mapping
$G:X\rightarrow Y$ using an adversarial loss, but constrains this mapping with
an inverse mapping $F:Y\rightarrow X$. It also introduces a cycle consistency
loss to enforce $F(G(X))\approx X$, and vice versa. In settings where paired
training data are not available, the authors showed promising qualitative
results. The authors in [27] and [28] use similar idea to solve the
unsupervised image-to-image translation tasks.

These approaches only tackle the problems of translating images between two
domains, and have two major drawbacks. First, when applied to $n$ domains,
these approaches need $n(n-1)$ generators to complete the task, which is
computationally inefficient. To train all models, it would either require a
significant amount of time to complete if the training is performed on one
GPU, or it will require a lot of hardware and computing resources if training
is run over multiple GPUs. Second, as each model is trained with only two
datasets, the training cannot benefit from the data of other domains.

Our work is inspired by multimodal learning [29], which shows that data
features can be better extracted using one modality if multiple modalities are
present at feature learning time. The intuition of our method is that if we
can encode the information of different domains together and generate a high-
level feature space, it would be possible to decode the high-level features to
build images of different domains. In this work, rather than generating images
from random noise, we incorporate an encoding process into a GAN model. The
image-to-image translation can be achieved by first encoding real images into
high-level features, and then generating images of different domains using the
high-level features through a decoding process. The encoding process and the
decoding process are constrained by a weight-sharing technique that both the
highest layer and the lowest layer are shared across the two encoders as well
as the two generators. Sharing the high-level layers makes sure that the
generated images are semantically correct, while sharing the low-level layers
ensures that important low-level features be captured and transferred between
domains. Our model is trained end-to-end using data from all $n$ domains.

###  2.3 Background

###  2.4 Cross-Domain Generative Adversarial Network

To conduct unsupervised multi-domain image-to-image translation, a direct
approach is to train a CycleGAN for every two domains. While this approach is
straightforward, it is inefficient as the number of training models increases
quadratically with the number of domains. If we have $n$ domains, we have to
train $n(n-1)$ generators, as shown in Fig. 2.1(a). In addition, since each
model only utilizes data from two domains to train, the training cannot
benefit from the useful features of other domains.

To tackle these two problems, a possible way is to encode useful information
of all domains into common high level features, and then to decode the high-
level features into images of different domains. Inspired by work [29] from
mutimodal learning, where training data are from multiple modalities, we
propose to build a multi-domain image translation model that can encode
information of multiple domains into a set $Z$ of high-level features, and
then use features in $Z$ to reconstruct data of different domains or to do
image-to-image translation. The overview of the model applied to 4 domains is
shown in Fig. 2.1(b), where only one model is used.

In this section, we first present our proposed CD-GAN model, then describe how
image translation can be performed across domains, and finally introduce our
cross-domain training method.

####  2.4.1 CD-GAN with Double Layer Sharing

We first describe how to apply our model to multi-domain image-to-image
translation in general then illustrate it using two domains as an example. As
shown in Fig. 2.2(a), our proposed CD-GAN model consists of a pair of encoders
followed by a pair of GANs. Taking domain $X$ and $Y$ as an example, the two
encoders $E_{X}$ and $E_{Y}$ encode domain information from $X$ and $Y$ into a
set of high-level features contained in a set $Z$. Then from a high-level
feature $z$ in space $Z$, we can generate images that fall into domain $X$ or
$Y$. The generated images are then evaluated by the corresponding
discriminators $D_{X}$ and $D_{Y}$ to see whether they look real and cannot be
identified as generated ones. For example, following the red arrows, the input
image $\bm{x}$ is first encoded into a high-level feature $\bm{z}_{x}$, then
$\bm{z}_{x}$ is decoded to generate the image $\hat{\bm{y}}$. The image
$\hat{\bm{y}}$ is the translated image in domain $Y$. Similar processes exist
for image $\bm{y}$.

Our model is also constrained by a reconstruction process shown in Fig.
2.2(b). For example, following the red arrows, the input image $\bm{x}$ is
first encoded into a high-level feature $\bm{z}_{x}$, then $\bm{z}_{x}$ is
decoded to generate the image $\bm{x}\prime$, which is a reconstruction of the
input image. Similar processes exist for image $\bm{y}$.

Learning with deep neural networks involves hierarchical feature
representation. In order to support flexible cross-domain image translation
and also to improve the training quality, we propose the use of double-layer
sharing where the highest-level and the lowest-level layers of the two
encoders share the same weights and so does the two generators. By enforcing
the layers that decode high-level features in GANs to share weights, the
images generated by different generators can have some common high-level
semantics. The layers that decode low-level details then map the high-level
features to images in individual domains.

Sharing weights of low-level layers has the benefit of transferring low-level
features of one domain to the other, thus making the image-to-image
translation more close to real images in the respective domains. Besides,
sharing layers reduces the complexity of the model, making it more resistant
to the over-fitting problem.

####  2.4.2 Conditional Image Generation

In state-of-the-art techniques, like CycleGAN, each domain is described by a
specific generator, thus there is no need to inform the generator which domain
the input image is generated to. However, in our model, multiple domains share
two generators. For an input image, we have to include an auxiliary variable
to guide the generation of image for a specific domain. The only information
we have is the domain labels. To make use of this information, the inputs of
the model are not images $\bm{x}$, $\bm{y}$, but image pairs
$(\bm{x},\bm{l}_{y})$ and $(\bm{y},\bm{l}_{x})$ where the labels $\bm{l}_{y}$
and $\bm{l}_{x}$ inform the generators which domains to generate an image for.
These image pairs are not the same as the image pairs of supervised image-to-
image generation tasks, which are $(\bm{x},\bm{y})$. Thus no matter which
domain images are the input, the model can always generate images of a domain
of interest.

(a)

(b)

Figure 2.2: The proposed CD-GAN model. (a) The translation mappings: the input
image $\bm{x}$ is first encoded as a latent code $\bm{z}_{x}$ through
$E_{X}(\bm{x})$, which is then decoded into a translated image $\hat{\bm{y}}$
through $G_{Y}(\bm{z}_{x},\bm{l}_{y})$. The process is identified with red
arrows. There is a similar process for the image $\bm{y}$. $D_{X}$ and $D_{Y}$
are adversarial discriminators for the respective domains to evaluate whether
the translated images are realistic. (b) The reconstruction mappings: the
input image $\bm{x}$ is first encoded as a latent code $\bm{z}_{x}$ through
$E_{X}(\bm{x})$, which is then decoded into a reconstructed image
$\bm{x}\prime$ through the generator $G_{X}(\bm{z}_{x},\bm{l}_{x})$. The
process is signified in red arrows. A similar process exists for image
$\bm{y}$. Note: the dashed lines indicate that the two layers share the same
parameters.

We denote the data distributions as $\bm{x}\sim p(\bm{x})$ and $\bm{y}\sim
p(\bm{y})$. As illustrated in Fig. 2.2, our model includes four mappings, two
translation mappings $X\rightarrow Z\rightarrow Y$, $Y\rightarrow Z\rightarrow
X$ and two reconstruction mappings $X\rightarrow Z\rightarrow X$,
$Y\rightarrow Z\rightarrow Y$. The translation mappings constrain the model by
a GAN loss, while the reconstruction mappings constrain the model by a
reconstruction loss. To further constrain the auxiliary variable, we introduce
a classification loss by applying a classifier to classify the real or
generated images into different domains. The intuition is that if images are
generated with the guidance of the auxiliary variable, then it can be
correctly classified into the domain specified by the auxiliary variable.
Next, we introduce these model losses in more details as follows.

GAN Losses Following the translation mapping $X\rightarrow Z\rightarrow Y$, we
can translate image $\bm{x}$ from domain $X$ to $\hat{\bm{y}}$ of domain $Y$
using $\bm{z}_{x}=E_{X}(\bm{x})$, $\hat{\bm{y}}=G_{Y}(\bm{z}_{x},\bm{l}_{y})$.
With the purpose of improving the quality of the generated samples, we apply
adversarial loss. We express the objective as:

$\displaystyle\mathcal{L}_{GAN_{Y}}$ $\displaystyle=\mathbb{E}_{\bm{y}\sim
p(\bm{y})}\log(D_{Y}(\bm{y}))$ (2.2) $\displaystyle+\mathbb{E}_{\bm{x}\sim
p(\bm{x})}\log(1-D_{Y}(G_{Y}(E_{X}(\bm{x}),\bm{l}_{y})))$

where $G_{Y}$ tries to generate images
$\hat{\bm{y}}=G_{Y}(\bm{z}_{x},\bm{l}_{y})$ that look similar to images from
domain $Y$, while $D_{Y}$ aims to distinguish between translated samples
$\hat{\bm{y}}$ and real samples $\bm{y}$. The similar adversarial loss for
$Y\rightarrow Z\rightarrow X$ is

$\displaystyle\mathcal{L}_{GAN_{X}}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim
p(\bm{x})}\log(D_{X}(\bm{x}))$ (2.3) $\displaystyle+\mathbb{E}_{\bm{y}\sim
p(\bm{y})}\log(1-D_{X}(G_{X}(E_{Y}(\bm{y}),\bm{l}_{x})))$

The total GAN loss is:

$\mathcal{L}_{GAN}=\mathcal{L}_{GAN_{X}}+\mathcal{L}_{GAN_{Y}}$ (2.4)

Reconstruction Loss The reconstruction mappings $X\rightarrow Z\rightarrow X$,
$Y\rightarrow Z\rightarrow Y$ encourage the model to encode enough information
to the high-level feature space $Z$ from each domain. The input can then be
reconstructed by the generators. The reconstruction process of domain $X$ is
$\bm{z}_{x}=E_{X}(\bm{x})$, $\bm{x}\prime=G_{X}(\bm{z}_{x},\bm{l}_{x})$.
Similar reconstruction process exists for domain $Y$. With $l_{2}$ distance as
the loss function, the reconstruction loss is:

$\displaystyle\mathcal{L}_{rec}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim
p(\bm{x})}(||\bm{x}-G_{X}(E_{X}(\bm{x}),\bm{l}_{x})||_{2})$ (2.5)
$\displaystyle+\mathbb{E}_{\bm{y}\sim
p(\bm{y})}(||\bm{y}-G_{Y}(E_{Y}(\bm{y}),\bm{l}_{y})||_{2})$

Latent Consistency Loss With only the above losses, the encoding part is not
well constrained. We constrain the encoding part using a latent consistency
loss. Although $\bm{x}$ is translated to $\hat{\bm{y}}$, which is in domain
$Y$, $\hat{\bm{y}}$ is still semantically similar to $\bm{x}$. Thus, in the
latent space $Z$, the high-level feature of $\bm{x}$ should be close to that
of $\hat{\bm{y}}$. Similarly, the high-level feature of $\bm{y}$ in domain $Y$
should be close to the high-level feature of $\hat{\bm{x}}$ in domain $X$. The
latent consistency loss is the following:

$\displaystyle\mathcal{L}_{lcl}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim
p(\bm{x})}(||E_{X}(\bm{x})-E_{Y}(G_{Y}(E_{X}(\bm{x}),\bm{l}_{y}))||)$ (2.6)
$\displaystyle+\mathbb{E}_{\bm{y}\sim
p(\bm{y})}(||E_{Y}(\bm{y})-E_{X}(G_{X}(E_{Y}(\bm{y}),\bm{l}_{x}))||)$

Classification Loss We consider $n$ domains as $n$ categories in the
classification problems. We use a network $C$, which is an auxiliary
classifier, on top of the general discriminator $D$ to measure whether a
sample (real or generated) belongs to a specific fine-grained category. The
output of the classifier $C$ represents the posterior probability
$P(c|\bm{x})$. Specifically, there are four classification losses, i.e., for
real data $\bm{x}$, $\bm{y}$, and generated data $\hat{\bm{x}}$,
$\hat{\bm{y}}$. For image-label pairs ($\bm{x}$, $\bm{l}_{x}$) and ($\bm{y}$,
$\bm{l}_{y}$) with $\bm{l}_{x}\sim p(\bm{l}_{x})$ and $\bm{l}_{y}\sim
p(\bm{l}_{y})$ our goal is to translate $\bm{x}$ to $\hat{\bm{y}}$ with label
$\bm{l}_{y}$, and to translate $\bm{y}$ to $\hat{\bm{x}}$ with label
$\bm{l}_{x}$. The four classification losses are:

$\displaystyle\mathcal{L}_{c}$ $\displaystyle=-\mathbb{E}_{\bm{x}\sim
p(\bm{x}),\bm{l}_{x}\sim p(\bm{l}_{x})}[\log P(\bm{l}_{x}|\bm{x})]$ (2.7)
$\displaystyle=-\mathbb{E}_{\bm{y}\sim p(\bm{y}),\bm{l}_{y}\sim
p(\bm{l}_{y})}[\log P(\bm{l}_{y}|\bm{y})]$
$\displaystyle=-\mathbb{E}_{\bm{x}\sim p(\bm{x}),\bm{l}_{y}\sim
p(\bm{l}_{y})}[\log P(\bm{l}_{y}|G_{Y}(E_{X}(\bm{x}),\bm{l}_{y}))]$
$\displaystyle=-\mathbb{E}_{\bm{y}\sim p(\bm{y}),\bm{l}_{x}\sim
p(\bm{l}_{x})}[\log P(\bm{l}_{x}|G_{X}(E_{Y}(\bm{y}),\bm{l}_{x}))]$

This loss can be used to optimize discriminators $D_{X}$, $D_{Y}$, generators
$G_{X}$, $G_{Y}$, and encoders $E_{X}$, $E_{Y}$.

Cycle Consistency Loss Although the minimization of GAN losses ensures that
$G_{Y}(E_{X}(\bm{x}),\bm{l}_{y})$ produce a sample $\hat{\bm{y}}$ similar to
samples drawn from $Y$, the model still can be unstable and prone to failure.
To tackle this problem, we further constrain our model with a cycle-
consistency loss [18]. To achieve this goal, we want mapping from domain $X$
to domain $Y$ and then back to domain $X$ to reproduce the original sample,
i.e., $G_{X}(E_{Y}(G_{Y}(E_{X}(\bm{x}),\bm{l}_{y})),\bm{l}_{x})\approx\bm{x}$
and $G_{Y}(E_{X}(G_{X}(E_{Y}(\bm{y}),\bm{l}_{x})),\bm{l}_{y})\approx\bm{y}$.
Thus, the cycle-consistency loss is:

$\displaystyle\mathcal{L}_{cyc}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim
p(\bm{x})}[||G_{X}(E_{Y}(G_{Y}(E_{X}(\bm{x}),\bm{l}_{y})),\bm{l}_{x})-\bm{x}||]$
(2.8) $\displaystyle+\mathbb{E}_{\bm{y}\sim
p(\bm{y})}[||G_{Y}(E_{X}(G_{X}(E_{Y}(\bm{y}),\bm{l}_{x})),\bm{l}_{y})-\bm{y}||]$

Final Objective of CD-GAN To sum up, the goal of our approach is to minimize
the following objective:

$\displaystyle\mathcal{L}(E,G,D)$
$\displaystyle=\mathcal{L}_{GAN}+\alpha_{0}\mathcal{L}_{rec}+\alpha_{1}\mathcal{L}_{lcl}+\alpha_{2}\mathcal{L}_{c}+\alpha_{3}\mathcal{L}_{cyc}$
(2.9)

where $E$, $G$, and $D$ signify encoders $E_{X}$, $E_{Y}$, generators $G_{X}$,
$G_{Y}$, and discriminators $D_{X}$, $D_{Y}$, and $\alpha_{0}$, $\alpha_{1}$,
$\alpha_{2}$, $\alpha_{3}$ control the relative importance of the losses. Same
as solving a regular GAN problem, training the model involves the solving of a
min-max problem, where $E_{X}$,$E_{Y}$, $G_{X}$, and $G_{Y}$ aim to minimize
the objective, while $D_{X}$ and $D_{Y}$ aim to maximize it.

$E^{\ast},G^{\ast}=arg\min_{E,G}\max_{D}\mathcal{L}(E,G,D)$ (2.10)

####  2.4.3 Cross-Domain Training

Our proposed model has two encoder-generator pairs, but we have data from $n$
domains. To train the model using samples of all domains equally, we introduce
a cross-domain training algorithm. As shown in Fig. 2.1(b), there are 4
domains. At each iteration, we randomly select two domains $R$ and $S$, and
feed training data of these two domains into the model. At the next iteration,
we might take another two domains $P$ and $Q$, and perform the same training.
We train the model using all data samples of $4$ domains at every epoch for
several iterations. The training algorithm is shown in Algorithm 1. Cross-
domain training ensures the model to learn a generic feature representation of
all domains by training the model equally on independent domains.

Algorithm 1 Joint domain training on CD-GAN using mini-batch stochastic
gradient descent

Require: Training samples from $n$ domains

Initialize $\bm{\theta}_{E}^{X}$, $\bm{\theta}_{E}^{Y}$,$\bm{\theta}_{G}^{X}$,
$\bm{\theta}_{G}^{Y}$,$\bm{\theta}_{D}^{X}$, and $\bm{\theta}_{D}^{Y}$ with
the shared network connection weights set to the same values.

while Training loss has not converged do

Randomly draw two domains $X$ and $Y$ from $n$ domains

Randomly draw $N$ samples from the two domains,
{$\bm{x}_{1},\bm{x}_{2},\ldots\bm{x}_{N};\bm{y}_{1},\bm{y}_{2},\ldots\bm{y}_{N}$}

Get the domain labels of the samples from the two domains,
$\\{\bm{l}_{X}^{i},\bm{l}_{Y}^{i}\\}_{i=1}^{N}$

(1) Update $\bm{D}_{X},\bm{D}_{Y}$ with fixed
$\bm{G}_{X},\bm{G}_{Y},\bm{E}_{X},\bm{E}_{Y}$

Generate fake samples using the real ones
$\displaystyle\hat{\bm{x}}_{i}=G_{X}(E_{Y}(\bm{y}_{i}),\bm{l}_{x}^{i}),\hat{\bm{y}}_{i}=G_{Y}(E_{X}(\bm{x}_{i}),\bm{l}_{y}^{i}),i=1\ldots
N$

Update $\bm{\theta}_{D}=(\bm{\theta}_{D}^{X},\bm{\theta}_{D}^{Y})$ according
to the following gradients
$\displaystyle\nabla_{\theta_{D}}\bigg{[}\frac{1}{N}\sum_{i=1}^{N}\Big{[}-\log
D_{X}(\bm{x}_{i})-\log(1-D_{X}(\hat{\bm{x}}_{i}))-\log D_{Y}(\bm{y}_{i})$
$\displaystyle-\log(1-D_{Y}(\hat{\bm{y}}_{i}))+\alpha_{2}\big{[}\log
P(\bm{l}_{x}|\bm{x}_{i})+\log P(\bm{l}_{y}|\bm{y}_{i})\big{]}\Big{]}\bigg{]}$

(2) Update $\bm{E}_{X},\bm{E}_{Y},\bm{G}_{X},\bm{G}_{Y}$ with fixed
$\bm{D}_{X},\bm{D}_{Y}$

Update
$\bm{\theta}_{E,G}=(\bm{\theta}_{E}^{X},\bm{\theta}_{E}^{Y},\bm{\theta}_{G}^{X},\bm{\theta}_{G}^{Y})$
according to the following gradients
$\displaystyle\nabla_{\theta_{E,G}}\bigg{[}\frac{1}{N}\sum_{i=1}^{N}\Big{[}\log(1-D_{X}(\hat{\bm{x}}_{i}))+\log(1-D_{Y}(\hat{\bm{y}}_{i}))$
$\displaystyle+||\bm{x}_{i}-G_{X}(E_{X}(\bm{x}_{i}),\bm{l}_{x}^{i})||_{2}+||\bm{y}_{i}-G_{Y}(E_{Y}(\bm{y}_{i}),\bm{l}_{y}^{i})||_{2}$
$\displaystyle+||E_{X}(\bm{x}_{i})-E_{Y}(\hat{\bm{y}}_{i})||+||E_{Y}(\bm{y}_{i})-E_{X}(\hat{\bm{x}}_{i})||$
$\displaystyle+\log P(\bm{l}_{x}|\hat{\bm{x}}_{i})+\log
P(\bm{l}_{y}|\hat{\bm{y}}_{i})$
$\displaystyle+\alpha\big{[}||\bm{x}_{i}-G_{X}(E_{Y}(\hat{\bm{y}}_{i}),\bm{l}_{x}^{i})||+||\bm{y}_{i}-G_{Y}(E_{X}(\hat{\bm{x}}_{i}),\bm{l}_{y}^{i})||\big{]}\Big{]}\bigg{]}$

end while

###  2.5 experiment

In this section, we conduct experiments over three datasets to compare our
proposed model with reference models in terms of image translation quality and
efficiency.

####  2.5.1 Datasets

To evaluate the scalability and effectiveness of our model, we test it on a
variety of multi-domain image-to-image translation tasks using the following
datasets:

Alps Seasons dataset [30] is collected from images on Flickr. The images are
categorized into four seasons based on the provided timestamp of when it was
taken. It consists of four categories: Spring, Summer, Fall, and Winter. The
training data consists of 6053 images of four seasons, while the test data
consists of 400 images.

Painters dataset [18] includes painting images of four artists Monet, Van
Gogh, Cezanne, and Ukiyo-e. We use 2851 images as the training set, and 200
images as the test set.

CelebA dataset [31] contains ten thousand identities, each of which has twenty
images, i.e., two hundred thousand images in total. Each image in CelebA is
annotated with 40 face attributes. We resize the initial $178\times 218$ size
images to $256\times 256$. We randomly select 4000 images as test set and use
all remaining images for training data.

We run all the experiments on a Ubuntu system using an Intel i7-6850K, along
with a single NVIDIA GTX 1080Ti GPU.

####  2.5.2 Reference Models

We compare the performance of our proposed CD-GAN with that of two reference
models:

CycleGAN [18] This method trains two generators $G:X\rightarrow Y$ and
$F:Y\rightarrow X$ in parallel. It not only applies a standard GAN loss
respectively for $X$ and $Y$, but applies forward and backward cycle
consistency losses which ensure that an image $\bm{x}$ from domain $X$ be
translated to an image of domain $Y$, which can then be translated back to the
domain $X$, and vice versa.

DualGAN [28] This method uses a dual-GAN mechanism, which consists of a primal
GAN and a dual GAN. The primal GAN learns to translate images from domain $X$
to domain $Y$, while the dual-GAN learns to invert the task. Images from
either domain can be translated and then reconstructed. Thus a reconstruction
loss can be used to train the model.

UNIT [32] This method consists of two VAE-GANs with a fully shared latent
space. To complete the task of image-to-image translation between $n$ domains,
it needs to be trained $\frac{n\times{(n-1)}}{2}$ times.

DB [33] This method addresses the multi-domain image-to-image translation
problem by introducing $n$ domain-specific encoders/decoders to learn an
universal shared-latent space.

####  2.5.3 Evaluation Metrics

There is a challenge to evaluate the quality of synthesized images [34].
Recent works have tried using pre-trained semantic classifiers to measure the
realism and discriminability of the generated images. The idea is that if the
generated images look to be more close to real ones, classifiers trained on
the real images will be able to classify the synthesized images correctly as
well. Following [35, 19, 36], to evaluate the performance of the models in
classifying generated images quantitatively, we apply the metric
classification accuracy. For each experiment, we generate enough number of
images of different domains, then we use a pre-trained classifier which is
trained on the training dataset to classify them to different domains and
calculate the classification accuracy.

####  2.5.4 Network Architecture and Implementation

The design of the architecture is always a difficult task [25]. To get a
proper model architecture, we adopt the architecture of the discriminator from
[19] which has been proven to be proficient in most image-to-image generation
tasks. It has 6 convolutional layers. We keep the discriminator architecture
fixed and vary the architectures of the encoders and generators. Following the
design of the architectures of the generators in [19], we use two types of
layers, the regular convolutional layers and the basic residual blocks [37].
Since the encoding process is the inverse of the decoding process, we use the
same layers for them but put the layers in the inverse orders. The only
difference is the first layer of the encoder and the last layer of the
generator. We apply $64$ channels (corresponding to different filters) for the
first layer of the encoders, but $3$ channels for the last layer of the
generators since the output images have only $3$ RGB channels. We gradually
change the number of convolutional layers and the number of residual blocks
until we get a satisfying architecture. We don’t apply weight sharing
initially. The performance of different architectures is evaluated on the
Painters dataset and shown in Fig. 2.3. We can see that when the model has 3
regular convolutional layers and 4 basic residual blocks, the model has the
best performance. We keep this architecture fixed for other datasets.

Figure 2.3: The accuracy on varying number of residual blocks and number of
convolutional layers.

We then vary the number of weight-sharing layers in the encoders and the
generators. We change the number of weight-sharing layers from 1 to 4. Sharing
1 layer means sharing the highest layer and the lowest layers in the encoder
pair. Sharing 2 layers means sharing the highest and lowest two layers. The
same sharing method applies for the generator pair (not including the output
layer). The results are shown in table 2.1. We found that sharing 1 layer is
enough to have a good performance.

Table 2.1: Classification accuracy on number of shared layers in encoders and generators. # of shared layers | acc. % (Painters) | acc. % (Alps Seasons)
---|---|---
0 | 49.75 | 29.95
1 | 52.54 | 33.78
2 | 52.81 | 33.54
3 | 51.13 | 33.06

In summary, for the testbed evaluation, we use two encoders each consisting of
3 convolutional layers and 4 basic residual blocks. The generators are
composed with 4 basic residual blocks and 3 fractional-strided convolutional
layers. The discriminators consist of a stack of 6 convolutional layers. We
use LeakyReLU for nonlinearity. The two encoders share the same parameters on
their layers 1 and 7, while the two generators share the same parameters on
layers 1 and 6, which is the lowest-level layer before the output layer. The
details of the networks are given in table 2.2. We evaluate various network
architectures in the evaluation parts. We fix the network architecture as in
Table 2.2.

Table 2.2: Network architecture for the multi-modal unsupervised image-to-image translation experiments. $cxkysz$ denote a Convolution-InstanceNorm-ReLU layer with $x$ filters, kernel size $y$, and stride $z$. $Rm$ denotes a residual block that contains two $3\times 3$ convolutional layers with the same number of filters on both layers. $un$ denotes a $3\times 3$ fractional-strided-Convolution-InstanceNorm-ReLU layer with $n$ filters, and stride $\frac{1}{2}$. $n_{d}$ denotes number of domains. $Y$ and $N$ denote whether the layer is shared or not. Layer | Encoders | Generators | Discriminators
---|---|---|---
1 | $c64k7s1(Y)$ | $R256(Y)$ | $c64k3s2(N)$
2 | $c128k3s2(N)$ | $R256(N)$ | $c128k3s2(N)$
3 | $c256k3s2(N)$ | $R256(N)$ | $c256k3s2(N)$
4 | $R256(N)$ | $R256(N)$ | $c512k3s2(N)$
5 | $R256(N)$ | $u256(N)$ | $c1024k3s2(N)$
6 | $R256(N)$ | $u128(Y)$ | $c(1+n_{d})k2s1(N)$
7 | $R256(Y)$ | $u3(N)$ |

We use ADAM [38] for training, where the training rate is set to 0.0001 and
momentums are set to 0.5 and 0.999. Each mini-batch consists of one image from
domain $X$ and one image from domain $Y$. Our model has several hyper-
parameters. The default values are $\alpha_{0}=10$, $\alpha_{1}=0.1$,
${\alpha}_{2}=0.1$, and ${\alpha}_{3}=10$. The hyper-parameters of the
baselines are set to the suggested values by the authors.

####  2.5.5 Quantitative Results

We evaluate our model on different datasets and compare it with baseline
models.

#####  2.5.5.1 Comparison on Painters Dataset

To compare the proposed model with baseline models Painters dataset, we first
train the state-of-the-art VGG-11 model [39] on training data and get a
classifier of accuracy 94.5%. We then score synthesized images by the
classification accuracy against the domain labels these photos were
synthesized from. We generate around 4000 images for every 5 hours and the
classification accuracies are shown in Fig. 2.4.

Figure 2.4: The classification accuracy on Painters dataset. The 7 models are
the proposed model with the lowest and the highest layer sharing, the lowest
layer sharing only, the highest layer sharing only, CycleGAN, DualGAN, UNIT,
and DB.

We can see that our model achieves the highest classification accuracy of
52.5% when using both the highest layer and lowest layer sharing, with the
training time less than the other reference models in reaching the peak.

#####  2.5.5.2 Comparison on Alps Seasons Dataset

We train VGG-11 model on training data of Alps Seasons dataset and get a
classifier of accuracy 85.5% trained on the training data. We then classify
the generated images by our model and the classification accuracies are shown
in Fig. 2.5.

Figure 2.5: The classification accuracy on Alps Seasons dataset. The 7 models
are the prosed model with lowest and highest level layers sharing, lowest
level layers sharing, highest level layers sharing, CycleGAN, DualGAN, UNIT,
and DB.

Similar to Fig. 2.4, our model achieves the highest classification accuracy of
33.8% with the training time less than the baseline models in reaching the
peak.

####  2.5.6 Analysis of the loss function

We compare the ablations of our full loss. As GAN loss and cycle consistency
loss are critical for the training of unsupervised image-to-image translation,
we keep these two losses as the baseline model and do the ablation experiments
to see the importance of other losses.

Table 2.3: Ablation study: classification accuracy of Painters and Alps Seasons datasets for different losses. The following abbreviations are used: R:reconstruction loss, LCL: latent consistency loss, C: classification loss. Loss | acc.% (Painters) | acc. % (Alps Seasons)
---|---|---
Baseline | 35.23 | 20.81
Baseline + R | 36.86 | 21.59
Baseline + LCL | 44.42 | 25.05
Baseline + C | 43.63 | 24.01
Baseline + R + LCL | 45.79 | 27.19
Baseline + R + C | 44.82 | 26.63
Baseline + LCL + C | 50.74 | 32.51
Baseline + R + LCL + C | 52.54 | 33.78

As shown in Tabel 2.3, the reconstruction loss $R$ is least important with
accuracy improvement of about 4.6% on Painters dataset and 3.7% on Alps
Seasons dataset. The latent consistency loss $LCL$ brings the model an
accuracy improvement of 26.1% on Painters dataset and 20.4% on Alps Seasons
dataset. The accuracy is improved by 23.8% on Painters dataset and 15.4% on
Alps Seasons dataset by the classification loss $C$.

####  2.5.7 Qualitative Results

We demonstrate our model on three unsupervised multi-domain image-to-image
translation tasks.

Painting style transfer (Fig. 2.6) We train our model on Painters dataset and
use it to generate images of size $256\times 256$. The model can transfer the
painting style of a specific painter to the other painters, e.g., transferring
the images of Cezanne to images of other three painters Monet, Ukiyoe and
Vangogh. In Fig. 2.7, we also compare our model with other reference models
when given the same test image.

Figure 2.6: Painters translation results. The original images are displayed
with a dashed square around. The other images are generated according to
different painters. Figure 2.7: Painters translation results. The original
images are displayed with a dashed square around. The other images are
generated according to different painters.

Season transfer (Fig. 2.8) The model is trained on the Alps Seasons dataset.
We use the trained model to generate images of different seasons. For example,
we generate an image of summer from an image of spring and vice versa. In Fig.
2.9, we also compare our model with other reference models when given the same
test image.

Figure 2.8: Alps Seasons translation results. The original images are
displayed with a dashed square around. The other images are generated
according to different seasons. Figure 2.9: Alps Seasons translation results.
The original images are displayed with a dashed square around. The other
images are generated according to different seasons.

Attribute-base face translation (Fig. 2.10) We train the model on CelebA
dataset for attribute-based face translation tasks. We choose 4 attributes,
black hair, blond hair, brown hair, and gender. We then use our model to
generate images with these attributes. For example, we transfer an image with
a man wearing black hair to a man with blond hair, or transfer a man to a
woman.

Figure 2.10: Attribute-base face translation results. The original images are
displayed with a dashed square around. The other images are generated
according to different face attributes.

###  2.6 Conclusion

In this chapter, we propose a Cross-Domain Generative Adversarial Networks
(CD-GAN), a novel and scalable model to conduct unsupervised multi-domain
image-to-image translation. We show its capability of translating images from
one domain to many other domain using several datasets. It still has some
limitations. First, training could be unstable due to the training problem of
GAN model. Second, the diversity of the generated images are constrained by
the cycle consistency loss. We plan to address these two problems in the
future work.

##  Chapter 3 Neural Machine Translation

###  3.1 Introduction

Syntactic information has been shown to improve the translation quality in NMT
models. On the source side, syntax can be incorporated in multiple ways —
either directly during encoding [40, 41, 42], or indirectly via multi-task
learning to produce syntax informed representations [43, 44, 45, 46]. On the
target side, however, incorporating the syntax is more challenging due to the
additional complexity in inference when decoding over latent states. To avoid
this, existing methods resort to approximate inference over the latent states
using a two-step decoding process [47, 48, 49, 50]. Typically, the first stage
decoder produces a beam of latent states, which serve as conditions to feed
into the second stage decoder to obtain the target words. Thus, training and
inference in these models can only explore a limited sub-space of the latent
states.

In this work, we introduce LaSyn, a new target side syntax model that allows
exhaustive exploration of the latent states to ensure a better translation
quality. Similar to prior work, LaSyn approximates the co-dependence between
syntax and semantics of the target sentences by modeling the joint conditional
probability of the target words and the syntactic information at each
position. However, unlike prior work, LaSyn eliminates the sequential
dependence between the latent variables and simply infers the syntactic
information at a given position based on the source text and the partial
translation context. This allows LaSyn to search over a much larger set of
latent state sequences. In terms of time complexity, unlike typical latent
sequential models, LaSyn only introduces an additional term that is linear in
the size of latent variable vocabulary and the length of the sentence.

We implement LaSyn by modifying a transformer-based encoder-decoder model. The
implementation uses a hybrid decoder that predicts two posterior
distributions: the probability of syntactic choices at each position
$P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$, and the probability of the
word choices at each position conditioned on each of the possible values for
the latent states
$P(\mathbf{y}_{n}|\mathbf{z}_{n},\mathbf{x},\mathbf{y}_{<n})$. The model
cannot be trained by directly optimizing the data log-likelihood because of
its non-convex property. We devise a neural expectation maximization (NEM)
algorithm, whose E-step computes the posterior distribution of latent states
under current model parameters, and M-step updates model parameters using
gradients from back-propagation. Given some supervision signal for the latent
variables, we can modify this EM algorithm to obtain a regularized training
procedure. We use parts-of-speech (POS) tag sequences, automatically generated
by an existing tagger, as the source of supervision.

Because the decoder is exposed to more latent states during training, it is
more likely to generate diverse translation candidates. To obtain diverse
sequences, we can decode the most likely translations for different POS tag
sequences. This is a more explicit and effective way of performing diverse
translation than other methods based on diverse or re-ranking beam search [51,
52], or coarse codes planning [53].

We evaluate LaSyn on four translation tasks. Evaluations show that LaSyn
outperforms models that only use partial exploration of the latent states for
incorporating target side syntax. A diversity based evaluation also shows that
when using different POS tag sequences during inference, LaSyn produces more
diverse and meaningful translations compared to existing models.

###  3.2 Related Work

Attention-based Neural Machine Translation (NMT) models have shown promising
results in various large scale translation tasks [54, 55, 56, Vaswani17] using
an Seq2Seq structure. Many Statistical Machine Translation (SMT) approaches
have leveraged benefits from modeling syntactic information [57, 58, 59].
Recent efforts have demonstrated that incorporating syntax can also be useful
in neural methods as well.

One branch uses features on the source side to help improve the translation
performance [41, 60, 42]. Sennrich et al. [41] explored linguistic features
like lemmas, morphological features, POS tags and dependency labels and
concatenate their embeddings with sentence features to improve the translation
quality. In a similar vein, Morishita et al. [60] and Eriguchi et al. [42],
incorporated hierarchical subword features and phrase structure into the
source side representations. Despite the promising improvements, these
approaches are limited in that the trained translation model requires the
availability of external tools during inference – the source text needs to be
processed first to extract syntactic structure [43].

Another branch uses multitask learning, where the encoder of the NMT model is
trained to produce multiple tasks such as POS tagging, named-entity
recognition, syntactic parsing or semantic parsing [43, 44, 45, 46]. These can
be seen as models that implicitly generate syntax informed representations
during encoding. With careful model selection, these methods have demonstrate
some benefits in NMT.

The third branch directly models the syntax of the target sentence during
decoding [47, 48, 49, 50, 61, 62]. Aharoni et al. [50] treated constituency
trees as sequential strings and trained a Seq2Seq model to translate source
sentences into these tree sequences. Wang et al. [48] and Wu et al. [49]
proposed to use two RNNs, a Rule RNN and a Word RNN, to generate a target
sentence and its corresponding tree structure. Gu et al. [47] proposed a model
to translate and parse at the same time. However, apart from the complex tree
structure to model, they all have a similar architecture as shown in Figure
3.1(b), which limits them to only exploring a small portion of the syntactic
space during inference.

LaSyn uses simpler parts-of-speech information in a latent syntax model,
avoiding the typical exponential search complexity in the latent space with a
linear search complexity and is optimized by EM. This allows for better
translation quality as well as diversity. Similar to our work, [63] and [64]
proposed a latent attention mechanism to further reduce the complexity of
model implementation by taking a top-K approximation instead of the EM
algorithm as in LaSyn.

###  3.3 A Latent Syntax Model for Decoding

In a standard sequence-to-sequence model, the decoder directly predicts the
target sequence $\mathbf{y}$ conditioned on the source input $\mathbf{x}$. The
translation probability $P(\mathbf{y}|\mathbf{x})$ is modeled directly using
the probability of each target word $\mathbf{y}_{n}$ at time step $n$
conditioned on the source sequence $\mathbf{x}$ and the current partial target
sequence $\mathbf{y}_{<n}$ as follows:

$P(\mathbf{y}|\mathbf{x};\bm{\theta})=\prod_{n=1}^{N}P(\mathbf{y}_{n}|\mathbf{x},\mathbf{y}_{<n};\bm{\theta})$
(3.1)

where, $\bm{\theta}$ denotes the parameters of both the encoder and the
decoder.

In this work, we model syntactic information of target tokens using an
additional sequence of variables, which captures the syntactic choices111The
variables can be used to model any linguistic information that can be
expressed as choices for each word position (e.g., morphological choices). at
each time step. There are multiple ways of incorporating this additional
information in a sequence-to-sequence model.

(a) Full co-dependence model.

(b) Two-step decoding model.

(c) LaSyn: Our Latent syntax model

Figure 3.1: Target-side Syntax Models: (a) An ideal solution that captures
full co-dependence between syntax and semantics. (b) A widely-used two-step
decoding model [48, 49, 50]. (c) LaSyn, our latent syntax model that uses non-
sequential latent variables for exhaustive search of latent states.

An ideal solution should capture the co-dependence between syntax and
semantics. In a sequential translation process, the word choices at each time
step depend on both the semantics and the syntax of the words generated at the
previous time steps. The same dependence also holds for the syntactic choices
at each time step. Figure 3.1(a) shows a graphical model that captures this
full co-dependence between the syntactic variable sequence
$\mathbf{z}_{1},\dots,\mathbf{z}_{N}$ and the output word sequence
$\mathbf{y}_{1},\dots,\mathbf{y}_{N}$. Such a model can be implemented using
two decoders, one to decode syntax and the other to decode output words. The
main difficulty, however, is that inference in this model is intractable since
it involves marginalizing over the latent $\mathbf{z}$ sequences.

To keep inference tractable, existing approaches treat syntactic choices
$\mathbf{z}$ as observed sequential variables [47, 48, 49, 50], as shown in
Figure 3.1(b). These models use a two-stage decoding process, where for each
time step they first produce most likely latent state $\mathbf{z}_{n}$ and
then use this as input to a second stage that decodes words. However, this
process is unsatisfactory in two respects. First, the inference of syntactic
choices is still approximate as it does not explore the full space of
$\mathbf{z}$. Second, these models are not well-suited for controllable or
diverse translations. Using such a model to decode from an arbitrary
$\mathbf{z}$ sequence is a divergence from its training, where it only learns
to decode from a limited space of $\mathbf{z}$ sequences.

####  3.3.1 Model Description

Our goal is to design a model that allows for exhaustive search over syntactic
choices. We introduce LaSyn, a new latent model shown in Figure 3.1(c). The
syntactic choices are modeled as true latent variables i.e., unobserved
variables. Compared to the ideal model in Figure 3.1(a), LaSyn includes two
simplifications for tractability: (i) The dependence between successive
syntactic choices is modeled indirectly, via the word choices made in the
previous time steps. (ii) The word choice at each position depends only on the
syntactic choice at the current position and the previous predicted words.
Dependence on previous syntactic choices is modeled indirectly.

Under this model, the joint conditional probability of the target word
$\mathbf{y}_{n}$ together with its corresponding latent syntactic choice
$\mathbf{z}_{n}$222Note that $\mathbf{z}_{n}\in V_{\mathbf{z}}$, where
$V_{\mathbf{z}}$ is the vocabulary of latent syntax for the target, which
differs from language to language.  is given by:

$\displaystyle
P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})=P(\mathbf{y}_{n}|\mathbf{z}_{n},\mathbf{x},\mathbf{y}_{<n})\times
P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$ (3.2)

We implement LaSyn by modifying the Transformer-based encoder-decoder
architecture [Vaswani17]. As shown in Figure 3.2, LaSyn consists of a shared
encoder for encoding source sentence $\mathbf{x}$ and a hybrid decoder that
manages the decoding of the latent sequence $\mathbf{z}$ (left branch) and the
target sentence $\mathbf{y}$ (right branch) separately.

The encoder consists of the standard self-attention layer, which generates
representations of each token in the source sentence $\mathbf{x}$. The hybrid
decoder consists of a self-attention layer that encodes the output generated
thus far (i.e., the partial translation), followed by a inter-attention layer
which computes the attention across the encoder and decoder representations.

The decoder’s left branch predicts the latent variable distribution
$P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$ by applying a simple linear
transformation and softmax on the inter-attention output, which contains
information about the encoded input $\mathbf{x}$ and the partial translation
$\mathbf{y}_{<n}$.

The right branch predicts the target word distribution
$P(\mathbf{y}_{n}|\mathbf{z}_{n},\mathbf{x},\mathbf{y}_{<n})$ using the inter-
attention output and the embeddings of all the available choices for
$\mathbf{z}_{n}$. The choices for $\mathbf{z}_{n}$ are represented as
embeddings that can be learned during training. We then combine the inter-
attention output and the latent choice embeddings through an Add operation,
which is a simple composition function that captures all combinations of
additive interactions between the two. The dimension of the inter-attention is
$n\times d_{model}$ and that of the latent embeddings is
$|V_{\mathbf{z}}|\times d_{model}$, where $|V_{\mathbf{z}}|$ is the total
number of choices for $\mathbf{z}_{n}$ or the size of the latent variable
vocabulary. We broadcast them to the same dimension
$n\times|V_{\mathbf{z}}|\times d_{model}$ and then simply add them together
point-wise as shown in Figure 3.2. This is then fed to a linear transform and
softmax over the target word vocabulary.

Figure 3.2: The architecture of LaSyn.

####  3.3.2 Inference with Exhaustive Search for Latent States

When using additional variables to model target side syntax, exact inference
requires marginalizing over these additional variables.

$P(\mathbf{y}|\mathbf{x})=\sum_{\mathbf{z}\in
F(\mathbf{z})}P(\mathbf{y}|\mathbf{z},\mathbf{x})P(\mathbf{z}|\mathbf{x})$
(3.3)

To avoid this exponential complexity, prior works use a two-step decoding
process with models similar to the one shown in Figure 3.1(b). They use greedy
or beam search to explore a subset $B(\mathbf{z})$ of the latent space to
compute the posterior distribution as follows:

$P(\mathbf{y}|\mathbf{x})\simeq\sum_{\mathbf{z}\in
B(\mathbf{z})}P(\mathbf{y}|\mathbf{z},\mathbf{x})P(\mathbf{z}|\mathbf{x})$
(3.4)

Finding the most likely translation using LaSyn also requires marginalizing
over the latent states. However, because the latent states in LaSyn don’t
directly depend on each other, we can exhaustively search over the latent
states. In particular, we can show that when $\mathbf{y}$ is fixed (observed),
the $\\{\mathbf{z}_{n}\\}_{n=1}^{N}$ variables are d-separated [65] i.e., are
mutually independent. As a result, the time complexity for searching latent
sequence $\mathbf{z}$ drops from ${|V_{\mathbf{z}}|}^{N}$ to
$N|V_{\mathbf{z}}|$.

The posterior distribution for the translation probability at a time step $n$
can be computed as follows:

$\displaystyle P$ $\displaystyle(\mathbf{y}_{n}|\mathbf{x},\mathbf{y}_{<n})$
$\displaystyle=\sum_{\mathbf{z}_{n}\in
F(\mathbf{z}_{n})}P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$
(3.5) $\displaystyle=\sum_{\mathbf{z}_{n}\in
F(\mathbf{z}_{n})}P(\mathbf{y}_{n}|\mathbf{z}_{n},\mathbf{x},\mathbf{y}_{<n})\times
P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$

where, $F(\mathbf{z}_{n})$ is the full search space of latent states
$\mathbf{z}_{n}$ and the joint probability is factorized as specified in
Equation 3.2.

For decoding words, we use standard beam search333Note our primary goal is to
perform exhaustive search in the latent space. Search in the target vocabulary
space remains exponential in our model. with
$P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$ as the beam
cost. With this inference scheme, we can easily control decoding for
diversity, by feeding different $\mathbf{z}$ sequences to the right branch of
the decoder and decode diverse $\mathbf{y}_{n}$ by directly using
$P(\mathbf{y}_{n}|\mathbf{z}_{n},\mathbf{x},\mathbf{y}_{<n})$ as the beam
cost. Unlike the two-step decoding models which only evaluate a small number
of $\mathbf{z}_{n}$ values at each time step (constrained by beam size), LaSyn
evaluates all possible values for $\mathbf{z}_{n}$ at each time step, while
avoiding the evaluation of all possible sequences.

####  3.3.3 Training with Neural Expectation Maximization

The log-likelihood of LaSyn’s parameters $\bm{\theta}$444This includes the
trainable parameters of the encoder, decoder, and the latent state embeddings.
computed on one training pair $\langle\mathbf{x},\mathbf{y}\rangle\in D$ is
given by:

$\displaystyle\mathcal{L}(\mathbf{\bm{\theta}})$ $\displaystyle=\log
P(\mathbf{y}|\mathbf{x};\mathbf{\bm{\theta}})$ (3.6)
$\displaystyle=\sum_{n=1}^{N}\log
P(\mathbf{y}_{n}|\mathbf{x},\mathbf{y}_{<n};\mathbf{\bm{\theta}})$
$\displaystyle=\sum_{n=1}^{N}\log\sum_{\mathbf{z}_{n}\in
V_{\mathbf{z}}}P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$

Directly optimizing the log-likelihood function (equation 5.1) with respect to
model parameter $\bm{\theta}$ is challenging because of the highly non-convex
function $P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$ and the
marginalization over $\mathbf{z}_{n}$.555Note that marginalization is an issue
during training, unlike in inference. As $P(y_{n},z_{n})$ is already an non-
convex function with respect to $\theta$, summing $P(y_{n},z_{n})$ over
different values of $z_{n}$ makes the function more complicated. Besides, we
also need to compute gradients to update the parameters and computing the
gradient of a log-of-sum function is costly and unstable. During the
translation, we only need to compute the value of $\mathcal{L}(\theta)$ as
score for beam searching. Therefore, the marginalization is not an issue.
Alternatively, we optimize the system parameters by Expectation Maximization
(EM).

Using Jensen’s inequality, equation (5.1) can be re-written as:

$\displaystyle\mathcal{L}(\mathbf{\bm{\theta}})$
$\displaystyle=\sum_{n=1}^{N}\log\sum_{\mathbf{z}_{n}\in
V_{\mathbf{z}}}Q(\mathbf{z}_{n})\frac{P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})}{Q(\mathbf{z}_{n})}$
(3.7) $\displaystyle\geq$ $\displaystyle\sum_{n=1}^{N}\sum_{\mathbf{z}_{n}\in
V_{\mathbf{z}}}Q(\mathbf{z}_{n})\log\frac{P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})}{Q(\mathbf{z}_{n})}$
$\displaystyle=$ $\displaystyle\mathcal{L}_{lower}(Q,\bm{\theta})$

where $\mathcal{L}_{lower}(Q,\bm{\theta})$ is the lower bound of the log-
likelihood and $Q$ is any auxiliary probability distribution defined on
$\mathbf{z}_{n}$. $\bm{\theta}$ is omitted from the expression for simplicity.

We set $Q(\mathbf{z}_{n})=P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{\leq
n};\bm{\theta}^{old})$, the probability of the latent state computed by the
decoder (shown in the left branch in Figure 3.2). Substituting this in
equation (3.7), we obtain the lower bound as

$\displaystyle\mathcal{L}_{lower}(Q,\bm{\theta})=$ (3.8)
$\displaystyle\sum_{n=1}^{N}\sum_{\mathbf{z}_{n}\in
V_{\mathbf{z}}}P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{\leq
n};\bm{\theta}^{old})\times\log
P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n};\bm{\theta})$
$\displaystyle-P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{\leq
n};\bm{\theta}^{old})\times\log P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{\leq
n};\bm{\theta}^{old})$
$\displaystyle=\mathcal{Q}(\bm{\theta},\bm{\theta}^{old})+C$

where

$\displaystyle\mathcal{Q}(\bm{\theta},\bm{\theta}^{old})=$
$\displaystyle\sum_{n=1}^{N}\sum_{\mathbf{z}_{n}\in
V_{\mathbf{z}}}P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{\leq
n};\bm{\theta}^{old})$ (3.9) $\displaystyle\times\log
P(\mathbf{y}_{n},\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n};\bm{\theta})$

EM algorithm for optimizing $\mathcal{Q}(\bm{\theta},\bm{\theta}^{old})$
consists of two major steps. In the E-step, we compute the posterior
distribution of $\mathbf{z}_{n}$ with respect to $\bm{\theta}^{old}$ by

$\displaystyle\gamma$
$\displaystyle(\mathbf{z}_{n}=i)=P(\mathbf{z}_{n}=i|\mathbf{x},\mathbf{y}_{\leq
n})$ (3.10)
$\displaystyle=\frac{P(\mathbf{y}_{n},\mathbf{z}_{n}=i|\mathbf{x},\mathbf{y}_{<n})}{\sum_{\mathbf{z}_{n}=j}P(\mathbf{y}_{n},\mathbf{z}_{n}=j|\mathbf{x},\mathbf{y}_{<n})}$

where $\gamma(\mathbf{z}_{n}=i)$ is the responsibility of $\mathbf{z}_{n}=i$
given $\mathbf{y}_{n}$, and can be calculated by equation (3.2).

In the M-step, we aim to find the configuration of $\bm{\theta}$ that would
maximize the expected log-likelihood using the posteriors computed in the
E-step. In conventional EM algorithm for shallow probabilistic graphical
model, the M-step is generally supposed to have closed-form solution. However,
we model the probabilistic dependencies by deep neural networks, where
$\mathcal{Q}(\bm{\theta},\bm{\theta}^{old})$ is highly non-convex and non-
linear with respect to network parameters $\bm{\theta}$. Therefore, there
exists no analytical solution to maximize it. However, since deep neural
network is differentiable, we can update $\bm{\theta}$ by taking a gradient
ascent step:

$\displaystyle\bm{\theta}^{new}=\bm{\theta}^{old}+\eta\frac{\partial\mathcal{Q}(\bm{\theta},\bm{\theta}^{old})}{\partial\bm{\theta}},$
(3.11)

The resulting algorithm belongs to the class of generalized EM algorithms and
is guaranteed (for a sufficiently small learning rate $\eta$) to converge to a
(local) optimum of the data log likelihood [66].

####  3.3.4 Regularized EM training

The EM training we derived does not assume any supervision for the latent
variables $\mathbf{z}$. This can be seen as inferring the latent syntax of the
target sentences by clustering the target side tokens into $|V_{\mathbf{z}}|$
different categories. Given some token-level syntactic information, we can
modify the training procedure to regularize the generation of latent sequence
$P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$ such that true latent sequences
have higher probabilities under the model. In this work, we consider parts-of-
speech sequences of the target sentences for regularization.

The regularized EM training objective is thus redefined as

$\displaystyle\mathcal{L}_{total}(\mathbf{\bm{\theta}})=\mathcal{L}_{lower}(\bm{\theta})+\lambda\mathcal{L}_{\mathbf{z}}(\bm{\theta}),$
(3.12)

where $\mathcal{L}_{lower}(\bm{\theta})$ is the EM lower bound in equation
(3.7) and $\mathcal{L}_{\mathbf{z}}(\bm{\theta})$ denotes cross entropy loss
between $P(\mathbf{z}_{n}|\mathbf{x},\mathbf{y}_{<n})$ and the true POS tag
sequences and $\lambda$ is a hyper-parameter that controls the impact of the
regularization.

This regularized training algorithm is shown in Algorithm 2.

Algorithm 2 Training NMT with latent POS tag sequences through Regularized
Neural EM

Objective: Maximize the log likelihood function
$\mathcal{Q}(\bm{\theta},\bm{\theta}^{old})$ with respect to
$\mathbf{\bm{\theta}}$ over observed variable $\mathbf{y}$ and latent variable
$\mathbf{z}$, governed by parameters $\bm{\theta}$.

Input: Parallel corpus training data $\langle\mathbf{x},\mathbf{y}\rangle$;
POS tag sequence $\mathbf{z}$ of target sentence; the number of EM update
steps per batch $K$.

Initialize: Initialize random values for the parameters $\bm{\theta}^{old}$.

while Training loss has not converged do

Select $\langle\mathbf{x},\mathbf{y}\rangle\in D$, parse $\mathbf{z}$ of
$\mathbf{y}$.

for $k\leftarrow 1$ to $K$ do

1\. E-step: Evaluate $\gamma(\mathbf{z}_{n}=i)$ for $i\leq|V_{\mathbf{z}}|$.

2\. M-step: Evaluate $\bm{\theta}^{new}$ given by equation (3.11).

3. Let $\bm{\theta}^{old}\leftarrow\bm{\theta}^{new}$.

end for

end while

###  3.4 Evaluation

We evaluate LaSyn on four translation tasks, including three with moderate
sized datasets IWSLT 2014 [67] German-English (De-En), English-German (En-De),
English-French (En-Fr), and one with a relatively larger dataset, the WMT 2014
English-German (En-De). We describe the datasets in more details in the
appendix.

We compare against three types of baselines: (i) general Seq2Seq models that
use no syntactic information, (ii) models that incorporate source side syntax
directly, and multitask learning models which include syntax indirectly, and
(iii) models that use syntax on the target side. We also define a LaSyn
Empirical Upper Bound (EUB), which is our proposed model using true POS tag
sequences for inference.

We use BLEU as the evaluation metric [68] for translation quality. For diverse
translation evaluation, we utilize distinct-1 score [69] as the evaluation
metric, which is the number of distinct unigrams divided by total number of
generated words.

For all translation tasks, we choose the base configuration of Transformer
with $d_{model}=512$. During training, we choose Adam optimizer [70] with
$\beta_{1}=0.9$, $\beta_{2}=0.98$ with initial learning rate is 0.0002 with
4000 warm-up steps. We describe additional implementation and training details
in the Appendix.

####  3.4.1 Results on IWSLT’14 Tasks

Table 3.1 compares LaSyn versions against some of the state-of-the-art models
on the IWSLT’14 dataset. LaSyn-K rows show results when varying the number of
EM update steps per batch ($K$).

On the De-En task, LaSyn provides a 1.7 points improvements over the
Transformer baseline, demonstrating that the LaSyn’s improvements come from
incorporating target side syntax effectively. This result is also better than
a transformer-based source side syntax model by 1.5 points. LaSyn results are
also better than the published numbers for LSTM-based models that use multi-
task learning for source side and models that uses target side syntax. Note
that since the underlying architectures are different, we only present these
results to show that the results with LaSyn are comparable with other models
that have incorporated syntax.

On the En-De task, our model achieves 29.2 in terms of BLEU score, with 2.6
points improvement over the Transformer baseline and 2.4 points improvement
over Transformer-based Source Side Syntax model. Compared with NPMT [71],
which is a BiLSTM based model, we achieve 3.84 point improvement.

On the En-Fr task, our model set a new state-of-the-art with a BLEU score of
40.6, which is 1.7 points improvement over the second best model which uses
Transformer to incorporate source side syntax knowledge. Our model also
surpasses the basic Transformer model by about 2.1 points.

We notice that across all tasks, the performance of our model improves with
number of EM update steps per batch ($K$). With larger $K$ values, we get
better lower bounds $\mathcal{L}_{lower}(\bm{\theta})$ on each training batch,
thus leading to better optimization. For update steps beyond $K>5$, the
performance does not improve any more.

Last, the EUB row indicates the performance that can be obtained when feeding
in the true POS tags. The large improvement here shows the potential for
improvement when modeling target side syntax.

Method Type | Model | BLEU
---|---|---
De-En | En-De | En-Fr
BiLSTM |  BiLSTM [72] | – | – | 34.8
Dual Learning [73] | 32.35 | – | –
AST [74] | – | – | 38.03
NPMT [71] | – | 25.36 | –
Multi-Task (BiLSTM) |  MTL-NMT [45] | 27.78 | – | –
Source Side Syn. (Transformer) |  Source-NMT [41] | 33.5 | 26.8 | 38.9
Target Side Syn. (BiLSTM) |  DSP-NMT [53] | 29.78 | – | –
Tree-decoder [48] | 32.65 | – | –
Transformer | Transformer | 33.3 | 26.6 | 38.5
LaSyn (Unsupervised) | 30.8 | 25.2 | 34.5
LaSyn (K=1) | 34.63 | 28.1 | 39.7
LaSyn (K=3) | 34.91 | 28.9 | 40.4
LaSyn (K=5) | 35.0 | 29.2 | 40.6
LaSyn EUB | 51.4 | 47.3 | 54.2
Table 3.1: IWSLT’14 English-German and English-French results \- shown are the
BLEU scores of various models on TED talks translation tasks. We highlight the
best model in bold.

Table 3.2 shows one example where LaSyn produces correct translations for a
long input sentence. The output of LaSyn is close to the reference and the
output of LaSyn when given the gold POS tag sequence is even better,
demonstrating the benefits of modeling syntax. The transformer model however
fails to decode the later portions of the long input accurately.

SRC |  letztes jahr habe ich diese beiden folien gezeigt , um zu veranschaulichen , dass die arktische eiskappe , die für annähernd drei millionen jahre die grösse der unteren 48 staaten hatte , um 40 prozent geschrumpft ist .
---|---
REF |  last year i showed these two slides so that demonstrate that the arctic ice cap , which for most of the last three million years has been the size of the lower 48 states , has shrunk by 40 percent .
Transformer |  last year , i showed these two slides to illustrate that the arctic ice caps that had the size of the lower 48 million states to 40 percent .
LaSyn |  last year , i showed these two slides to illustrate that the arctic ice cap , which for nearly three million years had the size of the lower 48 states , was shrunk by 40 percent .
LaSyn (groundtrue POS) |  last year i showed these two slides just to illustrate that the arctic ice cap , which for nearly about the last three million years has been the size of the lower 48 states , has shrunk by 40 percent .
Table 3.2: Translation examples on IWSLT’14 De-En dataset from our model and
the Transformer baseline. We put correct translation segment in blue and
highlight the wrong one in red.

####  3.4.2 Speed

We compare the speeds of our (un-optimized) implementation of LaSyn with a
vanilla transformer with no latents in its decoder. Table 3.3 shows the
training time per epoch, and the inference time for the whole test set.
computed on the IWSLT’14 De-En task. When $K=1$, LaSyn takes almost twice as
much time as the vanilla transformer for training. Increasing $K$ increases
training time further. For inference, LaSyn takes close to four times as much
time compared to the vanilla Transformer. In terms of complexity, LaSyn only
adds a linear term (in POS tag size to the decoding complexity. Specifically,
its decoding complexity is $B\times O(m)\times O(|V_{\mathbf{z}}|\times N)$
where $B$ is beam size, $m$ is a constant proportional to the tag set size and
$N$ is output size. As the table shows, empirically, our current
implementation incurs $m\simeq 4$. We leave further optimizations for future
work.

Model | Training Time/Epoch$\downarrow$ | Inference Time$\downarrow$
---|---|---
Transformer |  3.6 min |  12.8 s
LaSyn (K=1) | 6.3 min | 56.1 s
LaSyn (K=3) | 18.1 min | 55.6 s
LaSyn (K=5) | 28.0 min | 55.6 s
Table 3.3: IWSLT’14 De-En training and inference speed evaluation.
$\downarrow$ means the smaller the better. We highlight the best model in
bold.

####  3.4.3 Diversity

We compare the diversity of translations using distinct-1 score [69], which is
simply the number of distinct unigrams divided by total number of generated
words. We use our model to generate 10 translations for each source sentence
of the test dataset. We then compare our results with baseline Transformer.
The result is shown in Table 3.4. Much like translation quality, LaSyn’s
diversity increases with number of EM updates and is better than diversity of
the transformer and a source side encoder model.

Model | distinct-1
---|---
De-En | En-De | En-Fr
Transformer | 0.231 | 0.242 | 0.258
Source-NMT | 0.232 | 0.244 | 0.260
LaSyn (Unsupervised) | 0.228 | 0.231 | 0.239
LaSyn (K=1) | 0.237 | 0.251 | 0.265
LaSyn (K=3) | 0.241 | 0.253 | 0.270
LaSyn (K=5) | 0.245 | 0.255 | 0.273
LaSyn EUB | 0.328 | 0.516 | 0.354
Table 3.4: IWSLT’14 En-De/De-En/En-Fr diversity translation evaluation. We
highlight the best model in bold.

#####  3.4.3.1 Controlling Diversity with POS Sequences

One of the main strengths of LaSyn is that it can generate translations
conditioned on a given POS sequence. First, we present some examples that we
generate by decoding over different POS tag sequences. Given a source
sentence, we use LaSyn to provide the most-likely target pos tag sequence.
Then, we obtain a random set of valid POS tag sequences that differ from this
maximum likely sequence by some edit distance. For each of these randomly
sampled POS tag sequences, we let LaSyn generate a translation that fits the
sequence. Table 3.5 shows some example sentences. LaSyn is able to generate
diverse translations that reflect the sentence structure implied by the input
POS tags. However, in trying to fit the translation into the specified
sequence, it deviates somewhat from the ideal translation.

Figure 3.3: Diversity vs. Translation Quality: BLEU and distinct-1 scores for
targets decoded using POS sequences of increasing edit distance.

To understand how diversity plays against translation quality, we also conduct
a small scale quantitative evaluation. We pick a subset of the test dataset,
and for each source sentence in this subset, we sample $10$ POS tag sequences
whose edit distance to their corresponding Top-1 POS tag sequence equal to a
specific value, we then use them to decode $W$ translations. We calculate
their final BLEU and distinct-1 scores. The results are shown in Figure: 3.3.
As the edit distance increases, diversity increases dramatically but at the
cost of translation quality. Since the POS tag sequence acts as a template for
generation, as we move further from the most likely template, the model
struggles to fit the content accurately. Understanding this trade-off can be
useful for re-ranking or other scoring functions.

SRC |  und natürlich auch , wie nimmt gestaltung einfluss auf die wahrnehmung .
---|---
REF |  and of course how design affects perception .
$0$ |  cc in nn wrb nn vbz vbz nn .
|  and of course how design is affecting perception .
$20$ |  ls rb vb in dt nn vbz jj jj jj .
|  i also think that the design is affecting perception .
$30$ |  rb , dt nn nn dt nn in prp rb .
|  also , the way design adds influence in perception too .
$30$ |  prp vbz in prp vb vbn rb in nn .
|  it turns out it included design impact on perception .
Table 3.5: Examples of translations decoded from specified POS sequences with
different edit distances (shown as values in first column). SRC: source
sentence. REF: reference sentence.

####  3.4.4 Results on WMT’14 En-De

To assess the impact on a larger dataset, we show results on the WMT’14
English-German in table 3.6. Compared to the previously reported systems, we
see that our transformer implementation is a strong baseline. LaSyn produces
small gains, with the best gain at K=5 – a BLEU score improvement of 0.6. This
demonstrates that syntactic information can contribute more to the increase of
translation quality on a smaller dataset.

Model | BLEU
---|---
BiRNN+GCN [61] | 23.9
ConvS2S [75] | 25.16
MoE [76] | 26.03
Transformer (base) | 27.3
LaSyn (K=1) (base) | 27.6
LaSyn (K=3) (base) | 27.8
LaSyn (K=5) (base) | 27.9
Table 3.6: WMT’14 English-German results \- shown are the BLEU scores of
various models on TED talks translation tasks. We highlight the best model in
bold.

###  3.5 Conclusion

Modeling target-side syntax through true latent variables is difficult because
of the additional inference complexity. In this work, we presented LaSyn, a
latent syntax model that allows for efficient exploration of a large space of
latent sequences. This yields significant gains on four translation tasks,
IWSLT’14 English-German, German-English, English-French and WMT’14 English-
German. The model also allows for better decoding of diverse translation
candidates. This work only explored parts-of-speech sequences for syntax.
Further extensions are needed to tackle tree-structured syntax information.

##  Chapter 4 Generic Image Captioning

###  4.1 Introduction

Research on image captioning to generate textual descriptions of images has
made a great progress in recent years thanks to the introduction of encoder-
decoder architectures [77, 78, 79, 80, 81, 82, 83, 84]. Existing models are
generally trained and evaluated on datasets created for image captioning like
COCO [85, 86] and Flickr [87] that only contain generic object categories but
not pair-wise relations of the objects in the image.

To equip the captioning model with relation information, some more recent
studies resort to the scene graph generation [88, 89] to provide the graph
representations of real-world images with the semantic summaries of objects
and their pair-wise relationships. For example, the graph in Figure 4.1
encodes the key objects in the image such as people (‘man’), their possessions
(‘hair’ and ‘shirt’, both possessed by the man), and their activities (the man
is ‘holding’ a ‘racket’). The graph representation has been applied to improve
the image related tasks that involve natural language [90, 91]. When it comes
to the task of image captioning, recent studies [92, 93, 94] propose to first
use a scene graph generation model well-trained on Visual Genome [95] dataset
to predict the pair-wise relationships existing in the COCO image and then use
a Graph Convolutional Net (GCN) to encode the relation information. Typically,
the object region features and the relation representations are then merged
together via concatenation or convolution to feed into a decoder for
generating a sentence using the Maximum Likelihood Estimation (MLE). These
methods typically suffer from at least one of three main weaknesses: (i) There
are mis-alignments between the image objects and the relation labels, because
the regions containing the objects do not correspond to those used to predict
the relations; (ii) Given that the goal of using a GCN is to extract the
relation information, the training of model for GCN is less effective by only
using the objective to optimize the captioning without considering the object
relationship; (iii) The encoder itself cannot extract the relations between
objects but relying on other pre-trained models to do it, which makes the
captioning less explainable. As another observation, recent studies [96, 97,
98, 99, 100] have pointed out that good metric scores can be achieved with a
strong decoder, without the need of underlying encoder to truly understand the
visual content. Thus, it becomes less likely to determine if the models are
really learning some important relationships through the encoder or they just
follow some language rules by the decoder. To be more concrete, for a
generated sentence like ‘a man is riding a bike’, can the model really tell
the difference among ‘riding’, ‘rolling’ or ‘on’ or it just follows some
language expression rules (i.e., ‘riding’ is more commonly used than ‘rolling’
and ‘on’)?

Regularizing the encoder with a relation-centric objective is essential since
it can not only guide the encoder to learn representations with relation
information embedded, but also explicitly express the pair-wise relationships
and explain the generation of some relational words. In this paper, we propose
JoGANIC: the RElational transFORMER that learns a scene graph to express the
object relationships in the process of decoding a sentence description.
JoGANIC incorporates both image captioning and scene graph generation
components via a novel transformer encoder. Different from conventional
Transformer [101] that only uses the image captioning as the final objective
to train both the encoder and the decoder, JoGANIC uses a scene graph
generation objective to guide the encoder to learn better relational
representations. Since the image captioning and scene graph generation are two
distinct tasks, directly using the Multi-Task Learning paradigm is non-
trivial. We propose a sequential training algorithm that guides the JoGANIC to
learn both tasks step by step.

Our work has three main contributions. (i) We propose to generate scene graphs
as a way to enrich the captions that they together can better describe the
images; (ii) We design a novel relational Transformer (JoGANIC) that can
better learn the image features for captioning with the relationships embedded
via an auxiliary scene graph generation task; (iii) We propose a sequential
training algorithm that guides the JoGANIC to accomplish both tasks in three
consecutive steps. Experimental results show that JoGANIC can achieve better
performance than state-of-the-art methods on both image caption generation and
scene graph generation. We will release the source code.

###  4.2 Background and Related Work

In this section, we first introduce the background knowledge of scene graph
generation, and then discuss the related work on image captioning and the
application of scene graphs in image captioning.

####  4.2.1 Scene Graph Generation

Figure 4.1: A scene graph containing entities, such as man, hair or racket,
that are localized in the image with bounding boxes and the relationships
between those entities, such as with, wearing and holding.

A scene graph, $G$, as shown in Figure 4.1, is a structural representation of
the semantic contents in an image [95]. It consists of:

  * •

a set of bounding boxes $B=\\{b_{1},\ldots,b_{m}\\}$,
$b_{i}=(x_{i1},y_{i1},x_{i2},y_{i2})$111$x_{i1},y_{i1}$ are the top-left
coordinates of $b_{i}$, while $x_{i2},y_{i2}$ are the bottom-right
coordinates..

  * •

a corresponding set of objects $O=\\{o_{1},\ldots,o_{m}\\}$, where
$o_{i}\in\mathcal{C}$ is the class label assigned to the bounding box $b_{i}$
and $\mathcal{C}$ is the set containing all label categories.

  * •

a set of pair-wise relationships $R=\\{\ldots,r_{i\rightarrow j},\ldots\\}$,
with $r_{i\rightarrow j}\in\mathcal{R}$ representing the relationship between
a start node $(b_{i},o_{i})$ and an end node $(b_{j},o_{j})$. $\mathcal{R}$ is
the set of relation types, including the ‘background’ predicate, which
indicates that there is no edge between the specified objects.

Scene graph [102, 103, 89] is often generated with a few procedures: object
detection (detecting $b_{i}$), classification (classifying $o_{i}$) and
predicate (relation label) prediction to determine $r_{i\rightarrow j}$ given
$(b_{i},o_{i})$ and $(b_{j},o_{j})$. Most of the methods on scene graph
generation have been developed on the Visual Genome [95] dataset, which
provides annotated scene graphs for $100$K images, consisting of over $1$M
instances of objects and $600$K relations. Since only a small portion of
images in this data set also exist in the COCO captioning dataset [85],
directly using a multi-task learning scheme on both tasks is challenging.

####  4.2.2 Image Captioning

State-of-the-art approaches [77, 104, 79, 105, 106, 83, 84] mainly use
encoder-decoder frameworks with attention to generate captions for images. Xu
et al. [83] developed soft and hard attention mechanisms to focus on different
regions in the image when generating different words. Similarly, Anderson et
al. [77] used a Faster R-CNN [107] to extract regions of interest that can be
attended to. Yang et al. [84] used self-critical sequence training for image
captioning.

Various Transformer-based [101] models have achieved promising success on the
image captioning task [108, 104, 109, 110]. Cornia et al. [108] proposed a
meshed-memory transformer that learns a multi-level representation of the
image regions, and uses a mesh-like connectivity at decoding stage to exploit
low- and high-level features. Li et al. [110] introduced the entangled
attention that enables the Transformer to exploit semantic and visual
information simultaneously. He et al. [104] introduced the image transformer,
which consists of a modified encoding transformer and an implicit decoding
transformer to adapt to the structure of images. Herdade et al. [109]
introduced the object transformer, that explicitly incorporates information
about the spatial relationship between detected objects through geometric
attention.

Some research studies have been using scene graphs for image captioning [111,
92, 94, 112]. Zhong et al. [112] proposes a scene graph decomposition method
that decomposes a scene graph into a set of sub-graphs, with each sub-graph
capturing a semantic component of the input image. By selecting important sub-
graphs, different target sentences are decoded. Wang et al. [92] uses two
encoders, one is a ResNet image encoder, the other is a GCN encoder for the
relation labels of the objects. The two encoders are then attached with an
LSTM decoder and combined together using attention. Yao et al. [94] proposes
to use two GCNs to encode the spatial and semantic relations in an image. Then
the features from the two encoders are merged together via attention to get
the final feature. Guo et al. [111] proposed to explicitly model the object
interactions in semantics and geometry based on Graph Convolutional Networks
(GCNs). Yang et al. [93] proposed the scene graph auto-encoding technique to
learn a dictionary that helps to encode the desired language prior, which
guides the encoding-decoding pipeline.

JoGANIC differs from the previous methods in two aspects: (i) JoGANIC can not
only generate a caption but also a scene graph to capture the relationships
between the objects without using an external scene graph generator. (ii)
JoGANIC integrates the scene graph generation with image captioning using a
sequential training algorithm to better learn the relational image features
step by step.

###  4.3 Design of JoGANIC

In this section, we first discuss the problems of existing schemes and propose
the basic architecture to construct the JoGANIC that combines the scene graph
generation with image captioning to learn relational features of images. We
then present the scene graph generation task which is used to first pre-train
the encoder of JoGANIC and later used as an auxiliary objective to generate
captions. Finally, we describe the image captioning task as well as the
training algorithm to train on both tasks.

####  4.3.1 A Relational Encoding Learning Idea

In a standard image captioning model based on encoder-decoder structure, the
decoder directly predicts the target sequence $\mathbf{y}$ conditioned on the
source input $\mathbf{x}$. The captioning probability
$P(\mathbf{y}|\mathbf{x})$ is modeled directly using the probability of each
target word $\mathbf{y}_{i}$ at the time step $i$ conditioned on the source
input sequence $\mathbf{x}$ and the current partial target sequence
$\mathbf{y}_{1:i-1}$ as follows:

$\small
P(\mathbf{y}|\mathbf{x};\bm{\theta})=\prod_{i=1}^{N}P(\mathbf{y}_{i}|\mathbf{x},\mathbf{y}_{1:i-1};\bm{\theta})$
(4.1)

where $\bm{\theta}$ denotes the parameters of the model222Through-out this
paper, we omit $\bm{\theta}$ for simplicity..

In general, $\mathbf{x}$ are image features that can be obtained by feeding an
image $X$ to a pre-trained CNN (e.g. ResNet) encoder $\mathbf{x}=CNN(X)$. To
integrate relational information into the image features, some state-of-the-
art methods first use a well-trained scene graph generation model to extract a
graph $g$ from the same image and then use a Graph Convolutional Net (GCN) to
encode $g$ to vectors, shown in Figure 4.2(a). Then the image features and the
relational features are merged together $\mathbf{x}=[CNN(X);GCN(g)]$, where
$[\cdot;\cdot]$ can be concatenation, attention or convolution. This
straightforward way of integrating relational information may suffer from a
few problems. First, training a GCN with the objective of getting the image
caption (Maximum Likelihood Estimation, i.e., MLE) might be less effective.
The goal of using a GCN is to capture the relational information existing in
the image, while the likelihood function used to estimate the probability
distribution is not directly relevant to the relation of objects in an image.
Second, if the model no longer has access to the pre-trained scene graph
generator (e.g., using a different dataset), the caption generation might not
be feasible any more. Third, recent studies show that good metric scores can
be obtained with a strong decoder, without the underlying encoder to truly
understand the visual content. This indicates that the caption objective alone
cannot effectively guide the training of the encoder to accurately extract the
relationships between the objects.

(a) Two-encoder captioning model.

(b) JoGANIC: Our Relational Transformer model.

Figure 4.2: Models trying to encode relation information: (a) A widely-used
two-encoder captioning model that one encoder is a GCN used to encode the
scene graphs into relational features and the other is a pre-trained CNN (i.e.
ResNet) for images. (b) JoGANIC, our Relational Transformer that generates a
scene graph and a corresponding caption.

The above analysis motivates us to design a model that can generate scene
graphs together with the learning of image caption using the same dataset
without the need of an external scene graph generator, but with the objective
of producing the good scene graph. In this way, we can also better train the
encoder to well understand the visual contents without being misled by the
good results from a stronger decoder. We propose a second objective $P(g|X)$
for good scene graph generation and apply it to the encoder learning. We use a
Transformer model where the encoder is used to generate scene graphs and the
decoder is applied to generate captions. By incorporating this ‘RElational’
objective, our transFORMER can also better learn the image features with
relation information embedded. The simplified model structure of JoGANIC is
shown in Figure 4.2(b).

Figure 4.3: The architecture of JoGANIC. JoGANIC consists of a (a) Faster
R-CNN object detection model to provide $m$ bounding boxes, object labels and
object region features, a (b) Transformer encoder to generate $m\times(m-1)$
pair-wise relations for the objects in the image and a (c) Transformer decoder
for generating captions. The generated scene graph is shown in (d).

####  4.3.2 Scene Graph Generation

We first lay out the mathematical formulation of the scene graph generation
problem. As introduced in Section 4.2.1, for an image of $m$ objects, its
visually grounded scene graph consists of tuples $\big{(}r_{i\rightarrow
j},(b_{i},o_{i}),(b_{j},o_{j})\big{)}_{i,j=1,i\neq j}^{m}$333$i\in{1\ldots
m}$, $j\in{1\ldots m}$, but $i\neq j$., with $b_{i}$ being the bounding box of
the object $i$, $o_{i}$ the label and $r_{i\rightarrow j}$ the predicate
(relation label) between objects $i$ and $j$. Thus, the scene graph generation
objective can be derived as follows:

$\displaystyle P(g|X)=$ $\displaystyle\prod_{(i,j)}^{m}P(r_{i\rightarrow
j}|(b_{i},o_{i}),(b_{j},o_{j}),X)$ (4.2)
$\displaystyle\times\prod_{k}^{m}P(o_{k}|b_{k},X)P(b_{k}|X)$

To simplify the learning process and improve the effectiveness of training, we
conveniently use Faster-RCNN to model $P(o_{k}|b_{k},X)$ and $P(b_{k}|X)$
together. The negative log-likelihood of the model’s parameters computed on
the training data is then given by:

$\displaystyle\mathcal{L}_{g}$ $\displaystyle=-\log
P(g|X)=-\sum_{(i,j)}^{m}\log P(r_{i\rightarrow
j}|(b_{i},o_{i}),(b_{j},o_{j}),X)$ (4.3) $\displaystyle-\sum_{k}^{m}[\log
P(b_{k}|X)+\log P(o_{k}|b_{k},X)]$

We model $P(b_{k}|X)$ and $P(o_{k}|b_{k},X)$ with object detector Faster R-CNN
[107] to automatically generate a set of bounding boxes $b_{k}$ and the
corresponding object labels $o_{k}$ from an image $X$ (Figure 5.3(a)). In
practice, training a Faster R-CNN is highly non-trivial. To simplify the
training of our proposed JoGANIC, we first train a Faster R-CNN till its
convergence. Then with its parameters learnt, we train the model
$P(r_{i\rightarrow j}|(b_{i},o_{i}),(b_{j},o_{j}),X)$ (Figure 5.3(b)) with
$(b_{i},o_{i})$, $(b_{j},o_{j})$ and $X$ as the input. In theory, if the
groundtruth $b_{k}$ and $o_{k}$ are provided, we can eliminate Faster R-CNN.
To feed in $b_{k}$, $o_{k}$ and $X$, we use a Pre-Processing Layer (Section
4.3.3.1) which converts the three inputs into vectors and then concatenate
them together to get the final input features.

Modeling $P(r_{i\rightarrow j}|(b_{i},o_{i}),(b_{j},o_{j}),X)$ is actually a
predicate classification task. We use the JoGANIC encoder (Section 4.3.3.2) to
first encode the input into high-level features and then apply a Post-
Processing Layer (Section 4.3.3.3) to get the final output for classification.
The features right before the Post-Procesing layer contain useful relational
information and are used as the input to the decoder for caption generation
(Section 4.3.4). We talk about more details about the model architecture in
Section 4.3.3.

####  4.3.3 Encoder Architecture

The encoder takes as input a set of $m$ image region tuples,
$(b_{i},o_{i},\mathbf{v}_{i})_{i=1}^{m}$, where $\mathbf{v}_{i}$ is defined as
the mean-pooled convolutional feature from region $i$ with dimension $2048$
and the number of region varies for different images. It consists of three
main components which process the input consecutively: (i) a Pre-Processing
Layer that takes the bounding boxes, object labels and image region features
as input and linearizes them to form the input vector; (ii) the Encoder Layers
further process the features with Multi-Head Attention to create a
contextualized representation of each object; and (iii) the Post-Processing
Layer where the object features are paired to make predictions of their
relationships.

#####  4.3.3.1 Pre-Processing Layer

Different from conventional image captioning model where only image region
features $\mathbf{v}_{i}$ are used as the input, we also use object labels
$o_{i}$ as well as bounding boxes $b_{i}$ as the input to encode both object
information and spatial relation information.

Given the bounding box $b=(x_{1},y_{1},x_{2},y_{2})$, to better represent its
location as well as size in the image, we normalize it with the size of the
image and convert it into a $9$-dimensional vector
$(\frac{c_{x}}{W},\frac{c_{y}}{H},\frac{w}{W},\frac{h}{H},\frac{x_{1}}{W},\frac{y_{1}}{H},\frac{x_{2}}{W},\frac{y_{2}}{H},\frac{wh}{WH})$,
where$(c_{x},c_{y})$ is the coordinate of the bounding box center, $(w,h)$ and
$(W,H)$ are the width and height of the bounding box and the image
respectively. We use a two-layer feed-forward net to encode it into vector
$\mathbf{v}_{b}$ of dimension $d_{b}$.

As object labels (e.g. ‘man’) are meaningful words from natural languages, we
use Glove [113] features, which contain pre-trained features to capture fine-
grained semantic and syntactic regularities in natural languages. The
dimension of the object label feature $\mathbf{v}_{l}$ is denoted as $d_{l}$.

Thus, the final image feature is represented as
$\mathbf{v}=f([\mathbf{v}_{i};\mathbf{v}_{b};\mathbf{v}_{l}])$, where
$[\cdot;\cdot;\cdot]$ is the concatenation operation, and $f(\cdot)$ is a
feed-forward network to map the feature to dimension $d_{h}$.

#####  4.3.3.2 Transformer Encoder Layer

Given a set of image features
$\mathbf{v}=\\{\mathbf{v}_{1},\ldots,\mathbf{v}_{m}\\}$ extracted using the
Pre-Processing Layer, we use a stack of $N_{E}$ Transformer [101] encoder
layers to obtain a permutation invariant encoding. Each encoder layer consists
of a multi-head self-attention layer followed by a small feed-forward network.
The multi-head self-attention layer itself consists of $h$ identical heads.
Each attention head first calculates the queries $\mathbf{Q}$, keys
$\mathbf{K}$ and values $\mathbf{V}$ for the $m$ input features as follows:

$\small\mathbf{Q}=\mathbf{X}\mathbf{W}_{Q},\mathbf{K}=\mathbf{X}\mathbf{W}_{K},\mathbf{V}=\mathbf{X}\mathbf{W}_{V},$
(4.4)

where $\mathbf{X}$ contains all the input vectors
$\mathbf{x}_{1},\ldots,\mathbf{x}_{m}$ stacked into a matrix and
$\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, $\mathbf{W}_{V}$ are learned projection
matrices. The image features are used as the input to the first self-attentive
layer. For the following layers, we use the output of the previous encoder
layer as the input to the current layer. The output of each head are then
computed via scaled dot-product attention without using any recurrence:

$\small
head(\mathbf{X})=Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=softmax(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}})\mathbf{V}$
(4.5)

where $d$ is a scaling factor. Eq. 4.4 and 4.5 are calculated for every head
independently. The output of all $h$ heads are then concatenated to one output
vector and multiplied with a learned projection matrix $\mathbf{W}_{O}$:

$\small MultiHead(\mathbf{X})=Concat(head_{1},\ldots,head_{h})\mathbf{W}_{O}$
(4.6)

The multihead attention is then fed into a point-wise feed-forward network
(FFN), which is applied to each output of the attention layer:

$\small\mathbf{h}=FFN(\mathbf{x})=\max(0,\mathbf{x}\mathbf{W}_{1}+\mathbf{b}_{1})\mathbf{W}_{2}+\mathbf{b}_{2}$
(4.7)

where $\mathbf{W}_{1}$, $\mathbf{b}_{1}$ and $\mathbf{W}_{2}$,
$\mathbf{b}_{2}$ are the weights and biases of two fully connected layers. In
addition, skip-connections and layer-norm are applied to the outputs of the
self-attention and the feed-forward layer.

Other choices like LSTM or convolutional layers are also feasible. In this
paper, we choose self-attention layers because self-attention operation can be
seen as a way of encoding pair-wise relationships between input features given
that the self attention weights depend on the pair-wise similarities between
the input features. This is coherent to the objective of the paper: to model
the relationships from the input objects.

#####  4.3.3.3 Post-Processing Layer

For an image of $m$ objects, there are $m(m-1)$ possible relationships. For
each possible relationship between object $i$ and $j$, we compute the
probability of the relationship of label $r_{i\rightarrow j}$. Since the
output of the last Transformer encoder layer $\mathbf{h}$ contains $m$
features $\mathbf{h}_{1},\ldots,\mathbf{h}_{m}$, directly predicting the
relationships using these features is impossible. We thus use a Post-
Processing Layer that maps $m$ object features into $m(m-1)$ pair-wise
features. We use a Linear layer to map $\mathbf{h}_{i}$ of dimension $d_{h}$
into $\mathbf{r}_{i}$ of dimension $2d_{h}$. By doubling the dimension of
$\mathbf{h}_{i}$, we can then equally split $\mathbf{r}_{i}$ into two parts,
head $\mathbf{r}_{i}^{h}$ and tail $\mathbf{r}_{i}^{t}$, with head standing
for the relationship starts from this node while tail stands for the
relationship ends at this node. Thus, for each pair-wise relationship
representation, we can have:

$\small\mathbf{r}_{i\rightarrow
j}=(\mathbf{W}_{r}[\mathbf{r}_{i}^{h};\mathbf{r}_{j}^{t}])\odot[\mathbf{v}_{i};\mathbf{v}_{j}]$
(4.8)

where $\odot$ is point-wise multiplication operation, $[\cdot;\cdot]$ is
concatenation and $\mathbf{W}_{r}$ is a trainable parameter. The distribution
is:

$\small P(r_{i\rightarrow
j}|(b_{i},o_{i}),(b_{j},o_{j}),X)=softmax(\mathbf{r}_{i\rightarrow j})$ (4.9)

####  4.3.4 Weighted Decoder for Image Captioning

As discussed in Section 4.3.1, the objective of our model for caption
generation is to minimize the negative log-likelihood of the correct caption
using the maximum likelihood estimation:

$\small\mathcal{L}_{c}=-\sum_{i=1}^{n}\log
p(\mathbf{y}_{i}|\mathbf{y}_{1:n-1},\mathbf{x})$ (4.10)

The decoder that is used to model Eq. 5.1 consists of a stack of $N_{D}$
decoder layers (Figure 5.3(c)). For each layer, to calculate the distribution
for the word at the time step $i$, it takes as input: the embeddings of all
previously generated words $\mathbf{y}_{0:i-1}$ and the context embeddings
$\mathbf{x}$ from the encoder.

For conventional Transformer [101], $\mathbf{x}=\mathbf{h}^{N_{E}}_{1:m}$,
which means the decoder layers only take the output of the final layer (i.e.
the $N_{E}$-th) as input. This might omit some of the useful features from the
lower encoder layers. In this paper, given the outputs from all the encoder
layers, $\\{\mathbf{h}^{1:N_{E}}_{1:m}\\}$, we take a weighted sum across all
layers to obtain the final image feature as:

$\small\mathbf{x}_{i}=\alpha_{l}\sum_{l=1}^{N_{E}}\mathbf{h}^{l}_{i}$ (4.11)

where $\alpha_{l}$ are weights obtained using a softmax layer.

####  4.3.5 Sequential Training with Inferred Labels

Training JoGANIC involves three steps: (i) training the Faster R-CNN object
detector on Visual Genome dataset; (ii) the trained Faster R-CNN is applied to
train the encoder with Eq. 4.3 on Visual Genome dataset; (iii) when the
encoder is well trained, it is further trained with the joint objective of
scene graph generation and image captioning following Eq. 4.12 on COCO
dataset. As the relation labels are not available on COCO dataset, we infer
the labels for objects in COCO dataset using the encoder trained in step (ii).

The overall loss function in step (iii) is:

$\small\mathcal{L}=\mathcal{L}_{c}+\lambda\mathcal{L}_{r}$ (4.12)

with $\lambda$ being a hyper-parameter.

One possible variant of this training algorithm is to only use
$\mathcal{L}_{c}$ without $\mathcal{L}_{r}$ as the training loss, as done in
the literature work. However, our Ablation studies in Tab. 4.2 show that this
variant cannot achieve as good results as those using the proposed training
algorithm. The main purpose of incorporating $\mathcal{L}_{r}$ is to ensure
that the relational features learned do not vary much in step (iii) while
being used to generate an accurate scene graph.

###  4.4 Experiments

####  4.4.1 Datasets

We evaluate JoGANIC on two large-scale publicly available datasets: MS COCO
[86] which is an image captioning dataset and Visual Genome [95] which is a
scene graph generation dataset.

COCO. The dataset is the most popular benchmark for image captioning, which
contains $82,783$ training images and $40,504$ validation images. There are
$5$ human annotated descriptions per image. As the annotations of the official
test set are not publicly available, we follow the widely used splits provided
by [80], where $5,000$ images are used for validation, $5,000$ for testing and
the rest for training. We convert all the descriptions in the training set to
lower case and discard rare words which occur less than 5 times, resulting in
the final vocabulary with $10,201$ unique words in the COCO dataset. We also
evaluate our model on the COCO online test server. The results of the online
evaluation are shown in the Supplementary.

Visual Genome. The dataset is a large-scale image dataset for modeling the
relationships between objects, which contains $108$K images with densely
annotated objects, attributes, and relations. To pre-train the Faster R-CNN
object detector, we take $98$K for training, $5$K for validation and $5$K for
testing. As part of images (about $50$K) in the Visual Genome are also found
in COCO, the split of the Visual Genome is carefully selected to avoid
contamination of the COCO validation and test sets. We perform extensive
cleaning and filtering of training data, and train Faster R-CNN over the
selected $1,600$ object classes. To pre-train the encoder of JoGANIC on the
scene graph generation task, we adopt the same data split for training the
object detector. Moreover, we select the top-$50$ frequent predicates in
training data. The semantic relation detection model is thus trained over the
$50$ relation classes plus a non-relation class.

####  4.4.2 Methods & Metrics

We compare against three types of baselines. (i) The CNN-LSTM [114] based
models: Up-Down [77] which uses attention over regions of interest, NBT [81]
that first generates a sentence ‘template’ and then fill in by visual concepts
identified by object detectors, Att2all [115] that uses self-critical sequence
training for image captioning, and AoA [116] which uses attention on attention
for encoding image regions and an LSTM language model; (ii) Transformer-based
models: $\mathcal{M}^{2}$-T [108] which uses a mesh-like connectivity to learn
prior knowledge, Image-T [104], an image transformer, Object-T [109] that
models the spatial relationship between objects, and ETA [110] which proposes
the entangled attention mechanism; (iii) The GCN-LSTM based models: VSUA [111]
that uses GCNs to model the semantic and geometric interactions of the
objects, GCN [94] which exploits pairwise relationships between image regions
through a GCN, and SGAE [93] which instead uses auto-encoding scene graphs.

For the caption generation evaluation, we follow the other baselines and use
the BLEU-1 and BLEU-4 [117], ROUGE [118], METEOR [119], CIDEr [120] and SPICE
[121] metrics.

To evaluate the Scene Graph Generation, we divide it into three sub-tasks
[89]: (i) Predicate Classification (PredCls), to predict $r_{i\rightarrow j}$
with $(b_{i},o_{i})$ and $(b_{j},o_{j})$ given; (ii) Scene Graph
Classification (SGCls), to predict the object labels $o_{i}$ and $o_{j}$ and
the relationship $r_{i\rightarrow j}$ with $b_{i}$ and $b_{j}$ given; (iii)
Scene Graph Detection (SGDet), to directly predict $(b_{i},o_{i})$,
$(b_{j},o_{j})$ and $r_{i\rightarrow j}$ from an image without the groundtruth
bounding boxes or object labels.

The evaluation metrics we report are recall @$x$, where $x=20,50,100$. Recall
@$x$ computes the fraction of times the correct relationship is predicted in
the top $x$ confident relationship predictions. It was first proposed in [122]
and then widely adopted in other papers [89, 103, 103]. We notice that mean
average precision (mAP) is another widely used metric. However, mAP is a
pessimistic evaluation metric because we can not exhaustively annotate all
possible relationships in an image. We thus do not report the results using
this metric.

#####  4.4.2.1 Implementation Details

Method | B-1 | B-4 | M | R | C | S
---|---|---|---|---|---|---
Up-Down [77] | 79.8 | 36.3 | 27.7 | 56.9 | 120.1 | 21.4
Att2all [115] | – – | 34.2 | 26.7 | 55.7 | 114.0 | – –
NBT [81] | 75.5 | 34.7 | 27.1 | 54.7 | 107.2 | 20.1
AoA [116] | 80.2 | 38.9 | 29.2 | 58.8 | 129.8 | 22.4
ETA [110] | 81.5 | 39.3 | 28.8 | 58.9 | 126.6 | 22.7
Object-T [109] | 80.5 | 38.6 | 28.7 | 58.4 | 128.3 | 22.6
Image-T [104] | 80.8 | 39.5 | 29.1 | 59.0 | 130.8 | 22.8
$\mathcal{M}^{2}$-T [108] | 80.8 | 39.1 | 29.2 | 58.6 | 131.2 | 22.6
GCN [94] | 80.5 | 38.2 | 28.5 | 58.3 | 127.6 | 22.0
SGAE [93] | 80.8 | 38.4 | 28.4 | 58.6 | 127.8 | 22.1
VSUA [111] | – – | 38.4 | 28.5 | 58.4 | 128.6 | 22.0
JoGANIC | 82.3 | 39.8 | 29.7 | 59.8 | 131.9 | 23.0
Table 4.1: Results on COCO dataset. We only report the single model results on
the ‘Karpathy’ test split. We highlight the best model in bold.

To represent image regions, we use Faster R-CNN with ResNet-101 finetuned on
the Visual Genome dataset, thus obtaining a $2048$-dimensional feature vector
for each region. To represent words, we use one-hot vectors and linearly
project them to the input dimensionality of $512$. The dimension of the
encoded bounding box, object label and the final image feature are
$d_{b}=100$, $d_{l}=300$, and $d_{h}=512$ respectively. We set the dimension
of each layer to $d=512$ and the number of heads to $h=8$. We use the same
number of encoders and decoders, thus having $N_{E}=N_{D}=3$. We employ
dropout with a probability $0.9$ after each attention and feed-forward layer.
Training with the overall objective function (Eq. 4.12) is done following the
learning rate scheduling strategy of [101] with a warmup equal to $10$K
iterations. Then, during CIDEr optimization, we use a fixed learning rate of
$5\times 10^{-6}$. We train all models using the Adam [123] optimizer. We use
Glove [113] embedding to initialize word embedding layer. The total number of
objects in one image varies from $10$ to $50$, depending on the IOU threshold
that is set to $0.3$. After some parameter tuning, we fix $\lambda=0.1$ at
which JoGANIC provides the best CIDEr score.

####  4.4.3 Evaluation

#####  4.4.3.1 General Caption Generation

We first evaluate our model with the general caption generation metrics. We
first compare the performances of our JoGANIC with those of several recent
proposals for image captioning on the COCO ‘Karpathy’ test split. As shown in
Tab. 4.1, in general, the Transformer-based models outperforms other two types
of baselines: the models using pre-trained CNN to encode image information and
the LSTM decoder with attention to decode a caption and those using GCN to
encode scene graph information and the LSTM decoder to generate a sentence.
This proves that Transformer can be used to better learn high-level image
features and is capable of decoding sentences with a higher quality. The GCN-
based models outperform the CNN-based ones but perform worse than the ones
using the Transformer, which indicates that the scene graph information is
useful for better learning the image features but still not well explored. Our
proposed JoGANIC outperforms all other models. For example, it provides an
improvement of $1.5$, $0.7$, $0.5$, $1.2$, $0.7$ and $0.4$ points over
baseline model $\mathcal{M}^{2}$-T [108] on $6$ metrics respectively. This
demonstrates the effectiveness of concurrently exploiting the Transformer
structure and scene graphs in extracting the relational image features.

#####  4.4.3.2 Ablation

Method | B-4 | M | R | C | S
---|---|---|---|---|---
Trans.-2 | 35.7 | 27.4 | 56.4 | 121.3 | 20.5
Trans.-3 | 36.5 | 27.8 | 57.0 | 123.6 | 21.1
Trans.-4 | 36.3 | 27.6 | 56.5 | 121.5 | 20.8
Trans.-5 | 36.1 | 27.5 | 56.8 | 121.9 | 20.7
Weighted Trans. | 37.3 | 28.4 | 57.5 | 125.4 | 21.7
JoGANIC | 39.8 | 29.7 | 59.8 | 131.2 | 23.0
JoGANIC ∗ | 38.5 | 28.7 | 58.3 | 128.9 | 22.1
JoGANIC $-\mathcal{L}_{r}$ | 38.9 | 28.8 | 58.7 | 129.4 | 22.3
JoGANIC $-$ Weighted | 39.3 | 29.2 | 58.9 | 130.3 | 22.5
Table 4.2: Ablation study and comparison of JoGANIC variants. Results are
reported on the ‘Karpathy’ test split. ∗ denotes that we fix the encoder
during the caption training. We highlight the best model in bold.

We first do ablation studies on the number of Transformer layers. We start
from the vanilla Transformer without using any other techniques proposed in
the paper. We vary the number of encoder and decoder layers from 2 to 6. As
shown in Tab. 4.2, the Transformer model with $3$ layers achieves the best
results. To evaluate the importance of keeping $\mathcal{L}_{c}$ in step (iii)
of Section 4.3.5, we conduct two ablation experiments. We first keep the
parameters of the encoder fixed, which means $\mathcal{L}_{r}$ is not used to
update the parameters of the encoder. We denote this variant as JoGANIC ∗. We
find that the CIDEr score drops from $131.2$ to $128.9$. We then remove
$\mathcal{L}_{r}$ and only use $\mathcal{L}_{c}$ to update the parameters of
the encoder and the decoder. We denote this case as JoGANIC
$-\mathcal{L}_{r}$. We find that the CIDEr score drops as well. We can thus
conclude that using $\mathcal{L}_{r}$ helps to improve the quality of caption
generation.

#####  4.4.3.3 Scene Graph Generation Evaluation

Method | SGGen | SGCls | PredCls
---|---|---|---
R@20 | R@50 | R@100 | R@20 | R@50 | R@100 | R@20 | R@50 | R@100
IMP | 14.6 | 20.7 | 24.5 | 31.7 | 34.6 | 35.4 | 52.7 | 59.3 | 61.3
MOTIFS | 21.4 | 27.2 | 30.3 | 32.9 | 35.8 | 36.5 | 58.5 | 65.2 | 67.1
VCTree | 22.0 | 27.9 | 31.3 | 35.2 | 38.1 | 38.8 | 60.1 | 66.4 | 68.1
JoGANIC | 25.4 | 33.0 | 37.2 | 36.6 | 40.1 | 41.1 | 60.5 | 66.7 | 68.1
Table 4.3: Scene graph generation evaluation results on Visual Genome dataset.
We highlight the best model in bold.

Generating scene graphs is essential not only because it provides a way of
explaining the relationships between the objects in the image, but also the
resource of the scene graphs when there is no other state-of-the-art scene
graph generator available. To evaluate the proposed JoGANIC in generating
scene graphs, we compare it with other state-of-the-art scene graph generation
methods. IMP [88] solves the scene graph inference problem using standard RNNs
and learns to iteratively improves its predictions via message passing. MOTIFS
[89] is a stacked bi-directional LSTM architecture designed to capture higher
order motifs in scene graphs. VCTree [103] is a dynamic tree structure that
places the objects in an image into a visual context which helps to improve
the scene graph generation task. The results are shown in Tab. 4.3. JoGANIC
achieves better results than all other three baselines on all the three tasks:
SGGen, SGCls and PredCls. This demonstrates the capability of our JoGANIC to
exploit the relationships between the objects in the image.

#####  4.4.3.4 Qualitative Evaluation

Figure 4.4: Examples of captions generated by JoGANIC and the
$\mathcal{M}^{2}$-T model, as well as the corresponding ground-truths. JoGANIC
generates a scene graph to show the pair-wise relationships.

In Figure 5.1, we show the image, groundtruth caption and the caption
generated by $\mathcal{M}^{2}$ Transformer and JoGANIC. Our model is able to
not only generate a meaningful and more accurate caption than the baseline,
but also generate a scene graph showing the relationships between the objects
in the image. With the graphs generated, captions become more expressive and
explainable. Interestingly, the graphs usually contain useful information to
describe the image. For instance, in the first example, the scene graph
generated by JoGANIC tells us that the player is wearing a shirt and he is
holding a racket. While in the second example, the graph shows that there is a
glass on the table and an arm from a person. These information cannot be
inferred from only the caption generated. Since the average number of words in
a sentence in the COCO dataset is $\approx 10$, it becomes very difficult for
a captioner to describe details from the image using such short sentences.
Thus, a scene graph can be a good complementing component to the caption.

###  4.5 Conclusion

Exploring object relationships for image captioning is a challenging task,
because it not only requires a strong encoder-decoder model to generate
accurate captions but also a scheme to embed the relational information in the
encoder. To effectively improve the image caption quality, we propose the use
of JoGANIC to integrate the extraction of object relationship and caption
generation into the same learning framework that the encoder can be more
accurately trained. Our method achieves significant gains over COCO dataset
compared to the state-of-the-art models. However, there is still a room to
improve the quality of image captioning. For example, to get more accurate
scene graphs, one might use the online crowd-sourcing tools like Amazon
Mechanical Turk to manually annotate the COCO dataset with relational labels.

##  Chapter 5 Fashion Captioning

###  5.1 Introduction

Motivated by the quick global growth of the fashion industry, which is worth
trillions of dollars111https://www.statista.com/topics/965/apparel-market-in-
the-us/, extensive efforts have been devoted to fashion related research over
the last few years. Those research directions include clothing attribute
prediction and landmark detection [124, 125], fashion recommendation [126],
item retrieval [127, 128], clothing parsing [129, 130], and outfit
recommendation [131, 132, 133].

Accurate and enchanting descriptions of clothes on shopping websites can help
customers without fashion knowledge to better understand the features
(attributes, style, functionality, benefits to buy, etc.) of the items and
increase online sales by enticing more customers
222https://www.lyfemarketing.com/blog/best-product-description/. However,
manually writing the descriptions is a non-trivial and highly expensive task.
Thus, the automatic generation of descriptions is in urgent need. Since there
exist no studies on generating fashion related descriptions, in this chapter,
we propose specific schemes on Fashion Captioning. Our design is built upon
our newly created FAshion CAptioning Dataset (FACAD), the first fashion
captioning dataset consisting of over $800$K images and $120$K descriptions
with massive attributes and categories. Compared with general image captioning
datasets (e.g. MS COCO [85]), the descriptions of fashion items have three
unique features (as can be seen from Fig. 5.1), which makes the automatic
generation of captions a challenging task. First, fashion captioning needs to
describe the fine-grained attributes of a single item, while image captioning
generally narrates the objects and their relations in the image (e.g., a
person in a dress). Second, the expressions to describe the clothes tend to be
long so as to present the rich attributes of fashion items. The average length
of captions in FACAD is 23 words while a sentence in the MS COCO caption
dataset contains 10.4 words in average. Third, FACAD has a more enchanting
expression style than MS COCO to arouse greater customer interests. Sentences
like “pearly”, “so-simple yet so-chic”, “retro flair” are more attractive than
the plain or “undecorated” MS COCO descriptions.

Figure 5.1: An example for Fashion Captioning. The images are of different
perspectives, colors and scenarios (shop-street). Other information contained
include a title, a description (caption) from a fashion expert, the color info
and the meta info. Words in color denotes the attributes used in sentence.
Note: Image Caption is generated by an image captioning model trained on MS
COCO dataset.

The image captioning problem has been widely studied and achieved great
progress in recent years. An encoder-decoder paradigm is generally followed
with a deep convolutional neural network (CNN) to encode the input images and
a Long Short Term Memory (LSTM) decoder to generate the descriptions [83, 80,
79, 109, 77]. The encoder-decoder model is trained via maximum likelihood
estimation (MLE), which aims to maximize the likelihood of the next word given
the previous words. However, MLE-based methods will cause the model to
generate “unmatched” descriptions for the fashion items, where sentences
cannot precisely describe the attributes of items. This is due to two reasons.
First, MLE treats the attribute and non-attribute words equally. Attribute
words are not emphasized and directly optimized in the training process,
however, they are more important and should be considered as the key parts in
the evaluation. Second, MLE maximizes its objective word-by-word without
considering the global semantic meaning of the sentence. This shortcoming may
lead to generating a caption that wrongly describes the category of the item.

To generate better descriptions for fashion items, we propose two semantic
rewards as the objective to optimize and train our model using Reinforcement
Learning (RL). Specifically, we propose an attribute-level semantic (ALS)
reward with an attribute-matching algorithm to measure the consistency level
of attributes between the generated sentences and ground-truth. By
incorporating the semantic metric of attributes into our objective, we
increase the quality of sentence generation from the semantic perspective. As
a second procedure, we propose a sentence-level semantic (SLS) reward to
capture the semantic meaning of the whole sentence. Given a text classifier
pretrained on the sentence category classification task, the high level
features of the generated description, i.e., the category feature, should stay
the same as the ground-truth sentence. In this chapter, we use the output
probability of the generated sentence as the groundtruth category as the SLS
reward. Since both ALS reward and SLS reward are non-differentiable, we seek
RL to optimize them.

In addition, to guarantee that the image features extracted from the CNN
encoder are meaningful and correct, we design a visual attribute predictor to
make sure that the predicted attributes match the ground-truth ones. Then
attributes extracted are used as the condition in the LSTM decoder to produce
the words of description.

This work has three main contributions.

  1. 1.

We build a large-scale fashion captioning dataset FACAD of over $800$K images
which are comprehensively annotated with categories, attributes and
descriptions. To the best of our knowledge, it is the first fashion captioning
dataset available. We expect that this dataset will greatly benefit the
research community, in not only developing various fashion related algorithms
and applications, but also helping visual language related studies.

  2. 2.

We introduce two novel rewards (ALS and SLS) into the Reinforcement Learning
framework to capture the semantics at both the attribute level and the
sentence level to largely increase the accuracy of fashion captioning.

  3. 3.

We introduce a visual attribute predictor to better capture the attributes of
the image. The generated description seeded on the attribute information can
more accurately describe the item.

###  5.2 Related Work

We provide a review on fashion related studies and image captioning.

#####  5.2.0.1 Fashion Studies

Most of the fashion related studies [133, 134, 124, 125, 126, 127, 129]
involve images. For outfit recommendation, Cucurull et al. [133] used a graph
convolutional neural network to model the relations between items in a outfit
set, while Vasileva et al. [134] used a triplet-net to integrate the type
information into the recommendation. Wang et al. [124] used an attentive
fashion grammar network for landmark detection and clothing category
classification. Yu et al. [126] introduced the aesthetic information, which is
highly relevant with user preference, into clothing recommending systems. Text
information has also been exploited. Han et al. [131] used title features to
regularize the image features learned. Similar techniques were used in [134].
But no previous studies focus on fashion captioning.