# The Dirichlet series of the arithmetic derivative

Es-said En-naoui
Faculty Of Science And Technics,University Sultan Moulay Slimane
Morocco
<EMAIL_ADDRESS>

###### Abstract

The main object of this paper is to give the generalized von mangoldt function
using the L-additive function which can help us to make it possible to
calculate The Dirichlet series of the arithmetic derivative $\delta$ and
Dirichlet series defined by:

$\sum\limits_{n\geq 1}\frac{f(n)\delta(n)}{n^{s}}$

where $f$ is a classical arithmetic function.

##  1 Introduction

The arithmetic derivative of a natural number $n$, denoted by $\delta(n)$ or
$D(n)$ or $n^{\prime}$, has been the subject of extensive study, from Barbeau,
E. J (see, e.g., [4]) to P. Haukkanen, J. K. Merikoski, and T. Tossavainen
(see, e.g., [2]) .

In this work, we address two major result: Dirichlet product of L-additive
function and generalized von Mangoldt function using L-additive function $f$.
In this article, we use the generalized function von Mangoldt using the
L-additive function to find alternative proof for many series expansions
depend on the arithmetic derivative function. The methods readily generalize,
and can be applied to other L-additive functions. Our principal result are
that :

$\sum\limits_{n\geq
1}\frac{\delta(n)}{n^{s}}=\zeta(s-1)\sum\limits_{p}\frac{1}{p^{s}-p}$

where $\delta$ is the arithmetic derivative function .

Let $n$ be a positive integer. Its arithmetic derivative is the function
$\delta\;:\;\mathbb{N}\rightarrow\mathbb{N}$ , defined by the rules :

  1. 1.

$\delta(p)=1$ for all primes $p$

  2. 2.

$\delta(mn)=m\delta(n)+n\delta(m)$ for all positive integers $m$ and $n$ (the
Leibnitz rule)

Let $n$ a positive integer , if $n=\prod_{i=1}^{s}p_{i}^{\alpha_{i}}$ is the
prime factorization of $n$, then the formula for computing the arithmetic
derivative of n is (see, e.g., [4, 17]) giving by :

$\delta(n)=n\sum\limits_{i=1}^{s}\frac{\alpha_{i}}{p_{i}}=n\sum\limits_{p^{\alpha}||n}\frac{\alpha}{p}$
(1)

A brief summary on the history of arithmetic derivative and its
generalizations to other number sets can be found, e.g., in [4, 17, 9].

Similarly, one can define the arithmetic logarithmic derivative [17] as

${\rm ld}(n)=\frac{\delta(n)}{n}.$

First of all, to cultivate analytic number theory one must acquire a
considerable skill for operating with arithmetic functions. We begin with a
few elementary considerations.

######  Definition 1.1 (arithmetic function).

An arithmetic function is a function $f:\mathbb{N}\longrightarrow\mathbb{C}$
with domain of definition the set of natural numbers $\mathbb{N}$ and range a
subset of the set of complex numbers $\mathbb{C}$.

######  Definition 1.2 (multiplicative function).

A function $f$ is called an multiplicative function if and only if :

$f(nm)=f(n)f(m)$ (2)

for every pair of coprime integers $n$,$m$. In case (2) is satisfied for every
pair of integers $n$ and $m$ , which are not necessarily coprime, then the
function $f$ is called completely multiplicative.

Clearly , if $f$ are a multicative function , then
$f(n)=f(p_{1}^{\alpha_{1}})\ldots f(p_{s}^{\alpha_{s}})$, for any positive
integer $n$ such that $n=p_{1}^{\alpha_{1}}\ldots p_{s}^{\alpha_{s}}$ , and if
$f$ is completely multiplicative , so we have :
$f(n)=f(p_{1})^{\alpha_{1}}\ldots f(p_{s})^{\alpha_{s}}$.

The functions defined above are widely studied in the literature, (see, e.g.,
[11, 13, 14, SC, 15, 16]).

######  Definition 1.3 (additive function).

A function $f$ is called an additive function if and only if :

$f(nm)=f(n)+f(m)$ (3)

for every pair of coprime integers $n$,$m$. In case (3) is satisfied for every
pair of integers $n$ and $m$ , which are not necessarily coprime, then the
function $f$ is called completely additive.

Clearly , if $f$ are a additive function , then
$f(n)=f(p_{1}^{\alpha_{1}})+\ldots+f(p_{s}^{\alpha_{s}})$, for any positive
integer $n$ such that $n=p_{1}^{\alpha_{1}}\ldots p_{s}^{\alpha_{s}}$ , and if
$f$ is completely additive , so we have :
$f(n)=f(p_{1})^{\alpha_{1}}+\ldots+f(p_{s})^{\alpha_{s}}$.

######  Definition 1.4 (L-additive function).

We say that an arithmetic function $f$ is Leibniz-additive (or, L-additive, in
short) (see, e.g., [2]) if there is a completely multiplicative function
$h_{f}$ such that

$f(mn)=f(m)h_{f}(n)+f(n)h_{f}(m)$ (4)

for all positive integers $m$ and $n$.

Then $f(1)=0$ since $h_{f}(1)=1$. The property (4) may be considered a
generalized Leibniz rule. For example, the arithmetic derivative $\delta$ is
L-additive with $h_{\delta}(n)=n$, since it satisfies the usual Leibniz rule

$\delta(mn)=n\delta(m)+m\delta(n)$

for all positive integers $m$ and $n$, and the function $h_{\delta}(n)=n$ is
completely multiplicative. Similarly, the arithmetic partial derivative
respect to the prime $p$ is L-additive with $h_{\delta_{p}}(n)=n$. Further,
all completely additive functions $f$ are L-additive with $h_{f}(n)=1$. For
example, the logarithmic derivative of $n$ is completely additive since

${\rm ld}(mn)={\rm ld}(m)+{\rm ld}(n).$

The term “L-additive function” seems to be new in the literature, yet Chawla
[5] has defined the concept of completely distributive arithmetic function
meaning the same as we do with an L-additive function. However, this is a
somewhat misleading term since a distributive arithmetic function usually
refers to a property that

$f(u\ast v)=(fu)\ast(fv),$ (5)

i.e., the function $f$ distributes over the Dirichlet convolution. This is
satisfied by completely multiplicative arithmetic functions, not by completely
distributive functions as Chawla defined them.

Because L-additivity is analogous with generalized additivity and generalized
multiplicativity (defined in [8]), we could, alternatively, speak about
generalized complete additivity (and also define the concept of generalized
complete multiplicativity).

In this paper, we consider L-additive functions especially from the viewpoint
that they are generalizations of the arithmetic derivative. In the next
section, we present their basic properties. In the last section, we study
L-additivity and the arithmetic derivative in terms of the Dirichlet
convolution.

######  Theorem 1.1.

Let $f$ be an arithmetic function. If $f$ is L-additive and $h_{f}$ is
nonzero-valued, then $f/h_{f}$ is completely additive.

###### Proof.

If $f$ satisfies (4) and $h_{f}$ is never zero, then

$\frac{f(mn)}{h_{f}(mn)}=\frac{f(m)h_{f}(n)+f(n)h_{f}(m)}{h_{f}(m)h_{f}(n)}=\frac{f(m)}{h_{f}(m)}+\frac{f(n)}{h_{f}(n)},$

∎

######  Theorem 1.2.

Let $n$ a positive integer , if $n=\prod_{i=1}^{s}p_{i}^{\alpha_{i}}$ is the
prime factorization of $n$ and $f$ is L-additive with
$h_{f}(p_{1}),\dots,h_{f}(p_{s})\neq 0$, then

$f(n)=h_{f}(n)\sum_{i=1}^{s}\frac{\alpha_{i}f(p_{i})}{h_{f}(p_{i})}.$

###### Proof.

(see, e.g., [2, Theorem 2.4]) ∎

The next step is to extend the L-additive function to the set of rational
numbers $\mathbb{Q}^{*}$ to us it in the Direchlet product of L-additive
function , We start from the positive rationals.

The shortest way is to use the theorem $1.1$ . Namely, if
$x=\prod_{i=1}^{s}p_{i}^{x_{i}}$ is a factorization of a rational number $x$
in prime powers, (where some $x_{i}$ may be negative) then we put :

$f(x)=h_{f}(x)\sum_{i=1}^{s}\frac{x_{i}f(p_{i})}{h_{f}(p_{i})}.$ (6)

and the same proof as in Theorem $1.1$ shows that this definition is still
consistent with the Leibnitz rule for every L-additive function $f$ with
$h_{f}\neq 0$.

######  Lemma 1.1.

Let $n$ a positive integer , if $n=\prod_{i=1}^{s}p_{i}^{\alpha_{i}}$ is the
prime factorization of $n$, then and $f$ is L-additive function with $h_{f}$
is nonzero-valued, then :

$f\bigg{(}\frac{1}{n}\bigg{)}=\frac{-f(n)}{h^{2}_{f}(n)}$ (7)

###### Proof.

Let $n$ a positive integer , if $n=\prod_{i=1}^{s}p_{i}^{\alpha_{i}}$ is the
prime factorization of $n$ , then we have by the formula $(6)$ :

$f\bigg{(}\frac{1}{n}\bigg{)}=h_{f}\bigg{(}\frac{1}{n}\bigg{)}\sum_{i=1}^{s}\frac{-\alpha_{i}f(p_{i})}{h_{f}(p_{i})}=-h_{f}\bigg{(}\frac{1}{n}\bigg{)}\sum_{i=1}^{s}\frac{\alpha_{i}f(p_{i})}{h_{f}(p_{i})}$

Since $f(n)=h_{f}(n)\sum_{i=1}^{s}\frac{\alpha_{i}f(p_{i})}{h_{f}(p_{i})}$
then ,
$\sum_{i=1}^{s}\frac{\alpha_{i}f(p_{i})}{h_{f}(p_{i})}=\frac{f(n)}{h_{f}(n)}$
, so we have :

$f\bigg{(}\frac{1}{n}\bigg{)}=-h_{f}\bigg{(}\frac{1}{n}\bigg{)}.\frac{f(n)}{h_{f}(n)}=-h_{f}(n).h_{f}\bigg{(}\frac{1}{n}\bigg{)}.\frac{f(n)}{h_{f}^{2}(n)}=\frac{-f(n)}{h^{2}_{f}(n)}$

because $h_{f}$ is multiplicative and
$h_{f}(n).h_{f}\bigg{(}\frac{1}{n}\bigg{)}=h_{f}\bigg{(}\frac{n}{n}\bigg{)}=h_{f}(1)=1$
. ∎

######  Theorem 1.3.

Lets $n$ and $m$ two positive integers with $m\neq 0$. and $f$ is L-additive
function with $h_{f}$ is nonzero-valued, then we have :

$f\bigg{(}\frac{n}{m}\bigg{)}=\frac{f(n)h_{f}(m)-f(m)h_{f}(n)}{h^{2}_{f}(m)}$
(8)

A L-additive can be well defined for rational numbers using this formula and
this is the only way to define a L-additive over rationals that preserves the
Leibnitz rule.

###### Proof.

If $n$ and $m$ two positive integers with $m\neq 0$ and $h_{f}$ is never zero,
then :

$f\bigg{(}\frac{n}{m}\bigg{)}=f\bigg{(}n.\frac{1}{m}\bigg{)}=h_{f}(n)f\bigg{(}\frac{1}{m}\bigg{)}+h_{f}\bigg{(}\frac{1}{m}\bigg{)}f(n)$

Since by the lemma 1.1 we have :
$f\big{(}\frac{1}{m}\big{)}=\frac{-f(m)}{h^{2}_{f}(m)}$ , and
$h_{f}\big{(}\frac{1}{m}\big{)}=\frac{1}{h_{f}(m)}$ , then

$f\bigg{(}\frac{n}{m}\bigg{)}=\frac{f(n)}{h_{f}(m)}-\frac{h_{f}(n)f(m)}{h^{2}_{f}(m)}=\frac{f(n)h_{f}(m)-f(m)h_{f}(n)}{h^{2}_{f}(m)}$

∎

for all positive integers $n$ and $m$ , The theorem 1.3 may be considered a
generalized Leibniz rule in the set of rational number $\mathbb{Q}$. This
terminology arises from the observation that the arithmetic derivative is
L-additive with $h_{\delta}=n$ ; it satisfies the usual Leibniz rule of
quotient :

$\delta(\frac{n}{m})=\frac{\delta(n)h_{\delta}(m)-\delta(m)h_{\delta}(n)}{h^{2}_{\delta}(m)}=\frac{m\delta(n)-n\delta(m)}{m^{2}}$

Further, all completely additive functions $f$ are L-additive with
$h_{f}(n)=1$ , then we can extended any completely addtive function to the set
of rational number $\mathbb{Q}$ by this formula :

$f\left(\frac{n}{m}\right)=f(n)-f(m)$

For example, the logarithmic derivative of $n$ is completely additive, then we
have :

$ld\left(\frac{n}{m}\right)=ld(n)-ld(m)$

##  2 L-additive functions in terms of the Dirichlet convolution

Above we have seen that many fundamental properties of the extanded of the
L-additive function to the set of rational number. We complete this article by
changing our point of view slightly and demonstrate that L-additive functions
can also be studied in terms of the Dirichlet convolutions by using the
theorem 1.3.

Let $f$ and $g$ be arithmetic functions. Their Dirichlet convolution is

$(f\ast g)(n)=\sum_{\begin{subarray}{c}a,b=1\\\
ab=n\end{subarray}}^{n}f(a)g(b)=\sum_{\begin{subarray}{c}d|n\end{subarray}}^{n}f(d)g\left(\frac{n}{d}\right).$

where the sum extends over all positive divisors $d$ of $n$ , or equivalently
over all distinct pairs $(a,b)$ of positive integers whose product is $n$.
In particular, we have $(f*g)(1)=f(1)g(1)$ ,$(f*g)(p)=f(1)g(p)+f(p)g(1)$ for
any prime $p$ and for any power prime $p^{m}$ we have :

$(f*g)(p^{m})=\sum\limits_{j=0}^{m}f(p^{j})g(p^{m-j})$ (9)

This product occurs naturally in the study of Dirichlet series such as the
Riemann zeta function. It describes the multiplication of two Dirichlet series
in terms of their coefficients:

$\bigg{(}\sum\limits_{n\geq
1}\frac{\big{(}f*g\big{)}(n)}{n^{s}}\bigg{)}=\bigg{(}\sum\limits_{n\geq
1}\frac{f(n)}{n^{s}}\bigg{)}\bigg{(}\sum\limits_{n\geq
1}\frac{g(n)}{n^{s}}\bigg{)}$ (10)

with Riemann zeta function or is defined by :

$\zeta(s)=\sum\limits_{n\geq 1}\frac{1}{n^{s}}$

These functions are widely studied in the literature (see, e.g., [3, 6, 7]).

We let $f(u\ast v)$ denote the product function of $f$ and $u\ast v$, i.e.,

$(f(u\ast v))(n)=f(n)(u\ast v)(n).$

######  Theorem 2.1.

An arithmetic function $f$ is completely additive if and only if

$f(u\ast v)=(fu)\ast v+u\ast(fv)$

for all arithmetic functions $u$ and $v$.

###### Proof.

(see, e.g., [15, Proposition 2]). ∎

Next theorems shows the dirichlet convolution of arithmetic function with
L-additive functions.

######  Theorem 2.2.

Lets $f$ and $g$ be two arithmetics functions. If $f$ is L-additive and
$h_{f}$ is nonzero-valued, then :

$\displaystyle(f\ast g)(n)=\frac{f(n)}{h_{f}(n)}(h_{f}\ast
g)(n)-\left(h_{f}\ast\frac{fg}{h_{f}}\right)(n)$ (11)

###### Proof.

Let $f$ and $g$ be two arithmetic functions. If $f$ is L-additive and $h_{f}$
is nonzero-valued, then applying the theorem 1.3 on
$f\left(\frac{n}{d}\right)$ we have :

$\displaystyle(g*f)(n)$
$\displaystyle=\sum\limits_{d|n}g(d)f\big{(}\frac{n}{d}\big{)}=\sum\limits_{d|n}g(d)\bigg{(}\frac{h_{f}(d)f(n)-h_{f}(n)f(d)}{h_{f}^{2}(d)}\bigg{)}$
$\displaystyle=\sum\limits_{d|n}g(d)\bigg{(}\frac{f(n)}{h_{f}(d)}-\frac{h_{f}(n)f(d)}{h_{f}^{2}(d)}\bigg{)}$
$\displaystyle=f(n)\sum\limits_{d|n}\frac{g(d)}{h_{f}(d)}-h_{f}(n)\sum\limits_{d|n}\frac{f(d)f(d)}{h_{f}^{2}(d)}$
$\displaystyle=f(n)\left(1\ast\frac{g}{h_{f}}\right)(n)-h_{f}(n)\left(1\ast\frac{f.g}{h_{f}^{2}}\right)(n)$
$\displaystyle=\frac{f(n)}{h_{f}(n)}\left(h_{f}\ast
g\right)(n)-\left(h_{f}\ast\frac{f.g}{h_{f}}\right)(n)$

∎

We can prove this formula using the theorem (2.1) , since the arithmetic
function $\frac{f}{h_{f}}$ is completely additive by the theorem (1.1) , so we
have :

$\frac{f}{h_{f}}\left(h_{f}\ast g\right)=\left(\frac{f}{h_{f}}h_{f}\ast
g\right)+\left(h_{f}\ast\frac{f}{h_{f}}g\right)=\left(f\ast
g\right)+\left(h_{f}\ast\frac{fg}{h_{f}}\right)$

######  Corollary 2.1.

If $f$ is L-additive and $h_{f}$ is nonzero-valued, then :

$f\ast\mu h_{f}=-h_{f}\ast\mu f$

###### Proof.

We substitute $g=\mu h_{f}$ into (2.2) gives :

$(f\ast\mu h_{f})(n)=\frac{f(n)}{h_{f}(n)}(h_{f}\ast\mu
h_{f})(n)-\left(h_{f}\ast\frac{f\mu h_{f}}{h_{f}}\right)(n)$

Since :

$\frac{f(n)}{h_{f}(n)}(h_{f}\ast\mu
h_{f})(n)=f(n)\left(1\ast\mu\right)f(n)\epsilon(n)=0$

then we have :

$(f\ast\mu h_{f})(n)=-\left(h_{f}\ast\mu f\right)(n)$

∎

As we are aware, the arithmetic derivative $\delta$ is an L-additive function
with $h_{\delta}(n)=Id(n)=n$ , then by using the theorem (2.2) we have this
corollary :

######  Corollary 2.2.

Given an arithmetic function $g$, then for every positive integer not null $n$
we have :

$(\delta\ast g)(n)=\frac{\delta(n)}{n}\bigg{(}Id\ast
g\bigg{)}(n)-\bigg{(}Id*\frac{g.\delta}{Id}\bigg{)}(n)$ (12)

###### Proof.

It suffices to notice that $h_{\delta}(n)=Id(n)=n$. ∎

Now, taking $g(n)=Id(n)=n$, then the corollary (2.2) becomes :

$(Id\ast\delta)(n)=\frac{\delta(n)}{n}\bigg{(}Id*Id\bigg{)}(n)-\bigg{(}Id*\frac{Id.\delta}{Id}\bigg{)}(n)=\delta(n)\tau(n)-\left(Id\ast\delta\right)(n)$

So we have this proven formula in the paper (see, e.g., [10, Proposition 6])
and (see, e.g., [15, Proposition 2]) :

$(Id\ast\delta)(n)=\frac{1}{2}\tau(n)\delta(n)$ (13)

where $\tau(n)$ is the divisor-number-function .

We know that $1\ast Id=\sigma$ where $\sigma(n)$ is the sum of the (positive)
divisors of $n$ and $1(n)=1$ for all positive integers $n$ , then by the
equality 13 we have :

######  Corollary 2.3.

For every integer $n$ not null we have :

$\big{(}\sigma*\delta\big{)}(n)=\frac{1}{2}\big{(}1*\tau.\delta\big{)}(n)$
(14)

######  Corollary 2.4.

For every positive integer $n$ not null we have :

$\delta(n)=\frac{1}{2}\big{(}Id.\mu*\tau.\delta\big{)}(n)$ (15)

###### Proof.

Let $n$ a positive integer not null , and $\mu$ the Mobius function .
Since

$\left(Id\ast\delta\right)(n)=\frac{1}{2}\tau(n)\delta(n)$

And :

$\left(Id.\mu\ast Id\right)(n)=\epsilon(n)$

then we have :

$Id.\mu\ast\left(Id\ast\delta\right)(n)=\left(Id.\mu\ast\frac{\tau\delta}{2}\right)(n)$

So :

$\delta(n)=\frac{1}{2}\big{(}Id.\mu*\tau.\delta\big{)}(n)\;\;\;\;since\;\;\left(\epsilon\ast\delta\right)(n)=\delta(n)$

where $\epsilon$ is the multiplicative identity
$\left(\epsilon(n)=\lfloor\frac{1}{n}\rfloor\right)$ ∎

######  Corollary 2.5.

For every integer $n$ not null we have :

$\big{(}Id*Id.\delta\big{)}(n)=\sigma(n)\delta(n)-\big{(}Id^{2}*\delta\big{)}(n)$
(16)

###### Proof.

For $g(n)=1(n)$, where $1(n)=1$ for all positive integers $n$, this reads by
corollary (2.2) :

$(1\ast\delta)(n)=\frac{\sigma(n)\delta(n)-(Id^{2}\ast\delta)(n)}{n}$

The corollary (2.5) is satisfied by multiplying the previous equality by id .
∎

######  Corollary 2.6.

For every integer $n$ not null we have :

$\big{(}Id.\mu\ast\delta\big{)}(n)=-\big{(}Id\ast\mu.\delta\big{)}(n)$ (17)

###### Proof.

For $g(n)=Id(n)\mu(n)$ and for all positive integers $n$, this reads by
corollary (2.2) :

$(\delta\ast Id.\mu)(n)=\frac{\delta(n)}{n}\big{(}Id\ast
Id.\mu\big{)}(n)-\bigg{(}Id*\frac{Id.\mu.\delta}{Id}\bigg{)}(n)$

Then :

$(\delta\ast
Id.\mu)(n)=\delta(n)\big{(}1\ast\mu\big{)}(n)-\big{(}Id\ast\mu.\delta\big{)}(n)$

Since we know that :

$\big{(}1\ast\mu\big{)}(n)=\epsilon(n)$

Therefore :

$(\delta\ast
Id.\mu)(n)=\delta(n)\epsilon(n)-\big{(}Id\ast\mu.\delta\big{)}(n)=-\big{(}Id\ast\mu.\delta\big{)}(n)$

Because $\delta(n)\epsilon(n)=0$ for all positive integers $n$ . ∎

We can prove this corollary just by substitute $f(n)=\delta(n)$ with
$h_{\delta}(n)=n$ into the corollary (2.1)

######  Corollary 2.7.

For every integer $n$ not null we have :

$\big{(}Id.\phi\ast\delta\big{)}(n)=n\delta(n)-\big{(}Id\ast\phi.\delta\big{)}(n)$
(18)

###### Proof.

For $g(n)=Id(n)\phi(n)$ and for all positive integers $n$, this reads by
corollary (2.2) :

$(\delta\ast Id.\phi)(n)=\frac{\delta(n)}{n}\big{(}Id\ast
Id.\phi\big{)}(n)-\bigg{(}Id*\frac{Id.\phi.\delta}{Id}\bigg{)}(n)$

Then :

$(\delta\ast
Id.\phi)(n)=\delta(n)\big{(}1\ast\phi\big{)}(n)-\big{(}Id\ast\phi.\delta\big{)}(n)$

Since we know that (see[19]) :

$\big{(}1\ast\phi\big{)}(n)=Id(n)=n$

Therefore :

$(\delta\ast Id.\phi)(n)=n\delta(n)-\big{(}Id\ast\phi.\delta\big{)}(n)$

∎

On the other hand, An arithmetic function $f$ completely additive is
L-additive function with $h_{f}(n)=1(n)=1$ for every integer not null, then we
have this corollary :

######  Corollary 2.8.

Lets $f$ and $g$ be two arithmetic functions. If $f$ is completely additive,
then for every positive integer not null $n$ by using the theorem 1.3 we have
:

$(f\ast g)(n)=f(n)(1\ast g)(n)-\left(1\ast fg\right)(n)$ (19)

as we knows , the derivative arithmetic function $ld$ is completely additive
with $h_{ld}(n)=1(n)$ then , for every arithmetic function $f$ we have this
result :

$(ld\ast f)(n)=ld(n)(1\ast f)(n)-\left(1\ast ld.f\right)(n)$ (20)

If multiplie both sides of the previous equality by $Id$ we get this formula :

$\left(\delta\ast Id.f\right)(n)=\delta(n)\left(1\ast f\right)(n)-\left(Id\ast
f\delta\right)(n)$ (21)

In the same way we can give many result about The prime omega function
$\Omega$ and the function $log$ .

##  3 Main Results : The generalized von mangoldt function using L-additive
function

Let $f$ L-additive function with $h_{f}$ is nonzero-valued, then Now we
defined the von Mangoldt function related to The function $f$ by :

$\Lambda_{f}(n)={\begin{cases}\frac{f(p)}{h_{f}(p)}&{\text{if }}n=p^{k}{\text{
for some prime }}p{\text{ and integer }}k\geq 1,\\\
0&{\text{otherwise.}}\end{cases}}$ (22)

then we have this result :

######  Theorem 3.1.

If $n\geq 1$ then we have :

$f(n)=h_{f}(n)\sum_{\begin{subarray}{c}d|n\end{subarray}}\Lambda_{f}(d)$

That mean by using dirichlet convolution : $f=h_{f}\ast h_{f}\Lambda_{f}$

###### Proof.

If $n=p_{1}^{\alpha_{1}}\ldots p_{s}^{\alpha_{s}}$ then we have :

$\sum_{\begin{subarray}{c}d|n\end{subarray}}\Lambda_{f}(d)=\sum_{\begin{subarray}{c}i=1\end{subarray}}^{s}\sum_{\begin{subarray}{c}k=1\end{subarray}}^{i}\Lambda_{f}(p_{i}^{k})=\sum_{\begin{subarray}{c}i=1\end{subarray}}^{s}\sum_{\begin{subarray}{c}k=1\end{subarray}}^{i}\frac{f(p_{i})}{h_{f}(p_{i})}=\sum_{\begin{subarray}{c}i=1\end{subarray}}^{s}\frac{if(p_{i})}{h_{f}(p_{i})}=\frac{f(n)}{h_{f}(n)}$

as claimed
∎

######  Theorem 3.2.

for every positive integer $n$ we have :

${\displaystyle\Lambda_{f}(n)=\sum_{d\mid
n}\frac{\mu\left(\frac{n}{d}\right)f\left(d\right)}{h_{f}\left(d\right)}}=-\sum_{\begin{subarray}{c}d|n\end{subarray}}\frac{\mu(d)f(d)}{h_{f}(d)}$

That is we have $\Lambda_{f}=\mu\ast\frac{f}{h_{f}}=-1\ast\frac{\mu f}{h_{f}}$

###### Proof.

By Theorem (3.1) applying Mobius inversion, we have
$\Lambda_{f}=\mu\ast\frac{f}{h_{f}}$ , Note that :

$\displaystyle\sum_{d\mid
n}\frac{\mu\left(\frac{n}{d}\right)f\left(d\right)}{h_{f}\left(d\right)}$
$\displaystyle=\sum_{d\mid
n}\mu(d)\frac{f\left(\frac{n}{d}\right)}{h_{f}\left(\frac{n}{d}\right)}=\sum\limits_{d|n}\mu(d)\frac{h_{f}(d)}{h_{f}(n)}\bigg{(}\frac{h_{f}(d)f(n)-h_{f}(n)f(d)}{h_{f}^{2}(d)}\bigg{)}$
$\displaystyle=\sum\limits_{d|n}\frac{\mu(d)h^{2}_{f}(d)f(n)}{h_{f}(n)h^{2}_{f}(d)}-\frac{\mu(d)h_{f}(n)h_{f}(d)f(d)}{h_{f}(n)h^{2}_{f}(d)}$
$\displaystyle=\frac{f(n)}{h_{f}(n)}\sum\limits_{d|n}\mu(d)-\sum\limits_{d|n}\frac{\mu(d)f(d)}{h_{f}(d)}$
$\displaystyle=\frac{f(n)\epsilon(n)}{h_{f}(n)}-\sum\limits_{d|n}\frac{\mu(d)f(d)}{h_{f}(d)}$
$\displaystyle=-\sum\limits_{d|n}\frac{\mu(d)f(d)}{h_{f}(d)}$

Hence $\Lambda_{f}(n)=-1\ast\frac{\mu f}{h_{f}}$. This completes the proof . ∎

######  Corollary 3.1.

Let $f$ an arithmetic function. If $f$ is L-additive and $h_{f}$ is nonzero-
valued, then :

$\displaystyle(\tau\ast\Lambda_{f})(n)=\frac{f(n)\tau(n)}{2h_{f}(n)}$ (23)

###### Proof.

Substituting $g(n)=h_{f}(n)$ into the theorem (2.2) give :

$(f\ast h_{f})(n)=\frac{f(n)}{h_{f}(n)}(h_{f}\ast
h_{f})(n)-\left(h_{f}\ast\frac{fh_{f}}{h_{f}}\right)(n)$

then :

$(f\ast h_{f})(n)=\frac{1}{2}f(n)\tau(n)$

Since we have that from the theorem (3.1) :

$f(n)=\left(h_{f}\ast h_{f}\Lambda_{f}\right)(n)$

Then we find that :

$\left(h_{f}\ast f\right)(n)=\left(h_{f}\ast h_{f}\ast
h_{f}\Lambda_{f}\right)(n)=h_{f}(n)\left(1\ast 1\ast\Lambda_{f}\right)(n)$

We conclude that :

$h_{f}(n)\left(\tau\ast\Lambda_{f}\right)(n)=\frac{1}{2}\tau(n)f(n)$

This completes the proof of Theorem ∎

As we knows the arithmetic function $f$ completely additive is L-additive
function with $h_{f}(n)=1(n)=1$ for every integer not null, then we have the
von Mangoldt function related to The function $f$ defined by :

$\Lambda_{f}(n)={\begin{cases}f(p)&{\text{if }}n=p^{k}{\text{ for some prime
}}p{\text{ and integer }}k\geq 1,\\\ 0&{\text{otherwise.}}\end{cases}}$ (24)

######  Corollary 3.2.

Let $f$ an arithmetic function completely additive, then :

$f(n)=\sum_{\begin{subarray}{c}d|n\end{subarray}}\Lambda_{f}(d)$

That mean by using dirichlet convolution : $f=1\ast\Lambda_{f}$

######  Corollary 3.3.

Let $f$ an arithmetic function completely additive, then we have:

${\displaystyle\Lambda_{f}(n)=\sum_{d\mid
n}\mu\left(\frac{n}{d}\right)f\left(d\right)=-\sum_{\begin{subarray}{c}d|n\end{subarray}}\mu(d)f(d)}$

That is we have $\Lambda_{f}=\mu\ast f=-1\ast\mu f$

The definition (22) may be considered a generalized von mangoldt function.
This terminology arises from the observation that the logarithm is L-additive
with $h_{log}(n)=1$; it satisfies the usual von mangoldt function denoted by
$\Lambda$

$\Lambda(n)=\Lambda_{log}(n)={\begin{cases}\frac{log(p)}{h_{log}(p)}=log(p)&{\text{if
}}n=p^{k}{\text{ for some prime }}p{\text{ and integer }}k\geq 1,\\\
0&{\text{otherwise.}}\end{cases}}$

By using the corollary (3.2) and (3.3) the Properties of the von mangoldt
function is hold and we have

$log=1\ast\Lambda\;\;\;and\;\;\;\Lambda=\mu\ast\log=-1\ast\mu\log$

Now we can defined the von mangoldt function associed to arithmetic derivative
$ld$ by :

$\Lambda_{ld}(n)={\begin{cases}\frac{1}{p}&{\text{if }}n=p^{k}{\text{ for some
prime }}p{\text{ and integer }}k\geq 1,\\\ 0&{\text{otherwise.}}\end{cases}}$

Substituting $h_{ld}(n)=1$ into the corollary (3.2) and (3.3) gives :

$ld(n)=\left(1\ast\Lambda_{ld}\right)(n)$ (25)

And :

$\Lambda_{ld}(n)=\left(\mu\ast ld\right)(n)=-\left(1\ast ld\mu\right)(n)$ (26)

For later convenience we introduce the prime function denoted by $F$ defined
by :

$F(s)=\sum\limits_{p}\frac{1}{p^{s+1}-p}$

and note that it converges for $Re(s)>0$ It is an analog of the Riemann zeta
function, described in (see, e.g., [20]), with the sum taken over prime
numbers instead of all natural numbers , then we have this result about the
von mangoldt function associed to arithmetic derivative

######  Lemma 3.1.

Let $s$ a number complex such that $Re(s)>0$ , then we have :

$\sum\limits_{n\geq 1}\frac{\Lambda_{ld}(n)}{n^{s}}=F(s)$

###### Proof.

Let $s$ a number complex such that $Re(s)>0$ , then

$\displaystyle\sum\limits_{n\geq 1}\frac{\Lambda_{ld}(n)}{n^{s}}$
$\displaystyle=\frac{\Lambda_{ld}(1)}{1^{s}}+\frac{\Lambda_{ld}(2)}{2^{s}}+\frac{\Lambda_{ld}(3)}{3^{s}}+\frac{\Lambda_{ld}(4)}{4^{s}}+\frac{\Lambda_{ld}(5)}{5^{s}}+\ldots+\frac{\Lambda_{ld}(16)}{16^{s}}+\ldots$
$\displaystyle=\frac{1}{2^{s+1}}+\frac{1}{3^{s+1}}+\frac{1}{2^{2s+1}}+\frac{1}{5^{s+1}}+\frac{1}{7^{s+1}}+\frac{1}{2^{3s+1}}+\ldots+\frac{1}{2^{4s+1}}+\ldots$
$\displaystyle=\sum\limits_{p}\sum\limits_{k\geq
1}\frac{1}{p^{ks+1}}=\sum\limits_{p}\frac{1}{p}\sum\limits_{k\geq
1}\frac{1}{p^{ks}}$
$\displaystyle=\sum\limits_{p}\frac{1}{p}\sum\limits_{k\geq
1}\bigg{(}\frac{1}{p^{s}}\bigg{)}^{k}=\sum\limits_{p}\frac{1}{p}.\frac{1}{p^{s}}.\frac{1}{1-\frac{1}{p^{s}}}$
$\displaystyle=\sum\limits_{p}\frac{1}{p^{s+1}-p}$

which completes the proof ∎

######  Theorem 3.3.

Let $s$ a number complex such that $Re(s)>0$ , then we have :

$\sum\limits_{n\geq 1}\frac{\delta(n)}{n^{s}}=\zeta(s-1)F(s-1)$

###### Proof.

By Using $\left(\ref{von-ld}\right)$ we have :

$\delta(n)=\left(Id\ast Id.\Lambda_{ld}\right)(n)$

then by the formula $\left(\ref{eq:5}\right)$ we have :

$\sum\limits_{n\geq 1}\frac{\delta(n)}{n^{s}}=\sum\limits_{n\geq
1}\frac{\left(Id\ast
Id.\Lambda_{ld}\right)(n)}{n^{s}}=\zeta(s-1)\sum\limits_{n\geq
1}\frac{\Lambda_{ld}(n)}{n^{s-1}}=\zeta(s-1)F(s-1)$

∎

######  Corollary 3.4.

for every $s\in\mathbb{C}$ where $Re(s)>2$ we have :

$\sum\limits_{n\geq 1}\frac{\tau(n)\delta(n)}{n^{s}}=2\zeta^{2}(s-1)F(s-1)$

###### Proof.

If we apply the relation (10) on the equality (2) ,then for every complex
number $s$ we get that :

$\sum\limits_{n\geq
1}\frac{\tau(n)\delta(n)}{n^{s}}=2\zeta(s-1)\sum\limits_{n\geq
1}\frac{\delta(n)}{n^{s}}\;\;\;\;\;\;\left(for\;\;Re(s)>2\right)$

then by the theorem (LABEL:dir-der) for every $s\in\mathbb{C}$ where $Re(s)>2$
we have :

$\sum\limits_{n\geq 1}\frac{\tau(n)\delta(n)}{n^{s}}=2\zeta^{2}(s-1)F(s-1)$

∎

######  Corollary 3.5.

for every $s\in\mathbb{C}$ where $Re(s)>2$ we have :

$\sum\limits_{n\geq
1}\frac{\mu(n)\delta(n)}{n^{s}}=\frac{-F(s-1)}{\zeta(s-1)}$

in the same way , If we apply the relation (10) to the corollary (2.6) we have
that :

$\zeta(s-1)\sum\limits_{n\geq
1}\frac{\mu(n)\delta(n)}{n^{s}}=-\sum\limits_{n\geq
1}\frac{\mu(n)}{n^{s-1}}\sum\limits_{n\geq
1}\frac{\delta(n)}{n^{s}}\;\;\;\;\;\;\left(for\;\;Re(s)>2\right)$

Since (see, e.g., [19]) :

$\sum\limits_{n\geq 1}\frac{\mu(n)}{n^{s}}=\frac{1}{\zeta(s)}$

Therefore by using the result of theorem (3.3) we have that :

$\sum\limits_{n\geq
1}\frac{\mu(n)\delta(n)}{n^{s}}=\frac{-F(s-1)}{\zeta(s-1)}\;\;\;\;\;\;\left(for\;\;Re(s)>2\right)$

######  Corollary 3.6.

for every $s\in\mathbb{C}$ where $Re(s)>3$ we have

$\sum\limits_{n\geq
1}\frac{\phi(n)\delta(n)}{n^{s}}=\frac{\zeta(s-2)}{\zeta(s-1)}\bigg{(}F(s-2)-F(s-1)\bigg{)}$

###### Proof.

We know from $\left(\cite[cite]{[\@@bibref{}{cat-serie}{}{}]}\right)$ that :

$\sum\limits_{n\geq 1}\frac{\phi(n)}{n^{s}}=\frac{\zeta(s-1)}{\zeta(s)}$

And by corollary (2.7) we have :

$\big{(}Id.\phi\ast\delta\big{)}(n)=n\delta(n)-\big{(}Id\ast\phi.\delta\big{)}(n)$

then :

$\sum\limits_{n\geq
1}\frac{\phi(n)\delta(n)}{n^{s}}=\frac{1}{\zeta(s-1)}\sum\limits_{n\geq
1}\frac{\delta(n)}{n^{s-1}}-\frac{\zeta(s-2)}{\zeta^{2}(s-1)}\sum\limits_{n\geq
1}\frac{\delta(n)}{n^{s}}\;\;\;\;\;\;\left(for\;\;Re(s)>3\right)$

∎

On the other hand as a consequence of the corollary (2.5) we have :

$\big{(}Id*Id.\delta\big{)}(n)=\sigma(n)\delta(n)-\big{(}Id^{2}*\delta\big{)}(n)$

then by apply the formula 10 and the result of the theorem 3.3 we have:

######  Corollary 3.7.

for every $s\in\mathbb{C}$ where $Re(s)>3$ we have :

$\sum\limits_{n\geq
1}\frac{\sigma(n)\delta(n)}{n^{s}}=\zeta(s-1)\zeta(s-2)\left(F(s-2)+F(s-1)\right)$

By using this equality let $f(n)=Id_{k}(n)$ then we have

######  Corollary 3.8.

Let $k\in\mathbb{N}$ , then for every $s\in\mathbb{C}$ where $Re(s)>k+2$ we
have :

$\sum\limits_{n\geq
1}\frac{\sigma_{k}(n)\delta(n)}{n^{s}}=\zeta(s-1)\zeta(s-k-1)\left(F(s-1)+F(s-k-1)\right)$

###### Proof.

Let $k\in\mathbb{N}$ ,applying equality (24) to $f(n)=Id_{k}(n)$ we obtain :

$\left(\delta\ast Id.Id_{k}\right)(n)=\delta(n)\left(1\ast
Id_{k}\right)(n)-\left(Id\ast Id_{k}\delta\right)(n)$

Since :

$\left(1\ast Id_{k}\right)(n)=\sigma_{k}(n)$

then we have :

$\left(\delta\ast Id_{k+1}\right)(n)=\delta(n)\sigma_{k}(n)-\left(Id\ast
Id_{k}\delta\right)(n)$

Now applying equality (10) we obtain:

$\sum\limits_{n\geq 1}\frac{\left(\delta\ast
Id_{k+1}\right)(n)}{n^{s}}=\sum\limits_{n\geq
1}\frac{\delta(n)\sigma_{k}(n)}{n^{s}}-\sum\limits_{n\geq 1}\frac{\left(Id\ast
Id_{k}\delta\right)(n)}{n^{s}}$

Therefore,

$\sum\limits_{n\geq
1}\frac{\delta(n)\sigma_{k}(n)}{n^{s}}=\zeta(s-k-1)\sum\limits_{n\geq
1}\frac{\delta(n)}{n^{s}}+\zeta(s-1)\sum\limits_{n\geq
1}\frac{\delta(n)}{n^{s-k}}$

Since by the theorem (3.3) we have:

$\sum\limits_{n\geq 1}\frac{\delta(n)}{n^{s}}=\zeta(s-1)F(s-1)$

then :

$\sum\limits_{n\geq
1}\frac{\delta(n)\sigma_{k}(n)}{n^{s}}=\zeta(s-k-1)\zeta(s-1)F(s-1)+\zeta(s-1)\zeta(s-k-1)F(s-k-1)$

and the proof is complete. ∎

the aim of this study is to giving some result that may help us to find a
relation defined by Dirichlet product which makes it possible to calculate the
series of Dirichlet $\sum\limits_{n\geq 1}\frac{f(n)\delta(n)}{n^{s}}$ for
many arithmetic function $f$.

##  4 Conclusion :

The von Mangoldt function $\Lambda_{f}$ related to the L-additive function $f$
is the best way to be solved us the problem of the Dirichlet series of the
arithmetic derivative

## References

  * [1] V. Ufnarovski and B. Åhlander.  How to differentiate a number,  J . Integer Seq., 6 , (2003).
  * [2] P. Haukkanen, J. K. Merikoski, and T. Tossavainen.  The arithmetic derivative and Leibniz additive functions,  Notes Number Theory Discrete Math., 24.3 , (2018).
  * [3] T. M. Apostol, Introduction to Analytic Number Theory, Springer-Verlag, New York (1976).
  * [4] E. J. Barbeau, Remarks on an arithmetic derivative, Canad. Math. Bull. 4(2), 117–122 (1961).
  * [5] L. M. Chawla, A note on distributive arithmetical functions, J. Nat. Sci. Math. 13(1), 11–17 (1973).
  * [6] M.Lewinter and J.Meyer.  Elementary number theory with programming ,  Wiley , (2016).
  * [7] Heng Huat Chan,  Analytic number theory for undergraduates ,  World Scientific Publishing Co. Pte. Ltd. , (2009).
  * [8] P. Haukkanen, A note on generalized multiplicative and generalized additive arithmetic functions, Math. Stud. 61, 113–116 (1992).
  * [9] P. Haukkanen, J. K. Merikoski, T. Tossavainen, On arithmetic partial differential equations, J. Integer Seq. 19, Article 16.8.6 (2016).
  * [10] Es-said En-naoui, Dirichlet product of derivative arithmetic with an arithmetic function multiplicative, https://arxiv.org/abs/1908.07345. (2019).
  * [11] I. Kiuchi, M. Minamide, On the Dirichlet convolution of completely additive functions. J. Integer Seq. 17, Article 14.8.7 (2014).
  * [12] J. Kovič, The arithmetic derivative and antiderivative, J. Integer Seq. 15, Article 12.3.8 (2012).
  * [13] V. Laohakosol, P. Tangsupphathawat, Characterizations of additive functions. Lith. Math. J. 56(4), 518–528 (2016).
  * [14] P. J. McCarthy, Introduction to Arithmetical Functions, Springer-Verlag, New York (1986).
  * [15] E. D. Schwab, Dirichlet product and completely additive arithmetical functions, Nieuw Arch. Wisk. 13(2), 187–193 (1995).
  * [16] H. N. Shapiro, Introduction to the Theory of Numbers, Wiley InterScience, New York (1983).
  * [17] V. Ufnarovski, B. Åhlander, How to differentiate a number, J. Integer Seq. 6, Article 03.3.4 (2003).
  * [18] Merikoski, Jorma K.; Haukkanen, Pentti; Tossavainen, Timo , Complete additivity, complete multiplicativity, and Leibniz-additivity on rationals Integers. (2021).
  * [19] Gould, H. W. and Shonhiwa, T. ,A catalog of interesting Dirichlet series Missouri J. Math. Sci.,  1:2–18 (2008).
  * [20] NIST Digital Library of Mathematical Functions. http://dlmf.nist.gov, Release 1.0.9 of, Online companion to.[21] (2014-08-29).
  * [21] Olver, F. W. J. and Lozier, D. W. and Boisvert, R. F. and Clark, C. W.,. http://dlmf.nist.gov, Release 1.0.9 of, Handbook of Mathematical Functions. Cambridge University Press, New York, NY, . Print companion to[20] (2010).


#  Lafite : Towards Language-Free Training for Text-to-Image Generation

Yufan Zhou 1, Ruiyi Zhang 2, Changyou Chen 1, Chunyuan Li 3,
Chris Tensmeyer 2, Tong Yu 2, Jiuxiang Gu 2, Jinhui Xu 1, Tong Sun 2
1 State University of New York at Buffalo 2 Adobe Research 3 Microsoft
Research, Redmond
{yufanzho, changyou<EMAIL_ADDRESS>
{ruizhang, tensmeye, tyu, jigu<EMAIL_ADDRESS><EMAIL_ADDRESS>The
research of the first and eighth authors was supported in part by NSF through
grants IIS-1910492.

###### Abstract

One of the major challenges in training text-to-image generation models is the
need of a large number of high-quality image-text pairs. While image samples
are often easily accessible, the associated text descriptions typically
require careful human captioning, which is particularly time- and cost-
consuming. In this paper, we propose the first work to train text-to-image
generation models without any text data. Our method leverages the well-aligned
multi-modal semantic space of the powerful pre-trained CLIP model: the
requirement of text-conditioning is seamlessly alleviated via generating text
features from image features. Extensive experiments are conducted to
illustrate the effectiveness of the proposed method. We obtain state-of-the-
art results in the standard text-to-image generation tasks. Importantly, the
proposed language-free model outperforms most existing models trained with
full image-text pairs. Furthermore, our method can be applied in fine-tuning
pre-trained models, which saves both training time and cost in training text-
to-image generation models. Our pre-trained model obtains competitive results
in zero-shot text-to-image generation on the MS-COCO dataset, yet with around
only 1% of the model size and training data size relative to the recently
proposed large DALL-E model.

##  1 Introduction

Automatic synthesis of realistic images from arbitrary text description is one
of the core aspirations in artificial intelligence. Most existing works
achieve the goal by consuming a large number of high quality image-text pairs
[53, 59, 56, 38, 7], which, however, often requires heavy workload of precise
human captioning and filtering. For instance, MS-COCO [27], the most commonly
used dataset in text-to-image generation tasks, requires over 70,000 worker
hours in gathering and annotating the captions. Even for less curated datasets
such as Google Conceptual Captions [41], it consists of 3.3 million image-text
pairs that are heavily filtered from 5 billion images from around 1 billion
English webpages. In practice, for a customized domain, it is infeasible to
collect such a large number of image-text pairs for model training, due to the
high cost of human captioning and filtering. This challenge renders the
unprecedented importance of the zero-shot text-to-image generation tasks,
where no domain-specific image-text pairs are used to train a model to
generate images in a given domain.

Recently, several attempts have been made to tackle zero-shot text-to-image
generation problem, by pre-training giant generative models on web-scale
image-text pairs, such as DALL-E [38] and CogView [7]. Both are auto-
regressive Transformer models built for zero-shot text-to-image generation, as
they can generate corresponding images given arbitrary text description
without training on domain-specific datasets. However, to ensure good
performance, these models require a gigantic scale of data collections, model
size and model training. Specifically, DALL-E contains over 12 billion
parameters and is trained on a dataset consisting of 250 million image-text
pairs; CogView is a model with 4 billion parameters trained on 30 million
image-text pairs. For this reason, hundreds of GPUs are required in training
these models, which significantly increases carbon footprint and decrease the
inclusivity: making it extremely difficult for more researchers to participate
the study of this topic.

It is therefore desired to provide affordable solutions to build text-to-image
generation models for the settings of limited image-text pair data, by
reducing the requirements on model size, data collections and model training.
In terms of data collections, in the ideal scenarios, the language-free
setting is probably the minimal and cheapest requirement, where only image
data is provided. This is important because collecting only image data is much
easier than constructing high-quality image-text pairs, given the ample
domain-specific image datasets available online.

|  |
---|---|---
Figure 1: Model size vs performance of zero-shot image-to-text generation on
the COCO dataset. Lafite has much smaller model size, especially when
considering trainable parameters (Left figure), but shows higher Inception
score (Middle figure) and lower FID (Right figure). Please refer to Section 4
for details.

To this end, we propose Lafite111 LAnguage-Free traIning for Text-to-image
gEneration, a generative adversarial approach to significantly lowering the
cost barrier and to building efficient text-to-image generation models, based
on the pre-trained CLIP model [37]. Specifically, $(i)$ we take advantages of
CLIP’s property on image-text feature alignment in the joint semantic space,
to construct pseudo image-text feature pairs; $(ii)$ we propose a text-to-
image GAN (Generative Adversarial Network) model [11] that can effectively
leverage pseudo image-text feature pairs. Our major contributions can be
summarized as followings:

  * •

We propose Lafite, a versatile system that works effectively in a large range
of text-to-image generation settings, including language-free, zero-shot and
fully-supervised learning.

  * •

To the best of our knowledge, Lafite is the first work that enables the
language-free training for the text-to-image generation task. We propose two
novel schemes to construct pseudo image-text feature pairs, and conduct
comprehensive study for the new setting. The effectiveness is validated with
quantitative results on several datasets with different training schemes
(training from scratch and fine-tuning from pre-trained generative models).

  * •

In zero-shot text-to-image generation settings, Lafite outperforms the prior
art DALL-E and CogView on the COCO benchmark, with less than 1% of the
trainable model parameter size (with frozen CLIP model weights). Please see
Figure 1 for comparisons.

  * •

In the standard fully supervised settings, Lafite outperforms several state-
of-the-art (SoTA) methods by a large margin. Surprisingly, even our language-
free model shows superior performance than most existing models that are
trained with full image-text pairs.

##  2 Related Work

#### Text-to-image generation

Existing models on text-to-image generation can be categorized into two
classes: fully-supervised text-to-image generation [53, 59, 56] and zero-shot
text-to-image generation [38, 7]. The SoTA in the full image-text pair setting
is still dominated by GAN variants [53, 59, 56]. GANs [11] have inspired many
advances in image synthesis [32, 18, 28, 23, 20]. For text-to-image synthesis,
the improved model performance is often benefited from large generative
adversarial image models [56] and pre-trained text encoders [30]. Recently,
excellent zero-shot text-to-image generation performance has been achieved in
DALL-E [38] and CogView [7]. The basic idea is to encode images into discrete
latent tokens using VQ-VAE[45, 39], and pre-train a huge-size auto-regressive
Transformers[46] to predict these discrete tokens based on paired text
sequences. Our Lafite is the first generative adversarial approach that
achieves SoTA on zero-shot generation.

#### Multi-modal feature learning

Learning a joint and aligned feature space for vision-and-language has been a
long standing problem in artificial intelligence [50, 42]. Inspired by the
BERT model [6], a number of methods attempt to learn generic multi-modal
fusion layers, given the pre-extracted visual region features and textual
encoder [31, 26, 43, 57, 21, 24]. These works aim at learning generic multi-
modal representations for downstream tasks like visual question answering [2,
14], image captioning [27, 1], visual commonsense reasoning [55]. Unlike the
aforementioned works, another line of works focus on the way of learning
visual representation from natural language supervisions, including both
generative [5] and discriminative [49, 48, 58] methods. The latter learns an
aligned visual-semantic space. This idea is recently scaled up in CLIP/ALIGN
[37, 16], which pave the way toward building a universal image-text
representation space. Our Lafite is built up in this universal space, and is
the first one to leverage its multi-modal alignment property for language-free
text-to-image generation.

#### CLIP for generation/manipulation.

The idea of multi-modal feature space also inspires some recent works on
generative models [10, 35, 9, 33]. All of these works are related to ours in
that the tools of pre-trained CLIP model and StyleGAN2 are employed. Our
Lafite is different in two aspects: $(i)$ The motivations and scenarios are
different. Existing works focus on latent optimization [10], image
manipulation [35], domain adaptation [9], image segmentation [33]. We present
the first study on training text-to-image generation models without the
requirement of paired captions. $(ii)$ The techniques are different. Though
image-text feature alignment property is leveraged in all works, Our Lafite is
the only one to generate pseudo features pairs in the joint multi-modal space,
none of existing works considers such a possibility.

##  3 Lafite: A Language-Free Paradigm

A natural idea to avoid human captioning in constructing image-text pair
training data is using an off-the-shelf image captioning model that can
automatically generate captions for the collected training images. However,
this is especially challenging due to the lack of a universal captioning model
that can $(i)$ bridge the modality gap between text and image to generate
high-quality captions; $(ii)$ generalize to diverse image domains with large
domain gaps. In this paper, we resort to solving an easier problem: one may
directly generate text features rather than text descriptions, to avoid the
use of image captioning models.

Throughout the paper, $({\mathbf{x}},{\mathbf{t}})$ denotes an image-text
pair, ${\mathbf{x}}^{\prime}$ is the corresponding generated image of
${\mathbf{t}}$. $G$ and $D$ denote the generator and discriminator
respectively. We use $f_{\text{img}}$ and $f_{\text{txt}}$ to denote the pre-
trained text encoder and image encoder, which map text descriptions and image
samples into a joint multi-modal feature space.
${\mathbf{h}}=f_{\text{txt}}({\mathbf{t}})$ denotes the real text feature,
${\mathbf{z}}\sim\mathcal{N}(\mathbf{0},\mathcal{I})$ denotes latent noise
sampled from the standard Gaussian distribution, serving as one input of the
generator. Our idea to achieve language-free training is to generate pseudo
text features ${\mathbf{h}}^{\prime}$, which aims to approximating
${\mathbf{h}}$, by leveraging the image-text feature alignment of a pre-
trained model. The generated features are then fed into the text-to-image
generator to synthesize the corresponding images. Without loss of generality,
we denote the mapping from input data to the multi-modal feature space as
translator $T$ in two settings. If only images ${\mathbf{x}}$ are provided
(_i.e_. language-free setting), we consider a pseudo text-feature generation
process $T:{\mathbf{x}}\rightarrow{\mathbf{h}}^{\prime}$; If image-text pairs
$({\mathbf{x}},{\mathbf{t}})$ are provided (_i.e_. standard fully-supervised
settings), we encode ground-truth text,
$T:{\mathbf{t}}\rightarrow{\mathbf{h}}$.

###  3.1 Pseudo Text-Feature Generation

To achieve the goal, a universal multimodal feature space is desired, where
features of paired texts and images are well aligned. The recently vision-and-
language models such as CLIP and ALIGN achieve this, by pre-training on
hundreds/thousands of millions of image-text pairs using contrastive learning.
The cosine similarity between matched image-text features is maximized, while
cosine similarity of the mis-matched pair is minimized. This naturally
provides a high-dimensional hyper-sphere222In our implementation, we normalize
the features extracted with CLIP by their L2 norm. for the multimodal
features, where paired image-text should be close to each other, with a small
angle between their feature vectors. This inspires us to explore the
potentials of generating pseudo text features
${\mathbf{h}}^{\prime}\in\mathcal{H}({\mathbf{x}})$ for a given image
${\mathbf{x}}$ on this hyper-sphere:
$\mathcal{H}({\mathbf{x}})=\\{{\mathbf{h}}^{\prime}|\mathrm{Sim}({\mathbf{h}}^{\prime},f_{\text{img}}({\mathbf{x}}))\geq
c\\}$, where $\mathrm{Sim}$ denotes cosine similarity, $c>0$ is a threshold.
This idea is illustrated in Figure 2. Based on the analysis, we consider two
schemes to generate pseudo text features.

Figure 2: The illustration that the generated pseudo text feature vector
${\mathbf{h}}^{\prime}\in\mathcal{H}({\mathbf{x}})$ (blue dashed arrow) should
have high cosine similarity with the image feature
$f_{\text{img}}({\mathbf{x}})$ (red solid arrow), _i.e_.
$\theta\leq\arccos{c}$.

#### Fixed perturbations

To generate pseudo text feature ${\mathbf{h}}^{\prime}$, we propose to perturb
the image feature $f_{\text{img}}({\mathbf{x}})$ with adaptive Gaussian noise:

$\displaystyle{\mathbf{h}}^{\prime}=\tilde{{\mathbf{h}}}/\|\tilde{{\mathbf{h}}}\|_{2},~{}~{}~{}\tilde{{\mathbf{h}}}=f_{\text{img}}({\mathbf{x}})+\xi\mathbf{\epsilon}\|f_{\text{img}}({\mathbf{x}})\|_{2}/\|\mathbf{\epsilon}\|_{2},$
(1)

where $\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ is the
Gaussian noise, $\xi>0$ is a fixed hyper-parameter representing the level of
perturbations, $\|\cdot\|_{2}$ denotes L2 norm. The added Gaussian noise is
adaptive in the sense that it is normalized to a hyper-sphere, then re-scaled
by the norm of image feature. We can prove that, with the adaptive noise, our
$\textsc{Lafite}_{\text{G}}$ can generate $\mathcal{H}({\mathbf{x}})$ with a
high probability which depends on $\xi,c$ and $d$. The formal theorem and its
proof are provided in the Appendix.

#### Trainable perturbations

It is natural to extend $\textsc{Lafite}_{\text{G}}$ to learn more adaptive
noise instead of using a vanilla Gaussian. To this end, we propose to train an
inference model which takes the image features as inputs and outputs the mean
and variance of the desired noise distribution. Specifically, the inference
model consists of two neural networks $r_{1}(\cdot)$ and $r_{2}(\cdot)$. With
the re-parameterization trick [22], the generation of pseudo text features is:

$\displaystyle{\mathbf{h}}^{\prime}$
$\displaystyle=\tilde{{\mathbf{h}}}/\|\tilde{{\mathbf{h}}}\|_{2},\text{where}$
(2) $\displaystyle\tilde{{\mathbf{h}}}$
$\displaystyle=f_{\text{img}}({\mathbf{x}})+r_{1}(f_{\text{img}}({\mathbf{x}}))+\mathbf{\epsilon}\odot\exp(r_{2}(f_{\text{img}}({\mathbf{x}}))),~{}$

where $\exp$ denotes element-wise exponent operation, and $\odot$ denotes
element-wise multiplication,
$\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ denotes noise
sampled from standard Gaussian. In practice, we construct $r_{1}(\cdot)$ and
$r_{2}(\cdot)$ with 4 fully-connected (FC) layers respectively, and train them
in a supervised way by maximizing the cosine similarity
$\mathrm{Sim}({\mathbf{h}}^{\prime},{\mathbf{h}})$ between generated text
features and real text features.

#### Discussion.

Both schemes have their own pros and cons. The trainable perturbation
generally yields better performance than the fixed perturbation. However, the
fixed perturbation is easier to use, without the requirement of training an
inference model on an additional dataset with annotated image-text pairs.
Further, the performance of trainable perturbation is influenced by the gap
between datasets used in training the inference model and the generative
model, as empirically verified in our ablation studies in the experiment
section.

###  3.2 Network Architectures

We propose to adapt the unconditional StyleGAN2 to a conditional generative
model for our goal. Note that although we discuss our model in a language-free
setting, it can be directly generalized to standard text-to-image generation
by using ${\mathbf{h}}$ (real text feature) instead of ${\mathbf{h}}^{\prime}$
(pseudo text feature).

#### Generator

It is shown in recent works [29, 51] that the StyleSpace of StyleGAN2 is a
well-disentangled intermediate feature space, whose dimensions are highly
independent. By leveraging this property, we propose a simple yet effective
approach to enable conditional generation: injecting new conditional
information directly into the StyleSpace, as illustrated in Figure 3.
Specifically, we choose to inject text information as follows. $(i)$ Random
noise vectors ${\mathbf{z}}\in{\mathcal{Z}}$ are transformed into an
intermediate latent space ${\mathcal{W}}$ via a so-called mapping network,
which consists of a sequence of FC layers. The ${\mathcal{W}}$ space is
claimed to better reflect the disentangled nature of the learned distribution.
Each ${\mathbf{w}}\in{\mathcal{W}}$ is further transformed to channel-wise
unconditional style codes ${\mathbf{s}}$, using a different learned affine
transformation for each layer of the generator. The space spanned by these
style parameters is often referred to as StyleSpace, or ${\mathcal{S}}$.
$(ii)$ For a conditional vector ${\mathbf{h}}^{\prime}$ from the image-text
joint semantic space of CLIP, it is transformed into condition codes
${\mathbf{c}}$, using a different learned 2-layer FC network for each
generator layer. $(iii)$ At each layer of the generator, we concatenate its
style and conditional codes to obtain $[{\mathbf{s}},{\mathbf{c}}]$, which is
is further transformed to channel-wise conditional style codes ${\mathbf{u}}$
, using a different learned affine transformation for each generator layer. We
refer to the space spanned by these style parameters as Conditional
StyleSpace, or ${\mathcal{U}}$. In sum, the generator $G$ synthesizes a fake
image as:

$\displaystyle{\mathbf{x}}^{\prime}=G({\mathbf{h}}^{\prime},{\mathbf{z}})$ (3)

#### Discriminator

In the text-to-image task, the discriminator ensures the generated image to
satisfy two criterias: photo-realistic to human perception and fidelity to the
text condition. To this end, we encode the input image ${\mathbf{x}}$ with a
shared discriminator backbone, then perform two tasks (each with a task-
specific FC layer), as illustrated in Figure 4. $(i)$
$f_{\text{d}}({\mathbf{x}})$ projects ${\mathbf{x}}$ into a scalar, indicating
the level of true or fake of an input image ${\mathbf{x}}$. This is a common
task shared in all GAN models; $(ii)$ $f_{\text{s}}({\mathbf{x}})$ embeds
${\mathbf{x}}$ into a semantic space, which is expected to be similar to the
semantic space of CLIP. We compute the inner product
$\langle{\mathbf{h}}^{\prime},f_{\text{s}}({\mathbf{x}})\rangle$ to indicate
how well the input image ${\mathbf{x}}$ is semantically aligned/conditioned
with the pseudo text feature. In summary, the discriminator output is defined
as:

$\displaystyle
D({\mathbf{x}},{\mathbf{h}}^{\prime})=\underbrace{f_{\text{d}}({\mathbf{x}})}_{\text{real
or
fake}}+\underbrace{\langle{\mathbf{h}}^{\prime},f_{\text{s}}({\mathbf{x}})\rangle}_{\text{semantic
alignment}}~{},$ (4)

Intuitively, $D({\mathbf{x}},{\mathbf{h}}^{\prime})$ yields a high value for
an image ${\mathbf{x}}$, when it is real (with large
$f_{\text{d}}({\mathbf{x}})$ values) and the semantic similarity between
${\mathbf{h}}^{\prime}$ and $f_{\text{s}}({\mathbf{x}})$ is high. Similar
ideas have been exploited in [17, 15, 56]. Different from these methods, our
model can utilize the pre-trained multi-modal feature space, which relieves
the difficulty for discriminator in learning semantically meaningful features.

Figure 3: The process of injecting text-conditional information into each layer of the generator, where FC denotes fully-connected layer. The green modules have their own trainable parameters per generator layer. We can view the original StyleGAN2 constructs its StyleSpace as the process from ${\mathbf{z}}$ to ${\mathbf{s}}$. We propose to inject the semantic conditional information and further build our Conditional StyleSpace, whose elements ${\mathbf{u}}$ will be used to modulate image generation. This figure illustrates the language-free setting, where real image is used to generate pseudo text feature ${\mathbf{h}}^{\prime}$; For the fully supervised text-to-image generation setting, real text is used for the extraction of text feature ${\mathbf{h}}$. Please refer to the definition of translator in Section 3 for details. |  |
---|---|---
(a) Discriminator output | (b) $\mathcal{L}_{\text{ConD}}$ | (c) $\mathcal{L}_{\text{ConG}}$
Figure 4: Illustration of discriminator outputs and training objectives for
the language-free setting.

###  3.3 Training Objectives

For a mini-batch of $n$ images $\\{{\mathbf{x}}_{i}\\}_{i=1}^{n}$,
${\mathbf{h}}^{\prime}_{i}$ is the corresponding generated pseudo text
features for the $i$-th image. Our model is trained in an adversarial manner,
with additional contrastive losses to ensure that the GAN feature space is
aligned with pre-trained CLIP. The first one is the standard conditional GAN
loss. The losses for the generator and discriminator are defined, with the
logits from (4), as:

$\displaystyle\mathcal{L}_{\text{G}}$
$\displaystyle=-\sum_{i=1}^{n}\log\sigma(D({\mathbf{x}}^{\prime}_{i},{\mathbf{h}}_{i}^{\prime})),$
(5) $\displaystyle\mathcal{L}_{\text{D}}$
$\displaystyle=-\sum_{i=1}^{n}\log\sigma(D({\mathbf{x}}_{i},{\mathbf{h}}_{i}^{\prime}))-\sum_{i=1}^{n}\log(1-\sigma(D({\mathbf{x}}^{\prime}_{i},{\mathbf{h}}_{i}^{\prime})))$

where $\sigma(\cdot)$ denotes the Sigmoid function.

To enforce that the discriminator-extracted feature
$f_{\text{s}}({\mathbf{x}})$ is semantically aligned in the pre-trained CLIP
feature space, we consider the following contrastive regularizer for the
discriminator:

$\displaystyle\mathcal{L}_{\text{ConD}}=-\tau\sum_{i=1}^{n}\log\dfrac{\exp(\mathrm{Sim}(f_{\text{s}}({\mathbf{x}}_{i}),{\mathbf{h}}_{i}^{\prime})/\tau)}{\sum_{j=1}^{n}\exp(\mathrm{Sim}(f_{\text{s}}({\mathbf{x}}_{j}),{\mathbf{h}}_{i}^{\prime})/\tau)},$
(6)

where $\mathrm{Sim}$ denotes the cosine similarity, $\tau$ is a non-negative
hyper-parameter. Intuitively, $\mathcal{L}_{\text{ConD}}$ enforces the
discriminator to output image feature $f_{\text{s}}({\mathbf{x}}_{i})$ that is
similar to the corresponding text feature ${\mathbf{h}}^{\prime}_{i}$.

We further utilize the pre-trained CLIP model to improve the semantic
correspondence of the generated images ${\mathbf{x}}_{i}^{\prime}$ and its
conditioned pseudo text feature ${\mathbf{h}}^{\prime}_{i}$. We define the
following contrastive loss for the generator with the same hyper-parameter
$\tau$ as (6):

$\displaystyle\mathcal{L}_{\text{ConG}}=-\tau\sum_{i=1}^{n}\log\dfrac{\exp(\mathrm{Sim}(f_{\text{img}}({\mathbf{x}}^{\prime}_{i}),{\mathbf{h}}_{i}^{\prime})/\tau)}{\sum_{j=1}^{n}\exp(\mathrm{Sim}(f_{\text{img}}({\mathbf{x}}^{\prime}_{j}),{\mathbf{h}}_{i}^{\prime})/\tau)}.$
(7)

With the above contrastive regularizers, the final training loss for the
generator and discriminator are defined as:

$\displaystyle\mathcal{L}_{\text{D}}^{\prime}$
$\displaystyle=\mathcal{L}_{\text{D}}+\gamma\mathcal{L}_{\text{ConD}}$ (8)
$\displaystyle\mathcal{L}_{\text{G}}^{\prime}$
$\displaystyle=\mathcal{L}_{\text{G}}+\gamma\mathcal{L}_{\text{ConD}}+\lambda\mathcal{L}_{\text{ConG}}$
(9)

where $\tau=0.5,\lambda=\gamma=10$ for language-free settings, and
$\tau=0.5,\lambda=10$, $\gamma=5$ for fully-supervised settings333Details
about hyper-parameter tuning are provided in the Appendix..

###  3.4 Training Details

Algorithm 1 Language-free training of Lafite

1: Input: An image dataset $\\{{\mathbf{x}}_{i}\\}_{i=1}^{N}$, pre-trained
encoders $f_{\text{txt}},f_{\text{img}}$, hyper-parameters $\tau>0$

2: while not converge do

3: Sample mini-batch $\\{{\mathbf{x}}_{i}\\}_{i=1}^{n}$;

4: Sample perturbation noise
$\\{\mathbf{\epsilon}_{i}\\}_{i=1}^{n}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$;

5: // Pseudo text feature generation

6: Generate ${\mathbf{h}}^{\prime}_{i}$ according to (1) or (2);

7: // Forward pass of G and D

8: Sample latent noise
$\\{\mathbf{{\mathbf{z}}}_{i}\\}_{i=1}^{n}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$;

9: Synthesize fake image ${\mathbf{x}}_{i}^{\prime}$ with G using (3);

10: Feed real/fake images to D using (4);

11: // Update G and D with gradient descent

12: Update D with (8);

13: Update G with (9);

14: end while

We summarize the language-free training schedule of Lafite in Algorithm 1. For
the settings with full image-text pairs, one may replace pseudo text feature
generation step with the ground-truth text feature
${\mathbf{h}}=f_{\text{txt}}({\mathbf{t}})$.

#### Pre-training.

To demonstrate the zero-shot task transfer ability of our model, we also
consider a variant that is pre-trained on the Google Conceptual Captions 3M
(CC3M) dataset [41], which consists of 3.3 millions of image-text pairs. For
pseudo text-feature generation with trainable perturbation, we also train its
inference model on CC3M. There is no image overlapping between the pre-
training and downstream datasets, which ensures the fairness when comparing
our method against others in transfer learning. For face domain, we pre-
trained a model on FFHQ dataset [19] which contains 70,000 images. The pre-
trained models can be fine-tuned with Lafite under language-free setting on
different datasets, which will be discussed in next section.

#### Data augmentation.

In practice, we also consider image data augmentation to improve extracted
image features $f_{\text{img}}({\mathbf{x}})$ in (1). We choose to use random
cropping and avoid using augmentations like color transformation, because they
may lead to mismatching between ${\mathbf{h}}^{\prime}$ and ${\mathbf{x}}$.
The details are summarized in Appendix.

##  4 Experiments

As the proposed Lafite is a versatile system, we conduct experiments under
different settings, including the proposed language-free setting, as well as
the zero-shot and fully-supervised text-to-image generation settings. Due to
the difference of two schemes to generate pseudo text features described in
Section 3.1, we denote our system in two variants: fixed perturbations as
$\textsc{Lafite}_{\text{G}}$ and trainable perturbations as
$\textsc{Lafite}_{\textsc{NN}}$, respectively. All of our experiments are
conducted on 4 Nvidia Tesla V100 GPUs, implemented using Pytorch [34]. CLIP-
ViT/B-32 is used in our methods unless specified. All the codes and pre-
trained models will be publicly available upon acceptance.

#### Datasets.

We consider a suite of datasets that are commonly used in literature [53, 59,
56, 54], including MS-COCO [4], CUB [47], LN-COCO [36], Multi-modal CelebA-HQ
(MM CelebA-HQ) [52]. All the images are scaled to resolution $256\times 256$.
Statistics of these datasets are summarized in Table 7 in the Appendix.

#### Evaluation metrics.

Following [38, 7], we report the blurred Fréchet Inception Distance (FID) [12]
and Inception Score (IS) [40] on MS-COCO dataset, which are computed using
30,000 generated images with randomly sampled text from validation set.
FID-$k$ means the FID is computed after blurring all the images by a Gaussian
filter with radius $k$.

Figure 5: Language-free text-to-image generation examples on MS-COCO
validation set.  Figure 6: Image generation with multi-modal conditions
(conditioned on both image and text).

###  4.1 Language-free Text-to-image Generation

We first study Lafite under the proposed language-free setting, in which only
images are provided in a given domain, and no paired caption is available
during training.

#### Captioning-based baseline:

As a baseline, we employed the SoTA image captioning model VinVL [57] to
generate some associated captions for images. Note that MS-COCO image-text
pairs were used to train the author-provided VinVL image captioning model, so
the MS-COCO comparison is unfairly biased in favor of the baseline due to this
information leakage. We compare this baseline method with our Lafite using the
same network architecture and hyper-parameter setting for fairness.

Model | IS $\uparrow$ | FID-0 $\downarrow$ | FID-1 $\downarrow$ | FID-2 $\downarrow$ | FID-4 $\downarrow$ | FID-8 $\downarrow$
---|---|---|---|---|---|---
Cap-Base | $15.83$ | $56.36$ | $54.99$ | $51.84$ | $44.81$ | $37.28$
Cap-Large | $16.95$ | $47.21$ | $42.35$ | $37.85$ | $31.59$ | $23.49$
$\textsc{Lafite}_{\text{G}}$ | $\mathbf{27.20}$ | $\mathbf{18.04}$ | $\mathbf{17.80}$ | $\mathbf{17.68}$ | $\mathbf{16.16}$ | $\mathbf{14.52}$
$\textsc{Lafite}_{\textsc{NN}}$ | $22.23$ | $26.56$ | $26.48$ | $25.82$ | $23.90$ | $19.27$

Table 1: Results of language-free setting on MS-COCO dataset. ‘Cap’ indicates
a text-to-image generation baseline method based on VinVL captioning.

The main results are in Table 1. Both variants of our Lafite significantly
outperform the captioning-based baseline method. The simple
$\textsc{Lafite}_{\text{G}}$ performs the best on this dataset, indicating the
generality of the method. For $\textsc{Lafite}_{\textsc{NN}}$, note that CC3M
is used to train the inference model, thus there is no information leakage in
$\textsc{Lafite}_{\textsc{NN}}$ method as we test
$\textsc{Lafite}_{\textsc{NN}}$ on the MS-COCO dataset. Some generated
examples are provided in Figure 5, from which we can see that our Lafite leads
to text-aligned generation though no text data is used during training,
verifying the effectiveness of the proposed method.

Model | IS $\uparrow$ | FID-0 $\downarrow$ | FID-1 $\downarrow$ | FID-2 $\downarrow$ | FID-4 $\downarrow$ | FID-8 $\downarrow$ | SOA-C $\uparrow$ | SOA-I $\uparrow$
---|---|---|---|---|---|---|---|---
DALL-E | $17.90$ | $27.50$ | $28.00$ | $45.50$ | $83.50$ | $85.00$ | - | -
CogView | $18.20$ | $27.10$ | $19.40$ | $13.90$ | $19.40$ | $23.60$ | - | -
Lafite | $\mathbf{26.02}$ | $\mathbf{26.94}$ | $22.97$ | $18.70$ | $\mathbf{15.72}$ | $\mathbf{14.79}$ | $37.37$ | $54.25$

Table 2: Results of zero-shot setting on MS-COCO dataset, the model is pre-
trained with image-text pairs from CC3M dataset.

| MS-COCO | CUB | LN-COCO | MM CelebA-HQ
---|---|---|---|---
Model | IS $\uparrow$ | FID $\downarrow$ | SOA-C $\uparrow$ | SOA-I $\uparrow$ | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$
AttnGAN | $23.61$ | $33.10$ | $25.88$ | $39.01$ | $4.36$ | $23.98$ | $20.80$ | $51.80$ | - | 125.98
Obj-GAN | $24.09$ | $36.52$ | $27.14$ | $41.24$ | - | - | - | - | - | -
DM-GAN | $32.32$ | $27.34$ | $33.44$ | $48.03$ | $4.75$ | $16.09$ | - | - | - | 131.05
OP-GAN | $27.88$ | $24.70$ | $35.85$ | $50.47$ | - | - | - | - | - | -
DF-GAN | - | $21.42$ | - | - | $5.10$ | $14.81$ | - | - | - | 137.60
XMC-GAN | $30.45$ | $9.33$ | $50.94$ | $71.33$ | - | - | $28.37$ | $14.12$ | - | -
Lafite | $\mathbf{32.34}$ | $\mathbf{8.12}$ | $\mathbf{61.09}$ | $\mathbf{74.78}$ | $\mathbf{5.97}$ | $\mathbf{10.48}$ | $26.32$ | $\mathbf{11.78}$ | $\mathbf{2.93}$ | $\mathbf{12.54}$

Table 3: Standard text-to-image generation on CUB, LN-COCO and MM CelebA-HQ
datasets.

Furthermore, we can actually perform generation conditioned on images: For a
given image, we generate an image-conditioned pseudo text feature vector with
Lafite. Passing this pseudo text feature vector to $G$ leads to generated
images that are similar to the given image. Consequently, Lafite enables image
generation with multi-modal conditions, i.e. it can be conditioned on both
image and text simultaneously. The implementation details are discussed in the
Appendix. Some generated examples are provided in Figure 6, more results are
provided in the Appendix.

###  4.2 Zero-Shot Text-to-image Generation

Zero-shot is a setting to evaluate a pre-trained text-to-image generation
model, without training the model on any of downstream data. MS-COCO dataset
is used for evaluating our model pre-trained on CC3M. The main results are
shown in Table 2. Compared to DALL-E [38] and CogView [7], Lafite achieves
better quantitative results in most cases. We also emphasize that our model
has only 75 millions of trainable parameters, while DALL-E has over 12
billions of parameters. Arguably, our pre-training dataset CC3M is much
smaller444Though we acknowledge that Lafite is based on an off-the-shelf
discriminate model CLIP, which is trained on 400 million image-text pairs,
compared to the pre-training dataset used in DALL-E, which contains 250
millions of image-text pairs.

###  4.3 Standard Text-to-image Generation

We now consider the standard text-to-image generation task, where all the
ground-truth image-text pairs are provided during training. We compare Lafite
against a series of competitive systems: AttnGAN [53], Obj-GAN [25], DM-GAN
[59], OP-GAN [13], DF-GAN [44] and XMC-GAN [56]. The main results evaluated by
FID and IS on different datasets are provided in Table 3. We also report the
Semantic Object Accuracy (SOA) on MS-COCO following previous works [13, 56].
Results of competitive models are directly cited from the corresponding
papers. It is clear that our proposed model consistently outperforms all other
methods, creating new SoTA results in standard text-to-image generation.

| MS-COCO | CUB | LN-COCO | MM CelebA-HQ
---|---|---|---|---
Methods | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$
| Training from Scratch
$\textsc{Lafite}_{\text{G}}$ | $\mathbf{27.20}$ | $\mathbf{18.04}$ | $\mathbf{4.32}$ | $\mathbf{27.53}$ | $\mathbf{18.49}$ | $38.95$ | $2.78$ | $\mathbf{32.75}$
$\textsc{Lafite}_{\textsc{NN}}$ | $22.23$ | $26.56$ | $4.06$ | $46.32$ | $18.17$ | $\mathbf{36.19}$ | $\mathbf{2.89}$ | $50.34$
| Fine-tuned from Pre-trained Model
$\textsc{Lafite}_{\text{G}}$ | $24.89$ | $20.89$ | $\mathbf{6.13}$ | $\mathbf{35.99}$ | $19.32$ | $34.96$ | $3.10$ | $\mathbf{15.74}$
$\textsc{Lafite}_{\textsc{NN}}$ | $\mathbf{26.55}$ | $\mathbf{17.44}$ | $4.36$ | $37.91$ | $\mathbf{20.02}$ | $\mathbf{33.76}$ | $\mathbf{3.19}$ | $29.42$

Table 4: Comparisons between two schemes for language-free training on
different datasets.

###  4.4 Adaptation of Pre-trained Models

#### Language-free model fine-tuning.

Compared with existing works, one key advantage of the pre-trained Lafite
model is that it naturally enables language-free model fine-tuning. The
results are provided in Table 4, where both $\textsc{Lafite}_{\text{G}}$ and
$\textsc{Lafite}_{\textsc{NN}}$ are investigated on different datasets. We see
that fine-tuning from the pre-trained model generally outperform training from
scratch. We also notice that performance of pre-trained Lafite largely depends
on the domain gap in pre-training and fine-tuning datasets. For example,
$\textsc{Lafite}_{\textsc{NN}}$ sometimes obtains worse results than
$\textsc{Lafite}_{\text{G}}$, especially when the fine-tuning dataset is
dissimilar to CC3M, _i.e_., CUB and MM CelebA-HQ. This indicates that the
inference model used for generating text features may have biases, because it
may over-fit to its training dataset CC3M.

Pre-trained Lafite is also highly training-efficient. For example, training
from scratch with Lafite on MS-COCO dataset requires around 4 days to reach
FID of 18, while fine-tuning only needs 3 hours. This becomes a critical
advantage especially when we require several text-to-image generation models
across different datasets.

#### Semi-supervised fine-tuning.

Adaptation of pre-trained Lafite is sample-efficient. One interesting question
is, how much percentage of image-text pairs do we need to outperform previous
SoTA XMC-GAN on MS-COCO dataset? To answer this question, we conduct
experiment in which only a portion of the images are associated with ground-
truth text. Our model is first pre-trained using all the images under the
language-free setting, then it is fine-tuned with varying percentages of
image-text pairs. The main results are summarized in Figure 7. Our method
outperforms XMC-GAN on both IS and FID when less than half of total of the
image-text pairs are employed.

(a) FID $(\downarrow)$

(b) IS $(\uparrow)$

Figure 7: Comparison of Lafite and prior art XMC-GAN. X-axis is the percentage
of image-text pairs in the full MS-COCO dataset. XMC-GAN has over 166 millions
trainable parameters, while our Lafite only has 75 millions trainable
parameters.

###  4.5 Ablation Study

#### Ablation study of training objectives

We first investigate the impact of each component in our objective functions.
The standard generator and discriminator losses are always employed, we ablate
by excluding $\mathcal{L}_{\text{ConG}}$ and $\mathcal{L}_{\text{ConD}}$ one
by one. The results are provided in Table 5. For both variants of Lafite, it
is observed the model performance could drop significantly.

Model | $\mathcal{L}_{\text{ConG}}$ | $\mathcal{L}_{\text{ConD}}$ | IS $\uparrow$ | FID $\downarrow$ | SOA-C $\uparrow$ | SOA-I $\uparrow$
---|---|---|---|---|---|---
$\textsc{Lafite}_{\text{G}}$ |  |  | $14.79$ | $33.03$ | $9.64$ | $18.40$
✓ |  | $17.78$ | $29.65$ | $16.53$ | $30.33$
|  | ✓ | $22.28$ | $21.25$ | $29.09$ | $43.77$
| ✓ | ✓ | $\mathbf{27.20}$ | $\mathbf{18.04}$ | $\mathbf{36.84}$ | $\mathbf{54.16}$
$\textsc{Lafite}_{\textsc{NN}}$ |  |  | $11.05$ | $72.03$ | $8.28$ | $14.46$
✓ |  | $20.02$ | $30.67$ | $26.60$ | $41.26$
|  | ✓ | $19.14$ | $33.88$ | $33.32$ | $49.86$
| ✓ | ✓ | $\mathbf{22.23}$ | $\mathbf{26.48}$ | $\mathbf{36.86}$ | $\mathbf{54.02}$

Table 5: Ablations of training losses on MS-COCO dataset, $\checkmark$ means
the component is used during training.

Model | Feature dim | IS $\uparrow$ | FID $\downarrow$ | SOA-C $\uparrow$ | SOA-I $\uparrow$
---|---|---|---|---|---
RoBERTa-Base | $768$ | $15.95$ | $29.55$ | $11.58$ | $22.89$
RoBERTa-Large | $1024$ | $14.11$ | $35.77$ | $7.72$ | $16.03$
CLIP(B-32) Text encoder | $512$ | $24.54$ | $16.21$ | $47.74$ | $61.86$
CLIP(B-16) Text encoder | $512$ | $24.90$ | $15.97$ | $47.80$ | $62.71$
CLIP(B-32) | $512$ | $31.88$ | $8.62$ | $59.51$ | $73.76$
CLIP(B-16) | $512$ | $\mathbf{32.34}$ | $\mathbf{8.12}$ | $\mathbf{61.09}$ | $\mathbf{74.78}$

Table 6: Results of using different pre-trained models on MS-COCO dataset.

#### Ablations of pre-trained text/image encoders

To demonstrate the importance of using a multi-modal feature-aligned pre-
trained model in our Lafite, we compare the CLIP model and other single-
modality models. We adopt the popular RoBERTa [30] as the baseline text
encoder, which was trained on a large text corpus only. Note that it is
infeasible to perform language-free training without the joint feature space.
Thus this experiment is based on fully-supervised text-to-image generation
setting. For a fair comparison, we also report the results of only using the
text encoder of CLIP while discarding the image encoder. In this setting,
there is no image encoder thus the $\mathcal{L}_{\text{ConG}}$ term is removed
from the objective function consequently. The results are reported in Table 6.
As expected, even if the image encoder of CLIP is not used, models with only
CLIP text encoder still significantly outperform models using RoBERTa. From
the results, we can conclude that: $(i)$ The feature space of CLIP is
semantically meaningful for text-to-image generation, thus only using text
encoder of CLIP still leads to better results than RoBERTa; $(ii)$ Text-to-
image generation results can be improved by using a feature-aligned joint
feature space (CLIP vs others), and can be further improved with a stronger
joint space (CLIP-ViT/B-16 outperforms CLIP-ViT/B-32, where ViT/B-16 and
ViT/B-32 are different designs of visual transformers [8]).

##  5 Conclusion

We have presented Lafite, an approach to build text-to-image generation
systems without domain-specific image-text pairs in training. We achieve the
goal by resorting to generating pseudo text features from images. Excellent
performance in a variety of text-to-image generations tasks have demonstrated
the effectiveness of Lafite, including language-free, zero-shot and fully
supervised settings. In particular, Lafite creates new SoTA in zero-shot
setting, with only 1% trainable parameter counts compared with recent advances
such as DALL-E/CogView. Lafite also outperforms prior arts in the fully-
supervised settings. We believe that language-free training is a promising
direction to enable broader application areas for text-to-image generation, as
it significantly lowers the burden on data collection. One interesting future
direction is to explore image synthesis in the wild, where long tail and open
set conditions are provided for generation.

## References

  * [1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.  nocaps: novel object captioning at scale.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8948–8957, 2019.
  * [2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [3] Eungchun Cho.  Inner product of random vectors.  International Journal of Pure and Applied Mathematics, 56(2):217–221, 2009.
  * [4] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  In EMNLP, 2014.
  * [5] Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11162–11173, 2021.
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [7] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang.  Cogview: Mastering text-to-image generation via transformers, 2021.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [9] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  arXiv preprint arXiv:2108.00946, 2021.
  * [10] Federico A Galatolo, Mario GCA Cimino, and Gigliola Vaglini.  Generating images from caption and vice versa via clip-guided generative latent space search.  arXiv preprint arXiv:2102.01645, 2021.
  * [11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [13] Tobias Hinz, Stefan Heinrich, and Stefan Wermter.  Semantic object accuracy for generative text-to-image synthesis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020.
  * [14] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
  * [15] Jongheon Jeong and Jinwoo Shin.  Training gans with stronger augmentations via contrastive discriminator.  In International Conference on Learning Representations, 2020.
  * [16] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  arXiv preprint arXiv:2102.05918, 2021.
  * [17] Minguk Kang and Jaesik Park.  Contragan: Contrastive learning for conditional image generation.  Advances in Neural Information Processing Systems, 33:21357–21369, 2020.
  * [18] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [19] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [20] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of styleGAN.  arXiv preprint arXiv:1912.04958, 2019.
  * [21] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  arXiv preprint arXiv:2102.03334, 2021.
  * [22] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [23] Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin.  ALICE: Towards understanding adversarial learning for joint distribution matching.  In Advances in Neural Information Processing Systems, 2017.
  * [24] Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  arXiv preprint arXiv:2107.07651, 2021.
  * [25] Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao.  Object-driven text-to-image synthesis via adversarial training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12174–12182, 2019.
  * [26] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [28] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  In Advances in neural information processing systems, pages 700–708, 2017.
  * [29] Yunfan Liu, Qi Li, Zhenan Sun, and Tieniu Tan.  Style intervention: How to achieve spatial disentanglement with style-based generators?, 2020.
  * [30] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [31] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  arXiv preprint arXiv:1908.02265, 2019.
  * [32] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [33] Daniil Pakhomov, Sanchit Hira, Narayani Wagle, Kemar E Green, and Nassir Navab.  Segmentation in style: Unsupervised semantic image segmentation with stylegan and clip.  arXiv preprint arXiv:2107.12518, 2021.
  * [34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32:8026–8037, 2019.
  * [35] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  arXiv preprint arXiv:2103.17249, 2021.
  * [36] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari.  Connecting vision and language with localized narratives.  In European Conference on Computer Vision, pages 647–664. Springer, 2020.
  * [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [38] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  arXiv preprint arXiv:2102.12092, 2021.
  * [39] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In Advances in neural information processing systems, pages 14866–14876, 2019.
  * [40] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  Advances in neural information processing systems, 29:2234–2242, 2016.
  * [41] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [42] Richard Socher and Li Fei-Fei.  Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 966–973. IEEE, 2010.
  * [43] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  arXiv preprint arXiv:1908.08530, 2019.
  * [44] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao.  Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis, 2021.
  * [45] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309–6318, 2017.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * [47] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [48] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik.  Learning two-branch neural networks for image-text matching tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):394–407, 2018.
  * [49] Liwei Wang, Yin Li, and Svetlana Lazebnik.  Learning deep structure-preserving image-text embeddings.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5005–5013, 2016.
  * [50] Jason Weston, Samy Bengio, and Nicolas Usunier.  Large scale image annotation: learning to rank with joint word-image embeddings.  Machine learning, 81(1):21–35, 2010.
  * [51] Zongze Wu, Dani Lischinski, and Eli Shechtman.  Stylespace analysis: Disentangled controls for stylegan image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863–12872, 2021.
  * [52] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  Tedigan: Text-guided diverse face image generation and manipulation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [53] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [54] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji.  Improving text-to-image synthesis using contrastive learning.  arXiv preprint arXiv:2107.02423, 2021.
  * [55] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6720–6731, 2019.
  * [56] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation, 2021.
  * [57] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579–5588, 2021.
  * [58] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  arXiv preprint arXiv:2010.00747, 2020.
  * [59] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5802–5810, 2019.

##  Appendix A Appendix

###  A.1 Theoretical Results

######  Theorem 1.

For a given threshold $c>0$, the generated text feature by
$\textsc{Lafite}_{\text{G}}$ satisfies
$\text{Sim}(f_{\text{img}}({\mathbf{x}}_{i}),{\mathbf{h}}^{\prime}_{i})\geq c$
with probability at least

$\displaystyle\text{Prob}(\text{Sim}(f_{\text{img}}({\mathbf{x}}_{i}),{\mathbf{h}}^{\prime}_{i})\geq
c)$ $\displaystyle=$ $\displaystyle
1-\int_{-1}^{(c-1)/\xi+c}\dfrac{\Gamma(d/2+1/2)}{\sqrt{\pi}\Gamma(d/2)}(1-x^{2})^{d/2-1}\text{d}x$

where $d$ is the dimension number of features,
$\Gamma(z)=\int_{0}^{\infty}x^{z-1}e^{-x}\text{d}x$ is the Gamma function.

###### Proof.

Without loss of generality, we omit the subscript for clearness.

$\displaystyle\text{Sim}(f_{\text{img}}({\mathbf{x}}),{\mathbf{h}}^{\prime})$
$\displaystyle=$ $\displaystyle\dfrac{\langle
f_{\text{img}}({\mathbf{x}}),{\mathbf{h}}^{\prime}\rangle}{\|f_{\text{img}}({\mathbf{x}})\|_{2}\|{\mathbf{h}}^{\prime}\|_{2}}$
$\displaystyle=$ $\displaystyle\dfrac{\langle
f_{\text{img}}({\mathbf{x}}),f_{\text{img}}({\mathbf{x}})+\xi\epsilon\|f_{\text{img}}({\mathbf{x}})\|_{2}/\|\epsilon\|_{2}\rangle}{\|f_{\text{img}}({\mathbf{x}})\|_{2}\Big{\|}f_{\text{img}}({\mathbf{x}})+\xi\epsilon\|f_{\text{img}}({\mathbf{x}})\|_{2}/\|\epsilon\|_{2}\Big{\|}_{2}}$
$\displaystyle=$
$\displaystyle\dfrac{\|f_{\text{img}}({\mathbf{x}})\|^{2}+\xi\epsilon^{\intercal}f_{\text{img}}({\mathbf{x}})\|f_{\text{img}}({\mathbf{x}})\|_{2}/\|\epsilon\|_{2}}{\|f_{\text{img}}({\mathbf{x}})\|_{2}\Big{\|}f_{\text{img}}({\mathbf{x}})+\xi\epsilon\|f_{\text{img}}({\mathbf{x}})\|_{2}/\|\epsilon\|_{2}\Big{\|}_{2}}$

Denote
$a=f_{\text{img}}({\mathbf{x}})/\|f_{\text{img}}({\mathbf{x}})\|_{2},b=\epsilon/\|\epsilon\|_{2}$,
then we have

$\displaystyle\text{Sim}(f_{\text{img}}({\mathbf{x}}),{\mathbf{h}}^{\prime})$
$\displaystyle=$
$\displaystyle\dfrac{\|f_{\text{img}}({\mathbf{x}})\|^{2}+\xi\epsilon^{\intercal}f_{\text{img}}({\mathbf{x}})\|f_{\text{img}}({\mathbf{x}})\|_{2}/\|\epsilon\|_{2}}{\|f_{\text{img}}({\mathbf{x}})\|_{2}\Big{\|}f_{\text{img}}({\mathbf{x}})+\xi\epsilon\|f_{\text{img}}({\mathbf{x}})\|_{2}/\|\epsilon\|_{2}\Big{\|}_{2}}$
$\displaystyle=$ $\displaystyle\dfrac{1+\xi a^{\intercal}b}{\|a+\xi b\|_{2}}$
$\displaystyle\geq$ $\displaystyle\dfrac{1+\xi
a^{\intercal}b}{\|a\|_{2}+\xi\|b\|_{2}}$ $\displaystyle=$
$\displaystyle\dfrac{1+\xi a^{\intercal}b}{1+\xi}$

Consequently,

$\displaystyle\text{Prob}(\text{Sim}(f_{\text{img}}({\mathbf{x}}_{i}),{\mathbf{h}}^{\prime}_{i})\geq
c)$ $\displaystyle\geq$ $\displaystyle\text{Prob}(\dfrac{1+\xi
a^{\intercal}b}{1+\xi}\geq c)$ $\displaystyle=$
$\displaystyle\text{Prob}(1+\xi a^{\intercal}b\geq c+c\xi)$ $\displaystyle=$
$\displaystyle\text{Prob}(a^{\intercal}b\geq(c-1+c\xi)/\xi)$

By the cumulative distribution function (CDF) of inner product of random
vectors on sphere [3], we know that

$\displaystyle\text{Prob}(a^{\intercal}b\leq
z)=\int_{-1}^{z}\dfrac{\Gamma(d/2+1/2)}{\sqrt{\pi}\Gamma(d/2)}(1-x^{2})^{d/2-1}\text{d}x$

where $d$ is the dimension number of features,
$\Gamma(z)=\int_{0}^{\infty}x^{z-1}e^{-x}\text{d}x$ is the Gamma function.
Thus we have

$\displaystyle\text{Prob}(\text{Sim}(f_{\text{img}}({\mathbf{x}}_{i}),{\mathbf{h}}^{\prime}_{i})\geq
c)$ $\displaystyle\geq$
$\displaystyle\text{Prob}(a^{\intercal}b\geq(c-1+c\xi)/\xi)$ $\displaystyle=$
$\displaystyle
1-\int_{-1}^{(c-1)/\xi+c}\dfrac{\Gamma(d/2+1/2)}{\sqrt{\pi}\Gamma(d/2)}(1-x^{2})^{d/2-1}\text{d}x,$

which completes the proof. ∎

###  A.2 Experiment Details

#### Datasets

The statistics of datasets are summarized in Table 7.

Algorithm 2 Image feature extraction process

1: Input: An image dataset $\\{{\mathbf{x}}_{i}\\}_{i=1}^{N}$, image
resolution $w\times w$, pre-trained $f_{\text{img}}$, hyper-parameters
$a>0,k\geq 1$

2: // Image feature generation

3: for $i=1$ to $n$ do

4: if use data augmentation then

5: Initialize ${\mathbf{h}}^{\prime}_{i}\leftarrow\mathbf{0}$;

6: for $j=1$ to k do

7:
${\mathbf{h}}^{\prime}_{i}\leftarrow{\mathbf{h}}^{\prime}_{i}+f_{\text{img}}(\text{CROP}({\mathbf{x}}_{i}))$,
where $\text{CROP}(\cdot)$ denotes randomly cropping image to be
$w^{\prime}\times w^{\prime}$, $w^{\prime}$ is an integer randomly sampled
from the range $\left[a,w\right]$;

8: end for

9: ${\mathbf{h}}^{\prime}_{i}\leftarrow{\mathbf{h}}^{\prime}_{i}/k$;

10: else

11: Initialize ${\mathbf{h}}^{\prime}_{i}\leftarrow
f_{\text{img}}({\mathbf{x}}_{i})$;

12: end if

13: end for

Dataset | #train | #validation | caption/image
---|---|---|---
MS-COCO | 82k | 40k | 5
CUB | 9k | 3k | 10
LN-COCO | 134k | 8k | 1
MM CelebA-HQ | 24k | 6k | 10

Table 7: Statistics of datasets. The last column indicates ratio of captions
vs images.

#### Image feature extraction

In practice, we use random cropping as data augmentation when we extract the
image features, which is presented in Algorithm 2. The pseudo text features
will be generated by perturb the average feature of augmented samples. In our
implementation, we set $k=1,a=256$ to extract image features used in
generating ${\mathbf{h}}^{\prime}$, while we set $k=1,a=128$ in contrastive
loss (7).

#### Hyper-parameter

The hyper-parameters are selected based on the performance on MS-COCO dataset.
Specifically, $\tau$ is selected from $\left[0.1,0.2,0.5,1.0,2.0\right]$,
$\lambda,\gamma$ are selected from $\left[0,1,2,5,10,20,50\right]$.

#### Exponential sharpening

In practice, we found that applying an extra exponential sharpening in
contrastive loss makes it easier to reproduce the experiment results, i.e. we
add an extra exponential operation right before the softmax function in (6)
and (7). Our implementation can be found at https://github.com/drboog/Lafite.

###  A.3 More Results

We provide the implementation details of image generation with multi-modal
conditions, an ablation study on discriminator, and more generated examples
under language-free setting.

#### Generation with multi-modal conditions

To generate an image conditioned on both a reference image and text
description, we first extract the text feature ${\mathbf{h}}_{1}$ from the
given text, and pseudo text feature ${\mathbf{h}}^{\prime}_{2}$ from the
image. Then ${\mathbf{h}}_{1},{\mathbf{h}}^{\prime}_{2}$ will be feed into the
pre-trained generator, leading to two conditional style codes
${\mathbf{u}}_{1}$ and ${\mathbf{u}}_{2}$. We construct a new conditional
style code, whose elements are randomly selected from the corresponding
elements in either ${\mathbf{u}}_{1}$ or ${\mathbf{u}}_{2}$. The new
conditional style code will be fed into the generator to generate the desired
image.

Note that generation conditioned on image is not reconstruction. Thus when
only a reference image is provided, the generated image may have differences
with the given image. However, they will share some visible characteristics
that are semantic meaningful as illustrated in our examples.

#### Ablation study on discriminator

We test the impact of each term of 4 under language-free setting. The results
are provided in Table 8, from which we can see that both terms are important,
while the ”real or fake” term seems to be more important.

RoF | SA | FID $\downarrow$ | IS $\uparrow$ | SOA-C $\uparrow$ | SOA-I $\uparrow$
---|---|---|---|---|---
✓ |  | $24.85$ | $23.74$ | $30.54$ | $48.72$
| ✓ | $25.42$ | $21.14$ | $23.14$ | $38.32$
✓ | ✓ | $18.04$ | $27.20$ | $36.84$ | $54.16$
Table 8: Ablation study on discriminator logits in language-free setting. RoF
denotes “real or fake” term, SA denotes “semantic alignment” term.

#### Generated examples

Some text-to-image generation results on CUB, MS-COCO, MM CelebA-HQ, LN-COCO
are provided in the following figures.

Figure 8: Generating examples on MS-COCO dataset. Figure 9: Generating
examples on CUB dataset. Figure 10: Generating examples on MM CelebA-HQ
dataset. Figure 11: Generating examples on LN-COCO dataset. Figure 12:
Generating images with multi-modal conditions (conditioned on both image and
text) on MS-COCO dataset. Figure 13: Generating images with multi-modal
conditions (conditioned on both image and text) on CUB dataset. Figure 14:
Generating images with multi-modal conditions (conditioned on both image and
text) on MM CelebA-HQ dataset. Figure 15: Generating images with multi-modal
conditions (conditioned on both image and text) on LN-COCO dataset.