# Cross-spectral purity of nonstationary vector optical fields: A similarity
with stationary fields

Rajneesh Joshi, 1 and Bhaskar Kanseri 2* 1 Department of Physics, Laxman Singh
Mahar Campus, Pithoragarh-262502, Soban Singh Jeena University Almora,
Uttarakhand, India
2Experimental Quantum Interferometry and Polarization (EQUIP), Department of
Physics, Indian Institute of Technology Delhi, Hauz Khas, New Delhi-110016,
India
*Corresponding author:-<EMAIL_ADDRESS>
###### Abstract

This study establishes a reduction formula for nonstationary cross-spectrally
pure vector light fields with any spectral bandwidth. The formation of a
reduction formula, analogous to that for stationary fields, does not apply to
the normalized two-time Stokes parameters of a nonstationary field that is
cross-spectrally pure. The current formula incorporates time-integrated
coherence parameters to ensure cross-spectral purity. The reduction formula
derived for nonstationary vector light fields with arbitrary spectral
bandwidth shares a similar mathematical structure to that of reduction
formulas used for stationary vector fields. Additionally, we examine the
requirement of strict cross-spectral purity for using a time-integrated
coherence function, which exhibits a mathematical expression similar to that
of strict cross-spectral purity in stationary vector fields. This
investigation sheds light on the cross-spectral purity of pulse-type fields,
which holds potential applications in the field of statistical optics.

††journal: Optics Communications

##  1 Introduction

Cross-spectral purity (CSP) is a fundamental concept in classical optics that
is extensively studied using Young’s type interferometer (YI) in the case of
stationary light fields [1, 2]. When the identical normalized spectra of light
at two spatial input points, denoted as $\textbf{r}_{1}$ and $\textbf{r}_{2}$,
interfere within the YI, the resulting normalized spectrum on the observation
plane differs from the input spectra, indicating cross-spectrally impure
fields [1]. However, under specific mathematical conditions, the normalized
spectra at input points $\textbf{r}_{1}$ and $\textbf{r}_{2}$ can resemble the
normalized spectra at a particular point R on the observation plane,
representing cross-spectrally pure optical fields [1].

L. Mandel initially proposed CSP for stationary scalar light fields in 1961,
where he observed an intriguing reduction in the complex degree of coherence
(DOC) [1]. Subsequently, it was discovered that the absolute value of the
complex spectral DOC remains the same across all frequencies of the optical
field [3]. Various techniques have been explored to investigate CSP in
stationary scalar fields [4, 5, 6, 7, 8, 9, 10], and the phenomenon has also
been studied in the context of scattering and ghost imaging [11, 12].

Subsequently, the concept of CSP was investigated for stationary vector light
fields, specifically electromagnetic (EM) fields, considering all the
polarization Stokes parameters [13, 14]. Similar to the scalar case, a
reduction formula exists for vector fields. In other words, the normalized
space-time coherence Stokes parameters, divided by the corresponding usual
Stokes parameters, can be expressed as the product of spatial and temporal
coherence functions. Additionally, the absolute value of the normalized space-
frequency coherence Stokes parameters, divided by the corresponding usual
Stokes parameters, remains constant across all frequencies of the optical
field. Strict CSP is another important aspect of optical fields [14, 15]. When
all the Stokes parameters exhibit CSP at a single point on the observation
plane of the YI, it is referred to as strict CSP. The equivalence between the
space-time and space-frequency electromagnetic degree of coherence (EMDOC) and
the degree of cross-polarization (DOCP) serves as a hallmark of strict CSP
[14, 16]. Various approaches have also been explored in investigating CSP for
stationary vector fields [16, 17, 18, 19].

##  2 CSP of nonstationary light fields

The most general form of the light field in optics is nonstationary or pulsed-
like optical fields. Previous studies have examined their fundamental
properties [20, 21, 22, 23, 24, 25]. These fields exhibit characteristics
related to coherence and polarization, which are influenced by specific time
instants ($t_{1}$, $t_{2}$) or frequencies ($\omega_{1}$, $\omega_{2}$), as
well as the time difference ($\Delta t$). A wavefront folding interferometer
(WFI) [26, 27] and scanning wavefront folding interferometers [28] are used
instead of YI in studying CSP for nonstationary light fields to avoid the
frequency-dependent factors and spectral interference in YI. The concept of
CSP was initially introduced by M. Koivurove et al. in 2019 for nonstationary
scalar light fields [29]. In CSP, the normalized spectral density remains the
same at input spatial points $\textbf{r}_{1}$, $\textbf{r}_{2}$, and a point R
on the observation plane of the WFI. Furthermore, the two-frequency complex
DOC should also be identical at $\textbf{r}_{1}$, $\textbf{r}_{2}$, and R.
These conditions can be satisfied if the two-frequency complex DOC can be
expressed as the product of space and frequency correlation functions.
Similarly, the two-time complex DOC can be expressed as the product of space
and time correlation functions.

Later, the phenomenon of CSP was expanded to nonstationary vector fields in
both the domains of space-time and space-frequency [30]. The normalized Stokes
parameters for CSP exhibit identical values at positions $\textbf{r}_{1}$,
$\textbf{r}_{2}$, and the point R. Moreover, the two-frequency Stokes
parameters normalized by their corresponding usual Stokes parameters also
share the same values at $\textbf{r}_{1}$, $\textbf{r}_{2}$, and R. These
criteria are met when the two-frequency Stokes parameters, normalized by their
corresponding usual Stokes parameters, can be represented as a product of
spatial and frequency correlation functions. Similarly, the two-time Stokes
parameters, normalized by their corresponding usual Stokes parameters, are
multiples of spatial and temporal correlation functions. The idea of strict
CSP, originally developed for stationary vector fields, is expanded to
nonstationary vector fields in the context of CSP [30]. If the two-frequency
Stokes parameters, when normalized by the zeroth usual Stokes parameters,
remain independent of frequency, and the two-time Stokes parameters,
normalized by the zeroth usual Stokes parameters, remain independent of time,
then it can be stated equivalently that the spatial components of both
parameters are equal. This condition is referred to as the strict CSP of
nonstationary vector fields [30]. Unlike stationary vector fields, strict CSP
is only observed at a time difference of zero for nonstationary vector fields.
[30]. The mathematical characteristics of the coherence function in CSP for
nonstationary scalar and vector fields differ from those observed in
stationary fields [29, 30]. However, a recent development involves deriving a
reduction formula for the coherence function of nonstationary scalar fields by
incorporating the time-integrated coherence function, which exhibits
similarities to the results obtained for stationary scalar fields [31]. This
advancement enables the extension of CSP to the EM domain, where polarization
properties can be considered.

In this paper, we examine the existence of a reduction formula for the CSP of
nonstationary vector fields, which encompasses all the Stokes parameters. Our
approach involves making use of time-integrated coherence functions.
Furthermore, a condition for strict CSP involving time-integrated quantities
is derived, exhibiting a mathematical form akin to the strict CSP observed in
stationary vector fields. This is a novel approach to finding the conditions
of CSP and strict CSP for nonstationary (modulated and fluctuating light beams
as well as ultrashort laser pulses) vector fields, which are the most general
fields and have numerous applications in science [32, 33].

Figure 1: Schematic configuration to study the cross-spectral purity of
nonstationary vector field, $\textbf{r}_{1}$, $\textbf{r}_{2}$ are spatial
input points, and R is a point on the output plane. The interferometer
produces a laterally sheared wavefront of the initial wavefront.

##  3 CSP of Stokes parameters

We consider the superposition of EM light fields through a WFI as shown in
Fig. 1. If two EM fields at spatial input positions $\textbf{r}_{1}$ and
$\textbf{r}_{2}$ interfere, the net output field at point R and time t can be
expressed as [30]

$\begin{bmatrix}E_{x}(\textbf{R},t)\\\
E_{y}(\textbf{R},t)\end{bmatrix}=\begin{bmatrix}E_{x}(\textbf{r}_{1},t-\tau_{1})\\\
E_{y}(\textbf{r}_{1},t-\tau_{1})\end{bmatrix}+\begin{bmatrix}E_{x}(\textbf{r}_{2},t-\tau_{2})\\\
E_{y}(\textbf{r}_{2},t-\tau_{2})\end{bmatrix},$ (1)

where $\tau_{1}=\frac{l_{1}}{c}$, $\tau_{2}=\frac{l_{2}}{c}$ are the times of
light to reach point R from $\textbf{r}_{1}$ and $\textbf{r}_{2}$,
respectively. The distances $l_{1}$ and $l_{2}$ represent the measurements
between point R and the input points $\textbf{r}_{1}$ and $\textbf{r}_{2}$,
respectively. In the space-frequency domain, the net field at the output of
WFI can be expressed as the combination of fields at two spatial positions in
the input. The output field becomes [30]

$\begin{bmatrix}E_{x}(\textbf{R},\omega)\\\
E_{y}(\textbf{R},\omega)\end{bmatrix}=\begin{bmatrix}E_{x}(\textbf{r}_{1},\omega)\\\
E_{y}(\textbf{r}_{1},\omega)\end{bmatrix}\exp(\iota\omega\tau_{1})+\begin{bmatrix}E_{x}(\textbf{r}_{2},\omega)\\\
E_{y}(\textbf{r}_{2},\omega)\end{bmatrix}\exp(\iota\omega\tau_{2}),$ (2)

where $\iota$ (iota) represents the imaginary number, $\omega$ denotes the
angular frequency. Assuming that the input spectral EM field is given as

$E_{i}(\textbf{r},\omega)=E_{0_{i}}(\textbf{r},\omega)\exp[\iota\phi_{i}(\textbf{r},\omega)],(i=x,y),$
(3)

where $E_{0_{i}}(\textbf{r},\omega)$ and $\phi_{i}(\textbf{r},\omega)$ denote
the complex amplitude and phase of the optical field, respectively. The form
of the wavefront phase is as $\phi_{i}(\textbf{r},\omega)=K\phi_{i}(r)$,
$k=\frac{\omega}{c}$ is wave vector and $\phi_{i}(r)$ represents the shape of
the wavefront [31]. Finally, from this relation, we readily find that

$\phi_{i}(\textbf{r},\omega)=\frac{\omega}{\omega_{0}}\phi_{i}(\textbf{r},\omega_{0}),(i=x,y),$
(4)

where $\omega_{0}$ denotes a fixed peak frequency of the spectrum. Using Eqs.
(3) and (4), Eq. (2) takes the form

$E_{i}(\textbf{R},\omega)=[E_{0_{i}}(\textbf{r}_{1},\omega)+E_{0_{i}}(\textbf{r}_{2},\omega)\exp(\iota\omega\tau_{i}(\textbf{r}_{1},\textbf{r}_{2}))]\exp[\iota\omega(\tau_{1}+\frac{\phi_{i}(\textbf{r}_{1},\omega_{0})}{\omega_{0}})],$
(5)

where

$\tau_{i}(\textbf{r}_{1},\textbf{r}_{2})=\frac{\phi_{i}(\textbf{r}_{2},\omega_{0})-\phi_{i}(\textbf{r}_{1},\omega_{0})}{\omega_{0}}+\tau_{2}-\tau_{1},$
(6a)
$\tau_{i}(\textbf{r}_{1},\textbf{r}_{2})=(\tau_{0})_{i}(\textbf{r}_{1},\textbf{r}_{2})+\Delta\tau.$
(6b)

As previously discussed, there are two conditions for the CSP of nonstationary
vector fields. However, our findings and analysis focus solely on the first
condition of CSP. Specifically, for nonstationary vector light fields, the
first condition of CSP for the Stokes parameters can be described as
follows:[30]

$s_{j}(\textbf{r}_{1},\omega)=s_{j}(\textbf{r}_{2},\omega)=s_{j}(\textbf{R},\omega),(j=0-3),$
(7)

where

$s_{j}(\textbf{r},\omega)=\frac{S_{j}(\textbf{r},\omega)}{\int_{0}^{\infty}S_{j}(\textbf{r},\omega)d\omega}$
(8)

are the Stokes parameters normalized with the corresponding usual Stokes
parameters. Keeping in mind that the spectral Stokes parameters
$S_{j}(\textbf{r},\omega)$ are defined as follows: [2]

$S_{0}(\textbf{r},\omega)=\langle
E^{*}_{x}(\textbf{r},\omega)E_{x}(\textbf{r},\omega)\rangle+\langle
E^{*}_{y}(\textbf{r},\omega)E_{y}(\textbf{r},\omega)\rangle,$ (9a)
$S_{1}(\textbf{r},\omega)=\langle
E^{*}_{x}(\textbf{r},\omega)E_{x}(\textbf{r},\omega)\rangle-\langle
E^{*}_{y}(\textbf{r},\omega)E_{y}(\textbf{r},\omega)\rangle,$ (9b)
$S_{2}(\textbf{r},\omega)=\langle
E^{*}_{y}(\textbf{r},\omega)E_{x}(\textbf{r},\omega)\rangle+\langle
E^{*}_{x}(\textbf{r},\omega)E_{y}(\textbf{r},\omega)\rangle,$ (9c)
$S_{3}(\textbf{r},\omega)=\iota[\langle
E^{*}_{y}(\textbf{r},\omega)E_{x}(\textbf{r},\omega)\rangle-\langle
E^{*}_{x}(\textbf{r},\omega)E_{y}(\textbf{r},\omega)\rangle],$ (9d)

here $\langle E^{*}_{i}(\textbf{r},\omega)E_{j}(\textbf{r},\omega)\rangle$,
$(i,j=x,y)$ are the elements of the coherency matrix $J(\textbf{r},\omega)$.
The asterisk and angular brackets in Eq. (9) indicate the complex conjugation
and time average, respectively. By substituting the values of field components
from Equation (5) into (9) and further assuming that both components of the EM
field share the same phase, denoted as
$\tau_{x}(\textbf{r}_{1},\textbf{r}_{2})=\tau_{y}(\textbf{r}_{1},\textbf{r}_{2})=\tau(\textbf{r}_{1},\textbf{r}_{2})$,
we can determine the Stokes parameters at point R as follows:

$S_{j}(\textbf{R},\omega)=S_{j}(\textbf{r}_{1},\omega)+S_{j}(\textbf{r}_{2},\omega)+2Re[(S_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)\exp(\iota\omega\tau(\textbf{r}_{1},\textbf{r}_{2}))],(j=0-3),$
(10)

where the two-frequency Stokes parameters in terms of polarization amplitudes
are defined as [22]

$(S_{0})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\langle
E^{*}_{0_{x}}(\textbf{r}_{1},\omega_{1})E_{0_{x}}(\textbf{r}_{2},\omega_{2})\rangle+\langle
E^{*}_{0_{y}}(\textbf{r}_{1},\omega_{1})E_{0_{y}}(\textbf{r}_{2},\omega_{2})\rangle,$
(11a)
$(S_{1})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\langle
E^{*}_{0_{x}}(\textbf{r}_{1},\omega_{1})E_{0_{x}}(\textbf{r}_{2},\omega_{2})\rangle-\langle
E^{*}_{0_{y}}(\textbf{r}_{1},\omega_{1})E_{0_{y}}(\textbf{r}_{2},\omega_{2})\rangle,$
(11b)
$(S_{2})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\langle
E^{*}_{0_{y}}(\textbf{r}_{1},\omega_{1})E_{0_{x}}(\textbf{r}_{2},\omega_{2})\rangle+\langle
E^{*}_{0_{x}}(\textbf{r}_{1},\omega_{1})E_{0_{y}}(\textbf{r}_{2},\omega_{2})\rangle,$
(11c)
$(S_{3})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\iota[\langle
E^{*}_{0_{y}}(\textbf{r}_{1},\omega_{1})E_{0_{x}}(\textbf{r}_{2},\omega_{2})\rangle-\langle
E^{*}_{0_{x}}(\textbf{r}_{1},\omega_{1})E_{0_{y}}(\textbf{r}_{2},\omega_{2})\rangle],$
(11d)

where the elements of the two-frequency cross-spectral density matrix are
represented by
$(W_{ij})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\langle
E^{*}_{0_{i}}(\textbf{r}_{1},\omega_{1})E_{0_{j}}(\textbf{r}_{2},\omega_{2})\rangle$,
$(i,j=x,y)$. The first equality of the CSP condition in Eq. (7) can be
expressed as follows: [30]

$S_{j}(\textbf{r}_{2},\omega)=C(\textbf{r}_{1},\textbf{r}_{2})S_{j}(\textbf{r}_{1},\omega),$
(12)

where $C(\textbf{r}_{1},\textbf{r}_{2})$ is a frequency independent
proportionality factor. Utilizing Eq. (12), Eq. (10) can be expressed as
follows:

$S_{j}(\textbf{R},\omega)=S_{j}(\textbf{r}_{1},\omega)[1+C(\textbf{r}_{1},\textbf{r}_{2})+2\sqrt{C(\textbf{r}_{1},\textbf{r}_{2})}Re[(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)\exp(\iota\omega\tau(\textbf{r}_{1},\textbf{r}_{2}))]],$
(13)

where the definition of
$(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})$ can be
expressed as [30]

$(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\frac{(S_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})}{\sqrt{S_{j}(\textbf{r}_{1},\omega_{1})}\sqrt{S_{j}(\textbf{r}_{2},\omega_{2})}},$
(14)

known as two-frequency Stokes parameters normalized by corresponding usual
Stokes parameters, which is a complex quantity and can be represented as

$(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=|(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})|\exp[\iota\alpha_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})],$
(15)

where $\alpha_{j}$ is phase of
$(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})$. Applying
Eq. (15) to (13), the Stokes parameters at point R can be written as

$\begin{split}S_{j}(\textbf{R},\omega)&=S_{j}(\textbf{r}_{1},\omega)[1+C(\textbf{r}_{1},\textbf{r}_{2})+2\sqrt{C(\textbf{r}_{1},\textbf{r}_{2})}\\\
&Re[|(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)|\exp(\iota\alpha_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)+\iota\omega\tau(\textbf{r}_{1},\textbf{r}_{2}))]].\end{split}$
(16)

From the identity, $\exp(\iota\theta)=\cos\theta+\iota\sin\theta$, we readily
find that

$\begin{split}S_{j}(\textbf{R},\omega)&=S_{j}(\textbf{r}_{1},\omega)[1+C(\textbf{r}_{1},\textbf{r}_{2})+2\sqrt{C(\textbf{r}_{1},\textbf{r}_{2})}|(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)|\\\
&\cos(\alpha_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)+\omega\tau(\textbf{r}_{1},\textbf{r}_{2}))].\end{split}$
(17)

By employing the second equality of the CSP condition, which states that
$S_{j}(\textbf{r}_{2},\omega)=S_{j}(\textbf{R},\omega)$, it becomes evident
that the bracketed term in Eq. (13) must be frequency-independent.
Consequently,

$(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)\exp(\iota\omega\tau(\textbf{r}_{1},\textbf{r}_{2}))=f(\textbf{r}_{1},\textbf{r}_{2},\Delta\tau),$
(18a)
$(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)\exp(\iota\omega[\tau_{0}(\textbf{r}_{1},\textbf{r}_{2})+\Delta\tau])=f(\textbf{r}_{1},\textbf{r}_{2},\Delta\tau).$
(18b)

Therefore, CSP exists at $\textbf{r}_{1}$ and $\textbf{r}_{2}$ if Eq. (12)
holds together with Eq. (18). It is observed that
$|(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)|$ becomes
frequency-independent. To achieve this, the choice of time delays $\Delta\tau$
should be such that the frequency-dependent term is eliminated in Eqs. (15)
and (18) [31].
Examining coherence functions in both the space-time and space-frequency
domains is an essential requirement in studying CSP. The two-time Stokes
parameters are defined as follows: [22]

$S_{0}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\langle
E^{*}_{x}(\textbf{r}_{1},t_{1})E_{x}(\textbf{r}_{2},t_{2})\rangle+\langle
E^{*}_{y}(\textbf{r}_{1},t_{1})E_{y}(\textbf{r}_{2},t_{2})\rangle,$ (19a)
$S_{1}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\langle
E^{*}_{x}(\textbf{r}_{1},t_{1})E_{x}(\textbf{r}_{2},t_{2})\rangle-\langle
E^{*}_{y}(\textbf{r}_{1},t_{1})E_{y}(\textbf{r}_{2},t_{2})\rangle,$ (19b)
$S_{2}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\langle
E^{*}_{y}(\textbf{r}_{1},t_{1})E_{x}(\textbf{r}_{2},t_{2})\rangle+\langle
E^{*}_{x}(\textbf{r}_{1},t_{1})E_{y}(\textbf{r}_{2},t_{2})\rangle,$ (19c)
$S_{3}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\iota[\langle
E^{*}_{y}(\textbf{r}_{1},t_{1})E_{x}(\textbf{r}_{2},t_{2})\rangle-\langle
E^{*}_{x}(\textbf{r}_{1},t_{1})E_{y}(\textbf{r}_{2},t_{2})\rangle],$ (19d)

where the elements of the two-time mutual coherence matrix are represented by
$\Gamma_{ij}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\langle
E^{*}_{i}(\textbf{r}_{1},t_{1})E_{j}(\textbf{r}_{2},t_{2})\rangle$,
$(i,j=x,y)$. Equation (24) in reference [31] can be extended for EM fields,
and two-time Stokes parameters govern the complete information of coherence
and polarization for EM fields [22]. The time-integrated version of two-point
Stokes parameters for the EM field can be expressed as [31]

$\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)=\frac{1}{2\pi}\int_{-\infty}^{\infty}S_{j}(\textbf{r}_{1},\textbf{r}_{2},t,\Delta
t)dt=\int_{0}^{\infty}S_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)\exp[-\iota\omega\Delta
t]d\omega,$ (20)

where $t=\frac{t_{1}+t_{2}}{2}$, and $\Delta t=t_{2}-t_{1}$. Putting
$\textbf{r}_{1}=\textbf{r}_{2}=\textbf{r}$, in Eq. (20) we obtain

$\bar{S}_{j}(\textbf{r},\textbf{r},\Delta
t)=\frac{1}{2\pi}\int_{-\infty}^{\infty}S_{j}(\textbf{r},\textbf{r},t,\Delta
t)dt=\int_{0}^{\infty}S_{j}(\textbf{r},\textbf{r},\omega,\omega)\exp[-\iota\omega\Delta
t]d\omega.$ (21)

After straightforward developments of Eqs. (12), (14), and (20), we obtain

$\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)=\sqrt{C(\textbf{r}_{1},\textbf{r}_{2})}\int_{0}^{\infty}S_{j}(\textbf{r}_{1},\omega)\mu_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)\exp[-\iota\omega\Delta
t]d\omega,$ (22)

where the Eq. (14) provides the expression for
$\mu_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)$ without any subscript.
This quantity is associated with the electric field, as indicated by the left-
hand side (LHS) of Eq. (3). On recalling Eqs. (3), (4), and (11) we obtain

$S_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=(S_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)\exp[\iota\omega\tau_{0}(\textbf{r}_{1},\textbf{r}_{2})].$
(23)

Hence, the value of Eq. (14) yields

$\mu_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=(\mu_{j})_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)\exp[\iota\omega\tau_{0}(\textbf{r}_{1},\textbf{r}_{2})].$
(24)

Inserting Eq. (18) into (24), the above equation can be written as

$\mu_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=f(\textbf{r}_{1},\textbf{r}_{2},\Delta\tau)\exp[-\iota\omega\Delta\tau].$
(25)

By substituting the Eqs. (21) and (25) in Eq. (22), we obtain

$\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)=\sqrt{C(\textbf{r}_{1},\textbf{r}_{2})}f(\textbf{r}_{1},\textbf{r}_{2},\Delta\tau)\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{1},\Delta
t+\Delta\tau).$ (26)

By employing Eqs. (12) and (21), we obtain

$\bar{S}_{j}(\textbf{r}_{2},\textbf{r}_{2},0)=C(\textbf{r}_{1},\textbf{r}_{2})\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{1},0).$
(27)

In analogy to the spectral Eq. (14), the definition of two-time Stokes
parameters normalized by corresponding usual Stokes parameters in terms of
time-integrated correlations are expressed as [30]

$\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)=\frac{\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)}{\sqrt{\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{1},0)}\sqrt{\bar{S}_{j}(\textbf{r}_{2},\textbf{r}_{2},0)}}.$
(28)

By putting the values from Eqs. (26) and (27), we get

$\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)=\frac{f(\textbf{r}_{1},\textbf{r}_{2},\Delta\tau)\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{1},\Delta
t+\Delta\tau)}{\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{1},0)}=f(\textbf{r}_{1},\textbf{r}_{2},\Delta\tau)\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{1},\Delta
t+\Delta\tau).$ (29)

By setting $\Delta t=-\Delta\tau$ in Eq. (29) and observing that
$\bar{\psi}_{j}(\textbf{r},\textbf{r},0)=1$ in Eq. (28), we can conclude that

$\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{2},-\Delta\tau)=f(\textbf{r}_{1},\textbf{r}_{2},\Delta\tau).$
(30)

Therefore, Eq. (29) gives us the following result:

$\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)=\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{2},-\Delta\tau)\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{1},\Delta
t+\Delta\tau).$ (31)

This represents the reduction formula for nonstationary light fields, which is
the key outcome of this paper. Equation (31) can be interpreted as the CSP of
Stokes parameters for nonstationary vector fields. It reveals that the time-
integrated spatiotemporal function can be expressed as a product of time-
integrated spatial and temporal-dependent functions. This finding is analogous
to the earlier discovery of CSP for stationary vector fields, expressed as
$\psi_{j}(\textbf{r}_{1},\textbf{r}_{2},\tau)=\psi_{j}(\textbf{r}_{1},\textbf{r}_{2},\tau_{j})\psi_{j}(\textbf{r}_{1},\textbf{r}_{1},\tau-\tau_{j})$.
Now, we can express the inverse form of Eq. (20) as follows:

$S_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=\frac{1}{2\pi}\int_{-\infty}^{\infty}\bar{S}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)\exp[\iota\omega\Delta t]d\Delta t.$ (32)

By using Eqs. (4) and (21), we find that

$\frac{S_{j}(\textbf{r},\omega)}{\bar{S}_{j}(r,r,0)}=\frac{S_{j}(\textbf{r},\omega)}{\int_{0}^{\infty}S_{j}(\textbf{r},\omega)d\omega}=s_{j}(\textbf{r},\omega).$
(33)

By inserting the Eqs. (28) and (33) into (14), we obtain

$\mu_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=\frac{1}{2\pi}\frac{1}{\sqrt{s(\textbf{r}_{1},\omega)s(\textbf{r}_{2},\omega)}}\int_{-\infty}^{\infty}\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{2},\Delta
t)\exp(\iota\omega\Delta t)d\Delta t.$ (34)

Using Eq. (28) and (33), we obtain as

$\frac{\bar{\psi}_{j}(\textbf{r},\textbf{r},\Delta
t)}{s_{j}(\textbf{r},\omega)}=\frac{\bar{S}_{j}(\textbf{r},\textbf{r},\Delta
t)}{\bar{S}_{j}(\textbf{r},\textbf{r},0)s_{j}(\textbf{r},\omega)}=\frac{\bar{S}_{j}(\textbf{r},\textbf{r},\Delta
t)}{S_{j}(\textbf{r},\omega)}.$ (35)

Specifically, by utilizing Eqs. (31), (32), and (35), we can derive the
expression for $\mu_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)$ as
follows:

$\mu_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=\bar{\psi}_{j}(\textbf{r}_{1},\textbf{r}_{2},-\Delta\tau)\exp(-\iota\omega\Delta\tau).$
(36)

This demonstrates that the absolute value of spectral coherence Stokes
parameters, normalized by their corresponding usual Stokes parameters, remain
constant for all frequencies of the optical field, similar to the behavior
observed in stationary vector fields. Therefore, by considering time-
integrated quantities, it is possible to derive a reduction formula for
nonstationary vector fields that exhibits similarities to the reduction
formula observed in stationary vector fields.

##  4 Strict CSP

Now, let us direct our focus towards strict CSP in the context of
nonstationary optical fields. Similar to the stationary cases, it is possible
to derive an expression for strict CSP in nonstationary optical fields by
incorporating time-integrated coherence functions. The two-frequency and two-
time Stokes parameters, normalized by their respective zeroth Stokes
parameters, are denoted as
$\eta_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})$ and
$\nu_{j}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})$, respectively. These
parameters are defined as follows:[30]

$\eta_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\frac{S_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})}{\sqrt{S_{0}(\textbf{r}_{1},\omega_{1})S_{0}(\textbf{r}_{2},\omega_{2})}},$
(37a)
$\nu_{j}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\frac{S_{j}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})}{\sqrt{S_{0}(\textbf{r}_{1},t_{1})S_{0}(\textbf{r}_{2},t_{2})}}.$
(37b)

Developing Eq. (37) with Eq. (36) and utilizing Eq. (28) in reference [30], we
can write a relationship between the space-time and space-frequency coherence
Stokes parameters normalized by zeroth Stokes parameters, which reads

$\eta_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=\bar{\nu}_{j}(\textbf{r}_{1},\textbf{r}_{2},-\Delta\tau)\exp(-\iota\omega\Delta\tau).$
(38)

The form of this relationship bears a resemblance to
$\eta_{j}(r_{1},r_{2},\omega)=\nu_{j}(r_{1},r_{2},\tau_{0})\exp({\iota\omega\tau_{0}})$,
which is observed in the case of stationary vector fields. By introducing the
EMDOC for nonstationary fields in space-frequency ($\mu_{\epsilon}$) and
space-time ($\gamma_{\epsilon}$) domain via the formulas [22, 34]

$\mu_{\epsilon}^{2}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\frac{1}{2}\sum_{j=0}^{3}|\eta_{j}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})|^{2},$
(39a)
$\gamma_{\epsilon}^{2}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\frac{1}{2}\sum_{j=0}^{3}|\nu_{j}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})|^{2}.$
(39b)

Putting the value from Eq. (38), yields the interesting result of

$\mu_{\epsilon}^{2}(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=\bar{\gamma}_{\epsilon}^{2}(\textbf{r}_{1},\textbf{r}_{2},-\Delta\tau).$
(40)

Therefore, the spectral EMDOC and the time-integrated version of space-time
EMDOCs are equivalent. Recently, a new condition has been introduced for
strict CSP [16], which involves the equality of the DOCP [35, 36] in both the
space-time and space-frequency domains. The EMDOC and DOCP are related to the
equation in the following manner [16]:

$\mu_{\epsilon}^{2}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})=\frac{1}{2}|\mu_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})|^{2}[1+P^{2}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})],$
(41a)
$\gamma_{\epsilon}^{2}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\frac{1}{2}|\psi_{0}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})|^{2}[1+P^{2}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})],$
(41b)

where the parameter
$\mu_{0}(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})$ can be obtained
from Eq. (14) (without the subscript) by setting $j=0$. Additionally, the
parameter
$\psi_{0}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})=\frac{S_{0}(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})}{\sqrt{S_{0}(\textbf{r}_{1},t_{1})}\sqrt{S_{0}(\textbf{r}_{2},t_{2})}}$.
$P(\textbf{r}_{1},\textbf{r}_{2},\omega_{1},\omega_{2})$ represents the two-
frequency DOCP, while $P(\textbf{r}_{1},\textbf{r}_{2},t_{1},t_{2})$
corresponds to the two-time DOCP. By applying the Eqs. (36), (40), and (41),
we obtain

$P(\textbf{r}_{1},\textbf{r}_{2},\omega,\omega)=\bar{P}(\textbf{r}_{1},\textbf{r}_{2},-\Delta\tau).$
(42)

Like EMDOC, the spectral DOCP and the time-integrated version of space-time
DOCP are equivalent. Equations (38), (40), and (42) represent the distinctive
characteristics of strict CSP. These equations serve as the other key findings
of this paper, highlighting the significance of strict CSP in nonstationary
vector fields.

##  5 Conclusion

In conclusion, this study establishes the conditions of cross-spectral purity
and strict cross-spectral purity by incorporating time-integrated coherence
parameters. The mathematical structure of time-integrated normalized two-time
Stokes parameters exhibits similarity to the condition observed in stationary
vector fields. The time-integrated version of normalized space-time coherence
Stokes demonstrates a reduction property, which can be expressed as the
product of spatial and temporal coherence functions. Furthermore, the absolute
value of the normalized spectral coherence Stokes parameter remains consistent
across all frequencies of the optical field. Similar to the strict CSP
condition observed in stationary vector fields, we establish this condition
for nonstationary vector fields as well. This is accomplished by ensuring that
the EMDOCs and DOCPs in both the space-time and space-frequency domains are
equal. Thus, our study highlights that the outcomes of cross-spectral purity
in nonstationary vector fields align with those in stationary vector fields
when considering time-integrated coherence functions.

## Disclosures

The authors declare no conflicts of interest.

## References

  * [1] L. Mandel, Concept of cross-spectral purity in coherence theory, J. Opt. Soc. Am. 51, 1342-1350 (1961).
  * [2] L. Mandel, and E. Wolf, Optical Coherence and Quantum Optics (Cambridge University Press, Cambridge 1995).
  * [3] L. Mandel, and E. Wolf, Spectral coherence and concept of cross-spectral purity, J. Opt. Soc. Am. 66, 529-535 (1976).
  * [4] D. F. V. James, and E. Wolf, Cross-spectrally pure light and the spectral modulation law, Opt. Comm. 138, 257-261 (1997).
  * [5] H.C. Kandpal, K. Saxena, D.S. Mehta, J.S. Vaishya and K. C. Joshi, Experimental verification of cross-spectral purity in coherence theory, Opt. Commun. 99, 157-161 (1993).
  * [6] E. Wolf, Young’s interference fringes with narrow-band light, Opt. Lett. 8, 250-252 (1983).
  * [7] H. C. Kandpal, Generation of cross-spectrally pure light by two diffusers moving in opposite directions and determination of diffuser surface roughness, Opt. Comm. 207, 73-76 (2002).
  * [8] A. T. Friberg, and E. Wolf, Relationships between the complex degrees of coherence in the space time and space frequency domains, Opt. Lett. 20, 623-625 (1995).
  * [9] B. Kanseri, and H. C. Kandpal, Experimental study of the relation between degrees of coherence in space-time and space-frequency domain, Opt. Express 18, 11838-11845 (2010).
  * [10] B. Kanseri, and H. C. Kandpal, Experimental observation of invarience of spectral degree of coherence with change in bandwidth of light, Opt. Lett. 35, 70-72 (2010).
  * [11] M. Lahiri, and E. Wolf, Effect of scattering on cross-spectral purity of light , Opt. Comm. 330, 165-168 (2014).
  * [12] H. Liu, J. Cheng, and S. Han, Cross-spectral purity and its influence on ghost imaging experiments, Opt. Comm. 373, 50-53 (2007).
  * [13] T. Hassinen, J. Tervo, and A. T. Friberg, Cross-spectral purity of electromagnetic fields, Opt. Lett. 34, 3866-3868 (2009).
  * [14] T. Hassinen, J. Tervo, and A. T. Friberg, Cross-spectral purity of Stokes parameter, Appl. Phy. B 105, 305-308 (2011).
  * [15] X. Peng, D. Ye, Y. Xin, Y. Chen, and M. Song, Cross-spectral purity of Stokes parameters, purity of partial polarization and statistical similarity, Optik 145, 42-48 (2017).
  * [16] R. Joshi, and B. Kanseri, Generatin of cross-spectrally pure electromagnetic fields using a pair of moving diffusers, J. Opt. Soc. Am. A 40, 204-210 (2023).
  * [17] T. Hassinen, J. Tervo, and A. T. Friberg, ”Purity of partial polarization in the frequency and time domains,” Opt. Lett. 38 (2013) 1221-1223.
  * [18] J. Chen, R. Lu, F. Chen, and J. Li, Cross-spectrally pure light, cross-spectrally pure fields and statistical similarity in electromagnetic fields, Journal of Modern Optics 61, 1164-1173 (2014).
  * [19] H. Partanen, B. J. Hoenders, A. T. Friberg, and T. Setälä, Young’s interference experiment with electromagnetic narrowband light, J. Opt. Soc. Am. A 35, 1379-1384 (2018).
  * [20] I. P. Christov, Propagation of partially coherent light pulses, Optica Acta. 33, 63-72 (1986).
  * [21] H. Lazunen, P. Vahimaa, and J. Tervo, Theory of spatially and spectrally partially coherent pulses, J. Opt. Soc. Am. A 22, 1536-1545 (2005).
  * [22] T. Voipio, T. Setälä, and A.T. Friberg, Partial polarization theory of pulsed optical beams, J. Opt. Soc. Am. A 30, 71-81 (2013).
  * [23] M. Koivurova, L. Ahad, G. Geloni, T. Setälä, J. Turunen, and A. T. Friberg, Interferometry and coherence of nonstationary light, Opt. Lett. 44, 522-525 (2019).
  * [24] C. Ding, M. Koivurova, T. Setälä, J. Turunen, and A. T. Friberg, Spectral invariance and scaling law for nonstationary optical fields, Phy. Rev. A 101, 033808(1)-033808(7) (2020).
  * [25] M. Lahiri, and E. Wolf, Quantum theory of optical coherence of nonstationary light in the space-frequency domain, Phy. Rev. A 82, 1-6 (2010).
  * [26] M. Guo, and D. Zhao, Polarization properties of stochastic electromagnetic beams modulated by a wavefront-folding interferometer, Opt. Exp. 26, 8581-8593 (2018).
  * [27] W. J. Bates, A wavefront shearing interferometer, Proceedings of the Physical Society 61, 940-950 (1948).
  * [28] M. Koivurova, H. Partanen, J. Lahyani, N. Cariou, and J. Turunen, Scanning wavefront folding interferometers, Opt. Express 27, 7738-7750 (2019).
  * [29] M. Koivurova, C. Ding, J. Turunen, and A. T. Friberg, Cross-spectral purity of nonstationary light, Phy. Rev. A 99, 1-6 (2019).
  * [30] R. Joshi, and B. Kanseri, Cross-spectral purity of nonstationary vector fields in space-time and space-frequency domains, J. Opt. Soc. Am. A 38, 271-276 (2021).
  * [31] J. Laatikainen, M. Luo, A. Halder, M. Koivurova, T. Setala, J. Turunen and A. T. Friberg, Reduction formula for Cross-spectral purity of nonstationary light fields, J. Opt. Soc. Am. A 40, 1260-1267 (2023).
  * [32] R. Mandal, X. Mohammadi, A. Wiktor, A. Singh, A. S. Pratap, Applications of Pulsed Light Decontamination Technology in Food Processing: An Overview. Appl. Sci. 10, 1-40 (2020).
  * [33] B, Wagner, M. Steiner, L. Markovic, and R. Crevenna, Successful application of pulsed electromagnetic fields in a patient with post-COVID-19 fatigue: a case report, Wien Med Wochenschr 172, 227-232 (2022).
  * [34] T. Setälä, J. Tervo, and A. T. Friberg, Contrasts of Stokes parameters in Young’s interference experiment and electromagnetic degree of coherence, Opt. Lett. 31, 2669-2671 (2006).
  * [35] T. Shirai, and E. Wolf, Correlations between intensity fluctuations in stochastic electromagnetic beams of any state of coherence and polarization, Opt. Commun. 272 289-292 (2007).
  * [36] T. Hassinen, J. Tervo, T. Setälä and A.T. Friberg, Hanbury Brown-Twiss effect with electromagnetic waves, Opt. Exp. 19 15188-15195 (2011).


# StyleRes: Transforming the Residuals for Real Image Editing with StyleGAN

Hamza Pehlivan Yusuf Dalva Aysegul Dundar
Bilkent University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

We present a novel image inversion framework and a training pipeline to
achieve high-fidelity image inversion with high-quality attribute editing.
Inverting real images into StyleGAN’s latent space is an extensively studied
problem, yet the trade-off between the image reconstruction fidelity and image
editing quality remains an open challenge. The low-rate latent spaces are
limited in their expressiveness power for high-fidelity reconstruction. On the
other hand, high-rate latent spaces result in degradation in editing quality.
In this work, to achieve high-fidelity inversion, we learn residual features
in higher latent codes that lower latent codes were not able to encode. This
enables preserving image details in reconstruction. To achieve high-quality
editing, we learn how to transform the residual features for adapting to
manipulations in latent codes. We train the framework to extract residual
features and transform them via a novel architecture pipeline and cycle
consistency losses. We run extensive experiments and compare our method with
state-of-the-art inversion methods. Qualitative metrics and visual comparisons
show significant improvements. Code: https://github.com/hamzapehlivan/StyleRes

##  1 Introduction

Generative Adversarial Networks (GANs) achieve high quality synthesis of
various objects that are hard to distinguish from real images [14, 21, 43, 22,
41]. These networks also have an important property that they organize their
latent space in a semantically meaningful way; as such, via latent editing,
one can manipulate an attribute of a generated image. This property makes GANs
a promising technology for image attribute editing and not only for generated
images but also for real images. However, for real images, one also needs to
find the corresponding latent code that will generate the particular real
image. For this purpose, different GAN inversion methods are proposed, aiming
to project real images to pretrained GAN latent space [30, 33, 15, 31, 37].
Even though this is an extensively studied problem with significant progress,
the trade-off between image reconstruction fidelity and image editing quality
remains an open challenge.

Input |  |  |  |
---|---|---|---|---
e4e [32] |  |  |  |
HFGI [34] |  |  |  |
HyperStyle [5] |  |  |  |
StyleRes (Ours) |  |  |  |

Figure 1: Comparison of our method with e4e, HFGI, and HyperStyle for the
pose, bob cut hairstyle, smile removal, and color change edits. Our method
achieves high fidelity to the input and high quality edits.

The trade-off between image reconstruction fidelity and image editing quality
is referred to as the distortion-editability  trade-off [32]. Both are
essential for real image editing. However, it is shown that the low-rate
latent spaces are limited in their expressiveness power, and not every image
can be inverted with high fidelity reconstruction [1, 27, 32, 34]. For that
reason, higher bit encodings and more expressive style spaces are explored for
image inversion [1, 2]. Although with these techniques, images can be
reconstructed with better fidelity, the editing quality decreases since there
is no guarantee that projected codes will naturally lie in the generator’s
latent manifold.

In this work, we propose a framework that achieves high fidelity input
reconstruction and significantly improved editability compared to the state-
of-the-art. We learn residual features in higher-rate latent codes that are
missing in the reconstruction of encoded features. This enables us to
reconstruct image details and background information which are difficult to
reconstruct via low rate latent encodings. Our architecture is single stage
and learns the residuals based on the encoded features from the encoder and
generated features from the pretrained GANs. We also learn a module to
transform the higher-latent codes if needed based on the generated features
(e.g. when the low-rate latent codes are manipulated). This way, when low-rate
latent codes are edited for attribute manipulation, the decoded features can
adapt to the edits to reconstruct details. While the attributes are not
edited, the encoder can be trained with image reconstruction and adversarial
losses. On the other hand, when the image is edited, we cannot use image
reconstruction loss to regularize the network to preserve the details. To
guide the network to learn correct transformations based on the generated
features, we train the model with adversarial loss and cycle consistency
constraint; that is, after we edit the latent code and generate an image, we
reverse the edit and aim at reconstructing the original image. Since we do not
want our method to be limited to predefined edits, during training, we
simulate edits by randomly interpolating them with sampled latent codes.

The closest to our approach is HFGI [34], which also learns higher-rate
encodings. Our framework is different as we learn a single-stage architecture
designed to learn features that are missing from low-rate encodings and we
learn how to transform them based on edits. As shown in Fig. 1, our framework
achieves significantly better results than HFGI and other methods in editing
quality. In summary, our main contributions are:

  * •

We propose a single-stage framework that achieves high-fidelity input
embedding and editing. Our framework achieves that with a novel encoder
architecture.

  * •

We propose to guide the image projection with cycle consistency and
adversarial losses. We edit encoded images by taking a step toward a randomly
sampled latent code. We expect to reconstruct the original image when the edit
is reversed. This way, edited images preserve the details of the input image,
and edits become high quality.

  * •

We conduct extensive experiments to show the effectiveness of our framework
and achieve significant improvements over state-of-the-art for both
reconstruction and real image attribute manipulations.

##  2 Related Work

GAN Inversion. GAN inversion methods aim at projecting a real image into GANs
embedding so that from the embedding, GAN can generate the given image.
Currently, the biggest motivation of the inversion is the ability to edit the
image via semantically rich disentangled features of GANs; therefore models
aim for high reconstruction and editing quality. Inversion methods can be
categorized into two; 1) methods that directly optimize the latent vector to
minimize the reconstruction error between the output and target image [9, 1,
2, 22, 28], 2) methods that learn encoders to reconstruct images over training
images [45, 32, 34]. Optimization based methods require per-image optimization
and iterative passes on GANs, which take several minutes per image.
Additionally, overfitting on a single image results in latent codes that do
not lie in GAN’s natural latent distribution, leading to poor editing quality.
Among these methods, PTI [28] takes a different approach, and instead of
searching for a latent code that will reconstruct the image most accurately,
PTI fine-tunes the generator in order to insert encoded latent code into well-
behaved regions of the latent space. It shows better editability; however, the
method still suffers from long run-time optimizations and tunings. Encoder
based methods leverage the knowledge of training sets while projecting images.
They output results with less fidelity to the input, but edits on the
projected latents are better quality. They also operate mostly in real-time.
They either project the latent code with a single feed-forward pass [45, 27,
32], or two stage feed-forward passes where the first encoder learns to embed
an image, the second learns to reconstruct the missing details [34], or
iterative forward passes where the network tries to minimize the
reconstruction loss at each pass by taking the original image and
reconstructed output as inputs [4, 5]. In this work, we propose a novel
encoder architecture that achieves significantly better results than state-of-
the-art via a single feed-forward pass on the input image.

Figure 2: When images are encoded to W+ space, as shown in (b), the
reconstructions miss many image details. However, the edits are in good
quality (c). Additional features can be learned in higher-rate latent codes.
For example, skip connections from encoder to generator in higher resolution
features can enable high-fidelity reconstruction to input images as shown in
(d). However, if the high-rate features do not transform with the edits, they
result in ghosting effects as shown in (e). In this work, we propose high-rate
encoding and transformation for successful inversions and edits (f).

Image Translation. There is quite an interest in image translation algorithms
that can change an attribute of an image while preserving a given content [35,
12, 25], especially for faces editing [8, 29, 38, 42, 24, 36, 13, 17, 3, 10].
These algorithms set an encoder-decoder architecture and train the models with
reconstruction and GAN losses [19, 8, 46, 24, 39]. Even though they are
successful at image manipulation, they are trained for a given task and rely
on predefined translation datasets. On the other hand, it is shown that GANs
that are trained to synthesize objects in an unconditional way can be used for
attribute manipulation [21, 6, 22] via their semantically rich disentangled
features. This makes pretrained GANs promising technology for image editing,
given that via their ability of image generation, they can achieve many
downstream tasks simultaneously. Many methods have been proposed for finding
latent directions to edit images [30, 33, 15, 31, 37]. These directions are
explored both in supervised [30, 3] and unsupervised ways [33, 15, 31, 37]
resulting in exploration of attribute manipulations beyond the predefined
attributes of labeled datasets. However, finding these directions is only one
part of the problem; to achieve real image editing, there is a need for a
successful image inversion method which is the topic of this paper.

Figure 3: StyleRes encodes missing features for high-fidelity reconstruction
of given input via the first encoder, $E_{1}$. Those encoded features are the
ones which could not be encoded to low-rate W+ space via $E_{0}$ due to the
information bottleneck. Through the second encoder, $E_{2}$, StyleRes learns
to transform features based on the manipulated features. During training,
latent codes are edited by interpolating encoded W+’s with randomly generated
ones by StyleGAN’s mapping network. During inference, they are edited with
semantically meaningful directions discovered by methods such as InterfaceGAN
and GANSpace. Note that StyleGAN generator is shown as two parts just for the
ease of visualizing the diagram. First part includes the layers that generate
features to $64\times 64$ and the second part generates the higher resolution
features and final image.

##  3 Method

In Section 3.1, we describe the motivation for our approach. The model
architecture and the training procedure are presented in Sections 3.2 and 3.3,
respectively.

###  3.1 Motivation

Current inversion methods aim at learning an encoder to project an input image
into StyleGAN’s natural latent space so that the inverted image is editable.
However, it is observed by previous works that when images are encoded into
GAN’s natural space (low-rate W or W+ space), their reconstructions suffer
from low fidelity to input images, as shown in Fig. 2(b). On the other hand,
if the image is encoded to higher feature maps of pretrained GANs, they will
not be editable since they are not inverted to the semantic space of GANs. On
the other hand, low-rate inversion is editable, as shown in Fig. 2(c).

StyleGAN, during training and inference, relies on direct noise inputs to
generate stochastic details. These additional noise inputs provide stochastic
image details such as for face images, the exact placement of hairs, stubble,
freckles, or skin pores which are not modeled by W+ space. During inversion to
W+ space alone, that mechanism is discarded. One can tune the noise maps as
well via the reconstruction loss; however, then there will be no mechanism to
adapt those stochastic details to attribute manipulation. For example, the
freckles should move as the pose is manipulated, but with noise optimization
alone, it will not be possible.

We propose to embed images to both low-rate and high-rate embeddings.
Consistent with the design of StyleGAN, we aim at encoding the overall
composition and high-level aspects of the image to W+ space (low-rate
encoding). Our goal is to embed the stochastic image details and the diverse
background details, which are difficult to reconstruct from W+ space to higher
latent codes. However, this setting also requires a mechanism for encoded
image details to adopt manipulations in the image. Otherwise, it will cause a
ghosting effect, as shown in Fig 2(e). For example, in Fig 2(e1), the smile is
removed by W+ code edit; however, the higher-rate encodings stay the same and
cause artifacts around the mouth area. In Fig. 2(e2), the pose is edited but
higher detail encodings did not move and caused blur.

In this work, we design an encoder architecture and training pipeline to
achieve both learning residual features (the ones W+ space could not
reconstruct) and how to transform them to be consistent with the
manipulations.

###  3.2 StyleRes Architecture

Our method utilizes an encoder $E_{0}$ that can embed images into $W+$ latent
space and a StyleGAN generator $G$. In our setup, we utilize a pretrained
encoder for $E_{0}$ [32] and StyleGAN2 generator [22] and fix them in our
training. Because it is difficult to preserve image details only from low-rate
latent codes, we also extract high level features from the encoder. First, the
high and low rate features are extracted as $F_{0},W^{+}=E_{0}(x)$ using the
encoder $E_{0}$ from the input image $x$ as shown in Fig. 3. $F_{0}$ has the
spatial dimension of $64\times 64$ and provides us with more image details.
Next, from $F_{0}$, our aim is to encode residuals that are missing from the
image reconstruction. For this goal, we set an encoder, $E_{1}$, which takes
$F_{0}$ as input. Since the goal of $E_{1}$ is to encode residual features, we
also feed the generated features with $W^{+}$ from the StyleGAN generator,
$G_{W}=G_{0\rightarrow{}n}(W)$, where the arrow operator indicates the indices
of convolutional layers used from $G$.

$F_{a}=E_{1}(F_{0},G_{W+})$ (1)

With the inputs of $F_{0}$ and $G_{W+}$, $E_{1}$ can learn what the missing
features are by comparing the encoded features $F_{0}$ and generated ones,
$G_{W+}$, so far. While $E_{1}$ can learn the residual features, it is not
guided on how to transform them if images are edited. For that purpose, we
train $E_{2}$, which takes $F_{a}$ and edited features from the generator.
Since we do not target predefined edits (e.g., smile, pose, age), we simulate
the edits by taking random directions. Specifically, we sample a $z$ vector
from the normal distribution and obtain $W^{+}_{r}$ by StyleGAN’s mapping
network, $M$; $W^{+}_{r}=M(z)$. Next, we take a step towards $W^{+}_{r}$ to
obtain a mixed style code $W^{+}_{\alpha}$.

$W^{+}_{\alpha}=W^{+}+\alpha\frac{W^{+}_{r}-W^{+}}{10}$ (2)

where $\alpha$ controls the degree of the edit. During the training, we assign
$\alpha$ to $0$ (no edit) or a value in the range of $(4,5)$ with $50\%$
chance. We adjust $F_{a}$ to the altered generator features
$G_{\alpha}=G_{0\rightarrow{}n}(W_{\alpha})$ using a second encoder $E_{2}$
and obtain the final feature map F:

$F=E_{2}(F_{a},G_{\alpha})$ (3)

Finally, $F$ and $G_{\alpha}$ are summed, and given as an input to next
convolutional layers in the generator. Architectural details of $E_{1}$ and
$E_{2}$ are given in Supplementary.

Figure 4: We train StyleRes with (a) no editing based reconstruction and (b)
cycle consistency based reconstruction losses. Many details from the inference
pipeline are omitted for brevity. With reconstruction losses, the model learns
to preserve the image details. We additionally apply adversarial losses on
$x_{i}^{\prime}$; this way, when the image is edited, the transformation
network learn to output realistic images. With cycle consistency based
reconstruction, the network is regularized to keep the input details also
during edits. We use additional losses as explained in Section 3.4.

###  3.3 Training Phases

To train our model with the capabilities of high-fidelity inversion and high-
quality editing, we use two training phases, as shown in Fig. 4.

No Editing Path. In this path, we reconstruct images with no editing on
encoded $W^{+}$. This refers to the case where $\alpha=0$ in Eq. 2. This is
also the training path other inversion methods solely rely on. While this path
can teach the network to reconstruct images faithfully, it does not include
any edits and cannot guide the network to high-quality editability.

Cycle Translation Path. In this path, we edit images by setting $\alpha$ a
value in the range of $(4,5)$, which we found to work best in our ablation
studies. Via the edit, the generator outputs image $x_{i}^{\prime}$. Next, we
feed this intermediate output image to the encoder and reverse the edit by
inverting the addition presented in Eq. 2. The generator reconstructs
$x^{\prime\prime}$, which is supposed to match the input image $x$. The cycle
translation path is important because there is no ground-truth output for the
edited image, $x_{i}^{\prime}$. Adversarial loss can guide $x_{i}^{\prime}$ to
look realistic but will not guide the network to keep the input image details
if edited.

###  3.4 Training Objectives

Reconstruction Losses. For both no editing and cycle consistency path outputs,
our goal is to reconstruct the input image. To supervise this behavior, we use
$L_{2}$ loss, perceptual loss, and identity loss between the input and output
images. The first reconstruction loss is as follows, where $x$ is the input
image, $x^{\prime}$ and $x^{\prime\prime}$ are output images, as given in Fig.
4:

$\begin{split}\mathcal{L}_{rec-l2}=||x^{\prime}-x||_{2}+||x^{\prime\prime}-x||_{2}\end{split}$
(4)

We use perceptual losses from VGG ($\Phi$) at different feature layers ($j$)
between these images from the loss objective as given in Eq. 5.

$\small\mathcal{L}_{rec-p}=||\Phi_{j}(x^{\prime})-\Phi_{j}(x)||_{2}+||\Phi_{j}(x^{\prime\prime})-\Phi_{j}(x)||_{2}$
(5)

Identity loss is calculated with a pre-trained network $A$. $A$ is an ArcFace
model [11] when training on the face domain and a domain specific ResNet-50
model [32] for our training on car class.

$L_{rec-id}=(1-\langle A(x),A(x^{\prime})\rangle)+(1-\langle
A(x),A(x^{\prime\prime})\rangle)$ (6)

Adversarial Losses. We also use adversarial losses to guide the network to
output images that match the real image distribution. This objective improves
realistic image inversions and edits. We load the pretrained discriminator
from StyleGAN training, $D$, and train the discriminator together with the
encoders.

$\begin{split}L_{adv}=2\log{D(x)}+\log{(1-D(x^{\prime}))}\\\
+\log{(1-D(x^{\prime}_{i}))}\end{split}$ (7)

Feature Regularizer. To prevent our encoder from deviating much from the
original StyleGAN space, we regularize the residual features to be small:

$L_{F}=\sum_{F\in\phi}\|F\|_{2}$ (8)

Full Objective. We use the overall objectives given below. The hyper
parameters are provided in Supplementary.

$\begin{split}\underset{E_{1},E_{2}}{\min}\underset{D}{\max}\lambda_{a}\mathcal{L}_{adv}+\lambda_{r1}\mathcal{L}_{rec-l2}+\lambda_{r2}\mathcal{L}_{rec-p}\\\
+\lambda_{r3}\mathcal{L}_{rec-id}+\lambda_{f}\mathcal{L}_{F}\end{split}$ (9)

1\. Smile (-) |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
2\. Age (-) |  |  |  |  |  |  |  |  |
3\. Age (+) |  |  |  |  |  |  |  |  |
4\. Eye Openness |  |  |  |  |  |  |  |  |
5\. Pose(-) |  |  |  |  |  |  |  |  |
6\. Pose(+) |  |  |  |  |  |  |  |  |
7\. Beard |  |  |  |  |  |  |  |  |
8\. Color |  |  |  |  |  |  |  |  |
9\. Grass |  |  |  |  |  |  |  |  |
| Input | e4e [32] | HFGI [34] | HyperStyle [5] | StyleRes (Ours)

Figure 5: Qualitative results of inversion and editing. For each method, the
first column shows inversion, and the second shows InterfaceGAN [30] and
GANSpace [15] edits.

##  4 Experiments

Set-up. For datasets and attribute editing, we follow the previous work [34].
For the human face domain, we train the model on the FFHQ [21] dataset and
evaluate it on the CelebA-HQ [20] dataset. For the car domain, we use Stanford
Cars [23] for training and evaluation. We run extensive experiments with
directions explored with InterfaceGAN [30], GANSpace [15], StyleClip [26], and
GradCtrl [7] methods.

Evaluation. We report metrics for reconstruction and editing qualities. For
reconstruction, we report Frechet Inception Distance (FID) metric [16], which
looks at the realism by comparing the target image distribution and
reconstructed images, Learned Perceptual Image Patch Similarity (LPIPS) [44]
and Structural Similarity Index Measure (SSIM), which compares the target and
output pairs at the feature level of a pretrained deep network and pixel
level, respectively. Additionally, FIDs are calculated for adding and removing
the smile attributes on the CelebA-HQ dataset. By using the ground-truth
attributes, we add smile to images that do not have smile attribute and
calculate FIDs between smile addition edited images and smiling ground-truth
images. The same set-up is used for smile removal. On the Stanford cars
dataset, we change grass and color attribute of images, and since there is no
ground-truth attribute, we calculate the FIDs between the edited and original
images.

Table 1: Quantitative results of reconstruction and editing on CelebA-HQ
dataset. For reconstruction, we report FID, SSIM, and LPIPS scores. For
editing, we report FID metrics for smile addition (+) and removal (-).

| Reconstruction | Editing - FIDs
---|---|---
Method | FID | SSIM | LPIPS | Smile(+) | Smile(-)
pSp [27] | 23.86 | 0.75 | 0.17 | 32.47 | 34.0
e4e [32] | 30.22 | 0.71 | 0.21 | 38.58 | 39.68
ReStyle [4] | 24.82 | 0.73 | 0.20 | 30.35 | 33.69
HyperStyle [5] | 16.08 | 0.83 | 0.11 | 26.43 | 25.26
HFGI [34] | 12.17 | 0.85 | 0.13 | 25.22 | 27.10
StyleTransformer [18] | 21.82 | 0.75 | 0.17 | 34.32 | 34.61
FeatureStyle [40] | 11.33 | 0.90 | 0.10 | 27.20 | 26.15
StyleRes (Ours) | 7.04 | 0.90 | 0.09 | 23.52 | 21.80

Table 2: Quantitative results of reconstruction and editing on the Stanford
Cars Dataset. For reconstruction, we report FID, SSIM, and LPIPS scores. For
editing, we report FID metrics for grass addition and color change.

| Reconstruction | Editing - FIDs
---|---|---
Method | FID | SSIM | LPIPS | Grass | Color
e4e [32] | 14.04 | 0.50 | 0.32 | 18.02 | 29.79
ReStyle [4] | 13.38 | 0.57 | 0.30 | 16.01 | 21.34
HyperStyle [5] | 11.64 | 0.63 | 0.28 | 17.13 | 26.30
HFGI [34] | 9.41 | 0.83 | 0.16 | 14.84 | 26.65
StyleTransformer [18] | 14.01 | 0.57 | 0.28 | 19.47 | 19.94
StyleRes (Ours) | 7.60 | 0.83 | 0.14 | 10.64 | 18.86

Baselines. We compare our method with state-of-the-art image inversion methods
pSp [27], e4e [32], ReStyle [4], HyperStyle [5], HFGI [34], StyleTransformer
[18], and FeatureStyle [40]. We use the author’s released models. Hence, for
the Stanford car dataset, some methods are omitted from comparisons if the
models are not released. Among those, we only train HFGI for car model with
author’s released code since we base our main comparisons with it.

Quantitative Results. In Table 1, we provide reconstruction end editing scores
on the CelebA-HQ dataset. Our method achieves better results than all
competing methods on all metrics. Most significantly, we achieve better FID
scores on both reconstruction and editing qualities. While FeatureStyle
achieves comparable SSIM and LPIPS scores on reconstruction, the editing FIDs
are worse than our model. As shown in Table 2, we achieve significantly better
results than previous methods on the Stanford Car dataset as well. We also
compare the runtime of our method and previous methods. Our method is
significantly faster than HyperStyle (0.125 sec vs. 0.439 sec). That is
because HyperStyle refines its predicted weight offsets gradually via multiple
iterations. Our method is also faster than HFGI (0.125 sec vs. 0.130 sec), in
addition to achieving better scores. We run a single stage network inference,
whereas HFGI first generates an image via e4e and StyleGAN and provides the
error map to a second architecture. The table for runtimes is provided in
Supplementary.

Qualitative Results. We show visuals of inversion and editing results of our
method, e4e, HFGI, and HyperStyle in Fig. 5. We provide further comparisons in
Supplementary with other attribute editings and with other methods. Compared
to previous methods, our method achieves significantly better fidelity to the
input images and preserves the identity and details when edited. It is the
only method in these comparisons that preserves the background, earings
(second row), hands (second-fourth rows), and hats (fourth row). Our method
also achieves facial detail reconstruction; for example, in the fifth row, the
person has a mole at the corner of her mouth. Among the inversions, our method
is the only one preserving that. Furthermore, during the pose edit, it is
transformed correctly. On the seventh row, our method is the only one that
achieves the correct inversion and edit. In car examples, we again achieve
high fidelity to the input images both in inversion and editing. e4e and
HyperStyle do not reconstruct the image faithfully. On the other hand, HFGI
outputs artefacts during edits. We additionally show results with edits
explored by StyleClip [26] and GradCtrl [7] methods in Figs. 6 and 7,
respectively.

Input |  |  |  |
---|---|---|---|---
e4e [32] |  |  |  |
HFGI [34] |  |  |  |
HyperStyle [5] |  |  |  |
StyleRes (Ours) |  |  |  |

Figure 6: Comparison of our method with e4e, HFGI, and HyperStyle with
StyleClip edits [26]. The first column shows eyeglasses addition, the second
and third columns show bangs addition, and the last column shows bob cut
hairstyle results.

Input |  |  |
---|---|---|---
e4e [32] |  |  |
StyleRes (Ours) |  |  |
Input |  |  |
e4e [32] |  |  |
StyleRes (Ours) |  |  |

Figure 7: Additional results of our method and e4e with GradCtrl edits [7].
The first three rows show blue sky edit, and the others show tree background
edits.

Ablation Study. We run extensive experiments to validate the role of each
proposed design choices. We first experiment with an architecture that does
not have $E_{1}$ and $E_{2}$ modules. This experiment refers to the case where
we learn a network that does not take input from StyleGAN Part 1 features,
neither the original ($G_{W+}$) nor the edited features ($G_{\alpha}$). The
network directly takes higher layer features ($F_{0}$) from the encoder and
outputs them to the generator. As shown in Fig. 8, the network still tries to
learn residual features to achieve reconstructions; however, it achieves poor
edits. Without $E_{2}$ refers to the experiment with $E_{1}$ directly
outputting features to Generator Part 2 (as shown in Fig. 3). Without $E_{1}$
refers to the experiment where $E_{2}$ directly takes input from the encoder
($F_{0}$) and ($G_{\alpha}$) as defined in Sec. 3.2. None of the networks are
able to learn the residual features and how to transform them as well as our
final architecture since they are not designed to take the original and edited
features separately. We also experiment without cycle consistency losses. Fig.
9 shows visual results of methods trained with and without cycle consistency
constrain. We observe that with cycle consistency constrain, our network
achieves preserving fine image details even when images are edited.

Input |  |  |  |
---|---|---|---|---
w/o $E_{1}$ and $E_{2}$ |  |  |  |
w/o $E_{1}$ |  |  |  |
w/o $E_{2}$ |  |  |  |
StyleRes (ours) |  |  |  |

Figure 8: Ablation Study. Pose edit outputs of our final architecture and
architecture with missing modules. w/o $E_{1}$ or $E_{2}$, networks struggle
to transform features correctly. Figure 9: Ablation Study. Pose edit outputs
of models trained with and without cycle consistency loss. The model trained
with cycle consistency is better at preserving the details as shown in
enlarged boxes.

##  5 Conclusion

We present a novel image inversion framework and a training pipeline to
achieve high-fidelity image inversion with high-quality attribute editing. In
this work, to achieve high-fidelity inversion, we learn residual features in
higher latent codes that lower latent codes were not able to encode. This
enables preserving image details in reconstruction. To achieve high-quality
editing, we learn how to transform the residual features for adapting to
manipulations in latent codes. We show state-of-the-art results on a wide
range of edits explored with different methods both quantitatively and
qualitatively while achieving faster run-time than competing methods.

## Acknowledgement

This work has been funded by The Scientific and Technological Research Council
of Turkey (TUBITAK), 3501 Research Project under Grant No 121E097. A. Dundar
was supported by Marie Skłodowska-Curie Individual Fellowship.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432–4441, 2019.
  * [2] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8296–8305, 2020.
  * [3] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka.  Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows.  ACM Transactions on Graphics (ToG), 40(3):1–21, 2021.
  * [4] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Restyle: A residual-based stylegan encoder via iterative refinement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6711–6720, 2021.
  * [5] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18511–18521, 2022.
  * [6] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [7] Zikun Chen, Ruowei Jiang, Brendan Duke, Han Zhao, and Parham Aarabi.  Exploring gradient-based multi-directional controls in gans.  In European Conference on Computer Vision, pages 104–119. Springer, 2022.
  * [8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [9] Antonia Creswell and Anil Anthony Bharath.  Inverting the generator of a generative adversarial network.  IEEE transactions on neural networks and learning systems, 30(7):1967–1974, 2018.
  * [10] Yusuf Dalva, Said Fahri Altındiş, and Aysegul Dundar.  Vecgan: Image-to-image translation with interpretable latent directions.  In European Conference on Computer Vision, pages 153–169. Springer, 2022.
  * [11] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  * [12] Aysegul Dundar, Karan Sapra, Guilin Liu, Andrew Tao, and Bryan Catanzaro.  Panoptic-based image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8070–8079, 2020.
  * [13] Yue Gao, Fangyun Wei, Jianmin Bao, Shuyang Gu, Dong Chen, Fang Wen, and Zhouhui Lian.  High-fidelity and arbitrary face editing.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16115–16124, 2021.
  * [14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [15] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  Advances in Neural Information Processing Systems, 33, 2020.
  * [16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [17] Xianxu Hou, Xiaokang Zhang, Hanbang Liang, Linlin Shen, Zhihui Lai, and Jun Wan.  Guidedstyle: Attribute knowledge guided style manipulation for semantic face editing.  Neural Networks, 145:209–220, 2022.
  * [18] Xueqi Hu, Qiusheng Huang, Zhengyi Shi, Siyuan Li, Changxin Gao, Li Sun, and Qingli Li.  Style transformer for image inversion and editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11337–11346, June 2022.
  * [19] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  Eur. Conf. Comput. Vis., 2018.
  * [20] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [21] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [22] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [23] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [24] Xinyang Li, Shengchuan Zhang, Jie Hu, Liujuan Cao, Xiaopeng Hong, Xudong Mao, Feiyue Huang, Yongjian Wu, and Rongrong Ji.  Image-to-image translation via hierarchical style disentanglement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8639–8648, 2021.
  * [25] Guilin Liu, Aysegul Dundar, Kevin J Shih, Ting-Chun Wang, Fitsum A Reda, Karan Sapra, Zhiding Yu, Xiaodong Yang, Andrew Tao, and Bryan Catanzaro.  Partial convolution for padding, inpainting, and image synthesis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [26] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085–2094, 2021.
  * [27] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287–2296, 2021.
  * [28] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  ACM Transactions on Graphics (TOG), 42(1):1–13, 2022.
  * [29] Wei Shen and Rujie Liu.  Learning residual images for face attribute manipulation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4030–4038, 2017.
  * [30] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243–9252, 2020.
  * [31] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1532–1540, 2021.
  * [32] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  ACM Transactions on Graphics (TOG), 40(4):1–14, 2021.
  * [33] Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In International Conference on Machine Learning, pages 9786–9796. PMLR, 2020.
  * [34] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen.  High-fidelity gan inversion for image attribute editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11379–11388, 2022.
  * [35] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [36] Po-Wei Wu, Yu-Jing Lin, Che-Han Chang, Edward Y Chang, and Shih-Wei Liao.  Relgan: Multi-domain image-to-image translation via relative attributes.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 5914–5922, 2019.
  * [37] Zongze Wu, Dani Lischinski, and Eli Shechtman.  Stylespace analysis: Disentangled controls for stylegan image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863–12872, 2021.
  * [38] Taihong Xiao, Jiapeng Hong, and Jinwen Ma.  Elegant: Exchanging latent encodings with gan for transferring multiple face attributes.  In Proceedings of the European conference on computer vision (ECCV), pages 168–184, 2018.
  * [39] Guoxing Yang, Nanyi Fei, Mingyu Ding, Guangzhen Liu, Zhiwu Lu, and Tao Xiang.  L2m-gan: Learning to manipulate latent space semantics for facial attribute editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2951–2960, 2021.
  * [40] Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier.  A style-based gan encoder for high fidelity reconstruction of images and videos.  European conference on computer vision, 2022.
  * [41] Ning Yu, Guilin Liu, Aysegul Dundar, Andrew Tao, Bryan Catanzaro, Larry S Davis, and Mario Fritz.  Dual contrastive loss and attention for gans.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6731–6742, 2021.
  * [42] Gang Zhang, Meina Kan, Shiguang Shan, and Xilin Chen.  Generative adversarial network with spatial attention for face attribute editing.  In Proceedings of the European conference on computer vision (ECCV), pages 417–432, 2018.
  * [43] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In International conference on machine learning, pages 7354–7363. PMLR, 2019.
  * [44] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [45] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  In European conference on computer vision, pages 592–608. Springer, 2020.
  * [46] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman.  Multimodal image-to-image translation by enforcing bi-cycle consistency.  In Advances in neural information processing systems, pages 465–476, 2017.

Figure 10: Detailed architecture of our encoders. More information regarding dimensions are given in the corresponding text.  Table 3: Inference time of the competing methods are given. Method | Runtime (sec)
---|---
pSp [27] | 0.088
e4e [32] | 0.089
ReStyle [4] | 0.365
HyperStyle [5] | 0.437
HFGI [34] | 0.130
StyleTransformer [18] | 0.063
FeatureStyle [40] | 0.526
PTI [28] | 97.94
StyleRes (Ours) | 0.125
Table 4: Quantitative results of reconstruction and editing on CelebA-HQ
dataset. We compare with PTI. For reconstruction, we report FID, SSIM, and
LPIPS scores. For editing, we report FID metrics for smile addition (+) and
removal (-).

| Reconstruction | Editing - FIDs
---|---|---
Method | FID | SSIM | LPIPS | Smile(+) | Smile(-)
PTI [28] | 10.64 | 0.92 | 0.07 | 30.29 | 30.11
StyleRes (Ours) | 7.04 | 0.90 | 0.09 | 23.52 | 21.80

##  Appendix A Training Details

We use e4e [32] as the basic encoder and invert StyleGAN2 [22] generator. The
high level features $F_{0}$ are the input of the first map2style layer used in
e4e. $F_{0}$ has the spatial dimension of $128\times 64\times 64$. The
intermediate StyleGAN features are extracted as $G_{W}=G_{0\rightarrow 8}(W)$,
where the arrow operator indicates the indices of convolution layers used.
$G_{W}$ has the spatial dimension of $512\times 64\times 64$.

When we choose the _no editing path_ , we set $\lambda_{r1}=1.0$,
$\lambda_{r2}=0.001$, $\lambda_{r3}=0.1$ for the face and $\lambda_{r3}=0.5$
for the car dataset. When we choose the _cycle translation path_ , we set
$\lambda_{r1}=0.0$, meaning we do not use cycle consistency at the pixel
level, $\lambda_{r2}=0.0001$, $\lambda_{r3}=0.01$ for the face and
$\lambda_{r3}=0.05$ for the car dataset. At both paths, we set
$\lambda_{a}=0.1$. The regularizer coefficient is set to $\lambda_{f}=5.0$ for
the face dataset and $\lambda_{f}=3.0$ for the car dataset. The network is
trained with Adam optimizer, with a learning rate equal to $0.0001$. We halved
the learning rate at iterations 5000, 10000 and 15000.

Figure 11: Building blocks of our encoders. On the left, we demonstrate the
downsampling layers used in $E_{1}$. On the right, we show standard residual
layers used in both $E_{1}$ and $E_{2}$.

##  Appendix B Model Architecture

Our model consists of residual layers, which are visualized in Fig. 11.
Encoder $E_{1}$ takes the concatenated features $F_{0}$ and $G_{w}$, which has
a spatial dimension of 640x64x64. It then forwards the concatenated features
to $E_{2}$ using an encoder-decoder architecture. The first convolution layer
sets the channel size as 512, and it is not changed in the rest of the layers.
_DownResBlk_ reduces the resolution to half, and _Interpolate_ layer doubles
the resolution, by using linear interpolation.

$E_{2}$ takes both $F_{a}$ and $G_{\alpha}$ as inputs, and learns how to adapt
encoded features to the generator features. Before concatenating the input
features, their channel sizes are reduced to 256 using 3x3 convolution layers.
After the concatenation, the channel size becomes 512, and it does not change
throughout the $E_{2}$. The detailed architecture of our encoders is given in
Fig. 10.

##  Appendix C Evaluation Details

For the face images, the reconstruction evaluations are performed on the last
2000 images of the CelebA-HQ dataset, same as the most of the inversion
methods. For smile addition, we first find the ground truth smiling and non-
smiling images in the CelebA test set. We add a smile to non-smiling ground
truth images, and obtain fake smiling images. Finally, we compute the FID
between the ground-truth smiling and the fake smiling images. Similar setup is
used for smile removal as well. To add or remove a smile, we use the boundary
obtained by InterfaceGAN, with a factor of 3.

For the car images, the reconstruction and editing evaluations are performed
on the first 2000 images of the Stanford Cars test set. Because we do not have
ground-truth labels, we directly calculate FID between the original and edited
images. The edited images are obtained using GanSpace directions.

For HyperStyle and Restyle, we use 5 iterative iterations in the
reconstruction and editing, which is consistent with their training scheme.
For PTI, we deploy locality regularization, as introduced in their work, for
both reconstruction and editing.

We provide visuals obtained with various editing methods, namely InterfaceGan
[30], GanSpace [15], StyleClip [26] and GradCtrl [7]. Because our network is
trained with editing in mind, we can directly apply boundaries found by
different editing methods.

##  Appendix D Additional Results

In Table 4, we provide comparisons with PTI. PTI runs optimization for each
image and achieves better reconstruction scores in terms of SSIM and LPIPS.
However, our method achieves significantly better edit FIDs and reconstruction
FIDs. Furthermore, running evaluation with PTI method takes more than 2 days
on a single GPU, whereas our method finishes within 5 minutes. Run-time
comparisons are provided in Table 3. The running times are obtained by
averaging the reconstruction times of 2000 samples with batch size equal to 1
on a single NVIDIA GeForce GTX 1080 Ti GPU.

We provide visual comparisons on:

  1. 1.

smile editing with StyleTransformer, PTI, and FeatureStyle in Figs. 12, by
using InterfaceGAN.

  2. 2.

various editings with e4e, HFGI and HyperStyle in Figs. 13, by using
InterfaceGAN, GanSpace and StyleClip.

  3. 3.

car editings with e4e, HFGI and HyperStyle in Fig. 14, by using GanSpace.

Smile (+) |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
Smile (+) |  |  |  |  |  |  |  |  |
Smile (+) |  |  |  |  |  |  |  |  |
Smile (+) |  |  |  |  |  |  |  |  |
Smile (+) |  |  |  |  |  |  |  |  |
Smile (-) |  |  |  |  |  |  |  |  |
Smile (-) |  |  |  |  |  |  |  |  |
Smile (-) |  |  |  |  |  |  |  |  |
Smile (-) |  |  |  |  |  |  |  |  |
Smile (-) |  |  |  |  |  |  |  |  |
| Input | StyleTransformer [18] | PTI [28] | FeatureStyle [40] | StyleRes (Ours)

Figure 12: Qualitative results of inversion and editing. For each method,
first column shows inversion, and second shows editing.

Smile (+) |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
Age (-) |  |  |  |  |  |  |  |  |
Age (+) |  |  |  |  |  |  |  |  |
Pose (+) |  |  |  |  |  |  |  |  |
Pose (-) |  |  |  |  |  |  |  |  |
Beard |  |  |  |  |  |  |  |  |
Eye Openness |  |  |  |  |  |  |  |  |
Lipstick | 1  |  |  |  |  |  |  |  |
Eyeglasses |  |  |  |  |  |  |  |  |
Bangs |  |  |  |  |  |  |  |  |
Bob Cut |  |  |  |  |  |  |  |  |
| Input | e4e [32] | HFGI [34] | HyperStyle [5] | StyleRes (Ours)

Figure 13: Qualitative results of inversion and editing. For each method,
first column shows inversion, and second shows editing.

Car Color |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
Car Color |  |  |  |  |  |  |  |  |
Car Color |  |  |  |  |  |  |  |  |
Car Color |  |  |  |  |  |  |  |  |
Car Color |  |  |  |  |  |  |  |  |
Car Color |  |  |  |  |  |  |  |  |
Grass |  |  |  |  |  |  |  |  |
Grass |  |  |  |  |  |  |  |  |
Grass |  |  |  |  |  |  |  |  |
Grass |  |  |  |  |  |  |  |  |
Grass |  |  |  |  |  |  |  |  |
Grass |  |  |  |  |  |  |  |  |
| Input | e4e [32] | HFGI [34] | HyperStyle [5] | StyleRes (Ours)

Figure 14: Qualitative results of inversion and editing. For each method,
first column shows inversion, and second shows editing.