# Disentangled Face Attribute Editing via Instance-Aware Latent Space Search

Yuxuan Han1 Jiaolong Yang2222Work of JY was done in September 2020. Ying
Fu1111Corresponding Author<EMAIL_ADDRESS>1Beijing Institute of
Technology
2Mircosoft Research Asia {hanyuxuan<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent works have shown that a rich set of semantic directions exist in the
latent space of Generative Adversarial Networks (GANs), which enables various
facial attribute editing applications. However, existing methods may suffer
poor attribute variation disentanglement, leading to unwanted change of other
attributes when altering the desired one. The semantic directions used by
existing methods are at attribute level, which are difficult to model complex
attribute correlations, especially in the presence of attribute distribution
bias in GAN’s training set. In this paper, we propose a novel framework (IALS)
that performs Instance-Aware Latent-Space Search to find semantic directions
for disentangled attribute editing. The instance information is injected by
leveraging the supervision from a set of attribute classifiers evaluated on
the input images. We further propose a Disentanglement-Transformation ($DT$)
metric to quantify the attribute transformation and disentanglement efficacy
and find the optimal control factor between attribute-level and instance-
specific directions based on it. Experimental results on both GAN-generated
and real-world images collectively show that our method outperforms state-of-
the-art methods proposed recently by a wide margin. Code is available at
https://github.com/yxuhan/IALS.

###  1 Introduction

Figure 1: Our goal is to change the primal attribute of a given face (_e.g._ ,
age and gender here) while preserving other condition attributes (_e.g._ ,
eyeglasses and expression). Despite the latent space directions of the primal
and condition attributes in the InterfaceGAN method have been orthogonalized,
changing primal attribute still leads to unwanted condition attribute change.
Our IALS method can produce satisfactory disentangled editing results.

The task of face attribute editing aims to alter a given face image towards a
given attribute, such as age, gender, expression, and eyeglasses. A successful
editing should not only output high-quality results with accurate target
attribute, but also well preserve all other image content characterized by the
complementary attributes. Face attribute editing has attracted much attention
in recent years and numerous algorithms have been proposed Shen and Liu
(2017); Choi et al. (2018); Zhang et al. (2018); Bahng et al. (2020); Awiszus
et al. (2019); Gu et al. (2019). Notwithstanding the promising results
demonstrated by these methods, this task is still quite challenging due to the
high output dimension and need for precisely disentangling factors of
variation corresponding to different face attributes.

Face image synthesis has achieved tremendous success recently with Generative
Adversarial Networks (GANs) Goodfellow et al. (2014). State-of-the-art GAN
models Karras et al. (2019, 2020) can generate high-fidelity face images from
the learned latent space. This motivates several works Shen et al. (2020);
Voynov and Babenko (2020); Shen and Zhou (2020); Härkönen et al. (2020) to
edit face attribute by reusing the knowledge learned by GAN models.
Specifically, for each attribute, they search a corresponding direction in the
GAN latent space, such that moving a latent code along this direction can lead
to the desired change of this attribute in the generated images. Although the
target attributes can be changed effectively by these methods, disentangled
editing is still problematic. As illustrated in Figure 1, eyeglasses may
appear when age is changed from young to old, despite the latent space
direction of age have been made orthogonal to eyeglasses via projection Shen
et al. (2020). There are at least two possible reasons for this issue: _i_)
the distribution bias of GAN’s training set (_e.g._ , elder people tend to
wear eyeglasses in FFHQ Karras et al. (2019)), and _ii_) the attribute-level
directions cannot handle complex attribute distributions and are not effective
for attribute variation disentanglement.

In this paper, we propose a novel framework to search the semantic directions
in GAN latent space for disentangled face attribute editing. Instead of
naively using fixed, _attribute-level_ directions, we opt for dynamically
searching _instance-aware_ directions, where instance refers to the input
image to be edited. The intuition behind is that by leveraging the instance
information, the complementary attributes of the instance can be explicitly
and effectively preserved, leading to disentangled editing results. To render
the directions instance-aware, we consider the instance-specific direction
obtained by back-propagating the gradient of off-the-shelf attribute CNN
classifiers on the input image, and introduce a control factor to balance the
attribute-level and instance-specific direction components. We propose a
Disentanglement-Transformation ($DT$) metric to quantitatively evaluate the
editing results and select the control factor that leads to the highest $DT$
metric.

We test our method on both GAN-generated images and real ones, the latter of
which are achieved by GAN inversion Abdal et al. (2019) and re-generation.
Experiments show that our method can achieve high-quality disentangled face
editing results, outperforming state-of-the-art methods on both GAN-generate
and real-world images by a wide margin. In summary, our contributions include:

  * •

We propose a novel face attribute editing framework that searches for
instance-aware semantic direction in GAN latent space, which explicitly
promotes attribute variation disentanglement;

  * •

We propose a Disentanglement-Transformation ($DT$) metric to quantitatively
evaluate the editing efficacy and optimize our algorithm by leveraging this
metric;

  * •

We achieve high-quality results on both GAN-generated and real images that
significantly outperform existing methods.

###  2 Related Work

Face attribute editing aims to manipulate the interested face attribute while
preserving the rest. To achieve this goal, previous methods often leverage the
conditional GAN model Mirza and Osindero (2014); Odena et al. (2017).These
methods usually design the loss function Shen and Liu (2017); Bahng et al.
(2020); Zhu et al. (2017) or network architecture Choi et al. (2018); Zhang et
al. (2018); Liu et al. (2019); Lin et al. (2019); He et al. (2019) manually to
improve the output quality. Recently, 3D prior (_e.g._ 3DMM Blanz and Vetter
(1999)) is also introduced to encode the synthetic image Deng et al. (2020);
Tewari et al. (2020). These methods can generate high-quality results, but the
diversity of controllable attributes is limited by the 3D priors (_e.g._ , it
cannot well model and edit gender information). Other recent methods leverage
a high-quality synthetic face image dataset for disentangled representation
training Kowalski et al. (2020).

Another set of works Shen et al. (2020); Voynov and Babenko (2020); Shen and
Zhou (2020); Plumerault et al. (2020); Härkönen et al. (2020); Goetschalckx et
al. (2019) propose to edit face attribute by moving the latent code along a
specific semantic direction in the latent space of well-trained unconditional
GAN model Karras et al. (2018, 2019, 2020); Goodfellow et al. (2014). Shen _et
al._ Shen et al. (2020) learn an SVM boundary to separate the latent space
into the opposite semantic label and output the normal vector of the SVM
boundary as the semantic direction. Härkönen _et al._ Härkönen et al. (2020)
sample a collection of latent codes and perform PCA on them to find principle
semantic directions. However, these methods search the semantic directions on
the attribute level, which cannot handle complex attribute correlations. Our
method dynamically searches instance-aware semantic direction, which is
effective for attribute variation disentanglement. It can edit various face
attributes and obtain high-quality results.

Figure 2: The overview of our face attribute editing framework. The left part
shows our instance-aware semantic direction searching method in an incremental
update scheme. Here, $d$ and $\hat{d}(z)$ stand for the attribute-level and
instance-aware semantic directions, respectively. The right part illustrates
the instance-specific semantic direction search process.

###  3 Method

This section introduces the novel face attribute editing framework proposed by
this paper. We begin by introducing face attribute editing with GANs and
briefly revisiting prior methods using attribute-level latent directions,
after which we present our instance-aware direction search algorithm.

####  3.1 Semantic Direction for Attribute Editing

Given a pretrained generator $G$ from a state-of-the-art GAN model, _e.g._ ,
StyleGAN Karras et al. (2019), which maps a latent vector $z$ to a face image,
attribute editing can be achieved by moving $z$ along a certain direction in
the latent space. For real images, one can first embed them into the latent
space to obtain the latent vector $z$ and then modify it. The key is to find
suitable semantic directions for attribute editing.

Let $\mathcal{A}$ denote a collection of face attributes, _e.g._ , age,
gender, expression, and eyeglasses. For each attribute $X\in\mathcal{A}$,
existing methods seek for a direction $d_{X}$ corresponding to $X$. For
example, the InterfaceGAN method Shen et al. (2020) first generates a large
corpus of images $G(z)$ by randomly sampling $z$. Then it labels the
attributes of these images using a set of CNN binary classifiers $H(\cdot)$.
Finally, it trains a SVM to separate each attribute label in the GAN latent
space using these samples and outputs the normal vector of the SVM boundary as
the semantic direction $d_{X}$ for each attribute. To achieve disentangled
editing, it proposes to edit the primal attribute $A$ while preserving one
condition attribute $B$ (or more) via direction orthogonalization:

$d_{A|B}=d_{A}-<d_{A},d_{B}>d_{B},$ (1)

where $<\cdot,\cdot>$ denotes inner product. Directions with more than one
condition attribute can be obtained similarly as discussed in Shen et al.
(2020).

It can be seen that the directions so-obtained are at attribute level. They
are fixed for each attribute and are instance-agnostic. We instead propose to
incorporate instance information into direction search for better editing
performance.

####  3.2 Instance-Aware Semantic Direction Search

Our instance-aware semantic direction consists of two parts: _instance-
specific_ and _attribute-level_ directions. Next we first introduce the two
components and then describe how to combine them.

#####  3.2.1 Instance-Specific Semantic Direction

The generator $G$ maps a latent code $z$ to an image and an attribute
classifier $H$ maps an image $x\in\mathcal{X}$ to an attribute label. We can
bridge the GAN latent space and attribute space via compositing $H$ and $G$,
_i.e._ $H(G(\cdot))$. For attribute $X$, we can search for the instance-
specific semantic direction for instance $z$, denoted by $d_{X}(z)$, via
minimizing the following loss:

$\mathop{\arg\min}\limits_{d_{X}(z)}L(H(G(z+d_{X}(z))),y),$ (2)

where $y$ is the target attribute label and $L(\cdot,\cdot)$ is the
classification loss with binary cross entropy function

$L(x,y)=-y\log{x}-(1-y)\log(1-x).$ (3)

We simply use gradient descent to search for $d_{X}(z)$ in Eq. (2), where
$d_{X}(z)$ is updated by using an incremental direction updating scheme. To
streamline the presentation, more details of incremental update is deferred to
a later section. We further normalize $d_{X}(z)$ as the final instance-
specific semantic direction for $z$:

$\displaystyle d_{X}(z)$
$\displaystyle=\frac{-\nabla_{z}L(H(G(z)),y)}{\|\nabla_{z}L(H(G(z)),y)\|_{2}}$
(4) $\displaystyle=(2y-1)\frac{\nabla_{z}H(G(z))}{\|\nabla_{z}H(G(z))\|_{2}}.$

Eq. (4) shows that opposite directions can be obtained with $y=0$ and $1$,
respectively.

#####  3.2.2 Attribute-Level Semantic Direction

Attribute-level directions aggregate the information across all training
instances thus have higher resistance to noise. Similar to previous method, we
also leverage attribute-level directions for attribute editing and use the one
computed by InterfaceGAN as the default option.

Here we propose another way to compute the attribute-level direction. The
intuition is to bridge the gap between the local instance-level and the global
attribute-level information via sampling and averaging. Specifically, we can
first randomly sample a set of GAN latent codes $z$ and generate face images
by them accordingly. Then, we compute the instance-specific direction for each
sample and each attribute according to Eq. (4). Finally, for each attribute,
we can average the instance-specific directions from all samples as the
attribute-level direction $d_{X}$. We find that these $d_{X}$ lead to similar
attribute editing quality compared to the counterparts computed by
InterfaceGAN, but this approach is simpler and easier to implement and
eliminates the need for training SVMs.

Figure 3: An example $DT$ curve with gender as the primal attribute and
expression as the condition attribute.

#####  3.2.3 Instance-Aware Semantic Direction

Our instance-aware semantic direction is constructed by injecting instance-
specific information into the direction search process. Specifically, we
formulate it as the combination of the aforementioned attribute-level and
instance-specific directions:

$\hat{d_{X}}(z)=\lambda d_{X}+(1-\lambda)d_{X}(z),$ (5)

where $\lambda\in[0,1]$ is the control factor to balance these two components.
For conditional attribute editing, we rewrite Eq. (1) with our instance-aware
semantic direction as

$\hat{d_{A|B}}(z)=\hat{d_{A}}(z)-<\hat{d_{A}}(z),\hat{d_{B}}(z)>\hat{d_{B}}(z),$
(6)

where $A$ and $B$ are the primal and conditional attributes, respectively. We
set two different values for $\lambda_{1}$ and $\lambda_{2}$ for these two
attributes respectively, considering we might require a different quantity of
instance information when editing or disentangling face attributes. Note that
Eq. (6) degenerates to Eq. (1) when $\lambda_{1}=\lambda_{2}=1$. In practice,
we first solve for the $(\lambda_{1},\lambda_{2})$ pair which can produce best
results on a sample set, after which we fix them for the facial attribute
editing task. Next, we discuss how to solve the $(\lambda_{1},\lambda_{2})$
pair.

As mentioned previously, a good attribute editing should _i_) transform the
primal attribute $A$ to target label accurately, and _ii_) preserve the
condition attribute $B$ as much as possible. According to these criteria, we
propose to use a Disentanglement-Transformation ($DT$) curve to evaluate the
editing results for a pair of primal and condition attributes on a set of
samples. In a $DT$ curve, the $x$-axis represents the transformation accuracy
$p$, _i.e._ , the ratio of samples for which the primal attribute has been
transformed into target label correctly in the editing results. The $y$-axis
represents the disentanglement accuracy $q$, _i.e._ , the ratio of samples for
which the condition attribute on the edited images is consistent with their
original ones.

Specifically, we randomly sample a set of latent codes in the GAN latent space
and obtain the corresponding images generated by $G$. Then we edit these
images by changing the latent code $z$ to $z+k\cdot n\cdot\hat{d_{A|B}}(z)$,
where $k\in\mathbb{R}$ is the step size and $n\in\mathbb{N}$ is the number of
steps. The transformation and disentanglement accuracy ($p_{n}$, $q_{n}$) are
evaluated by CNN classifiers for the attributes. By continuously changing $n$
and evaluating ($p_{n}$, $q_{n}$), we obtain a $DT$ curve as illustrated in
Figure 3 (in practice, the $DT$ curve of our method is obtained via varying
the incremental updating steps; see next section).

With $DT$ curves generated, we can further compute the AUC (Area Under Curve)
of $DT$, and we choose the $(\lambda_{1},\lambda_{2})$ pair that maximizes the
average AUC for all possible primal-condition attribute pairs via:

$\mathop{\arg\max}\limits_{\lambda_{1},\lambda_{2}}\frac{1}{N}\sum_{A,B\in\mathcal{A}}\int^{1}_{0}q(A,B,\lambda_{1},\lambda_{2},p)\textup{d}p,$
(7)

where $N$ is the normalization factor. We numerically estimate the continuous
integral in Eq. (7) using quadrature.

Figure 4: Ablation study of the incremental updating scheme.  Figure 5:
Qualitative comparison of face attribute editing results between our method
and other competitors on GAN-generated images. By our method, not only the
primal attributes are successfully changed but also the condition attributes
are preserved much better than STGAN, GANSpace and InterfaceGAN.

####  3.3 Incremental Direction Updating

Here, we present an incremental instance-aware direction search scheme for our
editing method. We alternate between searching for the semantic direction and
updating the latent code (and the output image) . Given the current latent
code $z$ and generated image $G(z)$, we search for the new direction
$\hat{d}(z)$ and update the latent code as $z^{\prime}=z+k\cdot\hat{d}(z)$.
The whole process is illustrated in Figure 2.

###  4 Experiment

In this section, we evaluate our IALS method and compare it against the state-
of-the-arts. Due to space limitation, more experimental results and
discussions are included in the _suppl. material_.

####  4.1 Settings and Implementation Details

We consider five face attributes in our experiment, _i.e._ , expression, age,
gender, eyeglasses, and pose, as in Shen et al. (2020). We will focus on the
former four attributes in our experiments as we find pose is well disentangled
from other attributes in GAN latent space (similar observations are found in
Shen et al. (2020); Deng et al. (2020)). We test our framework on the
$\mathcal{W}$ space of StyleGAN generator trained on the FFHQ Karras et al.
(2018) and CelebA-HQ Karras et al. (2019) datasets. The attribute classifiers
$H(\cdot)$ are ResNet-18 He et al. (2016) networks trained on the CelebA
dataset Liu et al. (2015). The $DT$ metrics are computed by another set of
attribute classifiers with ResNet-50 structure. We empirically set the step
size of incremental updating in our method to $0.1$ in the following
experiments.

####  4.2 Ablation Study

In this part, we investigate the effect of direction control factor and
incremental updating scheme in our IALS framework.

#####  4.2.1 Control Factor Pairs

To study the behavior of different combination weights for attribute-level and
instance-specific directions, and solve for control factors $\lambda_{1}$ and
$\lambda_{2}$, we evaluate the $DT$ metric with $\lambda_{1}$ and
$\lambda_{2}$ evenly sampled in $[0,1]$ with a step size of 0.25 (hence 25
($\lambda_{1}$, $\lambda_{2}$) pairs in total). For each ($\lambda_{1}$,
$\lambda_{2}$) pair, we set $n_{max}=20$ (_i.e._ sample 20 points on the $DT$
curve) and $k=0.1$ and adopt the trapezoidal quadrature formula to approximate
the integral in AUC computation defined in Eq. (7).

Table 1 shows the average AUC of all $DT$ curves with different primal-
condition attribute combinations. We have the following three observations:
_i_) using attribute-level direction alone (_i.e.,_
$\lambda_{1}=\lambda_{2}=1$) is clearly non-optimal; _ii_) the attribute-level
information is more important to edit primal attribute ($\lambda_{1}\geq
0.75$); and _iii_) adding instance information to condition attribute
direction search significantly improves the editing results ($\lambda_{2}\leq
0.75$).

$\lambda_{1}$\$\lambda_{2}$ | 0.0 | 0.25 | 0.5 | 0.75 | 1.0
---|---|---|---|---|---
0.0 | 0.8393 | 0.8454 | 0.8445 | 0.8444 | 0.8302
0.25 | 0.8717 | 0.8675 | 0.8684 | 0.8706 | 0.8463
0.5 | 0.8882 | 0.8889 | 0.8886 | 0.8922 | 0.8660
0.75 | _0.9025_ | _0.8992_ | _0.9010_ | _0.9003_ | 0.8742
1.0 | _0.8999_ | _0.9003_ | _0.9006_ | _0.9013_ | 0.8662
Table 1: The average AUC (Area Under Curve) of all $DT$ curves with different
$(\lambda_{1},\lambda_{2})$ choices.

###### Discussion

Table 1 shows a performance plateau for a range of $\lambda$’s
($\lambda_{1}\geq 0.75$, $\lambda_{2}\leq 0.75$), indicating that our method
is insensitive to parameter selection within a reasonable range. It also shows
that adding moderate instance-level information for the condition attribute
would significantly better than using attribute-level information alone
($\lambda_{2}=1$), demonstrating the efficacy of our framework. In the
following we simply use $(\lambda_{1},\lambda_{2})=(0.75,0)$ for our editing
method.

#####  4.2.2 Incremental Updating

We further study the efficacy of our incremental updating scheme for instance-
aware direction search. Figure 4 shows a typical result when changing age from
young to old while preserving the eyeglasses attribute of the original face
image. It can be seen that the results are clearly inferior if we do not use
incremental updating (_i.e._ , we keep using the instance-aware direction
obtained in the first iteration).

Figure 6: Qualitative comparison of face attribute editing results between our
method and other competitors on real images from FFHQ. Our method yields
higher fidelity results than STGAN and better attribute variation
disentanglement than GANSpace and InterfaceGAN. Photos by Flickr users Elio
Yañez (Creative Commons BY 2.0 License) and sexinhose (Public Domain Mark 1.0
License).

####  4.3 Comparison with the State-of-the-arts

We compare our IALS method with several state-of-the-art face attribute
editing method proposed recently, including InterfaceGAN Shen et al. (2020),
GANSpace Härkönen et al. (2020), and STGAN Liu et al. (2019).

Both InterfaceGAN and GANSpace are methods based on GAN latent space search.
InterfaceGAN is a supervised semantic direction search method as mentioned in
the previous section, while GANSpace is an unsupervised one which performs PCA
on the sampled latent codes to find principle semantic directions in the
latent space. We assign the directions found by GANSpace to interpretable
meanings following Shen and Zhou (2020). The STGAN method is based on image
generation using conditional GAN. It uses an encoder-decoder architecture with
a well-designed selective transfer unit for attribute editing. For these
methods, we use the code or pretrained models released by the authors for
evaluation and comparison.

#####  4.3.1 Qualitative Results

we compare our method with the competitors on synthetic images and real images
respectively. We adopt the conditional manipulation setup Shen et al. (2020)
with one primal attribute to be changed and one condition attribute to be
preserved for the semantic direction based methods, and directly send the
source image to the conditional-GAN based method.

Some typical qualitative results on GAN-generated images are provided in
Figure 5. In can be seen that as the degree of attribute change increases,
STGAN generated blurry images while the semantic direction based methods
outputted high-fidelity results. Furthermore, our method can well preserve the
condition attribute of the original image when editing the primal attribute
while the other two semantic direction based competitors often fail.
InterfaceGAN and GANSpace only use attribute-level direction while ignoring
the instance information when facial editing. By contrast, our method combines
instance-specific and attribute-level information which results in much better
disentangled facial editing results. The results also demonstrate that our
method outperforms state-of-the-art methods on GAN-generated images.

Figure 6 shows two typical examples to illustrate the attribute editing
efficacy of different methods on real images111For InterfaceGAN, GANSpace and
our method, we firstly embed the given images into the $\mathcal{W}+$ latent
space of StyleGAN using Abdal et al. (2019) and then edit them. where the goal
is to edit age while preserving the eyeglasses attribute. We find that STGAN
outputted blurry results again. On the other hand, the condition attributes
(_i.e._ eyeglasses) are changed by GANSpace and InterfaceGAN. In contrast, our
method obtains superior results in terms of both image quality and attribute
variation disentanglement. It produces high fidelity results with satisfactory
attribute modification and preservation.

Figure 7: Human satisfactory rate comparison between our method and other
competitors on different primal-condition attribute pairs (exp. and eye.
denote expression and eyeglasses respectively)

#####  4.3.2 Quantitative Results

To further evaluate the performance of our IALS method, we conducted a user
study. we recruited 100 people and presented them with 720 groups of data,
with each group consisting of 5 images - the original face image and the
facial editing results of our method and the other 3 competitors. Each person
was randomly assigned 18 groups of data and asked to choose all of the results
they are satisfied with according to three criteria: the result looks natural,
the primal attribute is well changed, and condition attribute is well
preserved. Note that we did not ask them to select the best result, as optimal
facial editing results are often not unique.

The results in Fig. 7 show that our method obtained the highest average
satisfactory rate (69.66% for ours vs. 62.06% for InterfaceGAN, 35.44% for
GANSpace and 6.48% for STGAN). It significantly outperforms the other methods
on some challenging attribute pairs such as gender as the primal attribute and
expression as the condition attribute (76.44% for ours vs. 51.92% for
InterfaceGAN).

###  5 Conclusion

We have proposed a novel IALS framework to achieve disentangled face attribute
editing with GAN latent space search. The key ingredient is the integration of
attribute-level and instance-specific directions, which leads to accurate
target attribute control and, more importantly, highly disentangled attribute
editing. To better integrate the attribute-level and instance-specific
directions, we introduce a Disentanglement-Transformation ($DT$) metric to
find suitable control factor. The experiments collectively show that our
approach obtains significantly better results than previous methods.

### Acknowledgements

This research was supported by the National Natural Science Foundation of
China under Grants No. 61827901 and No. 62088101.

## Supplementary Material

###  6 More Implementation Details

We consider five attributes in our implementation, _i.e._ , expression, age,
gender, eyeglasses and pose, as in Shen et al. (2020). Note that our method
can be extended to edit any other attributes in the presence of corresponding
CNN classifier. To search instance-specific directions, we train a set of
binary CNN classifiers $H(\cdot)$ on the CelebA dataset Liu et al. (2015) with
$256\times 256$ image resolution. Unlike other attributes, pose is not
annotated in CelebA. We therefore construct the training set for it by
ourselves. Specifically, we first use the landmarks provided by CelebA
annotations to estimate the Euler angles of the face images, and then label
images with yaw angle greater than $30^{\circ}$ as a positive sample and less
than $-30^{\circ}$ as a negative sample, respectively. In the training phase,
we first center-crop the original $178\times 218$ images and then resize them
to $256\times 256$ before feeding them to $H(\cdot)$ (we adopt the ResNet-18
He et al. (2016) architecture), similar to other attributes. Table 2 shows the
precision and recall of our trained classifiers for all attributes.

For instance-specific direction searching, we resize the images generated by
$G(\cdot)$ (_i.e._ , the StyleGAN Karras et al. (2019) generator) from
$1024\times 1024$ to $256\times 256$ before feeding them to $H(\cdot)$. In the
instance-aware direction constructing process, we first normalize the
instance-specific direction $d_{X}(z)$ and the attribute-level direction
$d_{X}$ to unit length, and then combine them by the control factor $\lambda$.
In addition, we normalize $\hat{d}_{X}(z)$ as the final instance-aware
semantic direction for $z$ on attribute $X$.

The pseudocode of our method is provided in Algorithm 1.

| Smiling | Gender | Age | Eyeglasses | Pose
---|---|---|---|---|---
Precision | 0.9407 | 0.9613 | 0.9609 | 0.9951 | 0.9884
Recall | 0.9094 | 0.9855 | 0.8593 | 0.9264 | 0.9906
Table 2: Performance of the CNN attribute classifiers (ResNet-18) trained on
CelebA. These classifiers are used to edit attributes in our method. Algorithm
1 Instance-Aware Semantic Direction Search for Face Attribute Editing

Input: latent vector $z$, StyleGAN generator $G$, CNN classifier $H$,
attribute-level direction $d_{X}$, target label $y$, control factor $\lambda$,
step size $k$, number of step $N$.
Output: a list $\mathcal{I}$ containing $N$ images with continuous attribute
variation towards the target label.

1: Let $i\leftarrow 0,~{}\mathcal{\mathcal{I}}\leftarrow\emptyset$.

2: while $i<N$ do

3: $d_{X}(z)=\frac{\partial L(H(G(z)),y)}{\partial z}$ (Eq. 4 in the main
paper)

4: $d_{X}(z)\leftarrow Normalize(d_{X}(z))$

5: $d_{X}\leftarrow Normalize(d_{X})$

6: $\hat{d}_{X}(z)=\lambda d_{X}+(1-\lambda)d_{X}(z)$ (Eq. 5)

7: $\hat{d}_{X}(z)=Normalize(\hat{d}_{X}(z))$

8: $z\leftarrow z+k\cdot\hat{d}_{X}(z)$

9: $\mathcal{I}\leftarrow AppendList(\mathcal{I},G(z))$

10: $i\leftarrow i+1$

11: end while

12: return $\mathcal{I}$

###  7 More Results and Comparisons

####  7.1 More Results of Our Method

We show more real-image editing results of our method on different attributes
(including pose) in Figs. 8 and 9. We can see that our method can obtain high-
quality disentangled results when editing various face attributes. It implies
the effectiveness of our method.

####  7.2 More Results on Multiple Condition Attributes

In Fig 10, we show the facial editing results of our method with a different
number of condition attributes. It can be observed that our method is able to
handle complex disentanglement between face attributes by adding more
condition attributes into the conditional manipulation operation using the
strategy presented in Shen et al. (2020).

####  7.3 More Ablation Study on Control Factors.

In Figs. 11 and 12, we show the qualitative ablation results of varying
$\lambda$ in the primal and condition directions, respectively. We can see
that _i_) our method would fail to change the desired primal attribute if we
abandon the attribute-level information, and _ii_) our method can effectively
obtain disentangled results when adding instance information, which is
consistent with the conclusion in the main paper.

####  7.4 More Comparison with the State-of-the-Arts

Here, we provide more qualitative results compared to the state-of-the-art
methods, including the semantic direction based methods (InterfaceGAN Shen et
al. (2020) and GANSpace Härkönen et al. (2020)), and the conditional-GAN based
method Liu et al. (2019) on GAN-generated images. We adopt the conditional
manipulation setup Shen et al. (2020) for the semantic direction based
methods, and directly send the source image to the conditional-GAN based
method, as in the main paper. Besides, the results in Figs. 13, 14, and 15
show that our method performs better than the other competitors.

####  7.5 More Comparison with InterfaceGAN on attribute-level direction

In the main paper, we point out that the attribute-level direction computed by
our method leads to similar editing quality with the counterpart computed by
InterfaceGAN Shen et al. (2020). Here, we show qualitative comparison in Fig.
16 and demonstrate their similar efficacy. Note that our process is easier to
implement, as we eliminate the need for training SVMs. Our main contribution
is to use the instance-aware semantic direction for face attribute editing.

###  8 More Details of the Incremental Updating Scheme

In the main paper, we propose an incremental instance-aware direction search
scheme, instead of keep using the instance aware direction $\hat{d}_{X}(z)$ of
the original latent code $z$ when editing. We treat the latent codes formed in
the editing process (_e.g._ $z+k\hat{d}_{X}(z)$, where $k$ is the step size)
as different instances and use their instance-aware direction to edit them.
For example, we denote $z^{\prime}=z+k\hat{d}_{X}(z)$ and adopt
$\hat{d}_{X}(z^{\prime})$ to edit $z^{\prime}$ instead of directly using
$\hat{d}_{X}(z)$. Note that the incremental updating scheme makes no
difference if we use the attribute-level direction alone in the editing
process.

###  9 Discussion about Condition Attribute Selection

In practice, one can freely select one or multiple condition attributes for
primal attribute editing, depending on the targeted editing goals. In general,
to achieve accurate and disentangled editing for a primal attribute, selection
of condition attribute is closely related to the data distribution of GAN’s
training set. For example, elder people tend to wear eyeglasses in FFHQ Karras
et al. (2019), so age and eyeglass attributes are likely to be entangled with
each other in the latent space. One can also follow this strategy for
condition attribute selection.

###  10 Limitations

We have demonstrated that our method is effective for disentangled face
attribute editing. It still has some limitations of the semantic direction
based facial editing algorithm. For example, our method cannot model non-
binary attributes (_e.g._ hair color and hairstyle), as it needs a binary
classification boundary for facial editing. In addition, we observed that the
image editing quality on embedded latent code obtained by GAN inversion
approaches Abdal et al. (2019, 2020) is slightly lower than random latent
codes from the GAN generator. A recent work from Zhu et al. (2020) proposed to
eliminate the above performance gap by adding a regularization term to
encourage the embedded latent code to be semantically meaningful, which we
will explore in our future work.

Figure 8: More qualitative results of our method on real image editing. The
images are taken from FFHQ.

Figure 9: Qualitative results of our method on real image editing. The images
are taken from FFHQ.

Figure 10: Qualitative results of our method on multiple condition attributes.
Figure 11: Qualitative ablation study on $\lambda_{1}$ (we choose
$\lambda_{1}=0.75$ in this paper based on the $DT$ metric).  Figure 12:
Qualitative ablation study on $\lambda_{2}$ (we choose $\lambda_{2}=0$ in this
paper based on the $DT$ metric).

Figure 13: More qualitative comparison with the state-of-the-art on GAN-
generated images.

Figure 14: More qualitative comparison with the state-of-the-art on GAN-
generated images.

Figure 15: More qualitative comparison with the state-of-the-art on GAN-
generated images.

Figure 16: Qualitative comparison of face image editing using the attribute-
level directions computed by our method and InterfaceGAN.

### References

  * Abdal et al. [2019] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In ICCV, pages 4431–4440, 2019.
  * Abdal et al. [2020] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In CVPR, pages 8296–8305, 2020.
  * Awiszus et al. [2019] Maren Awiszus, Hanno Ackermann, and Bodo Rosenhahn.  Learning disentangled representations via independent subspaces.  In ICCVW, 2019.
  * Bahng et al. [2020] Hyojin Bahng, Sunghyo Chung, Seungjoo Yoo, and Jaegul Choo.  Exploring unlabeled faces for novel attribute discovery.  In CVPR, pages 5821–5830, 2020.
  * Blanz and Vetter [1999] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In SIGGRAPH, pages 187–194, 1999.
  * Choi et al. [2018] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In CVPR, pages 8789–8797, 2018.
  * Deng et al. [2020] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong.  Disentangled and controllable face image generation via 3d imitative-contrastive learning.  In CVPR, pages 5154–5163, 2020.
  * Goetschalckx et al. [2019] Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola.  Ganalyze: Toward visual definitions of cognitive image properties.  In ICCV, pages 5744–5753, 2019.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NIPS, pages 2672–2680, 2014.
  * Gu et al. [2019] Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan.  Mask-guided portrait editing with conditional gans.  In CVPR, pages 3436–3445, 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * He et al. [2019] Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen.  Attgan: Facial attribute editing by only changing what you want.  IEEE Transactions on Image Processing, 28(11):5464–5478, 2019.
  * Härkönen et al. [2020] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  In NIPS, 2020.
  * Karras et al. [2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In ICLR, 2018.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  CVPR, pages 4401–4410, 2019.
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, pages 8110–8119, 2020.
  * Kowalski et al. [2020] Marek Kowalski, Stephan J. Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, and Jamie Shotton.  Config: Controllable neural face image generation.  In ECCV, 2020.
  * Lin et al. [2019] Yu-Jing Lin, Po-Wei Wu, Che-Han Chang, Edward Chang, and Shih-Wei Liao.  Relgan: Multi-domain image-to-image translation via relative attributes.  In ICCV, pages 5914–5922, 2019.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In ICCV, pages 3730–3738, 2015.
  * Liu et al. [2019] Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen.  Stgan: A unified selective transfer network for arbitrary image attribute editing.  In CVPR, pages 3673–3682, 2019.
  * Mirza and Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * Odena et al. [2017] Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional image synthesis with auxiliary classifier gans.  In ICML, pages 2642–2651, 2017.
  * Plumerault et al. [2020] Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot.  Controlling generative models with continuous factors of variations.  In ICLR, 2020.
  * Shen and Liu [2017] Wei Shen and Rujie Liu.  Learning residual images for face attribute manipulation.  In CVPR, pages 1225–1233, 2017.
  * Shen and Zhou [2020] Yujun Shen and Bolei Zhou.  Closed-form factorization of latent semantics in gans.  arXiv preprint arXiv:2007.06600, 2020.
  * Shen et al. [2020] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In CVPR, pages 9243–9252, 2020.
  * Tewari et al. [2020] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Perez, Michael Zollhofer, and Christian Theobalt.  Stylerig: Rigging stylegan for 3d control over portrait images.  In CVPR, pages 6142–6151, 2020.
  * Voynov and Babenko [2020] Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In ICML, 2020.
  * Zhang et al. [2018] Gang Zhang, Meina Kan, Shiguang Shan, and Xilin Chen.  Generative adversarial network with spatial attention for face attribute editing.  In ECCV, pages 422–437, 2018.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In ICCV, pages 2242–2251, 2017.
  * Zhu et al. [2020] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  ECCV, 2020.


# Using Machine Learning to Detect Rotational Symmetries from Reflectional
Symmetries
in 2D Images

Koen Ponse  0000-0002-6542-3042 LIACS, Leiden University
Leiden, The Netherlands
<EMAIL_ADDRESS>Anna V. Kononova  0000-0002-4138-7024 LIACS, Leiden
University
Leiden, The Netherlands
<EMAIL_ADDRESS>Maria Loleyt  Ornamika
Moscow, Russia
<EMAIL_ADDRESS>Bas van Stein  0000-0002-0013-7969 LIACS, Leiden
University
Leiden, The Netherlands
<EMAIL_ADDRESS>
###### Abstract

Automated symmetry detection is still a difficult task in 2021. However, it
has applications in computer vision, and it also plays an important part in
understanding art. This paper focuses on aiding the latter by comparing
different state-of-the-art automated symmetry detection algorithms. For one of
such algorithms aimed at reflectional symmetries, we propose post-processing
improvements to find localised symmetries in images, improve the selection of
detected symmetries and identify another symmetry type (rotational). In order
to detect rotational symmetries, we contribute a machine learning model which
detects rotational symmetries based on provided reflection symmetry axis
pairs. We demonstrate and analyze the performance of the extended algorithm to
detect localised symmetries and the machine learning model to classify
rotational symmetries.

###### Index Terms:

symmetry detection, machine learning, rotational symmetry, reflection symmetry

##  I Introduction

In everyday life, humans and animals use their innate perception of symmetry
to identify objects quickly. Also in computer vision, symmetry plays an
important role in understanding objects. For example, the knowledge that an
object is symmetrical, such as a car, means that we do not need to observe the
entire object in order to know how it looks like. Symmetry also plays a
significant role in art as it is one of the main principles with which artists
create the composition of an artwork [1]. Automated symmetry detection can
therefore improve various computer tasks and aid art scientists and cultural
heritage projects. In this paper we aim to help the latter by analysing the
current state-of-the-art automated symmetry detection algorithms on applied
art data. Furthermore, an algorithm extension is proposed to find local
reflection symmetries and a machine learning model is proposed to detect
rotational symmetries using reflection symmetry pairs.

Online archive of world ornaments and patterns Ornamika [2] has kindly
provided us with their database containing a set of diverse images of art
pieces. Labelling these images with symmetries and symmetry types will help
organise their database. This will, in turn, aid designers using the database.
Due to a large number of images in the database, an automated approach for
labelling the images is required. The Ornamika data set differs from most data
sets on which existing symmetry detection algorithms have been tested – most
papers test their algorithms on data sets containing real-world photos with
complex backgrounds, lighting and depth. In contrast, the Ornamika data set is
mainly comprised of ornaments on a 2D plane. Often the image features a flat
textile art piece without any irrelevant background objects. When an image in
the Ornamika data set is not a flat surface, it is often an object which has
been photographed en face in good lighting conditions. Since other papers have
used a different kind of data, it is currently unknown which algorithms
perform best on a data set like Ornamika’s. Additionally, it is difficult to
predict whether existing algorithms can be improved to yield better results.

Over the years, many papers [3, 4, 5, 6, 7] have been published proposing
various symmetry detection methods. Up until now, three symmetry detection
competitions have been held, with increased task complexity in each
competition. Further details about these competitions are provided in Section
III. Despite the amount of work in the field, automated symmetry detection
does still prove to be a difficult task, as we can conclude based on the last
symmetry detection competition in 2017 [8]. At the same time, an algorithm
proposed in 2006 is still considered competitive.

In this paper we discuss the current state-of-the-art algorithms for symmetry
detection in Section III. Next, we compare these algorithms on the Ornamika
data set to determine which algorithm works best on such art data. A global
reflection symmetry detection algorithm [9] is selected from this comparison
and several improvements are proposed in Section V.

We propose to use the algorithm output in order to detect rotational
symmetries using a machine learning model (Section V-B) and we show that our
model is able to detect rotational symmetries with an accuracy of over 90% on
the test set.

We note that, in theory, any reflection symmetry detection algorithm will be
able to use our trained model to detect rotational symmetries, as its input
only requires a set of coordinates of two lines and their respective symmetry
scores. Most reflection symmetry detection algorithms produce these attributes
to draw symmetry lines. Our final algorithm is validated on 200 images. The
results of this, together with the source code, can be found on
GitHub111https://github.com/Koen-Git/ColorSymDetect.

##  II Types of Symmetry

Symmetries can be categorised into four different types, namely [1]:

  * •

Reflection \- where the image appears mirrored alongside a straight axis.
Figure 1(a) shows an example where the left half of the image is a mirrored
version of the right half.

  * •

Rotational \- where elements are repeated around a centre point. Figure 1(b)
shows an example where all the spikes of the wheel are centred around the
middle of the ferris wheel.

  * •

Translation \- where elements are repeated while rotated in a different
direction. Figure 1(c) displays an example where the metallic street stones
are repeated in a different direction.

  * •

Glide reflection \- is a combination of reflection symmetry and translation
symmetry. Figure 1(d) shows an image where the footprints are mirrored along a
straight axis and are also translated in a forward direction.

(a) Reflection

(b) Rotation

(c) Translation

(d) Glide-reflection

Figure 1: The four types of symmetry with the symmetry axis shown in yellow
(source: [1]).

##  III Related Work

Three different symmetry detection competitions were held in 2011, 2013 and
2017 [10, 11, 8]. These symmetry detection competitions featured state-of-the-
art algorithms of their respective times. The first two symmetry competitions
featured reflection, rotation and translation symmetries. In both the
reflection and rotation categories Loy and Eklundh [4] came out as the winner.
The latest symmetry detection competition, held in 2017, featured the medial
axis detection category for the first time. Loy and Eklundh [4] still acted as
the baseline for reflection and rotational symmetries. A second baseline
algorithm [12] for reflection symmetry was introduced and outperformed Loy and
Ekhlundh on single reflection symmetries. Surprisingly Loy and Eklundh once
again won in detecting multiple symmetries, with Elawady et al. [9] coming in
second place. For translation symmetries, the scores of the challenger [13]
and baseline [14] algorithms were still relatively low, with
$F_{1}$-scores222$F_{1}$-score defined in (1) offers a single-value score for
system performance comparison, by using the precision and recall [8] of 0.19
and 0.20 respectively. In the new category (medial axis) two algorithms
managed to beat the baseline algorithm. Liu et al. [15] came in first place
with an $F_{1}$-score of 0.73. For comparison, humans achieved an
$F_{1}$-score of 0.80 [8]. Funk and Liu [16] acted as a second baseline in the
rotational symmetry category and the authors themselves claim better
performance over Loy and Eklundh. However, Funk and Liu only focus on finding
the rotational centres. Furthermore, Funk and Liu produces a dense heatmap of
possible symmetries instead of coordinates of possible symmetries.

Since then, Elawady has evaluated different versions of his algorithm on seven
different symmetry data sets, four for single symmetry detection and three for
multiple axis detection [1] – the results are listed in Table I. These results
show that Elawady’s algorithm has a slight edge in the last competition’s
multiple axis training data set (ICCV17m). The test set of this same
competition shows Loy has the higher performance [8]. Nonetheless, when
considering the other 6 data sets, we can see Elewady has a clear advantage in
each of them over Loy and Eklundh.

TABLE I: Elawady algorithm compared to Loy and Eklundh and Cicconet et al. on seven different data sets. Each cell lists the amount of top-ranked detected symmetry axis that corresponds to a ground truth in the data set. Between brackets the $\max\\{F1\\}$ scores are listed (data source: [1]). Data set (#images) | Loy [4] | Cicconet [17] | Elawady [9]
---|---|---|---
PSU (157) | 81 (0.514) | 90 (0.569) |  113 (0.724)
AVA (253) | 174 (0.690) | 124 (0.493) |  182 (0.729)
NY (176) | 98 (0.528) | 92 (0.526) |  135 (0.766)
ICCV17 (100) | 52 (0.507) | 53 (0.536) |  70 (0.713)
PSUm (142) | 69 (0.292) | 68 (0.159) |  75 (0.338)
NYm (63) | 32 (0.337) | 36 (0.237) |  40 (0.411)
ICCV17m (100) | 54 (0.273) | 39 (0.207) |  57 (0.285)
Total (991) | 560 (0.449) | 502 (0.390) |  672 (0.567)

We can note that $F_{1}$-scores are still very low during the last competition
for multiple and single reflection symmetry detection: $F_{1}$-scores of 0.3
and 0.52 respectively. Elawady’s results show higher $F_{1}$-scores: 0.73 for
single reflection data sets and 0.34 for multiple reflection data sets.
However, it is clear that there is still much room for improvement in this
field.

Detecting rotational symmetries has not received much attention in the last
symmetry competition. Loy and Eklundh [4] won this category in the first and
second symmetry competition and is considered the best baseline for this
category. Funk and Liu [16] claim better performance but only focus on finding
the rotational centres in their work.

##  IV Evaluation methods

In 2019 Elawady published his thesis [1] in which he states the evaluation
method for reflection symmetry used in the competitions is unclear. Elawady
explains how to evaluate symmetry axis to their corresponding ground truths.
Just like the competitions, he uses the $\max F_{1}$ score defined in (1) as a
global quality measure for comparing algorithms. The $F_{1}$ score is the
harmonic mean of precision $P$ and recall $R$:

$F_{1}=\frac{2P\cdot R}{P+R}$ (1)

Precision $P$ is defined as the number of true positives over the number of
true positives plus the number of false positives. While recall $R$ is defined
as the number of true positives over the number of true positives plus the
number of false negatives. For evaluation, one can alter thresholds of when a
detected symmetry is classified as a true positive. This creates a plot of
$F_{1}$ scores which illustrates the trade off between precision and recall.
The maximum in this plot is a convenient performance measurement for detection
algorithms.

Furthermore, Elawady emphasizes that a pair of similar symmetry lines should
not both be evaluated. Instead, only the higher ranked symmetry line (with a
higher score) should be considered. Next, he states that multiple detected
symmetry axes can be grouped to form a single ground truth. Both practices are
visualised in Figure 2. It is unclear whether the symmetry competitions also
adopted these methods, as they only list the precision and recall with their
corresponding $F_{1}$-scores on a single test set.

Figure 2: Example with two ground truths (red) and six detected symmetry lines
(blue). The detected symmetry lines are ordered by their respective symmetry
score produced by the algorithm. SC1 and SC2 are matched to GT1 (the
difference in their slope and their centres are within some threshold) and SC4
is matched to GT2. Furthermore, SC6 can be disregarded as it is similar to the
higher-ranked SC5. (source: [1])

##  V Methodology

Based on our literature review, we conclude that for reflection symmetries,
Elawady’s WaveletSym algorithm [9] achieves the highest performance on the
largest distribution of data sets while using a clear evaluation method.
Furthermore, we found that WaveletSym processes images in under ten seconds on
average. In Section V-B, we compare the computation time of our work with Loy
and Eklundh. We find that Loy and Eklundh’s method requires four minutes on
average to process an image. This time is comparable to our proposed work in
which we recursively use WaveletSym multiple times. For these reasons, we
consider WaveletSym currently state of the art for detecting reflection
symmetries333We are aware that the latest version of Elawady’s algorithm [1]
is even more sophisticated but since no open-source version is available, we
used a slightly older version of the algorithm for our research. Note that
this does not impact the performance of the proposed machine learning model.

In this Section we will examine the performance of WaveletSym on our data set
and propose post-processing improvements to this algorithm. Furthermore, we
propose a machine learning model to detect rotational symmetries using the
output of WaveletSym and compare the final performance of our model against
Loy and Eklundh [4].

###  V-A Wavelet-based Reflection Symmetry Detection

Elawady’s [9] WaveletSym algorithm produces a multidimensional array of
symmetry lines with their corresponding symmetry score. As shown in Figure 3,
these lines are then drawn on the image with the legend of the scores
attached.

Figure 3: Results as produced by [9] without any alterations. The top 5
discovered symmetries are drawn on the image. The numbers in the legend
represent each lines symmetry score, how strong of a symmetry is present.

#### Initial results

WaveletSym produces pretty satisfying results, as is shown in Figure 3. With
default settings, the top 5 symmetries are drawn on the image. We can see
WaveletSym often finds the main symmetry axis, with secondary symmetries being
found relatively accurate as well. We should note that in a data set mainly
containing art pieces like the Ornamika [2] data set, the main symmetry axis
is often the least interesting. This is because in a vast majority of the
images a reflection symmetry exists in the centre of the image.

#### Further results

When inspecting the resulting images in Figure 3 we observe a few shortcomings
of the WaveletSym algorithm in its current state.

  * •

As is described in [9], the algorithm only deals with global reflection
symmetries, while we are also interested in more localised symmetries.
Especially, when considering that the main global symmetry axis is often the
least interesting in our data set.

  * •

This algorithm only deals with reflection symmetries, but we are also
interested in rotational symmetries since many art images contain such
symmetries.

  * •

All symmetry lines with a score over 0 are returned by the algorithm, with no
way to filter on accurate symmetry lines. Yet, the scores themselves are not a
very meaningful metric when comparing scores from one image to another.

  * •

The algorithm produces many symmetry lines that are very similar to each
other.

#### Improvements

Despite these shortcomings, WaveletSym is still a promising algorithm, and
some alterations will mitigate these described problems. In the next
paragraph, we will go through these problems one by one and suggest
improvements to mitigate them as much as possible.

  * •

Localised symmetries: We will cut the images at certain points to find
localised symmetries in smaller parts of the images. For determining where to
cut the image, we could cut at as many places as possible. However, this
results in undesired symmetry lines and most notably increases computation
time exponentially. Instead, we settled on recursively cutting up the image
using the top three symmetry lines. This creates six new (smaller) images
which can be processed in the same manner. This resulted in a good balance
between computation time and retrieving accurate local symmetries.

  * •

Rotational symmetry: First, we observed that when two lines cross and are
perpendicular, there is likely a rotational symmetry around the intersection
as shown in the left subfigure of Figure 4. The rotational symmetries as drawn
with this naive approach lead to unwanted results, as shown in the right
subfigure of Figure 4. These results are partly caused by our earlier
described problem of similar symmetry lines being drawn close to each other;
this also causes multiple rotational symmetries to be drawn. The other problem
is that there are not necessarily rotational symmetries at the intersections
between two perpendicular lines. Requiring the two perpendicular lines to have
similar symmetry scores improves the performance. Besides these simple rules,
we contribute a machine learning approach for detecting rotational symmetries
with symmetry line pairs as an input. We will elaborate more on this in
Section V-B.

  * •

Filtering: To determine if symmetries should be drawn on the image, we can use
the normalised score for each image or sub-image. First, we check if the top
symmetry found reaches our threshold parameter. Then we check if the other
detected symmetry scores lie within a second threshold compared to the first
one. Symmetries that do not reach these thresholds will be removed.

  * •

Similar symmetries: Since WaveletSym is producing a lot of similar symmetry
lines, we can loop over each line and compare them to each other. If we find
lines with a similar slope and centre, we can discard the line with a lower
symmetry score. The same procedure can be performed on rotational symmetries,
i.e. circles drawn with a similar radius and centre.

Figure 4: Rotational symmetries drawn with with a naive approach. The red
lines represent the discovered reflection symmetries. Any perpendicular lines
will draw a yellow rotational symmetry in their intersection.

#### Results after improvements

Figure 5 shows images with all our proposed adjustments. The rotational
symmetries are detected in these images using the described rule-based
approach, not using machine learning. For this set, we have set our first
parameter, ’symThresholdAC’, to 0.20. This parameter dictates whether the top
symmetry of each image or sub-image should be kept or not. Any lines with
scores below this parameter are removed. Lines generated as part of the same
recursive loop are removed as well. We have set our second parameter,
’normThresholdAC’, to 0.70. This parameter dictates which other symmetry lines
are kept. Lines with scores relative to the top symmetry score which are lower
than this threshold, are removed. The third parameter, ’CircleSymThreshold’,
was set to 0.75; this implies the maximum difference in symmetry scores
between two lines can at most be 25% for them to create a rotational symmetry.

The resulting images show that with these adjustments, WaveletSym is capable
of finding localised symmetries and rotational symmetries. However, some
rotational symmetries are still not found, and some smaller ones are found
inaccurately. Additionally, some localised symmetries which are discovered are
inaccurate. The performance of this rotational symmetry detection method will
function as a good baseline for our proposed machine learning model, which we
will introduce next.

Figure 5: Resulting images of WaveletSym with our alterations outlined in
Section V-A. The blue lines are the discovered global symmetries. Other
colours represent different depths of the recursive function and, in turn,
represent discovered localised symmetries. The colours of the different depths
are mapped in the colour map to the right. Colours presented lower on the map
represent more localised symmetries.

###  V-B Machine Learning for Rotational Symmetries

Discovering rotational symmetries as described in the previous Section V-A
amounts to reasonable results. In order to improve these results further we
propose to use a machine learning model to learn when reflection symmetry
lines indicate a rotational symmetry at their intersection. For this
classification task, we first had to label symmetry line combinations
manually. We labelled lines that intersected and had a rotational symmetry at
the intersection as true. Lines were also required not to be parallel or close
to parallel. Other line pairs are labelled as false. In order to balance the
data set, we also rotated every true case by 0.5 degrees 720 times.

Thus, our input is comprised of $x$ and $y$ coordinates of two line pairs with
their corresponding symmetry score. We pre-process this data into several
(hand-crafted) features as the input for our model, namely: the difference of
the symmetry scores, the slopes of both lines and their difference, the
perpendicular slope of one of the lines and the difference of this
perpendicular slope to the other line, difference in length of the lines,
whether or not the lines intersect, and the difference of each line’s endpoint
to the intersecting point.

We compared eight different machine learning algorithms trained on the
training data and validated on a different set of validation data. These sets
contain 153785 and 517213 samples respectively. The results are available in
Table II333In Table II and III: all non listed parameters have default values,
using sklearn 0.24.2 (python 3.7.10). Each algorithm, except for K-Nearest
Neighbors, was also set with ”random_state=44” and ”class_weight=’balanced’”..

TABLE II: Accuracy scores of various algorithms on validation data. While SVC_auto achieves the highest accuracy, the ROC curve (Figure 6) shows that Random Forest and LinearSVC perform the best in terms of AUC. Algorithm |  Parameters33footnotemark: 3 |  Accuracy |  AUC
---|---|---|---
Log_Regression |  max_iter=10000 |  0.82181 |  0.50
Random Forest |  |  0.85381 |  0.94
SVC |  |  0.19588 |  0.51
SVC_alt |  gamma=”auto” |  0.88888 |  0.76
KNN |  |  0.86131 |  0.71
Decision Tree |  |  0.87025 |  0.67
SGDClassifier |  |  0.46510 |  0.58
LinearSVC |  max_iter=5000 |  0.88024 |  0.94
Figure 6: ROC Curve of the scores in Table II. Random Forest and LinearSVC
come out on top in terms of AUC.

From these model configurations, the Support Vector Classification (SVC), with
its ’gamma’ hyperparameter set to ”auto” achieved the highest accuracy.
Alternatively to evaluating the accuracy we can instead calculate the true-
positive rate $(\frac{TruePositives}{TruePositives+FalseNegatives})$ and the
false-positive rate $(\frac{FalsePositives}{FalsePositives+TrueNegatives})$.
Plotting both of these rates against each other at different threshold values
will produce a receiver operating characteristic curve (ROC curve). When
plotting the true positive rate on the y-axis of the curve, it is often
desired to have a large area under the curve (AUC), as this corresponds with a
high true positive rate and a low false positive rate. By further evaluating
the Support Vector Classification, we find that this algorithm is scoring
poorly when considering the area under curve score. Random Forest and
LinearSVC achieved better performance with scores of 0.94.

We re-evaluated the three top-performing algorithms using $36$ different
hyperparameter configurations, the results of this experiment can be found on
our Github page. From the hyperparameter optimization step, we evaluated the
top-performing Random Forest and LinearSVC configurations one more time on a
completely different test set, containing 263569 samples. The final results of
this test are found in Table III33footnotemark: 3, and the Receiver Operating
Characteristic curve is found in Figure 7. We conclude that Random Forest with
”max_depth=10” and ”criterion=’entropy’” performs the best on various sets of
our data. max_depth dictates the maximum depth of each decision tree in the
Random Forest algorithm. criterion=entropy changes the splitting criterion of
the trees to information gain.

TABLE III: Accuracy scores of the best performing algorithms with their hyper parameters tuned. These tests are performed on a separate test set, which is entirely different from the earlier validation set. algorithm |  parameters33footnotemark: 3 |  accuracy |  AUC
---|---|---|---
Random Forest |  max_depth_10, criterion=”entropy” |  0.97348 |  0.996
Random Forest |  min_samples_leaf=3, criterion=”entropy”, max_depth=20 |  0.87452 |  0.994
LinearSVC |  max_iter=20000 |  0.92990 |  0.969
LinearSVC |  max_iter=20000, dual=False, penalty=”l1” |  0.92191 |  0.973
Figure 7: ROC curves of the different hyper parameter configurations listed in
Table III.

Thus, instead of our earlier naive approach to detect rotational symmetries as
described in Section V-A, we can now use our newly trained Random Forest model
on each line pair that is found by the algorithm. Any true case will have the
centre of its symmetry at the intersection of the reflection symmetry line
pair.

Figure 8: Original WaveletSym on the left, proposed rule-based post processing
(Section V-A) in the middle, and the proposed machine learning model approach
on the right. Rotational symmetries are drawn in yellow. The colour of lines
on the right indicates different levels of localised reflection symmetries.
The proposed machine learning model accurately detects each global rotational
symmetry and even some localised symmetries in the 2nd and 5th images. The
rule-based approach found no accurate rotational symmetries. Figure 9:
Original WaveletSym on the left, proposed rule-based post processing (Section
V-A) in the middle, and the proposed machine learning model approach on the
right. Rotational symmetries are drawn in yellow. The colour of lines on the
right indicates different levels of localised reflection symmetries. Example
images where the proposed methods did not perform well.

We found that the proposed machine learning model performs well in identifying
rotational symmetries based on two given reflection symmetry lines. However,
this does mean that our detection method is still completely reliant on the
provided reflection symmetry lines. Due to this, we will now remove fewer
symmetry lines as we did in the previous Section (V-A) since we can use the
model to determine if these lines help detect rotational symmetries or not.
After the predictions are made, we will remove the low scoring symmetries as
usual. This provides us with a better detection rate for rotational symmetries
at the cost of computation time.

Figures 8 and 9 display various images with the original algorithm on the
left, the proposed improvements as described in Section V-A in the middle, and
on the right our improvements with the machine learning model for rotational
symmetries. We can observe that the machine learning model has improved the
detection rate of especially the global rotational symmetries. The results do,
however, show an increased amount of false positives. These false positives
are mostly caused by wrongly detected reflection symmetries. Figure 10 lists
images with their human labeled ground truths (blue lines). When using these
ground truths as input we retrieve the resulting rotational symmetries as
shown as yellow circles in Figure 10, showing that the proposed model works
well when the symmetry lines are correct.

Figure 10: Blue lines denote human labeled reflection symmetries as ground
truths. Yellow circles denote the found rotational symmetries by the proposed
machine learning model based on the ground truths.

#### Comparison with Loy and Eklundh

In Section III Loy and Eklundh [4] is described as a good benchmark for
rotational symmetry detection methods, as it has won two competitions. Figure
11 and 12 show images of our proposed machine learning model against Loy and
Eklundh’s method. Our method shows superior performance when it comes to
detecting rotational symmetries in images that are completely defined as a
rotational symmetry (first, third and fifth image in Figure 11). Rotational
symmetries have, by definition, an infinite amount of smaller rotational
symmetries with the same centre. Our model will accurately draw the largest
encompassing rotational symmetries in images where multiple rotational
symmetries exists inside each other. Loy’s method does not always find the
largest encompassing symmetry or detects a greater symmetry strength in the
wrong centre. Furthermore, in all of our tested images, Loy’s method found
rotational symmetries in each of them. Our method produces less false
positives when there are no rotational symmetries in an image. Examples of
this are shown in the first, second and third images in Figure 12. The last
two images in Figure 12 show better performance for Loy and Eklundh. Both
methods struggle with detecting multiple rotational symmetries. However, Loy’s
method has a slight edge in this category. Most of the secondary rotational
symmetries detected by our method are not accurate, with the current state of
reflection symmetry detection.

Since symmetries should often be detected quickly, or in our case, on a large
number of images, the speed of each algorithm is an important factor. Table IV
lists the average and median time taken by our approach and Loy and Eklundh’s
method to detect rotational and reflection symmetries in a set of 10 images.
As is clear from the low median, most rotational symmetries are detected very
quickly with our method. However, some outliers do significantly raise the
average time taken. It should be noted that our rotational symmetry detection
method requires reflection symmetries as an input. This means the reflection
symmetries need to be computed first. On the other hand, Loy and Eklundh’s
method is able to detect rotational symmetries independently. Furthermore,
during testing, we found that Loy and Eklundh’s algorithm required a lot of
computational power. Running two instances simultaneously caused our computer
to slow down severely during some processing stages. In contrast, our method
was efficiently running on 6 or 7 instances, only being constraint by
RAM444Our computer configuration contained an AMD Ryzen 3600, with 16GB of
RAM.. This means that, depending on the computer configuration, it will be
possible to decrease the average detection time by several factors. This
greatly benefits cases where lots of images need to be processed, such as with
our case for labelling the Ornamika database.

TABLE IV: Average and median computation times in seconds of Loy [4] and our approach in both rotation and reflection symmetry detection on a set of 10 images. Our approach requires reflection symmetries to be detected before being able to detect rotational symmetries. | reflection | rotation
---|---|---
| our work | Loy [4] | our work | Loy [4]
average | 171 | 239 | 256 | 95
median | 109 | 103 | 3 | 38
Figure 11: Proposed method (right) compared against Loy and Eklundh [4]
(middle). The left column contains the ground truths of the images drawn in
red. These images show superior or equal performance of our model over Loy and
Eklundh in detecting rotational symmetries. Figure 12: Proposed method (right)
compared against Loy and Eklundh [4] (middle). The left column contains the
ground truths of the images drawn in red. The first three images contain no
rotational symmetries. Our method is better able to detect this. The bottom
three images show superior performance of Loy and Eklundh over our model.

##  VI Conclusion

In this paper, we evaluated state-of-the-art symmetry detection algorithm
WaveletSym [9] on the images of the Ornamika data set [2] and proposed several
improvements. We made alterations to automate the identification of accurately
detected symmetries and removing others. We proposed a recursive method to
apply the algorithm on smaller parts of the image and, in turn, detect more
localised symmetries. We also used the reflection symmetry algorithm to detect
rotational symmetries based on a set of rules. To improve performance on this
rotational symmetry detection method, we proposed a machine learning model
based on the Random Forest classifier. With our model, rotational symmetries
are detected with an accuracy of 90% on the validation set and 97% on the test
set. We compared our method of detecting rotational symmetries against Loy and
Eklundh’s method and found our work to produce more accurate results when a
single rotational symmetry is considered.

We can conclude that, with our model, it is possible to detect rotational
symmetries with reflection symmetry detection methods. As our model is not
bound to a specific algorithm, an improvement in reflection symmetry detection
methods will yield improvements to rotational symmetry detection with our
model.

##  VII Future Perspectives

We found, through observations, lower-resolution images (or smaller parts of
images) to produce higher symmetry scores in general with waveletSym. This
could imply that a fixed threshold for identifying accurate symmetries is not
the right approach. Instead, a higher threshold on each smaller part of images
could improve the accuracy.

Because our method relies on reflection symmetry detection methods, its
performance should, in theory, increase as reflection symmetry detection
methods improve. Because of this, our model can be re-evaluated on any
improved reflection detection method that produces fixed $x,y$ coordinates
with symmetry scores.

## References

  * [1] M. Elawady, Reflection Symmetry Detection in Images: Application to Photography Analysis.  PhD thesis, Université de Lyon, 2019.
  * [2] “Ornamika Database.” https://ornamika.com/.  Accessed: 2021-06-18.
  * [3] M. J. Atallah, “On Symmetry Detection,” in Department of Computer Science Technical Reports (paper 396), Purdue University, 1984.
  * [4] G. Loy and J.-O. Eklundh, “Detecting Symmetry and Symmetric Constellations of Features,” in European Conference on Computer Vision, pp. 508–521, Springer, 2006.
  * [5] M. Cho and K. M. Lee, “Bilateral Symmetry Detection via Symmetry-Growing,” in Proceedings of the British Machine Vision Conference, pp. 4.1–4.11, BMVA Press, 2009.  doi:10.5244/C.23.4.
  * [6] Y. Ming, H. Li, and X. He, “Symmetry Detection via Contour Grouping,” in 2013 IEEE International Conference on Image Processing, pp. 4259–4263, IEEE, 2013.
  * [7] W. Ke, J. Chen, J. Jiao, G. Zhao, and Q. Ye, “SRN: Side-output Residual Network for Object Symmetry Detection in the Wild,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1068–1076, 2017.
  * [8] C. Funk, S. Lee, M. R. Oswald, S. Tsogkas, W. Shen, A. Cohen, S. Dickinson, and Y. Liu, “2017 iccv challenge: Detecting Symmetry in the Wild,” in 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 1692–1701, 2017.
  * [9] M. Elawady, C. Ducottet, O. Alata, C. Barat, and P. Colantoni, “Wavelet-based Reflection Symmetry Detection via Textural and Color Histograms,” in Proceedings, ICCV Workshop on Detecting Symmetry in the Wild, vol. 3, p. 7, 2017\.
  * [10] I. Rauschert, K. Brocklehurst, S. Kashyap, J. Liu, and Y. Liu, “First Symmetry Detection Competition: Summary and Results,” The Pennsylvania State University, PA, Tech. Rep. CSE11–012, 2011.
  * [11] J. Liu, G. Slota, G. Zheng, Z. Wu, M. Park, S. Lee, I. Rauschert, and Y. Liu, “Symmetry Detection from Realworld Images Competition 2013: Summary and Results,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 200–205, 2013.
  * [12] T. Sie Ho Lee, S. Fidler, and S. Dickinson, “Detecting Curved Symmetric Parts using a Deformable Disc Model,” in Proceedings of the IEEE international conference on computer vision, pp. 1753–1760, 2013.
  * [13] E. Michaelsen and M. Arens, “Hierarchical Grouping using Gestalt Assessments,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1702–1709, 2017.
  * [14] C. Wu, J.-M. Frahm, and M. Pollefeys, “Detecting Large Repetitive Structures with Salient Boundaries,” in European conference on computer vision, pp. 142–155, Springer, 2010.
  * [15] X. Liu, P. Lyu, X. Bai, and M.-M. Cheng, “Fusing Image and Segmentation Cues for Skeleton Extraction in the Wild,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1744–1748, 2017.
  * [16] C. Funk and Y. Liu, “Beyond Planar Symmetry: Modeling Human Perception of Reflection and Rotation Symmetries in the Wild,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 793–803, 2017.
  * [17] M. Cicconet, D. Geiger, K. C. Gunsalus, and M. Werman, “Mirror symmetry histograms for capturing geometric properties in images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2981–2986, 2014.