Shape generation and editing of two subjects randomly generated with LED-VAE,
which is one of the proposed local eigenprojection disentangled models. Left:
effects caused on the generated shapes by traversing two arbitrary latent
variables controlling the eyes and nose of the first random subject. Right:
example of shape editing performed manipulating the latent variables
controlling jaw, nose, and forehead of the second subject. The latent
manipulations are performed with a GUI that allows the manual modification of
the latent variables, but random per-attribute modifications can also be
performed. The edited shapes are always paired with their corresponding
displacement map highlighting the shape differences from the initial model.

# 3D Generative Model Latent Disentanglement
via Local Eigenprojection

Simone Foti10000-0003-3207-1965, Bongjin Koo1,20000-0002-3611-4988, Danail
Stoyanov10000-0002-0980-3227, and Matthew J. Clarkson10000-0002-5565-1252
1University College London, London, UK 2University of California, Santa
Barbara, Santa Barbara, USA

###### Abstract

Designing realistic digital humans is extremely complex. Most data-driven
generative models used to simplify the creation of their underlying geometric
shape do not offer control over the generation of local shape attributes. In
this paper, we overcome this limitation by introducing a novel loss function
grounded in spectral geometry and applicable to different neural-network-based
generative models of 3D head and body meshes. Encouraging the latent variables
of mesh variational autoencoders (VAEs) or generative adversarial networks
(GANs) to follow the local eigenprojections of identity attributes, we improve
latent disentanglement and properly decouple the attribute creation.
Experimental results show that our local eigenprojection disentangled (LED)
models not only offer improved disentanglement with respect to the state-of-
the-art, but also maintain good generation capabilities with training times
comparable to the vanilla implementations of the models. Our code and pre-
trained models are available at
github.com/simofoti/LocalEigenprojDisentangled.

<ccs2012> <concept>
<concept_id>10010147.10010371.10010396.10010398</concept_id>
<concept_desc>Computing methodologies Mesh geometry models</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010371.10010396.10010397</concept_id>
<concept_desc>Computing methodologies Mesh models</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010257.10010258.10010260.10010271</concept_id>
<concept_desc>Computing methodologies Dimensionality reduction and manifold
learning</concept_desc> <concept_significance>500</concept_significance>
</concept> <concept>
<concept_id>10010147.10010257.10010293.10010319</concept_id>
<concept_desc>Computing methodologies Learning latent
representations</concept_desc>
<concept_significance>500</concept_significance> </concept> </ccs2012>

[300]Computing methodologies Dimensionality reduction and manifold learning
[300]Computing methodologies Learning latent representations

###### keywords:

disentanglement, geometric deep learning, variational autoencoder, generative
adversarial networks

##  1 Introduction

In recent years digital humans have become central elements not only in the
movie and video game production, but also in augmented and virtual reality
applications. With a growing interest in the metaverse, simplified creation
processes of diverse digital humans will become increasingly important. These
processes will benefit experienced artists and, more importantly, will
democratise the character generation process by allowing users with no
artistic skills to easily create their unique avatars. Since digitally
sculpting just the geometric shape of the head of a character can easily
require a highly skilled digital artist weeks to months of work [GFZ*20], many
semi-automated avatar design tools have been developed. Albeit simpler and
faster to use, they inherit the intrinsic constraints of their underlying
generative models [FKSC22]. Usually based upon blendshapes [LMR*15, OBB20,
TDM11], principal component analysis (PCA) [BV99, PWP*19, LBB*17], variational
autoencoders (VAEs) [RBSB18, GCBZ19, ATJD19, CNH*20], or generative
adversarial networks (GANs) [CBZ*19, GLP*20, LBZ*20, ABWB19], these models are
either limited in expressivity or they cannot control the creation of local
attributes. Considering that deep-learning-based approaches, such as VAEs and
GANs, offer superior representation capabilities with a reduced number of
parameters and that they can be trained to encourage disentanglement, we focus
our study on these models.

By definition [BCV13, HMP*17, KM18], with a disentangled latent
representation, changes in one latent variable affect only one factor of
variation while being invariant to changes in other factors. This is a
desirable property to offer control over the generation of local shape
attributes. However, latent disentanglement remains an open problem for
generative models of 3D shapes [ATJD19] despite being a widely researched
topic in the deep learning community [HMP*17, KM18, KWKT15, EWJ*19, DXX*20,
WYH*21, RL21]. Most research on latent disentanglement of generative models
for the 3D shape of digital humans addresses the problem of disentangling the
pose and expression of a subject from its identity [ATJD19, ATDJ21, CNH*20,
ABWB19, ZYL*20, LYF*21, ZYHC22], but none of these works is able to provide
disentanglement over the latent variables controlling the local attributes
characterising the identity. Some control over the generation of local
attributes was achieved for generative models of 3D furniture by leveraging
complex architectures with multiple encoders and decoders independently
operating on different furniture parts [NW17, YML*20, RDC*21]. In contrast,
[FKSC22] recently proposed a method to train a single VAE while enforcing
disentanglement among sets of latent variables controlling the identity of a
character. This approach allows their Swap Disentangled VAE (SD-VAE) to learn
a more disentangled, interpretable, and structured latent representation for
3D VAEs of bodies and heads. However, although [FKSC22] disentangles subsets
of latent variables controlling local identity attributes, variables within
each set can be entangled and not orthogonal. In addition, their curated mini-
batching procedure based on attribute swapping is applicable only to
autoencoder-based architectures and it significantly increases the training
duration. In this work, we aim at overcoming these limitations by leveraging
spectral geometry to achieve disentanglement without curating the mini-
batching. In particular, we encourage the latent representation of a mesh to
equal the most significant local eigenprojections of signed distances from the
mean shape of the training data. Since the eigenprojections are computed using
the eigenvectors of combinatorial Laplacian operators, we require meshes to be
in dense point correspondence and to share the same topology. This is a
standard requirement for most of the traditional [BV99, BRZ*16, DPSD20,
GFZ*20, LMR*15, OBB20, PWP*19, PVS*21] and neural-network-based [FKD*20,
FKSC22, GCBZ19, RBSB18, ZWL*20, YLY*20] generative models, which not only
simplifies the shape generation process, but also the definition of other
digital humans’ properties that will be automatically shared by all the
generated meshes (e.g., UV maps, landmarks, and animation rigs).

To summarise, the key contribution of this work is the introduction of a novel
local eigenprojection loss, which is able to improve latent disentanglement
among variables controlling the generation of local shape attributes
contributing to the characterisation of the identity of digital humans. Our
method improves over SD-VAE by enforcing orthogonality between latent
variables and avoiding the curated mini-batching procedure, thus significantly
reducing the training times. In addition, we demonstrate the flexibility and
disentanglement capabilities of our method on both VAEs and GANs.

##  2 Related Work

###  2.1 Generative Models.

Blendshapes are still widely adopted for character animation or as consumer-
level avatar design tools because, by linearly interpolating between a
predefined set of artistically created shapes, the blend-weights can be easily
interpreted [LAR*14]. However, to compensate for the limited flexibility and
diversity of these models, large amounts of shapes are required. This makes
the models very large and only a limited number of shapes can be used in most
practical applications. An alternative approach capable of offering more
flexibility is to build models relying on principal component analysis (PCA)
[BV99, EST*20]. These data-driven models are able to generate shapes as linear
combinations of the training data, but the variables controlling the output
shapes are related to statistical properties of the training data and are
difficult to interpret. In recent years, PCA-based models have been created
from large number of subjects. For example, Lsfm [BRZ*16] and Lyhm [DPSD20]
were built collecting scans from $10,000$ faces and $1,212$ heads
respectively. The two models were later combined in Uhm [PWP*19], which was
subsequently enriched with additional models for ears, eyes, teeth, tongue,
and the inner-mouth [PVS*21]. Also, [GFZ*20] combined multiple PCA models, but
they were controlling different head regions and an anatomically constrained
optimisation was used to combine their outputs and thus create an interactive
head sculpting tool. PCA-based models of the body were also combined with
blendshapes in Smpl [LMR*15] and Star [OBB20], which were trained with $3,800$
and $14,000$ body scans respectively. PCA-based models generally trade the
amount of fine details they can represent with their size. The advent of
geometric deep learning techniques brought a new set of operators making
possible the creation of neural network architectures capable of processing 3D
data such as point-clouds and meshes. [RBSB18] introduced the first VAE for
the generation of head meshes. In its comparison against PCA, the VAE model
used significantly fewer parameters and exhibited superior performances in
generalisation, interpolation, and reconstruction. This pioneering work was
followed by many other autoencoders which differed from one another mostly by
their application domain and the mesh operators used in their architecture
[LBBM18, FKD*20, YFST18, ZWL*20, GCBZ19, DS19, TZY*22, BBP*19]. These mesh
operators were used also for generative models based on GAN architectures
[OBD*21, CBZ*19], but they appear to be less frequent than their VAE
counterparts. Most GAN architectures operate in the image domain by
representing 3D shapes in a UV space [MPN*20, LBZ*20].

###  2.2 Latent Disentanglement.

Most research on latent disentanglement is performed on generative models of
images [KSB18, KM18, KWKT15, EWJ*19, DXX*20, RL21, WYH*21]. The $\beta$-VAE
[HMP*17] is probably the simplest model used to improve disentanglement in a
VAE. Other simple methods that leverage statistical properties and do not
require supervision over the generative factors are for instance the DIP-VAEs
[KSB18] and the FactorVAE [KM18]. All methods above were re-implemented to
operate on meshes by [FKSC22], but they did not report good levels of
disentanglement with respect to the identity attributes. In the 3D realm,
there are currently two prominent streams of research: the one disentangling
the identity from the pose or expression of digital humans [ATJD19, ATDJ21,
CNH*20, ZYL*20, ZBP20, TSL21, JWCZ19, HHS*21, OFD*22], and the stream
attempting to disentangle parts of man-made objects [YML*20, NW17, LLW22,
RDC*21]. In both cases, the proposed solutions require complex architectures.
In addition, in the former category, current state-of-the-art methods do not
attempt to disentangle identity attributes. The latter category appears better
suited for this purpose, but the type of generated shapes is substantially
different because the generation of object parts needs to consider intrinsic
hierarchical relationship, and surface discontinuities are not a problem. More
similar to ours, is the method recently proposed by [FKSC22], where the latent
representation of a mesh convolutional VAE is disentangled by curating the
mini-batching procedure and introducing an additional loss. In particular,
swapping shape attributes between the input meshes of every mini-batch, it is
possible to know which of them share the same attribute and which share all
the others. This knowledge is harnessed by a contrastive-like latent
consistency loss that encourages subsets of latent variables from different
meshes in the mini-batch to assume the same similarities and differences of
the shapes created with the attribute swapping. This disentangles subsets of
latent variables which become responsible for the generation of different body
and head attributes. We adopt the same network architecture, dataset, and
attribute segmentation of SD-VAE. This choice is arbitrary and simplifies
comparisons between the two methods, which differ only in their
disentanglement technique.

Like VAEs, the research on GANs comes mostly from the imaging domain, where
good levels of control over the generation process were recently made
possible. Most of these models leverage segmentation maps [HMWL21, LLWL20,
LKL*21], additional attribute classifiers [HZK*19, SBKM21], text prompts
[RKH*21], or manipulate the latent codes and the parameter space of the pre-
trained model to achieve the desired results [KAL*21, HHLP20, SYTZ22, LKL*21].
We argue that while the first two approaches require more inputs and
supervision than our method, the last two offer less editing flexibility. In
fact, describing the shape of human parts is a difficult task that would
ultimately limit the diversity of the generated shapes, while the post-
training manipulation may limit the exploration of some latent regions. Only a
few methods explicitly seek disentanglement during training [AW20, VB20] like
ours. However, [AW20] is specifically designed for grid-structured data, like
images, and [VB20] still requires a pre-trained GAN and two additional
networks for disentanglement. In the 3D shapes domain, GAN disentanglement is
still researched to control subject poses and expressions [CTS*21, OBD*21] or
object parts [LLHF21]. However, they suffer the same problems described for 3D
VAEs: they have complex architectures and do not have control over the
generation of local identity attributes.

###  2.3 Spectral Geometry.

Spectral mesh processing has played an essential role in shape indexing,
sequencing, segmentation, parametrisation, correspondence, and compression
[ZVD10]. Spectral methods usually leverage the properties of the
eigenstructures of operators such as the mesh Laplacian. Even though there is
no unique definition for this linear operator, it can be classified either as
geometric or combinatorial. Geometric Laplacians are a discretisation of the
continuous Laplace-Beltrami operator [Cha84] and, as their name suggests, they
encode geometric information. Their eigenvalues are robust to changes in mesh
connectivity and are often used as shape descriptors[RWP06, GYP14]. Since they
are isometry-invariant, they are used also in VAEs for identity and pose
disentanglement [ATJD19, ATDJ21]. However, being geometry dependant, the
Laplace-Beltrami operator and its eigendecomposition have to be precomputed
for every mesh in the dataset. On the other hand, combinatorial Laplacians
treat a mesh as a graph and are entirely defined by the mesh topology. For
these operators, the eigenvectors can be considered as Fourier bases and the
eigenprojections are equivalent to a Fourier transformation [SNF*13] whose
result is often used as a shape descriptor. If all shapes in a dataset share
the same topology, the combinatorial Laplacian and its eigendecomposition need
to be computed only once. For this reason, multiple graph and mesh
convolutions [BZSL13, DBV16] as well as some data augmentation technique
[FKD*20] and smoothing losses [FKSC22] are based on combinatorial Laplacian
formulations.

##  3 Method

The proposed method introduces a novel loss to improve latent disentanglement
in generative models of 3D human shapes. After defining the adopted shape
representation, we introduce our local eigenprojection loss, followed by the
two generative models on which it was tested: a VAE and two flavours of GANs.

Figure 1: Schematic representation of the local eigenprojection, the operation
at the core of our local eigenprojection loss. The signed distance between a
given mesh $\mathbf{X}$ and a mean shape template is computed as
$sd(\mathbf{X})$. $sd(\mathbf{X})$ is locally eigenprojected into a vector
$\mathbf{z}^{\star}$ where each subset of variables is a spectral descriptor
of a shape attribute. The projection is performed by matrix-multiplying the
signed distance by $\mathbf{U}^{\star}_{\omega}$, the highest-variance
eigenvectors of each shape attribute $\omega$. The heads in the bottom part of
the figure represent one-dimensional vectors whose values are mapped with
diverging colour maps on the mean shape head. On the heads corresponding to
the columns of $\mathbf{U}^{\star}_{\omega}$, the black seams mark the
different attributes that we seek to control during the generation procedure.

###  3.1 Shape Representation.

We represent 3D shapes as manifold triangle meshes with a fixed topology. By
fixing the topology, all meshes
$\mathcal{M}=\\{\mathbf{X},\bm{\mathcal{E}},\bm{\mathcal{F}}\\}$ share the
same edges $\mathbf{\mathcal{E}}\in\mathbb{N}^{\varepsilon\times 2}$ and faces
$\bm{\mathcal{F}}\in\mathbb{N}^{\Gamma\times 3}$. Therefore, they differ from
one another only for the position of their vertices
$\mathbf{X}\in\mathbb{R}^{N\times 3}$, which are assumed to be consistently
aligned, scaled, and with point-wise correspondences across shapes.

###  3.2 Local Eigenprojection Loss.

We define $F$ arbitrary attributes on a mesh template by manually colouring
anatomical regions on its vertices. Thanks to the assumption of our shape
representation, the segmentation of the template mesh can be consistently
transferred to all the other meshes without manually segmenting them. Mesh
vertices can be then grouped per-attribute such that
$\mathbf{X}=\\{\mathbf{X}_{\omega}\\}_{\omega=1}^{F}$. Seeking to train
generative models capable of controlling the position of vertices
corresponding to each shape attribute $\mathbf{X}_{\omega}$ through a
predefined set of latent variables, we evenly split the latent representation
$\mathbf{z}$ in $F$ subsets of size $\kappa$, such that
$\mathbf{z}=\\{\mathbf{z}_{\omega}\\}_{\omega=1}^{F}$ and each
$\mathbf{z}_{\omega}$ controls its corresponding $\mathbf{X}_{\omega}$. To
establish and enforce a direct relationship between each $\mathbf{X}_{\omega}$
and $\mathbf{z}_{\omega}$ we rely on spectral geometry and compute low-
dimensional local shape descriptors in the spectral domain. We start by
computing the Kirchoff graph Laplacian corresponding to each shape attribute
as: $\mathbf{K}_{\omega}=\mathbf{D}_{\omega}-\mathbf{A}_{\omega}$, where
$\mathbf{A_{\omega}}\in\mathbb{N}^{N_{\omega}\times N_{\omega}}$ is the
adjacency matrix of attribute $\omega$,
$\mathbf{D}_{\omega}\in\mathbb{R}^{N_{\omega}\times N_{\omega}}$ its diagonal
degree matrix, and $N_{\omega}$ the number of its vertices. Values on the
diagonal of $\mathbf{D}_{\omega}$ are computed as
$\text{D}_{aa}=\sum_{b}\text{A}_{ab}$. The Kirchoff Laplacian is a real
symmetric positive semidefinite matrix that can be eigendecomposed as
$\mathbf{K}_{\omega}=\mathbf{U}_{\omega}\bm{\Lambda}_{\omega}\mathbf{U}_{\omega}^{T}$.
The columns of $\mathbf{U}_{\omega}\in\mathbb{R}^{N_{\omega}\times K}$ are a
set of $K$ orthonormal eigenvectors known as the graph Fourier modes and can
be used to transform any discrete function defined on the mesh vertices into
the spectral domain. The signal most commonly transformed is the mesh
geometry, which is the signal specifying the vertex coordinates. However, the
local eigenprojection
$\mathbf{\tilde{X}}_{\omega}=\mathbf{U}_{\omega}^{T}\mathbf{X}_{\omega}$ would
result in a matrix of size $K\times 3$ containing the spectral representations
of the $3$ spatial coordinates. Instead of flattening
$\mathbf{\tilde{X}}_{\omega}$ to make it compatible with the shape of the
latent representation, we define and project a one-dimensional signal: the
signed distance between the vertices of a mesh and the per-vertex mean of the
training set $\mathbf{M}$ (see Fig. 1). We have:

$sd(\mathbf{X})=\bm{\gamma}\|\mathbf{X}-\mathbf{M}\|_{2}\qquad\text{with}\quad\bm{\gamma}=\text{sign}(\big{\langle}\mathbf{X}-\mathbf{M},\mathbf{N}\big{\rangle}),$
(1)

where $\langle\cdot,\cdot\rangle$ is the inner product, and $\mathbf{N}$ are
the vertex normals referred to the mesh template with vertex positions
$\mathbf{M}$. If X was standardised by subtracting $\mathbf{M}$ and dividing
by the per-vertex standard deviation of the training set $\bm{\Sigma}$, being
$\odot$ the Hadamard product, Eq. 1 can be rewritten as:

$sd(\mathbf{X})=\bm{\gamma}\|\mathbf{X}\odot\bm{\Sigma}\|_{2}\qquad\text{with}\quad\bm{\gamma}=\text{sign}(\big{\langle}\mathbf{X}\odot\bm{\Sigma},\mathbf{N}\big{\rangle}).$
(2)

We assume that not all eigenprojections are equally significant when
representing shapes. Therefore, for each attribute $\omega$, we eigenproject
all the local signed distances $sd(\mathbf{X}_{\omega})$ computed over the
training set, and identify the $\kappa$ (with $\kappa\ll K$) spectral
components with the highest variance. While these spectral components are
responsible for most shape variations, the small shape differences represented
by other components can be easily learned by the neural-network-based
generative model. After eigenprojecting the entire training set, we select the
Fourier modes
$\mathbf{U}_{\omega}^{\star}\in\mathbb{R}^{N_{\omega}\times\kappa}$ associated
with the highest variance eigenprojections (Fig. 1) and use them to compute
the eigenprojection loss. During this preprocessing step we also compute the
mean and standard deviation of the highest variance local eigenrpojections,
which we denote by $\mathbf{m}_{\omega}^{\star}$ and
$\mathbf{s}_{\omega}^{\star}$ respectively. We thus define the local
eigenprojection loss as:

$\mathcal{L}_{\textit{LE}}(\mathbf{X},\mathbf{z})=\frac{1}{F\kappa}\sum_{\omega=1}^{F}\Big{\|}\mathbf{z}_{\omega}-\frac{(\mathbf{U}_{\omega}^{\star})^{T}sd(\mathbf{X}_{\omega})-\mathbf{m}_{\omega}^{\star}}{\mathbf{s}_{\omega}^{\star}}\Big{\|}_{1}$
(3)

Note that combinatorial Laplacian operators are determined exclusively by the
mesh topology. Since the topology is fixed across the dataset, the Laplacians
and their eigendecompositions can be computed only once. Therefore, the local
eigenprojection can be quickly determined by matrix-multiplying signed
distances by the precomputed $\mathbf{U}_{\omega}^{\star}$. Also, if the
Laplace-Beltrami operator was used in place of the Kirchoff graph Laplacian,
the eigendecomposition would need to be computed for every mesh. Not only this
would significantly increase the training duration, but backpropagating
through the eigendecomposition would be more complex as this would introduce
numerical instabilities [WDH*19]. Alternatively, an approach similar to
[MRC*21] should be followed.

Figure 2: Local eigenprojection distributions. All training meshes are locally
eigenprojected to observe the distributions of the elements in the resulting
vectors. Distributions are colour-coded according to the shape attribute they
are referred to. The segmentation of the shape attributes displayed next to
the distributions is rendered on the mean shape templates of the corresponding
dataset. The dashed distributions, which are obtained sampling a Gaussian, are
reported for comparison.

###  3.3 Mesh Variational Autoencoder.

Like traditional VAEs [KW13], our 3D-VAE is also built as a probabilistic
encoder-decoder pair parameterised by two separate neural networks. The
probabilistic encoder is defined as a variational distribution
$q(\mathbf{z}|\mathbf{X})$ that approximates the intractable model posterior.
It predicts the mean $\bm{\mu}$ and standard deviation $\bm{\sigma}$ of a
Gaussian distribution over the possible $\mathbf{z}$ values from which
$\mathbf{X}$ could have been generated. The probabilistic decoder
$p(\mathbf{X}|\mathbf{z})$ describes the distribution of the decoded variable
given the encoded one. During the generation process, a latent vector
$\mathbf{z}$ is sampled from a Gaussian prior distribution
$p(\mathbf{z})=\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})$ and an output
shape is generated by the probabilistic decoder. Since the decoder is used as
a generative model, it is also referred to as generator. Following this
convention, we define our architecture as a pair of non-linear functions
$\\{E,G\\}$, where $E:\mathcal{X}\rightarrow\mathcal{Z}$ maps from the vertex
embedding domain $\mathcal{X}$ to the latent distribution domain
$\mathcal{Z}$, and $G:\mathcal{Z}\rightarrow\mathcal{X}$ vice versa. Since
traditional convolutional operators are not compatible with the non-Euclidean
nature of meshes, we build both networks as in [FKSC22], using the simple yet
efficient spiral convolutions [GCBZ19] and sparse matrix multiplications with
transformation matrices obtained with quadric sampling [GCBZ19, RBSB18] (see
Supplementary Materials for more details).

As in [FKSC22], the 3D-VAE is trained minimising
$\mathcal{L}_{\textit{VAE}}=\mathcal{L}_{\textit{R}}(\mathbf{X},\mathbf{X}^{\prime})+\alpha\mathcal{L}_{\textit{L}}(\mathbf{X}^{\prime})+\beta\mathcal{L}_{\textit{KL}}(\bm{\mu},\bm{\sigma})$.
While $\alpha$ and $\beta$ are weighting constants, $\mathcal{L}_{\textit{R}}$
is the reconstruction loss, $\mathcal{L}_{\textit{L}}$ is a Laplacian
regulariser, and $\mathcal{L}_{\textit{KL}}$ is a Kullback–Leibler (KL)
divergence. In auto-encoder parlance, the reconstruction loss
$\mathcal{L}_{\textit{R}}(\mathbf{X},\mathbf{X}^{\prime})=\frac{1}{N}\|\mathbf{X}^{\prime}-\mathbf{X}\|_{\text{F}}^{2}$
encourages the output of the VAE to be as close as possible to its input by
computing the squared Frobenius norm between
$\mathbf{X}^{\prime}=G(E(\mathbf{X}))$ and $\mathbf{X}$. The KL divergence can
be considered as a regularisation term that pushes the variational
distribution $q(\mathbf{z}|\mathbf{X})$ towards the prior distribution
$p(\mathbf{z})$. Since both prior and posterior are assumed to be Gaussian
$\mathcal{L}_{\textit{KL}}(\bm{\mu},\bm{\sigma})=\bm{\sigma}^{2}+\bm{\mu}^{2}-\text{log}(\bm{\sigma}^{2})-1$.
The Laplacian loss
$\mathcal{L}_{\textit{L}}(\mathbf{X}^{\prime})=\frac{1}{N}\|\mathbf{TX}^{\prime}\|_{\text{F}}^{2}$
is a smoothing term computed on the output vertices $\mathbf{X}^{\prime}$ and
based on the Tutte Laplacian
$\mathbf{T}=\mathbf{D}^{-1}\mathbf{K}=\mathbf{I}-\mathbf{D}^{-1}\mathbf{A}$,
where $\mathbf{A}$, $\mathbf{D}$, and $\mathbf{K}$ are the adjacency, diagonal
degree, and Kirchoff Laplacian introduced in the previous paragraph and
computed on the entire mesh rather than on shape attributes.

Latent disentanglement is enforced by separately applying the local
eigenprojection loss to the encoder and generator. We thus define the total
loss as:

$\begin{split}\mathcal{L}=\mathcal{L}_{\textit{R}}(\mathbf{X},\mathbf{X}^{\prime})+\alpha\mathcal{L}_{\textit{L}}(\mathbf{X}^{\prime})+\beta\mathcal{L}_{\textit{KL}}(\bm{\mu},\bm{\sigma})+\\\
+\eta_{1}\mathcal{L}_{\textit{LE}}(\mathbf{X},\bm{\mu})+\eta_{2}\mathcal{L}_{\textit{LE}}(\mathbf{X^{\prime}},\bm{\mu}),\end{split}$
(4)

where $\eta_{1}$ and $\eta_{2}$ are two scalar weights balancing the
contributions of the two local eigenprojection losses. Note that
$\mathcal{L}_{\textit{LE}}(\mathbf{X},\bm{\mu})$ is backpropagated only
through $E$. This term pushes the predicted $\bm{\mu}$ towards the
standardised local eigenprojections of the input, while the KL divergence
attempts to evenly distribute the encodings around the centre of the latent
space. Similarly, $\mathcal{L}_{\textit{LE}}(\mathbf{X^{\prime}},\bm{\mu})$ is
backpropagated only through $G$ and it enforces the output attributes to have
an eigenprojection compatible with the predicted mean.

###  3.4 Mesh Generative Adversarial Networks.

We propose two flavours of 3D Generative Adversarial Networks: one based on
Least Squares GAN (LSGAN) [MLX*17] and one on Wasserstein GAN (WGAN) [ACB17].
Like VAEs, GANs also rely on a pair of neural networks: a generator-
discriminator pair $\\{G,D\\}$ in LSGAN and a generator-critic $\\{G,C\\}$
pair in WGAN. The architecture of the generators is the same as the one
adopted in the generator of the 3D-VAE. The architectures of $D$ and $C$ are
similar to $E$, but with minor differences in the last layers (see
Supplementary Materials). Nevertheless, all networks are built with the same
mesh operators of our 3D-VAE and [FKSC22, GCBZ19].

In the LSGAN implementation, $G$ samples an input latent representation from a
Gaussian distribution
$p(\mathbf{z})=\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})$ and maps it to
the shape space as $G(\mathbf{z})=\mathbf{X}^{\prime}$. While it tries to
learn a distribution over generated shapes, the discriminator operates as a
classifier trying to distinguish generated shapes $\mathbf{X}^{\prime}$ from
real shapes $\mathbf{X}$. Using a binary coding scheme for the labels of real
and generated samples, we can write the losses of $G$ and $D$ respectively as
$\mathcal{L}^{\textit{G}}_{\textit{LSGAN}}=\frac{1}{2}\mathbb{E}_{\mathbf{z}\sim
p(\mathbf{z})}[(D(G(\mathbf{z}))-1)^{2}]$ and
$\mathcal{L}^{\textit{D}}_{\textit{LSGAN}}=\frac{1}{2}\mathbb{E}_{\mathbf{X}\sim
p(\mathbf{X})}[(D(\mathbf{X})-1)^{2}]+\frac{1}{2}\mathbb{E}_{\mathbf{z}\sim
p(\mathbf{z})}[D(G(\mathbf{z}))^{2}]$. We also add the Laplacian
regularisation term $\mathcal{L}_{\textit{L}}(\mathbf{X}^{\prime})$ to smooth
the generated outputs. When seeking disentanglement, we train the
discriminator by minimising $\mathcal{L}^{\textit{D}}_{\textit{LSGAN}}$ and
the generator by minimising the following:

$\mathcal{L}^{G}_{\textit{LS}}=\mathcal{L}^{\textit{G}}_{\textit{LSGAN}}+\alpha\mathcal{L}_{\textit{L}}(\mathbf{X}^{\prime})+\eta\mathcal{L}_{\textit{LE}}(\mathbf{X^{\prime}},\mathbf{z}).$
(5)

In WGAN, $G$ still tries to learn a distribution over generated shapes, but
its critic network $C$, instead of classifying real and generated shapes,
learns a Wasserstein distance and outputs scalar scores that can be
interpreted as measures of realism for the shapes it processes. The WGAN
losses for $G$ and $C$ are
$\mathcal{L}^{\textit{G}}_{\textit{WGAN}}=-\mathbb{E}_{\mathbf{z}\sim
p(\mathbf{z})}[D(G(\mathbf{z})]$ and
$\mathcal{L}^{\textit{C}}_{\textit{WGAN}}=\mathbb{E}_{\mathbf{z}\sim
p(\mathbf{z})}[D(G(\mathbf{z}))]-\mathbb{E}_{\mathbf{X}\sim
p(\mathbf{X})}[D(\mathbf{X})]$ respectively. Similarly to the LSGAN
implementation, when enforcing disentanglement, the critic is trained
minimising $\mathcal{L}^{\textit{C}}_{\textit{WGAN}}$, while the generator
minimising:

$\mathcal{L}^{G}_{\textit{W}}=\mathcal{L}^{\textit{G}}_{\textit{WGAN}}+\alpha\mathcal{L}_{\textit{L}}(\mathbf{X}^{\prime})+\eta\mathcal{L}_{\textit{LE}}(\mathbf{X^{\prime}},\mathbf{z}).$
(6)

Note that to make $C$ a 1-Lipschitz function, and thus satisfies the
Wasserstein distance computation requirements, $C$ weights are clipped to the
range $[-c,c]$.

##  4 Experiments

Table 1: Quantitative comparison between our model and other state-of-the-art methods. All methods were trained on Uhm [PWP*19]. Diversity, JSD, MMD, COV, and 1-NNA evaluate the generation capabilities of the models, while VP evaluates latent disentanglement. The different metrics are computed as detailed in Sec. 4.3. Note that the training time does not consider the initialisation time.  Method |  Mean Rec. ($\downarrow$) |  Diversity ($\uparrow$) |  JSD ($\downarrow$) |  MMD ($\downarrow$) |  COV (%, $\uparrow$) |  1-NNA ($\Delta$%, $\downarrow$) |  VP (%, $\uparrow$) |  Training Time ($\downarrow$)
---|---|---|---|---|---|---|---|---
VAE | $\mathbf{0.61}$ | $4.23$ | $4.89$ | $1.53$ | $65.49$ | $1.17$ | $63.73$ | $\mathbf{1\textbf{h:}46\textbf{m}}$
LSGAN | — | $6.12$ | $\mathbf{1.14}$ | $1.65$ | $43.41$ | $22.04$ | $46.83$ |  $2$h:$23$m
WGAN | — | $4.04$ | $22.75$ | $1.36$ | $57.94$ | $23.98$ | $71.07$ |  $2$h:$22$m
DIP-VAE-I | $4.65$ | $4.74$ | $5.32$ | $\mathbf{1.24}$ | $55.57$ | $4.31$ | $35.60$ |  $1$h:$48$m
SD-VAE | $0.73$ | $4.23$ | $4.30$ | $1.56$ | $\mathbf{65.67}$ | $\mathbf{0.50}$ | $79.75$ |  $7$h:$21$m
LED-VAE | $1.46$ | $5.30$ | $2.27$ | $1.73$ | $49.83$ | $15.80$ | $\mathbf{80.75}$ |  $2$h:$53$m
LED-LSGAN | — | $\mathbf{6.38}$ | $2.09$ | $2.03$ | $43.41$ | $17.23$ | $79.75$ |  $2$h:$28$m
LED-WGAN | — | $5.77$ | $2.55$ | $1.81$ | $47.47$ | $14.95$ | $74.11$ |  $2$h:$28$m

###  4.1 Datasets.

Since our main objective is to train a generative model capable of generating
different identities, we require datasets containing a sufficient number of
subjects in a neutral expression (pose). Most open source datasets for 3D
shapes of faces, heads, bodies, or animals (e.g. Mpi-Dyna [PRMB15], Smpl
[LMR*15], Surreal [VRM*17], Coma [RBSB18], Smal [ZKJB17], etc.) focus on
capturing different expressions or poses and are not suitable for identity
disentanglement. For comparison, we rely on the $10,000$ meshes –with neutral
expression and pose– generated in [FKSC22] using two linear models that were
built using a large number of subjects: Uhm [PWP*19] and Star [OBB20] (Sec.
2.1). We also use the same data split with $90\%$ of the data for training,
$5\%$ for validation, and $5\%$ for testing. Since these data are generated
from PCA-based models, we also train our models on real data from the Lyhm
dataset [DPSD20] registered on the Flame [LBB*17] template. In addition, even
though it is beyond the scope of this work, we attempt to achieve
disentanglement through local eigenprojection also on Coma [RBSB18], a dataset
mostly known for its wide variety of expressions. All models and datasets are
released for non-commercial scientific research purposes.

###  4.2 Local Eigenprojection Distributions.

We observe that the eigenprojections are normally distributed for datasets
with neutral poses or expressions (Fig. 2). By standardising the
eigenprojections in Eq. 3 we ensure their mean and standard deviation to be
$0$ and $1$ respectively. Since we enforce a direct relation between the local
eigenprojections and the latent representations, this is a desirable property
that allows us to generate meaningful shapes by sampling latent vector from a
normal distribution. In order to explain why this property holds for datasets
with neutral poses and expressions, we need to hypothesise that shapes follow
a Gaussian distribution. This is a reasonable hypothesis for datasets
generated from PCA-based models, such as those obtained from Uhm and Star,
because vertex positions are computed as linear combinations of generative
coefficients sampled from a Gaussian. However, following the maximum entropy
explanation [Lyo14], it is also reasonable to assume that shapes in dataset
obtained capturing real people (like Lyhm), are normally distributed. [Lyo14]
argues that although the Central Limit Theorem is the standard explanation of
why many things are normally distributed, the conditions to apply the theorem
are usually not met or they cannot be verified. We assume that, like people’s
height, also body and head shapes are largely determined by genetics and
partially by environment and epigenetic effects. The selection pressure
determines an ideal shape with some variability to hedge against fluctuating
circumstances in the environment. This amounts to fixing the mean, and an
upper bound on the variance. Apart from that, the population will naturally
tend to a state of maximal disorder (i.e., maximum entropy). Therefore,
according to the maximum entropy explanation, human shapes are normally
distributed because the distribution maximising entropy subject to those
constraints is a normal distribution. If the shapes are normally distributed,
we can consider also vertex positions consistently sampled on the shape
surfaces to follow each a different Gaussian distribution centred at the
corresponding vertex coordinates on the mean shape. Considering that the
signed distance and the local eigenprojection are both linear operations, they
preserve normality, and for this reason also the local eigenprojections are
normal. Note that expressions are subject-specific deformations with a highly
non-linear behaviour [CBGB20]. There is no guarantee that these
transformations preserve the normality of the shape distribution. Therefore,
datasets containing expressions, such as Coma, may not satisfy the normality
assumption. In fact, we observe that the standardised eigenprojections have
more complex distributions which appear to be mixture of Gaussians (see Fig.
2). Intuitively, each Gaussian in the mixtures could be related to a different
subset of expressions.

Figure 3: Random samples and vertex-wise distances showing the effects of
traversing three randomly selected latent variables (see Supplementary
Material to observe the effects for all the latent variables).

###  4.3 Comparison with Other Methods.

We compare our local eigenprojection disentangled (LED) methods against their
vanilla implementations and against the only state-of-the-art method providing
control over the generation of local shape attributes: the swap disentangled
VAE (SD-VAE) proposed in [FKSC22]. The authors compared their SD-VAE with
other VAEs for latent disentanglement. Among their implementation of DIP-
VAE-I, DIP-VAE-II, and FactorVAE, the first one appeared to be the best
performing. Therefore, we report results for DIP-VAE-I. For a fair comparison,
all methods were trained on the same dataset (Uhm) using the same batch size
and the same number of epochs. In addition, they share the same architecture
with minor modifications for the GAN implementations (see Supplementary
Materials). The SD-VAE implementation, as well as the evaluation code and the
benchmark methods, are made publicly available at github.com/simofoti/3DVAE-
SwapDisentangled. All models were trained on a single Nvidia Quadro P5000,
which was used for approximately 18 GPU days in order to complete all the
experiments.

The reconstruction errors reported in Tab. 1 are computed as the mean per-
vertex L2 distance between input and output vertex positions. This metric is
computed on the test set and applies only to VAEs. We report the generation
capabilities of all models in terms of diversity, JSD, MMD, COV, and 1-NNA.
The diversity is computed as the average of the mean per-vertex distances
among pairs of randomly generated meshes. The Jensen-Shannon Divergence (JSD)
[ADMG18] evaluates the KL distances between the marginal point distributions
of real and generated shapes. The coverage (COV) [ADMG18] measures the
fraction of meshes matched to at least one mesh from the reference set. The
minimum matching distance (MMD) [ADMG18] complements the coverage by averaging
the distance between each mesh in the test set and its nearest neighbour among
the generated ones. The 1-nearest neighbour accuracy (1-NNA) is a measure of
similarity between shape distributions that evaluates the leave-one-out
accuracy of a 1-NN classifier. In its original formulation [YHH*19], it
expects values converging to $50\%$. However, following [FKSC22], in Tab. 1 we
report absolute differences between the original score and the $50\%$ target
value. All the generation capability metrics can be computed either with the
Chamfer or the Earth Mover distance. Since we did not observe significant
discrepancies between the metrics computed with these two distances, we
arbitrarily report results obtained with the Chamfer distance.

Figure 4: Effects of traversing each latent variable across different mesh
attributes. For each latent variable (abscissas) we represent the per-
attribute mean distances computed after traversing the latent variable from
its minimum to its maximum value. For each latent variable, we expect a high
mean distance in one single attribute and low values for all the others.

Observing Tab. 1 we notice that none of the models is consistently
outperforming the others. GANs generally report better diversity scores than
VAEs, but they are worse in terms of coverage and 1-NNA. GANs were also more
difficult to train and were prone to mode collapse. On the other hand, VAEs
appeared stable and required significantly less hyperparameter tuning. The
scores of our LED models were comparable with other methods, thus showing that
our loss does not negatively affect the generation capabilities. However, LED
models are consistently outperformed in terms of MMD, COV, and 1-NNA. These
metrics evaluate the quality of generated samples by comparing them to a
reference set. Since comparisons are performed on the entire output shapes, we
hypothesise that a shape with local identity attributes resembling each a
different subject from the test set is more penalised than a shape whose
attributes are plausibly obtained from a single subject. Note also that MMD,
COV, and 1-NNA appear to be inversely proportional to the diversity,
suggesting that more diverse generated shapes are also less similar to shapes
in the test set. LED-models report higher diversity because attributes can be
independently generated. This negatively affects MMD, COV, and 1-NNA, but the
randomly generated shapes are still plausible subjects (see Fig. 3 and
Supplementary Materials). Interestingly, SD-VAE appears to be still capable of
generating shapes with attributes resembling the same subject from the test
set, but at the expense of diversity and latent disengagement (see Sec. 4.4).

LED-LSGAN and LED-WGAN train almost as quickly as the vanilla LSGAN and WGAN.
Training LED-VAE takes approximately one hour more than its vanilla
counterpart because the local eigenprojection loss is separately
backpropagated through the encoder. However, since latent disentanglement is
achieved without swapping shape attributes during mini-batching, the training
time of LED-VAE is reduced by $61\%$ with respect to SD-VAE. Note that the
additional initialisation overhead of LED models ($3.72$ minutes) is
negligible when compared to the significant training time reduction over SD-
VAE, which is the only model capable of achieving a satisfactory amount of
latent disentanglement.

If we then qualitatively evaluate the random samples in Fig. 3, we see that
the quality of the meshes generated by LED-LSGAN and LED-WGAN is slightly
worse than those from LED-VAE. We attribute this behaviour to the –usually
undesired– smoothness typically introduced by 3D VAE models. In this case, the
VAE model itself acts as a regulariser that prevents the shape artefacts
introduced by the local eigenprojection disentanglement. In addition,
traversing the latent variables, we find that mesh defects tend to appear when
latent variables approach values near $\pm 3$ (see supplementary material
video). This might be a consequence of the reduced number of training data
with local eigenprojections with these values (see Fig. 2). Nonetheless, the
problem can be easily mitigated with the truncation trick, thus sampling
latent vectors from a Gaussian with standard deviation slightly smaller than
one.

Figure 5: Results of LED-VAE on other datasets. For each dataset are displayed
the effects of traversing latent variables (Uhm is reported in Fig. 4), three
random samples and three vertex-wise distances highlighting the effects of
traversing three latent variables (Uhm is reported Fig. 3). Mean distances are
plotted following the colour coding depicted in Fig. 2.

###  4.4 Evaluation of Latent Disentanglement.

Latent disentanglement can be quantitatively evaluated on datasets with
labelled data. However, such labels are not available for the disentanglement
of shape attributes and traditional metrics such as Z-Diff [HMP*17], SAP
[KSB18], and Factor [KM18] scores cannot be used. Since the Variation
Predictability (VP) disentanglement metric does not require labelled data and
it has shown good correlation with the Factor score [ZXT20], we rely on this
metric to quantify disentanglement across different models (see Tab. 1). The
VP metric averages the test accuracies across multiple few-shot trainings of a
classifier. The classifier takes as input the difference between two shapes
generated from two latent vectors differing in only one dimension and predicts
the varied latent dimension. We implement the classifier network with the same
architecture of our encoders, discriminators, and critiques. The network was
trained for $5$ epochs with a learning rate of $1e^{-4}$. As in [ZXT20], we
set $\eta_{VP}=0.1$, $N_{VP}=10,000$ and $S_{VP}=3$.

In addition, we qualitatively evaluate disentanglement as in [FKSC22] by
observing the effects of traversing latent variables (Fig. 3D Generative Model
Latent Disentanglement  via Local Eigenprojection, left). For each latent
variable, we compute the per-vertex Euclidean distances between two meshes.
After setting all latent variables to their mean value ($0$), the first mesh
is generated setting a single latent to its minimum ($-3$) and the second mesh
setting the same variable to its maximum ($+3$). The Euclidean distances can
be either rendered on the mesh surface using colours proportional to the
distances (Latent Traversals in Fig. 3 and Fig. 5), or plotted as their per-
attribute average distance (Fig. 4 and Fig. 5). When plotted, the average
distances isolated to each attribute provide an intuitive way to assess
disentanglement: good disentanglement is achieved when the traversal of a
single variable determines high mean distances for one attribute and low mean
distances for all the others. Observing Fig. 3 and Fig. 4, it is clear that
the only state-of-the-art method providing control over local shape attributes
is SD-VAE. Since the eigenvectors used in the local eigenprojection loss are
orthogonal, we improve disentanglement over SD-VAE. In fact, traversing latent
variables of LED models determines finer changes within each attribute in the
generated shapes. For instance, this can be appreciated by observing the eyes
of the latent traversals in Fig. 3, where left and right eyes are controlled
by different variables in LED-VAE, while by the same one in SD-VAE (more
examples are depicted in the Supplementary Materials). We also notice that the
magnitude of the mean distances reported in Fig. 4 for our LED models is
bigger than SD-VAE within attributes and comparable outside. This shows
superior disentanglement and allows our models to generate shapes with more
diverse attributes than SD-VAE. Our model exhibits good disentanglement
performances also on other datasets (Fig. 5).

###  4.5 Direct Manipulation

Like SD-VAE, also LED-VAE can be used for the direct manipulation of the
generated shapes. As in [FKSC22], the direct manipulation is performed by
manually selecting $\Upsilon$ vertices on the mesh surface
($S\circ\mathbf{X}^{\prime}=S\circ G(\mathbf{z})\in\mathbb{R}^{\Upsilon\times
3}$) and by providing their desired location
($\mathbf{Y}\in\mathbb{R}^{\Upsilon\times 3}$). Then,
$\min_{\mathbf{z}_{\omega}}\|S\circ G(\mathbf{z})-\mathbf{Y}\|_{2}^{2}$ is
optimised with the Adam optimiser for $50$ iterations while maintaining a
fixed learning rate of $lr=0.1$. Note that the optimisation is performed only
on the subset of latent variables $\mathbf{z}_{\omega}$ controlling the local
attribute corresponding to the selected vertices. If vertices from different
attributes are selected, multiple optimisations are performed. As it can be
observed in Fig. 6, LED-VAE is able to perform the direct manipulations
causing less shape changes than SD-VAE in areas that should remain unchanged.

Figure 6: Direct manipulation. Left: the user manually selects an arbitrary
number of vertices (blue) and specifies their desired position (red). Right:
results of the direct manipulation optimisation for LED-VAE and SD-VAE. For
each method, the output shape, a close-up of the manipulated attribute, and
the rendering of the per-vertex distances between the initial and manipulated
shapes are reported. The colour-map used to represent vertex distances is blue
where distances are zero and red where they reach their maximum value.

##  5 Conclusion

We introduced a new approach to train generative models with a more
disentangled, interpretable and structured latent representation that
significantly reduces the computational burden required by SD-VAE. By
establishing a correspondence between local eigenprojections and latent
variables, generative models can better control the creation and modification
of local identity attributes of human shapes (See Fig. 3D Generative Model
Latent Disentanglement  via Local Eigenprojection). Like the majority of
state-of-the-art methods, the main limitation of our model is the assumption
on the training data, which need to be consistently aligned, in dense point
correspondence, and with a fixed topology. Even though this is surely a
limitation, as we mentioned in Sec. 1, this assumption can simplify the
generation of other digital human’s properties. Among the different LED models
we proposed, we consider LED-VAE to be the most promising. This model is
simpler to train, requires less hyperparameter tuning, and generates higher-
quality meshes. We trained and tested this model also on other datasets, where
it showed equivalent performances. Datasets with expressions have complex
local eigenprojection distributions (Fig. 2) which are more difficult to
learn. In fact, random samples generated by LED-VAE trained on Coma present
mesh defects localised especially in areas where changes in expression
introduce significant shape differences characterised by a highly non-linear
behaviour (e.g. the mouth region). Controlling the generation of different
expressions was beyond the scope of this work and we aim at addressing the
issue as future work. We proved that our loss can be easily used with both
GANs and VAEs. Being efficient to compute and not requiring modifications to
the mini-batching procedure (like SD-VAE), it could be leveraged also in more
complex architectures for 3D reconstruction or pose and expression
disentanglement. In the LED-VAE the local eigenprojection loss is computed
also on the encoder (see how this improves disentanglement in the ablation
study provided with the supplementary materials). Having an encoder capable of
providing a disentangled representation for different attributes could greatly
benefit shape-analysis research in plastic surgery [OvdLP*22] and in genetic
applications [CRW*18]. Therefore, we believe that our method has the potential
to benefit not only experienced digital artists but also democratise the
creation of realistic avatars for the metaverse and find new applications in
shape analysis. Since the generation of geometric shapes is only the first
step towards the data-driven generation of realistic digital humans, as future
work, we will research more interpretable generative processes for
expressions, poses, textures, materials, high-frequency details, and hair.

## Acknowledgement

This research was funded in whole, or in part, by the Wellcome Trust
[203145Z/16/Z]. For the purpose of Open Access, the author has applied a CC BY
public copyright licence to any Author Accepted Manuscript version arising
from this submission.

## References

  * [ABWB19] Victoria Fernandez Abrevaya, Adnane Boukhayma, Stefanie Wuhrer and Edmond Boyer  “A Decoupled 3D Facial Shape Model by Adversarial Training”  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ Seoul, Korea (South): IEEE, 2019, pp. 9418–9427
  * [ACB17] Martin Arjovsky, Soumith Chintala and Léon Bottou  “Wasserstein generative adversarial networks”  In _Proceedings of the 34th International Conference on Machine Learning_ 70, Proceedings of Machine Learning Research  Sydney, Australia: PMLR, 2017, pp. 214–223
  * [ADMG18] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas and Leonidas Guibas  “Learning representations and generative models for 3d point clouds”  In _Proceedings of the 35th International Conference on Machine Learning_ 80, Proceedings of Machine Learning Research  Stockholm, Sweden: PMLR, 2018, pp. 40–49
  * [ATDJ21] Tristan Aumentado-Armstrong, Stavros Tsogkas, Sven Dickinson and Allan Jepson  “Disentangling Geometric Deformation Spaces in Generative Latent Shape Models”  arXiv, 2021
  * [ATJD19] Tristan Aumentado-Armstrong, Stavros Tsogkas, Allan Jepson and Sven Dickinson  “Geometric disentanglement for generative latent shape models”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ Seoul, Korea (South): IEEE, 2019, pp. 8181–8190
  * [AW20] Yazeed Alharbi and Peter Wonka  “Disentangled image generation through structured noise injection”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ Virtual: IEEE, 2020, pp. 5134–5142
  * [BBP*19] Giorgos Bouritsas et al.  “Neural 3d morphable models: Spiral convolutional networks for 3d shape representation learning and generation”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ Seoul, Korea (South): IEEE, 2019, pp. 7213–7222
  * [BCV13] Yoshua Bengio, Aaron Courville and Pascal Vincent  “Representation learning: A review and new perspectives”  In _IEEE transactions on pattern analysis and machine intelligence_ 35.8 IEEE, 2013, pp. 1798–1828
  * [BRZ*16] James Booth et al.  “A 3d morphable model learnt from 10,000 faces”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ Las Vegas, Nevada: IEEE, 2016, pp. 5543–5552
  * [BV99] Volker Blanz and Thomas Vetter  “A Morphable Model for the Synthesis of 3D Faces”  In _Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques_ , SIGGRAPH ’99  Los Angeles, California, USA: ACM Press/Addison-Wesley Publishing Co., 1999, pp. 187–194
  * [BZSL13] Joan Bruna, Wojciech Zaremba, Arthur Szlam and Yann LeCun  “Spectral Networks and Locally Connected Networks on Graphs”  arXiv, 2013
  * [CBGB20] Prashanth Chandran, Derek Bradley, Markus Gross and Thabo Beeler  “Semantic deep face models”  In _2020 International Conference on 3D Vision (3DV)_ Fukuoka, Japan: IEEE, 2020, pp. 345–354
  * [CBZ*19] Shiyang Cheng et al.  “MeshGAN: Non-linear 3D Morphable Models of Faces”  arXiv, 2019
  * [Cha84] Isaac Chavel  “Eigenvalues in Riemannian geometry”  Orlando, Florida: Academic press, 1984
  * [CNH*20] Luca Cosmo et al.  “LIMP: Learning Latent Shape Representations with Metric Preservation Priors”  In _European Conference on Computer Vision – ECCV 2020_ Online: Springer International Publishing, 2020, pp. 19–35  Springer
  * [CRW*18] Peter Claes et al.  “Genome-wide mapping of global-to-local genetic effects on human facial shape”  In _Nature genetics_ 50.3 Nature Publishing Group, 2018, pp. 414–423
  * [CTS*21] Haoyu Chen et al.  “Intrinsic-extrinsic preserved gans for unsupervised 3D pose transfer”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ Virtual: IEEE, 2021, pp. 8630–8639
  * [DBV16] Michaël Defferrard, Xavier Bresson and Pierre Vandergheynst  “Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering”  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , NIPS’16  Barcelona, Spain: Curran Associates Inc., 2016, pp. 3844–3852
  * [DPSD20] Hang Dai, Nick Pears, William Smith and Christian Duncan  “Statistical modeling of craniofacial shape and texture”  In _International Journal of Computer Vision_ 128.2 Springer, 2020, pp. 547–571
  * [DS19] Hang Dai and Ling Shao  “Pointae: Point auto-encoder for 3d statistical shape and texture modelling”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ Seoul, Korea (South): IEEE, 2019, pp. 5410–5419
  * [DXX*20] Zheng Ding et al.  “Guided variational autoencoder for disentanglement learning”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ Virtual: IEEE, 2020, pp. 7920–7929
  * [EST*20] Bernhard Egger et al.  “3d morphable face models—past, present, and future”  In _ACM Transactions on Graphics (TOG)_ 39.5 ACM New York, NY, USA, 2020, pp. 1–38
  * [EWJ*19] Babak Esmaeili et al.  “Structured disentangled representations”  In _The 22nd International Conference on Artificial Intelligence and Statistics_ Naha, Okinawa, Japan: PMLR, 2019, pp. 2525–2534  PMLR
  * [FKD*20] Simone Foti et al.  “Intraoperative Liver Surface Completion with Graph Convolutional VAE”  In _Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis_ Lima, Peru: Springer-Verlag, 2020, pp. 198–207
  * [FKSC22] Simone Foti, Bongjin Koo, Danail Stoyanov and Matthew J Clarkson  “3D Shape Variational Autoencoder Latent Disentanglement via Mini-Batch Feature Swapping for Bodies and Faces”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ New Orleans, Louisiana, USA: IEEE, 2022, pp. 18730–18739
  * [GCBZ19] Shunwang Gong, Lei Chen, Michael Bronstein and Stefanos Zafeiriou  “Spiralnet++: A fast and highly efficient mesh convolution operator”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ Seoul, Korea (South): IEEE, 2019
  * [GFZ*20] A. Gruber et al.  “Interactive Sculpting of Digital Faces Using an Anatomical Modeling Paradigm”  In _Computer Graphics Forum_ 39.5, 2020, pp. 93–102  Wiley Online Library
  * [GLP*20] Baris Gecer et al.  “Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks”  In _European Conference on Computer Vision_ Virtual: IEEE, 2020, pp. 415–433  Springer
  * [GYP14] Zhanheng Gao, Zeyun Yu and Xiaoli Pang  “A compact shape descriptor for triangular surface meshes”  In _Computer-Aided Design_ 53 Elsevier, 2014, pp. 62–69
  * [HHLP20] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen and Sylvain Paris  “Ganspace: Discovering interpretable gan controls”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 9841–9850
  * [HHS*21] Qixing Huang et al.  “ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ Virtual: IEEE, 2021, pp. 5815–5825
  * [HMP*17] Irina Higgins et al.  “beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework”  In _5th International Conference on Learning Representations_ Toulon, France: OpenReview.net, 2017
  * [HMWL21] Xun Huang, Arun Mallya, Ting-Chun Wang and Ming-Yu Liu  “Multimodal Conditional Image Synthesis with Product-of-Experts GANs”  In _arXiv preprint arXiv:2112.05130_ arXiv, 2021
  * [HZK*19] Zhenliang He et al.  “Attgan: Facial attribute editing by only changing what you want”  In _IEEE transactions on image processing_ 28.11 IEEE, 2019, pp. 5464–5478
  * [JWCZ19] Zi-Hang Jiang, Qianyi Wu, Keyu Chen and Juyong Zhang  “Disentangled representation learning for 3d face shape”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ Long Beach, California, USA: IEEE, 2019, pp. 11957–11966
  * [KAL*21] Tero Karras et al.  “Alias-free generative adversarial networks”  In _Advances in Neural Information Processing Systems_ 34 Curran Associates, Inc., 2021, pp. 852–863
  * [KM18] Hyunjik Kim and Andriy Mnih  “Disentangling by factorising”  In _International Conference on Machine Learning_ Stockhol, Sweden: PMLR, 2018, pp. 2649–2658  PMLR
  * [KSB18] Abhishek Kumar, Prasanna Sattigeri and Avinash Balakrishnan  “Variational Inference of Disentangled Latent Concepts from Unlabeled Observations”  In _International Conference on Learning Representations_ Vancouver, Canada: OpenReview.net, 2018
  * [KW13] Diederik P Kingma and Max Welling  “Auto-Encoding Variational Bayes”  arXiv, 2013
  * [KWKT15] Tejas D Kulkarni, William F. Whitney, Pushmeet Kohli and Josh Tenenbaum  “Deep Convolutional Inverse Graphics Network”  In _Advances in Neural Information Processing Systems_ 28 Montreal, Canada: Curran Associates, Inc., 2015
  * [LAR*14] John P Lewis et al.  “Practice and theory of blendshape facial models.”  In _Eurographics (State of the Art Reports)_ 1.8, 2014, pp. 2
  * [LBB*17] Tianye Li et al.  “Learning a model of facial shape and expression from 4D scans.”  In _ACM Trans. Graph._ 36.6, 2017, pp. 194–1
  * [LBBM18] Or Litany, Alex Bronstein, Michael Bronstein and Ameesh Makadia  “Deformable shape completion with graph convolutional autoencoders”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ Salt Lake City, Utah, USA: IEEE, 2018, pp. 1886–1895
  * [LBZ*20] Ruilong Li et al.  “Learning Formation of Physically-Based Face Attributes”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ Virtual: CVPR, 2020, pp. 3410–3419
  * [LKL*21] Huan Ling et al.  “EditGAN: High-Precision Semantic Image Editing”  In _Advances in Neural Information Processing Systems_ 34 Virtual: Curran Associates, Inc., 2021, pp. 16331–16345
  * [LLHF21] Ruihui Li, Xianzhi Li, Ka-Hei Hui and Chi-Wing Fu  “SP-GAN: Sphere-guided 3D shape generation and manipulation”  In _ACM Transactions on Graphics (TOG)_ 40.4 ACM New York, NY, USA, 2021, pp. 1–12
  * [LLW22] Shidi Li, Miaomiao Liu and Christian Walder  “EditVAE: Unsupervised Parts-Aware Controllable 3D Point Cloud Shape Generation”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 36.2, 2022, pp. 1386–1394
  * [LLWL20] Cheng-Han Lee, Ziwei Liu, Lingyun Wu and Ping Luo  “Maskgan: Towards diverse and interactive facial image manipulation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ Virtual: IEEE, 2020, pp. 5549–5558
  * [LMR*15] Matthew Loper et al.  “SMPL: A skinned multi-person linear model”  In _ACM transactions on graphics (TOG)_ 34.6 ACM New York, NY, USA, 2015, pp. 1–16
  * [LYF*21] Sandro Lombardi et al.  “LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies”  In _2021 International Conference on 3D Vision (3DV)_ Virtual: IEEE, 2021, pp. 278–288
  * [Lyo14] Aidan Lyon  “Why are normal distributions normal?”  In _The British Journal for the Philosophy of Science_ 65.3 Oxford University Press, 2014, pp. 621–649
  * [MLX*17] Xudong Mao et al.  “Least squares generative adversarial networks”  In _Proceedings of the IEEE international conference on computer vision_ Venice, Italy: IEEE, 2017, pp. 2794–2802
  * [MPN*20] Stylianos Moschoglou et al.  “3dfacegan: Adversarial nets for 3d face representation, generation, and translation”  In _International Journal of Computer Vision_ 128 Springer, 2020, pp. 2534–2551
  * [MRC*21] Riccardo Marin et al.  “Spectral shape recovery and analysis via data-driven connections”  In _International journal of computer vision_ 129 Springer, 2021, pp. 2745–2760
  * [NW17] C. Nash and C… Williams  “The Shape Variational Autoencoder: A Deep Generative Model of Part-Segmented 3D Objects”  In _Computer Graphics Forum_ 36.5 Chichester, GBR: The Eurographs Association & John Wiley Sons, Ltd., 2017, pp. 1–12
  * [OBB20] Ahmed A A Osman, Timo Bolkart and Michael J. Black  “STAR: A Sparse Trained Articulated Human Body Regressor”  In _European Conference on Computer Vision (ECCV)_ Virtual: Springer International Publishing, 2020, pp. 598–613
  * [OBD*21] Nicolas Olivier et al.  “FaceTuneGAN: Face Autoencoder for Convolutional Expression Transfer Using Neural Generative Adversarial Networks”  In _arXiv preprint arXiv:2112.00532_ arXiv, 2021
  * [OFD*22] Naima Otberdout et al.  “Sparse to dense dynamic 3d facial expression generation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ New Orleans, Louisiana, USA: IEEE, 2022, pp. 20385–20394
  * [OvdLP*22] Eimear O’Sullivan et al.  “Convolutional mesh autoencoders for the 3-dimensional identification of FGFR-related craniosynostosis”  In _Scientific reports_ 12.1 Nature Publishing Group, 2022, pp. 1–8
  * [PRMB15] Gerard Pons-Moll, Javier Romero, Naureen Mahmood and Michael J Black  “Dyna: A model of dynamic human shape in motion”  In _ACM Transactions on Graphics (TOG)_ 34.4 ACM New York, NY, USA, 2015, pp. 1–14
  * [PVS*21] Stylianos Ploumpis et al.  “Towards a Complete 3D Morphable Model of the Human Head”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 43.11, 2021, pp. 4142–4160
  * [PWP*19] Stylianos Ploumpis et al.  “Combining 3D Morphable Models: A Large scale Face-and-Head Model”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ Long Beach, California, USA: IEEE, 2019, pp. 10934–10943
  * [RBSB18] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal and Michael J Black  “Generating 3D faces using convolutional mesh autoencoders”  In _Proceedings of the European Conference on Computer Vision (ECCV)_ Munich, Germany: Springer International Publishing, 2018, pp. 704–720
  * [RDC*21] Dominic Roberts et al.  “LSD-StructureNet: Modeling Levels of Structural Detail in 3D Part Hierarchies”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ Virtual: IEEE, 2021, pp. 5836–5845
  * [RKH*21] Alec Radford et al.  “Learning Transferable Visual Models From Natural Language Supervision”  In _Proceedings of the 38th International Conference on Machine Learning_ 139, Proceedings of Machine Learning Research  Virtual: PMLR, 2021, pp. 8748–8763
  * [RL21] Travers Rhodes and Daniel Lee  “Local Disentanglement in Variational Auto-Encoders Using Jacobian $L\\_1$ Regularization”  In _Advances in Neural Information Processing Systems_ 34 Virtual: Curran Associates, Inc., 2021
  * [RWP06] Martin Reuter, Franz-Erich Wolter and Niklas Peinecke  “Laplace–Beltrami spectra as ‘Shape-DNA’of surfaces and solids”  In _Computer-Aided Design_ 38.4 Elsevier, 2006, pp. 342–366
  * [SBKM21] Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky and Gerard Medioni  “Gan-control: Explicitly controllable gans”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ Virtual: IEEE, 2021, pp. 14083–14093
  * [SNF*13] David I Shuman et al.  “The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains”  In _IEEE signal processing magazine_ 30.3 IEEE, 2013, pp. 83–98
  * [SYTZ22] Yujun Shen, Ceyuan Yang, Xiaoou Tang and Bolei Zhou  “InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44.4 IEEE, 2022, pp. 2004–2018
  * [TDM11] J. Tena, Fernando De la Torre and Iain Matthews  “Interactive Region-Based Linear 3D Face Models”  In _ACM Transactions on Graphics_ 30.4 New York, NY, USA: Association for Computing Machinery, 2011
  * [TSL21] Norman Joseph Tatro, Stefan C Schonsheck and Rongjie Lai  “Unsupervised Geometric Disentanglement via CFAN-VAE”  In _ICLR 2021 Workshop on Geometrical and Topological Representation Learning_ Virtual: OpenReview.net, 2021
  * [TZY*22] Qingyang Tan et al.  “Variational Autoencoders for Localized Mesh Deformation Component Analysis”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44.10, 2022, pp. 6297–6310
  * [VB20] Andrey Voynov and Artem Babenko  “Unsupervised discovery of interpretable directions in the gan latent space”  In _International conference on machine learning_ Virtual: PMLR, 2020, pp. 9786–9796  PMLR
  * [VRM*17] Gul Varol et al.  “Learning from synthetic humans”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ Honolulu, Hawaii, USA: IEEE, 2017, pp. 109–117
  * [WDH*19] Wei Wang et al.  “Backpropagation-friendly eigendecomposition”  In _Advances in Neural Information Processing Systems_ 32, 2019
  * [WYH*21] Tan Wang et al.  “Self-Supervised Learning Disentangled Group Representation as Feature”  In _Advances in Neural Information Processing Systems_ 34 Virtual: Curran Associates, Inc., 2021
  * [YFST18] Yaoqing Yang, Chen Feng, Yiru Shen and Dong Tian  “Foldingnet: Point cloud auto-encoder via deep grid deformation”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ Salt Lake City, Utah, USA: IEEE, 2018, pp. 206–215
  * [YHH*19] Guandao Yang et al.  “Pointflow: 3d point cloud generation with continuous normalizing flows”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ Seoul, Korea (South): IEEE, 2019, pp. 4541–4550
  * [YLY*20] Yu-Jie Yuan et al.  “Mesh variational autoencoders with edge contraction pooling”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ Virtual: IEEE, 2020, pp. 274–275
  * [YML*20] Jie Yang et al.  “DSM-Net: Disentangled Structured Mesh Net for Controllable Generation of Fine Geometry”  In _arXiv preprint arXiv:2008.05440_ arXiv, 2020
  * [ZBP20] Keyang Zhou, Bharat Lal Bhatnagar and Gerard Pons-Moll  “Unsupervised shape and pose disentanglement for 3d meshes”  In _European Conference on Computer Vision_ Virtual: Springer International Publishing, 2020, pp. 341–357  Springer
  * [ZKJB17] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs and Michael J Black  “3D menagerie: Modeling the 3D shape and pose of animals”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ Honolulu, Hawaii, USA: IEEE, 2017, pp. 6365–6373
  * [ZVD10] Hao Zhang, Oliver Van Kaick and Ramsay Dyer  “Spectral Mesh Processing”  In _Computer Graphics Forum_ 29.6, 2010, pp. 1865–1894
  * [ZWL*20] Yi Zhou et al.  “Fully convolutional mesh autoencoder using efficient spatially varying kernels”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 9251–9262
  * [ZXT20] Xinqi Zhu, Chang Xu and Dacheng Tao  “Learning disentangled representations with latent variation predictability”  In _European Conference on Computer Vision_ Virtual: Springer International Publishing, 2020, pp. 684–700
  * [ZYHC22] Mingwu Zheng, Hongyu Yang, Di Huang and Liming Chen  “ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ New Orleans, Louisiana, USA: IEEE, 2022, pp. 20343–20352
  * [ZYL*20] Zihui Zhang et al.  “Learning Distribution Independent Latent Representation for 3D Face Disentanglement”  In _2020 International Conference on 3D Vision (3DV)_ Virtual: IEEE, 2020, pp. 848–857