smequation

$\BODY$ (1)

smequation*

$\BODY$

# FaceDet3D: Facial Expressions with 3D Geometric Detail Prediction

ShahRukh Athar1 Albert Pumarola2 Francesc Moreno-Noguer2 Dimitris Samaras1

1Stony Brook University
2Institut de Robòtica i Informàtica Industrial  CSIC-UPC

###### Abstract

Facial Expressions induce a variety of high-level details on the 3D face
geometry. For example, a smile causes the wrinkling of cheeks or the formation
of dimples, while being angry often causes wrinkling of the forehead.
Morphable Models (3DMMs) of the human face fail to capture such fine details
in their PCA-based representations and consequently cannot generate such
details when used to edit expressions. In this work, we introduce FaceDet3D, a
first-of-its-kind method that generates - from a single image - geometric
facial details that are consistent with any desired target expression. The
facial details are represented as a vertex displacement map and used then by a
Neural Renderer to photo-realistically render novel images of any single image
in any desired expression and view. The project website can be found here.

##  1 Introduction

Modelling the geometry of the human face continues to attract great interest
in the computer vision and computer graphics communities. Strong PCA-based
priors make 3D morphable models (3DMMs) [3] robust, but at the same time over-
regularize them, so they fail to capture fine facial details, such as the
wrinkles on the forehead when the eyebrows are raised or bumps on the cheeks
when one smiles. Additionally, the lack of diversity in the texture space of
most available 3DMMs make it very hard to generate realistic renderings that
capture the large variations of color and texture we observe in human faces.
Recent methods [28, 29, 33, 32, 39, 5, 9, 16, 20, 11] address this by learning
richer shape and expression spaces using a variety of data modalities such as
in-the-wild images [29, 33, 32, 39] and videos [28]. However, despite using
more expressive shape and expression spaces, these models still fail to
capture fine details in geometry.

Figure 1: Facial detail prediction and rendering. Given a single input image
and a target expression (in this case ‘Happy’), our method predicts facial
geometric details consistent with the target expression and renders realistic
images. (Electronic zoom recommended)

Recent methods that accurately estimate facial geometric details from single
images [6, 34], while being unable to predict novel details under expression
change, have nonetheless enabled the large scale annotation of unpaired image
data with facial geometric details. Thus, it is now possible to train facial
detail prediction methods using unsupervised adversarial losses [7, 24].
Similarly, Neural Rendering [31] has made it possible to render 3D geometries
with photo-realistic detail via the use of high-dimensional Neural Texture
Maps (NTMs). However, NTMs are able to store fine details of the output image,
causing the rendered details to be completely independent of the geometric
details. They do not change even if the geometric details do, making neural
rendering unsuitable for rendering facial geometric details. In this paper we
introduce FaceDet3D, which is, to the best of our knowledge, the first
generative model that predicts facial geometric details for any target
expression and renders them realistically to the image space. The model is
made up of two components: 1) A detail prediction network that infers
plausible geometric facial details of a person as their expression changes,
given a single image of that person. 2) A rendering network that overcomes the
aforementioned shortcoming of neural rendering and explicitly conditions the
rendered facial details on the details of the 3D face geometry. This
conditioning is achieved through the use of the novel Augmented Wrinkle Loss
and the Detailed Shading Loss during training.

Our method is trained using only a large scale in-the-wild image dataset and a
much smaller video dataset captured in controlled conditions, without any 3D
data as supervision for the target expression geometry. An exhaustive
evaluation shows that once trained, our method is able to generate plausible
facial details for any desired anatomically consistent facial expression and
render it realistically in any desired view direction.

##  2 Related Work

We next describe the most related methods in facial geometry estimation,
geometric facial detail estimation and facial expression editing and
reanimation.

##### Facial Geometry Estimation

Traditional facial geometry estimation methods fit statistical 3D face models
to a given image, going back to the original 3D Face Morphable Model [3].
However, the PCA-based prior used by 3DMMs is not flexible enough to represent
fine facial geometric details such as wrinkles and skin bumps. Recent methods
[28, 29, 33, 32, 39, 5, 9, 16, 20, 11] leverage the power of deep-learning and
large-scale image and video datasets to regress parameters that generate
realistic 3D reconstructions or learn complex representations for face shape,
expression and texture. These methods generate more realistic results than
traditional 3DMM fitting, but still cannot capture fine facial details. In our
work we use the Basel Face Model [12] as the underlying geometry on top of
which we predict geometric details.

##### Geometric Facial Details Estimation.

Over the past few years there has been significant improvement in the realism
of 3D face geometries estimated from single images. In [26], the regressed
correspondence and depth maps are registered onto a template mesh, which is
further refined to generate the detailed facial geometry. In [34] facial
details are modelled with bump maps on top of a 3DMM base. Similarly, in
Facial Details Synthesis (FDS)[6], details are represented as vertex
displacements of an underlying 3DMM, trained using a combination of 3D data
and in-the-wild images. These methods, however, can only estimate the facial
details of the expression manifested in an image, but cannot predict novel
facial details for a different expression, which is the motivation of our
method.

##### Facial Expression Editing.

The success of image-to-image translation networks [15], adversarial training
[13] and cycle-consistency losses [38] have led to the development of
expression editing methods that use large-scale in-the-wild training datasets.
In [27], an unsupervised face model disentangles the input face into albedo,
normals and shading. Expressions are then edited by traversals in the
disentangled latent space. In [7], expressions are edited via adversarial
losses coupled with cycle consistency. In [24], a network edits input images
to target expressions, represented as Action Units [10]. [2] extends this work
by explicitly modeling skin motion followed by texture hallucination. In [8],
template images are used for editing. While these methods give photo-realistic
results, they are restricted to the 2D Image space and cannot be used to
manipulate 3D viewpoint.

##### Facial Reanimation Methods.

Facial renanimation methods use driving parameters from a source video to
reanimate a source image or video. [19], performs the reanmatiion using the
parameters of a 3DMM extracted from the source video. In [31], Neural
Rendering is used to reanimate videos using expression parameters extracted
from a source video. However, this renderer is needs to be retrained for each
target video. In [30], audio parameters drive the target video. In [36],
facial landmarks from a source video are used to animate single images. While
all these methods generate realistic re-animations, they do not model any
facial geometric detail. More closely related to our work is [23], which
regresses a set of FACS [10] based textures from single images and are able to
generate realistic renderings in any desired expression. Nevertheless, they do
not predict any facial geometric detail but instead generate the details in
the texture space.

##  3 FaceDet3D

###  3.1 Problem Formulation

A face image $I_{\mathbf{x}}\in\mathbb{R}^{H\times W\times 3}$ has expression
$\mathbf{x}$ represented by Action Units [10]. Its shape and expression
parameters in the Basel Face Model (BFM) [12] space are
$\\{\alpha_{s},\alpha_{e}\\}$. A vertex displacement UV map
$\mathcal{D}(I_{\mathbf{x}})\in\mathbb{R}^{H_{\mathcal{D}}\times
W_{\mathcal{D}}\times 3}$ encodes the facial details of the person shown in
$I_{\mathbf{x}}$ with the expression $\mathbf{x}$. We want to: (1) Generate
facial details, represented by a vertex displacement map,
$\mathcal{D}(I_{\mathbf{y}})$ as the expression of that person changes to
$\mathbf{y}$; (2) Render an image of the geometry with these details.

We use FDS [6] to extract the texture map, initial detail map and geometry of
the input image $I_{\mathbf{x}}$: T(I_x), D(I_x), α_s, α_e ←FDS(I_x) where,
$\mathcal{T}(I_{\mathbf{x}})\in\mathbb{R}^{H_{\mathcal{T}}\times
W_{\mathcal{T}}\times 3}$ is the texture map.

Next, we use the detail prediction network, $DetP(\cdot)$, to predict the
plausible facial detail map of the person in $I_{\mathbf{x}}$ for expression
$\mathbf{y}$ as: ~D(I_y) = DetP(D(I_x), x, y, ^α_e, Age(I_x), FaceID(I_x))
where, $\mathbf{y}$, $\hat{\alpha_{e}}$ are the target AUs and expression
parameters, $Age(I_{\mathbf{x}})$ are features extracted from an age
prediction network and $FaceID(I_{\mathbf{x}})$ is the facial embedding of
$I_{\bf{x}}$ extracted using [25]. Note, we do not have access to the ground
truth image $I_{\mathbf{y}}$, thus we predict a plausible detail map of
$I_{\mathbf{y}}$ (i.e $\mathcal{\widetilde{D}}(I_{y})$) using $DetP$. Once we
have $\mathcal{\widetilde{D}}(I_{y})$, we use it to displace the vertices
along their normal direction giving us the detailed geometry. Finally, we
render this detailed face geometry using a rendering network $R(\cdot)$: ~I_y
= R(T(I_x), ~D(I_y), α_s, ^α_e, y, c, l, γ) where, $c$ are the desired camera
and view parameters, $\gamma$ is the albedo PCA-space parameters of BFM[12],
and $l$ are the lighting parameters.

###  3.2 Detail Prediction

Figure 2: Detail Prediction Network. $DetP$ first extracts from its various
inputs high-dimensional features and concatenates them. This concatenated
feature map is then used to predict a coarse detail map
$\mathcal{D}(I_{\mathbf{y}})^{C}$. The coarse detail map is then refined via a
refinement module to predict a fine detail map
$\mathcal{D}(I_{\mathbf{y}})^{F}$ which is then added to the coarse detail
map,$\mathcal{D}(I_{\mathbf{y}})^{C}$, to give
$\widetilde{\mathcal{D}}(I_{\mathbf{y}})$.

$DetP$ takes the input detail map, $\mathcal{D}(I_{\mathbf{x}})$, the target
expression parameters $\hat{\alpha_{e}}$, the shape parameters
$\alpha_{shape}$, the face embedding $FaceID(I_{\mathbf{x}})$, the action unit
$\mathbf{x}$, and the age features $Age(I_{\mathbf{x}})$ and extracts features
from each of them. There features are concatenated in the channel dimension
and passed though the $Decoder$ which receives the target action unit
$\mathbf{y}$ via Adaptive Instance Normalization [14] layers. The $Decoder$
gives as output a hallucination $\mathcal{D}(I_{\mathbf{y}})^{C\text{-}Hal}$
and a mask $\mathcal{D}(I_{\mathbf{y}})^{C\text{-}Mask}$ which are combined
together to give $\mathcal{D}(I_{\mathbf{y}})^{C}$: D(Iy)C=
D(Iy)C-Mask⊙D(Iy)C-Hal\+ (1 D(Iy)C-Mask) ⊙D(Ix)
$\mathcal{D}(I_{\mathbf{y}})^{C}$ then goes through a refinement module that
generates $\mathcal{D}(I_{\mathbf{y}})^{F\text{-}Hal}$ and a mask
$\mathcal{D}(I_{\mathbf{y}})^{F\text{-}Mask}$ which are combined to give
$\mathcal{D}(I_{\mathbf{y}})^{F}$: D(Iy)F= D(Iy)F-Mask⊙D(Iy)F-Hal\+ (1
D(Iy)F-Mask) ⊙D(Ix) Finally,
$\widetilde{\mathcal{D}}(I_{\mathbf{y}})=\mathcal{D}(I_{\mathbf{y}})^{C}+\mathcal{D}(I_{\mathbf{y}})^{F}$.
With this two-stage approach $\mathcal{D}(I_{\mathbf{y}})^{C}$ is able to
capture coarse-scale detail changes while $\mathcal{D}(I_{\mathbf{y}})^{F}$
captures the finer scale changes. The masking mechanism used in both stages
ensures the preservation of the details that are not meant to be changed with
the expression.

####  3.2.1 Training Losses

Due to the lack of a large scale in-the-wild dataset of paired data with
expression change or 3D data, we cannot directly perform a regression using
ground-truth geometric facial details, $\mathcal{D}(I_{\mathbf{y}})$ of the
image $I_{\mathbf{y}}$. Therefore, we instead use an adversarial training
paradigm along with cycle-consistency losses similar to [2, 24] to predict the
plausible facial geometric details $\mathcal{\widetilde{D}}(I_{y})$ and to
ensure the prediction’s fidelity to the target expression and input features.
In order to speed up convergence, we weakly supervise the adversarial training
using randomly sampled frames from videos of the MUG [1] and the ADFES
datasets [35], ensuring that the sampling is sparse enough such that there is
significant change in expression with frames sampled from each video.

##### Expression Adversarial Loss.

In order to ensure the predicted facial geometric details,
$\mathcal{\widetilde{D}}(I_{\mathbf{y}})$, are consistent with the target
expression $\\{\mathbf{y},\hat{\alpha}_{e}\\}$, we use an expression
discriminator $\textrm{D}_{\textrm{Exp}}$. Given $\mathcal{D}(I_{\mathbf{x}})$
of some image $I_{\mathbf{x}}$ manifesting expression
$\\{\mathbf{x},\alpha_{e}\\}$, $\textrm{D}_{\textrm{Exp}}$, outputs the
following D_Exp(D(I_x)) = {r, ^x, ^α_e} where $r$ is a realism score and
$\hat{\mathbf{x}}\text{ and }\hat{\alpha_{e}}$ are the predicted AUs and
expression parameters, respectively. For brevity, we will use
$\textrm{D}_{\textrm{Exp}}(\mathcal{D}(I_{\mathbf{x}}))$ and
$\textrm{D}_{\textrm{Exp}}(\mathcal{D})$ interchangeably. We use the Non-
Saturating adversarial loss [13] along with the $R1$ gradient penalty [22] to
train $\textrm{D}_{\textrm{Exp}}$. Specifically, let
$\mathcal{P}_{\mathcal{D}}$ be the distribution of real facial geometric
detail maps, the adversarial loss for $\textrm{D}_{\textrm{Exp}}$ can be
written as: LDExpAdv= -E~D∼DetP(.)[log (1 DExpr(~D))]- ED ∼PD[log (DExpr(D))]+
ED ∼PD[||∇DExpr(D)||22] where $\textrm{D}_{\textrm{Exp}}^{r}$ is the realism
output head of $\textrm{D}_{\textrm{Exp}}$. In addition,
$\textrm{D}_{\textrm{Exp}}$ is trained to minimize the error of the predicted
AU and expression parameters LDExpExp= ED ∼PD[||[DExpAU(D) x||22]+ ED
∼PD[||[DExpαe(D) - αe||22] where
$\textrm{D}_{\textrm{Exp}}^{\textrm{AU}}\text{ and
}\textrm{D}_{\textrm{Exp}}^{\alpha_{e}}$ are the AU and expression parameter
output head of $\textrm{D}_{\textrm{Exp}}$ respectively. The Detail Prediction
Network, $DetP$ is trained to minimize the adversarial loss LDetPExpAdv= Ix,
{y, ^αe} Elog (DExpr(DetP(⋅))) and the expression losses LDetPAU= Ix, {y, ^αe}
E||DExpAU(DetP(⋅)) y||22 LDetPαe= Ix, {y, ^αe} E||DExpαe(DetP(⋅)) ^αe||22
where, $DetP(\cdot)$ is to be read as in Eq. (3.1) and $\mathbf{y}$ and
$\hat{\alpha}_{e}$ are the target AU and expression parameters respectively.

##### FaceID Loss.

The FaceID loss ensures that facial details characteristic of the subject’s
identity, that are invariant to expression change, are preserved in the
predicted details. A face embedding detection network,
$\textrm{D}_{\textrm{Face}}$, is trained to predict the $FaceNet$ [25]
features of an image $I_{\mathbf{x}}$ from its detail map
$\mathcal{D}(I_{\mathbf{x}})$ as follows LDFaceID= ED ∼PD||DFaceID(D(Ix))
FaceNet(Ix)||22

where $\textrm{D}_{\textrm{Face}}^{\textrm{ID}}$ is the face embedding output
head of $\textrm{D}_{\textrm{Face}}$. Additionally,
$\textrm{D}_{\textrm{Face}}$ is also trained with an adversarial loss with the
same loss as shown in Eq. (3.2.1). The detail prediction network, $DetP$ is
trained to minimize both the FaceID loss and the adversarial loss LDetPID= Ix,
{y, ^αe}E||DFaceID(DetP(⋅)) FaceNet(Ix)||22 LDetPIDAdv= Ix, {y, ^αe} Elog
(DFacer(DetP(.)) where $\textrm{D}_{\textrm{Face}}^{\textrm{r}}$ is the
realism output head of $\textrm{D}_{\textrm{Face}}$.

##### Age Loss.

The Age Loss ensures that the facial details predicted are consistent with the
subject’s age by ensuring the age feature embedding of the predicted detail
map matches that of the input details. An age feature prediction network,
$\textrm{D}_{\textrm{Age}}$ is trained to predict the features of an image
$I_{\mathbf{x}}$ extracted using a pre-trained age prediction network,
$AgeNet$, from its detail map $\mathcal{D}(I_{\mathbf{x}})$ as follows
LDAgeAge= ED ∼PD||DAgeAge(D(Ix)) AgeNet(Ix)||22 where
$\textrm{D}_{\textrm{Age}}^{\textrm{Age}}$ is the age feature output head of
$\textrm{D}_{\textrm{Age}}$. Additionally, $\textrm{D}_{\textrm{Age}}$ is also
trained with an adversarial loss of the same type as shown in Eq. (3.2.1). The
detail prediction network, $DetP$ is trained to minimize both the Age loss and
the adversarial loss LDetPAge= Ix, {y, ^αe}E||DAgeAge(DetP(.)) AgeNet(Ix)||22
LDetPAgeAdv= Ix, {y, ^αe} -Elog (DAger(DetP(.)) where
$\textrm{D}_{\textrm{Age}}^{\textrm{r}}$ is the realism output head of
$\textrm{D}_{\textrm{Age}}$.

##### Regression Loss.

In order to speed up training, we use a small amount of video data from MUG
[1] and ADFES [35] to directly regress the details map estimated by FDS [6] as
~D(Iyk) = DetP(D(Ixm), x, y, ^αe, Age(Ixm), FaceID(Ixm))LRegressDetP=
LapLoss(~D(Iyk), D(Iyk)) where, $\mathcal{D}(I_{\bf{y}}^{k})$ and
$\mathcal{D}(I_{\bf{x}}^{m})$ are the detail map of $k$-th frame
$I_{\bf{y}}^{k}$ and $m$-th frame $I_{\bf{x}}^{m}$ respectively. Training
solely on video data is not possible due to the significant bias the dataset
has towards younger subjects.

##### Superresolution Losses

The detail maps generated by FDS [6] are of resolution $4096\times 4096$ and
thus cannot be used directly for training due to GPU-memory constraints. To
get around this, we train $DetP$ on detail maps downsampled to $256\times
256$. Simultaneously, we finetune a superresolution network, $RCAN$ [37], to
super-resolve downsampled $256\times 256$ patches of $\mathcal{D}(I_{\bf{x}})$
by a factor of 4 L_SR^RCAN = L1(RCAN(D(I_x)_256^P, D(I_x)_1024^P) where
$\mathcal{D}(I_{\bf{x}})_{1024}^{P}$ is a randomly sampled patch of resolution
$1024\times 1024$ from the full-resolution detail map
$\mathcal{D}(I_{\bf{x}})$ and $\mathcal{D}(I_{\bf{x}})_{256}^{P}$ is its
downsampled version. During inference, we use $RCAN$ twice on the predicted
detail map $\mathcal{\widetilde{D}}(I_{\bf{y}})$ to upsample it to $4096\times
4096$ ~D(I_y)^HR = RCAN(RCAN(~D(I_y))) In the interest of brevity, we will use
$\mathcal{\widetilde{D}}(I_{\bf{y}})$ in lieu of
$\mathcal{\widetilde{D}}(I_{\bf{y}})^{HR}$ in the remainder of this text.

###  3.3 Rendering Network

Figure 3: The Rendering Network. The Rendering Network, $R$ first predicts a
Neural Texture Map $NTM$ from the given input texture map $\mathcal{T}$ using
$NTex_{\phi}$. The $NTM$ is then rasterized using both the proxy (geometry w/o
details) and the detailed geometry and input into an image rendering network
$Tex2Im$. $Tex2Im$ generates a rendering of the details
$\hat{I_{\mathbf{y}}}^{\mathcal{D}}$ and a low-resolution image
$\hat{I_{\mathbf{y}}}^{LR}$ that contains only detail-invariant image
textures. They are added together to generate the final rendered image
$\hat{I_{\mathbf{y}}}$.

The rendering network, $R$, consists of two subnetworks: (1) The Neural
Texture prediction network $NTex_{\phi}$ and (2) The image rendering network
$Tex2Im$.

##### Neural Texture Prediction.

$NTex_{\phi}$ predicts the Neural Textures given the texture map
$\mathcal{T}(I_{\bf{x}})$ of image $I_{\bf{x}}$: NTM = NTex_ϕ(T(I_x)); NTM
∈R^F×H×W where $NTM$ is the predicted neural texture map with $F$ channels.
Unlike in [31], where the $NTM$ is identity specific, $NTex_{\phi}$ can be
used on any $\mathcal{T}(I_{\bf{x}})$ regardless of identity to generate its
corresponding neural texture map. Through training, $NTex_{\phi}$ learns to
extract the appropriate high-dimensional texture features from
$\mathcal{T}(I_{\bf{x}})$ such that $NTM$ can be used to generate a realistic
render of the person in $I_{\bf{x}}$ in any desired expression and view.

##### Image rendering.

The image rendering network, $Tex2Im$, consists of two branches, the low-res
image renderer $g_{\theta}$ and the detail renderer $g_{\omega}$. The low-res
image renderer generates identity-specific image textures that are invariant
to the predicted geometric details, such as the skin-tone or eye color. The
detail renderer $g_{\omega}$ renders the facial geometric details obtained
from the detail prediction network, $DetP$, and adds them to the low-res image
generated by $g_{\theta}$. The separation of the image rendering network into
two branches allows each branch to focus on its respective task and leads to
high-quality renderings.

The low-res image renderer, $g_{\theta}$ inputs the $NTM$ sampled using the UV
map rasterized by the proxy geometry,
$G_{P}=\\{0\times\widetilde{\mathcal{D}}(I_{\mathbf{y}}),\alpha_{s},\hat{\alpha_{e}}\\}$,
i.e the geometry without any details, along with the shaded albedo also
rasterized by $G_{P}$: ^I_y^LR = g_θ(Sample(NTM, UVRaster(G_P,c)),y, γ, l)
where $\gamma$ are the coefficients of the albedo PCA-space of the BFM [12]
and $l$ are the lighting parameters. Since $g_{\theta}$ only uses inputs
dependent on $G_{P}$ it generates the image textures that are invariant to
details predicted by $DetP$.

The detail renderer, $g_{\omega}$ takes as input the $NTM$ sampled using the
UV map rasterized by the detailed geometry,
$G_{\mathcal{D}}=\\{\widetilde{\mathcal{D}}(I_{\mathbf{y}}),\alpha_{s},\hat{\alpha_{e}}\\}$
along with the shaded albedo also rasterized by $G_{\mathcal{D}}$: ^I_y^D =
g_θ(Sample(NTM, UVRaster(G_D,c)),y, γ, l)  where $\gamma$ and $l$ are the
albedo PCA-space and lighting parameters. The final output image is calculated
as: ^I_y = ^I_y^LR + ^I_y^D Since all of the detail invariant textures are
already generated by $g_{\theta}$ in $\hat{I_{\bf{y}}^{LR}}$, the detail
renderer, $g_{\omega}$ can solely focus on realistically rendering the details
predicted by $DetP$.

####  3.3.1 Training Losses

The renderings generated by $R$ must: (1) faithfully render the geometric
details onto the RGB space and (2) be realistic. Neural Rendering [31] is
designed to address (2) as the high-dimensional neural texture map is able
store the fine details of the output texture. Consequently, the details on the
rendered image become entirely conditional on the input texture map,
$\mathcal{T}(I_{\mathbf{x}})$, and ignore the detailed geometry
$G_{\mathcal{D}}$. This significantly hurts (1) causing the details on the
rendered image to remain unchanged even if the facial geometric details change
due to changes in facial expression. In order to ensure the output renderings
satisfy (2) we use, along with the branched architecture discussed in Sect
3.3, an Augmented Wrinkle Loss and the Detailed Shading Loss to ensure the
geometric details are faithfully rendered onto the output image.

##### Photometric Loss.

The Photometric Loss ensures the rendered images are realistic by re-rendering
a given image $I_{\bf{x}}$, producing $\hat{I}_{\bf{x}}$ and comparing it to
the ground truth. More specifically, given the texture map
$\mathcal{T}(I_{\mathbf{x}})$, detailed geometry
$G_{\mathcal{D}}=\\{\mathcal{D}(I_{\mathbf{x}}),\alpha_{s},\alpha_{e}\\}$ and
action unit $\bf{x}$, $I_{\bf{x}}$ is re-rendered using $R$ as follows ^Ix=
R(T(Ix), GD, x, c, l, γ) where $c$ and $l$ are the camera and lighting
parameters of $I_{\bf{x}}$. The re-rendered image $\hat{I}_{\mathbf{x}}$ is
then compared to $I_{\bf{x}}$ LPhoto= MSE(^Ix, Ix) + L1(^Ix, Ix)+ LapLoss(^Ix,
Ix) + PcptL(^Ix, Ix) where $LapLoss$ is the Laplacian Loss [21, 4] and $PcptL$
is the perceptual loss [17]. In order to ensure the low-res rendering captures
the image textures that are invariant to facial details as much as possible,
the photometric loss is also applied to the low-res output of $g_{\theta}$ i.e
$\hat{I_{\bf{y}}^{LR}}$ LPhotoLR= MSE(^IxLR, Ix) + L1(^IxLR, Ix)+
LapLoss(^IxLR, Ix) + PcptL(^IxLR, Ix)

Figure 4: Expression Change. Here we show the results of detail prediction and
rendering as the expression changes. The first column is the input image, the
second column is the reconstruction and the subsequent columns are the results
under different expressions. The first image row is the output rendering and
the second is shading of the detailed geometry. As one can see $DetP$ is able
to generate realistic details depending on the expression being manifested and
$R$ is able to faithfully render them to the image space. We zoom-in on a
subset of details in the final column for greater clarity. (Please view in
high resolution)

##### Augmented Wrinkle Loss.

In order to enforce the rendering of geometric details onto the rendered image
we add ‘fake’ wrinkles to an image $I_{\bf{x}}$ and force $R$ to generate the
same. Given the the detailed geometry of $I_{\bf{x}}$,
$G_{\mathcal{D}}=\\{\mathcal{D}(I_{\mathbf{x}}),\alpha_{s},\alpha_{e}\\}$, a
geometry with ‘fake’ details
$G_{\mathcal{D}}^{*}=\\{\mathcal{D}(I^{*}_{\mathbf{z}}),\alpha_{s},\alpha_{e}\\}$
using the geometric details from some random image $I^{*}_{\mathbf{z}}$ and
the lighting $l$ of $I_{\bf{x}}$, the ‘fake’ wrinkles are added as follows
Shading(Ix) = LSph(GD, l); Shading*(Ix) = LSph(GD*, l)Ix*= Shading*(Ix)
×(IxShading(Ix)) where $L_{Sph}$ is the spherical harmonic lighting function
and $l$ are the coefficients of the first 9 spherical harmonics. The
artificially wrinkled image $I_{\bf{}_{x}}^{*}$ is now re-rendered using $R$
^Ix*= R(T(Ix), GD*, x, c, l)LAugW= LapLoss(^Ix*, Ix*) In order to faithfully
reconstruct $I_{\bf{}_{x}}^{*}$, $R$ is forced to rely on the detailed
geometry $G_{\mathcal{D}}^{*}$, since the input texture map
$\mathcal{T}(I_{\mathbf{x}})$, and consequently the neural texture map contain
no information about the ‘fake’ wrinkles.

##### Detailed Shading Loss.

In addition to the Augmented Wrinkle Loss, we also try to predict the shading
of the detailed facial geometry from the output rendering
$\hat{I}_{\mathbf{x}}$ ^Shading(^Ix) = fθ(^Ix)LDSL= LapLoss(^Shading(^Ix),
Shading*(Ix)) where $f_{\theta}$ is a small convolutional network (CNN) with
only two layers and the shading $Shading^{*}(I_{\bf{}_{x}})$ is calculated as
in Eq. (3.3.1). We calculate this loss only over the skin region. Since,
$f_{\theta}$ is a small CNN with limited representational capacity, the
details must be quite visible on the rendered image $\hat{I}_{\mathbf{x}}$ in
order for them to be picked up by $f_{\theta}$ to generate an accurate shading
$\hat{Shading}(\hat{I}_{\mathbf{x}})$.

##### Expression Adversarial Loss.

In order to ensure that the rendered output conforms to the target expression
we use an expression adversarial loss. Given a rendered image,
$\hat{I}_{\mathbf{x}}=R(\mathcal{T}(I_{\mathbf{x}}),G_{\mathcal{D}},\mathbf{x},c,l)$,
manifesting the expression encoded by AU $\bf{x}$ an expression discriminator,
$\textrm{D}_{\textrm{Exp}}^{\textrm{RGB}}$, outputs D_Exp^RGB(^I_x) = {r, ^x}
where $r$ is a realism score and $\hat{\bf{x}}$ is the predicted AU. We use
the Non-Saturating adversarial loss [13] along with the $R1$ gradient penalty
[22] to train $\textrm{D}_{\textrm{Exp}}$. More specifically, let
$\mathcal{P}_{\mathcal{I}}$ be the distribution of real images, the
adversarial loss can then be written as LDExpRGBAdv= -E^Ix∼R(.)[log (1
DExpRGB, r(^Ix))]- EIx∼PI[log (DExpRGB, r(Ix))]+ EIx∼PI[||∇DExpRGB,r(Ix)||22]
where $\textrm{D}_{\textrm{Exp}}^{\textrm{RGB},\textrm{r}}$ is the realism
output head of $\textrm{D}_{\textrm{Exp}}^{\textrm{RGB}}$. In addition,
$\textrm{D}_{\textrm{Exp}}^{\textrm{RGB}}$ is trained to minimize the
predicted AU error LDExpRGBExp= EIx∼PI[||[DExpRGB,AU(Ix) x||22] where
$\textrm{D}_{\textrm{Exp}}^{\textrm{RGB},\textrm{AU}}$ is the AU output head
of $\textrm{D}_{\textrm{Exp}}^{\textrm{RGB}}$. The Rendering Network, $R$ is
trained to minimize adversarial loss LRExpAdv= EIxlog (DExpRGB,r(R(.)) and AU
loss LRAU= EIx||DExpRGB,AU(R(⋅)) y||22 where $R(\cdot)$ is to be read as in
Eq. (3.1).

Figure 5: Details vs. No Details. We compare the results of changing the
expression both with and without predicting details. (Please view in high
resolution) Figure 6: View Consistency. In this figure we demonstrate the
consistency of the details rendered by $R$. A subset of the predicted details
from $DetP$ are marked with blue rectangles. As can be seen, $R$ renders the
details in a consistent manner across views. (Please view in high resolution)
Figure 7: Details under varied lighting. In this figure we show how the
appearence of the predicted geometric facial details changes with lighting
conditions. (Please view in high resolution)

##  4 Experiments

We train the detail prediction network, $DetP$, and the rendering network $R$,
on 9,000 images from the FFHQ dataset [18]. Additionally, 3,000 frames were
sampled from the MUG [1] and the ADFES [35] datasets to speed-up training of
the detail prediction network $DetP$. Due to memory constraints, $DetP$ and
$R$ are trained independently. Upon publication we will release the code.

##### Expression change.

We first demonstrate the detail prediction and rendering results with
expression change. Fig 4 shows the result of changing the expression of some
input image to a variety of expressions. The first column of Fig 4 is the
input image, the second column shows the image reconstructed using the detail
map $\mathcal{D}(I_{\mathbf{x}})$ predicted by FDS [6]. In the subsequent
columns $\mathcal{D}(I_{\mathbf{x}})$ is used as input to generate
$\mathcal{\widetilde{D}}(I_{\mathbf{y}})$ where
$\mathbf{y}=\\{\text{Anger},\text{Disgust},\text{Fear},\text{Happy},\text{Sad},\text{Surprise}\\}$.
The first row shows the final rendered image, i.e the output of $R$ and the
second row shows the shaded geometry with the predicted details, i.e
$G_{\mathcal{D}}$. The predicted details and their corresponding rendering are
marked out with dashed red and blue rectangles. We zoom in the red rectangles
of the last column As seen in the second row of Fig 4, the details predicted
are consistent with the manifested expression. For example, ‘Anger’ and
‘Disgust’ (3rd and 4th column of Fig 4 respectively) show consistent wrinkling
around the forehead and the nose while ‘Happy’ (column 6 Fig 4) shows
consistent wrinkling around the cheeks. Zooming in the last column of Fig 4
highlights the realism of the predicted details produced by the rendering
network, $R$.

Further, in Fig 5 we show the utility of predicting details as the expression
changes. The first row of Fig 5 shows the image rendered with the predicted
details from $DetP$ as the expression changes. The second row of images shows
the images rendered without details, this is done by setting the predicted
detail map to zero. As seen by comparing the skin appearance marked with the
red rectangles, in the first row the skin changes realistically as the
expression of the person changes due to the changing facial geometric details,
while the skin in the second row remains unchanged, significantly hurting
realism. In the supplementary we show further results on expression animation
and encourage the reader to inspect them.

##### View Consistency and lighting variation.

In Fig 6 we show the consistency of the details rendered by $R$ across various
views in novel expressions. While the underlying face model ensures that the
detailed geometry is consistent in any view, there is no guarantee its
rendering generated by $R$ would be too. The first column of Fig 6 is the
input image, the second column is the reconstruction of the input in the
original expression and view and the third column shows the input image
manifesting a novel expression in the view of the input image. The subsequent
columns show the input image rendered with the target expression in novel
views. The blue rectangles around the details in the rendered image show they
are rendered with high fidelity to the shaded geometry and therefore look
consistent across a variety of views. Fig 6 shows that $R$ is able to maintain
a close to one-to-one correspondence while rendering geometric details to the
image space which ensures that they appear consistent across views. In Fig 7,
we show how the appearance of the facial geometry details change with changes
in lighting. While state-of-the-art lighting manipulation is beyond the scope
of this paper (as we do not disentangle lighting), Fig 7 demonstrates the
utility of predicting facial details on the 3D face geometry to facilitate
such a manipulation.

##### Ablation Studies.

We examine the utility of the Augmented Wrinkle Loss (AugW) and the Detailed
Shading Loss (DSL) in rendering the facial geometric details to the image
space. Fig 8 shows the results of training $R$ with and without the AugW and
DP losses. As seen by comparing the results in rows 2 and 3 of Fig 8, without
those losses $R$ fails to render the predicted geometric facial details .

Figure 8: Ablating AugW and DSL. We show that without AugW and DSL details are
not rendered to the image space. (Please view in high resolution)

##  5 Conclusion and Future Work

We have presented FaceDet3D, a method capable of predicting, from a single
image, plausible facial geometric details as the expression changes and render
them photo-realistically. Using adversarial losses along with weak supervision
we train a detail prediction network capable of predicting plausible facial
details for any target expression. A rendering network then renders these
details photo-realistically in a manner that is consistent with the target
expression and the predicted detailed geometry. The Augmented Wrinkle Loss and
Detailed Shading Loss force the rendering network to use cues from the
detailed geometry and not rely solely on the neural texture map to generate
the details, ensuring consistency in the appearance of the rendered details
across a variety of views. Due to the lack of explicit disentanglement of the
lighting within our method, the renderings we generate are entangled with the
input lighting conditions and therefore cannot be controlled independently.
The detail prediction network relies on the detail map estimated by FDS [6] as
input to predict the plausible details of the target expression, therefore it
cannot handle occlusions such as glasses or make-up very well as FDS [6] fails
in those conditions. In future work, we plan to incorporate explicit
disentanglement of lighting in order to enable greater control over the final
rendering along with explicit modelling occlusions and joint training of
$DetP$ and $R$.

##  6 Acknowledgements

This work is supported in part by a Google Daydream Research award, by the
Spanish government with projects HuMoUR TIN2017-90086-R and María de Maeztu
Seal of Excellence MDM-2016-0656, by NSF-IUCRC on CVDI, Medpod Inc, by NICHD
1R21 HD93912-01A1, the Partner University Fund and the SUNY2020 Infrastructure
Transportation Security Center.

## References

  * [1] Niki Aifanti, Christos Papachristou, and Anastasios Delopoulos.  The mug facial expression database.  WIAMIS, pages 1–4, 2010.
  * [2] S Athar, Z Shu, and D Samaras.  Self-supervised deformation modeling for facial expression editing.  In IEEE FG, 2020.
  * [3] Volker Blanz, Thomas Vetter, et al.  A morphable model for the synthesis of 3d faces.  In Proc. SIGGRAPH, 1999.
  * [4] Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam.  Optimizing the latent space of generative networks.  In Int. Conf. Mach. Learn., 2018.
  * [5] James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou.  3d face morphable models" in-the-wild".  In IEEE Conf. Comput. Vis. Pattern Recog. IEEE, 2017.
  * [6] Anpei Chen, Zhang Chen, Guli Zhang, Kenny Mitchell, and Jingyi Yu.  Photo-realistic facial details synthesis from single image.  In Int. Conf. Comput. Vis., 2019.
  * [7] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  * [9] Pengfei Dou, Shishir K Shah, and Ioannis A Kakadiaris.  End-to-end 3d face reconstruction with deep neural networks.  In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
  * [10] P. Ekman and W Friesen.  Facial action coding system: A technique for the measurement of facial movement.  In Consulting Psychologists Press, 1978.
  * [11] Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Freeman.  Unsupervised training for 3d morphable model regression.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [12] Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter.  Morphable face models-an open framework.  In IEEE FG, 2018.
  * [13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Adv. Neural Inform. Process. Syst., 2014.
  * [14] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In Int. Conf. Comput. Vis., 2017.
  * [15] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
  * [16] Aaron S Jackson, Adrian Bulat, Vasileios Argyriou, and Georgios Tzimiropoulos.  Large pose 3d face reconstruction from a single image via direct volumetric cnn regression.  In Int. Conf. Comput. Vis., 2017.
  * [17] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Eur. Conf. Comput. Vis., 2016.
  * [18] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  * [19] H. Kim, P. Garrido, A. Tewari, Weipeng Xu, Justus Thies, M. Nießner, Patrick Pérez, C. Richardt, M. Zollhöfer, and C. Theobalt.  Deep video portraits.  ACM Transactions on Graphics (TOG), 37:1 – 14, 2018.
  * [20] Hyeongwoo Kim, Michael Zollhöfer, Ayush Tewari, Justus Thies, Christian Richardt, and Christian Theobalt.  Inversefacenet: Deep monocular inverse face rendering.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [21] Haibin Ling and Kazunori Okada.  Diffusion distance for histogram comparison.  In IEEE Conf. Comput. Vis. Pattern Recog., 2006.
  * [22] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.  Which training methods for gans do actually converge?  In Int. Conf. Mach. Learn., 2018.
  * [23] K. Nagano, Jaewoo Seo, J. Xing, Lingyu Wei, Zimo Li, S. Saito, Aviral Agarwal, Jens Fursund, and H. Li.  pagan: real-time avatars using dynamic textures.  ACM Trans. Graph., 2018.
  * [24] Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francesc Moreno-Noguer.  Ganimation: One-shot anatomically consistent facial animation.  International Journal of Computer Vision, 128(3):698–713, 2020\.
  * [25] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In IEEE Conf. Comput. Vis. Pattern Recog., 2015.
  * [26] Matan Sela, Elad Richardson, and Ron Kimmel.  Unrestricted facial geometry reconstruction using image-to-image translation.  In Int. Conf. Comput. Vis., 2017.
  * [27] Z. Shu, E. Yumer, S. Hadap, K. Sunkavalli, E. Shechtman, and D. Samaras.  Neural face editing with intrinsic image disentangling.  In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
  * [28] Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt.  Fml: Face model learning from videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10812–10822, 2019.
  * [29] Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt.  Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [30] Justus Thies, M. Elgharib, A. Tewari, C. Theobalt, and M. Nießner.  Neural voice puppetry: Audio-driven facial reenactment.  In ECCV, 2020.
  * [31] Justus Thies, M. Zollhöfer, and M. Nießner.  Deferred neural rendering.  ACM Transactions on Graphics (TOG), 2019.
  * [32] Luan Tran, Feng Liu, and Xiaoming Liu.  Towards high-fidelity nonlinear 3d face morphable model.  In In Proceeding of IEEE Computer Vision and Pattern Recognition, Long Beach, CA, June 2019.
  * [33] Luan Tran and Xiaoming Liu.  Nonlinear 3d face morphable model.  In IEEE Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, June 2018.
  * [34] Anh Tuấn Trần, Tal Hassner, Iacopo Masi, Eran Paz, Yuval Nirkin, and Gérard Medioni.  Extreme 3d face reconstruction: Seeing through occlusions.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [35] Job Van Der Schalk, Skyler T Hawk, Agneta H Fischer, and Bertjan Doosje.  Moving faces, looking places: validation of the amsterdam dynamic facial expression set (adfes).  Emotion, 11(4):907, 2011.
  * [36] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In Int. Conf. Comput. Vis., 2019.
  * [37] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Eur. Conf. Comput. Vis., 2018.
  * [38] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Int. Conf. Comput. Vis., 2017.
  * [39] Wenbin Zhu, HsiangTao Wu, Zeyu Chen, Noranart Vesdapunt, and Baoyuan Wang.  Reda:reinforced differentiable attribute for 3d face reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.