# High Order Impedance Boundary Condition for the Three-dimensional Scattering
Problem in Electromagnetism

OUESLATI Soumaya  Department of Mathematics
University CY Cergy Paris
Cergy-Pontoise, France
<EMAIL_ADDRESS>DAVEAU Christian  Department of Mathematics
University CY Cergy Paris
Cergy-Pontoise, France
<EMAIL_ADDRESS>AUBAKIROV Abil  Department of Mathematics
University CY Cergy Paris
Cergy-Pontoise, France
<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a variational formulation with the use of high order
impedance boundary condition (HOIBC) to solve the scattering problem. We study
the existence and uniqueness of the solution. Then, a discretization of this
formulation is done with Rao-Wilton-Glisson (RWG). We give validations of the
HOIBC obtained with a 3D MoM code that show the improvement in accuracy over
the standard impedance boundary condition (SIBC) computations.

###### Index Terms:

boundary element method, scattering problem, Lagrange multipliers, high order
impedance boundary condition.

##  I Introduction

We consider the scattering problem of electromagnetic waves
$(\textbf{E},\textbf{H})$ by a perfect conducting body with a complex coating.
Let $\Omega$ be a bounded domain with a Lipschitz-continuous boundary
$\Gamma$. Let $n$ be the unit normal vector to $\Gamma$ directed to the
exterior of $\Omega$. We define $\Omega_{+}$ as the space of radiating
electric fields E solutions of Maxwell equations in
$\mathbb{R}^{3}\backslash\overline{\Omega_{-}}$ which have a tangential trace
on $\Gamma$.

Waves propagate with constant wave number $k$ in the exterior unbounded domain
$\Omega$. The electric field E satisfies the harmonic Maxwell equation

$\nabla\times(\nabla\times\textbf{E})-k^{2}\textbf{E}=0$

While the related magnetic field is given by

$\textbf{H}=\frac{1}{ik}\nabla\times\textbf{E}$

An electric field is said to be radiating if it satisfies the Silver-Müller
radiation condition:

$\lim_{r\rightarrow\infty}r(\textbf{E}\times\textbf{n}_{r}+\textbf{H})=0,$

where $r=|\mathbf{x}|$ and $\mathbf{n}_{r}=\dfrac{\mathbf{x}}{|\mathbf{x}|}$,
$\mathbf{x}\in\mathbb{R}^{3}$.
The coating is modeled by the following impedance boundary condition

$\textbf{E}_{t}-Z(n\times\textbf{H})=0\,\,\,\,\,on\,\,\,\,\Gamma.$ (1)

Here, $Z$ is the impedance operator that depends on incident angle, subscript
tg denotes tangent component on the surface defined as:

$\textbf{E}_{t}=n\times(\textbf{E}\times n).$

Generally, we take the constant impedance operator, known as standard or
Leontovitch impedance boundary conditions [1, Senior_1995, 6]. Our problem
writes as follows:

$\left\\{\begin{array}[]{llll}\nabla\times(\nabla\times\textbf{E})-k^{2}\textbf{E}=0\;\Omega_{+},\\\
\nabla\times(\nabla\times\textbf{H})-k^{2}\textbf{H}=0\;\Omega_{+},\\\
\textbf{E}_{t}-Z(n\times\textbf{H})=0\,\,\,\,\,on\,\,\,\,\Gamma.\\\
\lim_{r\rightarrow\infty}r(\mathbf{E}\times\mathbf{n}_{r}+\mathbf{H})=0.\end{array}\right.$
(2)

This paper deals with the implementation of high order impedance boundary
condition coupled by an integral equation.

##  II high order impedance boundary condition

In this section, we propose a new impedance boundary condition with integral
operators. In [1] the authors approximated impedance condition as a ratio of
polynomials of second order of the sine of the incidence angle in spectral
domain. They obtained for the case of a rotational invariant coating a high
order impedance boundary condition in spatial domain equations. We transformed
these equations with integral operators to get:

$(I+b_{1}L_{D}-b_{2}L_{R})\textbf{E}_{t}=(a_{0}I+a_{1}L_{D}-a_{2}L_{R})(\mathbf{n}\times\textbf{H}).$
(3)

This approximation contains the operators $L_{D}$ and $L_{R}$ are defined: for
all vector function $\mathbf{A}$ sufficiently smooth, such that
$\mathbf{A}.\mathbf{n}=0$

$L_{D}(\mathbf{A})=\nabla_{\Gamma}(\mathrm{div}_{\Gamma}\mathbf{A}),\,\,\,\,\,\,\,\,L_{R}(\mathbf{A})=\mathbf{\mathrm{rot}}_{\Gamma}(\mathrm{rot}_{\Gamma}\mathbf{A}).$

In [4], different methods to calculate these coefficients
$(a_{0},a_{j},b_{j})$ for 3 are presented. In the following, we want to
establish a new variational formulation, applying a boundary integral method
for the problem in 3D case with HOIBC.

##  III Variational formulation with HOIBC

We introduce current densities J and M on the boundary $\Gamma$ as follows:

$\mathbf{M}=\mathbf{E}\times\mathbf{n}\ \ ,\
\mathbf{J}=\mathbf{n}\times\mathbf{H}.$

We finally obtain the variational formulation

###### Problem 1

Find $U=(\mathbf{J},\mathbf{M},\tilde{\mathbf{J}},\tilde{\mathbf{M}})\in
V=[H^{-1/2}(div,\Gamma)\cap L^{2}(\Gamma)]^{4}$ and
$\lambda=(\boldsymbol{\lambda}_{J},\boldsymbol{\lambda}_{M})\in[H^{-1/2}(\Gamma)]^{2}$
such that

$\begin{cases}A(U,\Psi)+B^{T}(\lambda,\Psi)=F(\Psi)\\\
B(U,\lambda^{\prime})=0\end{cases}$ (4)

for all
$\Psi=(\boldsymbol{\Psi}_{J},\boldsymbol{\Psi}_{M},\tilde{\boldsymbol{\Psi}}_{J},\tilde{\boldsymbol{\Psi}}_{M})\in
V=[H^{-1/2}(div,\Gamma)\cap L^{2}(\Gamma)]^{4}$ and
$\lambda^{\prime}=(\boldsymbol{\lambda}^{\prime}_{J},\boldsymbol{\lambda}^{\prime}_{M})\in
W=[H^{-1/2}(\Gamma)]^{2}$.

The bilinear forms are defined by:

$B(U,\lambda^{\prime})=\int_{\Gamma}\boldsymbol{\lambda}^{\prime}_{J}\cdot(\tilde{\mathbf{J}}-\mathbf{n}\times\mathbf{J})ds+\int_{\Gamma}\boldsymbol{\lambda}^{\prime}_{M}\cdot(\tilde{\mathbf{M}}-\mathbf{n}\times\mathbf{M})ds$

and

$A(U,\Psi)=iZ_{0}\iint_{\Gamma}kG\
(\textbf{J}\cdot\boldsymbol{\Psi}_{J})-\frac{1}{k}G\
\mathrm{div}{\boldsymbol{\Psi}_{J}}\ \mathrm{div}{\textbf{J}}dsds^{\prime}$
(5) $+\frac{i}{Z_{0}}\iint_{\Gamma}kG\
(\boldsymbol{\Psi}_{M}\cdot\textbf{M})-\frac{1}{k}G\
\mathrm{div}{\boldsymbol{\Psi}_{M}}\ \mathrm{div}{\textbf{M}}dsds^{\prime}$
$+\iint_{\Gamma}\nabla^{\prime}G\cdot(\boldsymbol{\Psi}_{J}\times\textbf{M})dsds^{\prime}-i\iint_{\Gamma}\nabla^{\prime}G\cdot(\boldsymbol{\Psi}_{M}\times\textbf{J})dsds^{\prime}$
$+\frac{a_{0}}{2}\int_{\Gamma}\textbf{J}\cdot\boldsymbol{\Psi}_{J}ds+\frac{1}{2a_{0}}\int_{\Gamma}\textbf{M}\cdot\boldsymbol{\Psi}_{M}ds$
$-\frac{a_{1}}{2}\int_{\Gamma}\mathrm{div}_{\Gamma}\textbf{J}\
\mathrm{div}_{\Gamma}\boldsymbol{\Psi}_{J}ds-\frac{a_{2}}{2}\int_{\Gamma}\mathrm{div}_{\Gamma}\tilde{\textbf{J}}\
\mathrm{div}_{\Gamma}\tilde{\boldsymbol{\Psi}}_{J}ds$
$+\frac{b_{1}}{2}\int_{\Gamma}\mathrm{div}_{\Gamma}\tilde{\textbf{M}}\
\mathrm{div}_{\Gamma}\boldsymbol{\Psi}_{J}ds-\frac{b_{2}}{2}\int_{\Gamma}\mathrm{div}_{\Gamma}\textbf{M}\
\mathrm{div}_{\Gamma}\tilde{\boldsymbol{\Psi}}_{J}ds$
$-\frac{b_{1}}{2a_{0}}\int_{\Gamma}\mathrm{div}_{\Gamma}\tilde{\textbf{M}}\
\mathrm{div}_{\Gamma}\tilde{\boldsymbol{\Psi}}_{M}ds-\frac{b_{2}}{2a_{0}}\int_{\Gamma}\mathrm{div}_{\Gamma}\textbf{M}\
\mathrm{div}_{\Gamma}\boldsymbol{\Psi}_{M}ds$
$+\frac{a_{1}}{2a_{0}}\int_{\Gamma}\mathrm{div}_{\Gamma}\textbf{J}\
\mathrm{div}_{\Gamma}\tilde{\boldsymbol{\Psi}}_{M}ds-\frac{a_{2}}{2a_{0}}\int_{\Gamma}\mathrm{div}_{\Gamma}\tilde{\textbf{J}}\
\mathrm{div}_{\Gamma}\boldsymbol{\Psi}_{M}ds,$

where $G(x,y)$ is the Green kernel.
We use a theorem from [7] to prove existence and uniqueness of a solution to
the problem 4. In the first time, we prove the following result.

###### Lemma 1

The operator $A$ is continuous on $V\times V$ for all $\Psi\in V$ and we have
that

$|A(U,\Psi)|\leq C\|U\|_{V}\|\Psi\|_{V}$

###### Lemma 2

The operator $A$ is coercive on $V$. We have to show that there exist
$\alpha>0$ such that

$\Re[A(U,U^{*})]\geq\alpha\|U\|^{2}_{V}-C\|U\|^{2}_{V},\ \forall U\in V.$

###### Lemma 3

The operator $B$ verifies the inequality

$\sup_{\|U\|_{V}=1}|B(U,\lambda)|\geq\beta\|\lambda\|_{W},$

$\forall\lambda\in W=[H^{-1/2}(\Gamma)]^{3}\times[H^{-1/2}(\Gamma)]^{3}$ where
$U\in V=[H^{-1/2}(div,\Gamma)\cap L^{2}(\Gamma)]^{4}$ and $\beta>0$.

Therefore, we have the result.

###### Theorem 1

The problem (4) admits a unique solution $U\in V=[H^{-1/2}(div,\Gamma)\cap
L^{2}(\Gamma)]^{4}$ and $\lambda\in[H^{-1/2}(\Gamma)]^{2}$, if coefficients
satisfy

$\Re(a_{j})+\frac{|a_{0}||b_{j}+a^{*}_{j}/a^{*}_{0}|}{2}=0\ \ for\ \ j=1,2.$
(6)

##  IV Discretization of the variational problem with HOIBC

The first step consists to approach the surface of the obstacle by a surface
$\Gamma_{h}$ composed of finite number of two dimensional elements. These
elements are triangular facets denoted by $T_{i}$ for $i=1$ to $N_{T}$. Let
$\\{\mathbf{f}_{i}\\}_{i=1,N_{e}}$ be a function of Rao-Wilton-Glisson
functions. We decompose the electric and magnetic currents with thes basis
functions. For the Lagrange multipliers, we use basis functions proposed in
[2] whith is defined as follows:

$\mathbf{g}_{n}(x)=\left\\{\begin{array}[]{cc}(1-2\omega^{+}_{i+2}(x))(\nu_{n}\times\mathbf{n}^{+}),\
\ \ \ x\in T^{+}_{n}\\\
(1-2\omega^{-}_{i+2}(x))(\nu_{n}\times\mathbf{n}^{-}),\ \ \ \ x\in
T^{-}_{n}\end{array}\right.$ (7)

with $\nu_{n}$ is a direction vector of the edge $\mathbf{n}$ and
$\\{\omega_{i}\\}_{i=1,3}$ are barycentric coordinates of $x$ relative to
triangles $T^{+}_{n}$ or $T^{+}_{n}$.

Then, the discretized problem of (4) is:

$\begin{cases}A^{h}(U_{h},\Psi_{h})+B^{h}(\lambda_{h},\Psi_{h})=\sum_{i=1}^{N_{e}}<\mathbf{E}^{inc},\mathbf{f}_{i}>+\\\
\sum_{i=1}^{N_{e}}<\mathbf{H}^{inc},\mathbf{f}_{i}>\\\
B^{h}(U_{h},\lambda_{h})=0\end{cases}$ (8)

where

$A^{h}(U_{h},\Psi_{h})=\sum^{N_{e}}_{i,j=1}<Z_{0}(B-S)\textbf{f}_{j},\mathbf{f}_{i}>J_{j}+\sum^{N_{e}}_{i,j=1}<Q\textbf{f}_{j},\mathbf{f}_{i}>M_{j}$
$+Z^{-1}_{0}\sum^{N_{e}}_{i,j=1}<(B-S)\textbf{f}_{j},\mathbf{f}_{i}>M_{j}-\sum^{N_{e}}_{i,j=1}<Q\textbf{f}_{j},\mathbf{f}_{i}>J_{j}+\frac{a_{0}}{2}$
$\sum^{N_{e}}_{i,j=1}<\textbf{f}_{j},\mathbf{f}_{i}>J_{j}+\frac{1}{2a_{0}}\sum^{N_{e}}_{i,j=1}<\textbf{f}_{j},\mathbf{f}_{i}>M_{j}-\frac{a_{1}}{2}$
$\sum^{N_{e}}_{i,j=1}<\mathbf{div}_{\Gamma}\textbf{f}_{j},\mathbf{div}_{\Gamma}\mathbf{f}_{i}>J_{j}-\frac{a_{2}}{2}\sum^{N_{e}}_{i,j=1}<\mathbf{div}_{\Gamma}\textbf{f}_{j},\mathbf{div}_{\Gamma}\mathbf{f}_{i}>\tilde{J}_{j}+\frac{b_{1}}{2}$
$\sum^{N_{e}}_{i,j=1}<\mathbf{div}_{\Gamma}\textbf{f}_{j},\mathbf{div}_{\Gamma}\mathbf{f}_{i}>\tilde{M}_{j}-\frac{b_{2}}{2}\sum^{N_{e}}_{i,j=1}<\mathbf{div}_{\Gamma}\textbf{f}_{j},\mathbf{div}_{\Gamma}\mathbf{f}_{i}>M_{j}-$
$\frac{b_{1}}{2a_{0}}\sum^{N_{e}}_{i,j=1}<\mathbf{div}_{\Gamma}\textbf{f}_{j},\mathbf{div}_{\Gamma}\mathbf{f}_{i}>\tilde{M}_{j}-\frac{b_{2}}{2a_{0}}\sum^{N_{e}}_{i,j=1}<\mathbf{div}_{\Gamma}\textbf{f}_{j},\mathbf{div}_{\Gamma}\mathbf{f}_{i}>M_{j}$
$+\frac{a_{1}}{2a_{0}}\sum^{N_{e}}_{i,j=1}<\mathbf{div}_{\Gamma}\textbf{f}_{j},\mathbf{div}_{\Gamma}\mathbf{f}_{i}>J_{j}-\frac{a_{2}}{2a_{0}}\sum^{N_{e}}_{i,j=1}<\mathbf{div}_{\Gamma}\textbf{f}_{j},\mathbf{div}_{\Gamma}\mathbf{f}_{i}>\tilde{J}_{j}$

and

$B^{h}(U_{h},\lambda^{\prime}_{h})=\sum^{N_{e}}_{i,k=1}<\mathbf{g}_{k},\mathbf{f}_{i}>\tilde{J}_{i}-\sum^{N_{e}}_{i,k=1}<\mathbf{g}_{k},\mathbf{n}\times\mathbf{f}_{i}>J_{i}$
$+\sum^{N_{e}}_{i,k=1}<\mathbf{g}_{k},\mathbf{f}_{i}>\tilde{M}_{i}-\sum^{N_{e}}_{i,k=1}<\mathbf{g}_{k},\mathbf{n}\times\mathbf{f}_{i}>M_{i}$

In the second step, we express the problem in matrix form

$(B-S)_{i,j}=i\int\int_{\Gamma_{h}}kG(s,s^{\prime})\mathbf{f}_{j}(s^{\prime})\cdot\mathbf{f}_{i}(s)-\frac{1}{k}$
$G(s,s^{\prime})(\mathbf{div}_{\Gamma}\mathbf{f}_{i})(\mathbf{div}_{\Gamma}^{\prime}\mathbf{f}_{j})dsds^{\prime}$
$Q_{i,j}=-i\int\int_{\Gamma_{h}}[\mathbf{f}_{i}(s)\times\mathbf{f}_{j}(s^{\prime})]\cdot\nabla^{\prime}_{\Gamma}G(s,s^{\prime})dsds^{\prime}$
$I_{i,j}=\int_{\Gamma_{h}}\mathbf{f}_{i}\cdot\mathbf{f}_{j}ds;\ \ \ \
D_{i,j}=\int_{\Gamma_{h}}(\mathbf{div}_{\Gamma}\mathbf{f}_{j})(\mathbf{div}_{\Gamma}\mathbf{f}_{i})ds$
$C_{Hi,j}=\int_{\Gamma_{h}}\mathbf{g}_{i}\cdot\mathbf{f}_{j}ds;\ \ \ \
C_{Ki,j}=\int_{\Gamma_{h}}\mathbf{g}_{i}\cdot(\mathbf{n}\times\mathbf{f}_{j})ds,$
(9)

where $[C_{H}]$ is a non singular diagonal matrix. Therefore it is invertible.
Then, we define $[A1]$ and $[A2]$ by

$[A1]=[(B-S)]+\frac{a_{0}}{2}[I]-\frac{a_{1}}{2}[D],$ (10)
$[A2]=[(B-S)]+\frac{1}{2a_{0}}[I]-\frac{b_{2}}{2a_{0}}[D].$ (11)

Using (9), (10) and (11), we now present (8) in the following matrix form

$\left(\begin{array}[]{cccccc}[A1]&[Q]&0&\frac{b_{1}}{2}[D]&[C_{K}]^{T}&0\\\
{[Q]}^{T}&[A2]&-\frac{a_{2}}{2a_{0}}[D]&0&0&[C_{K}]^{T}\\\
0&-\frac{b_{2}}{2}[D]&-\frac{a_{2}}{2}[D]&0&[C_{H}]^{T}&0\\\
\frac{a_{1}}{2a_{0}}[D]&0&0&-\frac{b_{1}}{2a_{0}}[D]&0&[C_{H}]^{T}\\\
{[C_{K}]}&0&[C_{H}]&0&0&0\\\ 0&[C_{K}]&0&[C_{H}]&0&0\end{array}\right)$ (12)

##  V Elimination of auxiliary currents and the Lagrange multipliers

Now, we focus ours attention on elimination auxiliary currents and the
Lagrange multiplier. Using the definition of the basis function (7), we obtain
that

$\int_{\Gamma}\mathbf{g}_{n}(s)\cdot\tilde{\mathbf{J}}ds=\frac{|T^{+}_{n}|+|T^{-}_{n}|}{3}\tilde{J}_{n}.$

On the other side, we have

$\int_{\Gamma}\mathbf{g}_{n}(s)\cdot(\mathbf{n}\times\mathbf{J})ds=\int_{T^{+}_{n}\cup
T^{-}_{n}}\mathbf{g}_{n}(s)\cdot(\mathbf{n}\times\mathbf{J})ds,$

that is calculated with help of Gaussian quadrature. And we get next equation

$\frac{|T^{+}_{n}|+|T^{-}_{n}|}{3}\tilde{J}_{n}=\int_{T^{+}_{n}\cup
T^{-}_{n}}\mathbf{g}_{n}(s)\cdot(\mathbf{n}\times\mathbf{J})ds$ (13)

Thus, we obtain

$\tilde{J}_{n}=\frac{3}{|T^{+}_{n}|+|T^{-}_{n}|}\int_{T^{+}_{n}\cup
T^{-}_{n}}\mathbf{g}_{n}(s)\cdot(\mathbf{n}\times\mathbf{J})ds$ (14)

on each edge of the mesh. Consequently, we conclude that the auxiliary
currents can be rewritten as

$\begin{array}[]{cc}\overline{\tilde{J}}=-[C_{H}]^{-1}[C_{K}]\overline{J}\,\,\,\,\text{and}\,\,\,\,\overline{\tilde{M}}=-[C_{H}]^{-1}[C_{K}]\overline{M}\end{array}$
(15)

So (12) together with (15) yields the following expression of Lagrange
multipliers in terms of $\overline{J}$ and $\overline{M}$

$\overline{\lambda_{J}}=\frac{b_{2}}{2}[C_{H}]^{-T}[D]\overline{M}-\frac{a_{2}}{2}[C_{H}]^{-T}[D][C_{H}]^{-1}[C_{K}]\overline{J}$
(16)
$\overline{\lambda_{M}}=-\frac{a_{2}}{2a_{0}}[C_{H}]^{-T}[D]\overline{J}-\frac{b_{1}}{2a_{0}}[C_{H}]^{-T}[D][C_{H}]^{-1}[C_{K}]\overline{M}$
(17)

However, the discretization leads to dense linear systems.

The accurate computation of such matrix elements is an important. [8] have
been the evaluation of singular integrals.

##  VI  Parallel BEM with H-matrices using MPI and OpenMP

Because a dense coefficient HOIBC matrix described in the previous section,
the total computational cost and the memory footprint of the method are very
high. To remedy these problems, we developed a software framework that is
based in parallel Hierarchical Matrices (H-matrices) with ACA (Adaptive Cross
Approximation) [M_bebendorf] [9].

Figure 1: -matrix partition of a sphere mesh with 1434 edges.

In Figure 1, the matrix size is 2868x2868. Admissible blocks are shown in
green, non-admissible blocks in red.
H-matrix computing can be conveniently coupled with parallel computing
technologies to distribute the memory load, and to accelerate not only the
matrix filling but also the classical matrix/vector product in order to obtain
fast iterative solvers. In next section, We present numerical results which
show the effectiveness of both the parallelization and the H-matrix method.

##  VII Numerical results

We present now some numerical results obtained in 3D. As a first example, we
consider the case of a coated conducting sphere with a conductor radius of
$1.5\lambda_{0}$ and coating thickness of $0.0075\lambda_{0}$, with
$\epsilon_{r}=5$ and $\mu_{r}=1.0$. Figure 2 show the $\theta\theta$
components of the bistatic RCS for a plane wave incident from $\theta=0$.
Three solutions are included: the exact serie solution (MIE) and the solutions
of the methods of moment studied below with SIBC and the HOIBC. The Figure
clearly shows the increased accuracy of the HOIBC solution relative to the
SIBC solution. The SIBC gives only the average behavior of the scattered field
while the HOIBC accurately predicts the sidelobe behavior.

Figure 2: $\theta\theta$ component of the bistatic RCS for a coated conducting
sphere. Exact serie solution and HOIBC solution.

The second test is a coated conducting spheroid whose the radii are $0.5m$ and
$1m$ with a coating thickness of $0.17\lambda_{0}$, with $\epsilon_{r}=5$ and
$\mu_{r}=1.0$. Figures 3 and 4 show the $\theta\theta$ and $\phi\phi$
components of the monostatic RCS. Three solutions are included : a method of
moments solution called PMCHWT, the HOIBC solution and the SIBC solution. It
does not exits exact solution for this case.

We note that a slight difference between the PMCHWT solution and the HOIBC
solution and the SIBC solution is very poor. In both cases, excellent results
for HOIBC method are obtained for all angles of incidence.

Figure 3: $\theta\theta$ component of the monostatic RCS for a coated
conducting spheroid, PMCHWT solution, SIBC solution and HOIBC solution. Figure
4: $\phi\phi$ component of the monostatic RCS for a coated conducting
spheroid, PMCHWT solution, SIBC solution and HOIBC solution.

As a final example a coated conducting almond is considered whose the radii
are 0.25m and 2m with a coating thickness of $0.03$m, with $\epsilon_{r}=4$
and $\mu_{r}=1.0$. Figures 5 show the $\theta\theta$ component of the bistatic
RCS. Two solutions are included : the HOIBC solution and the SIBC solution.

Figure 5: $\theta\theta$ component of the bistatic RCS for a coated conducting
almond. Exact serie solution and HOIBC solution.

##  VIII conclusion

The validations are shown an important accuracy improvement of the HOIBC model
over the SIBC model in the case of materials of lower index and/or with small
losses.

## References

  * [1] Y. Rahmat-Samii and D. J. Hoppe, Impedance boundary conditions in electromagnetics, _Taylor & Francis_, (1995).
  * [2] A. Bendali, M. Fares and J. Gay, _A Boundary-Element Solution of the Leontovich Problem_ , IEEE Trans. Antennas Propagat., Vol. 47, pp. 1597–1605, October, (1999).
  * [3] M. Bebendorf, _Approximation of boundary element matrices_ , Numerische Mathematik, vol. 86, pp. 565–589 (2000).
  * [4] A. Aubakirov, Electromagnetic Scattering Problem with Higher Order Impedance Boundary Conditions and Integral Methods, UCP, (2014).
  * [5] T. B. A. Senior and J. L. Volakis, _Approximate boundary conditions in electromagnetics_ , Inst. Elect. Eng. Electromagn. Waves Series 41, vol. 10, n. 4, pp. 189-209 (1995).
  * [6] V. Lange,_Equations intégrales espace-temps pour les équations de Maxwell : calcul du champ diffracté par un obstacle dissipatif_ , PHD thesis Mathématiques appliquées, Bordeaux, 1, (1995).
  * [7] J. C. Nédélec, _Acoustic and electromagnetic equations: Integral representations for harmonic problems_ , Springer édition, Vol. 144, (2000).
  * [8] S. Oueslati and I. Balloumi and C. Daveau and A. Khelifi, _Analytical method for the evaluation of singular integrals arising from boundary element method in electromagnetism_ , International Journal of Numerical Modelling, Vol. 34 (2020).
  * [9] S. Oueslati, C. Daveau, B. Naisseline, _High order impedance boundary condition for the three scattering problem in electromagnetism with the adaptive cross approximation_ , Waves, Vienna, Austria , (2019).


# Single-Shot Implicit Morphable Faces with Consistent Texture
Parameterization

Connor Z. Lin  Stanford UniversityUSA NVIDIAUSA ,  Koki Nagano  NVIDIAUSA ,
Jan Kautz  NVIDIAUSA ,  Eric R. Chan  Stanford UniversityUSA NVIDIAUSA ,  Umar
Iqbal  NVIDIAUSA ,  Leonidas Guibas  Stanford UniversityUSA ,  Gordon
Wetzstein  Stanford UniversityUSA and  Sameh Khamis  NVIDIAUSA

(2023)

###### Abstract.

There is a growing demand for the accessible creation of high-quality 3D
avatars that are animatable and customizable. Although 3D morphable models
provide intuitive control for editing and animation, and robustness for
single-view face reconstruction, they cannot easily capture geometric and
appearance details. Methods based on neural implicit representations, such as
signed distance functions (SDF) or neural radiance fields, approach photo-
realism, but are difficult to animate and do not generalize well to unseen
data. To tackle this problem, we propose a novel method for constructing
implicit 3D morphable face models that are both generalizable and intuitive
for editing. Trained from a collection of high-quality 3D scans, our face
model is parameterized by geometry, expression, and texture latent codes with
a learned SDF and explicit UV texture parameterization. Once trained, we can
reconstruct an avatar from a single in-the-wild image by leveraging the
learned prior to project the image into the latent space of our model. Our
implicit morphable face models can be used to render an avatar from novel
views, animate facial expressions by modifying expression codes, and edit
textures by directly painting on the learned UV-texture maps. We demonstrate
quantitatively and qualitatively that our method improves upon photo-realism,
geometry, and expression accuracy compared to state-of-the-art methods.

Neural Avatars, Implicit Representations, Texture Maps, Animation, Inversion

††journal: TOG††journalyear: 2023††copyright: rightsretained††conference:
Special Interest Group on Computer Graphics and Interactive Techniques
Conference Conference Proceedings; Aug 6–10; 2023††booktitle: Special Interest
Group on Computer Graphics and Interactive Techniques Conference Conference
Proceedings (SIGGRAPH ’23 Conference Proceedings), August 6–10, 2023, Los
Angeles, CA, USA††doi: 10.1145/3588432.3591494††isbn:
979-8-4007-0159-7/23/08††ccs: Computing methodologies Modeling/Geometry Figure
1. Given a single input image, our method reconstructs a high-quality editable
3D digital avatar (columns 2 and 3) by combining implicit geometry
representations with explicit texture maps. The proposed approach naturally
supports novel view synthesis from large pose shifts, an expressive and non-
linear facial animation space (columns 4 through 6), direct user access to
texture map editing (column 7), and 3D asset extraction for further downstream
applications such as relighting (column 8). Original image courtesy of COD
Newsroom/flickr (top) and Malcolm Slaney/flickr (bottom).

##  1\. Introduction

Personalized avatar creation—the ability to map one’s facial features to a 3D
virtual replica that can be animated, customized, and rendered—is an emerging
technology with great promise for cinema, the metaverse, and telepresence.
Advances in this area may lead to digital twins with greater verisimilitude in
detail and in animation that are more easily integrated into downstream
applications and pipelines. Single-shot personalized avatar creation enables
reconstructing face avatars from individual RGB images with greater
convenience and flexibility than methods that require more specialized capture
setups or procedures.

Traditional approaches to animatable 3D avatar creation are often based on 3D
Morphable Models (3DMM) (Blanz and Vetter, 1999), which disentangle shape and
appearance variation into a low-dimensional face representation. Building on
these, more recent approaches often leverage either explicit (textured)
template meshes (Feng et al., 2021; Daněček et al., 2022; Li et al., 2017;
Grassal et al., 2022; Khakhulin et al., 2022; Tran and Liu, 2019) or neural
implicit representations (Park et al., 2019; Sitzmann et al., 2019; Mildenhall
et al., 2021). Template-based approaches enable easy asset extraction and
intuitive editing, but are often unable to capture high-quality geometry and
textures. Emerging implicit face models can achieve greater realism by
modeling more complex geometric features such as hair (Zheng et al., 2022a;
Cao et al., 2022b; Giebenhain et al., 2022). However, implicit face
representations often compromise on interpretability and are less intuitive to
control; the entangled latent spaces learned by these highly parameterized
models are difficult to edit.

Our approach aims to combine the interpretability and editability advantages
of template-based 3DMMs with the quality and topological flexibility of
implicit 3D representations. Crucially, we decouple appearance and geometry
into two branches of our network architecture. By incorporating a UV
parameterization network to learn continuous and consistent texture maps, we
can export avatars as textured meshes to support downstream applications such
as texture map editing and relighting in a traditional graphics pipeline (See
Figure 1). On the other hand, by representing geometry with an implicit signed
distance field (SDF), our facial shape is less limited by resolution and
topology compared to mesh-based approaches.

We show that our proposed hybrid representation effectively captures the
geometry, appearance, and expression space of faces. We demonstrate that
single-shot in-the-wild portrait images can be effectively mapped to avatars
based on our proposed representation, and that these avatars improve upon the
previous state-of-the-art in photo-realism, geometry, and monocular expression
transfer. Moreover, we demonstrate compelling capability for enabling direct
texture editing and disentangled attribute editing such as facial geometry and
appearance attributes.

In summary, contributions of our work include:

  * •

We propose a hybrid morphable face model combining the high-quality geometry
and flexible topology of implicit representations with the editability of
explicit UV texture maps.

  * •

We present a single-shot inversion framework to map a single in-the-wild RGB
image to our implicit 3D morphable model representation. The inverted avatar
supports novel view rendering, non-linear facial reanimation, disentangled
shape and appearance control, direct texture map editing, and textured mesh
extraction for downstream applications.

  * •

We demonstrate state-of-the-art reconstruction accuracy for photo-realistic
rendering, geometry, and expression accuracy in the single-view reconstruction
setting.

##  2\. Related Work

Table 1. Comparison to recent prior work. To the best of our knowledge, our
method is the first implicit 3D face model to generalize across single-image
inputs while supporting flexible topology and explicit texture map control.

|  Generalizable |  Single-Image |  Implicit
Representation |  Explicit Texture Control
---|---|---|---|---
EMOCA (2022) |  ✓ |  ✓ |  ✗ |  ✓
ROME (2022) |  ✓ |  ✓ |  ✗ |  ✗
Neural Parametric Head Models (2022) |  ✗ |  ✗ |  ✓ |  ✗
IM-Avatar (2022a) |  ✗ |  ✗ |  ✓ |  ✗
Neural Head Avatars (2022) |  ✗ |  ✗ |  ✓ |  ✓
Volumetric Avatars from a Phone Scan (2022b) |  ✓ |  ✗ |  ✓ |  ✓
HeadNeRF (2022) |  ✓ |  ✓ |  ✓ |  ✗
Ours |  ✓ |  ✓ |  ✓ |  ✓

###  2.1. Mesh-based 3D Morphable Models

The seminal work by Blanz and Vetter proposed a linear 3D Morphable Model
(3DMM) (Blanz and Vetter, 1999) that models facial shape and textures on a
template mesh using linear subspaces computed by principal component analysis
(PCA) from 200 facial scans. This low-dimensional facial shape and texture
space makes 3DMMs suitable for robustly capturing facial animation as well as
reconstructing 3D faces in monocular settings. To reconstruct shape, texture,
and lighting from a photo, previous work employed continuous optimization
using constraints such as facial landmarks and pixel colors (Romdhani and
Vetter, 2005; Garrido et al., 2013; Shi et al., 2014; Cao et al., 2014; Ichim
et al., 2015; Thies et al., 2016; Cao et al., 2016; Garrido et al., 2016; Li
et al., 2017) and more recently deep learning-based inference (Dou et al.,
2017; Tuan Tran et al., 2017; Tewari et al., 2017; Genova et al., 2018; Wu et
al., 2019; Tewari et al., 2019; Deng et al., 2019b; Feng et al., 2021; Daněček
et al., 2022; Luo et al., 2021; B R et al., 2021; Dib et al., 2021b, a). While
approaches relying on 3DMMs tend to be robust, they are ineffective for
reconstructing high-fidelity geometry and texture details due to the linearity
and low dimensionality of the model. Various other methods extended 3DMMs to
capture non-linear shapes (Chandran et al., 2020; Tran and Liu, 2018; Tran et
al., 2019; Tran and Liu, 2019; Wang et al., 2022b; Li et al., 2020; Tewari et
al., 2018), photo-realistic appearance using neural rendering or optimization
(Thies et al., 2019; Nagano et al., 2018; Gecer et al., 2019; Saito et al.,
2017), or reflectance and geometry details for relightable avatar generation
(Lattas et al., 2020; Yamaguchi et al., 2018; Huynh et al., 2018; Chen et al.,
2019). Recent approaches predict geometry offsets over the template mesh to
reconstruct non-facial regions such as hair (Grassal et al., 2022; Khakhulin
et al., 2022). We refer the reader to Egger et al. (2020) for an in-depth
survey of 3DMM techniques and Tewari et al. (2022) for a report of recent
advancements in neural rendering.

Since mesh-based 3DMMs represent geometry with a shared template mesh, their
fixed topology limits the ability to scale the model to capture complex
geometry such hair or fine-scale details. Additionally, their ability to
synthesize photo-realistic facial textures may be limited by the resolution of
the template mesh and discrete texture map. By parameterizing geometry with a
signed distance function and color with a continuous texture map, our method
is able to avoid such resolution issues and scale more efficiently with model
capacity while retaining 3DMM-like intuitive parameters to individually
control geometry and textures. Our consistent texture parameterization enables
not only direct texture editing in UV space, but also semantic correspondence
between our face model and an input image via facial landmarks, which can be
leveraged to improve single-shot reconstruction quality.

Figure 2. Our Pipeline. Avatars are represented by geometry, expression, and
color latent codes $\\{w_{geom},w_{expr},w_{color}\\}$ with each being 512
dimensional. At each 3D coordinate $p$ during sphere tracing, the SDF network
$f$ and UV parameterization network $g$ are conditioned on $w_{geom}$,
$w_{expr}$, and positional encoding $PE(p)$ to predict the signed distance
$SDF(p)$ and UV coordinates $UV(p)$, respectively. The inverse UV
parameterization network $g^{-1}$ regularizes the learned mapping to be a
surface parameterization $g^{-1}(UV(p);w_{geom},w_{expr})=p$, while the color
network $h$ predicts the associated RGB texture
$RGB(p)=h(UV(p);w_{color},w_{expr})$. After training, the avatar can be
rendered freely with direct control over its texture and facial expression, or
extracted as a stand-alone textured mesh asset.

###  2.2. Implicit Representations for Modeling and Rendering

While single-shot 3D reconstruction methods have explored various explicit 3D
representations such as voxels (Yan et al., 2016; Girdhar et al., 2016; Zhu et
al., 2017; Tulsiani et al., 2017; Wu et al., 2018; Yang et al., 2018), point
clouds (Fan et al., 2017), meshes (Xu et al., 2019), geometric primitives (Zou
et al., 2017; Niu et al., 2018), and depth maps (Wu et al., 2020), implicit
representations have recently been leveraged to achieve higher resolution
reconstruction using occupancy or signed distance fields (SDFs) (Xu et al.,
2019; Mescheder et al., 2019; Chen and Zhang, 2019). Implicit representations
such as neural radiance fields (NeRFs) (Mildenhall et al., 2021) and signed
distance fields (SDFs) (Park et al., 2019) have demonstrated high
reconstruction quality for 3D shapes and volumetric scenes. PIFu (Saito et
al., 2019) and follow-up works (Cao et al., 2022a; Saito et al., 2020) use
implicit fields to model human bodies and clothing. AtlasNet (Groueix et al.,
2018) demonstrated 3D shape generation by predicting a set of parametric
surface elements given an input image or point cloud. NeuTex (Xiang et al.,
2021) replaces the radiance prediction of NeRFs with a learned UV texture
parameterization conditioned on lighting direction. Although our method also
employs a UV cycle consistency loss, we 1) operate in a SDF setting and
condition our parameterization on geometry and expression latent codes to
generalize across samples rather than overfit to a single scene, 2) employ
sparse facial landmark constraints to facilitate learning a semantically
intuitive and consistent parameterization, and 3) explicitly leverage 2D to 3D
facial landmark correspondences enabled by the learned consistent
parameterization during single-image reconstruction. Implicit representations
have also given rise to higher quality 3D generative models (Chan et al.,
2022; Or-El et al., 2022; Xue et al., 2022), and follow-up work has studied
inverting an image into the latent space of a pre-trained 3D GAN (Roich et
al., 2022; Lin et al., 2022; Ko et al., 2023) for single-view 3D
reconstruction. However, without careful optimization and additional priors
(Xie et al., 2022; Yin et al., 2022), this 3D GAN inversion tends to be less
robust due to unknown camera poses (Ko et al., 2023) and multi-view nature of
NeRF training in the monocular setting. On the other hand, the compact face
representation of our model provides robust initialization in the single-shot
reconstruction setting.

###  2.3. Implicit Face Models

Compared to traditional mesh-based 3DMMs for face modeling, implicit
representations naturally offer flexible topology and non-linear expression
animation through latent code conditioning. While some approaches learn to
reconstruct an implicit 3DMM from an input 3D face scan (Zheng et al., 2022b;
Yenamandra et al., 2021; Alldieck et al., 2021; Zanfir et al., 2022; Cao et
al., 2022b; Giebenhain et al., 2022), other works have explored modeling an
implicit face model from RGB videos (Zheng et al., 2022a; Grassal et al.,
2022; Zheng et al., 2022c; Ma et al., 2022). However, the above approaches
either do not support or cannot generalize to single-shot in-the-wild images.
Multi-view methods have also been used to reconstruct implicit head models
(Kellnhofer et al., 2021; Athar et al., 2022; Hong et al., 2022; Athar et al.,
2021; Li et al., 2022; Ramon et al., 2021; Wang et al., 2022a). HeadNeRF (Hong
et al., 2022) is the closest to our work and learns a parametric head model
from multi-view images during training; at test-time, an input image can be
inverted for 3D reconstruction. However, HeadNeRF performs volumetric
rendering at a limited image resolution and relies on upsampling CNN modules,
resulting in flickering artifacts from depth error during novel view
synthesis. Furthermore, existing implicit morphable models do not support
texture manipulation beyond interpolation; by contrast, our learned explicit
texture paramterization enables intuitive and out-of-domain edits such as
adding tattoos or mustaches (see Fig. 1).

##  3\. Method

###  3.1. Implicit Morphable Face Parameterization

We disentangle each facial avatar into identity and expression, where identity
is encoded by geometry and color latent codes while expression is captured by
an expression latent code. To attain both high-quality geometry and
interpretable texture, our model consists of an implicit geometry branch and a
UV texture parameterization branch. The geometry branch contains a multilayer
perceptron (MLP) that maps 3D points $p$ to SDF values $SDF(p)$ during sphere
tracing. The UV texture branch consists of a parameterization MLP that maps
$p$ to spherical coordinates $UV(p)$, a parameterization regularizer MLP that
learns the inverse mapping from $UV(p)$ back to $p$, and a color network that
predicts the output RGB at $UV(p)$. See Figure 2 for a diagram of our model
pipeline. Please refer to the supplement for model architecture details.

We train our model on the Triplegangers (2022) 3D scan dataset for its volume
and diversity of subjects and expressions. Although the RenderPeople (2022)
dataset additionally models hair and clothing, it only contains 120 neutral
expression subjects, making it less suitable for reconstructing an avatar from
unconstrained in-the-wild photos. Our training samples consist of a 3D head
mesh, UV diffuse texture map, and six diffusely lit frontal RGB images. The
dataset contains 515 different subjects each with 20 expressions, for a total
of 10,300 data samples. Our full model learns an AutoDecoder dictionary of 515
geometry codes, 515 color codes, and 10,300 expression codes, as subjects
express the same sentiment differently. Different expressions for the same
training subject share the same geometry and color codes, allowing the model
to disentangle expression from the underlying geometry and texture. Please
refer to the supplement for examples of our training data.

###  3.2. Training Losses

Our model is trained on geometry, color, and regularization losses:

(1) $\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{geom}+\mathcal{L}_{color}+\mathcal{L}_{reg}$

Following Figure 2, let $f$ be the SDF MLP, $g$ the UV parameterization MLP,
$g^{-1}$ the inverse UV parameterization MLP, and $X$ the set of randomly
sampled surface points during training. The geometry loss consists of surface,
Eikonal (Gropp et al., 2020), normal, and UV losses. The surface loss
$\ell_{surf}$ optimizes the SDF zero level set, the Eikonal loss
$\ell_{eikonal}$ regularizes the SDF gradients, and the normal loss
$\ell_{normal}$ aligns the SDF gradients with the ground truth mesh normals
$\hat{n}$. The UV loss $\ell_{uv}$ regularizes the learned mapping to follow
an invertible surface parameterization, which enables correspondences between
texture and geometry used in our single-shot inversion pipeline, described in
Section 3.5.

(2) $\displaystyle\ell_{surf}$ $\displaystyle=\frac{1}{|X|}\sum_{x\in
X}|f(x)|$ (3) $\displaystyle\ell_{eikonal}$
$\displaystyle=\mathbb{E}_{x}(\|\nabla_{x}f(x)\|-1)^{2}$ (4)
$\displaystyle\ell_{normal}$ $\displaystyle=\frac{1}{|X|}\sum_{x\in
X}\|\nabla_{x}f(x)-\hat{n}(x)\|^{2}$ (5) $\displaystyle\ell_{uv}$
$\displaystyle=\frac{1}{|X|}\sum_{x\in X}\|x-g^{-1}(g(x))\|^{2}$ (6)
$\displaystyle\mathcal{L}_{geom}$
$\displaystyle=\ell_{surf}+\ell_{eikonal}+\ell_{normal}+\ell_{uv}$

The color loss consists of a reconstruction loss $\ell_{tex}$ on the ground
truth texture $\hat{T}$, as well as perceptual (Zhang et al., 2018) and
reconstruction losses $\ell_{img}$ over the facial region $I_{face}$ between
the ground truth image $\hat{I}$ and rendered image $I$ obtained via sphere
tracing:

(7) $\displaystyle\ell_{tex}$ $\displaystyle=\frac{1}{|X|}\sum_{x\in
X}\|\hat{T}(x)-h(g(x))\|^{2}$ (8) $\displaystyle\ell_{img}$
$\displaystyle=LPIPS(\hat{I}_{face},I_{face})+\|\hat{I}_{face}-I_{face}\|^{2}$
(9) $\displaystyle\mathcal{L}_{color}$ $\displaystyle=\ell_{tex}+\ell_{img}$

Finally, we enforce the compactness in the learned latent space by penalizing
the magnitude of the geometry, color, and expression codes:

(10) $\mathcal{L}_{reg}=\|w_{geom}\|^{2}+\|w_{color}\|^{2}+\|w_{expr}\|^{2}$

Figure 3. Single-shot inversion pipeline. We de-light the input image and
initialize the latent codes using a pre-trained encoder (top row). We then
perform PTI (Roich et al., 2022) to get the final reconstruction (bottom row).
Original image courtesy of Brett Jordan/flickr.

###  3.3. Learning UV Parameterizations

To learn an interpretable texture space and coherent semantic correspondence
across subjects, we add an auxiliary loss term to $\mathcal{L}_{reg}$ that
enforces the parameterization to be consistent through a sparse set of facial
landmark constraints:

(11) $\displaystyle\ell_{landmark}=\frac{1}{|L|}\sum_{x\in
L}\|\hat{g}(x)-g(x)\|^{2}+\|x-g^{-1}(g(x))\|^{2}$

The first term enforces the learned UV mapping to match the ground truth UV
mapping $\hat{g}$ for the set of 3D facial landmark points $L$, and the second
term enforces this mapping to be invertible. Fig. 8 demonstrates the
consistency of our learned UV parameterization. Although mostly consistent, it
is difficult to obtain perfect registrations around the inner mouth and eyes
due to the billboard geometry and errors originating from the ground truth
data.

###  3.4. Animation

After training, an avatar can be animated by manipulating its expression
latent code. For a source subject with expression code $w_{expr}$, target
expression code $w_{expr}^{\prime}$, and animation timesteps $t\in[0,1]$, we
define the expression animation trajectory by:

(12) $w_{expr}(t)=w_{expr}+t*(w_{expr}^{\prime}-w_{expr})$

Unlike traditional linear 3DMM approaches, our expression space follows non-
linear trajectories learned from high-quality 3D scans, as shown in Fig. 4.

Figure 4. Non-linear animation space. By linearly interpolating between source
and target expression codes, our model exhibits non-linear deformation
trajectories on the 3D mouth vertices visualized. Original image courtesy of
David Shankbone/flickr.

###  3.5. Single-Shot Inversion

In order to reconstruct and animate unseen subjects, we project an input RGB
image into the latent space of our pre-trained model and lightly fine-tune the
model weights similar to Pivotal Tuning Inversion (PTI) (Roich et al., 2022).
To handle unseen lighting conditions, we de-light the input image using LUMOS
(Yeh et al., 2022) and initialize the geometry, color, and expression codes
through a separately trained encoder. We empirically find this encoder
initialization to be important in obtaining robust results for in-the-wild
input images (See Figure 9).

#### Image Encoder.

We attain latent code initializations by training a DeepLabV3+ (Chen et al.,
2018) encoder to reconstruct each training image $\hat{I}$ and its
corresponding latent codes $\hat{W}$ already computed from the previous
AutoDecoder training stage:

(13) $\displaystyle\mathcal{L}_{enc}$
$\displaystyle=\|\hat{I}-I\|^{2}+\|\hat{W}-W\|^{2}$ (14) $\displaystyle W$
$\displaystyle=[w_{geom};w_{color};w_{expr}]$

One major challenge when inverting in-the-wild images is handling unseen
identities, accessories, hairstyles, and occlusion present in real-world
images, as Triplegangers contain limited identities with no variations in
hairstyles or background. Therefore, we augment the encoder’s training dataset
with synthetically augmented Triplegangers images from (Yeh et al., 2022),
which improves the robustness of the initialization and final inversion
reconstruction, shown in Fig. 9.

#### Optimization.

After initializing the latent codes for an input image $\hat{I}$ using our
encoder, we freeze the model weights and optimize the latent codes while
minimizing image, silhouette, multi-view consistency, facial landmark, and
regularization losses:

(15) $\displaystyle\ell_{img}$
$\displaystyle=LPIPS(\hat{I}_{face},I_{face})+\|\hat{I}_{face}-I_{face}\|^{2}$
(16) $\displaystyle\ell_{silhouette}$
$\displaystyle=\\!\\!\\!\sum_{x\in\hat{I}_{face}\wedge x\not\in
I_{face}}\\!\\!\\!f(x)$ (17) $\displaystyle\ell_{ID}$
$\displaystyle=ArcFace(\hat{I},I,I_{rand})$ (18)
$\displaystyle\ell_{landmark}$ $\displaystyle=\sum_{d\in
D(\hat{I})}\|d-proj_{2D}(g^{-1}(\hat{d}))\|^{2}$ (19)
$\displaystyle\ell_{reg}$
$\displaystyle=\|w_{geom}\|^{2}+\|w_{color}\|^{2}+\|w_{expr}\|^{2}$

where the silhouette loss $\ell_{silhouette}$ iterates over points contained
in the ground truth face region $\hat{I}_{face}$, but not in the predicted
face region $I_{face}$, to bring the points closer to the SDF zero level set.
ArcFace (Deng et al., 2019a) measures the face similarity between different
views and $I_{rand}$ is a predicted render from a randomly perturbed camera
pose. $D$ is an off-the-shelf facial landmark detector (King, 2009) and
$\hat{d}$ is the ground truth facial landmark UV mapping enforced in Eq. 11.
Note that our consistent UV parameterization directly enables correspondences
for the facial landmark alignment loss $\ell_{landmark}$; Fig. 10 demonstrates
the benefits of incorporating this loss. The regularization loss $\ell_{reg}$
is important to ensure that the optimized codes stay near the manifold of the
pre-trained latent space for expression animation. We obtain face masks using
a pre-trained BiSeNet (Yu et al., 2018) and optimize for 800 steps.

#### Fine-tuning.

To reconstruct finer details in the input image, we freeze the latent codes
after optimization and fine-tune the model weights on the above losses. We
omit the silhouette loss, as we find it tends to bloat the geometry when the
model weights are unfrozen. Although fine-tuning the model improves
reconstruction quality, it may also hinder its capability for animation or
novel view synthesis. Therefore, we only perform model fine-tuning for 60
steps.

##  4\. Results

Figure 5. Single-shot reconstruction on FFHQ with expression and pose
transfer. On the left, we show the input FFHQ source image, de-lit input image
using LUMOS (Yeh et al., 2022), and reconstruction results for each method. On
the right, we show monocular performance capture and retargeting, where we
reconstruct and transfer the expression and pose from a target image (right-
most column) to the source image identity (left-most column). On the left from
top to bottom, original images are courtesy of José Carlos Cortizo
Pérez/filckr, Montclair Film/flickr, Pham Toan/flickr, Javier Morales/flickr,
Khiet Nguyen/flickr, and Malcolm Slaney/flickr. On the right from top to
bottom, original images are courtesy of Adam Charnock/flickr, Daughterville
Festival/flickr, Delaney Turner/flickr, South African Tourism/flickr, Pat
(Cletch) Williams/flickr, and Collision Conf/flickr. Table 2. Quantitative
results on single-shot in-the-wild reconstruction (left) and self-expression
retargeting (right). Left: image, pose, and identity metrics are computed on
500 images sampled from FFHQ. Depth metrics are computed on the H3DS dataset.
Image, identity, and depth metrics are computed only on the facial region.
EMOCA is evaluated using its smaller face crop. Right: FACS coefficients and
facial landmarks are computed after expression and pose transfer on 32
expression pairs sampled from the Triplegangers test split.

Reconstruction |  LPIPS$\downarrow$ |  DISTS$\downarrow$ |  SSIM$\uparrow$ |  Pose$\downarrow$ |  ID$\uparrow$ |  L1 Depth$\downarrow$ |  RMSE Depth$\downarrow$
---|---|---|---|---|---|---|---
EMOCA | 0.1122 | 0.1268 | 0.9182 | 0.0681 | 0.0697 |  0.0300 |  0.0677
ROME | 0.1054 | 0.1130 | 0.9317 | 0.0600 | 0.3866 |  0.0237 |  0.0513
HeadNeRF | 0.1090 | 0.1199 | 0.9268 | 0.0606 | 0.2334 |  0.0379 |  0.0695
Ours (optimization-free) | 0.1427 | 0.1465 | 0.9053 | 0.0549 | 0.1082 |  0.0357 |  0.0658
Ours (encoder-free) | 0.0890 | 0.0921 | 0.9441 | 0.0533 | 0.4600 |  0.0241 |  0.0527
Ours | 0.0879 | 0.0905 | 0.9451 | 0.0563 | 0.4670 |  0.0228 |  0.0510

Retargeting |  FACS$\downarrow$ |  Facial Landmarks$\downarrow$
---|---|---
EMOCA |  4.712 |  0.2088
ROME |  3.204 |  0.1414
HeadNeRF |  3.848 |  0.1641
Ours |  1.733 |  0.1165

Figure 6. Ground truth geometry comparison on the H3DS dataset in the single-
view setting. Table 3. Quantitative comparison with FaceVerse (Wang et al.,
2022b) on 500 sampled FFHQ images for single-shot in-the-wild reconstruction.

Reconstruction | LPIPS$\downarrow$ | DISTS$\downarrow$ | SSIM$\uparrow$
---|---|---|---
FaceVerse | 0.1280 | 0.1119 | 0.9126
Ours | 0.0879 | 0.0905 | 0.9451

We present results of our proposed method with comparisons to EMOCA (Daněček
et al., 2022), ROME (Khakhulin et al., 2022) and FaceVerse (Wang et al.,
2022b), three recent mesh-based approaches for single-shot 3D avatar
generation, and HeadNeRF (Hong et al., 2022), an implicit approach using
neural radiance fields. Our method achieves higher fidelity texture and
geometry reconstruction in the facial region compared to the baselines.
Qualitatively and quantitatively, our method also demonstrates more faithful
expression and pose transfer between in-the-wild source and target images.
Finally, our learned texture map is intuitive to edit and propagates naturally
during animation.

###  4.1. Implementation Details

Our model is trained in two stages. In the first stage, we withhold the ground
truth multi-view images, as we find that supervising with both texture maps
and multi-view images negatively impacts the model’s ability to learn a
consistent UV mapping. In the second stage, we freeze the UV networks
$\\{g,g^{-1}\\}$ and supervise using the multi-view images to fine-tune the
learned texture maps while rendering image reconstructions at 768 $\times$ 512
resolution. Camera poses are provided with ground truth training data and we
estimate camera poses for in-the-wild FFHQ images using Deep3DFaceRecon (Deng
et al., 2019b). We perform sphere tracing for 50 steps per ray and use a
dimensionality of 512 for the geometry, color, and expression latent codes. We
train our AutoDecoder for 1000 epochs (approx. one week) and our inversion
encoder for 200 epochs (approx. one day) across 8 NVIDIA A40 GPUs. We use a
Triplegangers training/test split of 386/129 for the quantitative expression
experiments. Sphere tracing takes 8.5 seconds and inversion takes 3 hours per
image. See supplemental material for more details on training and model
architectures.

###  4.2. Single-Shot 3D Face Reconstruction and Animation

#### Qualitative Results

We show qualitative comparisons for single-shot reconstruction followed by
expression and pose transfer on FFHQ (Karras et al., 2019) images between the
proposed method, EMOCA, ROME, and HeadNeRF in Fig. 5 and Fig. 13.

Overall, our method is more photo-realistic and achieves higher expression
accuracy in facial reconstruction. EMOCA does not model the mouth interior and
relies on a pre-trained FLAME (Li et al., 2017) albedo model for texture. Our
model produces the most faithful expression transfer, demonstrating the
diversity of its learned expression space and generalization capabilities of
our method to in-the-wild data. HeadNeRF exhibits a large amount of identity
shift during pose transfer, whereas our method remains view-consistent after
large pose changes.

We also show a ground truth comparison of reconstructed geometry on the H3DS
(Ramon et al., 2021) dataset between our method and the baselines in Fig. 6.
HeadNeRF performs volumetric rendering at a low resolution and therefore
produces noisy depth results. Our geometry captures higher fidelity facial
geometry than ROME and captures the expression more faithfully (e.g., eye
blink) compared to EMOCA.

#### Quantitative Results

We report quantitative reconstruction and self-reenactment expression transfer
results in Table 2 and Table 3. The photometric (LPIPS (Zhang et al., 2018),
DISTS (Ding et al., 2020), SSIM (Wang et al., 2004)), pose error, and MagFace
(Meng et al., 2021) identity consistency (ID) metrics are calculated over a
dataset of 500 images from FFHQ. We compute L1 and RMSE depth error over all
subjects in the H3DS dataset. To evaluate self-reenactment expression error,
we randomly sample 32 source–target expression pairs over a test split of the
Triplegangers dataset and measure the L2 error for FACS (Ekman and Friesen,
1978) coefficients and facial landmarks. For details related to how each
metric is computed, please refer to the supplemental material.

On the FFHQ dataset, our proposed method achieves the best accuracy in terms
of LPIPS, DISTS, SSIM, and ID score. The optimization-free ablation struggles
to handle the considerably large domain shift between Triplegangers training
data and FFHQ in-the-wild images. Our model also exhibits the lowest depth
error on the H3DS dataset without relying on a 3D template mesh prior.
Finally, our model has the lowest FACS and facial landmark errors,
demonstrating the diversity of its learned expression space.

###  4.3. Ablations

In addition to the baselines mentioned, we compare our method to two ablations
for single-shot reconstruction. The first ablation is an optimization-free
inversion approach that only uses the learned encoder to directly map an input
image to the geometry, color, and expression codes
$\\{w_{geom},w_{color},w_{expr}\\}$. The second ablation is an encoder-free
inversion approach that omits the encoder and instead uses a mean
initialization for $\\{w_{geom},w_{color},w_{expr}\\}$ over the learned
AutoDecoder dictionary of latent codes.

Quantitative results for the ablations are reported in Table 2. The
optimization-free approach produces significantly worse photometric and depth
results, as there is a large domain gap between Triplegangers training data
and in-the-wild images; this causes the encoder to produce a coarse
reconstruction. The encoder-free approach performs better than the
optimization-free approach but is still worse than our full method in image
and geometry quality, demonstrating that the encoder initialization improves
the optimization reconstruction. Both ablations and our full method perform
similarly on pose accuracy.

#### Applications

As demonstrated in Fig. 5, our method directly supports monocular facial
performance capture and expression retargeting. Our hybrid representation
provides direct control over an intuitive texture map with a consistent
layout. Fig. 7 demonstrates an example workflow: a user reconstructs an input
image and modifies the learned texture map. The edits then continue to persist
smoothly across different facial animations. Textured meshes can be extracted
for further downstream applications such as re-lighting, as shown in the
teaser. Fig. 11 and Fig. 12 further demonstrate our model’s disentanglement
between geometry, texture, and expression with its capability of shape and
facial appearance transfer.

Figure 7. Texture editing. Top row: input image, learned texture map, and user
edited texture map. The learned texture map layout is intuitive and edits
propagate naturally during facial animation as shown in the bottom row.
Original image courtesy of Ed Kohler/flickr.

##  5\. Discussion

We have presented a new method for reconstructing 3D animatable and textured
faces from a single RGB image. The proposed approach combines implicit
representations with explicit texture maps to support explicit editing while
achieving better photo-realistic rendering, geometry, and expression
reconstruction than previous methods. We believe the proposed method makes
important contributions towards accessible creation of high-fidelity avatars
from in-the-wild images that are animatable, editable, and customizable for
downstream applications.

However, there are still limitations to the method. Firstly, the current
optimization process during inversion is significantly slower than encoder-
based methods. For real-time applications, more expressive representations
such as neural feature fields can be explored to enable optimization-free
inversion methods. Furthermore, the method relies on a de-lighting module from
Lumos to process in-the-wild images to generate a diffusely lit input image,
which may cause subjects to appear paler than expected. These limitations may
be alleviated through lighting augmentations of the training dataset to reduce
the domain gap and incorporating a lighting model such as spherical harmonics
into the representation. Finally, the results shown in this paper do not
capture hair or accessories due to limitations of the training dataset. While
not perfect, we refer to the supplemental material for a preliminary
demonstration of our representation’s capacity to handle hair and clothing on
the smaller RenderPeople dataset. As implicit representations such as neural
radiance fields excel at capturing the geometry and texture of thin
structures, it may be fruitful to combine our method with recent sparse view
implicit hair models (Wu et al., 2022; Kuang et al., 2022).

###### Acknowledgements.

We thank Simon Yuen and Miguel Guerrero for helping with preparing the 3D scan
dataset and assets, and Ting-Chun Wang for providing Lumos code. We also thank
Nicholas Sharp, Sanja Fidler and David Luebke for helpful discussions and
supports. This project was supported in part by a David Cheriton Stanford
Graduate Fellowship, ARL grant W911NF-21-2-0104, a Vannevar Bush Faculty
Fellowship, a gift from the Adobe corporation, Samsung, and Stanford HAI.

## References

  * (1)
  * Alldieck et al. (2021) Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. 2021\.  imghum: Implicit generative models of 3d human shape and articulated pose. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5461–5470.
  * Athar et al. (2021) ShahRukh Athar, Zhixin Shu, and Dimitris Samaras. 2021\.  Flame-in-nerf: Neural control of radiance fields for free view face animation.  _arXiv preprint arXiv:2108.04913_ (2021).
  * Athar et al. (2022) ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. 2022.  RigNeRF: Fully Controllable Neural 3D Portraits. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 20364–20373.
  * B R et al. (2021) Mallikarjun B R, Ayush Tewari, Hans-Peter Seidel, Mohamed Elgharib, and Christian Theobalt. 2021\.  Learning Complete 3D Morphable Face Models from Images and Videos. In _cvpr_.
  * Blanz and Vetter (1999) Volker Blanz and Thomas Vetter. 1999.  A morphable model for the synthesis of 3D faces. In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. 187–194.
  * Cao et al. (2014) Chen Cao, Qiming Hou, and Kun Zhou. 2014.  Displaced Dynamic Expression Regression for Real-Time Facial Tracking and Animation.  (2014).
  * Cao et al. (2022b) Chen Cao, Tomas Simon, Jin Kyu Kim, Gabe Schwartz, Michael Zollhoefer, Shun-Suke Saito, Stephen Lombardi, Shih-En Wei, Danielle Belko, Shoou-I Yu, et al. 2022b.  Authentic volumetric avatars from a phone scan.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–19.
  * Cao et al. (2016) Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou. 2016.  Real-Time Facial Animation with Image-Based Dynamic Avatars.  (2016).
  * Cao et al. (2022a) Yukang Cao, Guanying Chen, Kai Han, Wenqi Yang, and Kwan-Yee K Wong. 2022a.  JIFF: Jointly-aligned Implicit Face Function for High Quality Single View Clothed Human Reconstruction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2729–2739.
  * Chan et al. (2022) Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. 2022\.  Efficient geometry-aware 3D generative adversarial networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16123–16133.
  * Chandran et al. (2020) Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. 2020.  Semantic deep face models. In _2020 International Conference on 3D Vision (3DV)_. IEEE, 345–354.
  * Chen et al. (2019) Anpei Chen, Zhang Chen, Guli Zhang, Kenny Mitchell, and Jingyi Yu. 2019.  Photo-Realistic Facial Details Synthesis from Single Image. In _iccv_.
  * Chen et al. (2018) Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018.  Encoder-decoder with atrous separable convolution for semantic image segmentation. In _Proceedings of the European conference on computer vision (ECCV)_. 801–818.
  * Chen and Zhang (2019) Zhiqin Chen and Hao Zhang. 2019.  Learning implicit fields for generative shape modeling. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5939–5948.
  * Daněček et al. (2022) Radek Daněček, Michael J Black, and Timo Bolkart. 2022\.  EMOCA: Emotion Driven Monocular Face Capture and Animation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 20311–20322.
  * Deng et al. (2019a) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019a.  Arcface: Additive angular margin loss for deep face recognition. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4690–4699.
  * Deng et al. (2019b) Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. 2019b.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 0–0.
  * Dib et al. (2021a) Abdallah Dib, Gaurav Bharaj, Junghyun Ahn, Cédric Thébault, Philippe Gosselin, Marco Romeo, and Louis Chevallier. 2021a.  Practical face reconstruction via differentiable ray tracing. In _Computer Graphics Forum_ , Vol. 40. Wiley Online Library, 153–164.
  * Dib et al. (2021b) Abdallah Dib, Cedric Thebault, Junghyun Ahn, Philippe-Henri Gosselin, Christian Theobalt, and Louis Chevallier. 2021b.  Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 12819–12829.
  * Ding et al. (2020) Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. 2020\.  Image quality assessment: Unifying structure and texture similarity.  _IEEE transactions on pattern analysis and machine intelligence_ (2020).
  * Dou et al. (2017) Pengfei Dou, Shishir K. Shah, and Ioannis A. Kakadiaris. 2017\.  End-To-End 3D Face Reconstruction With Deep Neural Networks. In _cvpr_.
  * Egger et al. (2020) Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 2020\.  3d morphable face models—past, present, and future.  _ACM Transactions on Graphics (TOG)_ 39, 5 (2020), 1–38.
  * Ekman and Friesen (1978) Paul Ekman and Wallace V Friesen. 1978.  Facial action coding system.  _Environmental Psychology & Nonverbal Behavior_ (1978).
  * Fan et al. (2017) Haoqiang Fan, Hao Su, and Leonidas J Guibas. 2017.  A point set generation network for 3d object reconstruction from a single image. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 605–613.
  * Feng et al. (2021) Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. 2021\.  Learning an animatable detailed 3D face model from in-the-wild images.  _ACM Transactions on Graphics (ToG)_ 40, 4 (2021), 1–13.
  * Garrido et al. (2013) Pablo Garrido, Levi Valgaert, Chenglei Wu, and Christian Theobalt. 2013.  Reconstructing Detailed Dynamic Face Geometry from Monocular Video.  (2013).
  * Garrido et al. (2016) Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Pérez, and Christian Theobalt. 2016\.  Reconstruction of Personalized 3D Face Rigs from Monocular Video.  (2016).
  * Gecer et al. (2019) Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou. 2019.  GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction. In _cvpr_.
  * Genova et al. (2018) Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Freeman. 2018\.  Unsupervised Training for 3D Morphable Model Regression. In _cvpr_.
  * Giebenhain et al. (2022) Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, and Matthias Nießner. 2022.  Learning Neural Parametric Head Models.  _arXiv preprint arXiv:2212.02761_ (2022).
  * Girdhar et al. (2016) Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. 2016.  Learning a predictable and generative vector representation for objects. In _European Conference on Computer Vision_. Springer, 484–499.
  * Grassal et al. (2022) Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. 2022.  Neural head avatars from monocular RGB videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18653–18664.
  * Gropp et al. (2020) Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. 2020.  Implicit geometric regularization for learning shapes.  _arXiv preprint arXiv:2002.10099_ (2020).
  * Groueix et al. (2018) Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 2018.  A papier-mâché approach to learning 3d surface generation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 216–224.
  * Hong et al. (2022) Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang. 2022.  Headnerf: A real-time nerf-based parametric head model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 20374–20384.
  * Huynh et al. (2018) Loc Huynh, Weikai Chen, Shunsuke Saito, Jun Xing, Koki Nagano, Andrew Jones, Paul Debevec, and Hao Li. 2018\.  Mesoscopic Facial Geometry Inference Using Deep Neural Networks. In _cvpr_.
  * Ichim et al. (2015) Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015\.  Dynamic 3D Avatar Creation from Hand-Held Video Input.  (2015).
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4401–4410.
  * Kellnhofer et al. (2021) Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. 2021\.  Neural lumigraph rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4287–4297.
  * Khakhulin et al. (2022) Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. 2022.  Realistic one-shot mesh-based head avatars. In _European Conference on Computer Vision_. Springer, 345–362.
  * King (2009) Davis E. King. 2009\.  Dlib-ml: A Machine Learning Toolkit.  _Journal of Machine Learning Research_ 10 (2009), 1755–1758.
  * Ko et al. (2023) Jaehoon Ko, Kyusun Cho, Daewon Choi, Kwangrok Ryoo, and Seungryong Kim. 2023.  3d gan inversion with pose optimization. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2967–2976.
  * Kuang et al. (2022) Zhiyi Kuang, Yiyang Chen, Hongbo Fu, Kun Zhou, and Youyi Zheng. 2022.  DeepMVSHair: Deep Hair Modeling from Sparse Views. In _SIGGRAPH Asia 2022 Conference Papers_. 1–8.
  * Lattas et al. (2020) Alexandros Lattas, Stylianos Moschoglou, Baris Gecer, Stylianos Ploumpis, Vasileios Triantafyllou, Abhijeet Ghosh, and Stefanos Zafeiriou. 2020.  AvatarMe: Realistically Renderable 3D Facial Reconstruction ”In-the-Wild”. In _cvpr_.
  * Li et al. (2022) Moran Li, Haibin Huang, Yi Zheng, Mengtian Li, Nong Sang, and Chongyang Ma. 2022\.  Implicit Neural Deformation for Sparse-View Face Reconstruction.  (2022).
  * Li et al. (2020) Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al. 2020\.  Learning formation of physically-based face attributes. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 3410–3419.
  * Li et al. (2017) Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017.  Learning a model of facial shape and expression from 4D scans.  _ACM Trans. Graph._ 36, 6 (2017), 194–1.
  * Lin et al. (2022) Connor Z Lin, David B Lindell, Eric R Chan, and Gordon Wetzstein. 2022.  3D GAN Inversion for Controllable Portrait Image Animation.  _arXiv preprint arXiv:2203.13441_ (2022).
  * Luo et al. (2021) Huiwen Luo, Koki Nagano, Han-Wei Kung, Qingguo Xu, Zejian Wang, Lingyu Wei, Liwen Hu, and Hao Li. 2021\.  Normalized avatar synthesis using stylegan and perceptual refinement. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11662–11672.
  * Ma et al. (2022) Li Ma, Xiaoyu Li, Jing Liao, Xuan Wang, Qi Zhang, Jue Wang, and Pedro V Sander. 2022\.  Neural parameterization for dynamic human head editing.  _ACM Transactions on Graphics (TOG)_ 41, 6 (2022), 1–15.
  * Meng et al. (2021) Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. 2021\.  Magface: A universal representation for face recognition and quality assessment. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14225–14234.
  * Mescheder et al. (2019) Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019\.  Occupancy networks: Learning 3d reconstruction in function space. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4460–4470.
  * Mildenhall et al. (2021) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021\.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ 65, 1 (2021), 99–106.
  * Nagano et al. (2018) Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, Hao Li, Richard Roberts, et al. 2018\.  paGAN: real-time avatars using dynamic textures.  _ACM Trans. Graph._ 37, 6 (2018), 258–1.
  * Niu et al. (2018) Chengjie Niu, Jun Li, and Kai Xu. 2018.  Im2struct: Recovering 3d shape structure from a single rgb image. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4521–4529.
  * Or-El et al. (2022) Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman. 2022.  Stylesdf: High-resolution 3d-consistent image and geometry generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13503–13513.
  * Park et al. (2019) Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019\.  Deepsdf: Learning continuous signed distance functions for shape representation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 165–174.
  * Ramon et al. (2021) Eduard Ramon, Gil Triginer, Janna Escur, Albert Pumarola, Jaime Garcia, Xavier Giro-i Nieto, and Francesc Moreno-Noguer. 2021.  H3d-net: Few-shot high-fidelity 3d head reconstruction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5620–5629.
  * Renderpeople (2022) Renderpeople. 2022\.  _Renderpeople_.  https://renderpeople.com
  * Roich et al. (2022) Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. 2022\.  Pivotal tuning for latent-based editing of real images.  _ACM Transactions on Graphics (TOG)_ 42, 1 (2022), 1–13.
  * Romdhani and Vetter (2005) S. Romdhani and T. Vetter. 2005.  Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In _cvpr_.
  * Saito et al. (2019) Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019\.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2304–2314.
  * Saito et al. (2020) Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. 2020.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 84–93.
  * Saito et al. (2017) Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and Hao Li. 2017.  Photorealistic Facial Texture Inference Using Deep Neural Networks. In _cvpr_.
  * Shi et al. (2014) Fuhao Shi, Hsiang-Tao Wu, Xin Tong, and Jinxiang Chai. 2014\.  Automatic Acquisition of High-Fidelity Facial Performances Using Monocular Videos.  (2014).
  * Sitzmann et al. (2019) Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. 2019\.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Tewari et al. (2019) Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. 2019\.  FML: Face Model Learning from Videos. In _cvpr_.
  * Tewari et al. (2022) Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, W Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. 2022\.  Advances in neural rendering. In _Computer Graphics Forum_ , Vol. 41. Wiley Online Library, 703–735.
  * Tewari et al. (2018) Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt. 2018\.  Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2549–2559.
  * Tewari et al. (2017) Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. 2017.  MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In _iccv_.
  * Thies et al. (2019) Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019\.  Deferred Neural Rendering: Image Synthesis using Neural Textures.  (2019).
  * Thies et al. (2016) J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. 2016\.  Face2Face: Real-time Face Capture and Reenactment of RGB Videos. In _cvpr_.
  * Tran et al. (2019) Luan Tran, Feng Liu, and Xiaoming Liu. 2019.  Towards high-fidelity nonlinear 3D face morphable model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1126–1135.
  * Tran and Liu (2018) Luan Tran and Xiaoming Liu. 2018.  Nonlinear 3d face morphable model. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7346–7355.
  * Tran and Liu (2019) Luan Tran and Xiaoming Liu. 2019.  On learning 3d face morphable model from in-the-wild images.  _IEEE transactions on pattern analysis and machine intelligence_ 43, 1 (2019), 157–171.
  * Triplegangers (2022) Triplegangers. 2022\.  _triplegangers_.  https://triplegangers.com
  * Tuan Tran et al. (2017) Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gerard Medioni. 2017.  Regressing Robust and Discriminative 3D Morphable Models With a Very Deep Neural Network. In _cvpr_.
  * Tulsiani et al. (2017) Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. 2017.  Multi-view supervision for single-view reconstruction via differentiable ray consistency. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2626–2634.
  * Wang et al. (2022a) Daoye Wang, Prashanth Chandran, Gaspard Zoss, Derek Bradley, and Paulo Gotardo. 2022a.  Morf: Morphable radiance fields for multiview neural head modeling. In _ACM SIGGRAPH 2022 Conference Proceedings_. 1–9.
  * Wang et al. (2022b) Lizhen Wang, Zhiyuan Chen, Tao Yu, Chenguang Ma, Liang Li, and Yebin Liu. 2022b.  FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 20333–20342.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 4 (2004), 600–612.
  * Wu et al. (2019) Fanzi Wu, Linchao Bao, Yajing Chen, Yonggen Ling, Yibing Song, Songnan Li, King Ngi Ngan, and Wei Liu. 2019\.  MVF-Net: Multi-View 3D Face Morphable Model Regression. In _CVPR_.
  * Wu et al. (2018) Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum. 2018.  Learning shape priors for single-view 3d completion and reconstruction. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 646–662.
  * Wu et al. (2022) Keyu Wu, Yifan Ye, Lingchen Yang, Hongbo Fu, Kun Zhou, and Youyi Zheng. 2022\.  NeuralHDHair: Automatic High-fidelity Hair Modeling from a Single Image Using Implicit Neural Representations. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1526–1535.
  * Wu et al. (2020) Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. 2020\.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1–10.
  * Xiang et al. (2021) Fanbo Xiang, Zexiang Xu, Milos Hasan, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Hao Su. 2021\.  Neutex: Neural texture mapping for volumetric neural rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7119–7128.
  * Xie et al. (2022) Jiaxin Xie, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen. 2022.  High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization.  _arXiv preprint arXiv:2211.15662_ (2022).
  * Xu et al. (2019) Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. 2019.  Disn: Deep implicit surface network for high-quality single-view 3d reconstruction.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Xue et al. (2022) Yang Xue, Yuheng Li, Krishna Kumar Singh, and Yong Jae Lee. 2022\.  GIRAFFE HD: A High-Resolution 3D-aware Generative Model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18440–18449.
  * Yamaguchi et al. (2018) Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li. 2018\.  High-Fidelity Facial Reflectance and Geometry Inference from an Unconstrained Image.  (2018).
  * Yan et al. (2016) Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016.  Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision.  _Advances in neural information processing systems_ 29 (2016).
  * Yang et al. (2018) Guandao Yang, Yin Cui, Serge Belongie, and Bharath Hariharan. 2018\.  Learning single-view 3d reconstruction with limited pose supervision. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 86–101.
  * Yeh et al. (2022) Yu-Ying Yeh, Koki Nagano, Sameh Khamis, Jan Kautz, Ming-Yu Liu, and Ting-Chun Wang. 2022\.  Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation.  _ACM Transactions on Graphics (TOG)_ 41, 6 (2022), 1–21.
  * Yenamandra et al. (2021) Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, and Christian Theobalt. 2021\.  i3dmm: Deep implicit 3d morphable model of human heads. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12803–12813.
  * Yin et al. (2022) Fei Yin, Yong Zhang, Xuan Wang, Tengfei Wang, Xiaoyu Li, Yuan Gong, Yanbo Fan, Xiaodong Cun, Öztireli Cengiz, and Yujiu Yang. 2022\.  3D GAN Inversion with Facial Symmetry Prior.  _arxiv:2211.16927_ (2022).
  * Yu et al. (2018) Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. 2018\.  Bisenet: Bilateral segmentation network for real-time semantic segmentation. In _Proceedings of the European conference on computer vision (ECCV)_. 325–341.
  * Zanfir et al. (2022) Mihai Zanfir, Thiemo Alldieck, and Cristian Sminchisescu. 2022\.  PhoMoH: Implicit Photorealistic 3D Models of Human Heads.  _arXiv preprint arXiv:2212.07275_ (2022).
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 586–595.
  * Zheng et al. (2022b) Mingwu Zheng, Hongyu Yang, Di Huang, and Liming Chen. 2022b.  ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 20343–20352.
  * Zheng et al. (2022a) Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges. 2022a.  Im avatar: Implicit morphable head avatars from videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13545–13555.
  * Zheng et al. (2022c) Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J Black, and Otmar Hilliges. 2022c.  PointAvatar: Deformable Point-based Head Avatars from Videos.  _arXiv preprint arXiv:2212.08377_ (2022).
  * Zhu et al. (2017) Rui Zhu, Hamed Kiani Galoogahi, Chaoyang Wang, and Simon Lucey. 2017.  Rethinking reprojection: Closing the loop for pose-aware shape reconstruction from a single image. In _Proceedings of the IEEE International Conference on Computer Vision_. 57–65.
  * Zou et al. (2017) Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, and Derek Hoiem. 2017.  3d-prnn: Generating shape primitives with recurrent neural networks. In _Proceedings of the IEEE International Conference on Computer Vision_. 900–909.

.

Figure 8. UV parameterization consistency. We measure the mean L2 error over
32 FFHQ subjects between the learned texture map (top left) and the cycle
texture map (bottom left) obtained by mapping from UV $\rightarrow$ 3D
$\rightarrow$ UV. Figure 9. Encoder training data augmentation ablation.
Training the encoder with the synthetically augmented Triplegangers dataset
(Yeh et al., 2022) significantly improves our initialization, which is
important for converging to a high quality inversion result. Note the
difference in the final reconstructed geometry. Original image courtesy of
David Geitgey Sierralupe/flickr. Figure 10. Facial landmarks loss ablation.
Removing the facial landmarks loss during inversion reduces reconstruction
quality of the face contour (left and right jaws) and facial features such as
the eyes (right). Original image courtesy of Cena Mineira (left) and
BigBrother Junkie (right). Figure 11. Shape attribute transfer. We fix the
color and expression codes for the source subject and directly replace the
source geometry code with the target geometry code. Original images are
courtesy of Francesco Pierantoni/flickr (left col, top), Tim Regan (left col,
bottom), Bob n Renee/flickr (top row, left), and Sarah & Austin Houghton-
Bird/flickr (top row, right). Figure 12. Facial appearance attribute transfer.
We fix the geometry and expression codes for the source subject and directly
replace the source color code with the target color code. Original images are
courtesy of Lord Jim/flickr (left col, top), xiǎo cháo zhù/flickr (left col,
bottom), U.S. Army/flickr (top row, left), and U.S. Department of
Energy/flickr (top row, right). Figure 13. Zoomed in comparison with ROME
(Khakhulin et al., 2022) from Fig. 5. Our model captures the target expression
with higher fidelity and higher resolution textures (512$\times$512) compared
to ROME (256$\times$256). Figure 14. Gallery of single-shot reconstruction
results on FFHQ. On the left from top to bottom, images are courtesy of Kerry
Goodwin/flickr, Alex ”Khaki” Vance/flickr, Katherine Donovan/flickr, Wilson
Seed/flickr, SC IPHC/flickr, Commander, U.S. Naval Forces Europe-Africa/U.S.
6th Fleet/flickr, Ordiziako Jakintza Ikastola/flickr, Cena Mineira/flickr,
Report Verlag/flickr, Malcolm Slaney/flickr, Gitta Wilén/flickr, and Jill
Carlson/flickr. On the right from top to bottom, images are courtesy of Pawel
Loj/flickr, Santuario Torreciudad/flickr, Wilbur Ince/flickr, Existence
Church/flickr, Eden, Janine and Jim/flickr, Ehud Kenan/flickr, Aécio Neves
Presidente/flickr, VcStyle/flickr, Pawel Loj/flickr, Jason Aspinall/flickr,
Logan C/flickr, and RISE/flickr.