# Surfel-based Gaussian Inverse Rendering for Fast and Relightable Dynamic
Human Reconstruction from Monocular Videos

Yiqun Zhao, Chenming Wu, Binbin Huang, Yihao Zhi, Chen Zhao,
Jingdong Wang, , and Shenghua Gao Yiqun Zhao and Binbin Huang are with School
of Information Science and Technology, ShanghaiTech University, Shanghai,
China. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Chenming Wu,
Chen Zhao, and Jingdong Wang are with the Department of Computer Vision
Technology (VIS), Baidu Inc., Beijing, China. Yihao Zhi is with the School of
Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China.
Shenghua Gao is with the University of Hong Kong, Hong Kong SAR, China.
E-mail<EMAIL_ADDRESS>Manuscript received April 19, 2005; revised August 26,
2015.

###### Abstract

Efficient and accurate reconstruction of a relightable, dynamic clothed human
avatar from a monocular video is crucial for the entertainment industry. This
paper introduces the Surfel-based Gaussian Inverse Avatar (SGIA) method, which
introduces efficient training and rendering for relightable dynamic human
reconstruction. SGIA advances previous Gaussian Avatar methods by
comprehensively modeling Physically-Based Rendering (PBR) properties for
clothed human avatars, allowing for the manipulation of avatars into novel
poses under diverse lighting conditions. Specifically, our approach integrates
pre-integration and image-based lighting for fast light calculations that
surpass the performance of existing implicit-based techniques. To address
challenges related to material lighting disentanglement and accurate geometry
reconstruction, we propose an innovative occlusion approximation strategy and
a progressive training approach. Extensive experiments demonstrate that SGIA
not only achieves highly accurate physical properties but also significantly
enhances the realistic relighting of dynamic human avatars, providing a
substantial speed advantage. We exhibit more results in our project page:
https://GS-IA.github.io.

###### Index Terms:

Relightable Dynamic Human Reconstruction, PBR Properties Estimation, Novel
View Synthesis

##  1 Introduction

Figure 1: We achieve fast reconstruction of clothed human avatars with PBR
properties from a monocular video. SGIA takes a monocular video and initial
human pose and shape as input to estimate dynamic clothed humans’ PBR
properties, including geometry and materials. Leveraging a PBR-aware 2DGS
representation, our method enables fast training and rendering processes. By
utilizing the estimated PBR properties, we can not only deform the avatars
into different poses but also render them with realistic lighting conditions,
allowing for versatile and visually appealing outputs.

Reconstructing a clothed human avatar and estimating its Physically-Based
Rendering (referred to as PBR) properties from monocular videos is a
challenging task in computer vision and graphics. This process involves
disentangling geometry, PBR materials, and environment illumination. With
accurate PBR properties, avatars can be deformed into different poses and re-
rendered under novel illuminations, as shown in Fig. 1, offering significant
potential for virtual reality, movies, and games.

Early works aimed to estimate PBR properties from single images [1, 2, 3, 4,
5, 6, 7] or multi-view inputs [8, 9, 10, 11, 12, 13, 14, 15], often assuming
known lighting environment maps or scene geometry, which is impractical.
Radiance fields [16] have emerged as effective 3D-consistent scene
representations. Works in this direction [17, 18, 19, 20, 21, 22, 23, 24]
estimate geometry, material, and environment light for static scenes from
multi-view images but do not account for dynamics, crucial for modeling
clothed human avatars. Recent research [25, 26, 27] has advanced clothed human
animation by modeling radiance fields and PBR properties for dynamic humans,
but NeRF optimizes density rather than geometry directly [28], leading to
suboptimal geometry quality. Some works explore neural implicit surfaces [29,
30] for better geometry, but all these methods rely on dense point sampling
from MLPs [16] or multi-resolution hash-grids [31], slowing down training and
rendering due to extensive query operations.

Recently, Gaussian Splatting [32, 33] has revolutionized high-quality, real-
time rendering through its explicit representation and efficient splatting-
based rasterization on modern GPUs. Unlike prior methods [34, 35, 29] that
depend on complex backward skinning [36, 37] for transformations to a
canonical space [38], Gaussian primitives for clothed human avatars are easily
and efficiently animated via forward skinning, similar to the template mesh
[39]. Recent studies [40, 41, 42, 43, 44, 45, 46] have applied Gaussian
splatting techniques to reconstruct dynamic clothed human avatars from
monocular or multi-view videos, achieving remarkable rendering quality and
speed. However, these approaches are limited to capturing radiance fields and
fail to recover the full Physically-Based Rendering (PBR) properties of
dynamic humans, thus lacking generalization to novel illuminations.

Our goal is to efficiently and accurately reconstruct a relightable, dynamic
clothed human avatar from a monocular video, addressing two key challenges:
_i.e._ , disentangling coupled materials and shadow effects caused by lighting
occlusion, and achieving high-quality geometry of the clothed human avatar. To
this end, we propose the Surfel-based Gaussian Inverse Avatar (SGIA), inspired
by the efficiency of Gaussian representation. SGIA recovers both the radiance
field and Physically-Based Rendering (PBR) properties of clothed human avatars
from monocular videos, enabling avatars to adapt to any scene and varying
lighting conditions. We define a set of canonical PBR-awre 2DGS in a canonical
space, which are transformed into the observation space via forward linear
blend skinning based on different poses. To accurately model the dynamics of
clothed human avatars, we combine articulation from the template model Skinned
Multi-Person Linear Model (SMPL)[39] with learnable latent bones conditioned
on human poses, drawing inspiration from[40].

To effectively disentangle the coupled materials and shadow effects caused by
lighting occlusion, inspired by previous works [47, 48, 49, 50], we add
learnable parameters, such as albedo, roughness, and metallic, to each
Gaussian primitive for the clothed human avatar. While occlusion modeling can
be efficiently handled with spherical harmonics in static scenes, it becomes
challenging in dynamic scenes where occlusion relationships change over time.
Directly calculating occlusion with ray tracing [51] for each Gaussian is
accurate but inefficient. To achieve efficient occlusion calculations, we use
modern ray-casting algorithms on the limited vertices of the template mesh.
Our hybrid strategy combines point and mesh benefits: querying the nearest
points on the template mesh for each Gaussian and using surface points and
normal vectors to bake ambient occlusion from the template mesh. Given the
minimal occlusion difference between the clothed human and the unclothed SMPL
[39], our method balances efficiency and accuracy, effectively disentangling
materials and lighting.

To accurately reconstruct the surface geometry of clothed human avatars, which
is crucial for physically based inverse rendering, we propose to leverage the
2D Gaussian Splatting (referred to as 2DGS) [33] as our primitive
representation due to its surface-aligned nature, improving geometry
reconstruction quality. Each 2D surfel’s orientation serves as a local normal
direction. Initially, we optimized the rough shape using the original Gaussian
color. However, the geometry remains inaccurate after image reconstruction,
especially under high light conditions (Fig.3). Unlike previous works[47, 50,
49] that freeze Gaussian shape after the first stage, we refine the shape
attributes during the PBR optimization stage, leveraging the normal-correlated
appearance to enhance geometry. While the original normal consistency loss in
2DGS maintains a splat normal consistency with geometry, it can hinder normal
optimization. To address this, we propose a progressive optimization strategy
that achieves both accurate geometry and consistent splat normals.

By utilizing the materials, occlusion, and geometry for each splat, we
calculate the PBR color of each Gaussian in the observation space and apply
the splatting process [52, 33] designed for 2DGS to produce the PBR image. To
address the ambiguity between lighting and materials and achieve plausible PBR
material solutions, we use regularization terms to maintain material
smoothness on planar surfaces and ensure natural lighting. Extensive
experiments on synthetic and real-world datasets show that our method enables
fast, relightable clothed human avatar reconstruction within 40 minutes and
renders at $540\times 540$ resolution at 5 FPS, which is $100\times$ faster
than state-of-the-art NeRF-based human inverse rendering methods. Quantitative
evaluations demonstrate that our method achieves comparable PBR property
estimation accuracy to state-of-the-art methods. Comprehensive ablation
studies further validate our approach.

In summary, our main contributions are:

  * •

We propose SGIA, a surfel-based PBR properties modeling method for fast and
relightable dynamic clothed human reconstruction from monocular videos.

  * •

Our method tackles the challenges in material lighting disentanglement and
accurate geometry reconstruction by utilizing our proposed occlusion
approximation strategy and progressive training strategy.

  * •

Extensive experiments demonstrate that our method achieves fast training and
rendering while maintaining high-accuracy in PBR properties estimation and
human relighting at novel poses.

##  2 Related Work

###  2.1 Inverse rendering for static scenes.

Estimating the physical properties from images is a highly indeterminate
problem due to its under-constrained nature. Previous works on BTF [53, 54,
55, 56, 57] and SVBRDF [58, 11, 14, 59] estimation relay on specific
assumptions of specific viewpoints, lighting patterns, or capture setups.
Other works explore inverse rendering on single images [1, 2, 3, 4, 5, 6, 7]
or multi-view inputs [8, 9, 10, 11, 12, 13, 14, 15]. Recovering shape and
illumination from RGB images is heavily ill-posed due to the coupling of
shape, materials, and lighting in real-world captured images. These works can
only recover high-quality material for each view but fail to achieve realistic
and consistent multi-view relighting results under real-world natural
illuminations due to the lack of 3D consistent material fields.

With the rapid development of Neural Radiance Fields (referred to as NeRFs)
[16], lots of works [23, 17, 18, 19, 21, 20, 60, 61, 22, 62, 63, 64] based on
neural field have been proposed to achieve inverse rendering from multi-view
images. Specifically, NeRFactor [23] enabled full estimation of the physical
properties (geometry, albedo, materials, and lighting) by incorporating the
effect of shadows. Neural-PIL [21] explored to approximate pre-integration
light integration process by a neural network [65] to achieve fast calculation
of outgoing radiance. PhySG [22] proposed to use mixtures of spherical
Gaussians to approximate the light transport efficiently. InvRender [62]
proposed to model the indirect illumination by distilling a pre-trained NeRF
into MLPs to achieve more accurate light disentangling. TensoIR [24] took
advantage of tensor decompostion [66] to improve the reconstruction quality.
Due to the inaccurate geometry modeling in NeRFs, some other works [67, 68,
69, 70, 71] explored neural signed distance function (SDFs) rendering to
achieve more accurate geometry reconstruction during inverse rendering. Among
them, NeRO [67] applied the split-sum to constrain the learning of reflective
objects, and it can achieve high-quality geometry reconstruction for the high
reflective objects. The deep Marching Tetrahedra [72] provided an efficient
and differentiable mesh extraction from neural implicit surface
representation. With the benefit of differentiable mesh representation and the
differentiable mesh rasterizer, Nvdiffrec [73] proposed to conduct light
integration on the mesh surface to achieve rapid and photo-realistic
physically based rendering. NvdiffrecMC [74] showed that by exploring the
combination of mesh-based Mote-Carlo ray-tracing and multi-importance
sampling, the decomposition into shape, materials, and lighting can be further
improved.

More recently, Gaussian Splatting based methods [32, 75, 33, 76] have achieved
high-quality rendering and real-time rendering speed thanks to the forward
mapping splatting process. Some works [47, 48, 49, 50] explored to use either
the learnable normal vector [47, 50] or the shortest axis of the Gaussian [48,
49] to achieve geometry representation for inverse rendering. However, these
works only focused on the inverse rendering of the static scene, while our
work focuses on the inverse rendering of a dynamic clothed human avatar, which
is more challenging due to the avatar’s dynamic.

###  2.2 Clothed human avatar modeling.

In addition to the textured mesh, NeRFs [16] also became a useful
representation for photo-realistic clothed avatar reconstruction [77, 38, 78,
79, 80, 35, 34, 81, 82, 83, 84] from monocular or multi-view videos.
NeuralBody [38] assigned the latent code on the canonical space SMPL
[39]vertices and applied the LBS [39] to transform the clothed human avatars
to articulation space. For the ray sampling in NeRF [16], the world space
points are first transformed to articulated SMPL space to query the radiance.
They also rely on time-varied latent to fit the pose-dependent deformation.
Animatable NeRF [77] further proposed a pose-dependent deformation network to
fit the clothed human conditioned on human. With pose-dependent modeling, the
reconstructed avatar can be generalized to novel poses. They also utilize
inverse LBS, which can transform the sample points from SMPL articulated space
to the canonical space to query the radiance, thus achieving accurate and fast
radiance field modeling for humans. However, there exists a large ambiguity in
the inverse LBS calculation. InstantAvatar [35] proposed to incorporate the
accurate and root finding [37, 36] for inverse LBS and multi-resolution Hash
Grids for radiance fields modeling. Recent advances in Gaussian Splatting [32]
assigned the radiance to explicit points that can be easily animated by
manipulating the skinning weight from template mesh. The forward splatting
process does not need the transformation from articulated space to canonical
space. Therefore, both the ambiguous inverse LBS and time-consuming root-
finding in Snarf can be avoided. Most works [43, 40, 44, 85, 86] explored
similar techniques to achieve fast clothed human avatar reconstruction from
monocular videos. In contrast with these works, our work aims to model the PBR
properties rather than only the radiance. We also take the disentangling of
geometry, materials, and environmental lighting to achieve estimation of full
PBR properties.

###  2.3 PBR properties reconstruction of clothed human avatars.

Previous works [87, 88, 89, 90] explored neural networks to learn the
decomposition of materials and lighting from single portrait images. Recently,
High quality physical properties reconstruction of clothed human avatar [30,
91, 25, 26, 29, 27, 92, 93, 94, 79] have been widely explored with the
development of Neural Radiance Fields [16]. DS-NeRF [79] proposed to query the
lighting at world space to disentangle the light and texture. Relighing4D [25]
first explored the MLPs representation based on NeuralBody [38] to jointly
estimate the shape, lighting, and the albedo of dynamic humans from monocular
or sparse videos under unknown illuminations by disentangling visibility via
MLPs. RANA [92] pre-trained a mesh representation on ground truth 3d assets to
approximate the radiance transfer under different lighting. Relightable Avatar
[26] pretrained an occlusion approximation neural network from a large-scale
motion sequence dataset [95, 96]. Intrinsic Avatar [29] implemented a fast
secondary ray-tracing at articulated space to calculate the visibility and the
indirect illuminations. Different from these works, our method explores a
relightable clothed human avatar representation based on Gaussian splatting
[32] to achieve fast training and fast rendering. Concurrent with our work,
Animatable Gaussians [97] proposed a relighting extension that predicts
occlusion based on the human pose and position map with the 2D StyleUnet [98]
to achieve high-quality rendering. However, their work relied on multi-view
videos as input and complex poses during training to achieve pose
generalization, which is not suitable for our setting. PhysAvatar [99] also
explored the relightable Gaussian Avatar, but it relied on the multi-view
videos as input to reconstruct the mesh at the first timestamp.

##  3 Our Method

Figure 2:  Overview of our pipeline. We define the radiance ($c_{k}$) and
materials ($a_{k},r_{k},m_{k}$) at the canonical space as canonical PBR-aware
2DGS and deform them to the world space via Linear Blend Skinning (LBS). We
first optimize with Image reconstruction loss to get the initial shape of the
clothed avatar. Based on the rough shape, we optimize the Gaussian attributes
with Physically Based Rendering. To model the shadow effect and decouple the
materials from lighting, we propose to approximate the calculation of
occlusion from the template mesh. Additionally, we apply regularization,
_i.e._ , smooth loss, white loss, and normal consistency loss to get a
plausible solution for the PBR materials.

In this section, we first introduce the background of surfel-based
representation [33, 32]. Then, we describe our framework for relightable
dynamic clothed avatar representation and physically-based rendering. The
clothed avatar is built upon surfel-based representation [33] with an
articulated template [40] based on SMPL [39]. Motivated by the success of 2DGS
for high-quality geometry recovery, we propose a PBR-aware 2DGS that
incorporates the lighting, roughness, metallics, and albedo into the 2DGS for
monocular videos. We also propose a solution for ambient occlusion calculation
to enhance the rendering realism and better decoupling of the material and
lighting. Finally, we describe our progressive training strategy to achieve
geometrically accurate surface reconstruction and accurate albedo estimation.
We visualize the overall pipeline of our method in Fig. 2.

###  3.1 A revisit of 2DGS

Unlike 3DGS [32], 2DGS [33] adopt flat 2D Gaussians embedded in 3D space for
scene representation. The primitive distributes densities within a planar disk
(also known as surfel), which makes the primitive better alignment with
surfaces. This feature can improve the quality of our geometry reconstruction
and achieve better material estimation. In 2DGS [33], each primitive is
characterized by a central points $\mathbf{p}_{k}\in\mathbb{R}^{3}$ , two
principal tangential vectors $\mathbf{t}_{u},\mathbf{t}_{v}\in\mathbb{R}^{3}$
and two scaling factors $s_{u},s_{v}\in\mathbb{R}$. The scaling factors
correspond to the variances of the 2D Gaussian.

A 2D Gaussian is defined in a local tangent plane of the world space,
parameterized as :

$\mathbf{P}(u,v)=\mathbf{p}_{k}+s_{u}\mathbf{t}_{u}u+s_{v}\mathbf{t}_{v}v=[\mathbf{R}_{k}\mathbf{S}_{k},\mathbf{p}_{k}](u,v,1,1)^{T}$
(1)

where
$\mathbf{R}_{k}=[\mathbf{t}_{u},\mathbf{t}_{v},\mathbf{t}_{u}\times\mathbf{t}_{v}]\in
SO(3)$ denotes the rotation of each surfel,
$\mathbf{t}_{u}\times\mathbf{t}_{v}$ denotes the normal orientation of each
surfel, and the diagonal matrix
$\mathbf{S}_{k}=\text{diag}(s_{u},s_{v},0)\in\mathbb{R}^{3\times 3}$ denotes
the scale of each surfel. Each 2D Gaussian primitive also carries an opacity
$o$ and a view-dependent appearance $\mathbf{c}$.

For each point $\mathbf{u}=(u,v)$ in the local coordinate, its 2D Gaussian
value can be calculated by

$\mathcal{G}(\mathbf{u})=\exp(-\frac{u^{2}+v^{2}}{2})$ (2)

Then, 2D Gaussians are sorted based on the depth of their center and organized
into tiles. To render a pixel $x$, we then cast a ray into the scene to
compute the intersection $\mathbf{u}(x)$ on the splats and accumulate the
texture function with volumetric alpha blending:

$\mathbf{c}(x)=\sum_{i=1}\mathbf{c}_{i}o_{i}\mathcal{G}_{i}(\mathbf{u}(x))\prod_{j=1}^{i-1}(1-o_{j}\mathcal{G}_{j}(\mathbf{u}(x)))$
(3)

###  3.2 Clothed humans avatars as dynamic surfels animated by template models

We follow the recent advances [40, 100, 44, 41] to model the human as
animatable Gaussians. Specifically, we define the PBR-aware 2DGS in a
canonical space and transform them into world space via linear blend skinning
(LBS). The Gaussians are animated by a human template model [39] to achieve
avatar animation. In contrast to them, we choose the surfel-based primitive
[33] as our representation for better geometry accuracy. Furthermore, we
define additional parameters for intrinsic properties and lighting on each 2D
Gaussian to enable physically based rendering.

Canonical PBR-aware 2DGS Representations. We first utilize a set of PBR-aware
2D Gaussians ($\mathbf{G}$) with material parameters to represent the clothed
avatar in a canonical space:

$\mathbf{G}=\\{\mathbf{p}_{k},\mathbf{R}_{k},\mathbf{S}_{k},\mathbf{c}_{k},o_{k},\mathbf{a}_{k},r_{k},m_{k}\\}_{k=1}^{N_{\text{gs}}}$
(4)

$N_{\text{gs}}$ is the number of the gaussians.
$\mathbf{p}_{k},\mathbf{R}_{k},\mathbf{S}_{k},o_{k}$ is the parameters for the
shape of each Gaussian. In the first stage, we utilize $\mathbf{c}_{k}$ as
appearance to optimize the shape parameters of the Gaussians. At the second
stage, to achieve PBR materials estimation and physically based rendering, we
additionally introduce parameters albedo $\mathbf{a}_{k}\in\mathbb{R}^{3}$,
roughness $r_{k}\in\mathbb{R}$ and metallics $m_{k}\in\mathbb{R}$ to model the
decoupled appearance on each 2D Gaussian. We utilize $N_{\text{init}}=6980$
canonical SMPL template vertices and their corresponding normal vectors to
initialize the location and rotation of our canonical Gaussians. The
initialization of scaling factors is calculated in the canonical space
following [40].

Articulation via LBS. We utilize SMPL [39] as our articulation template model.
The SMPL model is defined by a function
$M(\boldsymbol{\theta},\boldsymbol{\beta}):\mathbb{R}^{|\boldsymbol{\theta}|\times|\boldsymbol{\beta}|}\rightarrow\mathbb{R}^{3N_{\text{init}}}$,
where the $\boldsymbol{\theta}\in\mathbb{R}^{3\times K}$ ($K$ denotes the
number of body joints) denotes the body pose parameters,
$\boldsymbol{\beta}\in\mathbb{R}^{10}$ denotes the coefficients for body
shape.

With the template model, the Gaussian points can be deformed to the
articulated space via linear blend skinning (LBS). For each canonical space
Gaussian point $\mathbf{p}_{k}$, we first calculate the articulation
transformation $\mathbf{A}_{k}=[\mathbf{A}_{k}^{rot},\mathbf{A}_{k}^{t}]$
based on the pose $\boldsymbol{\theta}$ where $\mathbf{A}_{k}^{rot}$ denotes
the rotation matrix and $\mathbf{A}_{k}^{t}$ denotes the translation vector
for the articulated transformation. The $\mathbf{A}_{k}$ for each Gaussian can
be calculated via the following equation:

$\mathbf{A}_{k}=\sum_{i=1}^{N_{b}}\widehat{\mathbf{W}}_{i}(\mathbf{p}_{k})\mathbf{B}_{i}$
(5)

$\mathcal{B}(\boldsymbol{\theta})=[\mathbf{B}_{1},\mathbf{B}_{2},\dots,\mathbf{B}_{N_{b}}]$
($\mathbf{B}_{i}\in SE(3)$) denotes the rigid transformation that moves each
canonical joint to the articulated space. $N_{b}$ is the number of joints in a
template model. $\widehat{\mathbf{W}}(\mathbf{p}_{k})$ denotes the blending
weight for the bones $\mathcal{B}(\boldsymbol{\theta})$. We model with
$\widehat{\mathbf{W}}(\mathbf{p}_{k})=\mathbf{W}(\mathbf{p}_{k})+\Delta\mathbf{W}(\mathbf{p}_{k})$.
$\mathbf{W}(\mathbf{p}_{k})\in R^{N_{b}}$ is the predefined skinning weight
queried in canonical space for each Gaussian, and the
$\Delta\mathbf{W}(\mathbf{p}_{k})$ is the learnable blending weight offset for
each Gaussian. We follow [40] and add this term to the predefined blending
$\mathbf{W}$ to model the actual instance deformation. We follow Instant
Avatar [35] to store the predefined skinning weight of canonical space in a
dense 3D volume. Then, we can deform the 2D Gaussians to the articulated space
via:

$\mathbf{p}_{k}^{*}=\mathbf{A}_{k}^{rot}\mathbf{p}_{k}+\mathbf{A}_{k}^{t},\quad\mathbf{R}_{k}^{*}=\mathbf{A}_{k}^{rot}\mathbf{R}_{k}.$
(6)

where $\mathbf{p}_{k}^{*}$ and $\mathbf{R}_{k}^{*}$ denotes the location and
rotation of 2D Gaussians in articulated space.

To approximate the clothing motion that can be captured from the monocular
video, we follow [40] and introduce $N_{\text{lb}}=4$ latent Bones
$\widetilde{\mathcal{B}}(\boldsymbol{\theta})$ parameterized with an MLP
conditioned on human pose $\boldsymbol{\theta}$. We also parameterized the
blending weight $\widetilde{\mathbf{W}}_{i}$ for each latent bones
$\widetilde{\mathcal{B}}_{k}(\boldsymbol{\theta})$. We therefore extend the
equation 5 to the overall articulation transformation:

$\mathbf{A}_{k}=\sum_{i=1}^{N_{b}}\widehat{\mathbf{W}}_{i}(\mathbf{p}_{k})\mathbf{B}_{i}+\sum_{i=1}^{N_{lb}}\widetilde{\mathbf{W}}_{i}(\mathbf{p}_{k})\widetilde{\mathbf{B}}_{i}$
(7)

###  3.3 Physically based rendering with image-based Lighting

Physically Based Inverse Rendering. Unlike previous physically-based cloth
avatars [29, 30], our lighting calculation does not need the time-consuming
Monte Carlo Sampling, the integration of ingoing radiance in the hemisphere.
We instead utilize the image-based lighting and split-sum approximation [101]
to tackle the intractable integral.

The rendering equation involves the calculation of the outgoing light from a
surfel at observation space $\mathbf{p}_{k}^{*}$ 111We omit $\mathbf{p}_{k}$
for simplified representation. All the
$\mathbf{L}_{o},\mathbf{L}_{i},\mathbf{n},\mathbf{a},m$ in the following
equation denotes the corresponding parameters on each surfel
$\mathbf{p}^{*}_{k}$ based on the BRDF materials, which can be written as:

$\mathbf{L}_{o}(\mathbf{w}_{o})=\int_{\Omega}f(\mathbf{w}_{o},\mathbf{w}_{i})\mathbf{L}_{i}(\mathbf{w}_{i})(\mathbf{w}_{i}\cdot\mathbf{n})d\mathbf{w}_{i}$
(8)

where $\mathbf{w}_{o}$ denotes the view direction of outgoing radiance
$\mathbf{L}_{o}(\mathbf{w}_{o})$, $\mathbf{w}_{i}$ denotes the direction of
the ingoing radiance $\mathbf{L}_{i}(\mathbf{w}_{i})$ sampled from the
hemisphere $\Omega$, $\mathbf{n}$ denotes the normal orientation of the
surfel. The BRDFs property $f$ can be divided into a specular term (the 1st
term in equation. 9) and a diffuse term (the 2nd term in equation. 9)
following the Disney-BRDF model [102]:

$f(\mathbf{w}_{o},\mathbf{w}_{i})=\frac{DFG}{4(\mathbf{w}_{i}\cdot\mathbf{n})(\mathbf{w}_{o}\cdot\mathbf{n})}+(1-m)\frac{\mathbf{a}}{\pi}$
(9)

where $D$ denotes the GGX [103] normal distribution function (NDF), $F$
denotes the Fresnel term, and $G$ denotes the geometric attenuation,
$\mathbf{a}$ is the albedo for each Gaussian, $m$ denotes the metallics for
each Gaussian. We can therefore separate the whole
$\mathbf{L}_{o}(\mathbf{w}_{o})$ into two separate parts (specular light
$\mathbf{L}_{s}$ in equation. 10 and diffuse light $\mathbf{L}_{d}$ in
equation. 11). We utilize the split-sum approximation [101] to approximate the
two terms separately. By approximation, we can separate the integral of the
light and BRDF product into two separate integrals, which we can pre-integrate
separately for rapid querying.

For the specular light term $\mathbf{L}_{s}$, we have

$\mathbf{L}_{s}\approx\int_{\Omega}\frac{DFG}{4(\mathbf{w}_{i}\cdot\mathbf{n})(\mathbf{w}_{o}\cdot\mathbf{n})}d\mathbf{w}_{i}\int_{\Omega}\mathbf{L}_{i}(\mathbf{w}_{i})D(\mathbf{w}_{i},\mathbf{w}_{o})(\mathbf{w}_{i}\cdot\mathbf{n})d\mathbf{w}_{i}$
(10)

The first term denotes the integral of the specular BRDFs. It only depends on
the $\mathbf{w}_{i}\cdot\mathbf{n}$ and the roughness $r$ and is independent
of the illumination. The second term denotes the integral of ingoing radiance
with Specular NDF $D$, and it depends on the reflection direction
$\mathbf{r}=2(\mathbf{w}_{i}\cdot\mathbf{n})\mathbf{n}-\mathbf{w}_{i}$ and
roughness $r$. It can also be pre-integrated and queried from a filtered
cubemap.

For the diffuse light term $\mathbf{L}_{d}$, we have:

$\mathbf{L}_{d}\approx\mathbf{a}(1-m)\int_{\Omega}L(\mathbf{w}_{i})\frac{\mathbf{w}_{i}\cdot\mathbf{n}}{\pi}d\mathbf{w}_{i}$
(11)

The integration term can also be efficiently calculated with pre-integration
and queried from a cubemap.

Occlusion Calculation. As there exists a lot of self-occlusion during the
animation of clothed human avatars, to achieve more realistic physically based
rendering, we should consider the occlusion. However, calculating the
occlusion with ray-tracing [51] for each Gaussian is time-consuming and
hinders rendering speed. To achieve occlusion calculation while also
maintaining the fast rendering nature of Gaussian Splatting, we proposed to
approximate the occlusion as the ambient occlusion from SMPL [39] mesh
$M(\boldsymbol{\theta},\boldsymbol{\beta})=(\mathbf{V},\mathbf{F})$. Baking
the ambient occlusion of the SMPL mesh can reduce the time for occlusion
calculation. Specifically, we first query the nearest surface points
$\mathbf{v}_{k}^{*}$ from mesh $M(\boldsymbol{\theta},\boldsymbol{\beta})$ for
each 2D Gaussian $\mathbf{p}_{k}^{*}$ at articulated space. We then utilize
the surface points $\mathbf{v}_{k}^{*}$ and its corresponding surface normal
$\mathbf{n}_{k}^{*}$ from mesh to bake the ambient occlusion for the $k$-th
Gaussian:

$AO_{k}=\frac{1}{\pi}\int
V_{\mathbf{v}_{k}^{*},\mathbf{w}}(\mathbf{n}_{k}^{*}\cdot\mathbf{w})d\mathbf{w}$
(12)

where the $V_{\mathbf{v}_{k}^{*},\mathbf{w}}\in[0,1]$ is the visibility
function at each surface points $\mathbf{v}_{k}^{*}$. The integral is
approximated by casting rays in $100$ random directions around each vertex.
The visibility of the specular light is, therefore, $1-AO_{k}$.

The overall outgoing radiance for each Gaussian is:

$\mathbf{c}_{k}^{\text{PBR}}=\mathbf{L}_{o}(\mathbf{w}_{o})=\mathbf{L}_{d}+(1-AO_{k})\mathbf{L}_{s}$
(13)

here the $\mathbf{w}_{o}$ denotes the outgoing direction, corresponding to the
view direction for each 2D gaussian in world space.

Environment Map Representation. We utilize a $6\times 64\times 64$ trainable
parameters as the environment map for Image-based lighting. We query the light
at different mip-level. The base level signifies the pre-integrated lighting
for the minimum supported roughness value. Subsequent smaller mip-levels are
derived from this base level using pre-filtering approach [101].

Figure 3:  Visualization of our PBR optimization stage. At the initial stage,
both the D-normal $\mathbf{N}$ (Normal from depth points) and the R-normal
(Rendered Splat Normal) are not accurate. After the Geometry Fix Stage
($\lambda_{\text{NC}_{2}}=0,\lambda_{\text{NC}_{1}}=1$). The Geometry is
repaired while the splat normal is still messy. We then set (
$\lambda_{\text{NC}_{1}}=0,\lambda_{\text{NC}_{2}}=1$), and finally achieve
that both the geometry and splat normal are consistent and accurate.

Physical based Rendering with splatting. We first project the 2D Gaussians
into image space with the camera parameters. We can then calculate the PBR
color $\mathbf{c}_{k}^{PBR}$ for each 2D Gaussian and apply the splatting as

$\mathbf{c}^{\text{PBR}}(x)=\sum_{i=1}\mathbf{c}_{i}^{\text{PBR}}o_{i}\mathcal{G}_{i}(\mathbf{u}(x))\prod_{j=1}^{i-1}(1-o_{j}\mathcal{G}_{j}(\mathbf{u}(x)))$
(14)

we can also get the corresponding albedo, roughness, metallic, and visibility
maps by following the same splatting process of 2DGS.

###  3.4 Reconstruct animatable PBR-aware 2DGS from monocular videos

Our training process consists of two stages. In the first stage, we apply an
Image Reconstruction loss to capture the rough shape of the dynamic humans. In
the second stage, we use Physically-Based Rendering (PBR) to estimate PBR
materials and achieve accurate surface reconstruction. We add additional
regularization terms to obtain a suitable solution for such a highly ambiguous
problem. Specifically, given $M$ frames from a monocular video and the
estimated poses of the human
$\\{(\mathbf{I}_{1},\boldsymbol{\theta}_{1}),\dots,(\mathbf{I}_{M},\boldsymbol{\theta}_{M})\\}$
we optimize
$\mathbf{G},\widehat{\mathbf{W}},\widetilde{\mathbf{B}},\widetilde{\mathbf{W}}$
with the ground-truth images.

Training Stage I – Image Reconstruction. In the first stage, we aim to
reconstruct the rough shape for clothed avatars with view-dependent color
represented as spherical harmonics. We apply the image reconstruction loss,
regularization term on these Gaussian parameters, and the normal-depth
consistency loss proposed by [33]. Denote the rendered image as
$\hat{\mathbf{I}}$, the training loss at this stage is:

$L_{\text{RF}}=\ell_{1}(\hat{\mathbf{I}},\mathbf{I})+\lambda_{\text{SSIM}}L_{\text{SSIM}}(\hat{\mathbf{I}},\mathbf{I})+L_{\text{reg}}+\lambda_{\text{NC}}L_{\text{NC}}$
(15)

we only render the foreground avatar and apply the $L_{\text{RF}}$ loss on the
foreground avatar and do not model the background. Due to Gaussians’ discrete
representation nature, without any regularization term, the reconstructed
cloth avatars may be overfitting to the sparse 2D observation. We therefore
follow previous works [40] and regularize the standard variation $\sigma_{k}$
of Gaussian attributes in the KNN neighborhood of each Gaussian
$\mathbf{G}_{k}$, which leads to:

$L^{\sigma}_{k}=\sum_{\text{attr}_{k}\in\\{\mathbf{R}_{k},\mathbf{S}_{k},o_{k},\mathbf{c}_{k},\Delta\mathbf{W}_{k}\\}}\lambda_{\text{attr}_{k}}\sigma(\text{attr}_{k})$
(16)

where $\sigma_{\text{attr}_{k}}$ denotes the standrd variation of the Gaussian
attributes in the KNN neighborhood. Furthermore, we encourage learnable
blending weight, and the latent bones for cloth animation for each Gaussian
should also be small, thus alleviating the overfitting for the cloth
deformation. We follow the previous work [35] to store the learnable blending
weight in a 3D voxel and apply a tri-linear interpolation to query the
blending weight $\Delta W_{k}$ for each Gaussian. We therefore regularize the
learnable blending weight to be small for both the SMPL bones and latent bones
with:

$L_{k}^{\text{norm}}=\lambda_{\widehat{\mathbf{W}}}\|\Delta\mathbf{W}_{k}\|_{2}+\lambda_{\widetilde{\mathbf{W}}}\|\widetilde{\mathbf{W}}_{k}\|_{2}$
(17)

The total regularization loss is:

$L_{\text{reg}}=\frac{1}{N_{\text{gs}}}\sum_{k=1}^{N_{\text{gs}}}L_{k}^{\sigma}+L_{k}^{\text{norm}}$
(18)

To ensure the orientation of each 2D Gaussians locally aligned with the actual
surface and suitable for physically based rendering. We follow 2DGS [33] and
align the splat’s normal with the gradients of the depth maps as follows:

$L_{\text{NC}}=\sum_{i}w_{i}(1-\mathbf{n}_{i}^{T}\mathbf{N})$ (19)

where
$w_{i}=o_{i}\mathcal{G}_{i}(\mathbf{u}(x))\prod_{j=1}^{i-1}(1-o_{j}\mathcal{G}_{j}(\mathbf{u}(x)))$
denotes the blending weight of the $i$-th intersection, $\mathbf{n}_{i}$ is
the normal of the splat facing the camera, $\mathbf{N}$ is normal derived from
depth map, In contrast to the $\ell_{1}$ loss proposed by previous works [47,
49], we find this normal-consistency loss can achieve more smooth surface and
consistent orientation for each gaussian along the ray, as validated in our
ablation study.

Training Stage II – PBR optimization. At the PBR stage, we first apply the
image reconstruction loss on the PBR image. Denote the rendered image of PBR
as $\mathbf{\hat{I}}_{\text{pbr}}(\mathbf{a},m,r,\mathbf{n})$, which is
computed from albedo, materials, and splat normal. We apply:

$\displaystyle
L_{\text{PBR}}=\ell_{1}(\hat{\mathbf{I}}_{\text{pbr}},\mathbf{I})+\lambda_{\mathbf{a}}L_{\text{smooth}}(\mathbf{a})+$
(20)
$\displaystyle\lambda_{r}L_{\text{smooth}}(r)+\lambda_{m}L_{\text{smooth}}(m)+$
$\displaystyle\lambda_{\text{NC}}L_{\text{NC}^{*}}+\lambda_{\text{white}}L_{\text{white}}$

Here the $L_{\text{NC}^{*}}$ is slightly different from the $L_{\text{NC}}$
used in the previous stage. With original normal-consistency loss from the
previous stage, we can not reconstruct totally accurate geometry, especially
under the extreme high light conditions as shown in Fig 3 and original 2DGS
[33] paper. At the PBR stage, the appearance of Gaussians are correlated with
the splat normal. Thus we can further improve the geometry during the
optimization. Therefore, we can not freeze the Gaussian shape during the PBR
optimization stage as proposed by GS-IR [47]. However, if we directly optimize
with Equation 19, the splat normal still needs to be consistent with the
inaccurate geometry and can not improve the geometry of clothed avatars. If we
discard this normal consistency constraint, the splat normal will be totally
messy during the optimization. Therefore, we propose a progressive
optimization strategy to achieve both accurate geometry and consistent splat
normal with following:

$\displaystyle
L_{\text{NC}^{*}}=\lambda_{\text{NC}_{1}}\sum_{i}w_{i}(1-\text{sg}(\mathbf{n}_{i}^{T})\mathbf{N})$
(21)
$\displaystyle+\lambda_{\text{NC}_{2}}\sum_{i}w_{i}(1-\mathbf{n}_{i}^{T}\mathbf{N})$

where $\text{sg}(\cdot)$ denotes the stop gradient. At the initial stage, we
set $\lambda_{\text{NC}_{2}}=0,\lambda_{\text{NC}_{1}}=1$, which means we
utilize the splat normal to optimize the geometry $\mathbf{N}$. Then we set
$\lambda_{\text{NC}_{1}}=0,\lambda_{\text{NC}_{2}}=1$ to force the splat
normal to be aligned with the fixed geometry. We visualize the splat normal
and depth derived normal during the optimization stage at Fig 3. To overcome
the ambiguity between the lighting and PBR materials and obtain a plausible
solution for PBR materials, we follow [48] to utilize a smoothness loss which
forces the materials and albedo to be smooth at the planar surfaces where the
image gradients are absent. This can be written as:

$\displaystyle
L_{\text{smooth}}(\mathbf{a})=\frac{1}{N_{\text{pixel}}}\sum_{i,j}|\partial_{\mathbf{x}}\mathbf{a}_{ij}|e^{-\|\partial_{\mathbf{x}}\mathbf{I}_{ij}\|}$
(22)
$\displaystyle+\frac{1}{N_{\text{pixel}}}\sum_{i,j}|\partial_{\mathbf{y}}\mathbf{a}_{ij}|e^{-\|\partial_{\mathbf{y}}\mathbf{I}_{ij}\|}$

where $N_{\text{pixel}}$ denotes the numbe of pixels for the image. I
represents the ground-truth images,
$\partial_{\mathbf{x}},\partial_{\mathbf{y}}$ represents the gradients of an
image or material map in $\mathbf{x}$ and $\mathbf{y}$ directions. We apply
the smoothness loss on the rendered albedo, metallic and roughness map.
Furthermore, We assume a near-natural white incident light to obtain a
plausible lighting materials disentangle. We therefore apply the white light
regularizations on the environment map as:

$L_{\text{white}}=\sum_{\text{channel}}(\mathbf{L}_{\text{channel}}-\frac{1}{3}\sum_{\text{channel}}\mathbf{L}_{\text{channel}}),{\text{channel}}\in\\{R,G,B\\}$
(23)

###  3.5 Novel pose animation under novel illuminations

After optimization, we can get the canonical PBR-aware 2DGS $\mathbf{G}$. To
achieve novel pose animation under novel illuminations, we first deform the
optimized canonical PBR-aware 2DGS to the observation space with the given
novel pose which is unseen in the training process following the equation 7.
We then use the environment map of the novel illuminations to build a cubemap.
For each viewpoint, we query the cubemap to calculate the PBR color for each
Gaussian at observation space with the equation 13. Finally, we apply the
splatting process (equation 14) to render the avatar at given novel
illuminations.

##  4 Experiments

###  4.1 Evaluation datasets

To validate the effectiveness of our proposed methods, we use a synthetic
dataset (RANA [92]) and two real-world datasets (PeopleSnapshot [104], ZJU-
MoCap [38, 105]) for performance evaluation.

RANA [92]. RANA [92] is a synthetic dataset that includes the ground-truth
albedo, normal, and relighting results at novel pose for evaluation. The
training view is a subject with an A-pose rotating in front of a camera in the
scene. The testing set includes the same subject under unknown illumination
with random poses. To quantitatively evaluate the physical material of the
reconstructed human, we use 8 subjects from the RANA dataset [92] by following
the same protocols in IntrinsicAvatar [29].

PeopleSnapshot [104] PeopleSnapshot captures subjects that hold a nearly
A-pose and rotate in front of the camera. The subjects are under natural
illumination from the real world. We follow previous works [40, 35] and select
4 subjects from the dataset for qualitative evaluation. After optimization,
our reconstructed avatar can be rendered in a novel pose under novel
illuminations.

ZJU-MoCap [38, 105] In ZJU-MoCap, the subjects captured in the video may
conduct very complex motions. The shadow effect during the complex motion
challenges our physically based rendering. We use the dataset to show that our
method can achieve human intrinsic disentangles from a monocular video with
complex human motions.

###  4.2 Baselines

We compare our method with two recently proposed methods, IntrinsicAvatar [29]
and R4D [25]. IntrinsicAvatar [29] is the most recent state-of-the-art method
with publicly available training codes for the physically based inversed
rendering of clothed humans. We also choose the Relighting4D (R4D) as our
baselines [25]. Since the original R4D implementation does not utilize the
alpha loss, we follow [29] to report a variant of R4D*222The training code of
RelightableAvatar [30] are not fully released with guidance at the current
stage, which we encountered some errors when run the code. They only release
one checkpoint on their MobileStage dataset for inference. Thus we do not
compare our method with it. that utilize an alpha loss.

###  4.3 Evaluation Metrics

Figure 4:  Qualitative Results on the RANA Dataset. We visualize the PBR
image, albedo, and normal from the training view. We then relight the clothed
human avatars at 2 different novel poses under the novel illuminations.  TABLE
I: Quantitative comparison to the baselines on the RANA dataset. The bold font
indicates the best performance. The underline indicates the second-best
performance. Our method costs minimal training time.

Method | Relighting (novel pose) |  Normal Error $\downarrow$ | Albedo | Training Time
---|---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
R4D [25] | 16.41 | 0.8023 | 0.2213 | 42.69∘ | 18.24 | 0.7780 | 0.2414 | 2 days
R4D* [25] | 18.32 | 0.8132 | 0.1913 | 27.38∘ | 18.23 | 0.8254 | 0.2043 | 2 days
IntrinsicAvatar [29] | 20.68 | 0.8818 | 0.1139 | 9.98∘ | 22.23 | 0.8862 | 0.1615 | 4 hours
Ours | 21.30 | 0.8871 | 0.1134 |  10.33∘ | 21.76 | 0.8908 | 0.1492 | 40 mins
|  |  |  |  |  |  |  |

Figure 5:  Qualitative comparisons on the PeopleSnapshot dataset. We visualize
the PBR image, albedo, and normal from the training view. We then relight the
clothed human avatars at training pose and novel pose under the novel
illumination.

We use the following metrics for performance evaluation on the synthetic
dataset RANA [92].

Albedo: PSNR/SSIM/LPIPS: We use the image quality metrics to evaluate the
albedo estimation from the training view. Since ambiguity exists between the
albedo scale and light intensity, we follow previous works [22, 29] to align
the estimated albedo with a scale of the ground-truth albedo.

Normal Estimation Error: The metric evaluates the normal errors between the
predicted normal and the ground-truth normal from the training view. We
empirically find that the rendered normal is better.

Relighting (novel pose): PSNR/SSIM/LPIPS: We render the clothed human in the
novel pose under the novel illumination from the test set. We then evaluate
the image quality between the predicted images and the ground-truth.

We only compared our method with baselines qualitatively on the real-world
datasets since there is no ground-truth albedo and material information.

###  4.4 Experimental Results

Figure 6:  Qualitative results on the ZJU-MoCap [38] dataset. In the left 6
columns, we show the rendered image, estimated albedo, roughness, metallic,
normal, and the approximated occlusion. In the right 3 columns, we visualize
the relighting results under different novel illuminations at different poses.
Figure 7:  Our clothed human avatars in novel views under novel illuminations.
The avatars are from the PeopleSnapshot [104] dataset. We visualize the
avatars from different views with different poses and under different unknown
illuminations from [106].

Training details. We first train 5,000 steps for the image reconstruction
stage. We apply the normal-depth consistency regularizer after 3,000 steps,
from which we can start to align the splat normal to the optimized geometry.
We then conduct 3,000 steps to optimize the PBR materials. The overall
training step is 8,000 steps, which consumes about 40 minutes on a single
Nvidia 3090Ti GPU (with peak 13GB GPU memory usage), which is almost 6 times
faster than IntrinsicAvatar. We set the $\lambda_{\text{NC}}=0.05$ for the
normal-consistency loss. The weights in the smooth loss
($\lambda_{\mathbf{a}},\lambda_{r},\lambda_{m}$) are set to be 0.02. The
weight of $\lambda_{\text{white}}$ is set to be 0.1. We follow the learning
rate in the original Gaussian attribute in Gaussian Splatting [32, 33]. For
the additional parameters of the PBR attribute and the learnable environment
light, we set the learning rate the same as that of the spherical harmonics
appearance. In the image reconstruction stage, the overall loss is eq. 15,
while in the PBR optimization stage, the overall loss is eq. 20.

Rendering time. The rendering time mainly depends on the number of Gaussians.
The time cost mainly comes from the occlusion calculation. For PeopleSnapshot
[104] with $540\times 540$ resolution, there are $\sim 50,000$ Gaussians after
optimization, the physically based rendering time is 0.2 seconds per image,
which is faster than all previous neural-rendering based works (20s per image
as shown in the Fig. 4).

####  4.4.1 Comparison on synthetic dataset

We compare with the baselines in Tab.I 333 We re-run the baselines [25, 29]
with the official implementation provided by the authors:
https://github.com/FrozenBurning/Relighting4D,
https://github.com/taconite/IntrinsicAvatar on our dataset splits and report
the reproduced both the quantitative and qualitative results. Our reported
relighting results are higher than those reported in previous works as we
employ a similar processing method to albedo by dilating the foreground
outward by 100 pixels. and Fig.4. We can observe that R4D* [25] cannot
reconstruct the surface geometry accurately. Since such a method relies on the
time-conditioned embedding during optimization, thus, it cannot generalize to
the novel poses. Moreover, its implicit representation makes it lack of the
details. While Intrinsic Avatar [29] can generalize to the novel pose and its
multi-level Hash-Grids [31] representation provide much more details, the
albedo details (in the man’s face on the left of Fig. 5) are still missing.
For the female on the right of Fig. 5, the dress surface reconstructed by [29]
tends to overfit the texture, while our methods can reconstruct the smooth
surface for the dress. Furthermore, our methods can bring high light during
relighting, which brings superiority for relighting (novel pose). The
quantitative results in Tab. I also demonstrate this. Compared with R4D and
R4D*, our methods reduce $75.8\%$ and $62.6\%$ normal errors and significantly
improve the PSNR of estimated albedo by $19.2\%$. Compared with
IntrinsicAvatar, our methods achieve comparable estimated albedo quality (a
lower PSNR (0.46 dB), but a higher SSIM and LPIPS) and more accurate
relighting quality (a higher PSNR (0.62 dB)). Further, compared with SOTA
methods, our methods can achieve a $5\times$ faster in training speed and a
$100\times$ faster in rendering speed.

####  4.4.2 Comparison on real-world dataset

We also compare our method with the baselines on the real-world nearly A-pose
dataset qualitatively in Fig. 5. The novel poses are selected from the AIST
dataset [107]. We can observe that the albedo estimation of R4D* [25] fails to
recover the plausible solution, its geometry is inaccurate. The relighting
results at both the training and test poses are unsuitable for the
environmental light. Compared with Intrinsic Avatar [29], our methods can
produce more plausible relighting results in addition to the $5\times$ faster
in training speed and $100\times$ faster in rendering speed.

####  4.4.3 PBR properties estimation results on the real-world datasets with
complex motions

We follow the setting in the InstantNVR [34] and choose view 4 from ZJU-MoCap
as the training view. We visualize the intrinsic estimation results in Fig. 6.
Our methods can produce plausible BRDF estimation results. We also render the
clothed human at novel illumination and get well-reproduced results. We also
visualize the approximated occlusion calculated from the template mesh to show
the effectiveness of our proposed human avatar occlusion approximation
strategy. We can observe from Fig. 6 that our method can achieve a plausible
PBR properties estimation from a monocular video with complex human motions,
and can also re-render them with an appearance suitable to the environment
lighting.

####  4.4.4 Reposed clothed human avatars in novel views under novel
illuminations

We select the motion sequence from the CAPE dataset [108, 95] and repose our
reconstructed clothed human avatars. We then select 3 different views and re-
render each avatar under 3 different environment lighting conditions. The
results in Fig. 7 demonstrate the robustness of our methods. The reconstructed
avatars from our methods can generalize well to the out-of-distribution pose
under totally novel illuminations while maintaining 3D consistency. We provide
a video in our supplementary to provide a temporally consistent visualization.

Figure 8:  Ablation studies of different training strategies. We visualize the
Albedo estimation, D-Normal (normal from depth points), and the R-Normal
(rendered normal of each splat).

###  4.5 Ablation studies

We conduct ablation studies on different representations (3DGS or 2DGS),
different normal consistency regularization($\ell_{1}$-loss or cosine
similarity loss), the effectiveness of occlusion calculation, the
effectiveness of the occlusion approximation strategies, the PBR
regularization loss, and our progressive training strategy.

####  4.5.1 Training strategy

TABLE II: Ablation study of our proposed training strategy

Method | Albedo |  Normal Error $\downarrow$
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
Freeze shape | 18.78 | 0.7661 | 0.2581 | 21.51∘
w/o prog. train | 20.21 | 0.8247 | 0.1977 | 12.89∘
Ours SGIA | 23.17 | 0.8839 | 0.1701 | 11.31∘
|  |  |  |

We first compare different training strategies for the PBR optimization stage
in the Tab. II and the Fig. 8. ”Freeze shape” means that we choose the
optimization strategy of GS-IR [47], which freeze the shape after image
reconstruction stage and only optimize the PBR materials of each Gaussian.
”w/o progressive” means that we do not utilize the strategy in (equation 21)
and directly utilize the normal-consistency loss (equation 19) from the first
stage. We can observe from the Fig. 8 that if we utilize the ”freeze shape”,
the geometry of Gaussian is fixed. The geometry from the first stage cannot
bring any improvement. If we do not utilize our proposed progressive training
strategy, the normal of each Gaussian is always enforced to align with the
inaccurate geometry in the first stage. The inaccurate geometry will hurt the
geometry optimization in the PBR stage. With our proposed progressive training
strategy, both the normal of each splat and the geometry of the avatar can be
aligned consistently to the accurate geometry. Moreover, the inaccurate
geometry also leads to inaccurate albedo estimation. The quantitative results
in Tab. II further demonstrate these.

####  4.5.2 Primitive representation

Figure 9:  Ablation studies of different primitive choices. We visualize the
rendering image, albedo, D-Normal (normal from depth points), and the R-Normal
(rendered normal of each splat).  TABLE III: Ablation study of different
primitive representation

Method | Albedo |  Normal Error $\downarrow$
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
PBR-aware 3DGS | 22.07 | 0.8739 | 0.1586 | 12.33∘
PBR-aware 2DGS | 22.43 | 0.8923 | 0.1545 | 11.71∘
|  |  |  |

We ablate different choices of primitive representation in Tab. III and Fig.
9. Since original 3DGS [32] do not have an attribute for splat orientation, we
follow the previous relightable 3DGS [47, 50] to assign a learnable normal
vector for each Gaussian and keep all the other strategy unchanged. We can
observe from Fig. 9 that with 3DGS as a primitive representation, though the
splat normal (R-normal) can be very smooth, there exists a lot of uneven
region in the Depth (D-normal). The R-normal and the D-Normal are not
consistent. However, with the 2DGS representation, the R-Normal and the
D-Normal can be both consistent and accurate. The Normal Error in Tab. III
also demonstrates this quantitatively.

####  4.5.3 Normal-consistency loss

TABLE IV: Ablation study of different consistency loss

Method | Albedo |  Normal Error $\downarrow$ | Relighting (novel pose)
---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
$\ell_{1}$-loss | 21.76 | 0.8798 | 0.1746 | 20.33∘ | 15.87 | 0.7844 | 0.2162
cosine-loss | 22.43 | 0.8923 | 0.1545 | 11.71∘ | 19.77 | 0.8642 | 0.1317
|  |  |  |  |  |  |

Figure 10:  Ablation studies of different normal-consistency loss choices. We
visualize the albedo, the D-Normal (normal from depth points), the R-Normal
(rendered normal of each splat), and the relighting results with novel poses
under a novel illumination.

We ablate the normal consistency loss in Tab IV and Fig. 10. ”$\ell_{1}$-loss”
means that we follow the previous relightable 3DGS [50, 47] and utilize the
$\ell_{1}$-loss to minimize the difference of the linear blended normal and
the normal from depth points. We keep the remaining factors unchanged. We can
observe that utilizing the $\ell_{1}$-loss, the normal of each splat cannot
align with the D-Normal well, while the inaccurate splat normal also leads to
bad relighting results as shown in the Fig. 10. The quantitative results from
Tab IV also provide evidence.

Figure 11:  Ablation study of our proposed occlusion calculation. We visualize
the Albedo and Relighting results under novel illuminations, both at novel
poses. The red rectangle indicates the coupled occlusion information in
albedo. The green rectangle indicates the decoupled occlusion information from
the albedo.  TABLE V: Ablation study of the effectiveness of the occlusion
calculation

Method | Albedo |  Normal Error $\downarrow$
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
w/o occlusion | 21.08 | 0.8616 | 0.1711 | 11.87∘
w/ occlusion | 21.31 | 0.8701 | 0.1702 | 10.69∘
|  |  |  |

####  4.5.4 Occlusion calculation

We ablate our proposed occlusion calculation in Tab. V and Fig. 11. We can
observe that the shadow information may be mixed in the albedo without the
occlusion calculation during the optimization stage. With the occlusion
calculation, our methods successfully disentangle the occlusion information
from the albedo. The quantitative results in Tab. V also prove the
effectiveness of decoupling the shadow effect and the PBR properties of our
occlusion approximation strategy.

####  4.5.5 Occlusion calculation from template mesh or clothed mesh

We ablate our approximation of the occlusion calculation from the template
mesh in Tab. VI and Fig. 12. For the fully clothed mesh case, we follow 2DGS
[33] and use the TSDF Fusion [109] to extract the mesh of the clothed human
avatar at the canonical pose. We query the points by using tri-linear
interpolation from the canonical skinning weight voxel to get the skinning
weight for each canonical vertex. We keep other regularization terms the same
as those in 2DGS and optimize the same steps.Tab. VI shows that the accuracy
of the estimated PBR properties with the occlusion calculated from the
template mesh only has a minor gap to the occlusion calculated from the fully
clothed mesh. However, the rendering time cost using the fully clothed mesh is
about $10\times$ longer than that with the template mesh. This is because the
number of vertices and faces in the fully clothed mesh can be $20\times$ more
than those in the template mesh. Fig. 12 shows that the calculated visibility
map and the estimated albedo do not have any significant difference. This
demonstrates that our proposed occlusion approximation strategy can achieve a
faster rendering speed without sacrificing the accuracy of the estimated PBR
properties.

Figure 12:  Ablation studies of different occlusion calculation strategies. At
the top of the figure, we visualize different canonical meshes used in
different occlusion calculation strategies. At the bottom of the figure, we
visualize the albedo and the visibility map of different occlusion calculation
strategies both at novel poses.  TABLE VI: Ablation study of the occlusion
calculation from the template mesh or the clothed mesh

Method | Albedo |  Rendering Time $\downarrow$
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
Full clothed mesh | 22.01 | 0.8831 | 0.1267 | 2s
Template mesh (Ours) | 21.93 | 0.8827 | 0.1259 | 0.2s
|  |  |  |

####  4.5.6 PBR regularization

We ablate our PBR regularization choices in the Tab. VII and Fig. 13. ”w/o.
white reg” means that we do not utilize the white light regularization in
Equation 23. ”w/o. smooth reg” means that we do not utilize the smooth
regularization in Equation 22. We can observe from Fig. 13 that without the
light regularization, the albedo estimation and the splat normal can be noisy.
Without the smooth regularization, the albedo estimation does not include
details (such as eyes on the face), the splat normal and the roughness are
totally messy. The roughness value varied even in the region that should have
the same material properties. The PBR rendering image may also fit concisely
with the ground-truth image. The quantitative results in Tab. VII also
demonstrate the effectiveness of regularization terms.

Figure 13:  Ablation study of PBR Regularization. We visualize the rendering
image, albedo, R-Normal (rendered normal of each splat), and the roughness.
TABLE VII: Ablation study of PBR regularization

Method | Albedo |  Rendering Time $\downarrow$
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
w/o white reg | 22.05 | 0.9045 | 0.1761 | 10.71∘
w/o smooth reg | 21.83 | 0.9046 | 0.1863 | 10.93∘
Ours Full | 22.44 | 0.8874 | 0.1758 | 9.4∘
|  |  |  |

##  5 Conclusion And Future Work

In this paper, we propose SGIA, a relightable clothed human avatar
representation that can achieve fast reconstruction of the PBR properties from
monocular videos. In contrast to all previous NeRF-based works that rely on
Monte-Carlo sampling to integrate the in-going radiance, our methods can be
easily equipped with pre-integration to achieve efficient lighting
computation. We approximate the occlusion by ray-casting from template mesh to
balance accuracy and speed. Furthermore, to achieve accurate geometry
reconstruction guided by PBR, we propose a simple but effective progressive
optimization strategy that improves the geometry and then aligns the splat
normal with the geometry. Overall, our methods can achieve five times the
training speed and 100 times the rendering speed compared with previous SOTA
approaches while maintaining high-quality PBR properties estimation and
realistic re-lighting results.

While our method efficiently reconstructs the PBR properties of clothed human
avatars from monocular videos, it is important to acknowledge its limitations.
One significant limitation is the lack of detailed facial expression capture.
Our approach does not leverage expression capture techniques, resulting in
less detailed facial animations for the avatars. Future work could enhance
realism and expressiveness by integrating facial expression techniques, as
explored in studies such as [110, 111, 112], with our full-body gesture
reconstruction. A promising direction is to combine the expressive Gaussian
Avatar representation with our full PBR representation pipeline to achieve
both relightable and expressive human avatar reconstruction.

## Acknowledgments

The authors would like to thank Umar Iqbal and Shaofei Wang for providing
details about preprocessing RANA dataset, Shaofei Wang for providing the
scripts of the camera trajectory setting for free viewpoint rendering on CAPE
dataset, Zibo Zhao and Shenhan Qian for the valuable discussion and feedbacks
at the initial stage of the project, Yahao Shi and Yanmin Wu for providing the
implementation details of GIR, Jingwei Xu for providing the guidance of the
draft pipeline figure.

## References

  * [1] J. T. Barron and J. Malik, “Shape, illumination, and reflectance from shading,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 8, pp. 1670–1687, 2014.
  * [2] Z. Li, Z. Xu, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker, “Learning to reconstruct shape and spatially-varying reflectance from a single image,” _ACM Transactions on Graphics_ , vol. 37, no. 6, pp. 269:1–269:11, 2018.
  * [3] Z. Li, M. Shafiei, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker, “Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [4] D. Lichy, J. Wu, S. Sengupta, and D. W. Jacobs, “Shape and material capture at home,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [5] S. Sang and M. Chandraker, “Single-shot neural relighting and svbrdf estimation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020.
  * [6] X. Wei, G. Chen, Y. Dong, S. Lin, and X. Tong, “Object-based illumination estimation with rendering-aware neural networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020.
  * [7] Y. Yu and W. A. P. Smith, “Inverserendernet: Learning single image inverse rendering,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [8] P. Goel, L. Cohen, J. Guesman, V. Thamizharasan, J. Tompkin, and D. Ritchie, “Shape from tracing: Towards reconstructing 3d object geometry and svbrdf material from images via differentiable path tracing,” in _Proceedings of the International Conference on 3D Vision (3DV)_ , 2020.
  * [9] K. Guo, P. Lincoln, P. L. Davidson, J. Busch, X. Yu, M. Whalen, G. Harvey, S. Orts-Escolano, R. Pandey, J. Dourgarian, D. Tang, A. Tkach, A. Kowdle, E. Cooper, M. Dou, S. R. Fanello, G. Fyffe, C. Rhemann, J. Taylor, P. E. Debevec, and S. Izadi, “The relightables: Volumetric performance capture of humans with realistic relighting,” _ACM Transactions on Graphics_ , vol. 38, no. 6, pp. 217:1–217:19, 2019.
  * [10] P.-Y. Laffont, A. Bousseau, and G. Drettakis, “Rich intrinsic image decomposition of outdoor scenes from multiple views,” _IEEE Transactions on Visualization and Computer Graphics (TVCG)_ , vol. 19, no. 2, pp. 210–224, 2013.
  * [11] H. P. Lensch, J. Lang, A. M. Sa, and H.-P. Seidel, “Planned sampling of spatially varying brdfs,” _Computer Graphics Forum_ , 2003.
  * [12] J. J. Park, A. Holynski, and S. M. Seitz, “Seeing the world in a bag of chips,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [13] J. Philip, M. Gharbi, T. Zhou, A. A. Efros, and G. Drettakis, “Multi-view relighting using a geometry-aware network,” _ACM Transactions on Graphics_ , vol. 38, no. 4, pp. 78:1–78:14, 2019.
  * [14] C. Schmitt, S. Donne, G. Riegler, V. Koltun, and A. Geiger, “On joint estimation of pose, geometry and svbrdf from a handheld scanner,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [15] X. Zhang, S. R. Fanello, Y.-T. Tsai, T. Sun, T. Xue, R. Pandey, S. Orts-Escolano, P. L. Davidson, C. Rhemann, P. E. Debevec, J. T. Barron, R. Ramamoorthi, and W. T. Freeman, “Neural light transport for relighting and view synthesis,” _ACM Transactions on Graphics_ , vol. 40, no. 1, pp. 1–17, 2021.
  * [16] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in _ECCV_ , 2020.
  * [17] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron, “Nerv: Neural reflectance and visibility fields for relighting and view synthesis,” in _CVPR_ , 2021.
  * [18] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. P. Lensch, “Nerd: Neural reflectance decomposition from image collections,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * [19] J. Knodt, J. Bartusek, S.-H. Baek, and F. Heide, “Neural ray-tracing: Learning surfaces and reflectance for relighting and view synthesis,” 2021.
  * [20] W. Yang, G. Chen, C. Chen, Z. Chen, and K.-Y. K. Wong, “Ps-nerf: Neural inverse rendering for multi-view photometric stereo,” in _European Conference on Computer Vision (ECCV)_ , 2022.
  * [21] M. Boss, V. Jampani, R. Braun, C. Liu, J. T. Barron, and H. P. Lensch, “Neural-pil: Neural pre-integrated lighting for reflectance decomposition,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [22] K. Zhang, F. Luan, Q. Wang, K. Bala, and N. Snavely, “PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting,” in _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [23] X. Zhang, P. P. Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and J. T. Barron, “Nerfactor: Neural factorization of shape and reflectance under an unknown illumination,” _ACM Transactions on Graphics (ToG)_ , vol. 40, no. 6, pp. 1–18, 2021.
  * [24] H. Jin, I. Liu, P. Xu, X. Zhang, S. Han, S. Bi, X. Zhou, Z. Xu, and H. Su, “Tensoir: Tensorial inverse rendering,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * [25] Z. Chen and Z. Liu, “Relighting4d: Neural relightable human from videos,” in _ECCV_ , 2022.
  * [26] W. Lin, C. Zheng, J.-H. Yong, and F. Xu, “Relightable and animatable neural avatars from videos,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2024, pp. 3486–3494.
  * [27] W. Sun, Y. Che, H. Huang, and Y. Guo, “Neural reconstruction of relightable human model from monocular video,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 397–407.
  * [28] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” _arXiv preprint arXiv:2106.10689_ , 2021.
  * [29] S. Wang, B. Antić, A. Geiger, and S. Tang, “Intrinsicavatar: Physically based inverse rendering of dynamic humans from monocular videos via explicit ray tracing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2024.
  * [30] Z. Xu, S. Peng, C. Geng, L. Mou, Z. Yan, J. Sun, H. Bao, and X. Zhou, “Relightable and animatable neural avatar from sparse-view video,” in _CVPR_ , 2024.
  * [31] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _ACM Trans. Graph._ , vol. 41, no. 4, pp. 102:1–102:15, Jul. 2022. [Online]. Available: https://doi.org/10.1145/3528223.3530127
  * [32] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” _ACM Transactions on Graphics_ , vol. 42, no. 4, July 2023. [Online]. Available: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * [33] B. Huang, Z. Yu, A. Chen, A. Geiger, and S. Gao, “2d gaussian splatting for geometrically accurate radiance fields,” _SIGGRAPH_ , 2024.
  * [34] C. Geng, S. Peng, Z. Xu, H. Bao, and X. Zhou, “Learning neural volumetric representations of dynamic humans in minutes,” in _CVPR_ , 2023.
  * [35] T. Jiang, X. Chen, J. Song, and O. Hilliges, “Instantavatar: Learning avatars from monocular video in 60 seconds,” _arXiv_ , 2022.
  * [36] X. Chen, T. Jiang, J. Song, M. Rietmann, A. Geiger, M. J. Black, and O. Hilliges, “Fast-snarf: A fast deformer for articulated neural fields,” _Pattern Analysis and Machine Intelligence (PAMI)_ , 2023.
  * [37] X. Chen, Y. Zheng, M. J. Black, O. Hilliges, and A. Geiger, “Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes,” in _International Conference on Computer Vision (ICCV)_ , 2021.
  * [38] S. Peng, Y. Zhang, Y. Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou, “Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans,” in _CVPR_ , 2021.
  * [39] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “SMPL: A skinned multi-person linear model,” _ACM Trans. Graphics (Proc. SIGGRAPH Asia)_ , vol. 34, no. 6, pp. 248:1–248:16, Oct. 2015.
  * [40] J. Lei, Y. Wang, G. Pavlakos, L. Liu, and K. Daniilidis, “Gart: Gaussian articulated template models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * [41] Z. Li, Z. Zheng, L. Wang, and Y. Liu, “Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * [42] S. Zheng, B. Zhou, R. Shao, B. Liu, S. Zhang, L. Nie, and Y. Liu, “Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * [43] S. Hu and Z. Liu, “Gauhuman: Articulated gaussian splatting for real-time 3d human rendering,” _arXiv preprint_ , 2023.
  * [44] Z. Qian, S. Wang, M. Mihajlovic, A. Geiger, and S. Tang, “3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting,” in _CVPR_ , 2024.
  * [45] Y. Yuan, X. Li, Y. Huang, S. De Mello, K. Nagano, J. Kautz, and U. Iqbal, “Gavatar: Animatable 3d gaussian avatars with implicit mesh learning,” _arXiv preprint arXiv:2312.11461_ , 2023.
  * [46] H. Pang, H. Zhu, A. Kortylewski, C. Theobalt, and M. Habermann, “Ash: Animatable gaussian splats for efficient and photoreal human rendering,” _arXiv preprint arXiv:2312.05941_ , 2023.
  * [47] Z. Liang, Q. Zhang, Y. Feng, Y. Shan, and K. Jia, “Gs-ir: 3d gaussian splatting for inverse rendering,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * [48] Y. Shi, Y. Wu, C. Wu, X. Liu, C. Zhao, H. Feng, J. Liu, L. Zhang, J. Zhang, B. Zhou, E. Ding, and J. Wang, “Gir: 3d gaussian inverse rendering for relightable scene factorization,” _Arxiv_ , 2023.
  * [49] Y. Jiang, J. Tu, Y. Liu, X. Gao, X. Long, W. Wang, and Y. Ma, “Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces,” _arXiv preprint arXiv:2311.17977_ , 2023.
  * [50] J. Gao, C. Gu, Y. Lin, H. Zhu, X. Cao, L. Zhang, and Y. Yao, “Relightable 3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing,” _arXiv:2311.16043_ , 2023.
  * [51] L. Bolanos, S.-Y. Su, and H. Rhodin, “Gaussian shadow casting for neural characters,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * [52] M. Zwicker, J. Rasanen, M. Botsch, C. Dachsbacher, and M. Pauly, “Perspective accurate splatting,” in _Proceedings-Graphics Interface_ , 2004, pp. 247–254.
  * [53] S. Bi, Z. Xu, P. Srinivasan, B. Mildenhall, K. Sunkavalli, M. Hašan, Y. Hold-Geoffroy, D. Kriegman, and R. Ramamoorthi, “Neural reflectance fields for appearance acquisition,” _arXiv_ , vol. 2008.03824, 2020.
  * [54] M. Boss, V. Jampani, K. Kim, H. Lensch, and J. Kautz, “Two-shot spatially-varying brdf and shape estimation,” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 3982–3991.
  * [55] A. Gardner, C. Tchou, T. Hawkins, and P. Debevec, “Linear light source reflectometry,” _ACM Transactions on Graphics_ , vol. 22, no. 3, pp. 749–758, 2003.
  * [56] A. Ghosh, T. Chen, P. Peers, C. A. Wilson, and P. Debevec, “Estimating specular roughness and anisotropy from second order spherical gradient illumination,” _Computer Graphics Forum_ , vol. 28, no. 4, pp. 1161–1170, 2009.
  * [57] D. Guarnera, G. C. Guarnera, A. Ghosh, C. Denk, and M. Glencross, “Brdf representation and acquisition,” in _Proceedings of the 37th Annual Conference of the European Association for Computer Graphics: State of the Art Reports_ , 2016, pp. 625–650.
  * [58] M. Haindl and J. Filip, _Visual Texture_. Springer-Verlag, 2013.
  * [59] M. Weinmann and R. Klein, “Advances in geometry and reflectance acquisition (course notes),” in _SIGGRAPH Asia Courses_ , 2015.
  * [60] Y. Yao, J. Zhang, J. Liu, Y. Qu, T. Fang, D. McKinnon, Y. Tsin, and L. Quan, “Neilf: Neural incident light field for physically-based material estimation,” in _European Conference on Computer Vision (ECCV)_ , 2022.
  * [61] M. Boss, A. Engelhardt, A. Kar, Y. Li, D. Sun, J. T. Barron, H. P. Lensch, and V. Jampani, “SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * [62] Y. Zhang, J. Sun, X. He, H. Fu, R. Jia, and X. Zhou, “Modeling indirect illumination for inverse rendering,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 18 643–18 652.
  * [63] Y. Liu, S. Peng, L. Liu, Q. Wang, P. Wang, T. Christian, X. Zhou, and W. Wang, “Neural rays for occlusion-aware image-based rendering,” in _CVPR_ , 2022.
  * [64] Y. Zhang, T. Xu, J. Yu, Y. Ye, J. Wang, Y. Jing, and W. Yang, “Nemf: Inverse volume rendering with neural microflake field,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2023.
  * [65] V. Sitzmann, J. N. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” in _Proc. NeurIPS_ , 2020.
  * [66] A. Chen, Z. Xu, A. Geiger, and H. Su, “Tensorf: Tensorial radiance fields,” in _European Conference on Computer Vision (ECCV)_ , 2022.
  * [67] Y. Liu, P. Wang, C. Lin, X. Long, J. Wang, L. Liu, T. Komura, and W. Wang, “Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images,” in _TOG_ , 2023.
  * [68] S. Mao, C. Wu, Z. Shen, and L. Zhang, “Neus-pir: Learning relightable neural surface using pre-integrated rendering,” _arXiv preprint arXiv:2306.07632_ , 2023.
  * [69] K. Zhang, F. Luan, Z. Li, and N. Snavely, “Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2022.
  * [70] Z. Yang, Y. Chen, X. Gao, Y. Yuan, Y. Wu, X. Zhou, and X. Jin, “Sire-ir: Inverse rendering for brdf reconstruction with shadow and illumination removal in high-illuminance scenes,” _arXiv preprint arXiv:2310.13030_ , 2023.
  * [71] H. Wang, W. Hu, L. Zhu, and R. Lau, “Inverse rendering of glossy objects via the neural plenoptic function and radiance fields,” in _CVPR_ , 2024.
  * [72] T. Shen, J. Gao, K. Yin, M.-Y. Liu, and S. Fidler, “Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [73] J. Munkberg, J. Hasselgren, T. Shen, J. Gao, W. Chen, A. Evans, T. Müller, and S. Fidler, “Extracting Triangular 3D Models, Materials, and Lighting From Images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 8280–8290.
  * [74] J. Hasselgren, N. Hofmann, and J. Munkberg, “Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising,” _arXiv:2206.03380_ , 2022.
  * [75] Z. Yu, T. Sattler, and A. Geiger, “Gaussian opacity fields: Efficient high-quality compact surface reconstruction in unbounded scenes,” _arXiv preprint arXiv:2404.10772_ , 2024.
  * [76] P. Dai, J. Xu, W. Xie, X. Liu, H. Wang, and W. Xu, “High-quality surface reconstruction using gaussian surfels,” in _SIGGRAPH 2024 Conference Papers_. Association for Computing Machinery, 2024.
  * [77] S. Peng, J. Dong, Q. Wang, S. Zhang, Q. Shuai, X. Zhou, and H. Bao, “Animatable neural radiance fields for modeling dynamic human bodies,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 314–14 323.
  * [78] L. Liu, M. Habermann, V. Rudnev, K. Sarkar, J. Gu, and C. Theobalt, “Neural actor: Neural free-view synthesis of human actors with pose control,” _ACM SIGGRAPH Asia_ , 2021.
  * [79] Y. Zhi, S. Qian, X. Yan, and S. Gao, “Dual-space nerf: Learning animatable avatars and scene lighting in separate spaces,” in _International Conference on 3D Vision (3DV)_ , Sep. 2022.
  * [80] C.-Y. Weng, B. Curless, P. P. Srinivasan, J. T. Barron, and I. Kemelmacher-Shlizerman, “HumanNeRF: Free-viewpoint rendering of moving people from monocular video,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 16 210–16 220.
  * [81] Y. Huang, H. Yi, Y. Xiu, T. Liao, J. Tang, D. Cai, and J. Thies, “TeCH: Text-guided Reconstruction of Lifelike Clothed Humans,” in _International Conference on 3D Vision (3DV)_ , 2024.
  * [82] Y. Huang, H. Yi, W. Liu, H. Wang, B. Wu, W. Wang, B. Lin, D. Zhang, and D. Cai, “One-shot implicit animatable avatars with model-based priors,” in _IEEE Conference on Computer Vision (ICCV)_ , 2023.
  * [83] S. Wang, K. Schwarz, A. Geiger, and S. Tang, “Arah: Animatable volume rendering of articulated human sdfs,” in _European Conference on Computer Vision_ , 2022.
  * [84] Y. Xiao, J. Xu, Z. Yu, and S. Gao, “Debsdf: Delving into the details and bias of neural indoor scene reconstruction,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2024.
  * [85] L. Hu, H. Zhang, Y. Zhang, B. Zhou, B. Liu, S. Zhang, and L. Nie, “Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * [86] J. Wen, X. Zhao, Z. Ren, A. Schwing, and S. Wang, “GoMAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh,” in _CVPR_ , 2024.
  * [87] H. Kim, M. Jang, W. Yoon, J. Lee, D. Na, and S. Woo, “Switchlight: Co-design of physics-driven architecture and pre-training framework for human portrait relighting,” _arXiv preprint arXiv:2402.18848_ , 2024.
  * [88] R. Pandey, S. Orts Escolano, C. Legendre, C. Haene, S. Bouaziz, C. Rhemann, P. Debevec, and S. Fanello, “Total relighting: learning to relight portraits for background replacement,” _ACM Transactions on Graphics (TOG)_ , vol. 40, no. 4, pp. 1–21, 2021.
  * [89] T. Sun, J. T. Barron, Y.-T. Tsai, Z. Xu, X. Yu, G. Fyffe, C. Rhemann, J. Busch, P. Debevec, and R. Ramamoorthi, “Single image portrait relighting,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 4, pp. 1–12, 2019.
  * [90] Y.-Y. Yeh, K. Nagano, S. Khamis, J. Kautz, M.-Y. Liu, and T.-C. Wang, “Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation,” _ACM Transactions on Graphics (TOG)_ , vol. 41, no. 6, pp. 1–21, 2022.
  * [91] S. Saito, G. Schwartz, T. Simon, J. Li, and G. Nam, “Relightable gaussian codec avatars,” in _CVPR_ , 2024.
  * [92] U. Iqbal, A. Caliskan, K. Nagano, S. Khamis, P. Molchanov, and J. Kautz, “Rana: Relightable articulated neural avatars,” in _ICCV_ , 2023.
  * [93] D. Luvizon, V. Golyanik, A. Kortylewski, M. Habermann, and C. Theobalt, “Relightable neural actor with intrinsic decomposition and pose control,” _arXiv_ , 2023.
  * [94] S. Qian, J. Xu, Z. Liu, L. Ma, and S. Gao, “Unif: United neural implicit functions for clothed human reconstruction and animation,” in _European Conference on Computer Vision_. Springer, 2022, pp. 121–137.
  * [95] G. Pons-Moll, S. Pujades, S. Hu, and M. Black, “Clothcap: Seamless 4d clothing capture and retargeting,” _ACM Transactions on Graphics, (Proc. SIGGRAPH)_ , vol. 36, no. 4, 2017. [Online]. Available: http://dx.doi.org/10.1145/3072959.3073711
  * [96] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “AMASS: Archive of motion capture as surface shapes,” in _International Conference on Computer Vision_ , Oct. 2019, pp. 5442–5451.
  * [97] Z. Li, Y. Sun, Z. Zheng, L. Wang, S. Zhang, and Y. Liu, “Animatable and relightable gaussians for high-fidelity human avatar modeling,” _arXiv preprint arXiv:2311.16096v4_ , 2024.
  * [98] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of StyleGAN,” in _Proc. CVPR_ , 2020.
  * [99] Y. Zheng, Q. Zhao, G. Yang, W. Yifan, D. Xiang, F. Dubost, D. Lagun, T. Beeler, F. Tombari, L. Guibas, and G. Wetzstein, “Physavatar: Learning the physics of dressed 3d avatars from visual observations,” _arXiv preprint arXiv:2404.04421_ , 2024.
  * [100] S. Hu and Z. Liu, “Gauhuman: Articulated gaussian splatting from monocular human videos,” _arXiv preprint arXiv:_ , 2023.
  * [101] B. Karis and E. Games, “Real shading in unreal engine 4,” in _Proceedings of Physically Based Shading Theory Practice_ , vol. 4, no. 3, 2013, p. 1.
  * [102] B. Burley and W. D. A. Studios, “Physically-based shading at disney,” in _ACM SIGGRAPH_ , vol. 2012, 2012, pp. 1–7.
  * [103] B. Walter, S. R. Marschner, H. Li, and K. E. Torrance, “Microfacet models for refraction through rough surfaces,” in _Proceedings of the 18th Eurographics Conference on Rendering Techniques_ , ser. EGSR’07. Goslar, DEU: Eurographics Association, 2007, p. 195–206.
  * [104] T. Alldieck, M. Magnor, W. Xu, C. Theobalt, and G. Pons-Moll, “Video based reconstruction of 3d people models,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun 2018, pp. 8387–8397.
  * [105] Q. Fang, Q. Shuai, J. Dong, H. Bao, and X. Zhou, “Reconstructing 3d human pose by watching humans in the mirror,” in _CVPR_ , 2021.
  * [106] Poly Haven, “Hdris,” 2024, accessed: 2024-06-13. [Online]. Available: https://polyhaven.com/hdris
  * [107] R. Li, S. Yang, D. A. Ross, and A. Kanazawa, “Learn to dance with aist++: Music conditioned 3d dance generation,” 2021.
  * [108] Q. Ma, J. Yang, A. Ranjan, S. Pujades, G. Pons-Moll, S. Tang, and M. J. Black, “Learning to Dress 3D People in Generative Clothing,” in _Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [109] Q.-Y. Zhou, J. Park, and V. Koltun, “Open3d: A modern library for 3d data processing,” 2018, cite arxiv:1801.09847Comment: http://www.open3d.org. [Online]. Available: http://arxiv.org/abs/1801.09847
  * [110] Y. Xu, P. Chandran, S. Weiss, M. Gross, G. Zoss, and D. Bradley, “Artist-friendly relightable and animatable neural heads,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * [111] S. Qian, T. Kirschstein, L. Schoneveld, D. Davoli, S. Giebenhain, and M. Nießner, “Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians,” _arXiv preprint arXiv:2312.02069_ , 2023.
  * [112] S. Saito, G. Schwartz, T. Simon, J. Li, and G. Nam, “Relightable gaussian codec avatars,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.