#  Towards Realistic Generative 3D Face Models

Aashish Rai1 Hiresh Gupta∗1 Ayush Pandey∗1 Francisco Vicente Carrasco1 Shingo
Jason Takagi2 Amaury Aubel2 Daeil Kim2 Aayush Prakash2 Fernando De la Torre1
1Carnegie Mellon University 2Meta Reality Labs
https://aashishrai3799.github.io/Towards-Realistic-Generative-3D-Face-Models

###### Abstract

††* equal contribution

In recent years, there has been significant progress in 2D generative face
models fueled by applications such as animation, synthetic data generation,
and digital avatars. However, due to the absence of 3D information, these 2D
models often struggle to accurately disentangle facial attributes like pose,
expression, and illumination, limiting their editing capabilities. To address
this limitation, this paper proposes a 3D controllable generative face model
to produce high-quality albedo and precise 3D shape leveraging existing 2D
generative models. By combining 2D face generative models with semantic face
manipulation, this method enables editing of detailed 3D rendered faces. The
proposed framework utilizes an alternating descent optimization approach over
shape and albedo. Differentiable rendering is used to train high-quality
shapes and albedo without 3D supervision. Moreover, this approach outperforms
most state-of-the-art (SOTA) methods in the well-known NoW and REALY
benchmarks for 3D face shape reconstruction. It also outperforms the SOTA
reconstruction models in recovering rendered faces’ identities across novel
poses by an average of $10\%$. Additionally, the paper demonstrates direct
control of expressions in 3D faces by exploiting latent space leading to text-
based editing of 3D faces.

Figure 1: 3D generative face model. a) High-resolution 3D shape and albedo
recovered from a StyleGAN2 generated image. Novel views can be rendered using
the estimated face model. b) Editing of 3D faces with text. This method allows
for 3D expression manipulation through guidance with the CLIP model.

##  1 Introduction

The success of language models like GPT-3 [41], and more recently, the release
of text-to-image models like GLIDE [35], DALLE-2 [43], or Imagen [45] have all
contributed to the enormous popularity of generative AI. Besides generating
images with unprecedented visual quality, these models also show remarkable
generalization ability to novel texts with complex compositions of concepts,
making them generalists for image synthesis. In the context of faces,
StyleGAN2 [28] has the capability of generating powerful face images that are
frequently indistinguishable from reality. While these 2D generative models
create high-quality faces for many applications of interest, such as facial
animation [27, 55], expression transfer [30, 52, 36] virtual avatars [34],
these 2D models often encounter difficulties when it comes to effectively
disentangle facial attributes like pose, expression, and illumination. As a
result, their capacity to edit such attributes is limited. Moreover, a 3D
representation (shape, texture) is crucial to many entertainment
industries—including games, animation, and visual effects— that are demanding
3D content at increasingly enormous scales to create immersive virtual worlds.
Recall that many applications of interest require 3D assets that are
consumable by a graphics engine (e.g., Unity [54], Unreal [11]).

To address this demand, recently, researchers have proposed generative models
to generate 3D faces [1, 50, 19]. Even though these algorithms perform well,
the lack of diverse and high-quality 3D training data has limited the
generalization of these algorithms and their use in real-world applications
[53]. Another line of research involves using parametric models like 3DMM[3],
BFM[39], FLAME[3], and derived methods [15, 32, 10, 51, 46] to approximate the
3D geometry and texture of a 2D face image. While these 3D face reconstruction
techniques can reasonably recover low-frequency details, they typically do not
recover high-frequency details. Also, predicting high-resolution texture maps
that capture details remains an unaddressed problem. Most of the works
focusing in this direction either emphasize mesh or texture. However, a
generative 3D face model that can generate both high-quality texture and a
detailed mesh with the same quality as 2D models is still missing.

This paper proposes a 3D generative model for faces using a self-supervised
approach that can generate high-resolution texture and capture high-frequency
details in the geometry. The method leverages a pretrained StyleGAN2 to
generate high-quality 2D faces (see Fig. 1 a). We propose a network,
AlbedoGAN, that generates light-independent albedo directly from StyleGAN2’s
latent space. For the shape component, the FLAME model [33] is combined with
per-vertex displacement maps guided by StyleGAN’s latent space, resulting in a
higher-resolution mesh. The two networks for albedo and shape are trained in
alternating descent fashion. The proposed method outperforms SOTA methods in
shape estimation, such as DECA [15] and MICA [61], by $20\%$ and $1.1\%$,
respectively. It’s worth noting that MICA only generates a neutral and frontal
smooth mesh, while the proposed algorithm can generate any expression. Fig.
1(a) shows how an image generated by StyleGAN2 can be uplifted to 3D with a
detailed shape and albedo, being able to render realistic 3D faces. Finally,
given the 3D face asset, our algorithm can edit the face in 3D. For example,
Fig. 1(b) illustrates expression manipulation through text-based editing
guided by the CLIP model [40]. Briefly stated, our main contributions are:

1\. A self-supervised method to leverage StyleGAN2 into a 3D generative model
producing high-quality albedo and a detailed mesh. We introduce AlbedoGAN, a
single-pass albedo prediction network that generates high-resolution albedo
and decouples illumination using shading maps.

2\. We show that our model outperforms existing methods in capturing high-
frequency facial details in a mesh. Moreover, the proposed method reconstructs
3D faces that recover identity better than SOTA methods.

3\. We propose a displacement map generator capable of decoding per-vertex
displacements directly from StyleGAN’s latent space using detailed normals of
the mesh.

4\. Since our entire architecture can generate 3D faces from StyleGAN2’s
latent space, we can perform face editing directly in the 3D domain using the
latent codes or text.

##  2 Related Work

Reconstructing a 3D Face from a single 2D image is an ill-posed problem that
has intrigued researchers for decades. Blanz et al. [3] took the first
significant step in this direction when they introduced 3D Morphable Models
(3DMM) [3] in 1999 as general face representation. Their work inspired decades
of work in estimating parameters for 3DMM to find a textured mesh that best
fits an input 2D face. While estimating parameters for parametric models like
3DMM and its advanced versions like FLAME [33] has been the bulk of the focus
for researchers, there has also been a good amount of work done in learning
volumetric representations (e.g., NeRF) for a face. In particular, the focus
has been on using Neural Implicit Representation [24, 17, 59, 44, 5, 60] to
learn density and radiance to represent a face. However, it has been widely
noticed that they are prone to generating artifacts and consume a lot of time
to render these detailed representations. Furthermore, these methods do not
generate a topologically uniform mesh, and therefore do not directly serve
applications in graphic engines, face animation, avatar creation, etc. Due to
the above-mentioned reasons, we do not consider implicit representation in our
research. In the following sections, we describe work that is related to
individual components of our framework.

Figure 2: AlbedoGAN. Pose-invariant albedo, $\textbf{A}_{GT}$, obtained by
texture extraction and synthesis 3.1, is used to train StyleGAN2 generator,
$\textbf{G}_{Ab}$, for a given latent code $\textbf{z}/\textbf{w}$. We use a
3DMM fitting, image blending, and Spherical Harmonics to extract
$\textbf{A}_{GT}$. $\textbf{D}_{real}$ and $\textbf{D}_{id}$ are introduced to
generate realistic images and identity consistent albedo, respectively.

###  2.1 Texture Generation for 3D Mesh

Most of the work done in synthesizing texture for a mesh can be broadly
divided into two parts: using a parametric texture model like 3DMM or Basel
Face Model (BFM) [15, 10, 32, 51] or a GAN-based approach [18, 20, 21, 23, 56,
31, 22, 7] to generate texture. Using a parametric model BFM [39], works like
[15, 10, 32, 51] learn an encoder to predict the parameters that generate a
texture that best fits the visible part of the face. Since the model uses
representation in low dimensional PCA-based space, they generate an
approximate texture that often lacks high-frequency details, which correspond
to low variation direction in the projected latent space and do not lead to
photo-realistic rendering. Recently, with the advent of GANs, there are works
that leverage them to extract texture [18, 20, 31]. TBGAN [20] trains a
Progressive GAN to generate a high-quality texture with a differentiable
renderer in a supervised setting. On the other hand, OSTeC [18] uses 3DMM as
initialization to generate a textured mesh and then uses GAN inversion with
StyleGAN2 to generate multiple views of this mesh and extract texture. This is
an extremely time-consuming and takes several minutes per image. We propose
AlbedoGAN, which is able to generate high-quality texture in a single pass in
a time-efficient manner. AlbedoGAN generates textures that maintain identity
over multiple poses - where most previous methods struggle.

###  2.2 3D Shape prediction from 2D Image

Similar to texture, Blanz et al.’s seminal work [3] can represent a face mesh
in a low-dimensional PCA-based space. This led to the development of a huge
corpus of work [10, 51, 46, 32] in 3D face reconstruction focused on learning
an encoder that could predict parameters for generating shapes using 3DMM,
given a 2D image. The encoder could be learned in a self-supervised way with
2D image losses [15, 32, 10, 51, 46], or with 3D supervision, [61]. After
3DMM[3], there have been newer parametric models BFM [39] and FLAME [33],
which have been learned from more subjects and encode the structure of the
face better. The explosive development of 3D Face Reconstruction led to the
creation of NoW benchmark [46] as a common ground for comparison across 3D
Face reconstruction methods. Currently, DECA [15], and MICA [61] show best
reconstruction on NoW Benchmark. DECA’s architecture is inspired by RingNet
[46]. However, instead of 3DMM, DECA uses FLAME [33], and it adds an encoder-
decoder network that learns to generate displacement maps to learn animatable
details in UV space. MICA [61], the current state-of-art model, leverages
ArcFace backbone [8] to learn the actual shape of the face by regressing it on
a high-quality 3D face scan dataset[6, 16, 4].

##  3 AlbedoGAN Training

Albedo constitutes one of the crucial parts of a 3D face model, since face
appearance is largely dictated by it. To generate high quality 3d models, we
need to generate albedo that generalize over pose, age, and ethnicity.
However, training such a diverse albedo generative model requires a massive
database of 3D scans, which is neither cost nor time effective. An efficient
way of extracting textures from existing 2D images is fitting a 3DMM and
capturing a UV mapped texture. However, this ”pseudo” texture does not
generalize well over poses nor disentangle shadows. In this paper, we leverage
3DMM fitting, image blending, and Spherical Harmonics lighting to capture
high-quality $1024\times 1024$ resolution albedo that generalizes well over
different poses and tackles shading variations.

This section describes albedo training, refer Fig. 2 for an overview. The
first step includes texture extraction and correction, 3.1, followed by the
use of a spherical harmonics model to extract albedo from texture (Section
3.2). Section 3.3 explains training AlbedoGAN - a StyleGAN2 model to generate
albedo corresponding to the given latent code
$\textbf{w}\in\mathbb{R}^{18\times 512}$.

###  3.1 Texture Extraction and Correction

First, we establish a correspondence between the input 2D image
$\textbf{I}\in\mathbb{R}^{w\times h\times 3}$ and the UV domain by taking
orthogonal projection of a mesh fitted on the given image using a 3DMM [33].
Using this correspondence, we get the RGB values for the UV texture and
perform barycentric interpolation to fill out the missing pixels.

Now, texture correction is performed to fill the occluded areas by leveraging
pose information. This happens by projecting the flipped input image and
fitted mesh, and collecting the pixels corresponding to the missing parts.
These pixels are blended to the original texture to get a complete pose-
invariant texture.

###  3.2 Albedo from Texture

As shown in Fig. 2, the next step includes obtaining an albedo and shading map
from the unevenly illuminated texture. Following previous works [15, 10], we
made the following assumptions: (1) The illumination model is Spherical
Harmonics (SH), (2) light source is distant and monochromatic, and (3) surface
reflectance is Lambertian. The shading map can thus be calculated as

$\textbf{S}_{ij}=\sum_{b=1}^{9}\textbf{L}_{b}\textbf{H}_{b}(\textbf{N}_{ij})$

where, $\textbf{H}_{b}:\mathbb{R}^{3}\rightarrow\mathbb{R}$ are the SH basis
function, $\textbf{L}_{b}\in\mathbb{R}^{3}$ are SH coefficients, and
$N_{ij}\in\mathbb{R}^{3}$ are surface normals. The relation between albedo
$\textbf{A}\in\mathbb{R}^{w\times h\times 3}$, texture
$\textbf{T}\in\mathbb{R}^{w\times h\times 3}$, and shading map
$\textbf{S}\in\mathbb{R}^{2}$ can then be defined as
$\textbf{T}_{ij}=\textbf{A}_{ij}\odot\textbf{S}_{ij}$, where, $\odot$ is the
Schur product.

Figure 3: Overview of our generative model. The AlbedoGAN generator,
$\textbf{G}_{Ab}$, is used to synthesize albedo $\textbf{A}_{AbGAN}$
corresponding to a latent code w. $\textbf{G}_{StlGAN}$ generates a 2D image,
$\textbf{I}_{StlGAN}$, given to the shape model, $\textbf{E}_{shape}$, to get
a detailed mesh, M’. Finally, a differentiable renderer (DR) is used to
synthesize $\textbf{I}_{ren}$ from the mesh $\textbf{M}^{\prime}$, albedo
$\textbf{A}_{AbGAN}$, lighting l, and pose p. Losses between
$\textbf{I}_{ren}$ and $\textbf{I}_{StlGAN}$ are used to train the shape model
and the AlbedoGAN via Alternating Descent.

###  3.3 Training

In this section, we address the AlbedoGAN model training procedure. Later, the
resulting model will be fine-tuned during the shape and displacement map
training process, taking into account geometry and more sophisticated Phong
illumination model.

Our approach, AlbedoGAN, is built upon a generative model that can synthesize
face images corresponding to a latent vector, to this end we selected
StyleGAN2. This model is best suited to our requirements as it uses a mapping
network $M_{L}$ that maps an input noise vector
$\textbf{z}\in\mathbb{R}^{512}$ to an intermediate latent vector
$\textbf{w}\in\mathbb{R}^{18\times 512}$. This mapping,
$\textbf{w}=M_{L}(\textbf{z})$, adds the ability for manipulation and better
projection. Consequently, we use w as latent space for AlbedoGAN.

Hence, face images are generated by randomly sampling w using a pretrained
StyleGAN2. The generated images act as input, $\textbf{I}_{StlGAN}$, see
Fig.2. The same latent codes, w, are used in the AlbedoGAN generator, to
produce $\textbf{A}_{AbGAN}$. As shown in Fig.2, $\textbf{I}_{StlGAN}$ is
passed through the texture extraction, 3.1 and albedo extraction, 3.2, steps
to obtain $\textbf{A}_{GT}$. This albedo is used as a ground truth for the
training of the real/fake discriminator, $\textbf{D}_{real}$. Additionally, we
constraint AlbedoGAN to generate identity consistent albedos by introducing an
identity discriminator, $\textbf{D}_{id}$. To this intent, we use the features
of a pretrained face recognition model [26] $F:\mathbb{R}^{w^{\prime}\times
h^{\prime}\times 3}\rightarrow\mathbb{R}^{512}$. Our identity loss is defined
as cosine distance between the identity features of predicted albedo
$\textbf{A}_{AbGAN}$ and $\textbf{A}_{GT}$ as:

$L_{id}(\textbf{A}_{AbGAN},\textbf{A}_{GT})=1-\frac{F(\textbf{A}_{AbGAN}).F(\textbf{A}_{GT})}{||F(\textbf{A}_{AbGAN})||_{2}||F(\textbf{A}_{GT})||_{2}}$
(1)

##  4 Alternating Descent in Albedo and Shape

In this section, we describe our regression method to 3D shape given a face
image and the Differentiable Rendering, DR, based approach to fine-tune
AlbedoGAN. This fine-tuning process for AlbedoGAN takes into account
expression, camera pose, and the Phong illumination model.

Unfortunately, jointly optimizing all the components (shape, albedo,
illumination, etc.) that produce the best rendered face that is consistent
with the input image is computationally expensive. Thus, we propose using
Alternating Descent for optimization. First, we optimize the shape for a few
iterations while freezing AlbedoGAN. Next, AlbedoGAN is fine-tuned using the
updated 3D shape, with more detailed normals of the shape. This alternating
optimization cycle is repeated throughout the course of the training process.
Next, we first describe albedo optimization, 4.1, followed by shape
optimization, 4.2.

###  4.1 Albedo optimization

To fine-tune AlbedoGAN using the information of the 3D shape and illumination
model, we first assume we have an estimate of a detailed 3D shape
$\textbf{M}^{\prime}$. As shown in Fig. 3, given an estimated mesh
$\textbf{M}^{\prime}$, predicted albedo $\textbf{A}_{AbGAN}$, pose p, and
light l, we can generate a detail rendered image $\textbf{I}_{ren}$ using DR,
$R$ as:

$\textbf{I}_{ren}=R(\textbf{M}^{\prime},\textbf{A}_{AbGAN},\textbf{p},\textbf{l})$

The overall loss function $\mathcal{L}$ is defined as a sum of the following
terms:

$\mathcal{L}=\lambda_{sym\\_rec}\mathcal{L}_{sym\\_rec}+\lambda_{id}\mathcal{L}_{id}+\lambda_{perc}\mathcal{L}_{perc}+\lambda_{lmk}\mathcal{L}_{lmk}$

Where each loss is defined as follows:

Symmetric Reconstruction Loss, $\mathcal{L}_{sym\\_rec}$: A simple supervision
function that encourages low-level similarity in the predicted image and the
corresponding ground truth and symmetry in the estimated albedo. We use the
Mean Squared Error (MSE) to calculate reconstruction error.

$\begin{multlined}\mathcal{L}_{sym\\_rec}(\textbf{I}_{ren},\textbf{I}_{StlGAN})=\\\
\textrm{MSE}(\textbf{I}_{StlGAN},\textbf{I}_{ren})+\underbrace{\textrm{MSE}(\textbf{I}^{{}^{\prime}}_{StlGAN},\textbf{I}^{{}^{\prime}}_{ren})}_{\text{
Albedo symmetry consistency
term}}\end{multlined}\mathcal{L}_{sym\\_rec}(\textbf{I}_{ren},\textbf{I}_{StlGAN})=\\\
\textrm{MSE}(\textbf{I}_{StlGAN},\textbf{I}_{ren})+\underbrace{\textrm{MSE}(\textbf{I}^{{}^{\prime}}_{StlGAN},\textbf{I}^{{}^{\prime}}_{ren})}_{\text{
Albedo symmetry consistency term}}$

Where $\textbf{I}^{{}^{\prime}}_{StlGAN}$ is the flipped ground truth obtained
through StyleGAN2, and $\textbf{I}^{{}^{\prime}}_{ren}$ is the rendered
estimated image through flipping $\textbf{A}_{AbGAN}$, pose p, and light l.

Identity Loss, $\mathcal{L}_{id}(\textbf{I}_{ren},\textbf{I}_{StlGAN})$: This
loss term is introduced with the intent of making the AlbedoGAN generator
learn to match the identity of the rendered face, $\textbf{I}_{ren}$, with the
ground truth, $\textbf{I}_{StlGAN}$. As in section 3.3, we use a pretrained
face recognition model [26] for feature extraction. The cosine distance, eq.
1, is used to calculate the identity loss while fine-tuning the model.

Perceptual Loss, $\mathcal{L}_{perc}$: This perceptual loss is introduced with
the goal of forcing AlbedoGAN to generate a $\textbf{A}_{AbGAN}$ that matches
the visual appearance of $\textbf{I}_{StlGAN}$. Motivated by the existing
research, we selected a VGG16 based feature extractor [47], a pretrained face
recognition model. We use the output of $relu3\\_3$ as the image features. The
loss is calculated by the L2 distance between the feature vectors from
$\textbf{I}_{ren}$ and $\textbf{I}_{StlGAN}$.

Landmark Loss, $\mathcal{L}_{lmk}$: AlbedoGAN is also fine tuned-using using
68 facial landmarks detected on the ground truth and the rendered image to
avoid misaligned generations. We used a SOTA face landmark detection [57] to
predict 68 landmarks on $\textbf{I}_{StlGAN}$ and $\textbf{I}_{ren}$. The loss
is calculated using MSE between the two set of landmarks.

###  4.2 Shape Model and Optimization

We proceed with the shape optimization, while freezing AlbedoGAN. We sample a
latent vector $\textbf{w}\in\mathbb{R}^{18\times 512}$ and use a pretrained
StyleGAN2 model, $\textbf{G}_{StlGAN}$, to generate a 2D face image,
$\textbf{I}_{StlGAN}$, and AlbedoGAN generator, $\textbf{G}_{AbGAN}$, to
generate albedo, $\textbf{A}_{AbGAN}$. Figure 3 describes the detailed
architecture of our shape model, $\textbf{E}_{shape}$. We leverage ArcFace
backbone [9] to predict the face shape (s), expression ($\psi$), lighting (l),
and camera pose (p) parameters for the given image $\textbf{I}_{StlGAN}$. The
shape embedding vector $\textbf{s}\in\mathbb{R}^{300}$, pose
$\textbf{p}\in\mathbb{R}^{6}$, and expression $\psi\in\mathbb{R}^{50}$
parameters are fed into a parametric face model that gives us a coarse mesh
representation ($\textbf{M}_{c}$) as described below:

$\textbf{M}_{c}(s,\psi)=\textbf{T}+\textbf{B}_{S}\textbf{s}+\textbf{B}_{\psi}\psi$
(2)

where $\textbf{M}_{c}$ represents the generated coarse mesh synthesized by a
3DMM decoder. Specifically, we use FLAME [33] as our mesh decoder, which
generates a coarse mesh with $N=5023$ vertices. This coarse mesh is computed
by using a template mesh, $\textbf{T}\in\mathbb{R}^{3N}$, representing a mean
human face and different principal components
$\textbf{B}_{s}\in\mathbb{R}^{3N\times 300}$ and
$\textbf{B}_{\psi}\in\mathbb{R}^{3N\times 50}$ corresponding to the shape and
expression terms respectively.

To capture high-frequency details in meshes, we learn a displacement generator
$\textbf{G}_{Disp}$ to augment the coarse mesh, $\textbf{M}_{c}$, with a
detailed UV displacement map $\textbf{D}_{Geom}\in[-0.01,0.01]^{n\times n}$.
Recent research [2, 58] has shown that StyleGAN’s latent space contains
information about high-frequency details of a face. Using this insight, we
predict displacement maps to capture expression and pose dependent per vertex
offsets. The latent code w is the same as used in AlbedoGAN and the StyleGAN2
model. Finally, we combine the displacement map along the vertex normals of
the mesh $\textbf{M}_{c}$ to get a detailed mesh $\textbf{M}^{\prime}$ by
adding them in the UV domain.

We use the detailed mesh, $\textbf{M}^{\prime}$, along with the predicted pose
p, light l parameters, and the synthesized albedo $\textbf{A}_{AbGAN}$ to
render an image $\textbf{I}_{ren}$ as described below:

$\textbf{I}_{ren}=R(\textbf{M}_{c},\textbf{A}_{AbGAN},\textbf{p},\textbf{l})$
(3)

We apply multiple 2D image-based losses, including identity loss, perceptual
loss, and landmark loss between $\textbf{I}_{StlGAN}$ and $\textbf{I}_{ren}$
to optimize the mesh representation in a self-supervised fashion. In addition
to the losses, we also calculate a shape center loss eq. 4 on images belonging
to identity $i$. In particular, eq. 4 tries to reduce the distance between
shape vector for all the images and their corresponding mean $\mu_{i}$.
Besides this, we also perform L2 regularization, eq. 5, on predicted shape s,
expression $\psi$, and displacement maps $\textbf{D}_{Geom}$ that enforce a
prior distribution towards the mean face.

$\mathcal{L}_{sc}=\sum_{i=0}^{N}\sum_{k=0}^{K}\left\|\textbf{s}_{i,k}-\mu_{i}\right\|^{2}_{2}$
(4)

$\mathcal{L}_{reg}={\lVert\textbf{s}\rVert}^{2}_{2}+{\lVert\psi\rVert}^{2}_{2}+{\lVert\textbf{D}_{Geom}\rVert}^{2}_{F}$
(5)

The overall loss function $\mathcal{L}$ is defined as a weighted sum:

$\mathcal{L}=\lambda_{id}\mathcal{L}_{id}+\lambda_{perc}\mathcal{L}_{perc}+\lambda_{lmk}\mathcal{L}_{lmk}+\lambda_{sc}\mathcal{L}_{sc}+\lambda_{reg}\mathcal{L}_{reg}$

##  5 Experiments

This section describes the implementation details, quantitative, and
qualitative evaluation of the shape and texture reconstruction models, along
with SOTA comparison.

###  5.1 Dataset

We randomly sampled $100K$, $512$-dimensional random vectors
$\textbf{z}\in\mathbb{R}^{512}$ from a Gaussian distribution and generated the
corresponding $\textbf{w}\in\mathbb{R}^{18\times 512}$ from the StyleGANs
mapping network as $\textbf{w}=M_{L}(\textbf{z})$. These intermediate latent
vectors w are used to generate $100K$ images $\in\mathbb{R}^{1024\times
1024\times 3}$ from a pretrained StyleGAN2 [29] generator. To ensure diversity
in the generated images across ethnicity, expression, age, and pose; we
followed the work in [42]. The texture-preprocessing step (as described in
Sec. 3.1) is used to get the complete GT-albedo corresponding to all the
samples in the dataset.

To train our shape model with 2D images, we chose $30K$ of the previously
sampled z vectors. Then we perform latent space editing in the w space to
generate a total of $11$ images (belonging to different expressions and poses)
per identity using StyleGAN2 implementation [42] of InterFaceGAN [48]. We
estimate $68$ landmarks on all the GT images using the FAN [57] landmark
detection.

###  5.2 Implementation Details

Albedo Generation: PyTorch [37] is used as the implementation framework on
CUDA enabled system with NVIDIA RTX $A4500$ GPUs. We use the PyTorch
implementation of StyleGAN2 111https://github.com/rosinality/stylegan2-pytorch
for albedo and image generation. To generate face images from StyleGAN2, we
use the official pretrained weights trained on the FFHQ dataset for
$1024\times 1024$ resolution. We use Adam optimizer to train the AlbedoGAN
with learning rate $\alpha_{gen}=\alpha_{disc}=2e^{-3}$ and
$\beta_{1}=0,\beta_{2}=0.99$. The generator was regularized after every $4$
iteration, while the discriminator after every $16$ training iterations.

In the first step, we train the AlbedoGAN from GT-albedo and use the GAN loss
and ID loss eq.1 for supervision. Similar to StyleGAN2, for the GAN loss, we
use the element-wise Softplus, which can be defined as
$Softplus(x)=\frac{1}{\beta}*\log(1+\exp^{(\beta*x)})$. The $\mathcal{L}_{ID}$
is calculated between the predicted albedo from the generator
$\textbf{A}_{AbGAN}$ and the GT-albedo $\textbf{A}_{GT}$. The $\lambda_{ID}$
was set to $1$ during this training. We trained the model on $8$ GPUs, and it
took around $32$ hours for complete training (batch size $32$).

This gave us a robust albedo generator capable of generating a pose-invariant
albedo corresponding to a given $\textbf{z}\in\mathbb{R}^{512}$ or
$\textbf{w}\in\mathbb{R}^{18\times 512}$.

Figure 4: (a) Randomly generated albedo from AlbedoGAN. (b) Patches of
randomly generated albedo. AlbedoGAN can generate high-quality albedo of $1K$
resolution.

Optimizing Shape: We train our shape model, $\textbf{E}_{shape}$, on the
synthetically generated images by StyleGAN2 capturing multiple images of the
same identity across varying expression & pose. We run a face detector [25] on
the input images and scale the face crops to a resolution of $224\times 224$
before passing them to our shape model. The shape model consist of an ArcFace
backbone that is initialized to the weights learned by [61] and a convolution-
styled decoder ($\textbf{G}_{Disp}$) respectively. The whole pipeline is
optimized using Adam Optimizer with a learning rate of $1e^{-4}$. The final
loss is calculated between rendered images $\textbf{I}_{ren}$ and GT
$\textbf{I}_{StlGAN}$, where $\lambda_{id}$ is set to $0.5$, and
$\lambda_{perc}$, $\lambda_{lmk}$, $\lambda_{sc}$ and $\lambda_{reg}$ are set
to $1,5,1$ and $1e^{-4}$ respectively.

Figure 5: Randomly generated coarse mesh, detailed mesh and rendered faces
from our model, for input 2D faces.

Fine-tuning AlbedoGAN using DR: Once we have a pretrained AlbedoGAN and a good
shape estimator, we now fine-tune the AlbedoGAN. This makes the albedo
generator learn to capture more details from the GT-face. PyTorch3D is used as
the differentiable renderer in all our experiments under this section. We kept
using $\mathcal{L}_{gan}$ from previous AlbedoGAN training but gave more
gravity to the rendering losses. The rendering loss is calculated between
$\textbf{I}_{StlGAN}$ and $\textbf{I}_{ren}$ along with a symmetric
reconstruction loss between $\textbf{A}_{GT}$ and $\textbf{A}_{AbGAN}$ to
maintain low-level consistency in the fine-tuned albedo. $\lambda_{sym\\_rec}$
and $\lambda_{ID}$ were set to $0.1$ for albedo based losses.
$\lambda_{sym\\_rec}$, $\lambda_{ID}$, $\lambda_{perc}$ and $\lambda_{lmk}$
were set to $1,1,1,0.5$ for rendering losses, respectively. We fine-tuned the
model for another $48$ hours, keeping the batch size of $8$. The results of
randomly generated textures from our AlbedoGAN are shown in Fig. 4.

Once we fine-tune the AlbedoGAN, we again train the shape model with updated
albedo weights and repeat this step multiple times until we get a final model
that can synthesize realistic-looking 3D faces corresponding to the given 2D
images.

Fig. 1(a), 5, 6 shows the reconstructed 3D faces generated from our model for
multiple poses. It is interesting to see how our model generalizes well over
different poses and generates realistic-looking 3D faces. Section B.3 in
supplementary demonstrates lighting and pose control in rendered faces using
corresponding parameters and shading maps. Some additional results on testing
our pipeline on real-world images using GAN inversion can be seen in
supplementary section C.

###  5.3 Evaluation of Shape and Texture

####  5.3.1 NoW Benchmark - shape reconstruction

NoW benchmark [46] is a standard benchmark to evaluate the accuracy of 3D
meshes estimated from 2D images. It consists of $2054$ images for $100$ test
subjects across different expressions, poses, and occlusions, split across two
sets for validation (20 subjects) and test (80 subjects). NoW provides 3D
ground truth meshes for each test subject, and the predicted mesh is rigidly
aligned with the ground truth mesh using 3D face landmarks. The per-vertex
error is then used for all the subjects to compute the mean, median, and
standard deviation of the errors. Table LABEL:table:recons_error depicts the
comparison of our model with the current SOTA methods, including DECA and
MICA. Fig. 6 illustrates the visual comparison among these methods. Our model
outperforms the DECA model by achieving a 23% better median error in coarse
mesh and a 20% better median error in the detailed mesh. Our approach can
reconstruct realistic-looking rendered faces, and model accurate head shapes,
especially for faces with big heads. We also observe an improvement over the
MICA model that was trained on 3D face scans [6, 16, 4] with 2300 subjects on
the NoW validation set. As illustrated in 6, our method produces a more
detailed mesh, capturing wrinkles, expression, pose and head shape correctly
by only training on synthetic images.

Table 1: Reconstruction error on the NoW Benchmark.

Method | Median (mm) | Mean (mm) | Std (mm)
---|---|---|---
Validation Set |  |  |
Deep3D [10] | 1.286 | 1.864 | 2.361
DECA [15] | 1.178 | 1.464 | 1.253
MICA [61] | 0.913 | 1.130 | 0.948
Ours | 0.903 | 1.122 | 0.957
Test Set |  |  |
Deep3D [10] | 1.11 | 1.41 | 1.21
DECA [15] | 1.09 | 1.38 | 1.18
Ours | 0.97 | 1.21 | 1.02
MICA [61] | 0.90 | 1.11 | 0.92

###  5.4 REALY 3D Benchmark

We also evaluated our method on the most recent REALLY benchmark [14] for
single-image 3D face reconstruction from frontal and side view images. Our
results, as presented in Tables 2, demonstrate a significant improvement over
DECA by 15% and MICA by 18%. Our method also stands in the top 3 (out of 18
methods) on the REALLY benchmark challenge outperforming most of the existing
methods. The only two methods better than ours are HRN [12] and Deep3D [13].
However, they both generate only frontal mesh, while our method generates a
complete head model and is trained in an unsupervised setting.

Table 2: Single image reconstruction error on REALY Benchmark for Frontal and
Side-view images (lower is better).

Method | @nose | @mouth | @forehead | @cheek | @all
---|---|---|---|---|---
Front View |  |  |  |  |
HRN | 1.722 | 1.357 | 1.995 | 1.072 | 1.537
Deep3D | 1.719 | 1.368 | 2.015 | 1.528 | 1.657
Ours | 1.656 | 2.087 | 2.102 | 1.141 | 1.746
GANFit | 1.928 | 1.812 | 2.402 | 1.329 | 1.868
DECA | 1.697 | 2.516 | 2.394 | 1.479 | 2.010
PRNet | 1.923 | 1.838 | 2.429 | 1.863 | 2.013
EMOCA | 1.868 | 2.679 | 2.426 | 1.438 | 2.103
MICA | 1.585 | 3.478 | 2.374 | 1.099 | 2.134
RingNet | 1.934 | 2.074 | 2.995 | 2.028 | 2.258
Side View |  |  |  |  |
HRN | 1.642 | 1.285 | 1.906 | 1.038 | 1.468
Deep3D | 1.749 | 1.411 | 2.074 | 1.528 | 1.691
Our | 1.576 | 2.218 | 2.142 | 1.112 | 1.762
PRNet | 1.868 | 1.856 | 2.445 | 1.960 | 2.032
DECA | 1.903 | 2.472 | 2.423 | 1.630 | 2.107
EMOCA | 1.867 | 2.636 | 2.448 | 1.548 | 2.125
MICA | 1.525 | 2.636 | 2.448 | 1.548 | 2.125
RingNet | 1.921 | 1.994 | 3.081 | 2.027 | 2.256

Table 3: Diversity values of randomly generated meshes for various methods.
Higher is better.

- | Deep3D [10] | DECA [15] | MICA [61] | Ours
---|---|---|---|---
DIV | 0.18 | 1.13 | 0.21 | 1.67

####  5.4.1 Diversity metrics

One of the important features of a good 3D face model is how diverse it’s
synthesized meshes are. Similar to prior works that generate 3D meshes [1,
50], we measure global diversity as the mean vertex distance over all possible
pairs of $n$ meshes. Table 3 reports the diversity values for $n=1000$ meshes
synthesized by MICA[61], DECA[15] and Deep3D[10].

As illustrated in Fig. 6, our model captures head shapes better than DECA
model which produces similar-looking head shapes. We observe a significant
improvement in diversity statistics over MICA that only predicts a smooth and
neutral mesh and was trained on limited 3D data.

Figure 6: 3D Face Reconstruction comparison with DECA[15] & MICA [61]. DECA is
unable to model head shape accurately and provides a smooth texture. MICA only
outputs a smooth, neutral and frontal mesh. Our method produces a more
detailed mesh with photo-realistic texture, capturing wrinkles, expressions
and head shape accurately.

####  5.4.2 Evaluation of Rendered Faces

We evaluate the quality of texture by comparing qualitatively to a recently
proposed method, OSTeC [18]. We also quantitatively compare texture and
rendered faces by rendering it on a mesh with other methods, including
LiftedGAN [49], DECA [15], and OSTeC [18].

Fig. 7 shows the visual comparison between OSTeC [18] and our model. OSTeC
uses latent optimization based GAN inversion which adds random artifacts in
the generated image like the beard appearing on some patches of the face.
Furthermore, this leads to inconsistency while stitching textures across
different poses.

For quantitative evaluation of our texture and rendered faces, we randomly
sampled 1K images from a pretrained StyleGAN2 generator. We use the same set
of z to generate rendered 3D faces using our method. The corresponding 2D face
images are used to perform 3D reconstruction using other methods[15, 18, 49].
To compare how well the texture preserves identity, we measure identity
similarity between the input image and the corresponding 3D faces rendered at
multiple poses, including the original pose, $0$°, $\pm 15$°, $\pm 30$°, and
$\pm 45$°. The observations can be found in Table 4. As inferred from the
table, our method not only performs better in capturing the identity for the
front poses but also for a large number of side poses. More evaluations can be
found in supplementary section A.

Method | same pose |  $0$° |  $\pm 15$° |  $\pm 30$° |  $\pm 45$°
---|---|---|---|---|---
LiftedGAN [49] | 0.90 | 0.88 | 0.854 | 0.83 | 0.804
DECA [15] | 0.869 | 0.799 | 0.76 | 0.69 | 0.63
OSTeC [18] | 0.952 | 0.939 | 0.921 | 0.906 | 0.88
Ours | 0.999 | 0.995 | 0.988 | 0.972 | 0.941

Table 4: ID similarity comparison between the input image and the
corresponding 3D face rendered at various poses. Figure 7: Qualitative
comparison of synthesized texture and rendered results with OSTeC [18]. OSTeC
produces a smooth texture by stitching multiple images, often leading to
artifacts. Our approach can synthesize a better textured mesh outperforming
OSTeC in preserving the details.  Figure 8: Generating different expressions
by manipulating $\textbf{w}\in\mathbb{R}^{18\times 512}$. The first column in
each row is the neutral expression corresponding to randomly sampled w. The
subsequent columns show different expressions for the same identity by varying
w.

###  5.5 3D face manipulation

Once our end-to-end pipeline is trained and is able to generate albedo and
mesh corresponding to a $\textbf{z}\in\mathbb{R}^{512}$ or
$\textbf{w}\in\mathbb{R}^{18\times 512}$, it opens up an avenue for latent
space manipulations in 3D rendered faces. This section shows examples of
editing 3D faces directly from the latent space or text.

Latent space manipulation: Given a latent code
$\textbf{w}\in\mathbb{R}^{18\times 512}$, we can get the modified latent code
$\textbf{w}_{edit}\in\mathbb{R}^{18\times 512}$ by
$\textbf{w}_{edit}=\textbf{w}+\alpha\textbf{n}$, where
$\textbf{n}\in\mathbb{R}^{512}$ is a vector orthogonal to the semantic
boundary and $\alpha\in\mathbb{R}$ is a constant. More details on latent space
manipulations using semantic boundaries can be found in [48, 42]. Fig. 8 shows
the result of latent space manipulations for generating different expressions
of the same identity.

Text based 3D Face Editing: Similar to latent space manipulations, we can
perform text-based 3D face editing using Contrastive Language-Image
Pretraining (CLIP) models [40]. We used StyleCLIP [38] model to enable text-
based editing. The text-guided latent optimization and latent residual mapper
strategies of StyleCLIP are implemented in $\textbf{w}\in\mathbb{R}^{18\times
512}$ space and make it easy for us to plug into our pipeline. This paper
shows results for the text-guided latent optimization approach, which can be
trained for random text queries. Given a w for a face, StyleCLIP can produce
an updated latent code $\textbf{w}^{\prime}\in\mathbb{R}^{18\times 512}$
corresponding to a given text query and input w. Fig. 1(c) illustrates some
sample results generated for the text-based 3D face editing. Refer to
supplementary Section B.5 for more results on text-based editing.

##  6 Conclusion and Future Work

In this paper, we attempt to develop a high-quality 3D face generation
pipeline. We propose AlbedoGAN that synthesizes albedo and generalizes well
over multiple poses capturing intrinsic details of the face. Our approach
generates meshes that capture high-frequency details like face wrinkles.
Comprehensive experiments demonstrate superiority of our method over others in
predicting detailed mesh and preserving the identity in reconstructed 3D
faces. As a consequence of using StyleGAN2 based pipeline, we bring style
editing, semantic face manipulations, and text-based editing in 3D faces.

While our pipeline can generate high-quality 3D faces from StyleGAN2’s latent
space, some issues still need to be addressed. Mesh-based representations are
unable to model details like hair. We foresee exploiting topologically uniform
mesh, and a NeRF-based approach should be able to capture such facial
features. We’ll extend our work to incorporate more complex illumination
models.

## References

  * [1] Victoria Fernández Abrevaya, Adnane Boukhayma, Stefanie Wuhrer, and Edmond Boyer.  A decoupled 3d facial shape model by adversarial training.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9419–9428, 2019.
  * [2] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Only a matter of style: Age transformation using a style-based regression model.  ACM Transactions on Graphics (TOG), 40(4):1–12, 2021.
  * [3] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces, 1999.
  * [4] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  Facewarehouse: A 3d facial expression database for visual computing.  IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013.
  * [5] Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5799–5809, June 2021.
  * [6] Hang Dai, Nick Pears, William Smith, and Christian Duncan.  Statistical modeling of craniofacial shape and texture.  International Journal of Computer Vision, 128(2):547–571, 2020\.
  * [7] Jiankang Deng, Shiyang Cheng, Niannan Xue, Yuxiang Zhou, and Stefanos Zafeiriou.  Uv-gan: Adversarial facial uv map completion for pose-invariant face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7093–7102, 2018.
  * [8] Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In CVPR, 2019.
  * [9] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  * [10] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In IEEE Computer Vision and Pattern Recognition Workshops, 2019\.
  * [11] Epic Games.  Unreal engine.
  * [12] Biwen et al.  A hierarchical representation network for accurate and detailed face reconstruction from in-the-wild images.  In CVPR, 2023.
  * [13] Yu et al.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In CVPRW, 2019.
  * [14] Zenghao et al.  Realy: Rethinking the evaluation of 3d face reconstruction.  In ECCV, 2022.
  * [15] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  In ACM Transactions on Graphics, (Proc. SIGGRAPH), volume 40, 2021\.
  * [16] Zhen-Hua Feng, Patrik Huber, Josef Kittler, Peter Hancock, Xiao-Jun Wu, Qijun Zhao, Paul Koppen, and Matthias Rätsch.  Evaluation of dense 3d reconstruction from 2d face images in the wild.  In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 780–786. IEEE, 2018.
  * [17] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8649–8658, 2021.
  * [18] Baris Gecer, Jiankang Deng, and Stefanos Zafeiriou.  Ostec: One-shot texture completion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7628–7638, June 2021.
  * [19] Baris Gecer, Alexandros Lattas, Stylianos Ploumpis, Jiankang Deng, Athanasios Papaioannou, Stylianos Moschoglou, and Stefanos Zafeiriou.  Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks.  In European conference on computer vision, pages 415–433. Springer, 2020.
  * [20] Baris Gecer, Alexander Lattas, Stylianos Ploumpis, Jiankang Deng, Athanasios Papaioannou, Stylianos Moschoglou, and Stefanos Zafeiriou.  Synthesizing coupled 3d face modalities by trunk-branch generative adversarial networks.  In Proceedings of the European conference on computer vision (ECCV), 2020.
  * [21] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou.  Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [22] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou.  Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1155–1164, 2019.
  * [23] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou.  Fast-ganfit: Generative adversarial network for high fidelity 3d face reconstruction.  arXiv preprint arXiv:2105.07474, 2021.
  * [24] Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies.  Neural head avatars from monocular rgb videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18653–18664, 2022.
  * [25] Jia Guo, Jiankang Deng, Alexandros Lattas, and Stefanos Zafeiriou.  Sample and computation redistribution for efficient face detection.  arXiv preprint arXiv:2105.04714, 2021.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [27] Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly.  Dynamic 3d avatar creation from hand-held video input.  ACM Transactions on Graphics (ToG), 34(4):1–14, 2015.
  * [28] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [29] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [30] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt.  Deep video portraits.  ACM Transactions on Graphics (TOG), 37(4):1–14, 2018.
  * [31] Jongyoo Kim, Jiaolong Yang, and Xin Tong.  Learning high-fidelity face texture completion without complete face texture.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13990–13999, 2021.
  * [32] Chunlu Li, Andreas Morel-Forster, Thomas Vetter, Bernhard Egger, and Adam Kortylewski.  To fit or not to fit: Model-based face reconstruction and occlusion segmentation from weak supervision.  In arXiv preprint arXiv:2106.09614, 2021.
  * [33] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4D scans.  ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), 2017\.
  * [34] Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh.  Deep appearance models for face rendering.  ACM Transactions on Graphics (ToG), 37(4):1–13, 2018.
  * [35] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  ICML, 2022.
  * [36] Nicolas Olivier, Kelian Baert, Fabien Danieau, Franck Multon, and Quentin Avril.  Facetunegan: Face autoencoder for convolutional expression transfer using neural generative adversarial networks.  arXiv preprint arXiv:2112.00532, 2021.
  * [37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [38] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085–2094, 2021.
  * [39] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.  A 3d face model for pose and illumination invariant face recognition.  In 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pages 296–301, 2009.
  * [40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [41] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [42] Aashish Rai, Clara Ducher, and Jeremy R Cooperstock.  Improved attribute manipulation in the latent space of stylegan for semantic face editing.  In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 38–43. IEEE, 2021.
  * [43] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [44] Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi.  Lolnerf: Learn from one look.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1558–1567, 2022.
  * [45] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  NeurIPS, 2022.
  * [46] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [47] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [48] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  In TPAMI, 2020.
  * [49] Yichun Shi, Divyansh Aggarwal, and Anil K Jain.  Lifting 2d stylegan for 3d-aware face generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6258–6266, 2021.
  * [50] Fariborz Taherkhani, Aashish Rai, Quankai Gao, Shaunak Srivastava, Xuanbai Chen, Fernando de la Torre, Steven Song, Aayush Prakash, and Daeil Kim.  Controllable 3d generative adversarial face model via disentangling shape and appearance.  arXiv preprint arXiv:2208.14263, 2022.
  * [51] Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Pérez, and Christian Theobalt.  Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction.  CoRR, abs/1703.10580, 2017.
  * [52] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner.  Face2face: Real-time face capture and reenactment of rgb videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2387–2395, 2016.
  * [53] Mukhiddin Toshpulatov, Wookey Lee, and Suan Lee.  Generative adversarial networks and their application to 3d face generation: a survey.  Image and Vision Computing, 108:104119, 2021.
  * [54] Unity Technologies.  Unity.
  * [55] Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly.  Realtime performance-based facial animation.  ACM transactions on graphics (TOG), 30(4):1–10, 2011.
  * [56] Xiangnan Yin, Di Huang, Zehua Fu, Yunhong Wang, and Liming Chen.  Weakly-supervised photo-realistic texture generation for 3d face reconstruction.  arXiv preprint arXiv:2106.08148, 2021.
  * [57] Xi Yin, Ying Tai, Yuge Huang, and Xiaoming Liu.  Fan: Feature adaptation network for surveillance face recognition and normalization.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [58] Yin Yu, Ghasedi Kamran, Wu HsiangTao, Yang Jiaolong, Tong Xi, and Fu Yun.  Expanding the latent space of stylegan for real face editing.  arXiv preprint arXiv:2204.12530, 2022.
  * [59] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges.  Im avatar: Implicit morphable head avatars from videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13545–13555, 2022.
  * [60] Yiyu Zhuang, Hao Zhu, Xusen Sun, and Xun Cao.  Mofanerf: Morphable facial neural radiance field.  arXiv preprint arXiv:2112.02308, 2021.
  * [61] Wojciech Zielonka, Timo Bolkart, and Justus Thies.  Towards metrical reconstruction of human faces.  In European Conference on Computer Vision, 2022.


2021

[1,2]Yuri A. Kulchitsky

1] Joint Institute for Nuclear Research, Dubna, Russia 2] B.I. Stepanov
Institute of Physics, National Academy of Sciences, Minsk, Belarus

#  Moments of multiplicity distributions for KNO scaling study using the ATLAS
results

<EMAIL_ADDRESS>Pavel Tsiareshka  [  [

###### Abstract

The normalised order-  $\ignorespaces q$ moments of primarily charged-particle
multiplicity distributions are studied for KNO scaling investigation in
$\ignorespaces pp$ collisions as deduced from the results of the ATLAS at the
LHC. The normalised moments for the LHC and low-energy experiments are
compared for the kinematic region with an absolute pseudorapidity less than
$\ignorespaces 2.5$ . The normalised moments show a increases linearly with
centre-of-mass energies, and therefore the KNO scaling is violated for the
full-scaled multiplicity region. The normalised moments for scaled
multiplicity greater than one average multiplicity are constant for the
highest centre-of-mass energies, and therefore the KNO scaling is concluded to
hold.

###### keywords:

proton-proton interaction, minimum-bias events, multiplicity distributions,
normalised moments, KNO scaling, ATLAS at LHC

The investigation of charged-particle distributions in proton-proton (
$\ignorespaces pp$ ) collisions probes the strong interaction at the low-
momentum transfer (non-perturbative region of quantum chromodynamics). The
study of normalised order-  $\ignorespaces q$ moments,  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ , of primary charged-particle multiplicity
distributions is sensitive to the KNO scaling.

The KNO scaling hypothesis means that at energy asymptotic, the probability
distributions  $\ignorespaces P(n,\sqrt{s})$ of producing  $\ignorespaces n$
particles in a certain collision process should demonstrate a scaling relation
Polyakov:1970lyy ; Koba:1972ng ; Proceedings:1973mza . The main assumption of
the KNO scaling is Feynman scaling Feynman:1969ej , where it was concluded
that for asymptotically large centre-of-mass (CM) energies with
$\ignorespaces\sqrt{s}\rightarrow\infty$ the mean total number of any kind of
particle logarithmically rises with the CM energy as  $\ignorespaces\langle
n\rangle\propto\ln{\sqrt{s}}$ . For this assumption, the multiplicity
distribution  $\ignorespaces P(n,\sqrt{s})$ was represented as

$P(n,\sqrt{s})=\frac{1}{\langle
n(\sqrt{s})\rangle}\Psi(z)+\mathcal{O}\left(\frac{1}{\langle
n(\sqrt{s})\rangle^{2}}\right),$ (1)

where  $\ignorespaces\langle n(\sqrt{s})\rangle$ is the average multiplicity
of primary particles at the CM energy,  $\ignorespaces\Psi(z)$ is the particle
distribution as a function of the scaled multiplicity  $\ignorespaces
z={n(\sqrt{s})}/{\langle n(\sqrt{s})\rangle}$ . The first term in (1) results
from the leading term in  $\ignorespaces\ln{\sqrt{s}}$ (KNO scaling
hypothesis), and the second term contains all other terms Grosse-
Oetringhaus:2009eis ; Kittel:2005fu . The multiplicity distributions become
simple rescaled copies of the universal function  $\ignorespaces\Psi(z)$ ,
depending only on the scaled multiplicity or an energy-independent function.
Asymptotically, when  $\ignorespaces\sqrt{s}\rightarrow\infty$ , the second
term in (1) tends to zero, and therefore the KNO scaling holds.

To precisely find the KNO scaling, it is important to study the normalised
order-  $\ignorespaces q$ moments of primary charged-particle multiplicity
distributions

$C_{\mathrm{q}}(\sqrt{s})=\frac{\langle n^{q}(\sqrt{s})\rangle}{\langle
n(\sqrt{s})\rangle^{\mathrm{q}}},$ (2)

where  $\ignorespaces q$ is the order of the moment. The moments
$\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ give an integral characteristic of
multiplicity distributions  $\ignorespaces P(n,\sqrt{s})$ , while the KNO
scaling distributions,  $\ignorespaces\Psi(z)$ , give differential dependence
from normalised multiplicity. The energy independence of  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ of multiplicity distributions of various orders
would imply observation of the KNO scaling. Therefore, the investigation of
$\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ moments is an absolutely essential
addition to the study of  $\ignorespaces\Psi(z)$ functions.

In this paper, the normalised order-  $\ignorespaces q$ moments are studied
using the primary charged-particle multiplicity distributions STDM-2010-01 ;
STDM-2010-06 ; STDM-2014-19 ; STDM-2015-02 ; STDM-2015-17  by ATLAS
PERF-2007-01  at the LHC Evans:2008zzb . Measurements of the primary charged-
particle distributions in ATLAS at  $\ignorespaces\sqrt{s}=0.9$ ,
$\ignorespaces 2.36$ ,  $\ignorespaces 7$ ,  $\ignorespaces 8$ , and
$\ignorespaces 13$ TeV were performed for the pseudorapidity region less than
$\ignorespaces 2.5$ and for two samples of events: with the primary charged-
particle multiplicity greater than or equal to  $\ignorespaces 2$ and
$\ignorespaces 1$ and with the charged-particle transverse momentum
$\ignorespaces p_{\mathrm{T}}$ greater than  $\ignorespaces 100$ and
$\ignorespaces 500$ MeV, respectively. The study of the KNO scaling using the
$\ignorespaces\Psi(z)$ scaled multiplicity function, which is defined in (1)
and calculated on the ATLAS data STDM-2010-01 ; STDM-2010-06 ; STDM-2014-19 ;
STDM-2015-02 ; STDM-2015-17 , was published in Kulchitsky:2022gkm  by these
authors. A comparison of  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ with the
results of the LHC and lower energy experiments for  $\ignorespaces pp$
interactions is presented.

The KNO scaling and  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ -moments were
studied by the CMS at  $\ignorespaces\sqrt{s}$ from  $\ignorespaces 0.9$ to
$\ignorespaces 7$ TeV in central pseudorapidity
$\ignorespaces\mid\eta\mid<0.5$ and more inclusive
$\ignorespaces\mid\eta\mid<2.4$ regions CMS:2010qvf . The KNO results were
investigated by ALICE at  $\ignorespaces\sqrt{s}$ from  $\ignorespaces 0.9$ to
$\ignorespaces 13$ TeV in pseudorapidity regions
$\ignorespaces\mid\eta\mid<0.5$ ,  $\ignorespaces\mid\eta\mid<0.8$ ,
$\ignorespaces\mid\eta\mid<1.0$ , and  $\ignorespaces\mid\eta\mid<1.5$
ALICE:2010cin ; ALICE:2015olq ; ALICE:2022xip ; Fan:2022bbp . For ALICE and
CMS experiments the KNO scaling is violated for energies from  $\ignorespaces
0.9$ to  $\ignorespaces 8$ TeV if taking into account more inclusive
pseoudorapidity regions because the  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$
-moments are increase with energy. The KNO scaling holds for the central
pseudorapidity region with  $\ignorespaces\mid\eta\mid<0.5$ and for the energy
region from  $\ignorespaces\sqrt{s}=0.9$ to  $\ignorespaces 7$ TeV on the CMS
data CMS:2010qvf . In this case, the  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$
-moments are independent of energy. The results of ALICE for  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ -moments in central pseudorapidity,
$\ignorespaces\mid\eta\mid<0.5$ , are slightly increased with energy for
$\ignorespaces q=4,\ 5$ .

(a)

(b)

(c)

Figure 1:  The normalised order-  $\ignorespaces q$ moments,  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ , in Eq. (2) of the primary charged-particle
multiplicity distributions measured by ATLAS for events collected at
$\ignorespaces\sqrt{s}=0.9,\ 2.36,\ 7,8$ , and  $\ignorespaces 13$ TeV for (a)
the pseudorapidity region  $\ignorespaces\mid\eta\mid<2.5$ . The results of
CMS CMS:2010qvf  and lower-energy experiments NA22 EHSNA22:1987syd , UA1
UA1:1989bou , and UA5 UA5:1985hzd ; UA5:1988gup  are included. (b) The ATLAS
results for  $\ignorespaces\mid\eta\mid<2.5$ ,  $\ignorespaces
n_{\mathrm{ch}}\geq 1$ ,  $\ignorespaces p_{\mathrm{T}}>500$ MeV. (c) The
ATLAS results for  $\ignorespaces\mid\eta\mid<2.5$ ,  $\ignorespaces
n_{\mathrm{ch}}\geq 2$ ,  $\ignorespaces p_{\mathrm{T}}>100$ MeV with
additional scaled multiplicity thresholds:  $\ignorespaces z>0.5$ and
$\ignorespaces z>1.0$ . The  $\ignorespaces C_{\mathrm{2}}$ ,  $\ignorespaces
C_{\mathrm{3}}$ and  $\ignorespaces C_{\mathrm{4}}$ ,  $\ignorespaces
C_{\mathrm{5}}$ results are shown in the bottom and top panels, respectively.
The vertical bars are the statistical uncertainties, and the squares are the
systematic uncertainties. The coloured symbols are the data. Fits of the
$\ignorespaces\log{\sqrt{s}}$ dependence of the  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ of the multiplicity distribution (assuming linear
dependence) are shown. In (a), for  $\ignorespaces\sqrt{s}=0.9$ TeV, data from
non-ATLAS experiments have been shifted down  $\ignorespaces\sqrt{s}$ for
clarity. The lines show the results of the fits for  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ with statistical and systematic uncertainties added
in quadrature.

The results of this analysis of the validity of KNO scaling are shown
quantitatively in Fig. 1 by the  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ of
the multiplicity distributions measured by the ATLAS and complemented with the
CMS measurements at  $\ignorespaces\sqrt{s}=0.9,\ 2.36$ , and  $\ignorespaces
7$ TeV CMS:2010qvf  and results of the lower-energy experiments NA22
EHSNA22:1987syd , UA1 UA1:1989bou , and UA5 UA5:1985hzd ; UA5:1988gup . The
$\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ calculations based on ATLAS results
for the kinematic region  $\ignorespaces\mid\eta\mid<2.5$ ,  $\ignorespaces
n_{\mathrm{ch}}\geq 2$ and  $\ignorespaces p_{\mathrm{T}}>100$ MeV are shown
in Fig. 1(a) and in Table 1. The ATLAS and CMS results agree within the error
range. The values of  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ with
$\ignorespaces q=2,\dots,5$ for all experiments linearly increase with
$\ignorespaces\log{\sqrt{s}}$ as illustrated by the fits in Fig. 1(a) and in
Table 2 (phase space (A)). Since, as mentioned above, the KNO scaling requires
that  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ be independent of energy, we
can state that the KNO scaling is violated at least for the full region of
scaled multiplicity.

Figure 1(b) shows for the first time the values of  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ calculated using multiplicity distributions measured
by ATLAS for the kinematic region  $\ignorespaces\mid\eta\mid<2.5$ ,
$\ignorespaces n_{\mathrm{ch}}\geq 1$ and  $\ignorespaces p_{\mathrm{T}}>500$
MeV. Similarly, as in Fig. 1(a) the values of  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ linearly increase with
$\ignorespaces\log{\sqrt{s}}$ as the function

$C_{\mathrm{q}}(\sqrt{s})=\alpha+\beta\sqrt{s}.$ (3)

The results of the fit are presented in Table 2 (phase space (B)). The
$\ignorespaces C_{\mathrm{q}}$ values at  $\ignorespaces\sqrt{s}=2.36$ TeV in
Fig. 1(b) are much smaller than those for other energies. This is because the
region of primary charged-particle multiplicity distributions at
$\ignorespaces 2.36$ TeV is smaller (up to  $\ignorespaces z\approx 3.5$ )
than that for higher CM energies (up to  $\ignorespaces z\approx 9$ )
Kulchitsky:2022gkm . Therefore, the  $\ignorespaces C_{\mathrm{q}}$ values at
$\ignorespaces\sqrt{s}=2.36$ TeV were not used in the fits. The
$\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ for  $\ignorespaces
p_{\mathrm{T}}>500$ MeV has a higher bias (  $\ignorespaces\alpha$ ) and slope
(  $\ignorespaces\beta$ ) of the fits than those for the minimum
$\ignorespaces p_{\mathrm{T}}$ threshold, with the bias increasing from
$\ignorespaces 1.1$ at  $\ignorespaces q=2$ up to  $\ignorespaces 2.1$ at
$\ignorespaces q=5$ , and the slope increasing from  $\ignorespaces 1.4$ at
$\ignorespaces q=2$ up to  $\ignorespaces 2.6$ at  $\ignorespaces q=5$ . This
is the result of stronger interactions with a higher  $\ignorespaces
p_{\mathrm{T}}$ threshold in case (B) than in case (A). This is because the
total transverse momentum,  $\ignorespaces\sum p_{\mathrm{T}}$ , for events
with  $\ignorespaces p_{\mathrm{T}}>500$ MeV (B) is higher than the same
observable for events with  $\ignorespaces p_{\mathrm{T}}>100$ MeV (A).

Figure 1(c) shows moments  $\ignorespaces C_{\mathrm{q}}$ for events with
$\ignorespaces n_{\mathrm{ch}}\geq 2$ ,  $\ignorespaces p_{\mathrm{T}}>100$
MeV, and for  $\ignorespaces z>0.5$ without the fraction of single and double
diffraction events, which was accepted by the ATLAS minimum-bias trigger
STDM-2010-01 ; STDM-2010-06 ; STDM-2014-19 ; STDM-2015-02 ; STDM-2015-17 . In
this case, the values of  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ are
systematically higher than those for full distributions with  $\ignorespaces
z>0$ and show a similar linear increase with  $\ignorespaces\log{\sqrt{s}}$ as
illustrated in Fig. 1(c). For multiplicity distributions for  $\ignorespaces
z>1.0$ , the values of  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ at the
highest energies,  $\ignorespaces\sqrt{s}=7,\ 8$ , and  $\ignorespaces 13$
TeV, are in agreement within error uncertainties, as can be seen in Fig. 1(c).
Therefore, the energy independence of the moments of various orders can be
considered as a confirmation of the KNO scaling. This is in agreement with the
conclusion in Kulchitsky:2022gkm  that the KNO scaling holds for the highest
energies.

Table 1:  The normalised order-  $\ignorespaces q$ moments,  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ , of the primary charged-particle multiplicity distributions measured by the ATLAS Collaboration for events at center-of-mass energies  $\ignorespaces\sqrt{s}=0.9,\ 2.36,\ 7,8$ , and  $\ignorespaces 13$ TeV for pseudorapidity region  $\ignorespaces\mid\eta\mid<2.5$ and for two different phase spaces  $\ignorespaces n_{\mathrm{ch}}\geq 2$ ,  $\ignorespaces p_{\mathrm{T}}>100$ MeV and  $\ignorespaces n_{\mathrm{ch}}\geq 1$ ,  $\ignorespaces p_{\mathrm{T}}>500$ MeV.  $\ignorespaces\sqrt{s}$ |  $\ignorespaces p_{\mathrm{T}}^{\mathrm{min}}$ |  $\ignorespaces C_{\mathrm{q}}$
---|---|---
[TeV] | [MeV] |  $\ignorespaces C_{\mathrm{2}}$ |  $\ignorespaces C_{\mathrm{3}}$ |  $\ignorespaces C_{\mathrm{4}}$ |  $\ignorespaces C_{\mathrm{5}}$
13 |  $\ignorespaces 100$ |  $\ignorespaces 1.799\pm 0.002^{+0.021}_{-0.016}$ |  $\ignorespaces 4.491\pm 0.008^{+0.119}_{-0.084}$ |  $\ignorespaces 13.74\pm 0.04^{+0.65}_{-0.52}$ |  $\ignorespaces 48.49\pm 0.15^{+1.52}_{-0.66}$
|  $\ignorespaces 500$ |  $\ignorespaces 2.065\pm 0.002^{+0.008}_{-0.007}$ |  $\ignorespaces 6.209\pm 0.009^{+0.058}_{-0.047}$ |  $\ignorespaces 23.57\pm 0.05^{+0.46}_{-0.34}$ |  $\ignorespaces 105.77\pm 0.29^{+3.05}_{-2.55}$
8 |  $\ignorespaces 100$ |  $\ignorespaces 1.741\pm 0.001^{+0.030}_{-0.029}$ |  $\ignorespaces 4.185\pm 0.007^{+0.131}_{-0.126}$ |  $\ignorespaces 12.32\pm 0.03^{+0.55}_{-0.54}$ |  $\ignorespaces 41.81\pm 0.14^{+2.40}_{-2.40}$
|  $\ignorespaces 500$ |  $\ignorespaces 2.000\pm 0.001^{+0.030}_{-0.030}$ |  $\ignorespaces 5.793\pm 0.008^{+0.161}_{-0.157}$ |  $\ignorespaces 21.12\pm 0.04^{+0.83}_{-0.81}$ |  $\ignorespaces 90.81\pm 0.21^{+4.56}_{-4.44}$
7 |  $\ignorespaces 100$ |  $\ignorespaces 1.712\pm 0.005^{+0.026}_{-0.016}$ |  $\ignorespaces 4.025\pm 0.022^{+0.124}_{-0.062}$ |  $\ignorespaces 11.58\pm 0.10^{+0.64}_{-0.37}$ |  $\ignorespaces 38.43\pm 0.43^{+3.24}_{-2.04}$
|  $\ignorespaces 500$ |  $\ignorespaces 1.985\pm 0.004^{+0.018}_{-0.015}$ |  $\ignorespaces 5.701\pm 0.020^{+0.127}_{-0.116}$ |  $\ignorespaces 20.59\pm 0.10^{+0.87}_{-0.88}$ |  $\ignorespaces 87.64\pm 0.52^{+6.90}_{-6.56}$
2.36 |  $\ignorespaces 500$ |  $\ignorespaces 1.759\pm 0.047^{+0.075}_{-0.070}$ |  $\ignorespaces 4.082\pm 0.227^{+0.352}_{-0.313}$ |  $\ignorespaces 11.00\pm 0.95^{+1.46}_{-1.25}$ |  $\ignorespaces 32.43\pm 3.84^{+5.94}_{-4.86}$
0.9 |  $\ignorespaces 100$ |  $\ignorespaces 1.530\pm 0.015^{+0.028}_{-0.021}$ |  $\ignorespaces 3.121\pm 0.057^{+0.102}_{-0.076}$ |  $\ignorespaces 7.77\pm 0.21^{+0.39}_{-0.30}$ |  $\ignorespaces 22.33\pm 0.77^{+1.72}_{-1.36}$
|  $\ignorespaces 500$ |  $\ignorespaces 1.752\pm 0.011^{+0.016}_{-0.015}$ |  $\ignorespaces 4.364\pm 0.052^{+0.091}_{-0.093}$ |  $\ignorespaces 13.72\pm 0.24^{+0.53}_{-0.51}$ |  $\ignorespaces 50.81\pm 1.20^{+2.64}_{-2.61}$
Table 2:  The fit parameters of energy dependence of the distributions of normalised order-  $\ignorespaces q$ moments  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ in Eq. (3) for two different phase spaces: (A)  $\ignorespaces\mid\eta\mid<2.5$ ,  $\ignorespaces n_{\mathrm{ch}}\geq 2$ ,  $\ignorespaces p_{\mathrm{T}}>100$ MeV, and (B)  $\ignorespaces\mid\eta\mid<2.5$ ,  $\ignorespaces n_{\mathrm{ch}}\geq 1$ ,  $\ignorespaces p_{\mathrm{T}}>500$ MeV.  $\ignorespaces C_{\mathrm{q}}$ | Phase Space |  $\ignorespaces\alpha$ |  $\ignorespaces\beta$
---|---|---|---
$\ignorespaces C_{\mathrm{2}}$ | (A) |  $\ignorespaces 1.54\pm 0.01$ |  $\ignorespaces 0.200\pm 0.015$
| (B) |  $\ignorespaces 1.76\pm 0.02$ |  $\ignorespaces 0.271\pm 0.017$
$\ignorespaces C_{\mathrm{3}}$ | (A) |  $\ignorespaces 3.31\pm 0.03$ |  $\ignorespaces 0.907\pm 0.031$
| (B) |  $\ignorespaces 4.42\pm 0.10$ |  $\ignorespaces 1.59\pm 0.10$
$\ignorespaces C_{\mathrm{4}}$ | (A) |  $\ignorespaces 8.86\pm 0.18$ |  $\ignorespaces 3.40\pm 0.13$
| (B) |  $\ignorespaces 14.0\pm 0.5$ |  $\ignorespaces 8.45\pm 0.56$
$\ignorespaces C_{\mathrm{5}}$ | (A) |  $\ignorespaces 25.2\pm 0.7$ |  $\ignorespaces 17.6\pm 1.4$
| (B) |  $\ignorespaces 52.5\pm 2.7$ |  $\ignorespaces 46.5\pm 3.2$

The results of studying  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ of primarily
charged-particle multiplicity distributions using the results of the ATLAS at
the LHC are presented. The normalised order-  $\ignorespaces q$ moments are
sensitive to the KNO scaling. The  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$
from the ATLAS, CMS, and low-energy experiments are compared for the kinematic
region with an absolute pseudorapidity less than  $\ignorespaces 2.5$ of
$\ignorespaces\sqrt{s}$ from  $\ignorespaces 0.2$ to  $\ignorespaces 13$ TeV.
For the full-scale multiplicity region, the  $\ignorespaces
C_{\mathrm{q}}(\sqrt{s})$ moments show a linear increase with
$\ignorespaces\sqrt{s}$ , therefore indicating that KNO scaling is violated.
The  $\ignorespaces C_{\mathrm{q}}(\sqrt{s})$ of the scaled multiplicity
larger than one average multiplicity is constant for the highest energies.
Thus, for the first time, it can be concluded that for
$\ignorespaces\mid\eta\mid<2.5$ and  $\ignorespaces z>1$ , the KNO scaling is
valid. Previously, the KNO scaling validity at LHC energies was observed for
$\ignorespaces\mid\eta\mid<0.5$ by CMS CMS:2010qvf .

We thank the ATLAS Collaboration for the excellent experimental results that
were used for this analysis. Our special thanks go to Edward K. Sarkisyan-
Grinbaum and Stanislav Tokar for several productive discussions.

## References

  * (1)
  * (2) A. M. Polyakov, A Similarity hypothesis in the strong interactions. 1\. Multiple hadron production in  $\ignorespaces e^{+}e^{-}$ annihilation, Zh. Eksp. Teor. Fiz. 59 (1970) 542.
  * (3) Z. Koba, H. B. Nielsen and P. Olesen, Scaling of multiplicity distributions in high-energy hadron collisions, Nucl. Phys. B 40 (1972) 317.
  * (4) Z. Koba, Multi-body phenomena in strong interactions – description of hadronic multi-body final states, in Proceedings CERN-JINR School of Physics, Ebeltoft, Denmark, 17-13 June 1973.
  * (5) R. P. Feynman, Very high-energy collisions of hadrons, Phys. Rev. Lett. 23 (1969) 1415, ed. by L. M. Brown.
  * (6) J. F. Grosse-Oetringhaus and K. Reygers, Charged-Particle Multiplicity in Proton-Proton Collisions, J. Phys. G 37 (2010) 083001, arXiv: 0912.0023 [hep-ex].
  * (7) W. Kittel and E. A. De Wolf, Soft multihadron dynamics, 2005, isbn: 978-981-256-295-1.
  * (8) ATLAS Collaboration, Charged-particle multiplicities in  $\ignorespaces pp$ interactions at  $\ignorespaces\sqrt{s}=900$ GeV measured with the ATLAS detector at the LHC, Phys. Lett. B 688 (2010) 21, arXiv: 1003.3124 [hep-ex].
  * (9) ATLAS Collaboration, Charged-particle multiplicities in  $\ignorespaces pp$ interactions measured with the ATLAS detector at the LHC, New J. Phys. 13 (2011) 053033, arXiv: 1012.5104 [hep-ex].
  * (10) ATLAS Collaboration, Charged-particle distributions in  $\ignorespaces pp$ interactions at  $\ignorespaces\sqrt{s}=8$ TeV measured with the ATLAS detector, Eur. Phys. J. C 76 (2016) 403, arXiv: 1603.02439 [hep-ex].
  * (11) ATLAS Collaboration, Charged-particle distributions in  $\ignorespaces\sqrt{s}=13$ TeV  $\ignorespaces pp$ interactions measured with the ATLAS detector at the LHC, Phys. Lett. B 758 (2016) 67, arXiv: 1602.01633 [hep-ex].
  * (12) ATLAS Collaboration, Charged-particle distributions at low transverse momentum in  $\ignorespaces\sqrt{s}=13$ TeV  $\ignorespaces pp$ interactions measured with the ATLAS detector at the LHC, Eur. Phys. J. C 76 (2016) 502, arXiv: 1606.01133 [hep-ex].
  * (13) ATLAS Collaboration, The ATLAS Experiment at the CERN Large Hadron Collider, JINST 3 (2008) S08003.
  * (14) L. Evans and P. Bryant, LHC Machine, JINST 3 S08001 (2008).
  * (15) Y. Kulchitsky and P. Tsiareshka, Study of the KNO scaling in  $\ignorespaces pp$ collisions at  $\ignorespaces\sqrt{s}$ from 0.9 to 13 TeV using results of the ATLAS at the LHC, Eur. Phys. J. C 82 (2022) 462, arXiv: 2202.06697 [hep-ex].
  * (16) CMS Collaboration, Charged Particle Multiplicities in  $\ignorespaces pp$ Interactions at  $\ignorespaces\sqrt{s}=0.9$ , 2.36 and 7 TeV, JHEP 01 (2011) 079, arXiv: 1011.5531 [hep-ex].
  * (17) ALICE Collaboration, Charged-particle multiplicity measurement in proton-proton collisions at  $\ignorespaces\sqrt{s}=0.9$ and 2.36 TeV with ALICE at LHC, Eur. Phys. J. C 68 (2010) 89, arXiv: 1004.3034 [hep-ex].
  * (18) ALICE Collaboration, Charged-particle multiplicities in proton–proton collisions at  $\ignorespaces\sqrt{s}=0.9$ to 8 TeV, Eur. Phys. J. C 77 (2017) 33, arXiv: 1509.07541 [nucl-ex].
  * (19) ALICE Collaboration, Multiplicity dependence of charged-particle production in  $\ignorespaces pp$ ,  $\ignorespaces p$ \-  $\ignorespaces Pb$ ,  $\ignorespaces Xe$ \-  $\ignorespaces Xe$ and  $\ignorespaces Pb$ \-  $\ignorespaces Pb$ collisions at the LHC, CERN-EP-2022-266, Geneva, 2022, arXiv: 2211.15326 [nucl-ex].
  * (20) ALICE Collaboration, F. Fan for the Collaboration, Particle production as a function of underlying-event activity and very forward energy with ALICE, Eur. Phys. J. Web Conf. 276 (2023) 01009, arXiv: 2208.11348 [nucl-ex].
  * (21) EHS/NA22 Collaboration, Phase Space Dependence of the Multiplicity Distribution in  $\ignorespaces\pi^{+}p$ and  $\ignorespaces pp$ Collisions at 250-GeV/c, Z. Phys. C 37 (1988) 215.
  * (22) UA1 Collaboration, A Study of the General Characteristics of  $\ignorespaces p\bar{p}$ Collisions at  $\ignorespaces\sqrt{s}=0.2$ -TeV to  $\ignorespaces 0.9$ -TeV, Nucl. Phys. B 335 (1990) 261.
  * (23) UA5 Collaboration, An Investigation of Multiplicity Distributions in Different Pseudorapidity Intervals in anti-p p Reactions at a CMS Energy of 540-GeV, Phys. Lett. B 160 (1985) 193.
  * (24) UA5 Collaboration, Charged Particle Multiplicity Distributions at 200-GeV and 900-GeV Center-Of-Mass Energy, Z. Phys. C 43 (1989) 357, ed. by R. Kotthaus and J. H. Kuhn.


# Unveiling Empirical Pathologies of Laplace Approximation for Uncertainty
Estimation

Maksim Zhdanov1, Stanislav Dereka1, 2, Sergey Kolesnikov1

###### Abstract

In this paper, we critically evaluate Bayesian methods for uncertainty
estimation in deep learning, focusing on the widely applied Laplace
approximation and its variants. Our findings reveal that the conventional
method of fitting the Hessian matrix negatively impacts out-of-distribution
(OOD) detection efficiency. We propose a different point of view, asserting
that focusing solely on optimizing prior precision can yield more accurate
uncertainty estimates in OOD detection while preserving adequate calibration
metrics. Moreover, we demonstrate that this property is not connected to the
training stage of a model but rather to its intrinsic properties. Through
extensive experimental evaluation, we establish the superiority of our
simplified approach over traditional methods in the out-of-distribution
domain.

## Introduction

Deep neural networks (DNNs) have emerged as a powerful tool in a myriad of
applications, ranging from computer vision (Liu et al. 2022) to natural
language processing (Vaswani et al. 2017). Their ability to model complex
patterns and make highly accurate predictions has revolutionized several
domains. However, a significant issue with DNNs is their tendency to be
overconfident in their predictions on out-of-distribution samples (Nguyen,
Yosinski, and Clune 2015), often leading to misguided trust in their
decisions, especially when they encounter data that deviate from their
training set (Hendrycks, Mazeika, and Dietterich 2018). This overconfidence
underlines the critical importance of uncertainty estimation and out-of-
distribution (OOD) detection in neural networks. Uncertainty estimation
involves quantifying the confidence of a neural network in its predictions,
providing a measure of reliability. OOD detection, on the other hand, focuses
on identifying instances that are significantly different from the training
data, which are often the cases where the network’s predictions are less
reliable.

In addressing these challenges, Bayesian approaches (Jospin et al. 2022) have
gained prominence for uncertainty estimation in neural networks. Among the
various Bayesian techniques, the simple and computationally effective Laplace
approximation (MacKay 1992) has been particularly influential. This approach
simplifies the complex Bayesian inference problem by approximating the
posterior distribution of the model’s weights with a Gaussian, centered at the
point of maximum likelihood. The availability of various types of this
approximation makes Bayesian inference computationally feasible for large
neural networks. Further developments in this area include modifications to
the classic Laplace Approximation, such as the last-layer Laplace
approximation (Kristiadi, Hein, and Hennig 2020). This variant focuses on
approximating the posterior only for the weights of the final layer of the
network, under the assumption that the last layer primarily determines the
uncertainty. This assumption allows for a significant reduction in
computational complexity while retaining much of the benefit of the full
Laplace approximation.

Figure 1: Uncertainty estimation on toy binary classification dataset with
test outliers for (left) MAP network, (middle) Standard LLLA, (right) LLLA
only with prior optimization. In the top-left corner for each setup, the
average predictive uncertainty between outlier samples is displayed. It can be
seen that using prior precision as the only source of covariance leads to a
wider uncertainty surface, at the same time preserving the posterior
landscape. Subsequently, it achieves better entropy estimates for outlier data
points.

Contrastingly, some research challenges the need for complex Bayesian methods
in uncertainty estimation. Notably, studies have shown that deep ensembles
(DE), a more straightforward non-Bayesian method, can provide uncertainty
estimates that are on par or better (Lakshminarayanan, Pritzel, and Blundell
2017). This raises the question of whether simpler methods might suffice in
practical applications. Our research contributes to this debate by suggesting
that merely optimizing the prior precision for Laplace approximation (Immer,
Korzepa, and Bauer 2021), bypassing the need to fit the Hessian matrix as in
traditional approaches, can lead to robust and reliable OOD detection. This
streamlined approach not only questions the status quo in Bayesian techniques
but also aligns with the increasing demand for efficient, deployable AI
systems that guarantee trustworthy and safe predictions (Li et al. 2023).

The key contributions of our study are as follows:

  * •

We uncover an intriguing phenomenon where the Laplace approximation, devoid of
Hessian matrix computations, outperforms various established versions of the
method.

  * •

We conduct empirical evaluations of this characteristic across OOD detection
tasks, using datasets like CIFAR-10, CIFAR-100, ImageNet-200, and a toy binary
classification dataset. Moreover, we show that our method preserves
calibration metrics.

  * •

We demonstrate that the observed phenomenon is unrelated to the learning
dynamics of the network, suggesting that the underlying cause is rooted in the
structural aspects of the models.

## Background

Consider a multi-class classification problem where
$\mathcal{D}=\\{x_{i},y_{i}\\}^{N}_{i=1}$ represents a dataset consisting of
independently and identically distributed (i.i.d) data points
$x_{i}\in\mathbb{R}^{D}$ and labels $y\in\\{k_{1},\dots,k_{C}\\}$, each
depicted as a one-hot vector in a $C$-dimensional space.

We define a neural network
$f_{\theta}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{C}$ parameterized by
$\theta$, with a posterior distribution
$p(\theta|D)~{}\propto~{}p(\mathcal{D}|\theta)p(\theta)$. According to
Bayesian principles, the standard training of a neural network associates with
maximum a posteriori (MAP) estimation. This involves finding
$\theta_{\text{MAP}}~{}=~{}\operatorname*{arg\,max}_{\theta}p(\theta|\mathcal{D})$,
which can be further expanded as
$\operatorname*{arg\,max}_{\theta}\operatorname{log}(p(\mathcal{D}|\theta))+\operatorname{log}(p(\theta))$.

The log-likelihood $\operatorname{log}(p(\mathcal{D}|\theta))$ corresponds to
the cross-entropy loss for classification tasks, and the prior
$\operatorname{log}(p(\theta))$ typically is chosen to be an isotropic
Gaussian distribution $\mathcal{N}(\theta|0,\sigma^{2}I)$.

### Laplace Approximation

The approximation of the posterior $q(\theta|\mathcal{D})$ often utilizes the
Laplace approximation (MacKay 1992). This method computes a second-order
expansion of $\operatorname{log}(p(\mathcal{D}|\theta))$ around
$\theta_{\text{MAP}}$, resulting in a Gaussian approximation to
$p(\theta|\mathcal{D})$. $\operatorname{log}(p(\mathcal{D}|\theta))$
decomposes as

$\operatorname{log}(p(\mathcal{D}|\theta))\approx\operatorname{log}(p(\mathcal{D}|\theta_{\text{MAP}}))+\frac{1}{2}(\theta-\theta_{\text{MAP}})^{T}\Sigma_{\theta}(\theta-\theta_{\text{MAP}}).$
(1)
$\Sigma_{\theta}\triangleq\nabla^{2}_{\theta}\operatorname{log}(p(\mathcal{D}|\theta))|_{\theta_{\text{MAP}}}+\lambda
I$ (2)

Here, $\Sigma_{\theta}$ is the Hessian of the log-likelihood with respect to
$\theta$ evaluated at $\theta_{\text{MAP}}$ and with diagonal prior precision
$\lambda$. In practice, optimizing $\lambda$ is achievable through methods
such as cross-validation or marginal likelihood (Immer, Korzepa, and Bauer
2021; Ritter, Botev, and Barber 2018; Immer et al. 2021). Thus, we approximate
$p(\theta|\mathcal{D})$ as
$\mathcal{N}(\theta|\theta_{\text{MAP}},\Sigma_{\theta}^{-1})$.

There are multiple approaches for efficient computation of the Hessian in deep
neural networks. Prominent variants include the generalized Gauss-Newton (GGN)
matrix (Schraudolph 2002), the Fisher information matrix (Amari 1998), and
block-diagonal factorizations like the Kronecker-factored approximate
curvature (K-FAC) (Martens and Grosse 2015).

### Last-Layer Approximation

Studies have shown that Bayesian methods applied solely to the final layer can
rival the performance of those applied across all layers (Kristiadi, Hein, and
Hennig 2020; Snoek et al. 2015). In the context of last-layer Laplace
approximation (LLLA) (Kristiadi, Hein, and Hennig 2020), the neural network
$f_{\theta}$ is considered as $f(x_{i})=W\nu(x_{i})$, where
$W\in\mathbb{R}^{C\times L}$ is the weight matrix of the last layer, and
$\nu\in\mathbb{R}^{L}$ represents the output from the penultimate layer. This
results in a covariance
$\Sigma_{\theta}\triangleq(\nabla^{2}_{\theta}\operatorname{log}(p(\mathcal{D}|\theta))|_{W_{\text{MAP}}}+\lambda
I)\in\mathbb{R}^{CL\times CL}$. Notably, the linearized predictive
distribution in the last-layer approximation is exact (Immer, Korzepa, and
Bauer 2021) and given by

$p(f(x_{i})|x_{i},\mathcal{D})=\mathcal{N}(f(x_{i})|W_{\text{MAP}}\nu(x_{i}),J(x_{i})^{T}\Sigma^{-1}J(x_{i})),$
(3)

where $J$ is the Jacobian at input $x_{i}$.

## Gaussian Approximation via Prior Precision

In this section, we present our main finding – an intriguing property of
Laplace approximation for modern deep neural networks, which suggests the
ability to obtain better uncertainty landscape and pointwise estimates relying
only on prior precision.

Consider the covariance matrix $\Sigma_{\theta}$ from Equation 2, which is a
sum of the Hessian and the product of prior precision with the identity
matrix. We put forth the hypothesis that in overparameterized models,
gradients in Equation 2 act as a bottleneck in terms of OOD detection
capabilities. This might be linked to gradient instabilities commonly observed
in training overparameterized models (Philipp, Song, and Carbonell 2018; Luo
2017; Ioffe and Szegedy 2015) and complex structure of neural network
posteriors (Izmailov et al. 2021; Laurent, Aldea, and Franchi 2023).

Method | CIFAR-10 | CIFAR-100 | ImageNet-200
---|---|---|---
Near OOD | Far OOD | Near OOD | Far OOD | Near OOD | Far OOD
LLLA (GGN) | 88.94 ± 0.30 | 91.54 ± 0.36 | 81.44 ± 0.09 | 80.00 ± 0.45 | 81.84 ± 0.10 | 89.39 ± 0.15
LLLA (EF) | 89.54 ± 0.36 | 92.09 ± 0.30 | 81.66 ± 0.10 | 80.17 ± 0.83 | 81.86 ± 0.11 | 89.45 ± 0.14
LLLA (K-FAC) | 88.02 ± 0.28 | 90.73 ± 0.52 | 80.30 ± 0.13 | 77.77 ± 0.53 | 81.73 ± 0.10 | 88.94 ± 0.14
Prior LLLA (Ours) | 90.01 ± 0.21 | 92.50 ± 0.38 | 81.45 ± 0.10 | 80.79 ± 0.46 | 81.88 ± 0.10 | 89.45 ± 0.14
Table 1: OOD detection results for different Laplace approximations. All
approximations are tested on multiple OOD sources. Mean and standard
deviations AUROC values are reported. The best results among different methods
are highlighted. Figure 2: Calibration scores. Expected Calibration Error
(ECE), Negative Log-Likelihood (NLL), and Brier score for (left) CIFAR-10,
(middle) CIFAR-100 and (right) ImageNet-200 and different Laplace
approximations. Neglecting the Hessian does not detrimentally affect model
calibration.

Our approach suggests constructing a posterior approximation that is based
exclusively on optimizing prior precision:

$\Sigma_{\theta}=\lambda I$ (4)

In the subsequent experiments, we empirically show that our method yields a
broader uncertainty landscape and increased entropy for out-of-distribution
samples.

## Experiments

In our experiments, we evaluate the proposed version of Laplace approximation
on several datasets against state-of-the-art methods. Further, we study its
connection to model training and evaluate calibration scores.

Figure 3: Epoch-wise comparison between standard LLLA and LLLA without Hessian
for OOD detection. Based on this experiment, we can state that the observed
phenomenon is unrelated to model training dynamics.

### Experimental Setup

Figure 4: Uncertainty estimation on toy sinusoidal regression dataset for (left) MAP network, (middle) Standard LLLA, (right) LLLA only with prior optimization. The observed property of Laplace approximation and its Hessian generalizes to regression tasks and does not hurt uncertainty estimates. Method | CIFAR-10 | CIFAR-100 | ImageNet-200
---|---|---|---
Near OOD | Far OOD | Near OOD | Far OOD | Near OOD | Far OOD
MSP | 88.03 ± 0.25 | 90.73 ± 0.43 | 80.27 ± 0.11 | 77.76 ± 0.44 | 83.34 ± 0.06 | 90.13 ± 0.09
ODIN | 82.87 ± 1.85 | 87.96 ± 0.61 | 79.90 ± 0.11 | 79.28 ± 0.21 | 80.27 ± 0.08 | 91.71 ± 0.19
ReAct | 87.11 ± 0.61 | 90.42 ± 1.41 | 80.77 ± 0.05 | 80.39 ± 0.49 | 81.87 ± 0.98 | 92.31 ± 0.56
VIM | 88.68 ± 0.28 | 93.48 ± 0.24 | 74.98 ± 0.13 | 81.70 ± 0.62 | 78.68 ± 0.24 | 91.26 ± 0.19
SHE | 81.54 ± 0.51 | 85.32 ± 1.43 | 78.95 ± 0.18 | 76.92 ± 1.16 | 80.18 ± 0.25 | 89.81 ± 0.61
Prior LLLA (Ours) | 90.01 ± 0.21 | 92.50 ± 0.38 | 81.45 ± 0.10 | 80.79 ± 0.46 | 81.88 ± 0.10 | 89.45 ± 0.14
Table 2: OOD detection results against non-Bayesian methods. All methods are
trained on the ID dataset and tested on multiple OOD sources. Mean and
standard deviations AUROC values are reported. The best results among
different methods are highlighted.

We employ the well-established OpenOOD v1.5 (Zhang et al. 2023) benchmark
featuring the most state-of-the-art OOD detection methods. It uses CIFAR-10,
CIFAR-100, and ImageNet-200 (a subset of ImageNet-1K) (Deng et al. 2009) as
in-distribution datasets and evaluates methods against carefully chosen OOD
datasets. For all of our datasets, we use ResNet-18 (He et al. 2016) trained
with SGD optimizer with a momentum of 0.9. The initial learning rate is set to
0.1 and then reduced to $10^{-6}$ with the Cosine Annealing scheduler
(Loshchilov and Hutter 2016). Training lasts 100 epochs on CIFAR-10 and
CIFAR-100 and 90 epochs on ImageNet-200. Additionally, we perform a multi-seed
evaluation with 3 random seeds and report the mean and standard deviation
values for each experiment to obtain a fair comparison.

### Toy Examples

#### Half-Moons Dataset.

To visually examine the changes removing Hessian brings to uncertainty
estimates, we employ a half-moons binary classification dataset with an
additional 10 test outliers. In this experiment, we use a simple 2-layer MLP
with ReLU activations and 20 hidden units. We compare the uncertainty
landscapes produced by 3 methods: MAP, LLLA, and the proposed Prior LLLA. The
results are presented in Figure 1. We demonstrate that our method preserves
the general structure of the predictive distribution alongside giving more
strict and precise uncertainty estimates for outliers.

#### Sinusoidal Regression Dataset.

Our observations are not limited to classification settings and generalize to
other tasks, such as a simple sinusoidal regression dataset, which we
demonstrate in Figure 4.

### OOD Detection

#### Laplace Approximations.

In this experiment, we compare standard LLLA with GGN, EF, and K-FAC against
LLLA only backed by prior precision in terms of OOD detection AUROC. We tune
prior precision for all of our methods via marginal likelihood method (Immer
et al. 2021). The results are presented in Table 1. It can be seen from the
results that our method (Prior LLLA) achieves better performance for OOD
detection problems compared to other Laplace approximation variations in 5 of
6 experiments.

#### Non-Bayesian Approaches.

In addition to Laplace approximations, we compare Prior LLLA with popular
post-hoc non-Bayesian OOD detection methods from OpenOOD, including Maximum
Softmax Probability (MSP) (Hendrycks and Gimpel 2016), ODIN (Liang, Li, and
Srikant 2017), ReAct (Sun, Guo, and Li 2021), VIM (Wang et al. 2022) and SHE
(Zhang et al. 2022). The comparison is shown in Table 2. Our method achieves
competitive performance in comparison with other baselines.

### Connection to Learning Dynamics

We empirically observe that the phenomena is not connected to the training
stage of a model but rather its intrinsic structure. In Figure 3, we build
approximations for different stages of training and show that even at the
start of training positive performance gap persists between the standard last-
layer Laplace approximation and the proposed one.

### Accuracy and Calibration

Neglecting the Hessian in the Laplace approximation does not affect in-
distribution classification accuracy, as the method maintains an optimal
solution around $\theta_{\text{MAP}}$. Importantly, it does not significantly
change in-distribution probabilities, impacting calibration scores (see Figure
2). It maintains calibration metrics relatively similar to different versions
of Laplace approximations.

## Conclusion

This work has uncovered an intriguing aspect of the Laplace approximation: the
redundancy of the Hessian matrix for out-of-distribution (OOD) uncertainty
estimation. Our empirical findings indicate that a minor modification in the
approximation significantly enhances OOD detection while preserving
calibration scores. Interestingly, this improvement is independent of the
training process, suggesting a deeper relationship between approximate second-
order gradients and the neural network’s intrinsic structure. By improving the
OOD detection capability of Laplace approximation, our research has strong
implications for building safe and deployable AI systems.

## References

  * Amari (1998) Amari, S.-I. 1998.  Natural gradient works efficiently in learning.  _Neural computation_ , 10(2): 251–276.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hendrycks and Gimpel (2016) Hendrycks, D.; and Gimpel, K. 2016.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_.
  * Hendrycks, Mazeika, and Dietterich (2018) Hendrycks, D.; Mazeika, M.; and Dietterich, T. 2018.  Deep anomaly detection with outlier exposure.  _arXiv preprint arXiv:1812.04606_.
  * Immer et al. (2021) Immer, A.; Bauer, M.; Fortuin, V.; Rätsch, G.; and Emtiyaz, K. M. 2021.  Scalable marginal likelihood estimation for model selection in deep learning.  In _International Conference on Machine Learning_ , 4563–4573. PMLR.
  * Immer, Korzepa, and Bauer (2021) Immer, A.; Korzepa, M.; and Bauer, M. 2021.  Improving predictions of Bayesian neural nets via local linearization.  In _International conference on artificial intelligence and statistics_ , 703–711. PMLR.
  * Ioffe and Szegedy (2015) Ioffe, S.; and Szegedy, C. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , 448–456. pmlr.
  * Izmailov et al. (2021) Izmailov, P.; Vikram, S.; Hoffman, M. D.; and Wilson, A. G. G. 2021.  What are Bayesian neural network posteriors really like?  In _International conference on machine learning_ , 4629–4640. PMLR.
  * Jospin et al. (2022) Jospin, L. V.; Laga, H.; Boussaid, F.; Buntine, W.; and Bennamoun, M. 2022.  Hands-on Bayesian neural networks—A tutorial for deep learning users.  _IEEE Computational Intelligence Magazine_ , 17(2): 29–48.
  * Kristiadi, Hein, and Hennig (2020) Kristiadi, A.; Hein, M.; and Hennig, P. 2020.  Being bayesian, even just a bit, fixes overconfidence in relu networks.  In _International conference on machine learning_ , 5436–5446. PMLR.
  * Lakshminarayanan, Pritzel, and Blundell (2017) Lakshminarayanan, B.; Pritzel, A.; and Blundell, C. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30.
  * Laurent, Aldea, and Franchi (2023) Laurent, O.; Aldea, E.; and Franchi, G. 2023.  A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors.  _arXiv preprint arXiv:2310.08287_.
  * Li et al. (2023) Li, B.; Qi, P.; Liu, B.; Di, S.; Liu, J.; Pei, J.; Yi, J.; and Zhou, B. 2023.  Trustworthy AI: From principles to practices.  _ACM Computing Surveys_ , 55(9): 1–46.
  * Liang, Li, and Srikant (2017) Liang, S.; Li, Y.; and Srikant, R. 2017.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _arXiv preprint arXiv:1706.02690_.
  * Liu et al. (2022) Liu, Z.; Mao, H.; Wu, C.-Y.; Feichtenhofer, C.; Darrell, T.; and Xie, S. 2022.  A convnet for the 2020s.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 11976–11986.
  * Loshchilov and Hutter (2016) Loshchilov, I.; and Hutter, F. 2016.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_.
  * Luo (2017) Luo, P. 2017.  Learning deep architectures via generalized whitened neural networks.  In _International Conference on Machine Learning_ , 2238–2246. PMLR.
  * MacKay (1992) MacKay, D. J. 1992.  A practical Bayesian framework for backpropagation networks.  _Neural computation_ , 4(3): 448–472.
  * Martens and Grosse (2015) Martens, J.; and Grosse, R. 2015.  Optimizing neural networks with kronecker-factored approximate curvature.  In _International conference on machine learning_ , 2408–2417. PMLR.
  * Nguyen, Yosinski, and Clune (2015) Nguyen, A.; Yosinski, J.; and Clune, J. 2015.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 427–436.
  * Philipp, Song, and Carbonell (2018) Philipp, G.; Song, D.; and Carbonell, J. G. 2018.  Gradients explode-deep networks are shallow-resnet explained.
  * Ritter, Botev, and Barber (2018) Ritter, H.; Botev, A.; and Barber, D. 2018.  A scalable laplace approximation for neural networks.  In _6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings_ , volume 6. International Conference on Representation Learning.
  * Schraudolph (2002) Schraudolph, N. N. 2002.  Fast curvature matrix-vector products for second-order gradient descent.  _Neural computation_ , 14(7): 1723–1738.
  * Snoek et al. (2015) Snoek, J.; Rippel, O.; Swersky, K.; Kiros, R.; Satish, N.; Sundaram, N.; Patwary, M.; Prabhat, M.; and Adams, R. 2015.  Scalable bayesian optimization using deep neural networks.  In _International conference on machine learning_ , 2171–2180. PMLR.
  * Sun, Guo, and Li (2021) Sun, Y.; Guo, C.; and Li, Y. 2021.  React: Out-of-distribution detection with rectified activations.  _Advances in Neural Information Processing Systems_ , 34: 144–157.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2022) Wang, H.; Li, Z.; Feng, L.; and Zhang, W. 2022.  Vim: Out-of-distribution with virtual-logit matching.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 4921–4930.
  * Zhang et al. (2022) Zhang, J.; Fu, Q.; Chen, X.; Du, L.; Li, Z.; Wang, G.; Han, S.; Zhang, D.; et al. 2022.  Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy.  In _The Eleventh International Conference on Learning Representations_.
  * Zhang et al. (2023) Zhang, J.; Yang, J.; Wang, P.; Wang, H.; Lin, Y.; Zhang, H.; Sun, Y.; Du, X.; Zhou, K.; Zhang, W.; et al. 2023.  OpenOOD v1. 5: Enhanced Benchmark for Out-of-Distribution Detection.  _arXiv preprint arXiv:2306.09301_.