††thanks: J.W. and H.Y. contributed equally to this manuscript.††thanks: J.W.
and H.Y. contributed equally to this manuscript.††thanks: Corresponding
authors††thanks: Corresponding authors

# Coexistence of zero-, one- and two-dimensional degeneracy in tetragonal SnO2
phonons

Jianhua Wang  School of Physical Science and Technology, Southwest University,
Chongqing 400715, China  Hongkuan Yuan  School of Physical Science and
Technology, Southwest University, Chongqing 400715, China  Minquan Kuang
School of Physical Science and Technology, Southwest University, Chongqing
400715, China  Tie Yang  School of Physical Science and Technology, Southwest
University, Chongqing 400715, China  Zhi-Ming Yu  Centre for Quantum Physics,
Key Laboratory of Advanced Optoelectronic Quantum Architecture and Measurement
(MOE), School of Physics, Beijing Institute of Technology, Beijing, 100081,
China  Beijing Key Lab of Nanophotonics $\&$ Ultrafine Optoelectronic Systems,
School of Physics, Beijing Institute of Technology, Beijing, 100081, China
Zeying Zhang<EMAIL_ADDRESS>College of Mathematics and Physics, Beijing
University of Chemical Technology, Beijing 100029, China  Xiaotian Wang
<EMAIL_ADDRESS>School of Physical Science and Technology, Southwest
University, Chongqing 400715, China

###### Abstract

Based on the dimension of degeneracy, topological electronic systems can
roughly be divided into three parts: nodal point, line and surface materials
corresponding to zero-, one- and two-dimensional degeneracy, respectively. In
parallel to electronic systems, the concept of topology was extended to
phonons, promoting the birth of topological phonons. Till date, few nodal
point, line and surface phonons candidates have been predicted in solid-state
materials. In this study, based on symmetry analysis and first-principles
calculation, for the first time, we prove that zero-, one- and two-dimensional
degeneracy co-exist in the phonon dispersion of one single realistic solid-
state material SnO2 with P42/mnm structure. In contrast to the previously
reported electronic systems, the topological phonons observed in SnO2 are not
restricted by the Pauli exclusion principle, and they experience negligible
spin-orbit coupling effect. Hence, SnO2 with multiple dimensions of degeneracy
phonons is a good platform for studying the entanglement among nodal point,
line and surface phonons. Moreover, obvious phonon surface states are visible,
which is beneficial for experimental detection.

The discovery of topological semimetals in electronic systems add1 ; add2 ;
add3 ; add4 ; add5  has led to extensive research on the classification of
topological states and the identification of materials with topological
electronic structures. Generally, topological semimetals obey the following
standard of classification add6 : (i) dimension of degeneracy (such as nodal
point, line or surface); (ii) degree of degeneracy (such as two-, three-,
four-, six- or eight-fold degeneracy); and (iii) the type of dispersion
(linear, quadratic or cubic). Thus far, thousands of realistic materials add7
; add8 ; add9 ; add10 ; add11 ; add12 ; add13 ; add14 ; add15 ; add16 ; add17
; add18 ; add19 ; add20  with topological electronic structures have been
proposed using the first-principles calculation. Furthermore, owing to the
development of symmetry-based indicator theories add18 ; add21 ; add22 ; add23
and compatibility relations add24 , they have been widely used to determine
the topological state in the electronic structure of most known materials.
Moreover, an encyclopedia of quasiparticles in three-dimensional materials has
been recently afforded by Yu et. al. add25  with detailed correspondence among
quasiparticles, symmetry conditions, effective models, and topological
properties. Despite important achievements in electronic fermionic
quasiparticles and symmetry analysis, the study of the topological bosonic
state in crystalline materials is still in its infancy. Particularly, the
ideal topological semimetal candidates in real electronic materials are very
limited due to the strong Fermi level constraint. In contrast, topological
bosonic states do not have this constraint and are expected to be great
platforms for realizing ideal topological states. However, only a few real
materials with topological phonons have been predicted add26 ; add27 ; add28 ;
add29 ; add30 ; add31 ; add32 ; add33 ; add34 ; add35 ; add36 ; add37 ; add38
; add39 , significantly obstructing the development of topological states in
phonon systems.

Figure 1: (a) and (b) crystal structures of rutile-type SnO2 with P42/mnm
structure with views of different sides; (c) Three-dimensional coordinate
system of BZ and some high symmetry points (A, Z, R, M, $\Gamma$ and X); (d)
the schematic of the zero-dimensional Dirac point (DP) and triply degenerate
point (TDP), one-dimensional Weyl nodal lines (WNLs), and two-dimensional
nodal surface phonons in momentum space. The WNLs possess a quantized $\pi$
Berry phase.

Topological phonons add40 ; add41 ; add42 ; add43 ; add44 ; add45  are the
basic emergent bosons of crystalline lattices. In parallel to electronic
systems, realistic solid-state materials add26 ; add27 ; add28 ; add29 ; add30
; add31 ; add32 ; add33 ; add34 ; add35 ; add36 ; add37 ; add38 ; add39  have
been predicted to comprise nodal point, line and surface phonons with zero-,
one- and two-dimensional degeneracy phonons in momentum space, respectively.
Moreover, double Weyl point phonons and helical nodal line phonons have been
experimentally confirmed in FeSi-type systems and MoB2 add26 ; add30 ,
respectively. However, the above-mentioned studies have only focused on
materials with one or two types of dimensions of the degeneracy phonons.
_Thus, a natural question is whether three types of dimensions of degeneracy
phonons exist in a single synthesized solid-state material._ This study
answers this question in the affirmative.

Although three types of fermions (i.e., zero-, one-, two-dimensional
degeneracy) have been proposed before in some electronic systems add46 ; add47
; add48 , they do not belong to the ideal candidate materials as the proposed
topological elements are not near the Fermi level. As discussed above, in
contrast to electronic systems, phonons are not limited by the Pauli exclusion
principle, and thus, the entire frequency range of the phonon dispersion can
be physically observed and used to search for topological elements. Herein, we
focus on the solid-state material SnO2 with P42/mnm structure add49  to prove
the presence of zero-, one- and two-dimensional degeneracy phonons in its
phonon dispersion. Furthermore, a systematic symmetry analysis for these
proposed phonons was performed to provide a deeper physics understanding.
Obvious phonon surface states are visible for SnO2; this will benefit their
detection in experiments.

In this study, we use the density functional theory to deal with the ground
state of the P42/mnm-type SnO2 material. For the exchange-correlation
functional, GGA-PBE formalism add50  is employed. The projector augmented-wave
method add51  is used for the interactions between ions and valence electrons,
and the energy cutoff is set to 600 eV. A $\Gamma$-centered $k$-mesh of 7
$\times$ 7 $\times$ 10 is used for Brillouin zone (BZ) sampling. A lattice
dynamic calculation based on the density functional perturbation theory within
the PHONOPY package add52  is performed to achieve SnO2 phonon dispersion in
its ground state. The [001] and [100] phonon surface states of SnO2 are
calculated by constructing a Wannier tight-binding Hamiltonian of phonons
add53 . A prerequisite for the symmetry analysis of the topological states is
to obtain the corresponding irreducible representation (IRR) of the phonon
spectra. Fortunately, Liu et. al. recently developed a SpaceGroupIrep package
add54  that can be used to calculate the IRR of the phonon spectra for all
high-symmetry momentum points. The SpaceGroupIrep package is based on the
notation of Ref. add55  and is compatible with Ref. add25 . We used this
package to obtain the required IRRs in this work.

This study focuses on the already prepared rutile-type SnO2 with P42/mnm
structure add49 ; the crystal structure with views of different sides of SnO2
is shown in Fig. 1(a) and 1(b). This material is totally relaxed herein and
the obtained lattice constants (a = b = 4.801 Å and c = 3.242 Å) are in good
agreements with the experimental ones (a = b = 4.737 Å and c = 3.185 Å) add49
. Note that a series of methods for growing SnO2 single crystals have been
previously proposed add56 , and we believe that the easily obtained SnO2
single crystal is a good platform to experimentally confirm the topological
elements mentioned in the following parts of this letter.

The SnO2 phonon dispersion along the $\Gamma$–X–M–A–R–X–$\Gamma$–Z–A–R–Z path
(see Fig. 1(c)) is shown in Fig. 2(a). Obviously, all phonon frequencies are
positive, denoting that SnO2 with P42/mnm structure is dynamical stable.
Careful observation shows that two two-fold degenerate phonon bands exist
along the X–M–A–R–X and R–Z paths around 20 THz, named as R1 and R2,
respectively. Moreover, two phonon band-crossing points exist: one along
$\Gamma$–Z and the other along M–A, named as R3 and R4, respectively. Phonons
are bosons, and thus, the entire frequency range of the phonon dispersion can
be considered to identify topological elements. Hence, we will separately
discuss the topological element in R1-R4.

Figure 2: (a) The phonon dispersion of SnO2 along the
$\Gamma$–X–M–A–R–X–$\Gamma$–Z–A–R–Z path. The two obvious phonon band
degenerated regions along X–M–A–R–X and R–Z paths are named as regions R1 and
R2, respectively. Two clear phonon band-crossing regions are present along
$\Gamma$–Z and M–A regions, which are named as R3 and R4, respectively. The
(co)representations for each band in R1-R4 are labeled. (b) The bulk and
surface BZ along the [100] and [001] directions of SnO2.

In R1, the two nondegenerate phonon bands along the $\Gamma$–X path merged
into a two-fold degenerate phonon band along the X–M–A–R–X path, and the
phonon bands at the X point exhibit a linear phonon band dispersion. As shown
in Fig. 2(b), the high-symmetry X–M–A–R–X path is present in the $k_{y}=\pi$
plane of the three-dimensional BZ. In Fig. 3(b), we selected two symmetry
points B and C (E and D) in the $k_{y}=\pi$ ($k_{y}=0$) plane and divided the
B–C (E–D) path into five parts. Then, some symmetry points along the B–C and
E–D paths were chosen. The phonon dispersions along e–a–e′, f–b–f′, g–c–g′ and
h–d–h′ are shown in Fig. 3(c). Obviously, two nondegenerate phonon bands
linearly cross at the a, b, c and d points; the a, b, c and d points lie in
the $k_{y}=\pi$ plane. Actually, the two nondegenerate phonon bands
degenerated in the entire $k_{y}=\pi$ plane and yielded two-dimensional nodal
surface phonons add57  in the momentum space (see the schematic diagram of
nodal surface phonons in Fig. 1(d)). Similar nodal surface phonons can also be
observed in the entire $k_{x}=\pi$ plane.

The nodal surface in the $k_{x}=\pi$ plane is due to Kramers degeneracy. Due
to the combination of two-fold screw-rotational symmetry
$\\{C_{2x}|\frac{1}{2}\frac{1}{2}\frac{1}{2}\\}$ and time-reversal symmetry
$\cal{T}$ (notice that ${\cal{T}}^{2}\equiv 1$ for spinless case),
${(\\{C_{2x}|\frac{1}{2}\frac{1}{2}\frac{1}{2}\\}\cal{T})}^{2}=-1$ for each
point in the $k_{x}=\pi$ plane; this affords Kramers-like band degeneracy on
the $k_{x}=\pi$ plane and the appearance of the nodal surface state.
Similarly, ${(\\{C_{2y}|\frac{1}{2}\frac{1}{2}\frac{1}{2}\\}\cal{T})}^{2}=-1$
in the $k_{y}=\pi$ plane, which guarantees another nodal surface in the
$k_{y}=\pi$ plane add15 .

Figure 3: (a) The selected B, C, D and E points in the bulk BZ. (b) Some
symmetry points of the B–C–D–E plane; a (e), b (f), c (g) and d (h) are
located at equal distances between B (E) and C (D). (c) Phonon dispersions
along the e–a–e′, f–b–f′, g–c–g′ and h–d–h′ paths. The a, b, c and d points
lie in the $k_{y}=\pi$ plane.

In the R2 region, a two-fold degenerate phonon band exists along the R–Z path.
This two-fold degenerate phonon band is present in the $k_{z}=\pi$ plane and
should belong to one pair of Weyl nodal lines (WNLs) in the $k_{z}=\pi$ plane
(see the shapes of the WNLs of the $k_{z}=\pi$ plane in Fig. 4(a)). Notice
that the R–Z path exhibits a combination of glide-mirror operation and time
reversal symmetry
$g=\\{\sigma_{y}|\frac{1}{2}\frac{1}{2}\frac{1}{2}\\}\cal{T}$, which acts on
the lattice momentum as
$g:(k_{x},k_{y},k_{z})\rightarrow(-k_{x},k_{y},-k_{z})$ and satisfies
$g^{2}=-1$ in the R–Z path. Additionally, this Kramers-like symmetry
guarantees the double band degeneracy on the corresponding line add58 .
Furthermore, we calculated the band representation of the R–Z high-symmetry
line (see Fig 2(a)). Based on the calculated band representation, the $k\cdot
p$ Hamiltonian for the WNL at the R–Z path can be expressed as

$\cal{H}_{\textit{WNL}}=\quad\begin{pmatrix}c_{1}k_{y}&c_{2}k_{z}+ic_{3}k_{x}\\\
c_{2}k_{z}-ic_{3}k_{x}&c_{1}k_{y}\end{pmatrix},$ (1)

where $c_{i}$’s are real parameters.

The Berry phase for a closed loop surrounding the WNLs is calculated based on
the following formula:

$P_{B}=\oint_{L}A(k)\cdot dk,$ (2)

where $A(k)=-i\langle\varphi(k)|\bigtriangledown_{k}|\varphi(k)\rangle$ is the
Berry connection and $\varphi(k)$ is the periodic part of the Bloch function.
For the WNLs in the $k_{z}=\pi$ plane, $P_{B}=\pi$ (see Fig. 1(d)), which
implies that the WNLs are topologically nontrivial. Fig. 4(c) shows the phonon
surface spectra along the $\widetilde{\rm{P}}$–$\widetilde{\rm{P}^{\prime}}$
path on the [001] surface; the positions of the two phonon band-crossings, at
P and $\rm{P}^{\prime}$, of the WNLs are marked by red circles. The phonon
dispersion of the bulk along the P–$\rm{P}^{\prime}$ path is shown in Fig.
4(b) for a reference. As shown in Fig. 4(c), the drumhead-shaped phonon
surface states (marked by black arrows) arising from the WNLs are obvious.

Figure 4: (a) The shapes of the WNLs (white lines) in the $k_{z}=\pi$ plane;
(b) the phonon dispersion of SnO2 along the P–$\rm{P}^{\prime}$ path. (c)
Phonon surface spectra along the
$\widetilde{\rm{P}}$–$\widetilde{\rm{P}^{\prime}}$ path on the [001] surface.
Figure 5: (a) and (c) Phonon dispersions along L–Z and F–H–F′, respectively;
(b) and (d) phonon surface spectra along the
$\overline{\rm{L}}$–$\overline{\rm{Z}}$ path on the [100] surface and
$\widetilde{\rm{F}}$–$\widetilde{\rm{H}}$–$\widetilde{\rm{F}^{\prime}}$ path
on the [001] surface, respectively. The black circle in (b) and the red circle
in (d) represent the positions of TDP and DP, respectively.

In R3, a phonon band-crossing point with three-fold degeneracy exists along
the $\Gamma$–Z path. The three-fold degenerate band-crossing point is actually
a triply degenerate point (TDP) add41 ; add59  (see Fig. 5(a)), which is
formed by a two-fold degenerate phonon band and a nondegenerate phonon band.
By calculating the band representation of $\Gamma$–Z, we find that TDP is
actually an accidental degeneracy formed by a two-dimensional IRR
$R_{5}^{\Lambda}$ and a one-dimensional IRR $R_{1}^{\Lambda}$, as shown in
Fig. 2(a). Using the obtained IRR, the $k\cdot p$ Hamiltonian of the TDP can
be directly written as

$\cal{H}_{\textit{TDP}=}\quad\begin{pmatrix}c_{1}k_{z}&c_{2}k_{y}&c_{2}k_{x}\\\
c_{2}k_{y}&c_{1}k_{z}&0\\\ c_{2}k_{x}&0&c_{1}k_{z}\end{pmatrix},$ (3)

where $c_{i}$ are real parameters.

Fig. 5(b) depicts the phonon surface spectra along the
$\overline{\rm{L}}$–$\overline{\rm{Z}}$ path on the [100] surface. The phonon
surface state around the TDP (black circle) is highlighted with black arrows.

Moreover, in R4, a four-fold degenerate phonon band-crossing point, formed by
two two-fold degenerate phonon bands, can be found along the M–A path. Such a
four-fold degenerate band-crossing along the M–A path should be a Dirac point
(DP) add60 . Similarly, we calculate the IRR of the four bands forming the DP
in the M–A path. The four bands forming the DP become a pair of two-fold
degenerate bands in the M–A path, for which the band representations are
$R_{7}^{V}R_{8}^{V}$ and $R_{10}^{V}$, as illustrated in Fig. 2(a).
$R_{7}^{V}R_{8}^{V}$ is a two-dimensional co-representation, where two one-
dimensional representations $R_{7}^{V}$ and $R_{8}^{V}$ are bound together by
the anti-unitary operation $\cal{IT}$ and $\cal{I}$ is the inversion symmetry.
The corresponding $k\cdot p$ Hamiltonian of DP in the M–A path is

$\cal{H}_{\textit{DP}=}\quad\begin{pmatrix}ck_{z}&0&\alpha(k_{x}+k_{y})&-i\alpha(k_{x}-k_{y})\\\
0&ck_{z}&\alpha^{\ast}(k_{x}+k_{y})&i\alpha^{\ast}(k_{x}-k_{y})\\\
\alpha^{\ast}(k_{x}+k_{y})&\alpha(k_{x}+k_{y})&ck_{z}&0\\\
i\alpha^{\ast}(k_{x}-k_{y})&-i\alpha(k_{x}-k_{y})&0&ck_{z}\end{pmatrix},$ (4)

where $c$ is a real parameter and $\alpha$ is a complex parameter.

Fig. 5(d) shows the phonon surface spectra along the
$\widetilde{\rm{F}}$–$\widetilde{\rm{H}}$–$\widetilde{\rm{F}^{\prime}}$ path
on the [001] surface (Fig. 1(d)); the DP position is marked by a red circle.
As reference, the phonon dispersion of the bulk along F–H–F′ is exhibited in
Fig. 5(c), and DP (at H point) belongs to type I with linear phonon band
dispersion. Two phonon surface states (marked with black arrows) originating
from DP are visible in Fig. 5(d).

In summary, we proposed that the already prepared SnO2 with P42/mnm structure
is a topological phononic material with zero-dimensional TDP and DP phonons,
one-dimensional WNL phonons, and two-dimensional nodal surface phonons. TDP
and DP are along the $\Gamma$–Z and M–A paths, respectively. One pair of WNLs
is in the $k_{z}=\pi$ plane, and the WNLs are topologically nontrivial due to
the $\pi$ Berry phase. Multiple nodal surface states are present in the
$k_{x/y}=\pi$ planes.

Before closing, note the following: (i) This study for the first time, to the
best of our knowledge, reports that realistic SnO2 add49  simultaneously hosts
zero-dimensional TDP and DP phonons, one-dimensional WNLs phonons and two-
dimensional nodal surface phonons. Although some previous studies proved that
zero-, one- and two-dimensional degeneracy coexisted in some electronic
systems add46 ; add47 ; add48 , not all the zero-, one- and two-dimensional
topological elements were located at or around the Fermi level. Some
additional technologies add47 , such as hole or electron doping method, should
be used to tune the topological elements to the Fermi level. The topological
phonons proposed herein do not follow the limitations of the Pauli exclusion
principle, and therefore, these topological phonons can be physically observed
in the entire range of the phonon frequencies. (ii) The zero-, one- and two-
dimensional topological elements coexist in some electronic systems, and their
degenerate electronic structures are usually broken by the effect of spin-
orbit coupling (SOC) add46 . However, the phonons, without SOC, provide us a
new platform to search for material samples with true zero-, one- and two-
dimensional topological elements in their phonon dispersion. (iii) Topological
electronic systems with different topological elements enjoy different
physical properties. Therefore, if zero-, one- and two-dimensional topological
elements coexisted in one single solid-state material, the study of the
entanglement among these three topological elements would be very interesting;
(iv) the nodal surface and WNL states shown herein are nearly flat, as shown
in Fig. 2(a). Moreover, as exhibited in Fig. 3(c), an important feature of the
proposed nodal surface phonons is that they host linear phonon band dispersion
along the direction normal to the surface add15 . (v) The phonon surface
states for the WNL, DP and TDP phonons are obvious, benefiting experimental
detection. (vi) Finally, since 1961, the preparation technology of high-
quality pure single crystals of SnO2 has rapidly developed add56 . Till date,
more than ten methods add56  have been developed to experimentally obtain pure
single crystals of SnO2. We believe that SnO2 is a good candidate for
investigating the zero-, one- and two-dimensional topological phonons. Thus,
the experimental confirmation of the predicted topological elements in SnO2
phonons is imminent.

_Acknowledgments_ Z. Z. is grateful for the support from the National Natural
Science Foundation of China (Grants No. 12004028). X. W. is grateful for the
support from the National Natural Science Foundation of China (No. 51801163)
and the Natural Science Foundation of Chongqing (No. cstc2018jcyjA0765).

## References

  * (1) Burkov, A. A. (2016). Nature Materials, 15(11), 1145-1148.
  * (2) Narang, P., Garcia, C. A. and Felser, C. (2021). Nature Materials, 20(3), 293-300.
  * (3) Gao, H., Venderbos, J. W., Kim, Y. and Rappe, A. M. (2019). Annual Review of Materials Research,49, 153-183.
  * (4) Lv, B. Q., Qian, T. and Ding, H. (2021). Reviews of Modern Physics, 93(2), 025002.
  * (5) Schoop, L. M., Pielnhofer, F. and Lotsch, B. V. (2018). Chemistry of Materials, 30(10), 3155-3176.
  * (6) Wang S., Wu W. and Yang S. (2019). Acta Physica Sinica, 68(22), 224206.
  * (7) Fang, C., Weng, H., Dai, X. and Fang, Z. (2016). Chinese Physics B, 25(11), 117106.
  * (8) Fang, C., Chen, Y., Kee, H. Y. and Fu, L. (2015). Physical Review B, 92(8), 081201.
  * (9) Burkov, A. A., Hook, M. D. and Balents, L. (2011). Physical Review B, 84(23), 235126.
  * (10) Chen, C., Wang, S. S., Liu, L., Yu, Z. M., Sheng, X. L., Chen, Z. and Yang, S. A. (2017). Physical Review Materials, 1(4), 044201.
  * (11) Zhu, Z., Liu, Y., Yu, Z. M., Wang, S. S., Zhao, Y. X., Feng, Y., et. al. (2018). Physical Review B, 98(12), 125104.
  * (12) Wu, W., Jiao, Y., Li, S., Sheng, X. L., Yu, Z. M. and Yang, S. A. (2019). Physical Review Materials, 3(5), 054203.
  * (13) Zhang, X., Yu, Z. M., Lu, Y., Sheng, X. L., Yang, H. Y. and Yang, S. A. (2018). Physical Review B, 97(12), 125143.
  * (14) Li, S., Yu, Z. M., Liu, Y., Guan, S., Wang, S. S., Zhang, X., et. al. (2017). Physical Review B, 96(8), 081106.
  * (15) Wu, W., Liu, Y., Li, S., Zhong, C., Yu, Z. M., Sheng, X. L., et. al. (2018). Physical Review B, 97(11), 115125.
  * (16) Gibson, Q. D., Schoop, L. M., Muechler, L., Xie, L. S., Hirschberger, M., Ong, N. P., et. al. (2015). Physical Review B, 91(20), 205128.
  * (17) Zhang, T., Jiang, Y., Song, Z., Huang, H., He, Y., Fang, Z., et. al. (2019). Nature, 566(7745), 475-479.
  * (18) Song, Z., Zhang, T. and Fang, C. (2018). Physical Review X, 8(3), 031069.
  * (19) Vergniory, M. G., Elcoro, L., Felser, C., Regnault, N., Bernevig, B. A. and Wang, Z. (2019). Nature, 566(7745), 480-485.
  * (20) Tang, F., Po, H. C., Vishwanath, A. and Wan, X. (2019). Nature, 566(7745), 486-489.
  * (21) Kruthoff, J., De Boer, J., Van Wezel, J., Kane, C. L. and Slager, R. J. (2017). Physical Review X, 7(4), 041069.
  * (22) Po, H. C., Vishwanath, A. and Watanabe, H. (2017). Nature Communications, 8(1), 1-9.
  * (23) Song, Z., Zhang, T., Fang, Z. and Fang, C. (2018). Nature Communications, 9(1), 1-7.
  * (24) Bradlyn, B., Elcoro, L., Cano, J., Vergniory, M. G., Wang, Z., Felser, C., et. al. (2017). Nature, 547(7663), 298-305.
  * (25) Yu, Z. M., Zhang, Z., Liu, G. B., Wu, W., Li, X. P., Zhang, R. W., et. al. arXiv preprint arXiv:2102.01517.
  * (26) Miao, H., Zhang, T. T., Wang, L., Meyers, D., Said, A. H., Wang, Y. L., et. al. (2018). Physical Review Letters, 121(3), 035302.
  * (27) Jin, Y., Wang, R. and Xu, H. (2018). Nano Letters, 18(12), 7755-7760.
  * (28) Wang, R., Xia, B. W., Chen, Z. J., Zheng, B. B., Zhao, Y. J. and Xu, H. (2020). Physical Review Letters, 124(10), 105303.
  * (29) Jin, Y. J., Chen, Z. J., Xia, B. W., Zhao, Y. J., Wang, R. and Xu, H. (2018). Physical Review B,98(22), 220103.
  * (30) Zhang, T. T., Miao, H., Wang, Q., Lin, J. Q., Cao, Y., Fabbris, G., et. al. (2019). Physical Review Letters, 123(24), 245302.
  * (31) Liu, Q. B., Fu, H. H., Xu, G., Yu, R. and Wu, R. (2019). The Journal of Physical Chemistry Letters, 10(14), 4045-4050.
  * (32) Liu, Q. B., Qian, Y., Fu, H. H. and Wang, Z. (2020). npj Computational Materials, 6(1), 1-6.
  * (33) Li, J., Wang, L., Liu, J., Li, R., Zhang, Z. and Chen, X. Q. (2020). Physical Review B,101(8), 081403.
  * (34) Liu, J., Hou, W., Wang, E., Zhang, S., Sun, J. T. and Meng, S. (2019). Physical Review B,100(8), 081204.
  * (35) Li, J., Liu, J., Baronett, S. A., Liu, M., Wang, L., Li, R., et. al. (2021). Nature Communications, 12(1), 1-12.
  * (36) Liu, P. F., Li, J., Tu, X. H., Li, H., Zhang, J., Zhang, P., et. al. (2021). Physical Review B, 103(9), 094306.
  * (37) Zheng, B., Xia, B., Wang, R., Chen, Z., Zhao, J., Zhao, Y. and Xu, H. (2020). Physical Review B, 101(10), 100303.
  * (38) Liu, Q. B., Wang, Z. and Fu, H. H. (2021). Physical Review B, 103(16), L161303.
  * (39) Jin, Y. J., Chen, Z. J., Xiao, X. L. and Xu, H. (2021). Physical Review B, 103(10), 104101.
  * (40) Liu, Y., Chen, X. and Xu, Y. (2020). Advanced Functional Materials, 30(8), 1904784.
  * (41) Singh, S., Wu, Q., Yue, C., Romero, A. H. and Soluyanov, A. A. (2018). Physical Review Materials, 2(11), 114204.
  * (42) Li, N., Ren, J., Wang, L., Zhang, G., H$\ddot{\rm{a}}$nggi, P. and Li, B. (2012). Reviews of Modern Physics, 84(3), 1045.
  * (43) Huber, S. D. (2016). Nature Physics, 12(7), 621-623.
  * (44) Liu, Y., Xu, Y. and Duan, W. (2018). National Science Review, 5(3), 314-316.
  * (45) Long, Y., Ren, J. and Chen, H. (2020). Physical Review Letters, 124(18), 185501.
  * (46) Yang, T., Liu, Y., Wu, Z., Wang, X. and Zhang, G. (2021). Materials Today Physics, doi: 10.1016/j.mtphys.2021.100348.
  * (47) Wang, X., Cheng, Z., Zhang, G., Wang, B., Wang, X. L. and Chen, H. (2020). Nanoscale, 12(15), 8314-8319.
  * (48) Yang, T., Ding, G., Cheng, Z., Wang, X. and Zhang, G. (2020). Journal of Materials Chemistry C, 8(23), 7741-7748.
  * (49) Baur, W. H. (1956). Acta Crystallographica,9(6), 515-520.
  * (50) Perdew, J. P., Burke, K. and Ernzerhof, M. (1998). Physical Review Letters, 80(4), 891.
  * (51) Bl$\ddot{\rm{o}}$chl, P. E. (1994). Physical review B, 50(24), 17953.
  * (52) Togo, A. and Tanaka, I. (2015). Scripta Materialia, 108, 1-5.
  * (53) Wu, Q., Zhang, S., Song, H. F., Troyer, M. and Soluyanov, A. A. (2018). Computer Physics Communications, 224, 405-416.
  * (54) Liu, G. B., Chu, M., Zhang, Z., Yu, Z. M. and Yao, Y. (2021). Computer Physics Communications, 107993.
  * (55) Bradley, C. and Cracknell, A. (2009). The mathematical theory of symmetry in solids: representation theory for point groups and space groups. Oxford University Press.
  * (56) Jarzebski, Z. M. and Marton, J. P. (1976). Journal of the electrochemical Society, 123(7), 199C.
  * (57) Wang, X., Zhou, F., Yang, T., Kuang, M., Yu, Z. M. and Zhang, G. (2021). arXiv preprint arXiv:2103.15495.
  * (58) Young, S. M. and Kane, C. L. (2015). Physical Review Letters, 115(12), 126803.
  * (59) Winkler, G. W., Singh, S. and Soluyanov, A. A. (2019). Chinese Physics B, 28(7), 077303.
  * (60) Chen, Z. J., Wang, R., Xia, B. W., Zheng, B. B., Jin, Y. J., Zhao, Y. J. and Xu, H. (2021). Physical Review Letters, 126(18), 185301.


th]]. relax[relaxrelaxreference99] S. Stieberger and T.R. Taylor, “New
relations for Einstein–Yang–Mills amplitudes,” Nucl. Phys. B 913, 151 (2016).
[arXiv:1606.09616 [hep-th]]. relax[relaxrelaxreference1010] P. Mazloumi and S.
Stieberger, “Einstein Yang-Mills amplitudes from intersections of twisted
forms,” JHEP 06, 125 (2022) [arXiv:2201.00837 [hep-th]].
relax[relaxrelaxreference1111] Z. Bern, L.J. Dixon, M. Perelstein and J.S.
Rozowsky, “Multileg one loop gravity amplitudes from gauge theory,” Nucl.
Phys. B 546, 423 (1999). [hep-th/9811140]. relax[relaxrelaxreference1212] S.
Stieberger, “Open & Closed vs. Pure Open String One–Loop Amplitudes,”
[arXiv:2105.06888 [hep-th]]. relax[relaxrelaxreference1313] E.D’Hoker and D.H.
Phong, “The Geometry of String Perturbation Theory,” Rev. Mod. Phys. 60, 917
(1988). relax[relaxrelaxreference1414] S. Stieberger and T.R. Taylor,
“Non–Abelian Born-Infeld action and type I - heterotic duality (2):
Nonrenormalization theorems,” Nucl. Phys. B 648, 3 (2003). [hep-th/0209064].
relax[relaxrelaxreference1515] J.R. Ellis, P. Jetzer and L. Mizrachi, “One
Loop String Corrections to the Effective Field Theory,” Nucl. Phys. B 303, 1
(1988). relax[relaxrelaxreference1616] A. Tsuchiya, “More on One Loop Massless
Amplitudes of Superstring Theories,” Phys. Rev. D 39, 1626 (1989).
relax[relaxrelaxreference1717] N.E.J. Bjerrum-Bohr, P.H. Damgaard, T.
Sondergaard and P. Vanhove, “The Momentum Kernel of Gauge and Gravity
Theories,” JHEP 1101, 001 (2011). [arXiv:1010.3933 [hep-th]].
relax[relaxrelaxreference1818] S. Hohenegger and S. Stieberger, “Monodromy
Relations in Higher-Loop String Amplitudes,” Nucl. Phys. B 925, 63 (2017).
[arXiv:1702.04963 [hep-th]]. relax[relaxrelaxreference1919] J. Broedel and A.
Kaderli, “Amplitude recursions with an extra marked point,” Commun. Num.
Theor. Phys. 16, no.1, 75-158 (2022) [arXiv:1912.09927 [hep-th]].
relax[relaxrelaxreference2020] P. Tourkine and P. Vanhove, “Higher-loop
amplitude monodromy relations in string and gauge theory,” Phys. Rev. Lett.
117, no. 21, 211601 (2016). [arXiv:1608.01665 [hep-th]].
relax[relaxrelaxreference2121] E. Casali, S. Mizera and P. Tourkine,
“Monodromy relations from twisted homology,” JHEP 1912, 087 (2019).
[arXiv:1910.08514 [hep-th]]. relax[relaxrelaxreference2222] A.G. Cohen, G.W.
Moore, P.C. Nelson and J. Polchinski, “Semi Off-shell String Amplitudes,”
Nucl. Phys. B 281, 127 (1987). relax[relaxrelaxreference2323] A.G. Cohen, G.W.
Moore, P.C. Nelson and J. Polchinski, “An Off-Shell Propagator for String
Theory,” Nucl. Phys. B 267, 143-157 (1986) relax[relaxrelaxreference2424] C.S.
Hsue, B. Sakita and M.A. Virasoro, “Formulation of dual theory in terms of
functional integrations,” Phys. Rev. D 2, 2857-2868 (1970).
relax[relaxrelaxreference2525] S. Stieberger and T.R. Taylor, “Disk Scattering
of Open and Closed Strings (I),” Nucl. Phys. B 903, 104 (2016).
[arXiv:1510.01774 [hep-th]]. relax[relaxrelaxreference2626] Y. Geyer, L.
Mason, R. Monteiro and P. Tourkine, “Loop Integrands for Scattering Amplitudes
from the Riemann Sphere,” Phys. Rev. Lett. 115, no. 12, 121603 (2015).
[arXiv:1507.00321 [hep-th]]. relax[relaxrelaxreference2727] Y. Geyer, L.
Mason, R. Monteiro and P. Tourkine, “One-loop amplitudes on the Riemann
sphere,” JHEP 1603, 114 (2016). [arXiv:1511.06315 [hep-th]].
relax[relaxrelaxreference2828] S. He and O. Schlotterer, “New Relations for
Gauge-Theory and Gravity Amplitudes at Loop Level,” Phys. Rev. Lett. 118
(2017) no.16, 161601 [arXiv:1612.00417 [hep-th]]; A. Edison, M. Guillen, H.
Johansson, O. Schlotterer and F. Teng, “One-loop matrix elements of effective
superstring interactions: $\alpha^{\prime}$–expanding loop integrands,” JHEP
12, 007 (2021) [arXiv:2107.08009 [hep-th]]. relax[relaxrelaxreference2929] Z.
Bern, L.J. Dixon, D.C. Dunbar, M. Perelstein and J.S. Rozowsky, “On the
relationship between Yang-Mills theory and gravity and its implication for
ultraviolet divergences,” Nucl. Phys. B 530, 401 (1998). [hep-th/9802162].
relax[relaxrelaxreference3030] S. Mizera, “Combinatorics and Topology of
Kawai-Lewellen-Tye Relations,” JHEP 1708, 097 (2017). [arXiv:1706.08527 [hep-
th]]. relax[relaxrelaxreference3131] F. Brown, “Single-valued Motivic Periods
and Multiple Zeta Values,” SIGMA 2, e25 (2014). [arXiv:1309.5309 [math.NT]].
relax[relaxrelaxreference3232] S. Stieberger, “Closed superstring amplitudes,
single-valued multiple zeta values and the Deligne associator,” J. Phys. A 47,
155401 (2014). [arXiv:1310.3259 [hep-th]]. relax[relaxrelaxreference3333] S.
Stieberger and T.R. Taylor, “Closed String Amplitudes as Single-Valued Open
String Amplitudes,” Nucl. Phys. B 881, 269 (2014). [arXiv:1401.1218 [hep-th]].
relax[relaxrelaxreference3434] F. Brown and C. Dupont, “Single-valued
integration and double copy,” J. Reine Angew. Math. 775 (2021), 145–196
[arXiv:1810.07682 [math.NT]]; “Single-valued integration and superstring
amplitudes in genus zero,” Commun. Math. Phys. 382 (2021) no.2, 815-874
[arXiv:1910.01107 [math.NT]]. relax[relaxrelaxreference3535] E.D’Hoker, M. B.
Green, Ö. Gürdogan and P. Vanhove, “Modular Graph Functions,” Commun. Num.
Theor. Phys. 11 (2017), 165-218 [arXiv:1512.06779 [hep-th]]; F. Brown, “A
class of non-holomorphic modular forms I,” [arXiv:1707.01230 [math.NT]]; “A
Class of Nonholomorphic Modular Forms II: Equivariant Iterated Eisenstein
Integrals,” Forum Math. Sigma 8 (2020), e31. relax[relaxrelaxreference3636]
C.P. Burgess and T.R. Morris, “Open and Unoriented Strings À La Polyakov,”
Nucl. Phys. B 291 (1987), 256-284; S.K. Blau, M. Clements, S. Della Pietra, S.
Carlip and V. Della Pietra, “The String Amplitude on Surfaces With Boundaries
and Crosscaps,” Nucl. Phys. B 301 (1988), 285-303.
relax[relaxrelaxreference3737] E.D’Hoker and J. Kaidi, “Lectures on modular
forms and strings,” [arXiv:2208.07242 [hep-th]].
relax[relaxrelaxreference3838] W. Magnus, F. Oberhettinger, and R.P. Soni,
Formulas and Theorems for the Special Functions of Mathematical Physics,
Springer 1966; A. Erdélyi, W. Magnus, F. Oberhettinger, and F.G. Tricomi,
Higher transcendental functions, Volume II, McGraw-Hill Book Company 1953.


# FaceQAN: Face Image Quality Assessment Through Adversarial Noise Exploration

Žiga Babnik  Faculty of Electrical Engineering, University of Ljubljana
Tržaška cesta 25, 1000 Ljubljana
<EMAIL_ADDRESS>Vitomir Štruc  Faculty of Electrical Engineering,
University of Ljubljana
Tržaška cesta 25, 1000 Ljubljana
<EMAIL_ADDRESS>Žiga Babnik1, Peter Peer2 and Vitomir Štruc1
1University of Ljubljana, Faculty of Electrical Engineering Tržaška cesta 25,
1000 Ljubljana, Slovenia
2University of Ljubljana, Faculty of Computer and Information Science, Večna
Pot 113, 1000 Ljubljana, Slovenia
E-mail: {ziga.babnik<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent state-of-the-art face recognition (FR) approaches have achieved
impressive performance, yet unconstrained face recognition still represents an
open problem. Face image quality assessment (FIQA) approaches aim to estimate
the quality of the input samples that can help provide information on the
confidence of the recognition decision and eventually lead to improved results
in challenging scenarios. While much progress has been made in face image
quality assessment in recent years, computing reliable quality scores for
diverse facial images and FR models remains challenging. In this paper, we
propose a novel approach to face image quality assessment, called FaceQAN,
that is based on adversarial examples and relies on the analysis of
adversarial noise which can be calculated with any FR model learned by using
some form of gradient descent. As such, the proposed approach is the first to
link image quality to adversarial attacks. Comprehensive (cross-model as well
as model-specific) experiments are conducted with four benchmark datasets,
i.e., LFW, CFP–FP, XQLFW and IJB–C, four FR models, i.e., CosFace, ArcFace,
CurricularFace and ElasticFace, and in comparison to seven state-of-the-art
FIQA methods to demonstrate the performance of FaceQAN. Experimental results
show that FaceQAN achieves competitive results, while exhibiting several
desirable characteristics. The source code for FaceQAN is available at
https://github.com/LSIbabnikz/FaceQAN.

##  I Introduction

In recent years, Face Recognition (FR) techniques have achieved excellent
results on datasets containing both high quality frontal images and images of
variable quality [1, 2, 3]. Face recognition in completely unconstrained
settings, on the other hand, remains challenging, as no quality guarantees can
be made for facial images captured in-the-wild [4, 5, 6]. Face Image Quality
Assessment (FIQA) methods aim to improve the face recognition performance in
such settings, by providing additional information to the FR models
corresponding to the quality of the input face images. Based on this
information, a FR model can reject low quality images, which often times cause
critical False Non-Match (FNM) errors [7].

According to ISO/IEC 29794-1, the quality of a biometric sample can be defined
using its character, fidelity or utility [8]. Contemporary FIQA methods
typically rely on the latter and most often define quality in terms of the
utility of the input face sample for the FR task. Here, the term utility is
used to describe the fitness of a sample to accomplish or fulfill a biometric
function [7]. In this setting, a single numerical score is commonly calculated
to capture the quality of the given face image [7]. From a conceptual view
point, contemporary state-of-the-art FIQA techniques can be broadly
partitioned into two distinct groups: i.e., $(i)$ regression-based methods and
$(ii)$ model-based approaches. Techniques from the first group learn a mapping
directly from the image space to automatically-generated pseudo ground-truth
quality labels. Several labeling approaches have been introduced over the
years to facilitate this process, the basis for which are
comparison/similarity scores calculated using a selected FR model. Recent
techniques, for example, consider either comparison-score distributions
generated over mated image pairs from a given dataset as the basis for the
reference quality labels [9, 10], or alternatively, the distribution of
similarity scores between probe samples and mated reference samples of a known
quality [11]. Techniques from the second group, on the other hand, integrate
the quality estimation process directly with the considered FR model and
commonly predict the quality of the input samples based on the uncertainty of
the computed face representations (features/embeddings) [12, 13, 14]. Such
techniques are tightly linked to the given FR model and usually designed
without the need for supervised learning. While the quality scores produced by
existing approaches from either group have been shown to be good predictors of
the utility of the input images for face recognition, they are still limited
by the validity of the pseudo ground truth labels and poor integration with
the targeted FR model.

High quality

Medium quality

Low quality

Noise Sample

Figure 1: Visualization of adversarial noise as a function of image quality. A
clear difference can be observed in the characteristics of the adversarial
noise between images of different quality. Higher quality images produce noise
masks with a more distinct face-like pattern, while the lower quality masks
are closer to random noise. FaceQAN takes advantage of the presented
observations to estimate the quality of the input images.

In this paper, we propose a novel, unsupervised FIQA approach called FaceQAN
(Face Image Quality Assessment Through Adversarial Noise Exploration) that
avoids the pseudo quality label generation process and, consequently, the
supervised learning process, by using information available in the input
facial image and a given (targeted) FR model only. The basis for FaceQAN are
adversarial examples, which can be generated for all modern FR models learned
through some form of gradient descent [15]. The proposed method works under
the assumption that the difficulty of adversarial example generation is
directly correlated with the quality of a given image w.r.t. a certain FR
model, as shown on Fig 1. In other words, good quality images are expected to
produce stable and robust representations that are difficult to perturb using
adversarial noise. Based on this insight, we make the following main
contributions in this paper:

  * •

We propose FaceQAN, a novel approach for generating quality scores for face
images that ensures competitive results on several datasets and in comparison
to a wide range of recent state-of-the-art FIQA techniques.

  * •

To the best of our knowledge, we are the first to link adversarial attacks to
the task of face image quality assessment and show that such linkage leads to
highly desirable FIQA characteristics.

##  II Related Work

In this section, we position FaceQAN with respect to the two main groups of
existing methods, i.e., regression- and model-based approaches. A more
comprehensive coverage of the field can be found in the recent survey paper in
[7].

Regression-Based FIQA. Techniques from this group utilize pseudo quality
(ground truth) labels for the prediction of quality scores. The ground-truth
label-generation process is usually automatic [11, 16, 10] but can also
involve human assessment [17, 18]. The most recent methods in the literature
utilize deep learning models paired with automatic extraction of ground truth
labels. One such method, called FaceQNet, presented by Hernandez-Ortega et al.
in [11], is based on a ResNet-$50$ model trained on pseudo labels generated
from the VGGFace2 [19] dataset. The ground truth labels are obtained by first
selecting the highest quality image of the individuals in the set, calculated
using third party ISO/IEC 19794-5 [20, 21] compliance software. For all other
images in the set, the quality is calculated as the normalized Euclidean
distance between the embeddings of the given image and of the highest quality
image of a particular individual. A pretrained ResNet-$50$ network is then
fine-tuned on the generated ground truth labels. A more advanced labeling
process is presented by Ou et al. [16] in the SDD-FIQA method. Here, the
authors suggest using both the inter-class and intra-class distances to
produce ground truth labels. For a given image, both mated and non-mated
comparison-score distributions are constructed first. The quality score is
then computed as the average Wasserstein distance between the distributions
over several runs. A face recognition model is finally fine-tuned on the
obtained ground truth labels, similar to FaceQNet.

While the discussed approaches perform well on standard datasets such as LFW
[22], they heavily rely on the construction of ground truth quality labels.
This process often introduces biases associated with either the utilized FR
model or the selected dataset. Since the end goal is to improve the face
recognition performance of a particular FR model, with its own associated
biases, such FIQA methods may not produce optimal quality scores for different
datasets and FR models. To avoid such shortcomings, the proposed FaceQAN
quality estimation approach relies only on the information contained in the
given image and a targeted FR model, avoiding any learning and, consequently,
the process of generating pseudo ground truth labels. In cases where the
targeted FR model is not available, the FaceQAN quality assessment process can
also be performed with an arbitrary surrogate FR model.

Model-Based FIQA. Solutions in this group typically combine the face
recognition and FIQA tasks, and learn to produce the embeddings and the
quality (or uncertainty) of the input image simultaneously. One such method,
PFE (Probabilistic Face Embeddings) presented by Shi and Jain [12], learns to
predict the uncertainty of a given image, by estimating the mean and variance
of the computed feature vectors. Here, the mean vector corresponds to the
embedding of the given image, whereas the variance presents the uncertainty of
the image in the feature space. A quality score is obtained by calculating the
harmonic mean over the variance vector. More recently Meng et al. [13]
presented MagFace an approach that incorporates the prediction of the image
uncertainty into the face-recognition learning process. The authors achieved
this by introducing a new loss function built upon the ArcFace loss, which
enables better separation for embeddings with higher magnitudes. The quality
score of a given image is calculated simply as the norm of its embeddings, if
the embedding model is trained using the presented loss. While such methods do
not need to explicitly extract quality ground truth labels, they enforce a
learning regime upon the targeted FR model. The proposed FaceQAN method, on
the other hand, is learning-free and applicable directly to any (pretrained)
targeted model [15] without the need for interventions into the learning
procedure.

Our approach is most closely related to SER-FIQ, presented by Terhörst et al.
in [14]. SER-FIQ uses the properties of dropout, a regularization technique
commonly used in modern convolutional neural networks (CNNs) to avoid
overfitting. More precisely, several embeddings are first produced for a
single input image using different sub-network layouts, generated using
dropout. The quality score is then computed as the sigmoid of the negative
normalized sum of distances over all pairs of produced embeddings. Similarly
to SER-FIQ, FaceQAN also aims to capture the uncertainty of the embeddings
(computed for an input face image) for quality assessment, but does so through
the analysis of a set of generated adversarial examples.

##  III Methodology

An ideal FIQA method should reflect the biases and performance of the targeted
FR model, while being generally applicable to different model topologies
trained with arbitrary learning objectives. Thus, only information originating
from the targeted FR model and the given input sample should be considered in
the quality estimation process. The main contribution of this work, FaceQAN,
presented in this section, follows the outlined logic by estimating the sample
quality using adversarial noise, which can be generated for all modern FR
networks, trained using some variant of gradient descent [15], as well as
information available directly in the input image. Different from most
adversarial methods, our goal is not to create a specific adversarial example,
but rather to measure the difficulty of adversarial example generation.

###  III-A Overview of FaceQAN

Figure 2: High-level overview of FaceQAN. FaceQAN estimates the quality of an
input sample $I$ by exploring the characteristics of adversarial examples
$I^{+}$ in the embedding space of the targeted FR model $M$. The final quality
is estimated through an aggregation function $F$ that considers the similarity
between the embedding of the input sample and $k$ adversarial examples, i.e.,
$i=1,\ldots,k$. Moreover, the impact of pose on quality is modeled explicitly
within FaceQAN through a face-symmetry estimation procedure based on image
flipping. The figure is best viewed electronically and in color.

Given a targeted FR model $M$ and an input face image $I\in\mathbb{R}^{w\times
h\times 3}$, the goal of face image quality assessment is to generate a
quality score $Q(I)\in\mathbb{R}$ that reflects the utility of $I$ with
respect to $M$. With FaceQAN, the quality score for an input image $I$ is
computed by analyzing the characteristics of adversarial examples $I^{+}$ in
the embedding space of the targeted FR models, i.e., $Q(M(I),M(I^{+}))$. While
the pixel space could also be considered for the exploration of the
adversarial noise, using the embedding space ensures tighter integration with
the targeted FR models and allows FaceQAN to better capture the utility of the
input face samples for FR.

The complete FaceQAN pipeline, illustrated in Fig. 2, consists of four
distinct steps: $(i)$ an attack initialization step (§III-B) that generates
multiple noisy images from the given input face to facilitate the attack
procedure, $(ii)$ an adversarial-example generation step (§III-C) that
produces the attack samples needed for the analysis, $(iii)$ a symmetry
estimation step (§III-D) that specifically accounts for the impact of pose on
image quality, and $(iv)$ a quality-score calculation step (§III-E) that
computes the final quality value based on the analysis of the generated
adversarial embeddings. All steps are described in-detail in the following
sections.

###  III-B Attack Initialization

Standard adversarial attacks typically require an input sample $I$, a FR model
$M$ and the ground truth class/identity label $y$ of the input sample to be
able to generate adversarial examples [23, 24, 25]. Because face image quality
assessment needs to be applicable to arbitrary facial images irrespective of
the identities/labels used to learn the model $M$, we utilize the similarity-
based adversarial attack (SAA) criterion, proposed recently by Wang et al. in
[26], as the basis for FaceQAN and adapt it to fit our problem. SAA allows us
to use the embedding of the input image $I$ as the ground truth label to
attack, i.e., $y=M(I)$, and to define an (angular) dissimilarity loss to drive
the adversarial noise generation process, i.e.:

$L(M(I^{*}),y)=1-\frac{M(I^{*})^{T}\cdot y}{\|M(I^{*})\|\|y\|},$ (1)

where $\|\cdot\|$ is the $L_{2}$ norm. Since adversarial examples are
generated by maximizing $L$, we ensure that the gradient of $L$ with respect
to the input $I^{*}$ is non-zero, and, therefore, define $I^{*}$ as a noise
perturbed version of the original input image $I$. As long as the noise
infused into $I$ is minute, the embedding of $I^{*}$ is expected to be close
to the embedding of the original input image $I$, as also illustrated in Fig.
2.

Let $I\in\mathbb{R}^{w\times h\times 3}$ be an input face image with values in
the range of $[-1,1]$. FaceQAN generates the noisy counterpart to $I$ required
for the attack initialization as:

$I^{*}=\lfloor
I+N\rfloor_{[-1,1]},\;\;\;N\sim\mathcal{U}(-\epsilon,\epsilon),$ (2)

where the noise $N$ is sampled from a uniform distribution $\mathcal{U}$,
$\epsilon$ is an open hyperparameter, and $\lfloor\cdot\rfloor_{[-1,1]}$
denotes a clipping operator that guarantees that the resulting noisy image
$I^{*}$ contains pixels in the range $[-1,1]$. Choosing a value of $\epsilon$
close to 0 assures that the amount of noise added to the image is minuscule
and that the change in the embedding is limited.

Because the added noise initializes the adversarial attack in an arbitrary
direction within the embedding space, we minimize this randomness by
generating a set of $k$ noisy images, i.e., $\\{I^{*}_{i}\\}_{i=1}^{k}$, by
sampling the noise independently from $\mathcal{U}$ $k$ times. This procedure
allows FaceQAN to explore the embedding space around $M(I)$ in various
directions and estimate the stability of the embedding for quality estimation.

###  III-C Adversarial Example Generation

In the second step, FaceQAN uses the set of noisy images
$\\{I^{*}_{i}\\}_{i=1}^{k}$ to generate a corresponding set of adversarial
examples $\\{{I}^{+}_{i}\\}_{i=1}^{k}$. We note again, that the use of
adversarial methods in FaceQAN deviates from the usual, as we are not
interested in generating adversarial examples that can fool a targeted FR
model $M$, but rather in the characteristics of the generated examples after
an attack with fixed attack hyperparameters. FaceQAN is in general applicable
with any adversarial approach, but we select the Fast Gradient Sign Method
(FGSM) [23] together with the Basic Iterative Method (BIM) [27] for the
implementation in this paper due to their simplicity and ease of
implementation.

Fast Gradient Sign Method. In our setting, FGSM generates an adversarial
example $I^{+}$ from $I$ such that the difference between the true label
$y=M(I)$ and the embedding of the adversarial example $M(I^{+})$ is maximized.
The procedure starts by first generating the embedding of the input image
$M(I)$ and then calculating the loss from Eq. (1) between the generated
prediction and the true label, i.e., $L(M(I),y)$. The calculated loss is then
back-propagated to the input $I$, for which the gradient $\nabla_{I}L(M(I),y)$
is computed. Finally, the adversarial example is constructed by defining the
adversarial noise $\mu$ as the sign function of the gradient, i.e.,
$sign(\nabla_{I}L(M(I),y)$, and in turn: $I^{+}=I+\epsilon\cdot\mu$, where
$\epsilon$ is again an open hyperparameter of the method, that controls the
amount of noise added to $I$. A smaller $\epsilon$ corresponds to a lower
amount of adversarial noise in the final adversarial example $I^{+}$.

Basic Iterative Method. We use BIM to improve the attack capabilities of FGSM
and to better explore the embedding space around $M(I)$. BIM extends FGSM by
creating adversarial examples over $l$ FGSM runs, where only the input $I$
changes between iterations. Additionally, the parameter $\epsilon$ is scaled
as $\frac{\epsilon}{l}$, to limit the overall amount of noise added to the
image. The initial iteration of BIM is identical to FGSM, whereas for all
consequent iterations $m\in[2,l]$ the input $I$ is defined as the adversarial
example $I^{+}_{m-1}$ produced by the previous iteration. Performing $l$
iterations, we obtain the final adversarial example, i.e.:
$I^{+}\xleftarrow[]{}BIM_{l}(I,y,M).$ The set of adversarial examples
$\\{{I}^{+}_{i}\\}_{i=1}^{k}$ is generated from the set of noisy images using
simple batch processing.

###  III-D Symmetry Estimation

A critical factor known to affect face recognition performance are pose
variations [28, 29, 30]. For FaceQAN, we, therefore, design an additional step
to explicitly account for the effect of pose in the input images. To avoid
unnecessary complexity, we use the following simple logic. Generally faces are
vertically symmetrical and as such, if presented with a fully frontal face
image, flipping the image horizontally should not have major effects on the
produced embedding. This does not hold true for larger pose variations. Thus,
we create a horizontally flipped image $I^{f}$ for each given input $I$ and
use its embedding alongside the embeddings of the adversarial set
$\\{{I}^{+}_{i}\\}_{i=1}^{k}$ when computing the final quality score.

###  III-E Quality Score Calculation

In the last step, the adversarial set $\\{{I}^{+}_{i}\\}_{i=1}^{k}$ and the
flipped image $I^{f}$ are passed through the model $M$ to obtain a set of
adversarial embeddings $\\{{y}^{+}_{i}\\}_{i=1}^{k}=\\{\,M(I^{+}_{i})\mid
i\in[1,k]\,\\}$ and flipped embedding $y_{f}=M(I^{f})$. Both are compared to
the embedding $y=M(I)$, using the cosine similarity, which well matches the
learning objectives of modern FR models:

$\cos\alpha_{i}=S_{i}=\frac{y^{T}\cdot
y^{+}_{i}}{\|y\|\|y^{+}_{i}\|},\;\;\;\cos\alpha_{f}=s_{f}=\frac{y^{T}\cdot
y_{f}}{\|y\|\|y_{f}\|},$ (3)

where the set $\\{S_{i}\\}_{i=1}^{k}$ contains similarities for each
adversarial example in $\\{{I}^{+}_{i}\\}_{i=1}^{k}$, and $\alpha_{i}$ and
$\alpha_{f}$ are the angles between the embeddings - see Fig. 2. The set of
similarities $\\{S_{i}\\}_{i=1}^{k}$ and $s_{f}$ are then used to calculate
the final quality score using the aggregation function
$F:\\{\\{S_{i}\\}_{i=1}^{k},s_{f}\\}\mapsto Q$, defined by:

$q_{adv}=\frac{\mu_{S}+1}{2}\cdot\lfloor(1-\sigma_{S})\rfloor_{[0,1]},\
Q=\left(\ q_{adv}\cdot s_{f}\right)^{p}$ (4)

where $\mu_{S}$ represents the mean over $\\{S_{i}\\}_{i=1}^{k}$ and
$\sigma_{S}$ is the corresponding standard deviation. Hence, $F$ considers the
mean of the similarity scores as well as their dispersion when computing the
quality score, and additionally weights the estimated quality with the
computed symmetry prediction. The main intuition behind the adversarial part
of this procedure is that lower quality images map to a poorly defined part of
the embedding space that is easily perturbed, i.e., the adversarial examples
have a low average and high standard deviation.

Because the cosine similarity is bound between $[-1,1]$, we re-scale both the
mean and standard deviation to $[0,1]$. The final quality is as such defined
on $[0,1]$, where higher values represent images of better quality with
respect to $M$. To ensure the final quality scores $Q(I)$ cover the full range
of values on $[0,1]$, we use a power law as the last computational step in
FaceQAN, where $p$ is an open hyperparameter. Note that this final step has no
impact on the quality-estimation process and only affects the range of the
aggregation function $F$.

##  IV Experiments and Results

TABLE I: Summary of the experimental setup

Dataset | #Images | #IDs | #Comparisons | Main Quality Factors†‡
---|---|---|---|---
Mated | Non-mated | Pose | O-E | B-R-N | Sc
LFW [22] | $13{\small,}233$ | $5{\small,}749$ | $3{\small,}000$ | $3{\small,}000$ | L | L | L | M
CFP-FP [31] | $7{\small,}000$ | $500$ | $3{\small,}500$ | $3{\small,}500$ | H | M | L | M
XQLFW [32] | $13{\small,}233$ | $5{\small,}749$ | $3{\small,}000$ | $3{\small,}000$ | L | L | H | M
IJB-C [33] | $23{\small,}124^{\dagger\dagger}$ | $3{\small,}531$ | $19{\small,}557$ | $15{\small,}638{\small,}932$ | H | H | H | Lr
†O-E - Occlusion, Expression; B-R-N - Blur, Resolution, Noise; Sc - Scale. |
‡L - Low; M - Medium; H - High; Lr - Large; Values estimated subjectively by the authors. |
†† number of templates, each containing several images |

Figure 3: ERC results at FMR=$0.001$. FaceQAN is evaluated using three state-
of-the-art FR models over four datasets and in comparison to seven state-of-
the-art baselines and yields highly competitive results. All results
correspond to cross-model (C) quality assessment experiments. For reference
purposes, model-specific FaceQAN results, marked (S), are also reported. The
figure is best viewed electronically and in color.

###  IV-A Experimental Setup

Evaluation Setting. We compare FaceQAN with seven state-of-the-art FIQA
methods. We chose FaceQNet [11], MagFace [13], PCNet [9], LightQNet [10], and
SDD-FIQA [16], as representatives of regression-based techniques, and PFE [12]
and SER-FIQ [14] as examples of model-based approaches. To ensure a fair
comparison, publicly available official implementations are used or code
obtained directly from the authors. As summarized in Table IV, we test FaceQAN
on four benchmark datasets with diverse (quality) characteristics, i.e.:
Labeled Faces in the Wild (LFW) [22], Celebrities in Frontal-Profile in the
Wild (CFP-FP) [31], Cross-Quality LFW (XQLFW) [32] and IJB-C (IARPA Janus
Benchmark – C) [33], similarly to [34]. The experiments are conducted with
three popular (state-of-the-art) open-source FR models, referred to as
ArcFace111https://github.com/deepinsight/insightface [35],
CurricularFace222https://github.com/HuangYG123/CurricularFace [36], and
ElasticFace333https://github.com/fdbtrs/ElasticFace [37] hereafter based on
their learning objectives. All three FR models are based on the ResNet100
architecture. ArcFace is trained on MS1MV3, ElasticFace on MS1MV2, whereas
CurricularFace is trained on CASIA-WebFace and MS1MV2. To enable a fair
comparison with the competing state-of-the-art FIQA methods, all results are
generated using a cross-model (C) setting, where quality scores are generated
using one FR model and performance is measured on another. To compute quality
scores for FaceQAN we use the CosFace1 [38] model trained with the cosface
objective with a ResNet100 architecture on the Glint360K dataset [39]. For
reference purposes, we also provide model-specific FaceQAN results (S), where
access to a target FR model is assumed and, hence, the same FR model is used
for quality prediction and performance scoring.

Performance Evaluation. Following standard methodology [13, 14, 16, 7], the
Error Versus Reject Characteristic (ERC) is used to score performance. ERC
measures the False Non Match Rate (FNMR) at a constant False Match Rate (FMR)
usually set to $0.001$, when increasing the fraction of unconsidered images.
Images are rejected based on their calculated quality label. Additionally, we
also measure the Area Under the Curve (AUC) for the ERC plots, where lower
scores imply better performance. As suggested in [7], we report AUC values at
several drop rates, i.e., $0.1$, $0.2$, $0.4$ and $0.8$. Here, smaller drop
rates are typically more relevant, since this is when the images of the lowest
quality are removed.

Implementation Details. Since we are interested only in exploring the
localized space around the computed face embedding, $M(I)$, a combination of a
small $\epsilon$ and low $l$ is chosen for the experiments. Specifically, we
set $\epsilon=0.001$ and $l=5$ as the values for the BIM parameters. Ideally,
the batch size should follow $k\gg d_{M(I)}$, where $d_{M(I)}$ represents the
dimensions of the embedding space, so that all directions around $M(I)$ are
well explored. However, because the dimensionality of the embedding spaces is
usually large ($>512$), we set $k=10$ to ensure a reasonable trade-off between
the extent of the embedding space covered and the computational complexity of
FaceQAN. We chose $p=5$ since this enables the full use of values on $[0,1]$.
The benchmark images are cropped and aligned as specified per the chosen face
recognition model. The same procedure is also used for the baselines unless a
different preprocessing approach was proposed or included in the paper
describing the baseline. All experiments are conducted on a desktop PC with an
Intel i9-10900KF CPU, $64$ GB of RAM and an Nvidia $3090$ GPU.

TABLE II:  Comparison with state-of-the-art - AUC@FMR$1$e-$3$ [$\times
10^{-3}$] ($\downarrow$). B \- Best overall, \- Best in COMP, - Best in CLOSE

FR Model | ArcFace – AUC@FMR1E-3 [$\times 10^{-3}$]
---|---
DT† | DR† | Comparison Baselines (COMP)‡ | Closely Related (CLOSE)‡
FQN | SDD | PFE | PCNet | MagFace | LQN | SER | FQ(C)† | FQ(S)†
LFW | $10\%$ | $0.49$ | $0.44$ | $0.45$ | $0.4$ | $0.47$ | $\mathbf{0.38}$ | $0.44$ | $0.43$ | $0.43$
$20\%$ | $0.87$ | $0.73$ | $0.82$ | $0.69$ | $0.89$ | $0.7$ | $0.77$ | $\mathbf{0.66}$ | $0.66$
$40\%$ | $1.49$ | $1.37$ | $1.4$ | $1.25$ | $1.43$ | $1.28$ | $1.3$ | $\mathbf{1.03}$ | $1.03$
$80\%$ | $2.52$ | $2.2$ | $2.36$ | $\mathbf{1.73}$ | $2.08$ | $2.34$ | $2.49$ | $1.84$ | $1.83$
CFP-FP | $10\%$ | $2.07$ | $1.28$ | $1.37$ | $1.3$ | $1.53$ | $1.65$ | $0.92$ | $\mathbf{0.81}$ | $0.82$
$20\%$ | $3.83$ | $1.74$ | $2.01$ | $1.84$ | $2.39$ | $2.75$ | $1.3$ | $\mathbf{0.96}$ | $0.98$
$40\%$ | $6.29$ | $2.15$ | $2.74$ | $2.46$ | $3.14$ | $4.33$ | $1.87$ | $\mathbf{1.17}$ | $1.17$
$80\%$ | $8.75$ | $3.0$ | $3.26$ | $2.95$ | $3.77$ | $5.95$ | $2.67$ | $\mathbf{1.7}$ | $1.74$
XQLFW | $10\%$ | $55.71$ | $54.99$ | $54.51$ | $53.26$ | $54.44$ | $53.84$ | $\mathbf{52.06}$ | $53.19$ | $53.24$
$20\%$ | $110.68$ | $107.41$ | $104.84$ | $94.59$ | $105.72$ | $97.17$ | $\mathbf{90.9}$ | $93.56$ | $93.7$
$40\%$ | $219.04$ | $188.38$ | $175.9$ | $152.36$ | $203.53$ | $157.45$ | $142.6$ | $\mathbf{139.48}$ | $143.3$
$80\%$ | $422.55$ | $286.99$ | $231.52$ | $199.0$ | $371.23$ | $210.83$ | $187.38$ | $\mathbf{175.51}$ | $181.18$
IJB-C | $10\%$ | $3.29$ | $2.64$ | $2.72$ | $2.51$ | $2.7$ | $2.64$ | $\mathbf{2.3}$ | $2.37$ | $2.41$
$20\%$ | $7.14$ | $4.58$ | $4.91$ | $4.4$ | $5.11$ | $4.51$ | $4.14$ | $\mathbf{4.14}$ | $4.25$
$40\%$ | $16.87$ | $7.72$ | $8.7$ | $7.86$ | $9.28$ | $7.94$ | $7.47$ | $\mathbf{7.32}$ | $7.49$
$80\%$ | $42.12$ | $13.56$ | $14.6$ | $13.7$ | $15.36$ | $13.01$ | $13.84$ | $\mathbf{12.11}$ | $12.21$
FR Model | CurricularFace – AUC@FMR1E-3 [$\times 10^{-3}$]
DT† | DR† | Comparison Baselines (COMP)‡ | Closely Related (CLOSE)‡
FQN | SDD | PFE | PCNet | MagFace | LQN | SER | FQ(C)† | FQ(S)†
LFW | $10\%$ | $\mathbf{0.32}$ | $0.38$ | $0.45$ | $0.37$ | $0.42$ | $0.38$ | $0.45$ | $0.37$ | $0.41$
$20\%$ | $\mathbf{0.57}$ | $0.65$ | $0.81$ | $0.72$ | $0.78$ | $0.71$ | $0.87$ | $0.58$ | $0.76$
$40\%$ | $1.04$ | $1.25$ | $1.35$ | $1.31$ | $1.36$ | $1.21$ | $1.54$ | $\mathbf{0.99}$ | $1.34$
$80\%$ | $1.9$ | $2.0$ | $2.38$ | $1.93$ | $2.01$ | $2.19$ | $3.0$ | $1.66$ | $\mathbf{1.53}$
CFP-FP | $10\%$ | $2.2$ | $1.56$ | $1.44$ | $1.37$ | $1.64$ | $1.8$ | $1.19$ | $1.55$ | $\mathbf{1.03}$
$20\%$ | $3.91$ | $2.09$ | $2.04$ | $1.83$ | $2.54$ | $2.85$ | $1.68$ | $2.45$ | $\mathbf{1.25}$
$40\%$ | $6.39$ | $2.5$ | $2.65$ | $2.39$ | $3.3$ | $4.17$ | $2.37$ | $3.96$ | $\mathbf{1.55}$
$80\%$ | $8.7$ | $3.37$ | $3.19$ | $3.0$ | $3.93$ | $5.61$ | $3.16$ | $5.99$ | $\mathbf{2.17}$
XQLFW | $10\%$ | $74.15$ | $73.89$ | $73.56$ | $\mathbf{67.69}$ | $73.85$ | $70.45$ | $70.22$ | $68.48$ | $70.66$
$20\%$ | $147.75$ | $146.09$ | $142.97$ | $\mathbf{124.55}$ | $146.23$ | $134.75$ | $132.63$ | $129.23$ | $136.07$
$40\%$ | $294.9$ | $269.95$ | $252.79$ | $213.54$ | $282.63$ | $227.54$ | $221.3$ | $\mathbf{211.55}$ | $236.28$
$80\%$ | $579.14$ | $429.54$ | $338.31$ | $293.0$ | $529.29$ | $281.82$ | $291.27$ | $\mathbf{272.03}$ | $316.06$
IJB-C | $10\%$ | $3.69$ | $2.91$ | $3.01$ | $2.75$ | $3.01$ | $2.91$ | $\mathbf{2.53}$ | $2.58$ | $2.62$
$20\%$ | $7.98$ | $5.04$ | $5.43$ | $4.87$ | $5.65$ | $4.93$ | $\mathbf{4.49}$ | $4.54$ | $4.63$
$40\%$ | $18.86$ | $8.37$ | $9.42$ | $8.58$ | $9.92$ | $8.49$ | $7.96$ | $\mathbf{7.95}$ | $8.05$
$80\%$ | $47.14$ | $14.21$ | $15.36$ | $14.51$ | $16.1$ | $13.61$ | $14.46$ | $12.86$ | $\mathbf{12.74}$
FR Model | ElasticFace – AUC@FMR1E-3 [$\times 10^{-3}$]
DT† | DR† | Comparison Baselines (COMP)‡ | Closely Related (CLOSE)‡
FQN | SDD | PFE | PCNet | MagFace | LQN | SER | FQ(C)† | FQ(S)†
LFW | $10\%$ | $\mathbf{0.57}$ | $0.68$ | $0.66$ | $0.72$ | $0.69$ | $0.59$ | $0.74$ | $0.59$ | $0.59$
$20\%$ | $0.97$ | $1.14$ | $1.17$ | $1.32$ | $1.18$ | $1.05$ | $1.23$ | $\mathbf{0.85}$ | $0.95$
$40\%$ | $1.68$ | $1.94$ | $1.77$ | $2.24$ | $1.86$ | $1.69$ | $2.0$ | $\mathbf{1.24}$ | $1.42$
$80\%$ | $2.94$ | $2.97$ | $2.79$ | $3.02$ | $2.53$ | $2.69$ | $3.57$ | $1.88$ | $\mathbf{1.61}$
CFP-FP | $10\%$ | $2.42$ | $1.78$ | $1.69$ | $1.71$ | $1.88$ | $2.05$ | $1.42$ | $\mathbf{1.16}$ | $1.22$
$20\%$ | $4.25$ | $2.44$ | $2.43$ | $2.34$ | $2.95$ | $3.26$ | $2.08$ | $\mathbf{1.45}$ | $1.55$
$40\%$ | $6.82$ | $2.95$ | $3.17$ | $2.99$ | $3.84$ | $4.76$ | $2.86$ | $1.84$ | $\mathbf{1.8}$
$80\%$ | $9.04$ | $3.87$ | $3.71$ | $3.49$ | $4.49$ | $6.26$ | $3.68$ | $2.3$ | $\mathbf{2.28}$
XQLFW | $10\%$ | $75.04$ | $74.67$ | $74.54$ | $73.4$ | $75.02$ | $73.23$ | $\mathbf{71.9}$ | $74.21$ | $74.36$
$20\%$ | $149.38$ | $147.68$ | $146.64$ | $131.99$ | $148.6$ | $135.74$ | $\mathbf{130.25}$ | $134.91$ | $144.68$
$40\%$ | $295.97$ | $282.37$ | $280.81$ | $\mathbf{219.83}$ | $291.61$ | $226.24$ | $225.59$ | $219.84$ | $247.56$
$80\%$ | $577.87$ | $460.28$ | $393.29$ | $301.86$ | $534.96$ | $297.76$ | $312.28$ | $\mathbf{288.4}$ | $320.85$
IJB-C | $10\%$ | $3.65$ | $2.79$ | $2.79$ | $2.66$ | $2.87$ | $2.79$ | $\mathbf{2.4}$ | $2.47$ | $2.51$
$20\%$ | $7.96$ | $4.78$ | $5.03$ | $4.6$ | $5.39$ | $4.68$ | $4.3$ | $\mathbf{4.28}$ | $4.32$
$40\%$ | $18.91$ | $8.0$ | $8.92$ | $8.14$ | $9.62$ | $8.16$ | $7.83$ | $\mathbf{7.46}$ | $7.48$
$80\%$ | $47.41$ | $13.99$ | $14.98$ | $14.04$ | $15.86$ | $13.37$ | $14.26$ | $12.28$ | $\mathbf{12.13}$
†DT – Dataset; DR – Drop Rate (or Ratio of unconsidered images). (C) – cross-
model; (S) – model-specific
‡FQN – FaceQNet; SDD – SDD- FIQA; LQN – LightQNet; SER – SER-FIQ, FQ - FaceQAN
(ours)

TABLE III: Analysis of the time complexity over CFP-FP using CosFace

Complexity | FaceQnet | SDD-FIQA | PFE | PCNet | MagFace | LightQnet | SER-FIQ
---|---|---|---|---|---|---|---
t [s] | $\mu$ | $0.0432$ | $0.0006$ | $0.0493$ | $0.0175$ | $0.0011$ | $0.0535$ | $0.1125$
$\sigma$ | $0.0026$ | $0.0004$ | $0.0275$ | $0.0004$ | $0.0004$ | $0.0468$ | $0.0418$

Complexity | FaceQAN (ours)
---|---
$k=2$ | $k=5$ | ${}^{\dagger}k=10$ | $k=50$ | $k=100$
t [s] $(\mu\pm\sigma$) | $0.21\pm 0.019$ | $0.23\pm 0.018$ | $0.30\pm 0.006$ | $1.00\pm 0.027$ | $1.87\pm 0.031$
$\dagger$ Configuration used in experiments

###  IV-B Comparison with the State-Of-The-Art

The ERC plots for all combinations of FR models and datasets are shown in Fig.
3 and the calculated AUC scores over the ERC plots (multiplied by $10^{3}$ for
readability) in Table II. Below we analyze these results from two aspects:
$(i)$ in comparison to all considered (supervised) baselines (COMP; see Table
II), and $(ii)$ in comparison to the closely-related SER-FIQ, which does not
rely on training (CLOSE).

TABLE IV: AUC [$\times 10^{-3}$] scores ($\downarrow$) of the ablation study

FR | Dataset | FIQA Model | Image Drop Rate
---|---|---|---
$10\%$ | $20\%$ | $40\%$ | $80\%$
ArcFace | LFW | FaceQAN | $\mathbf{0.43}$ | $\mathbf{0.66}$ | $\mathbf{1.03}$ | $\mathbf{1.83}$
w/o Symm. Est. | $0.53$ | $0.99$ | $1.5$ | $2.5$
CFP-FP | FaceQAN | $\mathbf{0.82}$ | $\mathbf{0.98}$ | $\mathbf{1.17}$ | $\mathbf{1.74}$
w/o Symm. Est. | $0.89$ | $1.25$ | $1.56$ | $2.21$
Cu. Face† | LFW | FaceQAN | $0.41$ | $0.76$ | $1.34$ | $\mathbf{1.53}$
w/o Symm. Est. | $\mathbf{0.35}$ | $\mathbf{0.67}$ | $\mathbf{1.31}$ | $1.63$
CFP-FP | FaceQAN | $\mathbf{1.03}$ | $\mathbf{1.25}$ | $\mathbf{1.55}$ | $\mathbf{2.17}$
w/o Symm. Est. | $1.06$ | $1.4$ | $1.75$ | $2.48$
ElasticFace | LFW | FaceQAN | $\mathbf{0.59}$ | $\mathbf{0.95}$ | $\mathbf{1.42}$ | $\mathbf{1.61}$
w/o Symm. Est. | $0.63$ | $1.05$ | $1.65$ | $2.03$
CFP-FP | FaceQAN | $\mathbf{1.22}$ | $\mathbf{1.55}$ | $\mathbf{1.8}$ | $\mathbf{2.28}$
w/o Symm. Est. | $1.38$ | $1.89$ | $2.62$ | $3.32$
$\dagger$ Cu.Face – CurricularFace

Baseline Comparisons (COMP). FaceQAN is the most convincing approach when
compared to the state-of-the-art methods from the COMP group on all four
dataset when considering the ArcFace model despite not relying on supervised
training or (pseudo) reference labels. Similar observations can also be made
for the CurricularFace and ElasticFace models, where FaceQAN again yields
highly competitive results on all four datasets in the cross-model (C) and
model-specific setting (S). The only exeption is the cross-model result with
CurricularFace on the CFP-FP dataset, where FaceQAN is less convincing. It is
also interesting to observe that the relative ranking of the competing FIQA
techniques changes with different FR models. FaceQAN, on the other hand, is
among the top performers with all three considered FR models.

Comparison to SER-FIQ (CLOSE). SER-FIQ and FaceQAN are the only models in our
experiments that require no supervision and estimate face quality solely based
on the input image and a FR model. As can be seen from Table II, the cross-
model version of FaceQAN(C) is overall the top performer, suggesting that the
adversarial examples generated with the CosFace model are particularly
informative for quality estimation and therefore often outperform FaceQAN in
the model-specific setting. SER-FIQ is most competitive on the XQLFW dataset.
However, it needs to be noted that XQLFW was designed by optimizing SER-FIQ
quality scores and is, therefore, highly biased towards this FIQA approach
[32]. Convincing results are also achieved with SER-FIQ on the large-scale
IJB-C dataset at the lowest drop rate, where the performance is comparable to
FaceQAN(C).

Time Complexity. In Table III we analyze the time complexity of FaceQAN in
comparison with the considered baselines and as a function of the number of
generated adversarial examples $k$. The analysis is done over the CFP-FP
dataset and with the CosFace model. As expected, the time complexity of
FaceQAN increases with an increase in $k$ and is overall somewhat higher than
that of the competing solutions due to the nature of the BIM approach selected
for our implementation.

###  IV-C Ablation Study

To demonstrate the impact of the symmetry estimation step on the performance
of FaceQAN(S), we perform an ablation study on the LFW and CFP-FP datasets
using the three selected FR models (ArcFace, CurricularFace, ElasticFace) with
and without (w/o) the symmetry scores $s_{f}$ from Eq. (3). As can be seen
from the results in Table IV, removing the step from the quality assessment
process has a detrimental effect on the AUC scores, which consistently
increase in comparison to the full FaceQAN model across all tested
combinations, except when using CurricularFace on the LFW dataset.

###  IV-D Qualitative Evaluation

Figure 4: Quality-based ranking with FaceQAN. Note how the ordering of the
XQLFW sample images based on the generated quality scores (shown below the
images) corresponds to the perceived face quality. Zoom-in for details.

Image Ranking. In Fig. 4 we show example images from XQLFW ordered according
to the computed quality scores. As can be seen, the images follow a reasonable
order in terms of perceived quality for all three FR models. Because the
ordering is meant to reflect the utility of the samples for face recognition,
it is interesting to see how FaceQAN(S) favors blurry frontal images in
certain settings over crisp, but less frontal (and w/o neutral expressions)
samples.

Quality-Score Distribution. Fig. 5 shows that the quality-score distributions
generated for the ArcFace, CurricularFace and ElasticFace models exhibit a
relatively similar shape, but have a somewhat different support on some
datasets. Overall, however, all models generate reasonably consistent score
distributions (also given the results in Fig. 4) that well capture the
quality/utility of the input data.

Figure 5: Quality-score distribution generated by FaceQAN. Distributions are
shown for (from left to right): LFW, CFP-FP, XQLFW, and IJB-C. The three FR
models produce distributions on different scales but of similar shape.

##  V Conclusion

We presented a novel approach to unsupervised face image quality assessment,
called FaceQAN. The proposed approach elegantly avoids the quality-label
generation and learning process used by many state-of-the-art FIQA techniques
by harnessing adversarial noise, which can be generated for any modern deep
learning model [15]. By comparing the embeddings of adversarial examples and
the original input sample, FaceQAN is able to calculate a quality score that
is an excellent predictor of the sample’s utility for face recognition.
Extensive experiments with several baselines, datasets and FR models have
shown that FaceQAN achieves highly competitive results, while being based on
minimal assumptions. As part of our future work, we plan to explore the use of
adversarial examples in supervised settings and the predictive power of
adversarial noise with (pseudo) reference quality labels.

## Acknowledgments

Supported by the ARRS Research Program P2–0250 (B) as well as the ARRS Junior
Researcher Program.

## References

  * [1] K. Grm, V. Štruc, A. Artiges, M. Caron, and H. K. Ekenel, “Strengths and weaknesses of deep learning models for face recognition against image degradations,” _IET Biometrics_ , vol. 7, no. 1, pp. 81–89, 2018.
  * [2] K. Grm and V. Struc, “Deep face recognition for surveillance applications,” _IEEE Intelligent Systems_ , vol. 33, no. 3, pp. 46–50, 2018.
  * [3] B. Meden, P. Rot, P. Terhörst, N. Damer, A. Kuijper, W. J. Scheirer, A. Ross, P. Peer, and V. Štruc, “Privacy–enhancing face biometrics: A comprehensive survey,” _IEEE Transactions on Information Forensics and Security_ , 2021.
  * [4] M. Wang and W. Deng, “Deep face recognition: A survey,” _Neurocomputing_ , vol. 429, pp. 215–244, 2021.
  * [5] F. Boutros, N. Damer, J. N. Kolf, K. Raja, F. Kirchbuchner, R. Ramachandra, A. Kuijper, P. Fang, C. Zhang, F. Wang, D. Montero, N. Aginako, B. Sierra, M. Nieto, M. E. Erakin, U. Demir, H. K. Ekenel, A. Kataoka, K. Ichikawa, S. Kubo, J. Zhang, M. He, D. Han, S. Shan, K. Grm, V. Štruc, S. Seneviratne, N. Kasthuriarachchi, S. Rasnayaka, P. C. Neto, A. F. Sequeira, J. R. Pinto, M. Saffari, and J. S. Cardoso, “MFR 2021: Masked Face Recognition Competition,” in _Proceedings of the IEEE International Joint Conference on Biometrics (IJCB)_ , 2021.
  * [6] K. Grm, W. J. Scheirer, and V. Štruc, “Face hallucination using cascaded super-resolution and identity priors,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 2150–2165, 2020.
  * [7] T. Schlett, C. Rathgeb, O. Henniger, J. Galbally, J. Fierrez, and C. Busch, “Face image quality assessment: A literature survey,” _ACM Computing Surveys_ , 2022.
  * [8] ISO/IEC JTC 1/SC 37 Biometrics, “Information Technology - Biometric Sample Quality - Part 1: Framework,” International Organization for Standardization, Standard ISO/IEC 29794-1:2016, 2016.
  * [9] W. Xie, J. Byrne, and A. Zisserman, “Inducing predictive uncertainty estimation for face verification,” in _British Machine Vision Conference (BMVC)_ , 2020.
  * [10] K. Chen, T. Yi, and Q. Lv, “Lightqnet: Lightweight deep face quality assessment for risk-controlled face recognition,” _IEEE Signal Processing Letters_ , vol. 28, pp. 1878–1882, 2021.
  * [11] J. Hernandez-Ortega, J. Galbally, J. Fierrez, R. Haraksim, and L. Beslay, “Faceqnet: Quality assessment for face recognition based on deep learning,” in _Proceedings of the IEEE International Conference on Biometrics (ICB)_ , 2019, pp. 1–8.
  * [12] Y. Shi and A. K. Jain, “Probabilistic face embeddings,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 6902–6911.
  * [13] Q. Meng, S. Zhao, Z. Huang, and F. Zhou, “Magface: A universal representation for face recognition and quality assessment,” in _CVF/IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 14 225–14 234.
  * [14] P. Terhorst, J. N. Kolf, N. Damer, F. Kirchbuchner, and A. Kuijper, “SER-FIQ: Unsupervised Estimation of Face Image Quality Based on Stochastic Embedding Robustness,” in _CVF/IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 5651–5660.
  * [15] N. Akhtar, A. Mian, N. Kardan, and M. Shah, “Advances in adversarial attacks and defenses in computer vision: A survey,” _IEEE Access_ , vol. 9, pp. 155 161–155 196, 2021.
  * [16] O. Fu-Zhao, X. Chen, R. Zhang, Y. Huang, S. Li, J. Li, Y. Li, L. Cao, and W. Yuan-Gen, “SDD-FIQA: Unsupervised Face Image Quality Assessment with Similarity Distribution Distance,” in _CVF/IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 7670–7679.
  * [17] P. Wasnik, R. Ramachandra, K. Raja, and C. Busch, “An empirical evaluation of deep architectures on generalization of smartphone-based face image quality assessment,” in _IEEE International Conference on Biometrics Theory, Applications and Systems (BTAS)_ , 2018, pp. 1–7.
  * [18] L. Best-Rowden and A. K. Jain, “Learning face image quality from human assessments,” _IEEE Transactions on Information forensics and security_ , vol. 13, no. 12, pp. 3064–3077, 2018.
  * [19] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” in _IEEE International Conference on Automatic Face & Gesture Recognition (FG)_, 2018, pp. 67–74.
  * [20] D. Maltoni, A. Franco, M. Ferrara, D. Maio, and A. Nardelli, “Biolab-icao: A new benchmark to evaluate applications assessing face image compliance to iso/iec 19794-5 standard,” in _IEEE International Conference on Image Processing (ICIP)_ , 2009, pp. 41–44.
  * [21] ISO/IEC JTC 1/SC 37 Biometrics, “Information technology - Biometric Data Initerchange Formats - Part 5: Face Image Data,” International Organization for Standardization, Standard ISO/IEC 19794-5:2011, 2011.
  * [22] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” University of Massachusetts, Amherst, Tech. Rep. 07-49, October 2007.
  * [23] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [24] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _International Conference on Learning Representations (ICLR)_ , 2018.
  * [25] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _IEEE Symposium on Security and Privacy (SSP)_ , 2017, pp. 39–57.
  * [26] H. Wang, S. Wang, Z. Jin, Y. Wang, C. Chen, and T. Massimo, “Similarity-based gray-box adversarial attack against deep face recognition,” in _IEEE International Conference on Automatic Face & Gesture Recognition (FG)_, 2021\.
  * [27] A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” in _Artificial intelligence safety and security_. Chapman and Hall/CRC, 2018, pp. 99–112.
  * [28] V. Štruc and N. Pavešić, “The complete gabor-fisher classifier for robust face recognition,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2010, pp. 1–26, 2010.
  * [29] J. Zhao, Y. Cheng, Y. Xu, L. Xiong, J. Li, F. Zhao, K. Jayashree, S. Pranata, S. Shen, J. Xing _et al._ , “Towards pose invariant face recognition in the wild,” in _CVF/IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 2207–2216.
  * [30] S. Banerjee, J. Brogan, J. Krizaj, A. Bharati, B. R. Webster, V. Struc, P. J. Flynn, and W. J. Scheirer, “To frontalize or not to frontalize: Do we really need elaborate pre-processing to improve face recognition?” in _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2018, pp. 20–29.
  * [31] S. Sengupta, J. C. Cheng, C. D. Castillo, V. M. Patel, R. Chellappa, and D. W. Jacobs, “Frontal to profile face verification in the wild,” in _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2016.
  * [32] M. Knoche, S. Hormann, and G. Rigoll, “Cross-quality lfw: A database for analyzing cross-resolution image face recognition in unconstrained environments,” in _IEEE International Conference on Automatic Face and Gesture Recognition (FG)_ , 2021, pp. 1–5.
  * [33] B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, J. Cheney _et al._ , “IARPA Janus Benchmark-C: Face dataset and protocol,” in _International Conference on Biometrics (ICB)_ , 2018, pp. 158–165.
  * [34] F. Boutros, M. Fang, M. Klemt, B. Fu, and N. Damer, “CR-FIQA: Face Image Quality Assessment by Learning Sample Relative Classifiability,” _arXiv preprint arXiv:2112.06592_ , 2021.
  * [35] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive Angular Margin Loss for Deep Face Recognition,” in _CVF/IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 4690–4699.
  * [36] Y. Huang, Y. Wang, Y. Tai, X. Liu, P. Shen, S. Li, J. Li, and F. Huang, “CurricularFace: Adaptive curriculum learning loss for deep face recognition,” in _CVF/IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 5901–5910.
  * [37] F. Boutros, N. Damer, F. Kirchbuchner, and A. Kuijper, “Elasticface: Elastic margin loss for deep face recognition,” 2021.
  * [38] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu, “CosFace: Large margin cosine loss for deep face recognition,” in _CVF/IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 5265–5274.
  * [39] X. An, X. Zhu, Y. Gao, Y. Xiao, Y. Zhao, Z. Feng, L. Wu, B. Qin, M. Zhang, D. Zhang _et al._ , “Partial FC: Training 10 million identities on a single machine,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 1445–1449.