# Explainability of the Implications of Supervised and Unsupervised Face Image
Quality Estimations Through Activation Map Variation Analyses in Face
Recognition Models

Biying Fu1, Naser Damer1,2
1Fraunhofer Institute for Computer Graphics Research IGD, Darmstadt, Germany
2Department of Computer Science, TU Darmstadt, Darmstadt, Germany
Email<EMAIL_ADDRESS>
###### Abstract

It is challenging to derive explainability for unsupervised or statistical-
based face image quality assessment (FIQA) methods. In this work, we propose a
novel set of explainability tools to derive reasoning for different FIQA
decisions and their face recognition (FR) performance implications. We avoid
limiting the deployment of our tools to certain FIQA methods by basing our
analyses on the behavior of FR models when processing samples with different
FIQA decisions. This leads to explainability tools that can be applied for any
FIQA method with any CNN-based FR solution using activation mapping to exhibit
the network’s activation derived from the face embedding. To avoid the low
discrimination between the general spatial activation mapping of low and high-
quality images in FR models, we build our explainability tools in a higher
derivative space by analyzing the variation of the FR activation maps of image
sets with different quality decisions. We demonstrate our tools and analyze
the findings on four FIQA methods, by presenting inter and intra-FIQA method
analyses. Our proposed tools and the analyses based on them point out, among
other conclusions, that high-quality images typically cause consistent low
activation on the areas outside of the central face region, while low-quality
images, despite general low activation, have high variations of activation in
such areas. Our explainability tools also extend to analyzing single images
where we show that low-quality images tend to have an FR model spatial
activation that strongly differs from what is expected from a high-quality
image where this difference also tends to appear more in areas outside of the
central face region and does correspond to issues like extreme poses and
facial occlusions. The implementation of the proposed tools is accessible here
111https://github.com/fbiying87/Explainable_FIQA_WITH_AMVA.

##  1 Introduction

Face recognition (FR) systems are becoming more widely used in our daily life,
be it for security-relevant areas such as border control or for unlocking your
personal devices such as smartphones. This spread relates highly to the
performance improvements due to advances made in deep-learning methods for FR
[6, 2, 3].

Low utility [18] face samples largely effects the performance of FR algorithms
[1]. Therefore, to improve the performance of FR and overcome the
aforementioned challenges, advances have been made to enhance the performance
and dependability by choosing to use high-quality face images. Face image
quality is also used to weight face embeddings and comparison scores when
performing multi-frame (video) [21, 5, 20] or multi-spectrum [19] face
verification. Facial image quality assessment (FIQA) methods have been
developed to evaluate this metric for facial images. Recent advances in the
development of FIQA methods with deep-learning-based approaches already show
good results in improving the FR performance. Few works focus also on building
FIQA methods based on interpretable reasoning in terms of uncertainty [26],
embedding robustness [27], or embedding magnitude that correlate to the sample
location with respect to its class [21]. However, little effort was put into
the explainability of these methods.

To address this unscouted field, we develop a set of novel explainability
tools. To start with, and to enable our tools to be deployed to both
supervised and unsupervised FIQA approaches, we do not look into the responses
of the FIQA approach itself, but rather the FR model behavior when processing
images with various FIQ decisions. Given that the general spatial activation
of FR models to low or high-quality images is rather similar, we take this
analysis to the higher derivative level by looking at the activation map
variation analyses. The newly proposed tools successfully target answering
three questions related to the explainability of FIQA decisions. These
questions address the following issues: 1) What makes the face images of high
quality (in comparison to low) in the view of different FIQA based on the FR
model behavior, 2) what makes the decision of high or low quality different
between different FIQAs based on the FR model behavior, and 3) how do the
activation mappings caused by face images in the FR model deviate from what is
expected from high-quality images and how does that reflect in their quality
score.

The structure of our work is as follows: we first introduce the related work
in terms of FIQA methods in Section 2. In section 3, we introduce our proposed
explainability tools. The experimental setup is described in Section 4, while
the analyses based on our proposed tools are followed in Section 5. We
conclude our paper in Section 6 where we shortly summarized our main findings.

##  2 Related Works

The estimation of face images utility with FIQA methods helps FR systems to
improve their performance by avoiding the processing of low-quality, and thus
low-utility captures.

Most recent works of FIQA methods mainly focused on enhancing the performance
of the proposed metric in terms of the FR accuracy when neglecting low-quality
samples. An example of that is the probabilistic Face Embeddings (PFEs)
proposed by Shi et al. [26] where they represented each face image as a
Gaussian distribution in the latent space, where the variance of the Gaussian
indicates the uncertainty in the feature space. This uncertainty is used as a
measure of quality. Another example is the SDD-FIQA [23] where Ou et al.
proposed a supervised method using generated quality pseudo-labels by
calculating the Wasserstein Distance between the intra-class similarity
distribution and inter-class similarity distributions of sample identities.
With these peseudo-labels, the network trains a regression network for the
quality prediction. Other FIQA metrics using the trustworthiness as in SER-FIQ
[27] or MagFace [21] by learning a universal feature embedding which magnitude
is a direct measure of quality, are further introduced in Section 4.

Recent works have tried to have a detailed look into the contribution of
facial parts to the estimated face image quality (FIQ). Fu et al. [10]
investigated the different face sub-regions (including eyes, mouth, and nose)
and showed their relative importance towards face utility by comparing general
image quality assessment (IQA) metrics on these areas. However, this work only
looked at IQA metrics and thus does not provide many insights on the
explainability of FIQA. Further works have looked into the spatial activation
maps of supervised FIQA and IQA methods and analyzed how they are affected
when facing different sample categories such as low and high-quality [9], or
even masked faces [11]. However, these efforts were limited to the supervised
FIQA methods and did not address the better performing unsupervised FIQA
methods, as the nature of the unsupervised approaches does not allow for
rational activation map analyses. Moreover, none of the previous FIQA methods
tried to provide an explaination of a quality decision by looking into the
spatial interpretation of the response of FR models to what is deemed as low
or high-quality samples, rather than just analyzing the FIQA behavior. In this
work, we propose a generalized methodology to enhance the explainability of
the behavior of FR models on low and high-quality samples, as well as face
image quality estimation decisions, by analyzing the FR activation mappings,
rather than these of FIQA.

##  3 Methodology

In this section, we propose a novel set of tools to explain the face image
quality and its effect on FR model behavior, independent of the underlying
working principles of the FIQA methods itself, by focusing on the response of
the FR models to samples labeled with different qualities. We leveraged the
process of activation mapping of a visualization network to display the scaled
activation weighted by the face embedding of the FR network. This mapping
links the content of face embeddings with the pixels in the input face image.
Based on the activation mapping, we draw statistic characteristics for images
with different face qualities and thus enable analyses of the response of FR
models to low and high face image qualities and the quality interpretation of
single samples based on FR model responses.

To illustrate our proposed method with a concrete example, let us assume that
we chose MagFace [21] as the underlying unsupervised FIQA method and used the
ResNet-100 [13] model trained with ArcFace loss [6] as the face recognition
model to extract the face embeddings. We further used ScoreCAM [28] as the
approach for the activation mapping process. The activation mapping visualized
the deepest convolution layer of the Res-Net100 and upsampled it to overlay to
the input layer. The scaled version of the activation measures how the output
changes to the face embedding. For each image, the activation mapping (AM)
provided an output activation map with each pixel value noted as
$a_{i,j},i=1:112,j=1:112$ of the size 112x112\. However, this concept can be
extended to any FIQA method and CNN-based FR model. More details about the
exact models used in this work and the reason for this selection are provided
later in Section 4.

Using the selected FIQA metric, e.g., MagFace, we calculate the face image
scores for a given face images database. The calculated face image scores are
used as ground truth to determine the group of low and high-quality images. In
our experiment, we chose the 10% of face images with the lowest and 10% of
face images with the highest FIQ scores. For simplicity, these two groups are
named H and L individually.

For each of these two individual groups of H and L images, we introduce and
define the mean activation mappings (MAM) and denote them as MAMH and MAML.
These maps have the same dimension as the input with 112x112 pixels. Each
element in the MAM is noted as $\overline{a_{i,j}}$ and it is derived from
equation (1) using the activation value of each single sample $a_{i,j}$ with
the running index $i=1:112$ and $j=1:112$:

$\overline{a_{i,j}}=\frac{1}{N}\sum_{k=1}^{N}a_{i,j}^{k},$ (1)

where $N$ is the number of images within the H or L groups respectively. As we
aim to measure the variability in the activation mapping, we introduce and
define the activation mapping variation map (AM-V). We further denote them as
AM-VH and AM-VL respectively, for high-quality or low-quality samples. These
maps also have the dimension of the input with a shape of 112x112 and each
element in the AM-V is the $\text{s}_{i,j}$ and is derived according to
equation (2):

$\text{s}_{i,j}=\sqrt{\frac{1}{N}\sum_{k=1}^{N}(a_{i,j}^{k}-\overline{a_{i,j}})^{2}},$
(2)

where $N$ has the same meaning as in Equation (1) and $a_{i,j}$ are extracted
from elements of the activation mapping.

In order to reduce the influence of outliers in the MAM, we further looked at
the median activation mapping (MDAM). The notation of MDAMH and MDAML are also
noting the MDAM for low and high quality sample sets. The element of the MDAM
is denoted as $\widetilde{a_{i,j}}$ and is derived as:

$\widetilde{a_{i,j}}=Median(a_{i,j}^{k}),k=1..N\text{.}$ (3)

We further introduce the activation mapping Median variation (AM-MV), called
AM-MVH and AM-MVL. The associated equation for each element of these maps
$\widetilde{s_{i,j}}$ is given as:

$\widetilde{s_{i,j}}=\sqrt{\frac{1}{N}\sum_{k=1}^{N}(a_{i,j}^{k}-\widetilde{a_{i,j}})^{2}},$
(4)

where both AM-MV maps have the same dimension of 112x112 pixels.

Both the defined AM-V and AM-MV maps present a visualization tool to look into
the spatial areas where a relatively large variation in the activation of the
FR occurs, with respect to a set of high or low-quality images. This will help
identify the spatial regions responsible for the certain quality decision,
despite the low consistency of these areas’ location across different images.

As will be shown later, the differences between the MAM (or MDAM) of image
sets of different qualities do not uncover a lot of explainability
information. Therefore, to uncover the spatial related differences between
these groups, we rather analyze the differences between the variations in the
activation mapping (AM-V or AM-MV). We introduce these differences as the
Differential activation mapping variation (D-AM-V) as in (5) and the
Differential activation mapping Median variation (D-AM-MV), as in (6):

$\text{D-AM-V}=|\text{AM-V}_{H}-\text{AM-V}_{L}|,$ (5)

and

$\text{D-AM-MV}=|\text{AM-MV}_{H}-\text{AM-MV}_{L}|,$ (6)

where both equations (5) and (6) can be extended to look at the differences of
variations of any sets of images, not only L and H, but also to sets of images
determined to be H or L by different FIQA approaches.

The proposed visualization maps provide a useful tool to analyze the
differences in FR model responses to sets of facial images belonging to
different sets, here sets with different FIQ determined by any FIQA
approaches, or sets of a certain FIQ label determined by different FIQA
approaches.

So far we introduced methods to visualize and analyze the behavior of FR
models between sets of images to enable a better understanding of FR response
to images of different quality levels and thus understanding the used FIQA. To
further analyze the quality decision of a single face image using its FR model
response, we introduce the activation deviation from the MAM (AD-MAM). The
elements of the AD-MAM is noted as $d_{i},j$ and is calculated of an image x
with its activation mapping element $a_{i,j}$ and its absolute deviation from
the mean activation mapping of the high quality sets $\overline{a_{i,j}^{H}}$,

$d_{i,j}=|a_{i,j}-\overline{a_{i,j}^{H}}|,$ (7)

this can be calculated for different sets of images that build the MAM, here
we focus our analyses on the MAMH, and thus we note our AD-MAM as AD-MAMH.

##  4 Experimental Setup

In this section, we first introduce the face image database used to evaluate
the methodology proposed in Section 3. This was followed by the description of
the three recent deep-learning-based FIQA metrics and one general IQA metric
used as examples in our work. Finally, we used Res-Net100 trained with ArcFace
loss as the basic FR model used as the backbone of our explainability efforts.
A short experiment overview is provided before introducing the final results
and more detailed analysis.

###  4.1 Database

VGGFace2 [4] dataset is a large-scale database containing face images with a
large variety in quality distribution which makes it a challenging FR
database. The images have high diversity in poses and complex acquisition
conditions. For the main analysis in this paper, we only used the official
test dataset containing 500 subjects. To reduce heavy computation and a more
balanced database, we randomly selected 30 face images of each subject
representing the full database. This made a total of 15000 images, the list of
randomly selected images from each identity will be made publicly available to
enable reproducibility.

Data preprocessing includes face detection, cropping, and alignment. The
Multi-task Cascaded Convolutional Networks (MTCNN) [30] framework is used to
detect faces from the VGGFace2. The detected faces are further cropped and
aligned using similarity transform to 112x112 pixels, such that all the face
images are standardized for comparison.

###  4.2 FIQA methods

In our experiment, we selected four different quality metrics, one is an
supervised general image quality called BRISQUE [22] as a baseline and three
specifically designed methods for face image quality assessment, FaceQnetV1
[15], MagFace [21], and SER-FIQ [27], the first is a supervised FIQA and the
later two are unsupervised. the following introduces these methods shortly.

BRISQUE [22] proposed by Anish et al. in 2012 is an opinion-aware image
quality assessment method trained with human opinion scores. The assessment of
the image is purely based on natural scene statistics learned from distortion-
generic images. The method is built on the finding by Rudermann [24] that
natural scene images have a luminance distribution similar to a normal
Gaussian distribution. Handcrafted features were derived to quantify the
deviation from the Gaussian due to image distortions. The quality estimations
by BRISQUE were previously found to have a strong correlation to face image
utility [9].

FaceQnet [15] proposed by Hernandez-Ortega et al. in 2019 is a supervised FIQA
method. The BioLab-ICAO framework was used to label the ground-truth score for
the training image according to the ICAO compliance level [17]. This score is
used to train the regression layer on top of the feature extraction layers.
FaceQnet is based on fine-tuning a pre-trained FR network (RseNet-50) and the
successive regression layer to associate an input image to a utility score
that determines the appropriateness of the input image to an FR model. In this
work, we used the latest version published in [14], i.e.
FaceQnetV1222https://github.com/uam-biometrics/FaceQnet.

SER-FIQ [27] is an unsupervised deep-learning-based FIQA approach that applies
stochastic variations on face representations learned from a deep-learning-
based FR model by using dropout. This method mitigates the need for any
automated or human labeling. The face image was passed to several sub-networks
of a modified FR network by using different dropout patterns. Images with high
utility are expected to possess similar face representations resulting in low
variance. Thus, this proposed metric linked the robustness of face embeddings
directly with FIQ.

MagFace [21] by Meng et al. is another recently proposed unsupervised FIQA
method based on using the adaptive loss incorporating the face image quality
to the magnitude of the face embedding. This method can derive both the face
representation and the face image quality from calculating the magnitude of
the face embedding. The loss optimizes the inter-class variability and intra-
class similarity. The MagFace version used in this work is trained on MS-
Celeb-1M database and used the ResNet-100 as the backbone.

All selected FIQA methods perform well on the VGGFace2 database. This can be
seen in the error versus reject characteristic (ERC) presented in Figure 1.
The ERC shows the relative performance of the FR system when rejecting
different ratios of the evaluation data with the lowest quality according to
each FIQA metric. Figure 1 shows the ERC with the false non-match rate (FNMR)
at different ratios of rejected (low quality) images using a fixed false-match
rate (FMR) at 0.1%. The error clearly decreased as the number of worst quality
samples are discarded.

Figure 1: Error versus reject curve shows the FNMR at fixed FMR=0.1% decays
with increasing number of worse quality images are discarded. All four FIQA
methods seem to perform well as a strong decrease in error is observed when
the predicted low-quality images were removed.

In the overlapping ratio matrix shown in Figure 2, we see further that the set
of the lowest and highest 10% sample images are not fully identical for
different FIQA methods, indicating that the proposed metrics in Section 3 are
derived from different base samples. The largest overlap is found for both
unsupervised FIQA methods, i.e. for MagFace and SER-FIQ.

Figure 2: We displayed the samples’ overlap ratio between the samples of the
highest quality (10% of the data) on the right between every pair of quality
estimation methods, and the same for the 10% of the lowest quality on the
left.

###  4.3 Face Recognition Solution

We use the ResNet-100 trained with ArcFace loss as the main FR solution to
visualize the activation mapping of the input face images. This ArcFace [6]
model is trained using the MS1M dataset [12]. The loss function applied
additive angular margin to improve the discriminative power of the FR model.
We chose this model because of its improved accuracy on LFW [16] 99.83% and
YTF DB [29] 99.02%.

###  4.4 Activation mapping method

ScoreCAM [28] is used as the activation mapping method to display the
activations of the deepest convolution layer of the Res-Net100 and upsampled
to overlay to the input layer. This choice is motivated by the extensive
successful use of ScoreCam as an activation mapping tool in various biometric
domains [8, 7]. ScoreCAM provides a scaled version of the activation for the
face embeddings. The weights of the scale factors are derived from this
embedding. We chose ScoreCAM as it seems to provide more realistic
explainability analyses to other methods such as CAM [31] and Grad-CAM [25].

###  4.5 Experiment overview

Given the methodology proposed in Section 3 and the experimental setup in
Section 4, we intend to show: (1) difference between high and low-quality
decision of FIQA based on AM-V/AM-MV and D-AM-MV, (2) differences within the
low and high-quality decisions across FIQA methods, and (3) individual sample
quality explainability with its AD-MAM of the individual FIQA method.

All four FIQA metrics were used to determine the face image quality of the
used database. Out of which we determine the 10% of the face images with the
highest and lowest face qualities to the group H and L, which makes 1500 face
images in each of these two groups. ScoreCAM builds the activation mapping
from the FR using ArcFace model for these input images of both groups H and L.
For the FR model trained with ArcFace loss, we used the official Pytorch
version from the official Github
333https://github.com/deepinsight/insightface.

##  5 Results and Analyses

This section is structured in three main parts related to explaining FIQ
estimation within and across the decisions of different FIQA methods. The
methodologies used as explainability tools are introduced in Section 3.

#### 1\. What makes the face images of high quality (in comparison to low) in
the view of different FIQA based on the FR model behaviour?

Figure 3 and Figure 4 depict the results for the MAM and MDAM for each FIQA
method individually, we do not notice major differences between MAM and MDAM.
Taking a look at the distributions for the mean activation mapping for H and
L, no visible significant differences are noticed. This acts as the major
motivation behind our proposed analyses based on activation map variation,
rather than the activation maps themselves. Even though the difference for the
MAM/MDAM between the H and L sets is not strongly visible, there are strong
variations in the AM-V/AM-MV. Comparing the value distributions for AM-VH and
AM-VL, we clearly observe that the variations for L are significantly larger
compared to H, the same can be observed for AM-MVH and AM-MVL. This might
indicate that the variability in activation mapping is stronger for low-
quality face images. Looking at the AM-V/AM-MV for L and H, we noticed that
the values are typically higher in L on the borders of the images, while it is
higher in the middle face region for H. This indicates that the lower utility
of L is based on the FR model focusing more often on the border areas rather
than the center, in comparison to the H set. This result is further confirmed
by looking at the D-AM-V mapping, where we can see that stronger deviations
are observed on the left and right borders of the face image.

Figure 3: The results are derived from the MAM for each FIQA metric
individually. Even though the MAMH and MAML are similar for both H and L
groups, a strong deviation can be observed in terms of AM-V, indicating low-
quality images have a stronger deviation from the mean compared to high-
quality images, especially on image borders. It is to be noted that for the
visualization purpose, these deviation maps AM-VH, AM-VL, and D-AM-V are
scaled.

Generally, all considered FIQA methods lead to similar MA-V/AM-MV and
D-AM-V/D-AM-MV observations, indicating that the effect of quality differences
(as per different FIQAs) on the FR model is of the same nature. Based on these
observations and to answer the question driving this subsection, we can notice
that despite the similarity of the general activation maps of low and high-
quality images (similar MAM/MDAM for L and H), what makes an image high
quality is the consistent low activation on the areas outside of the face
center, while low-quality images, despite general low activation in these
areas, have high variations of activation there. In simple words, low-quality
images do attract the attention of FR model in areas outside of the center
face area, however, in different locations and less consistently, which can be
caused by the different reasons for the degradation of quality.

Figure 4: The results are derived from the MDAM for each FIQA metric
individually. The same result is obtained as for MAM in Figure3, where even
stronger variations are observed from the median for low-quality images. Low-
quality images tend to attract the attention of FR model in areas outside of
the center face area, which can be caused by different reasons like e.g.,
postures or occlusions. The variation maps are re-scaled for better
visualization.

#### 2\. What makes the decision of high or low quality different between
different FIQAs based on the FR model behaviour?

In Figure 5 we see visualization of our D-AM-V mapping across FIQA methods.
The D-AM-V here differs than the one presented in Equation (5) by looking
across FIQA methods rather than quality sets, here the
$\text{D-AM-V}_{H}=\text{AM-V}_{H,FIQA-1}-\text{AM-V}_{H,FIQA-2}$ and
$\text{D-AM-V}_{L}=\text{AM-V}_{L,FIQA-1}-\text{AM-V}_{L,FIQA-2}$. D-AM-VH and
D-AM-VL show the differential activation mapping variations for H and L sets
individually between pairs of FIQA metrics. D-AM-VH and D-AM-VL between the
unsupervised MagFace and the supervised methods FaceQnet and BRISQUE show
similar tendencies. Samples selected to be H by the supervised methods tend to
cause higher activation variations in the top and bottom of the image when
compared to MagFace. Samples selected to be L by the supervised methods tend
to cause higher activation variations on the left and right edges of the face
image when compared to MagFace. This consistent activation variation on areas
out of the face center can rationalize the performance differences between
FaceQnet and BRISQUE on one side, and MagFace on the other side, see Figure 1.
There are fewer differences in the activation variations caused by both the H
and L set between the supervised (and with poorer performance) FIQA methods
(BRISQUE and FaceQnet) both in terms of magnitude (distribution shifts) and
clear spatial distribution. The same can be seen between the two high-
performing unsupervised methods (SER-FIQ and MagFace). This leads to answering
the question behind this sub-section by stating that compared to high
performing FIQA methods, lower performing FIQA methods have larger FR
activation variation on the edges of the face image.

Figure 5: Results are derived when comparing D-AM-V for H and L sets across
different FIQA methods. D-AM-VH and D-AM-VL between the unsupervised MagFace
and the supervised methods FaceQnet and BRISQUE show similar tendencies. D-AM-
VH and D-AM-VL tend to show stronger and more consistent differences between
supervised and unsupervised methods, hinting a link between the lower
performance of FaceQnet and BRISQUE and the focus of the FR model on areas
outside of the face center in an unexpected manner .

#### How do the activation mappings caused by face images in the FR model
deviate from what is expected from high quality images and how does that
reflect in their quality score?

In Figure 6 we depicted sample images of high and low face qualities. Each
sample subject, we provided one original image with the FIQ score from all
four FIQA metrics below, one image overlayed with the activation mapping from
the ArcFace FR solution, and four overlayed with the AD-MAMH map of each FIQA
method. These differential activation mappings are for BRISQUE, FaceQnet,
MagFace, and SER-FIQA in the correct ordering starting from upper left to
bottom right displayed for each sample subject. These AD-MAMH maps show areas
which could cause the degradation in qualities as they deviate from the mean
template and is emphasized in the visualization.

Figure 6: Selected face images with high and low face qualities are displayed.
Each sample subject contains one original image with quality scores
illustrated below, one image overlapped with the activation mapping derived
from ArcFace FR, and four overlayed AD-MAMH maps. These differential maps
indicate the deviations from the MAMH and show areas which could cause the
degradation in qualities, such as sunglasses, hat, and mustaches.

From Figure 6, it can be generally concluded that the AD-MAMH for all FIQA
methods contain larger and higher values for low quality images in comparison
to high quality images. Also in this comparison, the high value areas in the
AD-MAMH for low quality images tend to appear more in the areas around the
face rather than the center of the face. This is less apparent in the high
quality images. This larger and higher values in AD-MAMH corresponds to the
low quality estimated across the four FIQA methods and in many cases it is
related to less than optimal poses, face occlusions, and overall low image
sharpness. To answer the question motivating this sub-section, our analyses
and proposed explainability tools reveal that low quality images tend to have
a FR model activation map that strongly differs than that of what is expected
from a high quality image. This difference also tends to appear more in the
areas outside of the central face region.

##  6 Conclusion

Making the face image quality estimation explainable is a challenging task
that goes beyond analyzing the FIQA network itself. Most recent works put more
focus on designing FIQA methods that perform well without looking into what is
the response of an FR model to high or low-quality face image. In this work,
we presented a novel set of explainability tools to enhance the visual
explainability of FIQ estimation decisions based on the variation analyses in
FR models. The proposed tools can be applied for any FIQA method with any CNN-
based FR solution using activation mapping to exhibit the network’s activation
derived from the face embedding. By showing the intra-groups and cross-method
inter-groups statistics of the network’s activation, we try to relate
explainability to groups of H and L face quality image sets for the individual
FIQA method. We demonstrate that even though the MAM between H and L is small,
the variations in activation mapping for L are larger compared to H. This
points out that the low-quality images tend to cause the FR network to focus
on areas outside of the central face area, however, in an inconsistent manner,
as the reason causing the low quality can vary largely. We additionally link
this observation to the relative performance of different supervised and
unsupervised FIQA approaches. Finally, we look at the explainability of the
quality decision of individual face images by analyzing the differences
between their activation maps in FR models and the maps expected from high-
quality images, pointing out consistent differences in FR behavior between
high quality and low-quality face images.

Acknowledgements: This research work has been funded by the German Federal
Ministry of Education and Research and the Hessian Ministry of Higher
Education, Research, Science and the Arts within their joint support of the
National Research Center for Applied Cybersecurity ATHENE.

## References

  * [1] Lacey Best-Rowden and Anil K. Jain.  Learning face image quality from human assessments.  IEEE Trans. Inf. Forensics Secur., 13(12):3064–3077, 2018.
  * [2] Fadi Boutros, Naser Damer, Meiling Fang, Florian Kirchbuchner, and Arjan Kuijper.  Mixfacenets: Extremely efficient face recognition networks.  In International IEEE Joint Conference on Biometrics, IJCB 2021, Shenzhen, China, August 4-7, 2021, pages 1–8. IEEE, 2021.
  * [3] Fadi Boutros, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper.  Elasticface: Elastic margin loss for deep face recognition.  CoRR, abs/2109.09416, 2021.
  * [4] Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman.  Vggface2: A dataset for recognising faces across pose and age.  In 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018, Xi’an, China, May 15-19, 2018, pages 67–74. IEEE Computer Society, 2018.
  * [5] Naser Damer, Timotheos Samartzidis, and Alexander Nouak.  Personalized face reference from video: Key-face selection and feature-level fusion.  In Face and Facial Expression Recognition from Real World Videos \- International Workshop, FFER@ICPR 2014, Stockholm, Sweden, August 24, 2014, Revised Selected Papers, volume 8912 of Lecture Notes in Computer Science, pages 85–98. Springer, 2014.
  * [6] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF CVPR, pages 4690–4699, 2019.
  * [7] Meiling Fang, Naser Damer, Fadi Boutros, Florian Kirchbuchner, and Arjan Kuijper.  Iris presentation attack detection by attention-based and deep pixel-wise binary supervision network.  In International IEEE Joint Conference on Biometrics, IJCB 2021, Shenzhen, China, August 4-7, 2021, pages 1–8. IEEE, 2021.
  * [8] Meiling Fang, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper.  Real masks and spoof faces: On the masked face presentation attack detection.  Pattern Recognition, 123:108398, 2022.
  * [9] Biying Fu, Cong Chen, Olaf Henniger, and Naser Damer.  A deep insight into measuring face image utility with general and face-specific image quality metrics.  CoRR, abs/2110.11111, 2021.
  * [10] Biying Fu, Cong Chen, Olaf Henniger, and Naser Damer.  The relative contributions of facial parts qualities to the face image utility.  In 2021 International Conference of the Biometrics Special Interest Group (BIOSIG), pages 1–5. IEEE, 2021.
  * [11] Biying Fu, Florian Kirchbuchner, and Naser Damer.  The effect of wearing a face mask on face image quality.  CoRR, abs/2110.11283, 2021.
  * [12] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  MS-Celeb-1M: A dataset and benchmark for large-scale face recognition.  In European Conf. on Computer Vision, pages 87–102. Springer, 2016\.
  * [13] Dongyoon Han, Jiwhan Kim, and Junmo Kim.  Deep pyramidal residual networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6307–6315. IEEE Computer Society, 2017.
  * [14] Javier Hernandez-Ortega, Javier Galbally, Julian Fiérrez, and Laurent Beslay.  Biometric quality: Review and application to face recognition with faceqnet.  CoRR, abs/2006.03298, 2020.
  * [15] Javier Hernandez-Ortega, Javier Galbally, Julian Fierrez, Rudolf Haraksim, and Laurent Beslay.  FaceQnet: Quality assessment for face recognition based on deep learning.  In Int. Conf. on Biometrics (ICB), pages 1–8. IEEE, 2019.
  * [16] Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments.  In Workshop on Faces in ’Real-Life’ Images: Detection, Alignment, and Recognition, Marseille, France, Oct. 2008. Erik Learned-Miller and Andras Ferencz and Frédéric Jurie.
  * [17] ISO/IEC JTC1 SC17 WG3.  Portrait Quality - Reference Facial Images for MRTD.  International Civil Aviation Organization, 2018.
  * [18] ISO/IEC JTC1 SC37 Biometrics.  ISO/IEC 29794-1:2016 Information technology - Biometric sample quality - Part 1: Framework.  International Organization for Standardization, 2016.
  * [19] Khawla Mallat, Naser Damer, Fadi Boutros, and Jean-Luc Dugelay.  Robust face authentication based on dynamic quality-weighted comparison of visible and thermal-to-visible images to visible enrollments.  In 22th International Conference on Information Fusion, FUSION 2019, Ottawa, ON, Canada, July 2-5, 2019, pages 1–8. IEEE, 2019.
  * [20] Brianna Maze, Jocelyn C. Adams, James A. Duncan, Nathan D. Kalka, Tim Miller, Charles Otto, Anil K. Jain, W. Tyler Niggel, Janet Anderson, Jordan Cheney, and Patrick Grother.  IARPA janus benchmark - C: face dataset and protocol.  In 2018 International Conference on Biometrics, ICB 2018, Gold Coast, Australia, February 20-23, 2018, pages 158–165. IEEE, 2018.
  * [21] Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou.  Magface: A universal representation for face recognition and quality assessment.  In IEEE Conf. on Computer Vision and Pattern Recognition, 2021.
  * [22] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik.  No-reference image quality assessment in the spatial domain.  IEEE Trans. Image Process., 21(12):4695–4708, 2012.
  * [23] Fu-Zhao Ou, Xingyu Chen, Ruixin Zhang, Yuge Huang, Shaoxin Li, Jilin Li, Yong Li, Liujuan Cao, and Yuan-Gen Wang.  SDD-FIQA: unsupervised face image quality assessment with similarity distribution distance.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7670–7679. Computer Vision Foundation / IEEE, 2021.
  * [24] Daniel L Ruderman.  The statistics of natural images.  Network: Computation in Neural Systems, 5(4):517–548, 1994.
  * [25] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  Int. J. Comput. Vis., 128(2):336–359, 2020.
  * [26] Yichun Shi and Anil K. Jain.  Probabilistic face embeddings.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 6901–6910. IEEE, 2019.
  * [27] Philipp Terhörst, Jan Niklas Kolf, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper.  SER-FIQ: Unsupervised estimation of face image quality based on stochastic embedding robustness.  In Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, pages 5651–5660, 2020.
  * [28] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.  Score-CAM: Score-weighted visual explanations for convolutional neural networks.  In Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops, pages 24–25, 2020.
  * [29] Lior Wolf, Tal Hassner, and Itay Maoz.  Face recognition in unconstrained videos with matched background similarity.  In The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pages 529–534. IEEE Computer Society, 2011.
  * [30] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE Signal Process. Lett., 23(10):1499–1503, 2016.
  * [31] Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2921–2929. IEEE Computer Society, 2016.


# De-Biased Modelling of Search Click Behavior with Reinforcement Learning

Jianghong Zhou  Emory University<EMAIL_ADDRESS>,  Sayyed M. Zahiri
The Home Depot<EMAIL_ADDRESS>,  Simon Hughes  The Home Depot
<EMAIL_ADDRESS>,  Khalifeh Al Jadda  The Home Depot
<EMAIL_ADDRESS>,  Surya Kallumadi  The Home Depot
<EMAIL_ADDRESS>and  Eugene Agichtein  Emory University
<EMAIL_ADDRESS>
(2021)

###### Abstract.

Users’ clicks on Web search results are one of the key signals for evaluating
and improving web search quality, and have been widely used as part of current
state-of-the-art Learning-To-Rank(LTR) models. With a large volume of search
logs available for major search engines, effective models of searcher click
behavior have emerged to evaluate and train LTR models. However, when modeling
the users’ click behavior, considering the bias of the behavior is imperative.
In particular, when a search result is not clicked, it is not necessarily
chosen as not relevant by the user, but instead could have been simply missed,
especially for lower-ranked results. These kinds of biases in the click log
data can be incorporated into the click models, propagating the errors to the
resulting LTR ranking models or evaluation metrics. In this paper, we propose
the De-biased Reinforcement Learning Click model (DRLC). The DRLC model
relaxes previously made assumptions about the users’ examination behavior and
resulting latent states. To implement the DRLC model, convolutional neural
networks are used as the value networks for reinforcement learning, trained to
learn a policy to reduce bias in the click logs. To demonstrate the
effectiveness of the DRLC model, we first compare performance with the
previous state-of-art approaches using established click prediction metrics,
including log-likelihood and perplexity. We further show that DRLC also leads
to improvements in ranking performance. Our experiments demonstrate the
effectiveness of the DRLC model in learning to reduce bias in click logs,
leading to improved modeling performance and showing the potential for using
DRLC for improving Web search quality.

web search click models, reinforcement learning for click modeling, de-biased
estimation of search click behavior

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
44th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 11–15, 2021; Virtual Event, Canada††booktitle:
Proceedings of the 44th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR ’21), July 11–15, 2021, Virtual
Event, Canada††price: 15.00††doi: 10.1145/3404835.3463228††isbn:
978-1-4503-8037-9/21/07

##  1\. Introduction

To improve web search, it is important to understand how the users interact
with web search engines. Clicking on search results is one of the most
important interaction data, which forms the basis of many important user
behaviors, like reformulating or switching queries, clicking on different
items, and browsing the search results. One powerful way to utilize these
click logs is to construct a click model to measure and predict clicks on
existing or future results(Borisov et al., 2016). A click model can predict
future clicks of other users, help train a learning to rank (LTR) model, and
allow to automatically evaluate search result quality. However, modeling
users’ clicks is a challenging task, because the click logs are observational
data, collected in-situ with a live search engine, and exhibits multiple
biases(Joachims et al., 2017). Previous research on click modeling and
prediction did not directly address this issue or addressed the biases using
heuristics, resulting in poor model performance on live (unseen) query
traffic(Joachims et al., 2017).

As a summary, previous research focused on two directions. The first direction
is the Probabilistic Graphical Model Framework Based Methods (PGM) (Koller and
Friedman, 2009). Those methods model the search process as a sequence of
events, predicting the click based on some probability models and assumptions.
They are flexible and interpreted. The second direction considers a different
representation of the events (Borisov et al., 2016). The searching process is
represented by some vectors. This form allows the users to consider a variety
of features easily and to feed them to some rather stronger learning models,
such as neural networks (Chakraborty et al., 2000). However, the drawbacks of
these two directions are obvious. The PGM methods are limited by a rather weak
learning model and fewer features. The second direction cannot consider the
bias issue in an interpreted way.

To overcome the aforementioned issues, we propose our new model, DRLC for
training unbiased (or less biased) click models. DRLC is also a PGM-based
method. Therefore, DRLC can be organized in a flexible way for different
ranking scenarios and generate an interpretive model to reduce a variety of
biases. However, in comparison to the previous PGM method, DRLC is constructed
by a more dynamic system, which is the reinforcement learning (Sutton and
Barto, 2018; Zhou and Agichtein, 2020; Zhou et al., 2020). This allows DRLC to
takes advantage of stronger learning models (neural networks).

Another contribution of this paper is we propose a posteriori method to learn
the observation bias. Previous approaches normally apply some priori
probability models to estimate the observation probability or reduce the bias
(Dupret and Piwowarski, 2008; Agarwal et al., 2019). Those methods highly rely
on the selected priori models, which is hard to be general. In this paper, we
concentrate on the posteriori knowledge. When users are browsing the SERP
(search engine result page), the latter part is naturally unobserved.
Therefore, if we use an observation window to augment the dataset, we can have
some data with more bias and some data with less bias. This separation is
helpful to capture the bias. DRLC uses this method to reduce the bias.

##  2\. Related Work

We discuss two important methods related to our work: PGM framework based
methods and neural click model.

Probabilistic Graphical Models: Most of the traditional click models are based
on the PGM framework (Koller and Friedman, 2009). In PGM, the users’
interactions are organized as a sequence of events, such as document
examinations, skips, and clicks. The most frequent events that those models
consider are document examinations and clicks. Since it is instinctively
correct that a user’s clicks should be led by a document examination, these
two events are highly interdependent. Based on this assumption, some important
click models, including CCM, DCM, DBN, UBM are proposed (Dupret and
Piwowarski, 2008; Guo et al., 2009b; Chapelle et al., 2009).

Neural Click Models: The neural click model (NCM) is a method based on
artificial neural networks. The input data of the artificial neural network is
generally one or many vectors, even high dimensional tensor. Therefore,
different from the PGM based method, NCM represents the clicking process with
vectors (Borisov et al., 2016). (Borisov et al., 2016) attempts to use many
different neural networks to construct the click models, including Long Short-
Term Memory (LSTM) (Huang et al., 2015) and recurrent neural networks (RNN)
(Yang et al., 2016). Their empirical evaluations showed that the LSTM version
has better performance.

Additionally, NCM also expands to some complicated click models. For example,
(Chen et al., 2020) develops a session search click model (CACM), which
leverages the signals from previous search iterations to predict the next
search iteration of the search session.

##  3\. The Proposed Method

In this part, we introduce the proposed DRLC model. We first introduce the
implementation of the DRLC’s components, namely the value networks, which
include a bias network, and a de-biased network. Then, we overview the
proposed reinforcement learning (RL) approach to training a de-biased click
model.

###  3.1. Value Network Implementation

We chose to use Convolutional Neural Networks (CNNs) as our value networks.
The first CNN $C_{1}$is a bias network, which means this network considers the
observation bias: Some documents are not clicked because they are not
observed. The input features are bias features $B$ and the document features
$D$. $B$ is a vector with $1$ or $0$. $B$ represents the observation situation
of the search results. If the item is observed, the value is $1$ or it is $0$.
We use an observation window to observe the SERP. We first assume the
documents browsed by the window are observed. Then, by comparing the
predictions from two value networks, we further update $B$. $D$ is a vector,
representing the features of a document. Those features are related to both
the query and the document, such as the frequency of the query appearing in
the document. The output of $C_{1}$ is the click estimation with bias or the
possibility of clicking this document. The second CNN $C_{2}$ is a de-biased
network, whose output is the de-biased click estimation or the possibility of
clicking the item under the de-biased setting. The input of $C_{2}$ is just
$D$. ‘ The structure of the value networks is summarized in Fig. 1(a).

$\begin{array}[]{cc}\includegraphics[width=240.0pt]{F1.PNG}&\includegraphics[width=240.0pt]{click_model.PNG}\\\
(a)&(b)\end{array}$
Figure 1. (a) The structure of the DRLC value networks.; (b) Illustration of
the DRLC Reinforcement Learning process.

Pre-training: Since two networks focus on two aspects of the dataset, we pre-
train them in different ways. We pre-train $C_{1}$ with the whole training
dataset, which is a highly biased dataset. For the observation features $B$,
we assume that the users observe the documents sequentially with an
observation window. If the documents appeared in the window before, they are
denoted as observed. If not, they are not observed. Only when all the
documents in the observation window are examined, the window moves to the next
position, the process is presented in Fig. 1(b). The size of the observation
window is decided by the empirical estimation (In this paper, the size of the
window is 3.). $C_{2}$ is trained by a de-biased dataset. We assume the
documents before the last clicked document is more likely to be observed.
Therefore, we only pre-train $C_{2}$ with the documents before the last
clicked document.

CNN architecture The input of the $C_{1}$ is an $100\times 1$ vector (Bias
features $B$) and a $56\times 1$ vector (Document features $D$). The bias
features are demonstrated in Pre-train part. The document features are
generated from the URL provided in the dataset. The features we selected are
identical to the ones listed in (Qin et al., 2010). Firstly, we apply two
filter layers to join the features from $D$ and $B$. For each CNNs, they have
three convolutional blocks. Each convolutional block contains 16 filters of
kernel $3\times 1$ with stride 1, a batch normalization layer, and a ReLU
layer. The output layer is a fully connected network. The loss function is a
softmax function. The structures of CNNs cannot be too deep, considering the
scalability of the click model.

After pre-training, the networks are initialized to be trained by the RL
method.

###  3.2. Reinforcement Learning (RL) Framework

The RL framework of DRLC is illustrated in Fig. 1(b). The components of the RL
are defined as follows:

State $s$ is the click and observation state of the documents. We assume a
user’s attention can be modeled by an observation window, where users observe
items in the observation window with a higher possibility.

Action $a$ is selecting the state of a document at the position. Those states
include: (Observed, Click), (Observed, Not click), and (Unobserved, Not
click).

Transition $T$ changes the document state in $S$ based on $a$.

Reward $R$ is how well the estimated click probabilities match the empirical
distribution observed in the click logs.

(1) $R_{t}=(C_{t}-C_{t,1})^{2}+\beta O_{t}(C_{t}-C_{t,2})^{2}$

Where $t$ is the position of the document. $C_{t}$ is the click of the click
logs. $C_{t,1}$ is the click prediction from $C_{1}$. $C_{t,2}$ is the click
prediction from $C_{2}$. $\beta$ weighs the importance of the de-biased
prediction The value is set to $0.7$ based on the validation part of the
experiment. $O_{t}$ shows that whether this document is observed. We have two
assumptions of the observation. First, if the document is clicked in the click
logs, it is observed. Second, if $C_{t,1}/C_{t,2}<\theta$, the document is
unobserved. The first assumption is understandable. The second assumption is
based on the bias effect, which is $P(O_{t})C_{t,2}=C_{t,1}$. If $P(O_{t})$ is
small, it means the possibility of the observation is low. In this paper, we
empirically set $\theta$ as 0.3.

The goal of the RL is to learn a policy $\pi^{*}$ to maximize
$\mathbb{R}=\sum\gamma^{t}R_{t}$. In turn that means learning the value of
each state, corresponding to click probability.

Update: The $C_{1}$ are further trained by the results of the final state
$S_{T}$, where $T$ is the total number of the documents in the click log. The
$C_{2}$ is updated by the observed documents.

##  4\. Experiments and Discussion

In this section, we introduce the datasets, metrics, and baselines.

Datasets: We demonstrate the effectiveness of the proposed model in two open-
sourced dataset and one private dataset: ORCAS dataset (Craswell et al.,
2020), Yandex click dataset (Serdyukov et al., 2013) and the real interactive
dataset from a large e-commerce website (https://www.homedepot.com/). ORCAS is
a click-based dataset associated with the TREC Deep Learning Track. It covers
1.4 million of the TREC DL documents, providing 18 million connections to 10
million distinct queries. The Yandex click dataset comes from the Yandex
search engine, containing more than 30 million search sessions. Each session
contains at least one search query together with 10 ranked items. The private
Interactive Dataset (RID) is a 3-month search log. In this dataset, the users
normally search for several queries. For each query, the search engine returns
a list of products and then the user can interact with the results by
clicking, adding the items to the cart and ordering. Table 1 shows a sample of
the data. Additionally, with the products’ ID, we can further find the page of
the products and extract the text features. The features list can be found in
(Qin and Liu, 2013).

Table 1. The sample of the Home Depot website’s Interactive data. visitor ID | session id | date | time | searchterm | click sku | atc sku | order sku | product impression
---|---|---|---|---|---|---|---|---
1000 | 1000-mobile-1 | 6/1/2020 | 6:30 pm | everbilt dropcloth | 2034 |  |  | 3072—2034—2037—2036
1000 | 1000-mobile-1 | 6/1/2020 | 6:34 pm | pull down shades | 3022 | 3022 | 3022 | 3022—2051—3042—2071
1001 | 1001-mobile-1 | 6/1/2020 | 6:36pm | fence panel |  |  |  | 2030—1003—2029—1000
1001 | 1001-mobile-2 | 6/1/2020 | 6:36pm | fince dog ears | 2053 |  |  | 2055—2034—3034—2053

Metrics We evaluate the model from two aspects. The first aspect is based on
the click prediction. The second aspect is based on relevance. In terms of the
click prediction, we use Log-likelihood and perplexity as the evaluation
methodology. In terms of the relevance analysis, if the item is clicked, its
relevance score is 1, or it is 0. Then, we consider the scores calculated to
predict the click as the relevance score. We rank these scores and calculate
the NDCG as our relevance prediction metric (Borisov et al., 2016).

Baselines We use DBN, DCM, CCM, UBM and NCM as our baselines (Chapelle and
Zhang, 2009; Chuklin et al., 2013; Dupret and Piwowarski, 2008; Guo et al.,
2009a; Borisov et al., 2016). Those methods are the state-of-art click models
based on PGM and neural networks. The parameter settings are based on the
Pyclick package (https://github.com/markovi/PyClick). (We did not use CACM as
our baseline, because it is a session search click model, not a single search
click model (Chen et al., 2020). )

Table 2. The experiment results of DBN, DCM, CCM, UBM, NCM, URCM on ORCAS dataset, Yandex click dataset and the https://www.homedepot.com/ e-commerce website interactive dataset (RID). The best performance results are highlighted in bold font. All improvements are significant with $p<0.05$. Dataset | Model | Perplexity | Log-likelihood | NDCG@1 | NDCG@3 | NDCG@5 | NDCG@10
---|---|---|---|---|---|---|---
ORCAS Dataset | DBN | 1.4628 | -0.2273 | 0.596 | 0.606 | 0.623 | 0.655
| DCM | 1.4647 | -0.2894 | 0.609 | 0.618 | 0.639 | 0.662
| CCM | 1.4664 | -0.2778 | 0.615 | 0.626 | 0.637 | 0.671
| UBM | 1.4593 | -0.2203 | 0.599 | 0.608 | 0.628 | 0.656
| NCM | 1.4545 | -0.2186 | 0.617 | 0.625 | 0.639 | 0.677
| DRLC | 1.4326 | -0.2037 | 0.610 | 0.624 | 0.645 | 0.686
Yandex Click Dataset | DBN | 1.3562 | -0.2789 | 0.702 | 0.724 | 0.766 | 0.841
| DCM | 1.3605 | -0.3594 | 0.729 | 0.744 | 0.775 | 0.845
| CCM | 1.3688 | -0.3522 | 0.746 | 0.757 | 0.779 | 0.848
| UBM | 1.3422 | -0.2667 | 0.729 | 0.739 | 0.769 | 0.841
| NCM | 1.3406 | -0.2522 | 0.756 | 0.763 | 0.788 | 0.846
| DRLC | 1.3283 | -0.2393 | 0.729 | 0.754 | 0.776 | 0.848
RID dataset | DBN | 1.3777 | -0.2267 | 0.543 | 0.578 | 0.598 | 0.605
| DCM | 1.3764 | -0.2873 | 0.566 | 0.587 | 0.603 | 0.611
| CCM | 1.3872 | -0.2983 | 0.511 | 0.601 | 0.608 | 0.621
| UBM | 1.3899 | -0.2637 | 0.538 | 0.612 | 0.618 | 0.632
| NCM | 1.3937 | -0.2433 | 0.556 | 0.617 | 0.623 | 0.638
| DRLC | 1.3554 | -0.2232 | 0.616 | 0.624 | 0.645 | 0.648

Results and Discussion The results of the experiment are summarized in Table
2. The empirical results show that DRLC outperforms all baselines in terms of
clicking prediction by 3.4% to 5.2%. Based on the T-test, this improvement is
substantial. For the ranking prediction, DRLC outperforms the other baselines
mostly.

Based on the empirical results, DRLC can predict the click, both in accuracy
and relevance, better than the previous methods. We attribute the improvement
of the click prediction to our assumptions of observation. In past, the
unobserved data is hard to train, because it is almost impossible to manually
label the data as observed documents or unobserved ones. However, in our RL
framework, the users browse the results sequentially. It is reasonable to
assume that the documents are unobserved when the users are still observing
the top documents. In this way, we can train the networks to classify whether
the documents are observed. Besides, when the de-biased prediction and the
biased prediction have a huge difference, it also indicates that the document
is highly possible to be unobserved.

Additionally, in the RL framework, the initial State is updated during the
training process to further improve the value networks.

##  5\. Conclusions

In this paper, we propose DRLC, which is a de-biased click model based on
reinforcement learning. The model aggregates the advantages of the previous
PGM methods and neural network methods, modeled by some novel assumptions of
users’ clicking and the observation bias. The empirical evaluation shows that
DRLC is the state-of-art method in terms of click prediction, indicating that
the RL, CNNs, and the proposed assumptions in this paper are helpful to
improve the performance of the click models.

In the future, we can further consider other methods to de-bias the dataset.
For example, we can apply counterfactual learning to train the de-bias
network, which is theoretically unbiased (Agarwal et al., 2019). DRLC can also
be a trainer of some novel LTR models, providing a de-biased dataset.

## References

  * (1)
  * Agarwal et al. (2019) Aman Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Li, Marc Najork, and Thorsten Joachims. 2019\.  Estimating position bias without intrusive interventions. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. 474–482.
  * Borisov et al. (2016) Alexey Borisov, Ilya Markov, Maarten De Rijke, and Pavel Serdyukov. 2016.  A neural click model for web search. In _Proceedings of the 25th International Conference on World Wide Web_. 531–541.
  * Chakraborty et al. (2000) B Chakraborty, R Kaustubha, A Hegde, A Pereira, W Done, R Kirlin, A Moghaddamjoo, A Georgakis, C Kotropoulos, and Pitas Xafopoulos. 2000\.  Bishop, CM, Neural Networks for Pattern Recognition, Oxford University Press, New York, 1995. Carreira-Perpiñán M., Mode-Finding for Mixtures of Gaussian Distributions, IEEE transaction on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, November 2000, 1318-1323.  _IEEE transaction on Pattern Analysis and Machine Intelligence_ 22, 11 (2000), 1318–1323.
  * Chapelle et al. (2009) Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009.  Expected reciprocal rank for graded relevance. In _Proceedings of the 18th ACM conference on Information and knowledge management_. 621–630.
  * Chapelle and Zhang (2009) Olivier Chapelle and Ya Zhang. 2009.  A dynamic bayesian network click model for web search ranking. In _Proceedings of the 18th international conference on World wide web_. 1–10.
  * Chen et al. (2020) Jia Chen, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020.  A context-aware click model for web search. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 88–96.
  * Chuklin et al. (2013) Aleksandr Chuklin, Anne Schuth, Katja Hofmann, Pavel Serdyukov, and Maarten De Rijke. 2013.  Evaluating aggregated search using interleaving. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 669–678.
  * Craswell et al. (2020) Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. 2020.  ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search.  _arXiv preprint arXiv:2006.05324_ (2020).
  * Dupret and Piwowarski (2008) Georges E Dupret and Benjamin Piwowarski. 2008.  A user browsing model to predict search engine click data from past observations.. In _Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval_. 331–338.
  * Guo et al. (2009b) Fan Guo, Chao Liu, Anitha Kannan, Tom Minka, Michael Taylor, Yi-Min Wang, and Christos Faloutsos. 2009b.  Click chain model in web search. In _Proceedings of the 18th international conference on World wide web_. 11–20.
  * Guo et al. (2009a) Fan Guo, Chao Liu, and Yi Min Wang. 2009a.  Efficient multiple-click models in web search. In _Proceedings of the second acm international conference on web search and data mining_. 124–131.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF models for sequence tagging.  _arXiv preprint arXiv:1508.01991_ (2015).
  * Joachims et al. (2017) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017\.  Unbiased learning-to-rank with biased feedback. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_. 781–789.
  * Koller and Friedman (2009) Daphne Koller and Nir Friedman. 2009.  _Probabilistic graphical models: principles and techniques_.  MIT press.
  * Qin and Liu (2013) Tao Qin and Tie-Yan Liu. 2013.  Introducing LETOR 4.0 datasets.  _arXiv preprint arXiv:1306.2597_ (2013).
  * Qin et al. (2010) Tao Qin, Tie-Yan Liu, Jun Xu, and Hang Li. 2010\.  LETOR: A benchmark collection for research on learning to rank for information retrieval.  _Information Retrieval_ 13, 4 (2010), 346–374.
  * Serdyukov et al. (2013) Pavel Serdyukov, Georges Dupret, and Nick Craswell. 2013\.  WSCD2013: workshop on web search click data 2013. In _Proceedings of the sixth ACM international conference on Web search and data mining_. 787–788.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Yang et al. (2016) Tzu-Hsuan Yang, Tzu-Hsuan Tseng, and Chia-Ping Chen. 2016\.  Recurrent neural network-based language models with variation in net topology, language, and granularity. In _2016 International Conference on Asian Language Processing (IALP)_. IEEE, 71–74.
  * Zhou and Agichtein (2020) Jianghong Zhou and Eugene Agichtein. 2020.  RLIRank: Learning to Rank with Reinforcement Learning for Dynamic Search. In _Proceedings of The Web Conference 2020_. 2842–2848.
  * Zhou et al. (2020) Jianghong Zhou, Eugene Agichtein, and Surya Kallumadi. 2020\.  Diversifying Multi-aspect Search Results Using Simpson’s Diversity Index. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2345–2348.