2021

[1]Anuj Rai

1]Computer Science and Engineering, Indian Institute of Technology Indore,
Simrol, Indore, 453552, Madhya Pradesh, India

2]Computer Science and Engineering, The LNM Institute of Information
Technology, Sumel, Jaipur, 302031, Rajasthan, India

# An Open Patch Generator based Fingerprint Presentation Attack Detection
using Generative Adversarial Network

<EMAIL_ADDRESS>Ashutosh Anshul<EMAIL_ADDRESS>Ashwini
Jha<EMAIL_ADDRESS>Prayag Jain<EMAIL_ADDRESS>Ramprakash
Sharma<EMAIL_ADDRESS>Somnath Dey<EMAIL_ADDRESS>[  [

###### Abstract

The low-cost, user-friendly, and convenient nature of Automatic Fingerprint
Recognition Systems (AFRS) makes them suitable for a wide range of
applications. This spreading use of AFRS also makes them vulnerable to various
security threats. Presentation Attack (PA) or spoofing is one of the threats
which is caused by presenting a spoof of a genuine fingerprint to the sensor
of AFRS. Fingerprint Presentation Attack Detection (FPAD) is a countermeasure
intended to protect AFRS against fake or spoof fingerprints created using
various fabrication materials. In this paper, we have proposed a Convolutional
Neural Network (CNN) based technique that uses a Generative Adversarial
Network (GAN) to augment the dataset with spoof samples generated from the
proposed Open Patch Generator (OPG). This OPG is capable of generating
realistic fingerprint samples which have no resemblance to the existing spoof
fingerprint samples generated with other materials. The augmented dataset is
fed to the DenseNet classifier which helps in increasing the performance of
the Presentation Attack Detection (PAD) module for the various real-world
attacks possible with unknown spoof materials. Experimental evaluations of the
proposed approach are carried out on the Liveness Detection (LivDet) 2015,
2017, and 2019 competition databases. An overall accuracy of 96.20%, 94.97%,
and 92.90% has been achieved on the LivDet 2015, 2017, and 2019 databases,
respectively under the LivDet protocol scenarios. The performance of the
proposed PAD model is also validated in the cross-material and cross-sensor
attack paradigm which further exhibits its capability to be used under real-
world attack scenarios.

###### keywords:

Biometrics, Fingerprint, Presentation Attack Detection, Generative Adversarial
Networks.

##  1 Introduction

Fingerprint biometric systems are beneficial compared with conventional
approaches as they are trustworthy, autonomous, and user-friendly. These
properties make them useful in many real-life applications such as border
security, national identification, and various payment systems. Nowadays, we
can also spot these systems in various banking applications for customer
identification and authentication such as Automated Teller Machines (ATMs),
customer account opening through biometric-based Aadhar identification, etc.
Its applicability in such a wide range of applications makes it vulnerable to
various security threats, such as theft of identity, account hacking,
unauthorized access, and many more. A Presentation Attack (PA) is one of the
possible threats, which is carried out by presenting an artificial artifact of
a genuine user’s fingerprint to the sensor to get access to the system.
Nowadays, various spoofing materials including rubber fingers, 3D printed
papers, and high-definition images and videos of fingerprints that are capable
to deceive a system are available at a very low cost. Gelatine, Wood Glue,
Latex, and Siligum are some of the inexpensive materials available to create
an artificial replica of a genuine user’s fingerprint. This attack
necessitates the use of altered fingerprints, as well as fingerprint patterns
that have been edited or deleted, to influence the identification process.
This problem of PAs can be addressed with two kinds of methods i.e. hardware-
based and software-based methods. The hardware-based methods utilize extra
hardware devices for the extraction of features such as temperature, humidity,
and pulse rate from the fingertip. The utilization of additional hardware
costs in terms of money, as well as maintenance, makes these approaches less
popular for Presentation Attack Detection (PAD). On the other side, software-
based methods rely only on the input fingerprint samples for PAD which makes
these approaches user-friendly as well as cost-effective. Due to the
widespread applicability and cost-effectiveness of the software-based
approaches, our work is focused on developing a software-based approach to
deal with the PAs in the fingerprint recognition system.

The state-of-the-art software-based approaches include statistical feature-
based methods, perspiration and pore based-methods, handcrafted feature-based
methods, and deep learning-based methods. The perspiration property of the
fingerprint is affected by the pressure that is applied to the fingertip as
well as the temperature of the surrounding environment. Extraction of these
feature values requires multiple impressions of the same fingerprint. Pore-
based methods require a sensing device with the capability of capturing high-
resolution (around 1000 pixels per inch) images of fingerprints which
proportionally increases the cost of the biometric system. The existing
feature-based methods are comprised of a variety of statistical and
handcrafted features. The application of different sensing devices causes
significant variations in the quality of fingerprint samples which affects the
performance of these methods. This phenomenon can be seen in the fingerprint
images captured using the orcanthus sensor. The poor quality of images in the
orcanthus dataset makes it difficult to extract accurate hand-crafted features
from them. These methods are capable to detect the spoof fingerprint in the
intra-sensor and intra-material experimental scenarios but do not resemble the
same performance in the cross-sensor or cross-material paradigm. Various
researchers have adopted deep learning-based systems to deal with this problem
due to their superior image classification capability. These models have a
series of convolutional layers that extracts minute features from fingerprint
images. Although, deep learning is a strong tool, as proved in Uliyan et al
(2020)Nogueira et al (2016) but most of these methods are not capable of
dealing with the problem of FPAD in cross-material and cross-sensor paradigms
due to less generalizable capability across different sensor and materials.
The main objective of the proposed method is to empower a deep-learning
classifier in such a way that makes it capable to detect the spoof samples
fabricated with unknown material as well as captured with some unknown
sensors.

In this paper, we propose a deep learning-based approach that outperforms
various existing methods for FPAD. Since detection of fingerprint samples
fabricated with unknown spoof materials i.e. open-set PAD is a key issue for
the security of fingerprint recognition systems, we have developed a unique
generator called an Open-Patch Generator (OPG) that enables a classifier to
deal with the problem of open-set as well as cross-sensor PAs. We have used
the Generative Adversarial Network (GAN), which works on the unsupervised
learning method of machine learning. GAN consists of two sub-models that are
generator and discriminator. The generator creates fingerprint samples, while
the discriminator determines if the input sample is real or created from the
generator model. One more benefit of using GAN to create these samples is that
the generated sample does not belong to a particular class of samples while
properties of input samples are present in it. Since the utilization of GAN
models costs in terms of time and computational resources, we have used it for
the generation of spoof samples only. A game-play between both these sub-
models enables the generator to create more realistic fingerprint samples
which are further utilized to train DenseNet classifiers. DenseNet is a deep
CNN architecture in which each layer is feed-forward connected to the other
layers. The feature maps from all previous layers are fed as input to each
layer, and the layer’s feature maps are fed as input to the succeeding layers.
We have used DenseNet as a classifier due to its capability to deal with the
problem of vanishing gradient. This problem persists with the input images
that have less feature or texture information.

##  2 Related Work

PAs are one of the major security threats to Automatic Fingerprint Recognition
Systems (AFRS). To protect the AFRS against PAs, various methods are proposed
in the literature. In recent years, various software-based methods are
proposed by various researchers to solve the problem of PAs in AFRS. According
to the methodology and utilization of resources, these methods can be further
divided into categories such as statistical feature-based methods,
perspiration and pore-based methods, handcrafted feature-based methods, and
deep learning-based methods. Some of the methods belonging to these categories
as well as their limitations are mentioned in this section.
Statistical feature-based methods: Choi et al. Choi et al (2009), suggested a
novel fingerprint liveness detection method based on statistical features such
as histogram, directional contrast, ridge thickness, and ridge signal. These
features are combined and used for the training of the SVM classifier on a
customized fingerprint database. Park et al. Park et al (2018b), utilized
statistical features such as deviation, variance, skewness, kurtosis, hyper
skewness, and hyper flatness along with three additional features i.e. average
brightness, standard deviation, and differential image, to train an SVM
classifier for the liveness detection on ATVS Fake Fingerprint database. Their
method have not been tested in cross-material and cross-sensor paradigms.
Marasco et al. Marasco and Sansone (2012) proposed a feature-based solution by
focusing on some static characteristics of fingerprint images. The texture of
an image provides some visual information related to the variation of grey-
level intensity and orientation. They retrieved certain first-order
statistical features from the fingerprint image, such as energy, entropy,
median, variance, etc., as well as some intensity-based features to train
different classifiers on LivDet 2009 database. Statistical feature-based
methods are quite sufficient to be utilized as an important feature but none
of these methods have shown the capability to detect spoofs in cross-material
and cross-sensor paradigms. Therefore, the problem remains unsolved for the
novel spoof materials and sensors by these methods.
Perspiration and pore-based methods: Pores are small holes that are present in
live fingerprints. This property is utilized by researchers to measure the
liveness of a fingerprint sample. Derakshini et al. Derakhshani et al (2003)
utilize the prevalence of perspiration pattern and sweat diffusion pattern
using ridge signal extracted from fingerprint samples to discriminate between
live and spoof fingerprints. Espinoza et al. Espinoza and Champod (2011)
demonstrate the use of pores for the detection of PAs. Sweat pore is a natural
pattern that persists in human fingers and remains missing in spoof
fingerprints. Authors have utilized one of the pore properties such as number
of pores as a feature for the detection of spoof fingerprints. The proposed
method is validated using a custom-made database. Abhyankar et al. Abhyankar
and Schuckers (2009) proposed a wavelet-based method for the detection of
fingerprint PAs. They exploited the sweat feature of fingerprints to
discriminate between live and spoof samples. Sweat is a natural property that
is not found in spoof fingerprints and hence it plays a vital role in
discrimination between live and spoof fingerprint samples. Marcialis et al.
Marcialis et al (2010) proposed a pore analysis method for the detection of
PAs. The system captures two fingerprint impressions at an interval of five-
second and then detects the number of pores in them. The proposed work is
evaluated using a custom-made database consisting of 8960 live and 7760 spoof
fingerprint images.
The major disadvantage with pore features as a liveness detector is that it
can only be detected in an image that is captured with a high-resolution
sensor while the presence of wetness or perspiration features in a fingerprint
depends on the atmosphere and external temperature. Also, these methods are
not capable to detect a spoof fingerprint sample in cross-material and cross-
sensor paradigms.
Handcrafted feature-based methods: Dubey et al. Dubey et al (2016) proposed a
shape and texture feature-based method. In their study, Speeded Up Robust
Feature [SURF] and Pyramid Extension of Histogram of Gradient [PHOG] are
utilized to extract shape information which works as a distinctive feature for
the detection of PAs. The ridges and valleys present in live and spoof
fingerprints have different shapes due to the different elasticity levels
possessed by the human skin and fabrication materials. Along with the above-
mentioned features, the gabor wavelet is used to analyze the texture
information. They evaluated the proposed method on LivDet 2011 and 2013
databases. The proposed method performs well in intra-sensor but have not been
tested in cross-material and cross-sensor paradigms. Rattani et al. Rattani
and Ross (2014) proposed a novel method to detect PAs against the samples
generated with the help of unknown materials. They suggested a novel material
detector which is a multi-class classifier to predict a fingerprint as a live,
spoof, or unknown. The proposed method have shown better performance for the
limited set of fingerprints but it have not been tested on any of the
available standard fingerprint databases. In another work, Rattani et al.
Rattani et al (2015) suggested the use of a Weibull-calibrated SVM as a
classifier. This SVM is a combination of 1-class SVM and binary SVM. This
modification has shown a significant improvement in performance as compared to
their previous work. Sharma et al. Sharma and Dey (2019) used quality-related
features such as ridge-valley smoothness, number of abnormal ridges and
valleys, ridge-valley clarity, frequency-domain analysis, orientation-
certainty level, and gabor quality which play a vital role in the detection of
spoof fingerprints. The live and spoof fingerprints have a different range of
values for these features due to the natural properties of live fingerprints
which are not present in their spoofs. The proposed method is validated on the
LivDet 2009, 2013, and 2015 databases. In continuation to Sharma and Dey
(2019), Sharma et al. Sharma et al (2020) proposed to extract white circular
patches, ridge ending, ridge bifurcation, and center intensity difference for
spoof detection. The proposed model is evaluated on LivDet 2015 databases.
Galbally et al. Galbally et al (2014) suggested 25 novel features for liveness
detection in three authentication systems i.e. face, iris, and fingerprint.
The proposed work have shown good performance in the intra-sensor paradigm.
Ghiani et al. Ghiani et al (2013) utilized a local image descriptor called
Binary Statistical Image Feature (BSIF). This feature is obtained by a set of
natural filters on fingerprint images whose output is then converted to binary
data. The proposed method is tested on Livdet 2011 database but not tested in
the cross-material and cross-sensor paradigm. In another work, Sharma et al.
Sharma and Dey (2021) utilized various hand-crafted texture descriptors such
as Weber Local Descriptor (WLD), Binary Statistical Image Features (BSIF),
Local Phase Quantization (LPQ), Local Contrast Phase Descriptor (LCPD), and
Rotation Invariant Co-occurrence among Adjacent Local Binary Pattern (RICLBP)
for FPAD. The results on LivDet 2011, 2013, and 2015 databases shows that none
of the texture descriptors performs uniformly well for the cross-sensor,
cross-material experimental scenarios. Xia et al. Xia et al (2017) suggested a
gradient-based method that extracts the second and third order co-occurrence
matrix of gradients and utilized it as a feature for the training of the SVM
classifier. This method is tested on LivDet 2009 and 2011 databases. In
another work, Xia et al. Xia et al (2020) proposed a local descriptor that
extracts intensity variance as well as gradient-based features. The co-
occurrence probability of these two features is assessed to form a feature
vector which is further used to train an SVM classifier. This method performs
well as compared to their previous approach when validated using LivDet 2011
and 2013 databases but not tested in open-set and cross-sensor paradigms.
Gragnaniello et al. Gragnaniello et al (2013) also utilizes the weber
descriptor to extract digital excitation and gradient information. Despite
calculating the co-occurrence matrix, it computes a joint histogram of them to
build feature vectors. This method is validated using LivDet 2011 and 2013
databases. Yuan et al. Yuan et al (2019) proposed a method that relies on the
creation of two co-occurrence matrices using the laplacian operator. This
operator is used to compute image gradient values for different quantization
operators. These matrices are utilized as a feature for the training of the
back-propagation neural network. This method is validated using LivDet 2013
database. Kim et al. Kim (2017) proposed a novel image descriptor that
utilizes the local coherence of a given fingerprint image. The creation of
spoof fingerprints using different fabrication materials results in a
significant difference between fingerprints. This method uses a local
coherence pattern as a feature to train SVM. The proposed method is validated
using ATVS and LivDet 2009, 2011, 2013, and 2015 databases. Sharma et al.
Sharma and Selwal (2021) have suggested a hybrid approach to deal with the
problem of PAs. They suggested a novel feature named Local Adaptive Binary
Pattern (LABP) which is an enhanced version of the existing Local Binary
Pattern (LBP). They have combined this feature with existing BSIF and Complete
Local Binary Pattern (CLBP) and used them for the training of the SVM
classifier to generate a global liveness score for fingerprint samples. The
proposed method is validated on LivDet 2009, 2011, 2013, and 2015 databases.
The methods discussed in this category perform well for the intra-sensor and
material scenarios but do not maintain the same performance in cross-sensor
and cross-material scenarios.
Deep learning-based methods: Arora et al. Arora (2019), proposed a robust
framework to detect PAs in fingerprint biometric systems which involves
contrast enhancement using histogram equalization as a pre-processing
technique. They have used VGG architecture as a classifier. Authors have
validated this work on a variety of fingerprint databases which include FVC
2006, ATVSFFp, finger vein database, LivDet 2013, and 2015 databases. Uliyan
et al. Uliyan et al (2020) proposed a deep features-based spoof detection
method. It utilizes a Deep Boltzmann Machine [DBM] for the extraction of
features from fingerprint images that is capable to find a complex
relationship among the features. The proposed method exhibits better
performance when compared with hand-crafted feature-based methods but suffers
from poor performance in cross-sensor and cross-material PAD scenarios. Gomez-
Barrero et al. Gomez-Barrero et al (2018) proposed a multi-modal architecture
which relies on the surface of the fingerprint within the shortwave infrared
region of the Short Wave Infra Red (SWIR) sensor as well as Laser Speckle
Contrast Image (LSCI) technology. The SWIR sensor captures an RGB image of the
fingerprint which is classified by Resnet and VGG19 classifiers. In addition
to this, averaged LSCI image undergoes through some feature extraction modules
and then the extracted features are used by SVM for liveness score generation.
Scores generated by both models are fused to get a global liveness score.
Chugh et al. Chugh et al (2018) proposed a deep learning-based method that
uses minutiae-centered fingerprint images for the training and testing of a
MobileNet CNN classifier. A fingerprint is cropped into a finite number of
patches based on the number of minutiae points. The extracted patches are fed
to a MobileNet CNN model which generates a liveness score for every patch. The
score produced by the classifier for all the patches are fused together to
compute global score. This method have been tested on LivDet 2011, 2013, and
2015 databases as well as the Michigan State University’s FPAD database. In
continuation to their previous work, Chugh et al. Chugh and Jain (2021)
proposed another method for the detection of spoofs created using unknown
fabrication materials. Since, novel and cheap materials are being discovered
nowadays, it is hard to generalize an FPAD model for all kind of spoofs. They
have utilized an image synthesis technique mentioned in Huang and Belongie
(2017) to create new fingerprint samples which contribute to better training
of deep learning classifier. The method is tested on LivDet 2015 and 2017
databases. Nogueira et al. Nogueira et al (2016), utilized pre-trained CNN
architectures using style transfer. Their method involves various
architectures such as VGG, Alexnet, and CNN with SVM. The method is tested
over LivDet 2009, 2011, and 2013 databases.

The above study concludes that deep learning-based methods outperform the
methods belonging to other categories. The possession of convolutional layers
enables them to extract discriminating features. Some of the proposed methods
Arora (2019), Uliyan et al (2020), and Nogueira et al (2016) performs well for
the LivDet databases but still their performance is not well for the PAD in
cross-material and cross-sensor paradigms.

##  3 Proposed Work

In this paper, an OPG wrapper that is composed of a GAN is proposed for the
detection of PAs in fingerprint recognition systems. The proposed wrapper is
capable to generate fingerprint samples of other possible unknown spoof
materials. The proposed method comprises three steps: 1. Training of
Wasserstein’s GAN model as an OPG wrapper, 2. Generating spoof fingerprint
patches using the OPG wrapper and, 3. Training the section-wise DenseNet
classifiers with training dataset along with the spoof fingerprint patches
created by the OPG wrapper. The liveness score of a fingerprint image is
obtained by the fusion of scores generated for all section-wise patches. In
the following sections, descriptions of the GAN, Wasserstein’s GAN, OPG, and
DenseNet classifier are provided.

###  3.1  Generative Adversarial Network (GAN)

The GANs are a combination of two models, the generator and the discriminator.
Both the generator and the discriminator models are trained to complement each
other such that their performances will be improved in the subsequent epochs.
The generator takes the noise vector as input and generates new data samples,
whereas the discriminator predicts the possible image source S (real images
from the training dataset or OPG-generated samples) of the given input sample.
The log-likelihood function for predicting the image source S is given in Eq.
1. Here, $E[X]$ denotes the expectation value of variable $X$, $P$ is the
probability distribution over image source S, $X_{real}$ denotes an image from
input dataset and $X_{fake}$ denotes an image generated by the generator.

$L=E[logP(S=real\|X_{real})]+E[logP(S=fake\|X_{fake})]$ (1)

The discriminator model of the GAN tries to maximize the above equation while
the generator model tries to minimize the second term of the above equation.

###  3.2  Wasserstein’s GAN with Gradient Penalty

Although GANs are proven to be successful generators, it always remained a
challenging problem to train them. Therefore, to make the generators more
efficient, various techniques Salimans et al (2016)Metz et al (2016)Arjovsky
et al (2017)Poole et al (2016)Arjovsky and Bottou (2017) suggested different
generator objectives for training GANs. Arjowsky et al. Arjovsky and Bottou
(2017) have examined the convergence properties of the GAN optimized value
functions. Arjowsky et al. Arjovsky et al (2017), further, proposed a new
variation of basic GAN called Wasserstein’s GAN which uses Earth-Mover’s
distance as a loss function in place of Binary Cross Entropy Loss. Basic GANs
suffer from the problem of vanishing gradient and mode collapse which are
removed by using Earth-Mover’s distance as a loss function. In this
architecture, the generator tries to maximize the distance between the
probability distributions for real samples as well as generated samples while
the discriminator tends to minimize this distance. However, W-GANs sometimes
may fail to converge or generate low-quality images due to weight clipping.
Gulrajani et al. Gulrajani et al (2017) introduced gradient penalty as an
alternative to weight clipping which overcomes the shortcomings present in
W-GAN.

###  3.3 Open Patch Generator (OPG)

The purpose of devising an OPG is to generate spoof fingerprint patches to
improve CNN’s learning capabilities against the samples of unknown spoof
materials and unknown sensors. This process involves the training of separate
W-GAN models for every sensor. A W-GAN for a particular sensor A is trained
with spoof fingerprint patches captured from every sensor except sensor A so
that no data from the testing sensor gets involved in the training process.
For instance, training a W-GAN corresponding to HiScan includes spoof
fingerprint patches from all the sensors available in the training dataset
used, except HiScan. The wrapper is finally used to generate new generalized
spoof samples. This method helps the model in generalized training as the
models are trained with samples from multiple sensors and materials. The
protocol of using the OPG is associated with the testing sensor. For example,
devising a model to be tested on HiScan will require the use of patches
generated from W-GAN corresponding to HiScan from the OPG. Since the training
sample used to train W-GAN corresponding to HiScan doesn’t include samples
from the same sensor, this protocol ensures that no sample from the testing
domain is a part of the training, thus, maintaining zero correlation between
training and testing datasets. A more detailed explanation is presented while
discussing intra-sensor and cross-sensor experimental scenarios in section
4.4.

###  3.4  DenseNet

DenseNet Huang et al (2017) is a CNN model in which each layer is connected
directly with every other layer. The traditional CNN architectures suffer from
the problem of vanishing gradient. A vanishing gradient is an issue that
appears in deep CNNs when their depth is increased. Due to the involvement of
a large number of activation functions, the gradient of the loss function
approaches zero. As fingerprint images have limited texture and color
information as compared with imagenet datasets, CNNs face the problem of
vanishing gradient while being applied for classification on these images. The
dense connections among the layers as well as batch-normalization operation in
the transition block of the DenseNet classifier reduce the problem of
vanishing gradient and make it suitable to be used for fingerprint PAD.
The DenseNet model (version-121) is comprised of four dense blocks. Each block
has 6, 12, 24, and 16 dense layers as shown in Fig. 1. The internal
architecture of densenet is depicted in Fig. 2. Each dense layer consists of
three sequential operations including Batch-Normalization, Rectified Linear
Unit (ReLU), and Convolution. Thus it is defined by BN - ReLU - Conv where
Conv indicates convolutional layers with a $3\times 3$ kernel. Each layer
collects feature maps from all preceding layers. The output feature map can be
denoted with Eq. 2.

$X_{n}=F_{n}[A_{0},A_{1},A_{2},A_{3},.....A_{n_{-}1}]$ (2)

where $A_{0},A_{1},A_{n-1}$ represents the concatenation of all feature maps
from layers 0, 1, 2, …, n-1. $F_{n}$, denotes a function that performs BN -
ReLU - Conv ($3\times 3$). In addition, a bottleneck layer is used after each
dense block to perform 1x1 convolution with $2\times 2$ average pooling. As a
result, the densenet 121 model completely replicates the procedure as: BN -$>$
ReLU -$>$ Conv($3\times$3) -$>$ BN-$>$ ReLU -$>$ Conv($1\times 1$)…n-times.
The transition block consists of one more operation which is pooling. The
internal architecture of the transition block is depicted in Fig. 3.

Figure 1: Architecture of DenseNet-121 classifier

Figure 2: Composition of dense block

Figure 3: Composition of transition block

###  3.5 Training

In the training phase, first, the proposed OPG is trained to create new
fingerprint patches belonging to the spoof class. The generated spoof patches
along with the training dataset are used for the training of the DenseNet CNN
classifier. The entire process is described in the following sections.

####  3.5.1  Minutiae-centered patch extraction

The fingerprint minutiae points (ridge bifurcation and ridge endings) are
detected using the National Institute of Standards and Technology’s Biometric
Image Software (NBIS) Ko (2007). NBIS requires input images to be in .jpg
format, the training and testing images are first converted to the .jpg format
and then fed to the NBIS. We have extracted minutiae-centered patches instead
of random patches as minutiae are the most important features for fingerprint
matching. One more advantage of using patches in this way is that it makes the
data uniform irrespective of the different dimensions of fingerprint images
captured with different types of sensors. The output of the mindtct package of
NBIS includes detected minutiae points, each represented by four values- (i)
x-coordinate ($x_{i}$) value, (ii) y-coordinate ($y_{i}$) value, (iii)
orientation angle ($\theta_{i}$), and (iv) quality ($q_{i})$ ranging from 0 to
100. The quality value output helps to filter out low-quality minutiae points
by setting up a quality threshold.

We selected minutiae points for a set of sample fingerprint images using a
quality threshold varying between 0 to 20. Fig. 4 shows the selected minutiae
points at quality thresholds of 0, 10, 15, and 20. In Fig. 4, it can be seen
that the threshold value of 10 contains many false minutiae points while at
the threshold value of 20 many genuine minutiae points are rejected. The image
depicting the minutiae points at threshold value 15, shows that most of the
true minutiae are preserved at this threshold while neglecting the spurious
minutiae points. By further analyzing the threshold value between 10-20, it is
observed that the quality threshold of 15 gives the best set of true minutiae
points. Hence, the quality threshold 15 is selected to determine the eligible
minutiae points. After minutiae detection, patches with minutiae points as
their center are extracted. We decided the size of the patch to be $96\times
96$ by performing an experiment over patches of different sizes, that is,
$64\times 64$, $96\times 96$, and $128\times 128$. The results of the
aforementioned experiment are discussed in section 4.2.

(a) threshold - 0 (c) threshold - 15

(b) threshold - 10 (d) threshold - 20

Figure 4: Detected minutiae points for quality threshold values (a) 0, (b) 10,
(c) 15, (d) 20

####  3.5.2  Section-wise grouping of patches

The segmented part of the fingerprint of size $x\times y$ is divided into 9
sections each of size $x/3\times y/3$ and the minutiae-centered patches are
then mapped into these sections based on the location of the minutiae in the
segmented image. Patches belonging to the same section are similar in terms of
orientations and ridge flow. So mapping the patches section-wise, helps the
model to learn salient features precisely. Also, since all the sections are
independent, parallel training and testing for section-wise patches would
reduce time complexity.

####  3.5.3 Implementation Details

The proposed algorithm is implemented in Python and the models were
implemented using the Tensorflow-Keras library. All training and testing were
done over NVIDIA TESLA P100 GPU. Each W-GAN was trained for 125 epochs which
took around 7-8 hours to converge. The noise input to W-GAN is 128,
1-dimensional random values from the normal distribution. The DenseNet model
was trained for 100 epochs which took a maximum of 2 hours to complete its
training. To optimize the parameters, the Adam optimizer is used with a
learning rate initialized as 0.0001 and weight decay as 0.0004. The optimum
threshold values for the global Liveness Score for every model were estimated
using the DET curve. Since all the threshold values were very close to 0.5,
the threshold value is fixed to be 0.5.

####  3.5.4  OPG training and patch generation

OPG includes training a W-GAN model associated with every section and for
every sensor. For a given dataset with n sensors, the OPG consists of $9\times
n$ W-GAN models labeled $WGAN_{ij}(0\leq i\leq n-1,0\leq j\leq 8)$.
$WGAN_{ij}$ associated with $i^{th}$ sensor is trained with patches of
$j^{th}$ section from all the n-1 sensors expect the $i^{th}$ sensor to avoid
its involvement in the training process.

Figure 5: Block diagram depicting the training process of W-GAN model Figure
6: Fingerprint patches generated by the proposed OPG

Fig. 5 shows the training of the W-GAN with detailed structure of section 0
W-GAN model. W-GAN models for other sections are similar in structure but
trained with the patches of respective sections. After training, the generator
of the trained models is used to generate synthetic spoof samples. Fig. 6
shows some of the patches generated by the proposed OPG for different
datasets, sensors, and fingerprint sections.

####  3.5.5  Training DenseNet Classifier

The OPG acts as a wrapper over the spoof detector classifier model. The
wrapper feeds the generated synthetic samples to the classifier model by
augmenting the samples to training data. This makes the classifier robust
against fingerprint samples that might be created using various unknown
spoofing materials. CNN DenseNet-121 architecture is used as a classifier. It
has been modified for the problem of binary classification because the
original classifier is designed for 1000 classes. For the same, a block of 8
layers comprising Pooling, Batch-Normalization, Dropout, Dense, Dense, Batch-
Normalization, Dropout, and Dense layers have been added on top of it in the
mentioned order. The last Dense layer consist of a single neuron to represent
the class of input fingerprints to be either live or a spoof. Finally, the
compiled DenseNet models are trained from scratch for each section. The number
of synthetic spoof samples provided by the OPG is equal to the number of spoof
samples present in the training dataset. The training flow is presented in
Fig. 7.

Figure 7: Block diagram depicting the training process of DenseNet classifier

In the figure, the classifier refers to the DenseNet models with the 8
additional layers on top of it.

##  4 Experimental Results

###  4.1  Database and performance metrics

To assess the performance of the proposed approach, experiments are carried
out on the Liveness Detection Competition (LivDet) 2015 Mura et al (2015),
2017 Yambay et al (2018), and 2019 Orrù et al (2019) databases. Each database
consists of multiple datasets captured with various sensing devices. The
fingerprint samples in all the datasets are arranged separately for the
purpose of training and testing. Each dataset comprises fingerprint samples
with different quality and resolution as per the configuration of the sensor.
The detailed composition of the LivDet databases is provided in Table 4.1.

The performance of the proposed model is evaluated in accordance with the
ISO/IEc IS 30107 standards 30107-3:2017 (en). The values reported for each
experiment are the Attack Presentation Classification Error Rate (APCER)
denoted with Eq. 3 which represents the percentage of misclassified spoof
fingerprint images while the Bonafide Presentation Classification Error Rate
(BPCER) denoted with Eq. 4 represents the percentage of misclassified live
fingerprint images.

$APCER=\frac{\textnormal{Number of mis-classified fake
samples}}{\textnormal{Total fake samples}}\times{100}$ (3)

$BPCER=\frac{\textnormal{Number of mis-classified live
samples}}{\textnormal{Total live samples}}\times{100}$ (4)

The overall performance of the system is measured by Average Classification
Error(ACE) which is an averaged sum of APCER and BPCER. The equation denoting
the formulation of ACE is denoted with Eq. 5.

$ACE=\frac{APCER+BPCER}{2}$ (5)

For live fingerprint patches, the threshold is set to 0.5 according to the
binary classification criteria. Thus, a sample having a classification score
greater than 0.5 is classified as a live fingerprint while a sample with a
classification score less than 0.5 is considered as a spoof one. The ACE can
be further utilized to derive the accuracy of the model using the formula
mentioned below.

$Accuracy=100-ACE$ (6)

Table 1: Database description of LivDet 2015, 2017, and 2019 databases.

Database |  Sensor | Live | Spoof | Spoofing Materials
---|---|---|---|---
LivDet 2015 |  CrossMatch |  1000/1000 |  1473/1448 |  Body Double, Eco flex, Playdoh, OOMOO, Gelatin
Digital Persona |  1000/1000 |  1000/1500 |  Ecoflex, Latex, Gelatine, Woodglue, Liquid Ecoflex, RTV
GreenBit |  1000/1000 |  1000/1500
HiScan |  1000/1000 |  1000/1500
LivDet 2017 |  GreenBit |  1000/1700 |  1200/2040 |  Body double, Ecoflex, Woodglue, Gelatine, Latex, Liquid ecoflex
Orcanthus |  1000/1700 |  1180/2018 |  Body double, Ecoflex, Latex, Woodglue, Gelatine, Liquid ecoflex
Digital Persona |  999/1692 |  1199/2028 |  Body Double, Ecoflex, Woodglue, Gelatine, Latex, Liquid ecoflex
LivDet 2019 |  GreenBit |  1000/1020 |  1200/1224 |  Body double, Ecoflex, Woodglue, Mix1, Mix2, Liquid ecoflex
Orcanthus |  1000/990 |  1200/1088 |  Body double, Ecoflex, Woodglue, Mix1, Mix2, Liquid ecoflex
Digital Persona |  1000/1099 |  1000/1224 |  Ecoflex, Gelatine, Woodglue, Latex, Mix1, Mix2, Liquid ecoflex

###  4.2 Patch-Size Selection

The optimal size of the minutiae centered patches are decided using the
experiments performed on Greenbit sensor dataset of LivDet 2015. Experimental
results using three different patch size of $64\times 64$, $96\times 96$,
$128\times 128$ are provided in Table 2. The proposed model provides the best
ACE for the $96\times 96$ patch size. The performance for the $64\times 64$
and $128\times 128$ is significantly lower than the $96\times 96$ patch size.
This is because $64\times 64$ patch size do not provide an appropriate amount
of ridge-valley information and those with size $128\times 128$ cause
overlapping of fingerprint patches. The better performance of $96\times 96$
patch size can also be seen in the Receiver Operating Characteristic (ROC)
curve plotted in Fig. 8. Therefore, we have selected the minutiae centered
patch size of $96\times 96$ for all further experiments.

Table 2: Performance of the proposed method on fingerprint patches of
different sizes

Patch Size |  BPCER (%) |  APCER (%) |  ACE (%)
---|---|---|---
$64\times 64$ |  5.63 |  1.07 |  3.35
$96\times 96$ |  2.21 |  2.34 |  2.27
$128\times 128$ |  5.43 |  0.53 |  2.98

Figure 8: ROC Curve of the proposed method on fingerprint patches of different
size

###  4.3 Testing Procedure

The testing of an input fingerprint involves the utilization of trained
DensNet models for each of the nine sections.

Fig. 9 shows the testing pipeline. Minutiae-centered patches are extracted
from input test images and are mapped to their respective section. The
section-wise grouped patches are fed to the trained DenseNet classifier
dedicated for a particular section.

Figure 9: Block diagram depicting the testing process of DenseNet classifier

The classifier model generates a liveness score for each input patch. The
individual liveness score for every patch is then averaged to produce a global
liveness score (S) for the entire fingerprint image. The formulation of the
global liveness score (S) is denoted with Eq. 7.

$S=1/n\sum_{i=1}^{\text{$n$}}s_{i}$ (7)

Here, $s_{i}$ is the liveness score of $i^{th}$ patch and n is the total
number of patches extracted. An input image is predicted to be live if its
global liveness score is more than a threshold value of 0.5 and predicted
spoof otherwise.

###  4.4 Comparative Analysis

The performance of the proposed model is assessed in three different scenarios
that demonstrate its robustness against samples created with unknown spoof
materials and captured with unknown sensors. All the details of these setups
are mentioned in the following sections.

####  4.4.1 Intra-Sensor and Known Spoof Material

In this setup, all the training, as well as testing images are acquired with
the same sensor and the testing dataset contains the spoof fingerprint samples
of the materials which are used during the training. We distributed all the
minutiae-centered patches to nine section-wise models which introduce parallel
processing and result in less processing time for fingerprints for real-life
applications. Table 3 presents the performance of the proposed model. Since
only LivDet 2015 consists of spoof fingerprint samples created with known and
unknown materials, the column “APCER (Known)” indicates the performance of the
proposed model in this setup. An ACE of 4.61% is achieved in this setup which
indicates its capability to detect the spoofs created with the known
materials.

Table 3: Intra-Sensor performance on LivDet 2015 database

Sensor |  BPCER (%) |  APCER (%) |  APCER (Known) (%) |  APCER (Unknown) (%) |  ACE (%)
---|---|---|---|---|---
CrossMatch |  2.08 |  2.78 |  3.46 |  1.83 |  2.49
Digital Persona |  4.42 |  7.87 |  5.48 |  12.7 |  6.50
HiScan |  3.71 |  3.89 |  7.5 |  5.62 |  3.82
GreenBit |  2.21 |  2.34 |  2 |  3.4 |  2.27
Average |  3.10 |  4.22 |  4.61 |  5.88 |  3.80

####  4.4.2 Intra-Sensor and Unknown Spoof Material

In this configuration, training and testing samples are collected from the
same sensor but the testing dataset have some or all spoof samples belonging
to other materials that are not utilized in the training of the model. In this
scenario, testing any model stresses the model’s performance in a real-world
context where a spoof fingerprint may be generated using any novel substance.
LivDet 2015 is considered as semi-cross-material dataset since only one-third
of testing spoof samples captured by all four sensors are created using
unknown material. LivDet 2017 and 2019 are completely cross-material
databases. The performance of the proposed model in this evaluation setup is
represented by the column labeled “APCER (Unknown)” for the LivDet 2015
database in Table 3 while the performance over LivDet 2017 and 2019 is
reported in Table 4 and 5. The ACE of 5.88%, 5.03%, and 7.10% are reported on
LivDet 2015, 2017, and 2019 databases, respectively.

Table 4: Intra-Sensor Performance on LivDet 2017 database

Sensor |  BPCER (%) |  APCER (%) |  ACE (%)
---|---|---|---
GreenBit |  5.02 |  5.47 |  5.26
Digital Persona |  6.09 |  6.58 |  6.36
Orcanthus |  5.37 |  1.87 |  3.47
Average |  5.45 |  4.63 |  5.03

Table 5: Intra-Sensor performance on LivDet 2019 database

Sensor |  BPCER (%) |  APCER (%) |  ACE (%)
---|---|---|---
GreenBit |  2.65 |  0.16 |  1.29
Digital Persona |  5.89 |  29.0 |  18.45
Orcanthus |  5.05 |  0.09 |  2.45
Average |  4.52 |  10.12 |  7.10

####  4.4.3 Cross-Sensor

In this experimental setup, training and testing images are captured using
different sensors. It checks the capability of the model to detect the PAs in
case the testing samples are captured with different sensors than the training
samples. Table 6 lists the outcomes of the model in this scenario for LivDet
2015, while Tables 7 and 8 lists the results for LivDet 2017 and 2019
databases.

Table 6: Cross-Sensor performance on LivDet 2015 database

Sensor [Training (Testing)] |  Accuracy (%)
---|---
CrossMatch (Digital Persona) |  52.38
CrossMatch (HiScan) |  57.95
CrossMatch (GreenBit) |  59.17
GreenBit (CrossMatch) |  71.23
GreenBit (Digital Persona) |  87.47
GreenBit (HiScan) |  70.44
HiScan (CrossMatch) |  63.38
HiScan (Digital Persona) |  81.66
HiScan (GreenBit) |  85.32
Digital Persona (CrossMatch) |  70.14
Digital Persona (HiScan) |  60.12
Digital Persona (GreenBit) |  74.89
Average |  69.51

Table 7: Cross-Sensor performance on LivDet 2017 database.

Sensor Training (Testing) |  | Accuracy
---
GreenBit (Orcanthus) |  58.73
GreenBit (Digital Persona) |  88.011
Orcanthus (GreenBit) |  50.054
Orcanthus (Digital Persona) |  55.699
Digital Persona (GreenBit) |  80.31
Digital Persona (Orcanthus) |  57.41
Average |  65.09

Table 8: Cross-Sensor performance on LivDet 2019 database

Sensor [Training (Testing)] |  Accuracy (%)
---|---
GreenBit (Orcanthus) |  58.229
GreenBit (Digital Persona) |  88.141
Orcanthus (GreenBit) |  50.089
Orcanthus (Digital Persona) |  56.175
Digital Persona (GreenBit) |  79.679
Digital Persona (Orcanthus) |  53.417
Average |  64.455

####  4.4.4 Comparison with existing approaches in Intra-Sensor paradigm

The obtained results of the proposed approach are compared to state-of-the-art
FPAD techniques tested on LivDet 2015, 2017, and 2017 databases. Comparison on
LivDet 2015 is presented in Table 9. Table 9 indicates that our proposed
method outperforms the methods discussed in Kim (2017), Yuan et al (2019),
Ojala et al (2002), Gragnaniello et al (2013), Xia et al (2020), Xia et al
(2017), Sharma and Selwal (2021), Park et al (2018a), Uliyan et al (2020),
Mura et al (2015), Sharma and Dey (2019) and Sharma and Selwal (2021). The
aforementioned comparison is also depicted graphically in Fig. 10. Similarly,
comparison on LivDet 2017 which is mentioned in Table 10 shows that our
proposed method outperforms Chugh et al. Chugh et al (2018) on the Orcanthus
dataset, Zhang et al. Zhang et al (2019) on Digital Persona and Orcanthus
datasets. The overall performance of our proposed method is comparatively
equivalent to the existing works on LivDet 2017. This comparison is also
depicted graphically in Fig. 11. The result is not compared on LivDet 2019
database since no method is tested on this database in this scenario to the
best of our knowledge.

Table 9: Comparison with state-of-the-art methods over LivDet 2015 database.

Various Methods |  HiScan |  Cross-Match |  Green-Bit |  Digital Persona |  Average
---|---|---|---|---|---
Kim et al. Kim (2017) |  - |  - |  - |  - |  86.39
Yuan et al. Yuan et al (2019) |  - |  - |  - |  - |  88.99
Ojala et al. Ojala et al (2002) |  - |  - |  - |  - |  88.18
Gragnaniello et al. Gragnaniello et al (2013) |  - |  - |  - |  - |  85.73
Xia et al. Xia et al (2020) |  90.36 |  89.18 |  95.47 |  86.28 |  90.33
Xia et al. Xia et al (2017) |  - |  - |  - |  - |  86.39
Sharma et al. Sharma and Dey (2019) |  95.22 |  98.07 |  95.7 |  94.16 |  95.78
Jung et al. Jung and Heo (2018) |  95.80 |  98.60 |  96.20 |  90.50 |  95.23
Park et al. Park et al (2018a) |  |  - |  - |  - |  86.39
Uliyan et al. Uliyan et al (2020) |  - |  95.00 |  - |  - |  95.00
Sharma et al. Sharma and Selwal (2021) |  95.5 |  95.00 |  96.7 |  97.2 |  96.10
LivDet 2015 winner Mura et al (2015) |  94.36 |  98.10 |  95.40 |  93.72 |  95.39
Proposed Method |  96.18 |  97.51 |  97.63 |  93.50 |  96.20

Figure 10: Comparative analysis of proposed method on LivDet 2015 database in
intra-sensor paradigm Table 10: Comparison with state-of-the-art methods over
LivDet 2017 database.

Various Methods | Orcanthus | Digital Persona | GreenBit | Average
---|---|---|---|---
Chugh et al. Chugh et al (2018)* |  94.51 |  95.12 |  96.68 |  95.43
Zhang et al. Zhang et al (2019)* |  93.93 |  92.89 |  95.20 |  94.00
Proposed Method |  96.53 |  93.64 |  94.74 |  94.97

*The findings of Zhang et al. Zhang et al (2019) and Chugh et al. Chugh et al (2018) are referred from Chugh et al. Chugh and Jain (2021).

Figure 11: Comparative analysis of proposed method on LivDet 2017 database in
intra-sensor paradigm

####  4.4.5 Comparison with existing approaches in Cross-Sensor paradigm

A comparative analysis of the cross-sensor experimental performance is given
in Table 11 which indicates that the proposed model’s performance is
comparable with Zhang et al (2019) and Chugh et al (2018) on LivDet 2017
database in cross-sensor paradigm. The aforementioned comparison is also
depicted graphically in Fig. 12. The results of the cross-sensor paradigm on
LivDet 2015 and 2019 databases could not be compared as no other existing
approach have performed experiments in the cross-sensor experimental setup.
The comparative analysis shows that the involvement of the proposed OPG
enables the DenseNet classifier to detect PAs in cross-sensor and cross-
material paradigms.

Table 11: Comparison with state-of-the-art methods on LivDet 2017 database in
cross-sensor paradigm.

Sensor Training (Testing) |  Zhang et al. Zhang et al (2019)* |  Chugh et al. Chugh et al (2018)* |  | Accuracy
---
GreenBit (Orcanthus) |  43.98 |  49.43 |  58.73
GreenBit (Digital Persona) |  80.39 |  89.37 |  88.011
Orcanthus (GreenBit) |  68.82 |  69.93 |  50.054
Orcanthus (Digital Persona) |  62.30 |  57.99 |  55.699
Digital Persona (GreenBit) |  87.9 |  89.54 |  80.31
Digital Persona (Orcanthus) |  44.30 |  49.32 |  57.41
Average |  64.61 |  67.59 |  65.09

* The findings of Zhang et al. Zhang et al (2019) and Chugh et al. Chugh et al (2018) are referred from Chugh et al. Chugh and Jain (2021).

Figure 12: Comparative analysis of proposed method on LivDet 2017 database in
cross-sensor paradigm

We have also reported the Detection Error Trade-off curves for all the
datasets of LivDet 2015, 2017, and 2019 databases which are depicted in Fig.
13. It plots APCER vs BPCER and helps in assessing the model’s performance. It
can be observed that the curves for different sensors of LivDet 2015 and
LivDet 2017 are similar and close to each other which shows that the model
gives consistent performance across all the sensors in the databases. We can
also analyze the high-security performance of the model using the DET curve.
For a high-security system, a low APCER is preferred. It can be observed that
at a low APCER, around 1%, for LivDet 2015, the BPCER is around 15 to 30% for
CrossMatch, GreenBit, and HiScan sensors and reaches around 50% for the
Digital Persona sensor. For LivDet 2017, the BPCER is in the range of 20 to
40% for all sensors in the case of 1% APCER. For LivDet, 2019 the BPCER is
between 1% and 5% for Orcanthus and GreenBit, and 50% for Digital Persona at
1% APCER.

Figure 13: Detection Error Trade-off (DET) curves for LivDet databases (a)
2015, (b) 2017 and (c) 2019

## Conclusion

In this work, we have proposed an OPG wrapper that augments the training
dataset with GAN-generated spoof samples which helps in building a generalized
PAD model. The training of the DenseNet classifier with the training dataset
along with the OPG-generated patches improves its capability to detect spoof
fingerprints. It also empowers the classifier to detect fingerprints, created
with materials that are unseen at the time of training. The proposed model is
validated through a set of experiments using various standard databases which
results in improvement over the current state-of-the-art methods. The division
of fingerprints into nine sections and processing them in parallel makes it
suitable for the various real-time applications required to ensure the
security of fingerprint-based recognition systems.

## Declaration

  * •

The publicly available databases that are used for the validation of this work
can be accessed by the following link: https://livdet.org/registration.php

  * •

Conflict of Interest: The authors declare that they have no conflict of
interest

## References

  *   * 30107-3:2017 (en) 30107-3:2017(en) I (2017) Information technology — biometric presentation attack detection — part 3: Testing and reporting
  * Abhyankar and Schuckers (2009) Abhyankar A, Schuckers S (2009) Integrating a wavelet based perspiration liveness check with fingerprint recognition. Pattern Recognition 42(3):452–464
  * Arjovsky and Bottou (2017) Arjovsky M, Bottou L (2017) Towards principled methods for training generative adversarial networks. In: 5th International Conference on Learning Representations, (ICLR), pp 1–17
  * Arjovsky et al (2017) Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: 34th International Conference on Machine Learning (ICML), pp 214–223
  * Arora (2019) Arora S (2019) Fingerprint spoofing detection to improve customer security in mobile financial applications using deep learning. Arabian Journal for Science and Engineering 45(4):2847–2863
  * Choi et al (2009) Choi H, Kang R, Choi K, et al (2009) Fake-fingerprint detection using multiple static features. Optical Engineering 48(4):047202–15
  * Chugh and Jain (2021) Chugh T, Jain AK (2021) Fingerprint spoof detector generalization. IEEE Transactions on Information Forensics and Security 16:42–55
  * Chugh et al (2018) Chugh T, Cao K, Jain AK (2018) Fingerprint spoof buster: Use of minutiae-centered patches. IEEE Transactions on Information Forensics and Security 13(9):2190–2202
  * Derakhshani et al (2003) Derakhshani R, Schuckers SA, Hornak LA, et al (2003) Determination of vitality from a non-invasive biomedical measurement for use in fingerprint scanners. Pattern Recognition 36(2):383–396
  * Dubey et al (2016) Dubey RK, Goh J, Thing VLL (2016) Fingerprint liveness detection from single image using low-level features and shape analysis. IEEE Transactions on Information Forensics and Security 11(7):1461–1475
  * Espinoza and Champod (2011) Espinoza M, Champod C (2011) Using the number of pores on fingerprint images to detect spoofing attacks. In: International Conference on Hand-Based Biometrics (ICHB), pp 1–5
  * Galbally et al (2014) Galbally J, Marcel S, Fierrez J (2014) Image quality assessment for fake biometric detection: Application to iris, fingerprint, and face recognition. IEEE Transactions on Image Processing 23(2):710–724
  * Ghiani et al (2013) Ghiani L, Hadid A, Marcialis GL, et al (2013) Fingerprint liveness detection using binarized statistical image features. In: IEEE 6th International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp 1–6
  * Gomez-Barrero et al (2018) Gomez-Barrero M, Kolberg J, Busch C (2018) Towards multi-modal finger presentation attack detection. In: International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), pp 547–552
  * Gragnaniello et al (2013) Gragnaniello D, Poggi G, Sansone C, et al (2013) Fingerprint liveness detection based on weber local image descriptor. In: IEEE Workshop on Biometric Measurements and Systems for Security and Medical Applications, pp 46–50
  * Gulrajani et al (2017) Gulrajani I, Ahmed F, Arjovsky M, et al (2017) Improved training of wasserstein gans. In: 31st International Conference on Neural Information Processing Systems (NIPS), p 5769–5779
  * Huang et al (2017) Huang G, Liu Z, Maaten LVD, et al (2017) Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2261–2269
  * Huang and Belongie (2017) Huang X, Belongie S (2017) Arbitrary style transfer in real-time with adaptive instance normalization. In: IEEE International Conference on Computer Vision (ICCV), pp 1510–1519
  * Jung and Heo (2018) Jung HY, Heo Y (2018) Fingerprint liveness map construction using convolutional neural network. Electronics Letters 54(9):564–566
  * Kim (2017) Kim W (2017) Fingerprint liveness detection using local coherence patterns. IEEE Signal Processing Letters 24(1):51–55
  * Ko (2007) Ko K (2007) User’s guide to nist biometric image software (nbis). https://doi.org/10.6028/NIST.IR.7392
  * Marasco and Sansone (2012) Marasco E, Sansone C (2012) Combining perspiration- and morphology-based static features for fingerprint liveness detection. Pattern Recognition Letters 33(9):1148–1156
  * Marcialis et al (2010) Marcialis GL, Roli F, Tidu A (2010) Analysis of fingerprint pores for vitality detection. In: 20th International Conference on Pattern Recognition, pp 1289–1292
  * Metz et al (2016) Metz L, Poole B, Pfau D, et al (2016) Unrolled generative adversarial networks. CoRR abs/1611.02163
  * Mura et al (2015) Mura V, Ghiani L, Marcialis GL, et al (2015) Livdet 2015 fingerprint liveness detection competition. In: IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp 1–6
  * Nogueira et al (2016) Nogueira RF, de Alencar Lotufo R, Campos Machado R (2016) Fingerprint liveness detection using convolutional neural networks. IEEE Transactions on Information Forensics and Security 11(6):1206–1213
  * Ojala et al (2002) Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7):971–987
  * Orrù et al (2019) Orrù G, Casula R, Tuveri P, et al (2019) Livdet in action - fingerprint liveness detection competition 2019. In: International Conference on Biometrics (ICB), pp 1–6
  * Park et al (2018a) Park E, Cui X, Kim W, et al (2018a) End-to-end fingerprints liveness detection using convolutional networks with gram module. CoRR abs/1803.07830
  * Park et al (2018b) Park Y, Jang U, Lee EC (2018b) Statistical anti-spoofing method for fingerprint recognition. Soft Computing 22(13):4175–4184
  * Poole et al (2016) Poole B, Alemi AA, Sohl-Dickstein J, et al (2016) Improved generator objectives for gans. CoRR abs/1612.02780
  * Rattani and Ross (2014) Rattani A, Ross A (2014) Automatic adaptation of fingerprint liveness detector to new spoof materials. In: IEEE International Joint Conference on Biometrics (IJCB), pp 1–8
  * Rattani et al (2015) Rattani A, Scheirer WJ, Ross A (2015) Open set fingerprint spoof detection across novel fabrication materials. IEEE Transactions on Information Forensics and Security 10(11):2447–2460
  * Salimans et al (2016) Salimans T, Goodfellow I, Zaremba W, et al (2016) Improved techniques for training gans. In: Advances in Neural Information Processing Systems, pp 2234–2242
  * Sharma and Selwal (2021) Sharma D, Selwal A (2021) Hyfipad: a hybrid approach for fingerprint presentation attack detection using local and adaptive image features. The Visual Computer 38(8):2999–3025
  * Sharma and Dey (2019) Sharma R, Dey S (2019) Fingerprint liveness detection using local quality features. The Visual Computer 35(10):1393–1410
  * Sharma and Dey (2021) Sharma R, Dey S (2021) A comparative study of handcrafted local texture descriptors for fingerprint liveness detection under real world scenarios. Multimedia Tools and Applications 80:1–20
  * Sharma et al (2020) Sharma RP, Anshul A, Jha A, et al (2020) Investigating fingerprint quality features for liveness detection. In: Mining Intelligence and Knowledge Exploration (MIKE), pp 296–307
  * Uliyan et al (2020) Uliyan DM, Sadeghi S, Jalab HA (2020) Anti-spoofing method for fingerprint recognition using patch based deep learning machine. Engineering Science and Technology, an International Journal 23(2):264–273
  * Xia et al (2017) Xia Z, Lv R, Zhu Y, et al (2017) Fingerprint liveness detection using gradient-based texture features. Signal, Image and Video Processing 11(2):381–388
  * Xia et al (2020) Xia Z, Yuan C, Lv R, et al (2020) A novel weber local binary descriptor for fingerprint liveness detection. IEEE Transactions on Systems, Man, and Cybernetics: Systems 50(4):1526–1536
  * Yambay et al (2018) Yambay D, Schuckers S, Denning S, et al (2018) Livdet 2017 - fingerprint systems liveness detection competition. In: IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp 1–9
  * Yuan et al (2019) Yuan C, Sun X, Wu QM (2019) Difference co-occurrence matrix using bp neural network for fingerprint liveness detection. Soft Comput 23(13):5157–5169
  * Zhang et al (2019) Zhang Y, Shi D, Zhan X, et al (2019) Slim-rescnn: A deep residual convolutional neural network for fingerprint liveness detection. IEEE Access 7:91476–91487


11footnotetext: The first two authors contributed equally.

# Secure Federated Learning against Model Poisoning Attacks via Client
Filtering

Duygu Nur Yaldiz$\phantom{}{}^{*}$ , Tuo Zhang$\phantom{}{}^{*}$, Salman
Avestimehr
Department of Electrical and Computer Engineering
University of Southern California
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Given the distributed nature, detecting and defending against the backdoor
attack under federated learning (FL) systems is challenging. In this paper, we
observe that the cosine similarity of the last layer’s weight between the
global model and each local update could be used effectively as an indicator
of malicious model updates. Therefore, we propose CosDefense, a cosine-
similarity-based attacker detection algorithm. Specifically, under CosDefense,
the server calculates the cosine similarity score of the last layer’s weight
between the global model and each client update, labels malicious clients
whose score is much higher than the average, and filters them out of the model
aggregation in each round. Compared to existing defense schemes, CosDefense
does not require any extra information besides the received model updates to
operate and is compatible with client sampling. Experiment results on three
real-world datasets demonstrate that CosDefense could provide robust
performance under the state-of-the-art FL poisoning attack.

##  1 Introduction

The gist of Federated Learning (FL) is to train a model coordinated by a
server while preserving the clients’ data privacy Zhang et al. (2021).
However, this substantial property introduces new challenges. Since the server
does not have access to the client data due to privacy concerns, FL is
vulnerable to data or model poisoning attacks, in which the attacker send
corrupted updates and contaminates the global model. Given the distributed
nature of FL, it is challenging to detect and correct these failures under the
vanilla FL framework McMahan et al. (2016); Zhang et al. (2022).

Several solutions have been proposed to defend the server from model poisoning
attacks to relax the security challenge for the FL framework. Server-side
robust aggregation approaches aim to detect outliers by inspecting the client
updates, and filtering the malicious updates before model aggregation such as
Blanchard et al. (2017). Besides completely filtering out before model
aggregation, approaches proposed by Xu & Lyu (2021); Cao et al. (2021);
Regatti et al. (2021); Fung et al. (2020); Prakash et al. (2020) diminish the
aggregation coefficients of the clients that are likely to be malicious.
However, existing approaches have some critical drawbacks or unfeasible
assumptions as we summarize in Table 1.

In this work, we propose CosDetect, a cosine similarity based outlier
detection algorithm, to tackle the fundamental issues of existing defense
methods. We provide an intriguing finding that the weight inside the last
layer of the local model update is more sensitive to the local data
distribution than other layers. Based on this crucial observation, we propose
that the last layer of local updates from the malicious clients should be
outliers compared to the ones from the benign clients. By calculating the
cosine similarity of the last layer between each collected model update and
the last global model, it is possible to filter the poisoning updates before
model aggregation.

Table 1: Comparison of CosDefense with existing backdoor defense FL methods. |  | Compatibility with
---
Client Sampling
| Validation Data
---
at the Server
| Information of
---
Attacker Number
| Client Score
---
Maintenance
at the Server
Krum Blanchard et al. (2017) | $\surd$ | $\times$ | $\surd$ | $\times$
Multi-Krum Blanchard et al. (2017) | $\surd$ | $\times$ | $\surd$ | $\times$
Median Yin et al. (2018) | $\surd$ | $\times$ | $\times$ | $\times$
RFFL Xu & Lyu (2021) | $\times$ | $\times$ | $\times$ | $\surd$
FoolsGold Fung et al. (2020) | $\surd$ | $\times$ | $\times$ | $\surd$
FLTrust Cao et al. (2021) | $\surd$ | $\surd$ | $\times$ | $\times$
ByGars Regatti et al. (2021) | $\surd$ | $\surd$ | $\times$ | $\surd$
SageFlow Park et al. (2021) | $\surd$ | $\surd$ | $\times$ | $\times$
CosDefense (Our Method) | ✔ | ✖ | ✖ | ✖

As shown in Table 1, the proposed cosine similarity based outlier detection
scheme, though simple, has equipped CosDetect with multiform merits compared
to prior strategies: (1) CosDetect does not require representative benign data
at the server to distinguish the malicious updates. It is only built on
accessing the global model parameters and the clients’ updates. (2) CosDetect
does not need the precise number of malicious clients per communication round
in advance for robust performance. (3) CosDetect does not rely on the previous
clients’ records to cluster the malicious clients. Instead, CosDetect
eliminates the outliers only based on the current round information, which
makes our algorithm compatible with client selection and brings more
flexibility for the implementation.

##  2 Approach Overview

###  2.1 Layer-wise Cosine Similarity for Model Weights

Figure 1: The average cosine similarity of model weight in various layers
between one client and the other nine clients. All 10 models are trained
independently for 1000 iterations without synchronization.

We first focus on a critical question: how would the attack be reflected on
the server side during FL training? Under the vanilla FL setting McMahan et
al. (2016), the only information the server holds during the training are the
collected model updates and the following aggregated model. A previous study
Zhao et al. (2020) notes that label information for the training data can be
computed analytically from the gradients of the last layer inside the machine
learning model under the centralized setting. Following this direction, one
intriguing finding from our empirical study is that compared to other layers,
the last layer’s weight is more sensitive to the input data distribution. We
quantify the similarity of model weights by cosine similarity, which is the
dot product of the two vectors divided by the product of individual norms as
$cos(\alpha)=\frac{<x,y>}{||x||\cdot||y||}$ where $\alpha$ is the angle
between vectors $x$ and $y$. Figure 1 shows the average cosine similarity for
each layer in the model across independent clients. In this experiment, ten
clients train a four-layer CNN-based model independently without any model
synchronization on the MNIST dataset, which has been non-iid partitioned (See
Section 3.1 for details). We observe that with the increment of the iteration,
the input-side layers show higher similarity than the output-side layers, and
the last layer has the lowest similarity score, because the local data
distribution among all clients mainly varies on the label distribution. These
observations provide critical insight that the local data label distribution
could be efficiently reflected in the last layer’s weight compared to the
other layers.

###  2.2 CosDefense: Filtering the Malicious Model Update via Cosine
Similarity

Based on the crucial observation we described in the previous section, we
propose that the last layer of the local models from the attackers should be
outliers compared to the ones from the benign models. Therefore, it is
possible to filter the malicious model update on the server by calculating the
cosine similarity between each collected model update and the last global
model. When calculating the cosine similarity between global model weights and
local model update, we have the following:

$\displaystyle cos(\alpha_{i})$
$\displaystyle=\frac{<\theta_{t},g_{t+1}^{i}>}{||\theta_{t}||\cdot||g_{t+1}^{i}||}=\frac{<\theta_{t},\theta_{t+1}^{i}-\theta_{t}>}{||\theta_{t}||\cdot||\theta_{t+1}^{i}-\theta_{t}||}=\frac{<\theta_{t},\theta_{t+1}^{i}>-<\theta_{t},\theta_{t}>}{||\theta_{t}||\cdot||\theta_{t+1}^{i}-\theta_{t}||}$
(1)

where $\alpha_{i}$ denotes the angle between global model weights
($\theta_{t}$) and local model update of client $i$ ($g_{t+1}^{i}$). If all
the participated clients are benign nodes, as the communication round $t$ goes
to infinity, the global model should be converged to an optimal point based on
the received local model update. Therefore, the cosine similarity between
global model weights and the local model update tends to be decreasing during
the training, as the difference between $\theta_{t}$ and $\theta_{t+1}^{i}$
(local model for client $i$) goes smaller for each client.

However, the cosine similarity scores from malicious clients do not follow
this phenomenon. As their aim is to prevent convergence, their local
optimization direction would be different compared to the benign client models
or the global model. As a result, the difference between $\theta_{t}$ and
$\theta_{t+1}^{i}$ from malicious nodes becomes more extensive than the benign
nodes, making the cosine similarity score between the malicious client update
and the global weights tend to be increasing during the training.

Figure 2: The trace of the average cosine similarity score of the last layer’s
weight between all received local updates and global model.

To demonstrate our thoughts, we trace the average cosine score of the last
layer local updates and that of the global model under benign and poisoning
attack cases ( Figure 2). Under both cases, the total client number is 100,
and the sampling rate is 0.1 in each round. For the poisoning attack case, we
launch the IPM attack Xie et al. (2019) at round 200, with 30% of the
participants being malicious clients. For a better illustration, Figure 2 is
smoothed by the moving averaged filter with window size 40. The average cosine
similarity increases sharply as the attack happens and is much higher than the
scores under the all-benign case, which confirms our speculation.

We observed that cosine similarity between global model parameters and the
benign local model updates tends to be smaller during the training, which
indicates that the direction of benign updates is less than perpendicular to
the global model, making it move toward a different direction in each round.
However, the average cosine similarity scores increase after the attack,
indicating that cosine similarity scores from malicious clients are much
larger than the scores from benign clients. The large cosine similarity values
represent that malicious client updates are aligned with the global model
parameters and do not update the global model in different directions but keep
it still. As a result, if there are enough malicious clients in the system,
their updates considerably diminish the benefit of benign client updates,
preventing the convergence of the global model.

Following these directions, we propose CosDefense, a server-side clustering-
based defense method as shown in the Appendix C. In each round $t$, after the
server receives all the updates from the sampled clients, it calculates the
cosine similarity between the last layer’s global weights and the last layer’s
local updates for each client. Then, the server clusters clients based on
their cosine similarity scores either as malicious or benign. If the cosine
similarity score of a client is much higher than others, then the server
labels that client as malicious for round $t$ and exclude that client from
aggregation. CosDefense only detects malicious clients before aggregation and
excludes them, allowing the server to perform any aggregation method with
benign client updates. Hence, CosDefense algorithm is compatible with any
aggregation method.

##  3 Evaluations

###  3.1 Experiment Setup

We conduct the evaluations on three datasets and two models: MNIST LeCun et
al. (1998) and Fashion-MNIST Xiao et al. (2017) with a four-layer CNN-based
model following the previous related work Li et al. (2022), and CIFAR-10
Krizhevsky (2009) with ResNet-18 model He et al. (2015). We generate the non-
iid partition for all datasets following previous FL works Li et al. (2022);
Fang et al. (2019) with default $q$ as 0.5, where the higher $q$ represents a
higher non-iid degree. We compare our defense method with two representative
robust aggregation rules: Krum Blanchard et al. (2017) and Clipping-Median. We
do not include the baselines such as FLTrust Cao et al. (2021) or RFFL Xu &
Lyu (2021) because they either need validation data on the server or do not
compatible with the client sampling. Further details of the baseline
implementation and parameter selection can be found in Appendix D

###  3.2 Defense Performance with the State-of-the-Art Model Poisoning Attack

To evaluate the defense performance, we evaluate the CosDefense on top of the
state-of-the-art model poisoning attack methods, Inner Product Manipulation
(IPM) Xie et al. (2019) attack. In the following experiments, we set the
attacker control 30% of the participants as malicious nodes, and the attack
would launch at round 200. The server randomly samples 10 clients from 100 for
local model training in each communication round.

Evaluation Results: Figure 3 shows the accuracy curve over 1000 communication
rounds for Krum, Clipping-Median, and CosDefense for MNIST, Fashion-MNIST, and
CIFAR-10. The detailed numerical results and ablation study related to the
non-iid degree are shown in Appendix E. We have three observations from the
results. (1) As Krum and Clipping-Median use all layers of the updates, the
results of CosDefense indicate that the similarity of the last layer between
the collected model update and global model is more sensitive to attacks,
which could effectively be used to filter out the malicious model updates. (2)
After the IPM attack begins, all the defense strategies drop the accuracy
significantly. However, the proposed CosDefense strategy has a much faster
convergence speed and shorter recovery time to resume the learning compared to
Krum and Clipping-Median. (3) For the most challenging dataset CIFAR-10, after
the IPM attack happens, CosDefense not only recovers the best accuracy before
round 200, but also has an increasing trend on the curve, while other defense
baselines even could not reach the best accuracy before the attack.

|  |
---|---|---
Figure 3: Accuracy performance over 1000 rounds for Krum, Clipping-Median,
CosDefense, and FedAvg on MNIST (left), FMNIST (middle), and CIFAR-10 (right)
under IPM attack.  Figure 4: Accuracy performance on MNIST dataset under
different numbers of attackers perform IPM attacks.

Impact of the number of the attackers. Some previous works on untargeted model
poisoning assume that there is a large fraction of attackers among
participants Fang et al. (2019); Xie et al. (2019). Therefore, we also
investigate the number of attackers’ impact on defense performance. In this
section, we conduct experiments on non-iid partitioned MNIST with $q=0.5$, and
vary the number of attackers from 10% to 40% of the total participants.
Results of this study are provided in Figure 4, showing the final global model
accuracy of defense methods under various settings. We observe that the two
defense baselines remain robust when the ratio of attackers is less than 30%.
As the ratio reaches 30%, both Krum and Clipping Median collapse on the
accuracy performance and fail to defend the model convergence, while
CosDefense still provides robust accuracy performance. One notable point is
that the CosDefense achieves the best accuracy compared to other baselines
among all different ratios of attackers. The experiment results demonstrate
that CosDefense is robust and reliable under both small-scale and large-scale
attack scenarios.

##  4 Conclusion

We presented CosDefense, a cluster-based defense scheme that could filter the
malicious updates out of the model aggregation on the server. Experiment
results on real-world datasets demonstrate that CosDefense can provide robust
performance under the state-of-the-art FL poisoning attack. We picture the
limitations and potential future work in Appendix F.

##  5 Acknowledgement

This material is based upon work supported by Defense Advanced Research
Projects Agency (DARPA) under Contract No. HR001120C0156, ARO award
W911NF1810400, ONR Award No. N00014-16-1-2189. The views, opinions, and/or
findings expressed are those of the author(s) and should not be interpreted as
representing the official views or policies of the Department of Defense or
the U.S. Government.

## References

  * Biggio et al. (2012) Battista Biggio, Blaine Nelson, and Pavel Laskov.  Poisoning attacks against support vector machines.  ICML, 2012.
  * Blanchard et al. (2017) Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer.  Machine learning with adversaries: Byzantine tolerant gradient descent.  NeurIPS, 2017.
  * Cao et al. (2021) Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong.  Fltrust: Byzantine-robust federated learning via trust bootstrapping.  NDSS, 2021.
  * Fang et al. (2019) Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong.  Local model poisoning attacks to byzantine-robust federated learning.  In _USENIX Security Symposium_ , 2019.
  * Fung et al. (2020) Clement Fung, Chris JM Yoon, and Ivan Beschastnikh.  The limitations of federated learning in sybil settings.  23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID), 2020.
  * He et al. (2015) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770–778, 2015.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ , 86:2278–2324, 1998.
  * Li et al. (2022) Henger Li, Xiaolin Sun, and Zizhan Zheng.  Learning to attack federated learning: A model-based reinforcement learning attack framework.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=4OHRr7gmhd4.
  * McMahan et al. (2016) H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _International Conference on Artificial Intelligence and Statistics_ , 2016.
  * Naseri et al. (2020) Mohammad Naseri, Jamie Hayes, and Emiliano De Cristofaro.  Toward robustness and privacy in federated learning: Experimenting with local and central differential privacy.  arXiv, 2020.
  * Park et al. (2021) Jungwuk Park, Dong-Jun Han, Minseok Choi, and Jaekyun Moon.  Sageflow: Robust federated learning against both stragglers and adversaries.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , volume 34, pp. 840–851, 2021.
  * Prakash et al. (2020) Saurav Prakash, Hanieh Hashemi, Yongqin Wang, Murali Annavaram, and Salman Avestimehr.  Secure and fault tolerant decentralized learning.  2020\.
  * Regatti et al. (2021) Jayanth Regatti, Hao Chen, and Abhishek Gupta.  Bygars: Byzantine sgd with arbitrary number of attackers.  International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML, 2021.
  * Sun et al. (2021) Jingwei Sun, Ang Li, Louis DiValentin, Amin Hassanzadeh, Yiran Chen, and Hai Li.  Fl-wbc: Enhancing robustness against model poisoning attacks in federated learning from a client perspective.  NeurIPS, 2021.
  * Sun et al. (2019) Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H. Brendan McMahan.  Can you really backdoor federated learning?  arXiv, 2019.
  * Tolpegin et al. (2020) Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu.  Data poisoning attacks against federated learning systems.  In _Computer Security – ESORICS 2020_ , pp. 480–501. Springer International Publishing, 2020.  ISBN 978-3-030-58951-6.
  * Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _ArXiv_ , abs/1708.07747, 2017.
  * Xie et al. (2019) Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta.  Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation.  In _35th Conference on Uncertainty in Artificial Intelligence_ , 2019\.
  * Xu & Lyu (2021) Xinyi Xu and Lingjuan Lyu.  A reputation mechanism is all you need: Collaborative fairness and adversarial robustness in federated learning.  International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML, 2021.
  * Yin et al. (2018) Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett.  Byzantine-robust distributed learning: Towards optimal statistical rates.  In Jennifer Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 5650–5659. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/yin18a.html.
  * Zhang et al. (2021) Tuo Zhang, Lei Gao, Chaoyang He, Mi Zhang, Bhaskar Krishnamachari, and Salman Avestimehr.  Federated learning for the internet of things: Applications, challenges, and opportunities.  _IEEE Internet of Things Magazine_ , 5:24–29, 2021.
  * Zhang et al. (2022) Tuo Zhang, Tiantian Feng, Samiul Alam, Sunwoo Lee, Mi Zhang, Shrikanth S Narayanan, and Salman Avestimehr.  Fedaudio: A federated learning benchmark for audio tasks.  _arXiv preprint arXiv:2210.15707_ , 2022.
  * Zhao et al. (2020) Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen.  idlg: Improved deep leakage from gradients.  _ArXiv_ , abs/2001.02610, 2020.
  * Zizzo et al. (2020) Giulio Zizzo, Ambrish Rawat, Mathieu Sinn, and Beat Buesser.  Fat: Federated adversarial training.  NeurIPS Workshop on Scalability, Privacy, and Security in Federated Learning (SpicyFL), 2020.

##  Appendix A Preliminary

###  A.1 Federated Learning

In this paper, we consider the federated learning setting that is similar to
the vanilla FedAvg McMahan et al. (2016), in which the FL system is composed
of a server with $\mathcal{K}$ clients, whose data is only locally kept
without sharing. The clients cooperate in training a global model
$\mathcal{W}_{g}$ with parameters $\theta$. We consider the following
distributed optimization problem: $\min\limits_{\theta}f(\theta)$, where
$f(\theta):=\sum_{i=1}^{\mathcal{K}}p_{i}F_{i}(\theta)$. The $F_{i}(\cdot)$
represents the local objective of client $i$ and $p_{i}$ denotes the
aggregation weight of client $i$ satisfying $p_{i}\geq 0$ and
$\sum_{i=1}^{\mathcal{C}}p_{i}=1$.

The federated learning algorithm runs as follows: in each communication round
$t$, the server randomly selects a subset $S_{t}$ from available clients for
client sampling and sends the global model parameters $\theta_{t}$ to the
selected clients. Each selected client $i\in S_{t}$ initializes its local
model parameters as $\theta_{t}$ and performs local model updates
$\theta^{i}_{t+1}=\theta_{t}-\eta\frac{1}{B}\sum_{z\in
b_{i}}\nabla_{\theta}\ell(\theta_{t};z)$, where $\eta$ is learning rate,
$b_{i}$ is a mini-batch of size $B$ randomly sampled from local data
$\mathcal{D}_{i}$. After local update finishes, each client $i\in S_{t}$ send
the model update $g^{i}_{t+1}:=\theta^{i}_{t+1}-\theta_{t}$ to the server.
Lastly, the server performs the aggregation function over the received model
updates and obtains a new global model such that
$\theta_{t+1}\leftarrow\theta_{t}-G_{t}$, where $G_{t}\leftarrow
Aggr(\\{g^{i}_{t+1}\\}_{i\in S_{t}})$. The newly updated model parameters
$\theta_{t+1}$ is then used to perform the next round of FL training. The FL
training is ended when the round number reaches $\mathcal{T}$.

###  A.2 Threat Model

We assume that the attacker’s goal is to diminish global model accuracy
regardless of specific inputs or classes, in which malicious clients perform
untargeted attacks. In this work, we assume that $p$ percentage of the total
number of clients is malicious, where $0\leq p\leq 1$. Let the malicious
clients be denoted by $\mathcal{M}$ and the set of malicious clients as
$\mathcal{A}$. All malicious clients share the same purpose and perform the
same type of attack. They are either coordinated by an external attacker or by
a leader client. During the FL training, the malicious attackers send crafted
local updates $\left\\{\widetilde{g}^{t}_{k}\right\\}_{k\in\mathcal{A}}$ to
the server to purposely maximize the global optimization function $f(\theta)$.
The crafted local updates are generated from their local data distributions,
which differ from the benign clients’ local distributions. The illustration of
the threat model is shown in Figure 5.

Figure 5: Threat model of the data poisoning attack in FL. After receiving the
global model from the server, the malicious clients purposely upload
contaminated model updates to poison the model.

##  Appendix B Related Works

Poisoning Attacks in FL. In federated learning, the attackers may control a
fraction of the clients and manipulate the local data or training process to
purposely poison the global model performance by injecting the wrong local
model update. This kind of attack is called the model poisoning attack, in
which the attacker contaminates the global model. Poisoning attacks are mainly
categorized into targeted and untargeted attacks according to the attacker’s
aim. Targeted attacks aim to keep global model performance high, while the
model considerably misbehaves on a specific set of inputs. This can be done
either by injecting backdoor triggers to the inputs regardless of the original
class or flipping the labels of only one class to a similar label (e.g.,
flipping 1 to 7 in MNIST or ’cat’ to ’dog’ in CIFAR10). On the other hand, the
purpose of untargeted attacks is to decrease the global model’s performance
significantly. Label-flip Biggio et al. (2012), and gradient modification
(e.g., sending random noise to the server instead of real update) Blanchard et
al. (2017) attacks are examples of this type. Moreover, Li et al. (2022)
utilizes reinforcement learning to create an optimal untargeted attack,
however, building the framework to generate online attack is costly in terms
of both time and computation. Another state-of-the-art untargeted model
poisoning attack is Inner Product Manipulation (IPM) proposed by Xie et al.
(2019). The aim of the IPM is to modify the local updates of the malicious
clients such that the inner product between the true gradient and the
aggregated updates is negative.

Defenses for Poisoning Attacks in FL. Several solutions have been proposed to
mitigate the effect of the attacks, so that model performance is not
negatively affected. Client-side defenses modify the local training algorithm
and rely on benign clients Zizzo et al. (2020); Sun et al. (2021). For
example, Sun et al. (2021) finds the model parameters that the attack is
hidden and perturbs those parameters during local training of benign clients.
Some server-side defense methods decrease the aggregation weights of the
malicious client updates Regatti et al. (2021); Cao et al. (2021); Xu & Lyu
(2021). While others perform clustering based on different techniques and
aggregate only benign clients Blanchard et al. (2017); Tolpegin et al. (2020).
For instance, Multi-Krum Blanchard et al. (2017) calculates the total
Euclidean distance from the $n-f-2$ nearest neighbors for each client update,
where $n$ is the number of clients and $f$ is the estimated malicious client
number. Then, only the $n-f$ updates with the minimum distance is aggregated.
Another work Sun et al. (2019) applies norm-clipping to the client updates
before the aggregation, based on the observation that norm of malicious
updates are large. Also, differential privacy methods (e.g., adding noise to
updates) are shown to be mitigating model poisoning attacks Sun et al. (2019);
Naseri et al. (2020).

However, some current solutions have critical drawbacks in terms of real-world
feasibility. For instance, algorithms proposed by Regatti et al. (2021); Cao
et al. (2021); Park et al. (2021) rely on representative benign data on the
server, which is hard to accomplish in actual FL deployment, where the server
seldom stores the ground-true validation data. Moreover, Xu & Lyu (2021) is
incompatible with client selection, an inevitable necessity for cross-device
FL where thousands of clients participate. Also, Krum and Multi-Krum Blanchard
et al. (2017) require the knowledge of the number of malicious clients in the
FL setting , which is not feasible. Lastly, methods proposed by Xu & Lyu
(2021); Fung et al. (2020); Regatti et al. (2021) maintains reputation scores
for each client on the server side, which is a serious overhead since
thousands of clients may participate in the FL system. Overall, there is a gap
for a defense solution in the FL setting that does not require server data, is
compatible with client selection, and is self-sufficient for detecting
malicious clients.

##  Appendix C Algorithm

CosDefense algorithm is presented in the Algorithm 1. More specifically, after
calculating the cosine similarity scores of each client, the server takes
their absolute value before and performs min-max normalization, which is
beneficial for separating the actual malicious and benign clients better.
After these two adjustments, we define the threshold value as the mean of
current scores and cluster clients accordingly. If the client score exceeds
the mean value, it is labeled malicious. Otherwise, the server considers it
benign and aggregates its weights for the next global model.

Algorithm 1 CosDefense

Input: Global model parameters $\theta_{t}$, local updates $g^{i}_{t+1}$ of
sampled clients $i\in S_{t}$, and aggregation method $Aggr(\cdot)$

Initialize: Benign client set $B=\\{\\}$ß

for each client $i$ in $S_{t}$ do

$cs_{i}\leftarrow|cosine\ similarity(\theta_{t,L},g_{t+1,L}^{i})|$

for each client $i$ in $cs_{i}$ do

$cs_{i}\leftarrow(cs_{i}-min(cs))/(max(cs)-min(cs))$

threshold $\leftarrow mean(cs)$

for each client $i$ in $S_{t}$ do

if $cs_{i}<$ threshold then

$B\leftarrow B\cup\\{i\\}$

$G_{g}\leftarrow Aggr(\\{g_{t+1}^{i}\\}_{i\in B})$

$\theta_{t+1}\leftarrow\theta_{t}-G_{t}$

Output: $\theta_{t+1}$

##  Appendix D Experimental Details

Datasets and partitioning. Dataset statistics are provided in Table 2. We
generate the non-iid partition for all datasets following previous FL works Li
et al. (2022); Fang et al. (2019). Specifically, suppose there are $C$ various
classes in a dataset. We evenly break all the clients into C groups, in which
each group is assigned $1/C$ of training samples as follows. A training point
with label $c$ would be assigned to $c$-th group with probability $q\geq 1/C$
and to each of the rest groups with probability $(1-q)/(1-C)$. The attacking
clients would be evenly distributed across $C$ groups. A higher $q$ value
stands for a higher non-iid degree. In the experiments above, we set the
default $q$ equals to 0.5 and partition the datasets into 100 clusters.

Table 2:  Dataset statistics.  Dataset | Train Clients | Train Examples | Test Examples | Input Size
---|---|---|---|---
MNIST | 100 | 60,000 | 10,000 | 1x28x28
Fashion-MNIST | 100 | 60,000 | 10,000 | 1x28x28
CIFAR-10 | 100 | 50,000 | 10,000 | 3x32x32

Baseline implementations and parameter selections. We follow the same
implementation details for Krum and Clipping-Median as in the previous work Li
et al. (2022). For Krum, we set the number of the attacker parameter $f$ to
its real value in each round for the best performance. For the Clipping-
Median, we first apply norm-clipping Sun et al. (2019) then perform
coordinate-wise median Yin et al. (2018), since it is shown to be more
powerful than the vanilla median Li et al. (2022). Moreover, we provide the
results for no defense case, where the server only performs FedAvg
aggregation. For all experiments, the communication rounds fix to 1000, and
the local training iteration number is 1 with batch size 128. The aggregation
function is FedAvg McMahan et al. (2016). The default local learning rate is
0.01 for all experiments.

Configurations. All experiments are conducted by CPU/GPU simulation. The
simulation experiments are conducted on a computing server with eight GPUs.
The server is equipped with AMD EPYC 7502 32-Core Processor and 1024G memory.
The GPU is NVIDIA RTX A4000.

##  Appendix E Experimental Results

The final global model accuracy of each method is provided in the table 3.

Table 3:  Global model accuracy comparison between No Defense, Krum, Clipping-Median, and CosDefense.  | No Defense | Krun | Clipping-Median | CosDefense
---|---|---|---|---
MNIST | 9.8% | 9.8% | 19.29% | 83.37%
F-MNIST | 10% | 10% | 21.24% | 67.63%
CIFAR-10 | 10% | 10% | 20.89% | 52.02%

Impact of the non-iid degree. In order to investigate the impact of data
heterogeneity on CosDefense, we partition the MNIST dataset with $q$ equals
0.1, 0.3, and 0.5, respectively, and run all three defense strategies on top
of the IPM attack for the evaluation. We set 30% of the devices as malicious
clients, the same as in the previous section. As shown in Figure 6, we observe
that CosDefense is stable with the change of the non-iid degree, and
outperforms the other two defense baselines on accuracy performance among all
scenarios. The results indicate that the CosDefense is resistant to the impact
of data heterogeneity compared to the other defense baselines.

Figure 6: Accuracy performance on MNIST dataset under different non-iid
partition levels.

##  Appendix F Limitations and Future works.

One of the directions for future work is the combination of CosDefense with
the existing secure aggregation frameworks. The challenging point is that
under the secure aggregation frameworks, each model update is locally
encrypted and uninspectable by a server. Instead, only the sum of the model
update would be revealed to the server when sufficient updates have been
received, which impedes the implementation of CosDefense on top of it.