# Finger-NestNet: Interpretable Fingerphoto Verification on Smartphone using
Deep Nested Residual Network

Raghavendra Ramachandra Hailin Li
Norwegian University of Science and Technology (NTNU), Norway
E-mail: {raghavendra.ramachandra, hailin.li} @ ntnu.no

###### Abstract

Fingerphoto images captured using a smartphone are successfully used to verify
the individuals that have enabled several applications. This work presents a
novel algorithm for fingerphoto verification using a nested residual block -
Finger-NestNet. The proposed Finger-NestNet architecture is designed with
three consecutive convolution blocks followed by a series of nested residual
blocks to achieve reliable fingerphoto verification. This paper also presents
the interpretability of the proposed method using four different visualization
techniques that can shed light on the critical regions in the fingerphoto
biometrics that can contribute to the reliable verification performance of the
proposed method. Extensive experiments are performed on the fingerphoto
dataset comprised of 196 unique fingers collected from 52 unique data subjects
using an iPhone6S. Experimental results indicate the improved verification of
the proposed method compared to six different existing methods with EER =
1.15%.

##  1 Introduction

The rapid evolution of smartphone technology with highly sophisticated cameras
aided reliable biometric authentication. Even though some smartphones in the
market have dedicated biometrics sensors, the popularity of smartphone camera-
based biometrics has paved the way for several applications because of the
scalability, flexibility, and user convenience. This has resulted in several
smartphone applications based on the different biometric characteristics such
as face [23], periocular [27], visible iris [21], multimodal biometrics [22]
and fingerphotos [30]. Among these biometric characteristics, fingerphoto
biometrics has gained significant interest as an alternative to touch-based
fingerprint authentication due to the COVID-19 pandemic. The fingerphoto
authentication employs the built-in camera (normally back-side) to capture the
image that can be used for the authentication. Figure 1 shows the example of a
fingerphoto image captured using an iPhone 6S back camera with various indoor
and outdoor scenarios.

Fingerphoto verification is a challenging task due to unconstrained capture
scenarios. The capture quality varies significantly depending on the capture
conditions like indoor, outdoor, flash on/off and external light. In addition,
the motion blur during fingerphoto capture is highly anticipated due to the
small micro-motion of the figure. Further, the fingerphoto images are captured
with varied orientation and distance, introducing the challenges of resolving
the valley with ridges that further limits the applicability of extracting the
minutia-based features. Therefore, the fingerphoto verification systems
indicate a degraded performance compared to the conventional fingerphoto
verification system.

Figure 1: Illustration of fingerphoto captured using smartphone with varying
environment conditions(brightness of light) Table 1: State-of-the-art
fingerphoto recognition methods on smartphone

Authors |  year |  Method |  Database and types of attacks
---|---|---|---
Lee et al. [14] | 2006 |  Color based segmentation and contrast between ridges and valleys |  150 fingers, 840 images.
Derawi et.al [9] | 2012 |  Matching using VeriFinger SDK |  1320 images from each capture device
Li et.al [15] | 2012 |  Matching using VeriFinger SDK and NFIS |  2100 images from 100 fingers
Raghavendra et. al [20] | 2013 |  Mean Shift Segmentation (MSS) and matching through minutiae feature based method |  1800 samples from 25 subjects
Tiwari and Gupta [6] | 2015 |  Matching using scale-invariant features |  156 images from 50 fingers
Sankaran et.al [25] | 2015 |  ScatNet |  5100 images from 128 fingers
Carney et.al [6] | 2017 |  Four-finger capturing scheme |  1400 touchless fingerprints
Chopra et.al [7] | 2018 |  ResNet50 representation based matching approach |  3450 fingerphotos from 230 fingers
Pankaj et al [29] | 2018 |  LBP, HOG, BSIF and VeriFinger SDK |  720 fingerphotos from 48 fingers
Deb et.al [8] | 2019 |  Evaluation of Fingerphoto-to-slap-fingerprint matching |  309 subjects, 7,976 images
Wild et.al [31] | 2019 |  Verifinger SDK |  4310 fingerprint images from 108 users
Birajadar et.al [5] | 2019 |  Source-AFIS, NBIS-NIST, Verifinger SDK |  200 subjects
Al-Nima et.al [2] | 2020 |  Shallow CNN model to recognize a middle finger and index finger |  IIITD database
Malhotra et.al [16] | 2020 |  Deep Scattering Network based feature extraction, and Random Decision Forest to authenticate finger-selfies |  19456 images (17024 fingerphotos and 2432 live scan fingerprints) from 304 fingers
Kauba et.al [12] | 2021 |  TensorFlow Object Detection API and transfer learning, MobileNetV2 |  Training, validation, and test datasets consisted of 1176 images, 168 images, and 784 images, respectively
Attrish et.al [4] | 2021 |  Fuse CNN-based and minutiae based feature score |  IITI-CFD containing 105 train and 100 test subjects
Priesnitz et.al [19] | 2022 |  A fingerprint recognition workflow which can process four fingers and useability issues |  1360 samples from 29 subjects

The smartphone-based fingerphoto verification is exhaustively studied in the
literature, resulting in several algorithms. Table 1 summarizes the
fingerphoto verification techniques developed based on smartphone data. Early
work on fingerphoto verification is proposed in [14] that explored the
applicability of fingerphoto for authentication. Since then, several
algorithms based on both local features like minutiae and global features like
Local Binary Patterns (LBP) [29], Binarized Statistical Image Features (BSIF)
[29], Histogram of Gradients (HoG) [29], Scale Invariant features [28],
Scattered wavelets [16]. The advances in deep learning have further motivated
the researchers to introduce Convolutional Neural Networks (CNN) for
fingerphoto verification on a smartphone. To this extent, the transfer
learning is performed using MobileNet V2 architecture for fingerphoto
verification [12]. The use of a shallow CNN network with single-layer
convolution [2] is introduced, which has indicated a reliable verification
performance. Further, the hybrid combination of deep features and minutiae-
based features [4] are introduced for reliable fingerphoto verification.
Recent work on fingerphoto verification introduced novel features based on the
haemoglobin [18] in the finger that can be quantified using color channels.

Figure 2: Block diagram of the proposed Finger-NestNet for contactless
fingerphoto verification

This research proposes a novel algorithm for fingerphoto verification from
smartphone-captured data. Based on the available literature, the recent
approaches are based on the pre-trained deep CNN; however, the fingerphoto
verification still indicates poor performance. Local features like minutiae
depend more on the quality of the fingerphoto, and similar observation holds
for global features-based fingerphoto verification systems. Further, we also
present a detailed analysis of how interpreting and visualizing the
activations resulted in the comparison score computed by the proposed method.
To the best of our knowledge, this is the first work on the fingerphoto that
discusses the interpretability of results and paves the way towards
explainability. The following are the main contributions of this work:

  * •

Proposed a novel contactless fingerphoto verification algorithm based on the
Finger-NestNet designed using nested residual blocks.

  * •

Introduced a new fingerphoto dataset comprised of 196 unique fingers collected
from 52 unique data subjects. The fingerphoto dataset is collected using the
rear camera of the iPhone 6S smartphone in multiple sessions with an indoor
scenario.

  * •

Extensive experiments are carried out by comparing the proposed method with
six different existing approaches. The existing approaches include both
academic systems based on the deep learning and Commercial-Off-The-Shelves
(COTS) system including NeuroTechnology Verifinger [1].

  * •

Extensive analysis of interpreting the proposed method is presented using four
different visualising techniques.

The rest of the paper is organised as follows: Section 2 presents and
discusses the proposed Finger-NestNet architecture, Section 3 discusses both
qualitative and quantitative results of the proposed and state-of-the-art
(SOTA) techniques. An extensive analysis of four different visualising
techniques is presented to interpret the outcome of the proposed method.
Finally, Section 4 draws the conclusion.

##  2 Proposed Method

Figure 2 shows the functional block diagram of the proposed Finger-NestNet
architecture for contactless fingerphoto verification. The goal of the
proposed method is to leverage the residual blocks[11] to achieve reliable
fingerphoto verification. We are motivated to design our network with stacked
residual blocks as they can learn the identity function through the skip
connection and address the vanishing gradients. Further, the stacked use of
residual blocks has proven to guarantee the generalization of unseen data
[10].

As shown in Figure 2, the proposed architecture has three consecutive layers
of convolution-group normalization-ReLu-Max Pooling that can learn the high-
level features. The first block is designed to have a convolution layer with a
kernel size of $21\times 21$, which is then reduced by half in the subsequent
blocks (e.g., kernel size of $14\times 14$ in block-2 and of $7\times 7$ in
block 3). The use of dissimilar kernels allows the network to learn different
high-level features. We have employed group normalization after each
convolution layer to independently normalize the mini-batch of data across
each observation that can overcome the sensitivity to network initialization.
The residual blocks are connected following the three blocks of the
convolutions. The six different residual blocks are connected serially with
skip connections. The residual blocks employed in this work have both direct
and indirect connections passed through the convolution layer that is summed
together at the output. The residual blocks 1, 2, 4 and 5 indicate a similar
architecture design but vary with the parameters to allow high-level feature
learning. The residual block-1 will connect the skip connection directly from
input to output, and the main stem is comprised of convolution-group
normalization-ReLu(Rectified Linear units)-convolution-group normalization
layers connected in series. The outcome of the additional layer that combines
the features from the main stem and the skip connection is further passed
through the non-linear activation ReLu that improves the robustness to
vanishing gradients. The convolution layer in the main stem in residual block
1 is kernel size $3\times 3$ with the feature maps size of 32. The residual
block-2 is identical to residual block-1. However, with residual block-3, we
introduce the convolution-group normalization layer in the skip connection to
extract the coarse features further. The convolution layer employed in the
skip connection has a kernel size of $1\times 1$ with a feature map size of
64. The residual block 4 is identical to residual block-2, but the number of
feature map sizes is increased to 64. Residual block 5 and 6 shares the same
architecture that of residual block 3 and 4, respectively. However, the
feature size of a residual block of 5 and 6 correspond to 128. The schematic
increase in the feature map size allows the network to precisely capture the
robust non-linear feature maps. The outcome of the residual block is then
normalized, and a fully connected layer is implemented with 128 nodes.

Table 2: Quantitative performance of the proposed contactless fingerphoto
verification

Algorithms |  EER(%)
---|---
AlexNet Features [12, 4] | 46.12
DenseNet Features [12, 4] | 24.78
MobileNet Features [12, 4] | 8.26
ResNet50 Features [12, 4] | 15.33
Deep Scattering Network [16] | 9.33
COTS [1] | 3.84
Proposed Method | 1.15

Finally, the SoftMax function is used at the output layer corresponding to the
number of unique fingerphotos. In this work, we have employed cross-entropy as
it is well suited for applications with more than two classes [17]. The
proposed Finger-NestNet is optimized using Adam optimizer [13] that takes
advantage of adaptive gradient and Root Mean Square (RMS) propagation.

##  3 Experiments and Results

This section discusses the qualitative and quantitative results of the
proposed contactless fingerphoto verification system. The quantitative
performance of the proposed method is benchmarked against deep learning-based
state-of-the-art techniques that include the deep scattering network [16], and
the pre-trained deep CNNs [12, 4]. We compare the performance of the proposed
method with the COTS (NeuroTechnology Verifinger) [1]. The verification
performance of the contactless fingerphoto algorithms is presented in terms of
False Non-Match Rate (FNMR), False Match Rate (FMR) and Equal Error Rate
(EER). The qualitative results are presented by interpreting the proposed
Finger-NestNet using four different visualization methods such as Grad-Cam
[26], Occlusion Sensitivity [32], Locally Interpretable Model-agnostic
Explanations (LIME) [24] and Gradient Attribution [3].

Experiments are carried out on the contactless fingerphoto dataset introduced
in [29] that is further augmented to have 196 unique fingers corresponding to
the 52 data subjects. The data is collected using the iPhone 6S with the back
camera that can record video at 240 fps with the flash turned on. Session-1
consists of 200 images per unique fingerphoto, and Session-2 consists of 200
images per unique fingerphoto. The fingerphotos are collected in indoor
scenarios with standard lighting settings in the two sessions. The contactless
fingerphoto systems are trained on the Session-1 data and tested solely on the
Session-2 data.

Figure 3: DET performance of the proposed and SOTA contactless fingerphoto
method Figure 4: Visualisation of the proposed Finger-NestNet with matched
verification (a) high verification match (b) normal verification match

Table 2 indicates the quantitative performance of the proposed method and the
state-of-the-art for smartphone-based contactless fingerphoto verification.
Figure 3 shows the ROC curves indicating the verification performance of the
proposed and existing contactless fingerphoto verification. The performance of
the proposed method is compared against five different deep learning-based
approaches that are well exploited in the literature for reliable contactless
fingerphoto verification. Based on the obtained results, it can be observed
that:

  * •

The state-of-the-art deep learning-based methods vary with the type of pre-
trained networks employed.

  * •

Among the pre-trained deep learning approaches, the AlexNet-based pre-trained
network has indicated degraded performance. The best performance among the
deep learning approaches is noted with the MobileNet features with EER =
8.25%.

  * •

Deep scattering network indicated comparable results than the deep feature-
based methods with EER = 9.33%.

  * •

The proposed method has indicated the best performance with EER = 1.15%. Thus,
the proposed method performs better than the existing contactless fingerphoto
verification approaches.

Figure 5: Visualisation of the proposed Finger-NestNet for (a) Enroled sample
(b) probe sample (correctly matched to enroled sample) (c) Probe sample
(falsely matched to enroled sample)

To interpret the performance of the proposed Finger-NestNet, we have evaluated
four different techniques to understand better how the proposed network can
translate the given contactless fingerphoto to the decision. In this work, we
have visualized the proposed Finger-NestNet with two different experiments:
(a) the genuine comparison that has resulted in high comparison scores and
normal comparison scores that has passed the verification threshold (b) match
and miss-matched comparison.

Figure 4 shows the qualitative results interpreting the decision achieved by
the proposed Finger-NestNet. Figure 4 (a) visualize the network activations
when the two fingerphotos from the same finger are compared had yielded a high
comparison score, whereas Figure 4 (b) shows the visualization of the network
when two fingerphotos from the same finger resulted in a normal value of
comparison score. As indicated in the Figure 4 following can be noticed:

  * •

The highest match scores are attributed to the activations in the identical
region of the fingerphoto image. The Grad-CAM map (see Figure 4 (a)) indicates
the larger gradient values in the top regions of the fingerphotos, especially
from the last ReLU layer of the proposed network. Further, the Grad-CAM map of
the earlier region (first convolution layer) highlights the edge and line type
features at the same finger region. Visualizing occlusion sensitivity between
two fingerphoto images highlights the same region of the two fingerphoto
images that have contributed to the comparison scores. The high-resolution
occlusion sensitivity that will highlight the more precision region also
indicates the similarity between the two fingerphotos that are compared. This
further justifies the high comparison scores. Further, it is also interesting
that high activations are noted around the core region of the fingerphoto
samples. Next, we consider the third visualization method using the locally
interpretable model-agnostic explanations (LIME) technique. The red area in
Figure 4 (a) indicates the highest important region that can contribute to the
decision. It is interesting to note that the LIME method also indicated the
similar areas (with different importance) between the fingerphotos that have
resulted in the high value of comparison score. The red colour maps can be
noted for the same regions (even though an area of the region is different).
Lastly, we present the visualization with gradient attribution that can
generate a pixel-resolution map corresponding to the most critical pixels in
the fingerphoto images. Figure 4 (a) indicates the gradient activation in the
same region that has contributed to the highest comparison score. Thus, based
on the four different visualizing methods, we notice that the activations in
the same regions have contributed to the highest comparison score.

  * •

Figure 4 (b) illustrates the case for the reduced comparison score magnitude.
Here, we can notice that even a small change in the activation region due to
the change in image characteristics will reflect a reduction of the comparison
score magnitude.

  * •

It is interesting to notice that the core region in the fingerphoto has
contributed mainly to achieving a high comparison score. Further, this
observation appears to be consistent with all four visualization methods
employed in this work. Thus, the best performance of the proposed method can
be attributed to the availability of good-quality information in the
fingerphoto core region.

Figure 5 shows the illustration of the visualization techniques on both
correctly and falsely matched fingerphotos samples by the proposed Finger-
NestNet. Figure 5 (a) shows the visualization indicating the activations of
the proposed method with enrolled fingerphoto, Figure 5 (b) probe samples that
are correctly verified and Figure 5 (c) probe samples that are falsely
verified to the enrolled sample. We can notice from Figure 5 (c) that the
activations are similar (even though not the same) to that of the enroled
fingerphoto image that has resulted in the false verification.

##  4 Conclusions

Reliable fingerphoto biometric smartphone verification is challenging due to
unconstrained data capture. This work presents a novel method for fingerphoto
biometric verification using nested residual blocks. The interpretability of
the proposed method is exclusively discussed using four different visualizing
techniques. The visualization methods consistently indicate the importance of
the core region in the fingerphoto image that has contributed to improved
performance. Extensive experiments are carried out on the fingerphoto dataset
comprised of 196 unique fingers collected from 52 unique data subjects using
iPhone 6FS. The performance of the proposed method is compared with six
different existing methods including COTS. Experimental results demonstrate
the best performance of the proposed method with EER = 1.15%.

## References

  * [1] Verifinger sdk.  https://www.neurotechnology.com/verifinger.html, 2000.  Neurotechnology.
  * [2] Raid Rafi Omar Al-Nima, Saba Qasim Hasan, and Sahar Esmail.  Exploiting the deep learning with fingerphotos to recognize people.  International Journal of Advance Science and Technology, 29(7):13035–13046, 2020.
  * [3] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross.  Towards better understanding of gradient-based attribution methods for deep neural networks.  arXiv preprint arXiv:1711.06104, 2017.
  * [4] Aman Attrish, Nagasai Bharat, Vijay Anand, and Vivek Kanhangad.  A contactless fingerprint recognition system.  arXiv preprint arXiv:2108.09048, 2021.
  * [5] Parmeshwar Birajadar, Meet Haria, Pranav Kulkarni, Shubham Gupta, Prasad Joshi, Brijesh Singh, and Vikram Gadre.  Towards smartphone-based touchless fingerprint recognition.  Sādhanā, 44(7):1–15, 2019.
  * [6] Lewis A Carney, Josiah Kane, Jonathan F Mather, Asem Othman, Andrew G Simpson, Aryana Tavanai, Richard A Tyson, and Yiqun Xue.  A multi-finger touchless fingerprinting system: mobile fingerphoto and legacy database interoperability.  In Proceedings of the 2017 4th international conference on biomedical and bioinformatics engineering, pages 139–147, 2017.
  * [7] Shaan Chopra, Aakarsh Malhotra, Mayank Vatsa, and Richa Singh.  Unconstrained fingerphoto database.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 517–525, 2018.
  * [8] Debayan Deb, Tarang Chugh, Joshua Engelsma, Kai Cao, Neeta Nain, Jake Kendall, and Anil K Jain.  Matching fingerphotos to slap fingerprint images.  arXiv preprint arXiv:1804.08122, 2018.
  * [9] Mohammad Omar Derawi, Bian Yang, and Christoph Busch.  Fingerprint recognition with embedded cameras on mobile phones.  In International conference on security and privacy in mobile information and communication systems, pages 136–147. Springer, 2011.
  * [10] Fengxiang He, Tongliang Liu, and Dacheng Tao.  Why resnet works? residuals generalize.  IEEE Transactions on Neural Networks and Learning Systems, 31(12):5349–5362, 2020.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] Christof Kauba, Dominik Söllinger, Simon Kirchgasser, Axel Weissenfeld, Gustavo Fernández Domínguez, Bernhard Strobl, and Andreas Uhl.  Towards using police officers’ business smartphones for contactless fingerprint acquisition and enabling fingerprint comparison against contact-based datasets.  Sensors, 21(7):2248, 2021.
  * [13] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [14] Chulhan Lee, Sanghoon Lee, Jaihie Kim, and Sung-Jae Kim.  Preprocessing of a fingerprint image captured with a mobile camera.  In International conference on biometrics, pages 348–355. Springer, 2006.
  * [15] Guoqiang Li, Bian Yang, R Raghavendra, and Christoph Busch.  Testing mobile phone camera based fingerprint recognition under real-life scenarios.  NISK, 1:2, 2012.
  * [16] Aakarsh Malhotra, Anush Sankaran, Mayank Vatsa, and Richa Singh.  On matching finger-selfies using deep scattering networks.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 2(4):350–362, 2020.
  * [17] Manuel Martinez and Rainer Stiefelhagen.  Taming the cross entropy loss.  In German Conference on Pattern Recognition, pages 628–637. Springer, 2018.
  * [18] Keiichiro Nakazaki, Naoto Miura, Yusuke Matsuda, and Akio Nagasaka.  Fingerphoto recognition using cross-reference-matching multi-layer features.  In 2022 IEEE International Conference on Consumer Electronics (ICCE), pages 1–6. IEEE, 2022.
  * [19] Jannis Priesnitz, Rolf Huesmann, Christian Rathgeb, Nicolas Buchmann, and Christoph Busch.  Mobile contactless fingerprint recognition: implementation, performance and usability aspects.  Sensors, 22(3):792, 2022.
  * [20] Ramachandra Raghavendra, Christoph Busch, and Bian Yang.  Scaling-robust fingerprint verification with smartphone camera in real-life scenarios.  In 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pages 1–8. IEEE, 2013.
  * [21] Kiran B Raja, Ramachandra Raghavendra, Vinay Krishna Vemuri, and Christoph Busch.  Smartphone based visible iris recognition using deep sparse filtering.  Pattern Recognition Letters, 57:33–42, 2015.
  * [22] Raghavendra Ramachandra, Martin Stokkenes, Amir Mohammadi, Sushma Venkatesh, Kiran Raja, Pankaj Wasnik, Eric Poiret, Sébastien Marcel, and Christoph Busch.  Smartphone multi-modal biometric authentication: Database and evaluation.  arXiv preprint arXiv:1912.02487, 2019.
  * [23] Ajita Rattani and Reza Derakhshani.  A survey of mobile face biometrics.  Computers & Electrical Engineering, 72:39–52, 2018.
  * [24] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ” why should i trust you?” explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * [25] Anush Sankaran, Aakarsh Malhotra, Apoorva Mittal, Mayank Vatsa, and Richa Singh.  On smartphone camera based fingerphoto authentication.  In 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), pages 1–7. IEEE, 2015.
  * [26] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [27] Renu Sharma and Arun Ross.  Periocular biometrics and its relevance to partially masked faces: A survey.  arXiv preprint arXiv:2203.15203, 2022.
  * [28] Kamlesh Tiwari and Phalguni Gupta.  A touch-less fingerphoto recognition system for mobile hand-held devices.  In 2015 International Conference on Biometrics (ICB), pages 151–156. IEEE, 2015.
  * [29] Pankaj Wasnik, Martin Stokkenes, Mihkal Dunfjeld, Raghavendra Ramachandra, Kiran Raja, and Christoph Busch.  Baseline evaluation of smartphone based finger-photo verification system: A preliminary study of technology readiness.  In Proc. Norwegian Inf. Security Conf., pages 1–10, 2018.
  * [30] Axel Weissenfeld, Reinhard Schmid, Bernhard Kohn, Bernhard Strobl, and Gustavo Fernández Domínguez.  Case study of the acquisition of contactless fingerprints in a real police setting.  In 2022 International Conference of the Biometrics Special Interest Group (BIOSIG), pages 1–5. IEEE, 2022.
  * [31] Peter Wild, Franz Daubner, Harald Penz, and Gustavo Fernández Domínguez.  Comparative test of smartphone finger photo vs. touch-based cross-sensor fingerprint recognition.  In 2019 7th International Workshop on Biometrics and Forensics (IWBF), pages 1–6. IEEE, 2019.
  * [32] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.


# Deep Reinforcement Learning-based Energy Efficiency Optimization For Flying
LoRa Gateways

Mohammed Jouhari1, Khalil Ibrahimi2, Jalel Ben Othman3 4 5 and El Mehdi
Amhoud1<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>1School of
Computer Science, Mohammed VI Polytechnic University, Ben Guerir, Morocco
2Ibn Tofail University, Faculty of Sciences, Laboratory of Research in
Informatics (LaRI), Kenitra, Morocco
3CNRS, CentraleSupélec, Laboratoire des signaux et systémes, Université Paris-
Saclay, Gif-sur-Yvette, France
4University of Sorbonne Paris North, Villetaneuse, France
5College of Technological Innovation, Zayed University, Abu Dhabi, UAE

###### Abstract

A resource-constrained unmanned aerial vehicle (UAV) can be used as a flying
LoRa gateway (GW) to move inside the target area for efficient data collection
and LoRa resource management. In this work, we propose deep reinforcement
learning (DRL) to optimize the energy efficiency (EE) in wireless LoRa
networks composed of LoRa end devices (EDs) and a flying GW to extend the
network lifetime. The trained DRL agent can efficiently allocate the spreading
factors (SFs) and transmission powers (TPs) to EDs while considering the air-
to-ground wireless link and the availability of SFs. In addition, we allow the
flying GW to adjust its optimal policy onboard and perform online resource
allocation. This is accomplished through retraining the DRL agent using
reduced action space. Simulation results demonstrate that our proposed DRL-
based online resource allocation scheme can achieve higher EE in LoRa networks
over three benchmark schemes.

###### Index Terms:

LoRaWAN, Deep Reinforcement Learning, Energy Efficiency.

##  I Introduction

Long-range (LoRa) is one of the most popular low-power wide-area-network
(LPWAN) protocols due to its easy deployment and flexible management as well
as its open protocol stack. As a physical layer technology, LoRa adopts chirp
spread spectrum (CSS) techniques to propagate narrowband signals over a
specific channel bandwidth [1]. The signal could therefore travel further
while consuming less power, enabling the connection of thousands of devices
with long battery lives. Based on different spreading factors (SFs), multiple
LoRa end devices (EDs) can simultaneously share the same time slot and
frequency channel. Hence, various signal-to-noise ratio (SNR) limits are
required for each used SF, resulting in different data rate transmission and
transmission range [2]. For efficient LoRa communication, a decision on LoRa
PHY layer parameters should be taken to select the adequate SF, transmission
power (TP), and channel frequency depending on the wireless channel condition
and the distance between the EDs and the gateway (GW). Thus an intelligent and
proactive PHY LoRa parameters adjustment is needed owing to the time-variance
of data traffic load, and channel condition [3].

The next generation of Internet of Things (IoT) networks is expected to
benefit significantly from this technology [4, 5]. However, this type of
system generally involves using a large number of resource-constrained EDs to
provide reliable data delivery to potentially critical applications, making
LoRa resource allocation a complicated task [6, 7]. Various works focused on
LoRa resource management problems under different LoRa network scenarios and
assumptions. To cope with concurrent transmission and interference, a
cooperative SF assignment scheme was designed in [8] for a multi-operator
LoRaWAN deployment scenario. Game theory and gradient ascent-based iterative
algorithms were used to solve the SF assignment optimization. Authors in [9]
suggested a hybrid LoRa-based low-power mesh network where EDs use both sub-
GHz long-range radio and 2.4 GHz short-range radio. Furthermore, customized
TDMA and ANT protocols were considered for efficient data collection in dense
LoRa wireless networks. In contrast, authors in [10] allocate LoRa PHY
parameters in time slots for data transmission to alleviate LoRaWAN
scalability issues by reducing collisions and grouping acknowledgments.

Moreover, deep reinforcement learning (DRL) was suggested in [11] to
efficiently manage the grid power consumption of GWs in hybrid LoRa Networks.
Authors in [12] combined centralized RL with distributed multi-agent RL to
improve throughput and reduce the energy consumption of EDs. The authors in
[13] suggested freezing some layers when retraining an artificial neural
network model on real data. This technique is intended to optimize the energy
efficiency of LoRa systems by assigning the best power transmission to each
device. In this work, we consider a LoRa network composed of EDs that sense
data from the target region, and flying GW forwards collected data toward a
LoRa server to establish the communication between the EDs and the server. The
main contributions of this work are summarized as follows:

  * •

We formulate the problem of SF and TP allocation as one optimization problem
to maximize the energy efficiency (EE) in the LoRa network considering EDs
position and air-to-ground (A2G) channel.

  * •

We adopt a DRL agent to solve the EE optimization problem using the proximal
policy optimization (PPO) algorithm to handle stochastic policies and cope
with the high computation requirement of traditional RL.

  * •

We design an efficient MDP environment with states, action space, and reward
function that is adequate to the considered LoRa system for a stable learning
of the DRL agent.

  * •

We enable online resource management for moving flying GW by allowing fast
policy adaptation onboard using an adequate action space reduction scheme
(ASR).

This paper is outlined as follows: Section II describes the system model and
the problem formulation. DRL-based resource allocation scheme is detailed in
Section III, while Section IV evaluates the performance of our proposed
solutions. Finally, the paper is concluded in Section. IV.

##  II System Model And Problem Formulation

In this work, we consider a LoRa network composed of a set of $\mathcal{N}$
EDs and a single unmanned aerial vehicle (UAV) equipped with a LoRa GW to
establish communication between EDs and the server (Fig. 1). The UAV flies at
a fixed altitude and changes its horizontal position periodically depending on
the requirements of the mission and the events occurring in the monitored
area. We characterize the A2G link by its path loss to associate the distance
between the EDs and the GW with the link quality and LoRa specifications. This
will help to identify the optimal TP required by the EDs to efficiently
communicate their collected data to the GW. The path loss is composed of two
elements $l^{los}(i,g)$ and $l^{nlos}(i,g)$ which are respectively the line-
of-sight (LoS) and non-line of sight (nLoS) path loss and are given as follow:

$l^{los}(i,g)=l^{fs}(d_{r})+10\beta_{los}\log_{10}d_{i,g}+\chi_{\sigma_{los}},$
(1)
$l^{nlos}(i,g)=l^{fs}(d_{r})+10\beta_{nlos}\log_{10}d_{i,g}+\chi_{\sigma_{nlos}},$
(2)

where $\beta$ is the path loss exponent. $\chi_{\sigma_{los}}$ and
$\chi_{\sigma_{nlos}}$ are the shadowing random variables which are,
respectively, characterized as the Gaussian random variables with zero mean
and $\sigma_{los}$, $\sigma_{nlos}$ standard deviations. $l^{fs}$ represent
the free-space path loss in the reference distance $d_{r}$ which is calculated
in dB by $l^{fs}(d_{r})=20\log_{10}(\frac{d_{r}f4\pi}{c})$, where $f$ is the
carrier frequency in Hz, $c$ is the light speed, and $d_{r}$ the reference
distance of the free space path loss. Since we consider communication between
an UAV and multiple EDs anchored in the ground we assume that we have
$d\geqslant c/f$ in such a way both antennas are far away from each other.
Then according to the actual location of the UAV especially its flying
altitude that causes occlusion, the LoS probability can be calculated as
follows:

$P^{los}(\hat{\theta})=\frac{1}{1+\alpha e^{-\lambda(\hat{\theta}-\alpha)}},$
(3)

where $\alpha$ and $\lambda$ are the Sigmoid function parameters, while
$\hat{\theta}$ refers to the elevation angle of the GW given by
$\mathmbox{\hat{\theta}=\sin^{-1}(h_{g}/d_{i,g})}$. The probability value of
nLoS can be obtained by $P^{nlos}(\hat{\theta})=1-P^{los}(\hat{\theta})$. As
results, the overall A2G path loss is calculated using both $l^{nlos}(i,g)$,
$l^{nlos}(i,g)$:

$l^{a2g}=P^{los}(\hat{\theta})l^{los}(i,g)+P^{nlos}(\hat{\theta})l^{nlos}(i,g).$
(4)

Figure 1: LoRa EDs transmit collected data to the flying GW using their
allocated SF and TP

The received signal strength indicator (RSSI) at the ED from the transmitting
UAV which is our GW allows us to determine the quality of the LoRa
communication link and to adjust the receiver sensitivity through tuning LoRa
physical parameters. This relationship is defined based on the distance of the
UAV and the nature of the communication environment that influences the direct
visibility between the ED and the UAV. As described previously, the A2G path
loss is composed of the LoS and nLoS parts related to the UAV position and
altitude. Thus, the A2G path loss can also be obtained by substituting the
RSSI value from the TP of the GW ($p^{tr}_{gw}$) as following

$l^{a2g}(d_{i,g},\hat{\theta})=p^{tr}_{gw}-RSSI.$ (5)

The RSSI value obtained from Eq. (5) in the path loss function and the TP of
the GW should be greater than the receiver sensitivity to correctly decode the
received information. In another way, the maximum distance where the received
signal is decodable is obtained by substituting the sensitivity $\zeta_{n,m}$
instead of the RSSI in Eq. (5). To link the network topology with LoRa PHY
parameters we consider the signal-to-noise ratio between the GW and the EDs
using the SF $m$ defined as
$\mathrm{\rho}^{a2g}_{n,m}=\frac{p_{n,m}}{10^{0.1l^{a2g}(d_{n},\hat{\theta})}.\sigma^{2}}$.
$p_{n,m}$ is the TP of EDs $n$ at SF $m$ while $\sigma^{2}$ refers to the
Gaussian noise power. Hence, the signal-to-interference- plus-noise ratio
$\Upsilon$ is given by:

$\mathrm{\Upsilon}_{n,m}=\frac{\mathrm{\rho}^{a2g}_{n,m}}{\sum\limits_{k\in\mathcal{N}_{-n}}\psi_{k,m}\mathrm{\rho}^{a2g}_{k,m}+1},$
(6)

Where $\mathcal{N}_{-n}=\mathcal{N}\backslash\\{n\\}$ and $\psi_{k,m}$ is a
binary variable that indicates whether the ED k uses the SF m from the set of
SF $\mathcal{M}$. The channel capacity between ED $n$ and the GW can be
calculated by $\Gamma_{n,m}=\mathrm{BW}\log_{2}(1+\mathrm{\Upsilon}_{n,m}),$
where BW is the bandwidth. We define the energy efficiency (EE) of the system
by:

$\xi_{EE}=\frac{\Gamma}{P_{T}+P_{c}},$ (7)

where $P_{c}$, $P_{T}=\sum\limits_{n=1}p_{n,m}$ and $\Gamma$ are respectively
the circuit consumption power, the TP and the sum rate of the system over all
the logical channels given as $\Gamma=\sum\limits_{n=1}\Gamma_{n,m}.$

Our objective is to maximize the EE of the LoRa network by allocating the best
SF and TP depending on the ED position and the A2G channel quality. For this
we formulate the following optimization problem: ψ_n,m ∈Ψp_n,m ∈Pξ_EE(P1)
0≤p_n,m≤p_max ψ_n,m ∈{0, 1} ∀n,m ∑_m=1ψ_n,m ≤1,∀n ∑_n=1ψ_n,m ≤ϱ_max,∀m ζ_n,m
≤RSSI_n,m,∀n,m, where $\Psi=\\{\psi_{n,m}|m\in\mathcal{M},n\in\mathcal{N}\\}$,
$P=\\{p_{n,m}|n\in\mathcal{N},m\in\mathcal{M}\\}$ denote the optimization
variable sets of EDs clustering based on SF assignment and the TP allocation.
The optimization problem (P1) is constrained by the lower and the upper bounds
of the TP available for each ED (1b) and the binary allocation variable
$\psi_{n,m}\in\\{0,1\\}$ (1c). Constraint (1d) implies that each ED can adopt
at most one SF. To cope with the limited available SF multiple EDs can be
associated with the same SF (1e) [2]. The constraint (1f) guarantees that the
received signal power is detectable at the ED.

##  III DRL-based resource allocation

The problem (P1) is known to be NP-hard, which makes the efficient allocation
of TP and SF more complicated and costly in terms of computation resources and
time. Moreover, solving this problem requires full knowledge of the network
topology and the assigned SF in the future time step to deal with co-SF
interference. This is not practical in a real-life scenario. DRL techniques
have shown excellent performance as a tool to solve such online problems known
by their complex and dynamic spaces. It takes advantage of the interaction
with the environmental parameters (e.g., A2G link budget, energy costs) to
learn the most efficient policy from the statistical distribution of these
features and the system’s current state using its insights of the future
learned through experiences. Hence the DRL technique seems to be an adequate
solution to solve problem (P1) while reducing its complexity by making
suboptimal decisions following the learned policy.

Previously, we described the optimization (P1) to efficiently maximize the
system’s energy efficiency while considering the A2G link budget, co-SF
interferences, and the network topology. The optimization is performed by
assigning the adequate SF and TP that maximize the energy efficiency.
Accordingly, we designed a DRL agent to solve the problem mentioned above. In
particular, at each time step of the resource allocation process, the DRL
agent decides on the SF and TP to allocate for each EDs regarding the system’s
current state. The DRL agent ultimately aims to maximize the channel
throughput by reducing co-SF interferences between EDs while reducing the TP
regarding the distance between the EDs and the UAV-based GW. During the
training process, the agent receives rewards for each allocation made until
convergence owing to the optimal policy. The convergence of the DRL agent is
achieved when its learning curve gets flat and stops increasing. To implement
the solution, we abstracted the problem into a Markov Decision Process (MDP)
framework of the four-tuple ($\mathcal{S}$, $\mathcal{A}$, $R$,
$\mathcal{P}_{r}$). Such that, $\mathcal{S}$ depicts the state set of the
system, $\mathcal{A}$ the set of actions, $R$ is the immediate reward that can
be received for each action, and $\mathcal{P}_{r}$ the probability of
transition.

Figure 2: DRL architecture

###  III-1 MDP environment

The DRL agent interacts with the LoRa network (as shown by Fig. 2) presented
as an MDP environment, designed to generate a reward $R_{t}$ for each agent
action $\mathcal{A}_{t}$ at time-step t that impacts the next state on the
system $\mathcal{S}_{t+1}$. An episode consists of a consecutive sequence of
some time steps. Throughout multiple episodes, the agent is trained by
experiencing various scenarios to learn the optimal policy from its previous
actions and the received rewards. At each time step, the agent decides on
assigning the TP and SF for single LoRa ED as described in the problem (P1).
Episodes are independent, meaning that the cumulative reward is initiated to 0
at the beginning of each episode. Moreover, a new network topology is
considered in each episode. Hence, the episode length corresponds to the
number of active LoRa EDs in the network, which should remain the same during
the learning process. Since the agent does not have an overall overview of the
environment, thus the policy is obtained through learning from its interaction
with the environment.

###  III-2 State and action spaces

We set the state space as
$\mathcal{S}=\\{AllocatedSF(t),AllocatedTP(t),DataRate(t)\\}$ where
$AllocatedSF(t)$ and $AllocatedTP(t)$ are respectively the matrices of
allocated SF and TP at time step $t$. $DataRate(t)$ is a matrix of the current
data rate between the EDs and the GW after assigning resources to the ED
corresponding to the time step $t$. The DRL agent takes actions from the
action space $\mathcal{A}_{t}$ based on the current state of the system and
its policy $\Pi$. This action consists of allocating a combination of the SF
and the TP which can be expressed as $\mathcal{A}_{t}=\\{m_{t},p_{t}\\}$ such
as $m_{t}\in\\{7,8,9,10,11,12\\}$ and $p_{t}\in\\{2,5,8,11,14\\}$. LoRa ED
relevant to the current step is served through the allocated resources once
the decision is made. As each step is completed, the allocated SF and TP
matrices and the data rate map are updated based on the current decision. The
new matrices are then used as input for the next step.

###  III-3 Immediate reward function R

We formulate the immediate reward function to match the objective of our
optimization problem, aiming to maximize the energy efficiency of all EDs in
the network while respecting the availability of SF and the TP to meet the
sensitivity requirement. Specifically, when the DRL agent observes the state
of the system $\mathcal{S}_{t}$ an action $\mathcal{A}_{t}$ should be taken
that meets the two constraints:

$\begin{cases}C1:&\sum\limits_{i=1}^{i=t}\mathcal{A}_{i}(0)\leq
6,\,\text{Constraint \, (19e)}\\\ C2:&\zeta(\mathcal{A}_{t}(0))\leq
RSSI,\,\text{Constraint \, (19f).}\end{cases}$ (8)

(C1) indicates that a SF can not be assigned to more than 6 EDs which depicts
the channel capacity for handling the co-SF interference and matches the
constraint (19e). (C2) means that the transmitted signal strength using the TP
$\mathcal{A}_{t}(1)$ should be more significant than the receiver sensitivity
using the SF $\mathcal{A}_{t}(0)$. Therefore, the immediate reward is
formulated as follows:

$R_{t}=\frac{\sum\limits_{n=1}\Gamma_{n,m}(t)}{P_{c}+\sum\limits_{n=1}p_{n,m}(t)}.$
(9)

A null reward is provided when only one constraint is not respected.
Otherwise, a positive reward calculated from Eq. (9) is provided to the agent,
which helps to find the optimal policy by maximizing the cumulative rewards
through consecutive time steps.

###  III-4 DRL algorithm

As described previously, we are working on the time-variant LoRa system with a
dimensional set of actions that depends on the available SF and the TP level.
Hence, using traditional RL methods and saving all experienced MDP tuples in a
table can be challenging and requires high computation capability. To cope
with this, we adopted the DRL technique based on DNN to solve our optimization
problem. The considered DRL characterized by its dynamic state space is
deployed using the proximal policy optimization (PPO) [14], known for its high
performance for stochastic policies which makes it an adequate solution for
our problem.

Algorithm 1 PPO algorithm training process

1:\- Randomly initialize $\theta$ and get $\Pi_{\theta}$

2:\- Establish a sampling policy $\Pi_{\theta^{*}}$ such as
$\theta^{*}\leftarrow\theta$

3:for each episode e do

4: $\mathcal{S}_{0}\leftarrow$ NewTopology

5: for $t\in\mathcal{N}$ do

6: take $\mathcal{A}_{t}$ based on the policy $\Pi_{\theta^{*}}$

7: if $\mathcal{A}_{t}$ respects (C1) and (C2) then

8: Update $\mathcal{S}_{t}$ using the allocated resources

9: end if

10: end for

11: if Resources allocated to all EDs then

12: Calculate the current reward $R_{t}$

13: Observe the next state $\mathcal{S}_{t+1}$ and $R_{t}$

14: Save $(\mathcal{S}_{t},\mathcal{A}_{t},R_{t},\mathcal{S}_{t+1})$ in D

15: end if

16: \- Compute the estimator
$\hat{E}_{t}=\sum_{i=0}^{\infty}(\gamma\chi)^{i}\delta_{t+i}$

17: \- Where $\delta_{t+i}=R_{t}+\gamma
V(\mathcal{S}_{t+1},\theta)-V(\mathcal{S}_{t},\theta)$

18: \- Select mini-batch sample
$(\mathcal{S}_{t},\mathcal{A}_{t},R_{t},\mathcal{S}_{t+1})$ from D

19: \- Update $\theta$ based on the sampled data and maximizing:

20:
$L^{CLIP}(\theta)=\mathbb{E}[p_{t}(\theta)\hat{E}_{t},clip(p_{t}(\theta),1-\epsilon,1+\epsilon)\hat{E}_{t}]$

21: \- Where $p_{t}(\theta)$ is calculated as
$p_{t}(\theta)=\frac{\Pi(\mathcal{A}_{t}\|\mathcal{S}_{t},\theta)}{\Pi(\mathcal{A}_{t}\|\mathcal{S}_{t},\theta^{*})}$

22: \- Update DNN parameters: $\theta^{*}\leftarrow\theta$

23:end for

The training process of the PPO algorithm to reach convergence is described in
algorithm 1, including multiple steps with respect to our MDP model. We start
the training process by initializing the DNN weights to get initial PPO
policies used to generate samples for different episodes using $\theta^{*}$.
At each episode, a new network topology is considered, which consists of the
new position of EDs in the target area (line 4). An action $\mathcal{A}_{t}$
is taken based on the current policy $\Pi_{\theta^{*}}$. If the action
$\mathcal{A}_{t}$ respects the constraints (C1) and (C2), the resources will
be assigned to the concerned EDs, and the system state will be updated (lines
5-10). D is a replay memory where these samples are stored (lines 12-14). At
the end of each episode, the policy gradient estimator $\hat{E}_{t}$ is
calculated in lines 16-17. In line 18, samples are extracted from a random
mini-batch from the experience memory D. The parameter $\chi$ is used to
regulate the bias-variance trade-off. In contrast, $V(\mathcal{S}_{t},\theta)$
is the state value describing the expected return of the system in state t
using the parameters $\theta$. The objective function $L^{CLIP}(\theta)$ (line
20) is maximized for each batch sample to update the main policy
$\Pi_{\theta}$. The main goal of the clip function is to keep the policy
probability ratio in the range of (1-$\epsilon$) and (1+$\epsilon$). This
mechanism used to learn from past experiences, known as experience replay, is
vital to converge and stabilize learning over time. Finally, both policies are
synchronized by replacing the oldest with the new one.

(a)

(b)

(c)

Figure 3: Performance of DRL agent: (a) Training rewards for each network
density (b) EE comparison with benchmarking schemes (c) Probability
distribution of actions.

##  IV Performance Evaluation

In this section, we simulate our proposed model to evaluate its performances
and compare it to the random, distance-based, and genetic algorithms [15]. The
distance-based scheme allocates SF and TP based on the distance of EDs from
the flying GW. The EDs positions are generated randomly among the target area
of $2\,km^{2}$ at each episode in the training process. The flying GW is a UAV
located at the center of this area with an altitude of 0.3 km and is used to
collect data from active EDs using LoRa protocol. Parameters of the wireless
link and the DRL agent used in this simulation to evaluate our optimization
are fixed as follows, $\gamma=0.99$, $\epsilon=0.1$, $\chi=0.01$. Also, we
have considered the path loss exponent $\beta_{los}=2$ and $\beta_{nlos}=2.5$,
the Gaussian random variable $\sigma_{los}=5$ and $\sigma_{nlos}=20$, f = 868
GHz, BW = 125 kHz.

Fig. 3 (a) shows the convergence of the average cumulative reward of the DRL
agent in the training process versus the number of episodes. The agent is
trained through experiencing different network topologies of fixed UAV and
variable EDs positions for multiple consecutive episodes to learn the optimal
policy. As shown from the figure, the agent reached the convergence after 1200
episodes for different network densities and learned the best policy to
allocate SF and TP while considering the system’s energy efficiency and the
A2G LoRa wireless link. Moreover, networks with low density obtain low rewards
in the training process, which does not mean achieving low energy efficiency;
it depends on the agent learning of the relation between variable EDs
positions and the fixed flying GW. Fig. 3 (b) illustrates the performance
comparison of our proposed energy efficiency optimization solution using a DRL
agent versus traditional algorithms such as the random resource allocation and
distance based where the higher SF are assigned to further EDs. On the other
hand, the key idea of the genetic algorithm is to assign different SFs and TP
to different EDs such that constraints (C1) and (C2) are respected. This
algorithm is configured with a population size of 200, elite size of 20 and
mutation rate of 0.6, and 5000 generations. The proposed DRL-based
optimization solution outperforms the considered traditional scheme in terms
of energy efficiency due to the power of reinforcement learning in handling
the considered LoRa system, including the A2G wireless link, EDs position, the
available SFs, and the dimensional action space. In Fig. 3 (c), we plot the
probability distribution of actions for network densities of 6, and 8, knowing
that as defined previously an action is a tuple of (SF, TP). In Fig. 3 (c), we
plotted only the actions with a probability higher than 0.001. This will help
us to identify the most taken actions by the agent or the preferred actions
and reduce the action space dimension for future use, as we will show after.
As it can be noticed from the figure, all the SF can be assigned to the EDs
with different probabilities depending on their availability while respecting
the constraint (C1) and the distance of EDs from the GW. However, due to the
energy efficiency purpose, the agent favored the low TP level. Consequently,
TP of 11 dBm and 14 dBm were not taken despite being available in the action
space.

(a) N = 6

(b) N = 8

Figure 4: Training results of DRL on the new configuration

We consider a new scenario where the flying GW moves from its initial position
in the center of the target area to a new position in the corner. In Fig. 4,
we show the outcome of the training process using different network
configurations where the flying GW moves from its initial position 0 to the
new position 1 using the pre-trained DRL0 agent, which is trained on the
network configuration 0 to infer in the network configuration 1 results in
performance reduction of the system in terms of energy efficiency. To cope
with this, a typical solution can be used by training a new agent using the
new configuration, as shown by the green curve in Fig. 4 (a). This process is
generally performed offline, which requires time and computation resources to
ensure stable system training and convergence. However, time and computation
resources may not be available for flying GW to deal with the time-variance of
event occurrence in the target area. Thus we suggest continuing the training
of the pre-trained DRL0 agent (blue curve Fig. 4 (a)) to reduce both the
convergence time and the required computation resources to allow an online
policy adjustment onboard the UAV through a small training process. In
addition, we used the results of Fig. 3 (c) to reduce the action space of the
agent by eliminating the actions of low probability, which helped further
reduce the convergence time of the retraining process in the cost of the
energy efficiency (red curve Fig. 4 (a)). The experience is performed for
networks composed of 6 and 8 active EDs (Fig. 4 (b)) to show further the
system’s behavior against the action space reductions.

TABLE I: Results summary of our proposed DRL-based energy efficiency optimization Active EDs | Scenario 0 | Scenario 1 | Scenario 1 | Scenario 1 | Scenario 1
---|---|---|---|---|---
Trained DRL0 | Pretrained DRL0 | Retrained DRL0 | Retrained DRL0 with ASR | Trained DRL1
EE | Accuracy | EE | Accuracy | EE | Accuracy | EE | Accuracy | EE | Accuracy
6 | 5.14 | 100 | 5.05 | 100 | 5.26 | 100 | 5.12 | 99.7 | 5.07 | 100
8 | 5.35 | 99.8 | 4.95 | 91.1 | 5.49 | 100 | 5.19 | 98.1 | 5.32 | 100
10 | 5.82 | 99.9 | 5.66 | 87.9 | 5.85 | 100 | 5.73 | 94.4 | 5.79 | 98.83
12 | 6.02 | 88.3 | 5.77 | 79.5 | 5.87 | 95.5 | 5.72 | 77.9 | 6.19 | 97.29
14 | 6.1 | 93.7 | 5.15 | 77.7 | 5.92 | 99.1 | 5.95 | 84.7 | 6.23 | 94.23

Table. I summarizes the inference results of our proposed DRL-based energy
efficiency optimization solution to deal with moving flying LoRa GW. The
second column shows the test results of DRL0 trained and tested using the
network where the GW is located in the center of the area with different EDs
positions. Energy efficiency and accuracy are both considered as performance
metrics to evaluate the proposed solutions. EE depicts the average energy
efficiency of all active EDs in the network, and the accuracy is the
percentage of tests where the agent satisfied both constraints (C1) and (C2).
Obtained results show that networks with lower density achieve high accuracy
due to the max SF reuse in constraint (C1), which is easy to respect in
networks with low density. Although the EE increases with the density due to
the short distance between the EDs and the GW, which requires low TP to reach
the destination conducting to higher energy efficiency. This trend is almost
followed for any network configuration. The system performance degrades when
using the pre-trained DRL0 to allocate resources in network configuration 1.
This is a normal result since the agent is only trained to learn to allocate
resources for fixed GW and moving EDs. As shown in Table I, retraining the
DRL0 improves its network performance with the new GW position. However, to
reduce the convergence time of the DRL0 training on the new network
configuration, we used the ASR method. The obtained results improve those of
DRL0 while being lower than retrained DRL0 due to the set of actions removed
from the action space which conduct the agent to take inadequate action for
the energy efficiency of the system. The last column of the table shows that
training a new DRL agent does not provide good results in terms of EE owing to
the important convergence time required for the training process and the
computation resource required for that which is not available in flying GW.

##  V Conclusion

In this work, we proposed a DRL-based resource allocation to maximize the
energy efficiency in the LoRa network composed of EDs deployed randomly in the
target area and a single flying GW. This latter manages LoRa resources to
avoid interference between EDs and allows efficient data transmission.
Furthermore, the flying GW can change its position following the events
occurring in the target area to increase the throughput of EDs in this region,
reducing communication latency and improving the system’s energy efficiency.
In this work, we suggested using the PPO algorithm to solve the energy
efficiency optimization by assigning SF and TP to EDs in the network while
considering constraints on co-SF interference and the A2G LoRa wireless link.
The obtained results outperformed the existing ones in terms of energy
efficiency due to PPO’s ability to deal with stochastic behavior and
dimensionality. In addition, to enable a real-time decision of flying GW when
changing its position, we proposed retraining the pre-trained DRL agent using
ASR. In future works, we look forward to incorporating simultaneously two DRL
agents, for LoRa resource allocation and for data collection optimization.

## Acknowledgement

This work was sponsored by the Junior Faculty Development program under the
UM6P – EPFL Excellence in Africa Initiative, and partly supported by ASPIRE,
the technology program management pillar of Abu Dhabi’s Advanced Technology
Research Council (ATRC), via the ASPIRE Visiting International Professorship
program. Project ASPIRE/ZU R22036 EU2105.

## References

  * [1] P. A. Frangoudis, C. Tsigkanos, and S. Dustdar, “Connectivity Technology Selection and Deployment Strategies for IoT Service Provision Over LPWAN,” IEEE Internet Computing, vol. 25, no. 1, 2021.
  * [2] B. Su, Z. Qin, and Q. Ni, “Energy Efficient Resource Allocation for Uplink LoRa Networks,” in IEEE Global Communications Conference (GLOBECOM), pp. 1–7, 2018.
  * [3] A. Furtado, J. Pacheco, and R. Oliveira, “PHY/MAC Uplink Performance of LoRa Class A Networks,” IEEE Internet of Things Journal, vol. 7, no. 7, pp. 6528–6538, 2020.
  * [4] Y. Etiabi, E. M. Amhoud, and E. Sabir, “Huber estimator and statistical bootstrap based light-weight localization for IoT systems,” in International Symposium on Ubiquitous Networking, pp. 79–92, Springer, 2021\.
  * [5] Y. Etiabi, M. Jouhari, and E. M. Amhoud, “Spreading Factor and RSSI for Localization in LoRa Networks: A Deep Reinforcement Learning Approach,” arXiv preprint arXiv:2205.11428, 2022.
  * [6] M. Jouhari, N. Saeed, M.-S. Alouini, and E. M. Amhoud, “A Survey on Scalable LoRaWAN for Massive IoT: Recent Advances, Potentials, and Challenges,” IEEE Communications Surveys & Tutorials, pp. 1–1, 2023.
  * [7] M. Jouhari, K. Ibrahimi, and M. Benattou, “Implementation of bit error rate model of 16-qam in aqua-sim simulator for underwater sensor networks,” in Advances in Ubiquitous Networking 2: Proceedings of the UNet’16 2, pp. 123–134, Springer, 2017.
  * [8] H. Fawaz, K. Khawam, S. Lahoud, S. Martin, and M. E. Helou, “Cooperation for Spreading Factor Assignment in a Multioperator LoRaWAN Deployment,” IEEE Internet of Things Journal, vol. 8, no. 7, 2021.
  * [9] X. Jiang, H. Zhang, E. A. Barsallo Yi, N. Raghunathan, C. Mousoulis, S. Chaterji, D. Peroulis, A. Shakouri, and S. Bagchi, “Hybrid Low-Power Wide-Area Mesh Network for IoT Applications,” IEEE Internet of Things Journal, vol. 8, no. 2, pp. 901–915, 2021.
  * [10] K. Q. Abdelfadeel, D. Zorbas, V. Cionca, and D. Pesch, “ $FREE$ —Fine-Grained Scheduling for Reliable and Energy-Efficient Data Collection in LoRaWAN,” IEEE Internet of Things Journal, vol. 7, no. 1, pp. 669–683, 2020.
  * [11] R. Hamdi, E. Baccour, A. Erbad, M. Qaraqe, and M. Hamdi, “LoRa-RL: Deep Reinforcement Learning for Resource Management in Hybrid Energy LoRa Wireless Networks,” IEEE Internet of Things Journal, vol. 9, no. 9, pp. 6458–6476, 2022.
  * [12] G. Park, W. Lee, and I. Joe, “Network Resource Optimization with Reinforcement Learning for Low Power Wide Area Networks,” EURASIP J. Wirel. Commun. Netw., vol. 2020, sep 2020.
  * [13] L.-T. Tu, A. Bradai, O. B. Ahmed, S. Garg, Y. Pousset, and G. Kaddoum, “Energy Efficiency Optimization in LoRa Networks—A Deep Learning Approach,” IEEE Transactions on Intelligent Transportation Systems, pp. 1–13, 2022.
  * [14] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017\.
  * [15] B. Reynders, W. Meert, and S. Pollin, “Power and spreading factor control in low power wide area networks,” in 2017 IEEE International Conference on Communications (ICC), pp. 1–6, IEEE, 2017.