# A Deep Learning-based Pest Insect Monitoring System for Ultra-low Power
Pocket-sized Drones

Luca Crupi1, Luca Butera1, Alberto Ferrante1, and Daniele Palossi12 1Luca
Crupi, Luca Butera, Alberto Ferrante, and Daniele Palossi are with the Dalle
Molle Institute for Artificial Intelligence (IDSIA), USI and SUPSI, Lugano,
6962, Switzerland<EMAIL_ADDRESS>Palossi is also with the
Integrated Systems Laboratory (IIS), ETH Zürich, Zürich, 8092, Switzerland.

###### Abstract

Smart farming and precision agriculture represent game-changer technologies
for efficient and sustainable agribusiness. Miniaturized palm-sized drones can
act as flexible smart sensors inspecting crops, looking for early signs of
potential pest outbreaking. However, achieving such an ambitious goal requires
hardware-software codesign to develop accurate deep learning (DL) detection
models while keeping memory and computational needs under an ultra-tight
budget, i.e., a few $\text{\,}\mathrm{MB}$ on-chip memory and a few 100s
$\text{\,}\mathrm{mW}$ power envelope. This work presents a novel vertically
integrated solution featuring two ultra-low power System-on-Chips (SoCs),
i.e., the dual-core STM32H74 and a multi-core GWT GAP9, running two State-of-
the-Art DL models for detecting the Popillia japonica bug. We fine-tune both
models for our image-based detection task, quantize them in 8-bit integers,
and deploy them on the two SoCs. On the STM32H74, we deploy a FOMO-MobileNetV2
model, achieving a mean average precision (mAP) of 0.66 and running at
$16.1\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$
within $498\text{\,}\mathrm{mW}$. While on the GAP9 SoC, we deploy a more
complex SSDLite-MobileNetV3, which scores an mAP of 0.79 and peaks at
$6.8\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$
within $33\text{\,}\mathrm{mW}$. Compared to a top-notch RetinaNet-
ResNet101-FPN full-precision baseline, which requires 14.9$\times$ more memory
and 300$\times$ more operations per inference, our best model drops only 15%
in mAP, paving the way toward autonomous palm-sized drones capable of
lightweight and precise pest detection.

##  I Introduction

A crucial aspect of any modern farming activity is a precise and timely
intervention in the case of pest (insect) infestations to minimize the
production/economic damage and the environmental impact of the required
treatments [1]. For example, early identification of harmful bugs can lead to
ad-hoc treatments, such as spraying only part of the cultivated field or a few
plants, up to highly accurate treatments of only part of single trees/plants.
Conversely, traditional mass-scale farming productions have adopted coarse-
grained strategies, spraying chemicals in the entire cultivated field, even in
isolated or partial infestations. An important step forward in the accuracy of
treatments has been possible by disseminating traps in the cultivated area to
monitor the presence of specific pests, such as Cydia pomonella [2] or
Popillia japonica [3]. However, early approaches required costly and time-
consuming expert human operators’ intervention to manually inspect the traps
and collect information about the condition of the crop [4].

In the last decade, thanks to the advent of Internet-of-Things (IoT)
technologies, the diffusion of embedded ultra-low power Systems-on-Chips
(SoCs) and sensors’ miniaturization, trap-based precision agriculture has
become smarter by automating the monitoring procedure with embedded devices
integrated on the traps [5]. These battery-powered systems are usually
composed of an image sensor (e.g., color or infrared camera), a low-power
microcontroller unit (MCU), and a radio (e.g., WiFi, GSM, 4/5G) to stream data
(e.g., images) to a power-unconstrained remote server for the analysis.
Despite the paramount improvement compared to human-operated traps, these
State-of-the-Art (SoA) solutions still require costly external infrastructure,
i.e., mainframes and servers, and need high-throughput radio connectivity with
the additional disadvantage of draining small-capacity batteries typically
available on the traps [5].

Figure 1: Use case: a pocket-sized nano-drone inspecting a plant relying only
on onboard sensing and computational capabilities.  TABLE I: Network survey
(number of parameters, operations, throughput, and mAP) on the models used in
[6] and extended with two additional ones introduced in our work (in bold).
The GPU employed is an Nvidia GeForce RTX 2080.

Network | Input size | # Param. [$\text{\,}\mathrm{M}$] | # Op. [$\text{\,}\mathrm{G}\mathrm{M}\mathrm{A}\mathrm{C}$] | Device | Frame-rate [$\text{\,}\mathrm{Hz}$] | mAP
---|---|---|---|---|---|---
FasterRCNN-VGG16-FPN | $800\times 800\times 3$ | 31.90 | 275.23 | GPU | 11.90 | 0.92
FasterRCNN-ResNet101-FPN | $800\times 800\times 3$ | 60.20 | 167.41 | GPU | 11.23 | 0.92
FasterRCNN-DenseNet169-FPN | $800\times 800\times 3$ | 30.00 | 73.99 | GPU | 7.59 | 0.91
FasterRCNN-MobileNetV3-FPN | $800\times 800\times 3$ | 18.90 | 18.41 | GPU | 60.92 | 0.93
RetinaNet-VGG16-FPN | $800\times 800\times 3$ | 22.90 | 270.06 | GPU | 12.37 | 0.91
RetinaNet-ResNet101-FPN | $800\times 800\times 3$ | 51.20 | 174.85 | GPU | 11.83 | 0.93
RetinaNet-DenseNet169-FPN | $800\times 800\times 3$ | 21.00 | 65.10 | GPU | 18.49 | 0.92
RetinaNet-MobileNetV3-FPN | $800\times 800\times 3$ | 10.60 | 11.09 | GPU | 48.91 | 0.91
SSD-VGG16 | $300\times 300\times 3$ | 12.10 | 26.58 | GPU | 42.58 | 0.56
SSD-ResNet101 | $300\times 300\times 3$ | 32.30 | 42.39 | GPU | 29.76 | 0.87
SSD-DenseNet169 | $300\times 300\times 3$ | 11.80 | 24.51 | GPU | 28.59 | 0.92
SSD-MobileNetV3 | $300\times 300\times 3$ | 7.50 | 3.42 | GPU | 33.20 | 0.80
SSDLite-MobileNetV3 | 320$\times$240$\times$3 | 3.44 | 0.58 | GAP9 | 6.8 | 0.80
FOMO-MobileNetV2 | 96$\times$96$\times$3 | 0.02 | 0.01 | STM32 | 16.1 | 0.66

In this context, our work provides a vertically integrated system for accurate
pest detection employing ultra-constrained embedded MCUs, i.e., within a few
100s $\text{\,}\mathrm{mW}$ power envelope and a few $\text{\,}\mathrm{MB}$
on-chip memory, and cheap low-resolution cameras. Leveraging SoA deep learning
models for detecting Popillia japonica bugs, we present a novel hardware-
software co-design that is an ideal fit for both traditional traps and aboard
miniaturized palm-sized nano-drones, which can autonomously and dynamically
inspect crops, as drafted in Figure 1. Employing nano-drones, such as
miniaturized blimps [7] in greenhouses or autonomous quadrotors [8], for this
type of monitoring and detection activities has the additional advantage of
flexibility. For example, these tiny (less than $10\text{\,}\mathrm{cm}$ in
diameter) robotic platforms can easily reach locations where traditional traps
are not deployable or unreachable by bulky robotic arms, e.g., attached to
trucks and tractors.

Therefore, to achieve our goal, we must face both the challenges posed by
hardware-constrained devices and develop robust deep-learning models capable
of distinguishing between small dangerous bugs and harmless species (sometimes
only a few pixels in blurry images). For example, the Popillia japonica we
address in this work has similar visual features to the innocuous Cetonia
aurata and Phyllopertha horticola, shown in Figure 6.

On the one hand, we select two embedded devices, both compatible with the tiny
power envelope of battery-powered traps and nano-drones: the Arduino Portenta
H7 (STM32H74 dual-core MCU) and the Greenwaves Technologies (GWT) multi-core
GAP9 SoC. On the other hand, we survey the field of deep learning models for
pest detection, and we select two SoA models, a powerful SSDLite-MobileNetV3
[9] and a lighter FOMO-MobileNetV2 model [10]. The former is a top-scoring
convolutional neural network (CNN), which can run on the GAP9 SoC, while the
latter is more suitable for the STM32H74 MCU. In detail, our contributions can
be summarized as follows:

  * •

We start from models pre-trained on the COCO dataset for object detection
[11], and we then perform an additional fine-tuning stage with a custom
dataset ($3\text{\,}\mathrm{k}$ images) for the Popillia japonica detection.

  * •

We quantize, implement, and deploy both models on the two SoCs. On the
STM32H74, we execute and profile the FOMO-MobileNetV2 in float32 full-
precision and its quantized 8-bit version. On the GAP9, instead, we deploy the
SSDLite-MobileNetV3 in float16 and int8 (quantized), comparing the performance
between the general-purpose multi-core domain of the GAP9 and its
convolutional hardware accelerator (called NE16).

  * •

Finally, we perform a thorough assessment of both classification performances
and on-device inference, including power measurements on the two target
boards.

Our results highlight how both CNNs can achieve a top-notch mean average
precision (mAP) of 0.66 for the FOMO-MobileNetV2 and 0.79 for the SSDLite-
MobileNetV3, with minimal loss w.r.t. SoA solutions [6] (i.e., minimum loss of
0.01 of mAP) even in the case of strong 8-bit quantization. Furthermore, the
FOMO-MobileNetV2 running on the Portenta peaks with a throughput of
$17.5\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$@$494\text{\,}\mathrm{mW}$,
while the SSDLite-MobileNetV3 deployed on the GAP9 board achieves
$4\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$@$31.4\text{\,}\mathrm{mW}$
and
$6.8\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$@$33\text{\,}\mathrm{mW}$,
running on the multi-core cluster and the NE16, respectively. Therefore, the
proposed systems represent a viable option for long-lasting smart traps (up to
283 days with a
$1000\text{\,}\mathrm{mA}\text{\,}\mathrm{h}$@$3.7\text{\,}\mathrm{V}$
battery) and aboard nano-drones.

##  II Related work

Insect detection systems, using images and deep learning models, mostly
leverage SoA object detectors, as surveyed in [12]. For example, [13]
leverages a FasterRCNN-based detector [14], while [15] uses a Single Shot
Multibox Detector (SSD) [16], [17] uses RetinaNet [18], and [19] employs R-FCN
[20], for detecting small insects. Table I reports the models studied in [6]
extended by two smaller MobileNet-based models [10, 9] employed in our work
(last two bold lines). This broad comparison considers the size of the input
image, the number of parameters, the computational cost in MAC operations, the
device used for testing the model, and the detection mAP with the same testing
dataset used in our work. Even though most networks perform remarkably well in
mAP, i.e., up to 0.933, they require desktop-class devices to achieve a real-
time throughput (max
$61\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$).
This power-hungry class of computational devices is clearly unsuitable for IoT
battery-powered smart traps or nano-drones due to their power requirement
($\sim$$100\text{\,}\mathrm{W}$) and weight ($\sim$$1\text{\,}\mathrm{kg}$).

As for other IoT applications, images are often collected using camera-
equipped IoT nodes, but the actual pest detection algorithms run in the cloud
[21]. Nodes deployed in the fields need to be cheap - and, thus, extremely
limited in computational resources and memory - and have limited energy
requirements. These nodes are often powered by batteries that can be recharged
through small solar panels. This solution greatly limits the monitoring
frequency of the insects, as it would imply the frequent transmission of
relatively large images (e.g., 640$\times$480 pixels images [5]) over limited-
bandwidth networks, with its associated energy usage. For example, in the
iSCOUT111https://metos.global/en/iscout/ insect monitoring system, insect
count can run, at most, three times a day. This choice of processing images in
the cloud is often motivated by reasons other than technical (e.g., the
collection of images for improving the models), even in applications where
edge devices would have sufficient computational resources and energy for
running the models.

Instead, considering autonomous detection systems without any need for remote
computation and power-hungry communication streams, recent works allowed the
deployment of object detection algorithms on the edge [22, 23, 24]. Squeezed
edge YOLO [22] is a network that performs object detection within less than 1
million parameters and is deployable on very limited parallel ultra-low power
SoC. [23] introduces a Viola-Jones-based algorithm for detecting the Cydia
pomonella insect, running a battery-powered embedded system mounted on
conventional traps. Despite the simplicity of this approach, using pre-trained
patches on a reduced amount of data to detect insects, it peaks at
$2.5\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$
a GWT GAP8 SoC. Moreover, the approach is not suitable for solving our task
since we do not want only to detect but also to classify and discriminate
different insects. [25] explores a CNN-based classification system that relies
on a YOLO network to classify insects on drone images. Although accurate (up
to 0.92 of mAP), the method struggles to classify clusters of insects
correctly due to the suppression method that excludes a priori detections with
high intersection over union.

Finally, [24] presents an SSD-MobileNetV2 object detection algorithm fully
deployed on the GAP8 SoC running aboard an autonomous nano-drone to detect tin
cans and bottles. This system reaches up to
$1.6\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$
with an mAP of 0.5. Similarly, in [26], the authors employ the GAP8 SoC for an
automatic license plate recognition system using a MoileNetV2 with an SSDLite
detector. Similarly, also our CNNs are based on MobineNets [27], and for the
one we deploy on the GAP9 SoC, we adopt an SSDLite head achieving a peak
throughput of
$6.8\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$.
Ultimately, our work allows precise insect detection and classification tasks
on computationally constrained devices that can be used in smart battery-
powered IoT traps or aboard autonomous nano-drones.

##  III System implementation

In this section, we discuss our two system designs featuring two ultra-low
power embedded devices, i.e., a widely used dual-core Arduino Portenta H7 and
a novel multi-core GWT GAP9 SoC featuring the NE16 convolutional hardware
accelerator. Then, given these two devices’ significant memory and computing
power differences, we explore two alternative SoA CNNs for pest detection: a
lightweight FOMO-MobileNetV2 [10], which we deploy on the Portenta, and a more
complex SSDLite-MobileNetV3 [10, 9] for the more capable GAP9 SoC – almost
100$\times$ more MAC operations per inference than the FOMO-MobileNetV2.

###  III-A Platforms

Figure 2: Arduino Portenta H7 block diagram (A) and picture of the board (B).
Figure 3: GAP9 evaluation kit block diagram (A) and picture of the board (B).

The detection systems we propose are based on two widely different platforms:
the Arduino Portenta H7 board and the GAP9 evaluation kit. The Arduino
Portenta board is extended with the Portenta vision shield, which provides a
grayscale, ultra-low power Himax HM-01B0 camera with a maximum resolution of
320$\times$320 pixels, which is also available on the GAP9 development kit.
The main differences between the two boards arise when comparing their
computational resources. The Arduino Portenta, depicted in Figure 2, features
an STM32H747 SoC with two cores: a Cortex®M4, running at a
$240\text{\,}\mathrm{MHz}$, and a Cortex®M7, running at
$480\text{\,}\mathrm{MHz}$. The two cores can communicate via a remote
procedure call mechanism. Both cores have a double-precision floating point
unit (FPU) that allows the computations to be performed directly in floating
point arithmetic. The Portenta board features two off-chip memories, a
$16\text{\,}\mathrm{MB}$ FLASH and a $8\text{\,}\mathrm{MB}$ RAM. The
STM32H747’s memory hierarchy instead is composed by a $2\text{\,}\mathrm{MB}$
FLASH, a $1\text{\,}\mathrm{MB}$ L2 memory, and a $16\text{\,}\mathrm{kB}$ L1
memory.

TABLE II: Comparison between the STM32H747 and the GAP9.

|  STM32H747 |  GAP9
---|---|---
General-purpose cores |  2 |  10
Frequencies [$\text{\,}\mathrm{MHz}$] |  480 |  370
CNN accelerators |  None |  NE16
FPU availability |  Yes |  Yes
On-chip L1 memory [$\text{\,}\mathrm{kB}$] |  16 |  128
On-chip L2 memory [$\text{\,}\mathrm{MB}$] |  1 |  1.6
On-chip FLASH [$\text{\,}\mathrm{MB}$] |  2 |  2
Off-chip RAM [$\text{\,}\mathrm{MB}$] |  8 |  32
Off-chip FLASH [$\text{\,}\mathrm{MB}$] |  16 |  64

The GAP9 evaluation kit, depicted in Figure 3, features a GAP9 ultra-low power
SoC that relies on ten RISC-V cores equipped with single-precision FPUs. This
SoC is characterized by a power domain featuring one core, called fabric
controller, which orchestrates the work for the multi-core cluster and acts as
the interface with the external peripherals. The cluster, instead, is meant
for general-purpose intense parallel workloads, such as CNNs. This powerful
processor can deliver up to
$15.6\text{\,}\mathrm{G}\mathrm{O}\mathrm{p}\mathrm{/}\mathrm{s}$ and has an
on-chip L2 memory of $1.6\text{\,}\mathrm{MB}$. The off-chip memories
available on the GAP9 development kit (board) are $32\text{\,}\mathrm{MB}$ RAM
and $64\text{\,}\mathrm{MB}$ FLASH. Furthermore, the GAP9 SoC features the
NE16, an accelerator specifically tailored for linear algebra computations
that can consequently accelerate CNN operations but can be used only with int8
arithmetic. It can reach up to 150 8-bit MAC operations per second. Table II
summarizes the two boards’ specifications.

###  III-B Neural networks

Figure 4: SSDLite with MobileNetV3 backbone architecture. Figure 5: FOMO with
MobileNetV2 backbone architecture.

To address the insect detection and classification task, we use two CNNs: one
devoted to the Arduino Portenta H7 and one for the GAP9 development kit. This
choice is due to their differences, as highlighted in Section III-A. For this
work, we fine-tune, test, and deploy two architectures based on MobileNet [10,
9] and pre-trained on the COCO dataset [11]. Fine-tuning is a common transfer-
learning [28] technique that takes a model trained for a task, e.g., image
classification, on a specific domain, e.g., the COCO challenge [11] and uses
it for the same task but for a new domain exploiting the high-level knowledge
obtained in the pre-trained network.

The architecture of choice for the Arduino Portenta is a FOMO MobileNet V2
[10] with a width multiplier of 0.35. The network’s backbone is a MobileNetV2,
truncated after the 6th expands ReLU layer, with an output feature map as big
as 1/8th w.r.t to the input image. This feature map is then used by the faster
object more object (FOMO) detector to extract the objects of interest with
their object centroids, dimensions, and classes. The peculiarity of this
detector is that it relies on a fully convolutional neural network that is
trained on objects’ centroids instead of bounding boxes and is designed
specifically for constrained edge devices.

In the case of the FOMO MobileNetV2 network, we fine-tune four different
versions that change in the input type: 160$\times$160 RGB, 160$\times$160
grayscale, 96$\times$96 RGB, and 96$\times$96 grayscale images. The number of
MACs changes depending on the input and is 16.33, 14.49, 5.88, and
$5.21\text{\,}\mathrm{M}\mathrm{}\mathrm{M}\mathrm{A}\mathrm{C}$,
respectively. Also, the network parameters change depending on the number of
channels in the input, i.e., one for grayscale images and three for RGB ones,
which result in 20532 and 20820 parameters, respectively. The fine-tuning of
these networks is done using a stochastic gradient descent (SGD) optimization
algorithm, running for 100 epochs with a learning rate of 0.001.

Conversely, the architecture chosen for the GAP9 is the MobileNet V3 with an
SSDLite detector, which is a modification of the MobileNetV3 with SSD,
suggested in [6] for deployment on edge devices. In particular, the backbone
of the network is a MobileNet V3 [9] architecture that has
$3.44\text{\,}\mathrm{M}$ parameters and requires
$584\text{\,}\mathrm{M}\mathrm{}\mathrm{M}\mathrm{A}\mathrm{C}$ per inference,
as the architecture in [6]. The detector used as the network’s head is an
SSDLite [10] detector that is obtained, starting from the SSD model, by
replacing all the regular convolutions with depth-wise separable convolutions.
Replacing the SSD layer with the SSDLite one does not affect the mAP
performance, as reported in Table I. The SSDLite detector, which includes a
non-maximal suppression layer, is up to $3.3\times$ less computationally
expensive than the original SSD. The purpose of the non-maximal suppression
layer is to remove redundant and low-confidence predictions. The network’s
input is a 320$\times$240 pixels image, and it produces as output several
bounding boxes, together with a class label and the relative confidence, as
opposed to the FOMO network, which relies on centroids. We fine-tune the
network for 300 epochs with a learning rate of 0.00025 using the SGD
optimization algorithm. The learning rate scheduling policy considers a
momentum of 0.9, with a weight decay of 0.0005, a warmup time of 10 epochs,
and a minimum learning rate of 0.00005.

###  III-C Dataset

Figure 6: Samples of the three classes in our dataset [6].

Our fine-tuning labeled dataset comes from [6] and comprises more than 3,300
images of insects, selected through a strict filtering process from an initial
collection of more than 36,000 gathered through the internet. This filtering
ensures the near-absence of duplicates that could harm the model evaluation
fairness. It contains three insect classes: Popillia japonica, Cetonia aurata,
and Phyllopertha horticola. Popillia japonica is a dangerous pest insect; the
other two, while similar to Popillia japonica and often mistaken for it, are
not. We have 1,422, 1,318, and 877 samples for each class, respectively.
Figure 6 reports an example of the three classes of insects. We divided the
dataset into training and testing splits, with an 80-20% ratio.

###  III-D Deployment

To deploy the CNNs, we leverage two tools, one for each platform, since we
rely on tools that convert high-level Python code to C platform-specific code.
The first, for the Arduino Portenta board, is based on the Edge Impulse
project222https://edgeimpulse.com; the second, for the GAP9, is based on the
tool developed by GWT, namely, NN-Tool. The Edge Impulse tool provides an
easy-to-use interface to deploy neural networks on many embedded devices,
including the Arduino Portenta. The tool allows the selection, fine-tuning,
and deployment, in int8 or float32, of a custom or pre-designed neural
network. We deploy different variants of the networks in both fashions (int8
and float32) since the Arduino Portenta H7 SoC has hardware support for
double-precision floating point operations, i.e., an FPU in each core. Each
network is exported as a tflite model, and the inference phase is performed on
the embedded device in a MicroPython-based system, thanks to OpenMV. OpenMV,
with MicroPython, provides a USB streamer and the tensor movements in the
memory hierarchy with a reduced implementation effort but introduces
computational overhead. Furthermore, the OpenMV network execution does not use
the RAM to store the activations during the inferences. The tensors need to be
entirely in the L1 and L2 memory during the computation, thus avoiding the
tiling from the off-chip memories but limiting the execution of the network
within $1\text{\,}\mathrm{MB}$ of memory. For each network, we deploy an int8
quantized version of it, leveraging the reduced amount of memory needed for
the execution of these versions of the network.

NN-Tool is the deployment tool developed by GWT to deploy neural networks on
their platforms, such as GAP8 and GAP9 SoCs. It allows the deployment starting
from an onnx or tflite file. We finetune the COCO pre-trained MobileNet V3
with the SSDLite detector available in Pytorch on our dataset to obtain a
float32 version of the CNN. We then convert it into an onnx file that can be
used with NN-Tool. To obtain a model that is compliant with the layers
deployable with NN-Tool, such layers include convolutional, fully connected,
and dropout layers or skip connections, to name a few types of operations
supported. However, it does not support the deployment of the non-maximal
suppression layer included in the architecture, and as such, we remove the
operation from the onnx and apply it manually. NN-Tool also performs the
tiling and movement of tensors in the memory hierarchy to exploit all of its
levels. Since the platform has a single-precision FPU, the network is deployed
in both float16 and int8 fashions. The NE16 can be leveraged for CNN
applications since the NN-tool pipeline supports deploying networks
specifically tailored for this type of accelerator. Since it can be used only
with int8 networks, we deploy three architectures with NN-tool: float16, int8,
and int8 with NE16 hardware acceleration.

Even if the platforms have an FPU onboard and, as such, they can perform
floating point operations without relying on soft-float emulation, we perform
the quantization to int8 arithmetic since it is a convenient method to reduce
the memory occupation with a limited reduction in the accuracy of the network.
The quantization requires a calibration set for estimating the range of values
that each tensor can assume. Our calibration set is a subset of the training
set.

##  IV Results

###  IV-A Object detection performance

In this section, we evaluate the detection accuracy per each insect class,
where the dangerous Popillia japonica is the one we want to detect. We
evaluate the network accuracy considering the mAP, a standard metric for
object detection tasks, which ranges between 0 and 1. We consider an insect
correctly detected if the Intersection over Union (IoU) of the bounding box
produced by the network with the ground truth is at least 0.5, thus applying
the COCO standard for object detection [11]. We test the networks on our 660
samples test set, and we explore different types of input (i.e., image color
and resolution), as well as models deployed in float32, float16, and int8
(quantized). Figure 7 summarizes the mAP performance of all the tested
networks, with all combinations of grayscale and RGB inputs and considering
float32 and int8 quantized versions.

Figure 7: Comparison of the mean average precision (mAP).

Considering the models in float32 with RBG inputs, the SSDLite-MobileNetV3 is
the best-performing with an mAP score of 0.80 (input size 320$\times$240),
while the FOMO-MobileNetV2 achieves 0.78 in mAP (input size 160$\times$160),
and finally the smallest FOMO-MobileNetV2 marks an mAP of 0.67 (input size
96$\times$96). Then, assuming a cheaper monochrome camera, we assess the
performances of the CNNs by feeding grayscale inputs but still employing
float32 arithmetic. In this case, the SSDLite-MobileNetV3 (input size
320$\times$240), FOMO MobileNetV2 (input size 160$\times$160), and the FOMO-
MobileNetV2 (input size 96$\times$96), respectively score an mAP of 0.79,
0.62, and 0.55. The relative performances are kept the same, being the
SSDLite-MobileNetV3 still the most accurate, while all three networks drop in
mAP due to the grayscale input (losing from 0.01 to 0.16 in mAP). Finally,
considering the SSDLite-MobileNetV3 deployed in float16, to take full
advantage of the single-precision FPUs available on the GAP9 SoC, we observe
no drop in mAP compared with the double-precision implementation. Then
Instead, moving from float32 to int8 quantized versions of all CNNs, we see a
minimal drop in performance, always lower than 3%.

###  IV-B Embedded systems performance

In this section, we present a thorough performance assessment of all models
deployed on both devices, i.e., the Arduino Portenta (STM32H74 MCU) and the
GAP9 evaluation kit. Table III reports three configurations of the GAP9 SoC,
spanning the voltage from $0.65\text{\,}\mathrm{V}$ and
$0.8\text{\,}\mathrm{V}$, enabling different clock speeds, up to
$370\text{\,}\mathrm{MHz}$. In our experiments, we use the maximum efficiency
configuration since it achieves the longest battery duration given a target
throughput. In Table IV, we present our analysis regarding memory
requirements, latency for one-image inference, and the total power consumption
(including both compute unit and off-chip memories) for all CNNs/devices. For
each model, we report the data type used (float32 and int8 for the STM32H74,
and float16 and int8 for the GAP9), and the input image size ($\times$3 in
case of RGB images and $\times$1 for grayscale ones). For the GAP9, the peak
memory is computed considering the network’s layer with the maximum
requirements for input tensor, weights, and output tensor. Instead, for the
STM32H74, the computation also includes the input image (never freed for the
entire inference) and the memory footprint of a micro-python-based operating
system (OS) that presses on the same $1\text{\,}\mathrm{MB}$ L2 memory. While
for the GAP9 SoC, the deployment tool (NN-Tool) can organize data to exploit
both on-chip L2 memory ($1.6\text{\,}\mathrm{MB}$) and the off-chip ones (up
to $32\text{\,}\mathrm{MB}$), the STM32H74’s tool (Edge Impulse) can not
generate deployable code for CNNs requiring more than the
$1\text{\,}\mathrm{MB}$-L2 on-chip memory – this limitation is marked with a
in Table IV.

TABLE III: GAP9 configurations.

Configuration | Voltage [$\text{\,}\mathrm{V}$] | Freq CL [$\text{\,}\mathrm{MHz}$] | Freq FC [$\text{\,}\mathrm{MHz}$]
---|---|---|---
Min power | 0.65 | 150 | 150
Max efficiency | 0.65 | 240 | 240
Min latency | 0.80 | 370 | 370

TABLE IV: Latency and power consumption of all networks on both STM32H74 and
GAP9 MCUs.

Network | Data type | Input size | MCU |  Memory [$\text{\,}\mathrm{kB}$] | Latency [$\text{\,}\mathrm{ms}$] | Total power [$\text{\,}\mathrm{mW}$]
---|---|---|---|---|---|---
FOMO-MobileNetV2a | float32 |  96$\times$96$\times$1 | STM32H74 | 1070 | N.D. | N.D.
FOMO-MobileNetV2 | int8 |  96$\times$96$\times$1 | STM32H74 | 398 | 57 | 494
FOMO-MobileNetV2a | float32 |  96$\times$96$\times$3 | STM32H74 | 1045 | N.D. | N.D.
FOMO-MobileNetV2 | int8 |  96$\times$96$\times$3 | STM32H74 | 416 | 62 | 498
FOMO-MobileNetV2a | float32 |  160$\times$160$\times$1 | STM32H74 | 2654 | N.D. | N.D.
FOMO-MobileNetV2 | int8 |  160$\times$160$\times$1 | STM32H74 | 802 | 158 | 501
FOMO-MobileNetV2a | float32 |  160$\times$160$\times$3 | STM32H74 | 2862 | N.D. | N.D.
FOMO-MobileNetV2 | int8 |  160$\times$160$\times$3 | STM32H74 | 854 | 169 | 499
SSDLite-MobileNetV3 | float16 |  320$\times$240$\times$3 | GAP9 (CL) | 3622 | 462 | 41
SSDLite-MobileNetV3 | int8 |  320$\times$240$\times$3 | GAP9 (CL) | 1811 | 249 | 31
SSDLite-MobileNetV3 | int8 |  320$\times$240$\times$3 | GAP9 (NE16) | 1811 | 147 | 34

a Network not deployable (N.D.) due to its memory footprint exceeding the
$1\text{\,}\mathrm{MB}$ L2 available on the STM32H74. The memory footprint
includes the input and output tensors and weights of the largest network
layer, the input image, and the OS (in the case of the STM32H74).

On the STM32H74 MCU, we deploy the FOMO-MobilenNetV2, varying the input size
from 96$\times$96$\times$1 up to 160$\times$160$\times$3\. Among the
deployable CNNs on the STM32H74, the one using a monochrome 96$\times$96 image
with quantized (int8) representation is the smallest
($239\text{\,}\mathrm{kB}$ L2) and the fastest, achieving
$17.5\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$.
All the deployable networks require less than $501\text{\,}\mathrm{mW}$,
primarily due to MCU’s power consumption, which, in typical operating
conditions, consumes $\sim$$480\text{\,}\mathrm{mW}$ (using only the M7 core).

On the GAP9 SoC, we deploy the SSDLite-MobileNetV3 with a fixed input size of
320$\times$240$\times$3, considering three execution conditions: i) float16
running on general-purpose CL (with FPU hardware support), ii) int8 quantized
running on the CL, and iii) with int8 but exploiting the NE16 convolutional
accelerator. As expected, the float16 version is the most memory demanding,
requiring up to $\sim$$7\text{\,}\mathrm{MB}$ of memory, and the slowest.
Despite the platform’s FPU hardware support, the float16 network reaches only
$2.1\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$,
consuming $40.6\text{\,}\mathrm{mW}$. Instead, employing the NE16 accelerator
with the int8 data type, requires up to $3.4\text{\,}\mathrm{MB}$ of memory
and reaches up to
$6.8\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$
within only $34\text{\,}\mathrm{mW}$.

Finally, in Figure 8, we report the waveform of the power consumption of the
GAP9 SoC in three cases, i.e., with the float16 version, the int8 executed on
the general-purpose multicore cluster, and the int8 running on the NE16
accelerator, respectively in Figure 8-A, -B and -C. All three waveforms show
small periods of idleness (up to $41\text{\,}\mathrm{ms}$ in total), in
particular, at the beginning of the execution of the network, where the tensor
movements between memories do not overlap with the execution of the network.
The idle periods are reduced going forward in the execution since the size of
the tensors reduces.

Figure 8: Power waveforms for the three deployed networks on the GAP9, i.e.,
A) float16, B) int8 running on the CL, and C) int8 running on the NE16
accelerator.

###  IV-C Discussion

Compared to the best-in-class network, the RetinaNet-ResNet101-FPN full-
precision [6], our best model drops only 15% in accuracy evaluating it with
the mAP but reduces the number of operations by 299$\times$ and the memory
required by 14.9$\times$ allowing the deployment of the system on an ultra-low
power embedded system such as our GAP9-based board. On the other hand, the
FOMO MobileNetV2 reduces the number of operations by $\sim$20000$\times$ and
the memory by $\sim$2500$\times$ w.r.t. the RetinaNet-ResNet101-FPN full-
precision [6] with a reduction of mAP of 0.27. The impressive parameters’
reduction of both our networks, the FOMO MobileNet V2 and the SSDLite-
MobileNetV3, allows the deployment of our system in traps as an IoT device to
detect insects such as Popillia japonica, Cetonia aurata, and Phyllopertha
horticola. We envision a battery-powered insect detection system that uses one
of our platforms, the Arduino Portenta H7 with the FOMO MobileNetV2 or the
GAP9-based development kit running our MobileNetV3 with SSDLite, an ultra-low-
power camera such as the HIMAX HM-01B0 for image acquisition, and a LoRaWAN
module, e.g, the HTCC-AB01 board by CubeCell, for the transmission of the
detections. We estimate the duration of the battery considering a
$1000\text{\,}\mathrm{mA}\text{\,}\mathrm{h}$ battery rated at
$3.7\text{\,}\mathrm{V}$, which can provide up to
$7.4\text{\,}\mathrm{W}\text{\,}\mathrm{h}$, equivalent to
$13.3\text{\,}\mathrm{kJ}$, to our system.

Figure 9: Power breakdown considering the image acquisition, the processing,
and the transmission. A) GAP9 board, B) Arduino Portenta.

On the one hand, the energy consumption can be subdivided into
$3.82\text{\,}\mathrm{mJ}$ for the GAP9 SoC, $1.03\text{\,}\mathrm{mJ}$ for
the memory, $0.05\text{\,}\mathrm{mJ}$ for the camera, and
$2\text{\,}\mathrm{mJ}$ for the LoRaWAN transmission system, considering two
bytes sent each time. We highlight that the LoRaWAN transmission module is not
strictly required in the case of traps since we can also rely on a visual
signaling system, such as an LED positioned near the trap, that signals the
detection of the Popillia japonica. As a consequence, our GAP9-based system
consumes $6.9\text{\,}\mathrm{mJ}$ with the LoRa module or
$4.9\text{\,}\mathrm{mJ}$ without the LoRa module. The system for the Portenta
board consumes $31\text{\,}\mathrm{mJ}$ with the LoRa module or
$29\text{\,}\mathrm{mJ}$ without. Furthermore, we need to consider the energy
consumption in deep sleep mode for the GAP9 SoC, that is
$40\text{\,}\mathrm{J}$ per day, and for the Arduino Portenta, which is
$214\text{\,}\mathrm{J}$ per day. As such, the battery can last up to
$267\text{\,}\mathrm{}$ days with the LoRa transmission module or more than
$283\text{\,}\mathrm{}$ days in the case of LED-based signaling in the case of
the GAP9 board. For the Arduino Portenta, it reduces to $51\text{\,}\mathrm{}$
days with the LoRa transmissions and $58\text{\,}\mathrm{}$ without it. Figure
9 reports the power breakdown of the system described above. With the system
described above and running it once every $1\text{\,}\mathrm{min}$, we expect
up to 267 working days on the GAP9 SoC without recharging nor changing the
battery or up to 58 days on the Arduino Portenta.

Either our two detection systems can be deployed aboard nano-drones since they
can fit the power envelope ($\sim$$10\text{\,}\mathrm{W}$ including also the
motors) and the $\sim$$10\text{\,}\mathrm{g}$ payload available onboard. This
allows the envisioning of a pest-detection system based on a nano-drone that
flies into greenhouses and provides localized information for the use of
pesticides, drastically reducing their use related to widespread adoption.

##  V Conclusions

This work presents a novel hardware-software design for image-based pest
detection, deployable in battery-powered smart traps and aboard ultra-
constrained nano-drones. We present two system designs featuring two ultra-low
power embedded devices, i.e., a widely used dual-core Arduino Portenta H7 and
a novel multi-core GWT GAP9 SoC featuring the NE16 hardware accelerator. Given
these two devices’ significant memory and computing power differences, we
explore two alternative SoA CNNs. On the Portenta, we deploy a lightweight
FOMO-MobileNetV2
($5.88\text{\,}\mathrm{M}\mathrm{M}\mathrm{A}\mathrm{C}\mathrm{/}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}$),
capable of reaching 0.66 mAP on detecting the Popilla japonica bug, running at
$16.1\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$
and consuming $498\text{\,}\mathrm{mW}$. While on the GAP9 SoC, we deploy a
more complex SSDLite-MobileNetV3 CNN
($584\text{\,}\mathrm{M}\mathrm{M}\mathrm{A}\mathrm{C}\mathrm{/}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}$),
scoring an mAP of 0.79 with a throughput of
$6.8\text{\,}\mathrm{f}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{e}\mathrm{/}\mathrm{s}$
at $33\text{\,}\mathrm{mW}$. With our hardware-software codesign, we present a
fine-tuning procedure with a custom dataset for the detection task, an 8-bit
quantization stage for efficient exploitation of the two SoCs, and finally,
the implementation and deployment of our workloads. Compared to the huge
first-in-class RetinaNet-ResNet101-FPN
($174\,850\text{\,}\mathrm{M}\mathrm{M}\mathrm{A}\mathrm{C}\mathrm{/}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}$),
our best model drops only 15% in mAP, paving the way toward autonomous palm-
sized drones capable of lightweight and precise pest detection.

## References

  * [1] T. T. Høye, J. Ärje, K. Bjerge, O. L. Hansen, A. Iosifidis, F. Leese, H. M. Mann, K. Meissner, C. Melvad, and J. Raitoharju, “Deep learning and computer vision will transform entomology,” _Proceedings of the National Academy of Sciences_ , vol. 118, no. 2, p. e2002545117, 2021.
  * [2] D. Ju, D. Mota-Sanchez, E. Fuentes-Contreras, Y.-L. Zhang, X.-Q. Wang, and X.-Q. Yang, “Insecticide resistance in the cydia pomonella (l): Global status, mechanisms, and research directions,” _Pesticide Biochemistry and Physiology_ , vol. 178, p. 104925, 2021.
  * [3] E. P. on Plant Health (PLH), C. Bragard, K. Dehnen-Schmutz, F. Di Serio, P. Gonthier, M.-A. Jacques, J. A. Jaques Miret, A. F. Justesen, C. S. Magnusson, P. Milonas, _et al._ , “Pest categorisation of popillia japonica,” _Efsa Journal_ , vol. 16, no. 11, p. e05438, 2018.
  * [4] M. Preti, F. Verheggen, and S. Angeli, “Insect pest monitoring with camera-equipped traps: Strengths and limitations,” _Journal of pest science_ , vol. 94, no. 2, pp. 203–217, 2021.
  * [5] B. Ramalingam, R. E. Mohan, S. Pookkuttath, B. F. Gómez, C. S. C. Sairam Borusu, T. Wee Teng, and Y. K. Tamilselvam, “Remote insects trap monitoring system using deep learning framework and iot,” _Sensors_ , vol. 20, no. 18, p. 5280, 2020.
  * [6] L. Butera, A. Ferrante, M. Jermini, M. Prevostini, and C. Alippi, “Precise agriculture: Effective deep learning strategies to detect pest insects,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 9, no. 2, pp. 246–258, 2021.
  * [7] D. Palossi, A. Gomez, S. Draskovic, K. Keller, L. Benini, and L. Thiele, “Self-sustainability in nano unmanned aerial vehicles: A blimp case study,” in _Proceedings of the computing frontiers conference_ , 2017, pp. 79–88.
  * [8] L. Crupi, E. Cereda, A. Giusti, and D. Palossi, “Sim-to-real vision-depth fusion cnns for robust pose estimation aboard autonomous nano-quadcopters,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023, pp. 7711–7717.
  * [9] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, _et al._ , “Searching for mobilenetv3,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 1314–1324.
  * [10] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [11] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” pp. 740–755, 2014.
  * [12] A. C. Teixeira, J. Ribeiro, R. Morais, J. J. Sousa, and A. Cunha, “A systematic review on automatic insect detection using deep learning,” _Agriculture_ , vol. 13, no. 3, p. 713, 2023.
  * [13] F. Ali, H. Qayyum, and M. J. Iqbal, “Faster-pestnet: A lightweight deep learning framework for crop pest detection and classification,” _IEEE Access_ , 2023.
  * [14] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [15] L. Wang _et al._ , “Transfer learning-based lightweight ssd model for detection of pests in citrus,” _Agronomy_ , vol. 13, no. 7, p. 1710, 2023.
  * [16] W. Liu _et al._ , “Ssd: Single shot multibox detector,” in _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_. Springer, 2016, pp. 21–37.
  * [17] Y. Sun _et al._ , “Automatic in-trap pest detection using deep learning for pheromone-based dendroctonus valens monitoring,” _Biosystems engineering_ , vol. 176, pp. 140–150, 2018.
  * [18] T.-Y. Lin _et al._ , “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [19] Z. Shi _et al._ , “Detection and identification of stored-grain insects using deep learning: A more effective neural network,” _IEEE Access_ , vol. 8, pp. 163 703–163 714, 2020.
  * [20] J. Dai _et al._ , “R-fcn: Object detection via region-based fully convolutional networks,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [21] M. Dhanaraju, P. Chenniappan, K. Ramalingam, S. Pazhanivelan, and R. Kaliaperumal, “Smart farming: Internet of things (iot)-based sustainable agriculture,” _Agriculture_ , vol. 12, no. 10, 2022. [Online]. Available: https://www.mdpi.com/2077-0472/12/10/1745
  * [22] E. Humes _et al._ , “Squeezed edge yolo: Onboard object detection on edge devices,” 2023.
  * [23] M. Rusci _et al._ , “Parallel execution of the viola-jones algorithm on mcus for low-cost automated pest detection,” in _2023 IEEE Conference on AgriFood Electronics (CAFE)_. IEEE, 2023, pp. 35–39.
  * [24] L. Lamberti _et al._ , “Bio-inspired autonomous exploration policies with cnn-based object detection on nano-drones,” in _2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)_, 2023, pp. 1–6.
  * [25] F. Betti Sorbelli _et al._ , “Yolo-based detection of halyomorpha halys in orchards using rgb cameras and drones,” _Computers and Electronics in Agriculture_ , vol. 213, p. 108228, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0168169923006166
  * [26] L. Lamberti _et al._ , “Low-power license plate detection and recognition on a risc-v multi-core mcu-based vision system,” in _2021 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 2021, pp. 1–5.
  * [27] A. G. Howard _et al._ , “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [28] F. Zhuang _et al._ , “A comprehensive survey on transfer learning,” _Proceedings of the IEEE_ , vol. 109, no. 1, pp. 43–76, 2020.