# Comprehensive Analysis of the Object Detection Pipeline on UAVs

Leon Amadeus Varga
Cognitive Systems
University of Tuebingen, Germany
<EMAIL_ADDRESS>Sebastian Koch
Cognitive Systems
University of Tuebingen, Germany
<EMAIL_ADDRESS>Andreas Zell
Cognitive Systems
University of Tuebingen, Germany
<EMAIL_ADDRESS>
###### Abstract

An object detection pipeline comprises a camera that captures the scene and an
object detector that processes these images. The quality of the images
directly affects the performance of the object detector. Many works nowadays
focus either on improving the image quality or improving the object detection
models independently, but neglect the importance of joint optimization of the
two subsystems. The goal of this paper is to tune the detection throughput and
accuracy of existing object detectors in the remote sensing scenario by
focusing on optimizing the input images tailored to the object detector. To
achieve this, we empirically analyze the influence of two selected camera
calibration parameters (camera distortion correction and gamma correction) and
five image parameters (quantization, compression, resolution, color model,
additional channels) in remote sensing applications. For our experiments, we
utilize three UAV data sets from different domains and a mixture of large and
small state-of-the-art object detector models to provide an extensive
evaluation of the influence of the pipeline parameters. Finally, we realize an
object detection pipeline prototype on an embedded platform for an UAV and
give a best practice recommendation for building object detection pipelines
based on our findings. We show that not all parameters have an equal impact on
detection accuracy and data throughput, and that by using a suitable
compromise between parameters we are able to achieve higher detection accuracy
for lightweight object detection models, while keeping the same data
throughput.

Figure 1: Prototype system using an optimized detection pipeline.

##  1 Introduction

Micro Aerial Vehicles (MAVs) and Unmanned Aerial Vehicles (UAVs) can support
surveillance tasks in many applications. Examples are traffic surveillance in
urban environments [53], or search and rescue missions in maritime
environments [43], or the much more general task of remote sensing object
detection [13]. Two key components of these tasks are the camera system and
the object detector. The interaction, which is normally based on image data,
of these components is crucial. This work analyzes the impact of image data
settings. This kind of analysis is currently only available for the advanced
driver assistance system (ADAS) scenario (see section 2).
In most projects, the configuration of these components has to be conducted at
an early stage. A recording of a necessary data set should be already done
with the final camera system. This leads to a chicken-and-egg problem. The
optimal configuration depends highly on the application, but we provide some
intuitions of the general behavior for the remote sensing use case.
The emerging trend towards smart cameras, combining the camera and the
computing unit, confirms the need for classification and detection pipelines.
The lack of flexibility and the high price of these devices are still a
serious drawback and prevent their use in research or prototyping. In this
work, the detection pipeline is analyzed without smart cameras, but the
results could also be useful for projects that include them.
Besides the experiments (in a desktop and embedded environment) and the result
analysis, we applied the outcomes to build a proof-of-concept system. In this
work, we provide the most important findings. Additionally, in our
supplementary, we provide further details on the used detection models and
datasets, the code and a full list of experiments. Our contributions are:

  * •

We evaluated the impact of 7 parameters on three different data sets and four
models.

  * •

We achieve improved detection accuracy with similar data throughput on an
embedded platform and provide a best practice recommendation for the evaluated
parameters for the UAV object detection.

  * •

Finally, we demonstrate the usability with a prototype.

(a) 8 Bit

(b) 4 Bit

(c) 2 Bit

Figure 2: Example images for Quantization

##  2 Related work

The image processing pipeline has a large impact on the performance of object
detectors. Many parameters influence this pipeline. However, there is still a
lack of impact analysis for UAV-based systems. Yahyanejad _et al_. has
measured the impact of lens distortion correction for thermal images acquired
by a UAVs [48]. However, their results are only useful for a thermal camera
system.
Other existing camera parameter evaluations are mostly based on the autonomous
driving scenario [2, 9, 27, 28, 35, 36]. The findings of the autonomous
driving experiments cannot be directly applied to the remote sensing use-case.
For autonomous driving, the scene is often homogenous, and the object sizes
differ only slightly. Further, the distance to the objects is often only a
couple of meters. For remote sensing recordings, the distance to the scene is
often over 30 meters [43], resulting in tiny objects. Depending on the camera
angle, the images can vary strongly. Because of these differences, the results
of one scenario are not completely valid in the other, but can give a first
sign.
Balsinski _et al_. proposed a render framework to evaluate camera
architectures in simple autonomous driving scenarios [2]. Secci _et al_.
evaluated camera failures in the autonomous driving application [36]. Besides
these, the works of Liu _et al_. [27], Carlson _et al_. [9] and Saad _et al_.
[35] showed the still existing gap between synthetic and real-world data.
Thus, our experiments are based entirely on real-world recordings.
Buckler _et al_. analyzed the image pipeline regarding performance and energy
consumption [6]. In contrast to their work, we close the gap for a missing
camera parameter evaluation in the remote sensing application. Further, we
consider parameters, which are configurable for most off-the-shelf camera
systems.
In contrast to work that accelerates inference by, e.g., quantizing the neural
network [24, 7], our work focuses on the parameters of the object recognition
pipeline that affect the overall throughput time of the system, rather than on
the inference time itself.
In summary, our work differs from the aforementioned works in the following
aspects. We focus on the remote sensing use-case for UAVs and instead of using
synthetic data, we used established real-world data sets. The evaluated camera
parameters are configurable for most cameras.

##  3 Considered factors of influence

Our goal is to evaluate the impact of adjustable acquisition parameters on the
performance of object detectors in UAV remote sensing. We selected parameters
which have a direct influence on the recordings. Three parameters,
Quantization, Compression and Resolution, control the required memory
consumption. Further, we evaluated the impact of thorough camera setup and
calibration. We select radial Image Distortion which originates from the
camera lens and Gamma Correction to simulate different exposure settings for a
prerecorded dataset. Other camera effects and calibration parameters like
focus distance are also interesting, but harder to simulate with prerecorded
dataset. Thus, we limit ourselves to those chosen effects and measure the
impact of a correctly exposed and distortion corrected camera. Further, we
evaluate the impact of the selected Color Model. Multispectral recordings can
carry useful additional information; we therefore check how easily they can be
integrated into existing architectures and whether they help to improve the
prediction capability.
We limit our evaluation to these parameters because we believe that these
parameters have a direct influence on the recording and can be simulated on
existing datasets.

###  3.1 Quantization

The quantization, also called bit depth for images, defines the number of
different values per pixel and color (see Fig. 2). Quantization is used to
describe a continuous signal with discrete values [15]. Too few quantization
steps lead to a large quantization error, which results in a loss of
information. Therefore, it is necessary to find a balance between the required
storage space and the information it contains.
All used data sets provide images with a bit depth of 8bit per color channel.
We want to evaluate, whether we can reduce the bit depth further without
losing performance. This could reduce the data size. Hence, 8bit is the
maximal available quantization of the data sets. We provide experiments for
8bit, 4bit and 2bit quantization, which would decrease the size of the images
for a bitmap format by factor 2 (for 4bit) and 4 (for 2bit).

(a) No comp.

(b) 90 comp. qlty.

(c) 70 comp. qlty.

Figure 3: Example images for Compression

###  3.2 Compression

Compression is used to reduce the required space of a recording. There are two
types of compressions: lossless and lossy compressions. The image format PNG
belongs to lossless compressions techniques. The compression time for PNG is
very high, therefore it is often not considered for online compression. JPEG
is faster, but also a lossy compression. The compression removes information.
Nevertheless, JPEG is a common choice. It provides a sufficient trade-off
between loss of information, file size and speed.
The compression quality of JPEG is defined as a value between 1 and 100
(default: 90). Higher values result in less compression.
We check the relation between compression and detector performance. A lower
compression quality would cause smaller file sizes. This could directly lead
to a higher throughput of the system. For the inferences of models, the
decompressed data is used. A lower compression quality would, therefore,
mainly affect the throughput from the camera to the processing unit memory.
Since this is also a limiting factor for high-resolution or high-speed
cameras, it is worth checking. In our experiments, we evaluate the influence
of no compression and a compression quality for JPEG of 90 and 70 (see Fig.
3).

(a) Full resolution

(b) Max 768 pixels

(c) Max 256 pixels

Figure 4: Example images for image resolutions with max side length

###  3.3 Color Model

The color model defines the representation of the colors. One of the most
common one is RGB [18]. It is an additive color model, which is based on the
three colors red, green, and blue. For object detection and object
classification, this representation is mostly used [34, 41]. Besides RGB, non-
linear transformations of RGB are also worth considering. HLS and HSV are
based on a combination of hue and saturation. Both differ only in the
brightness’s definition value. YCbCr takes human perception into account and
encodes information more suitably for the human eye.
In some applications, HSV outperforms RGB. Cucchiara _et al_. [10] and Shuhua
_et al_. [38] could confirm this performance improvement. Kim _et al_.
demonstrated RGB outperforms the other color models (HSV, YCbCr, and CIE Lab)
in traffic signal detection. It appears, which color model is best depends
strongly on the application and the model.
The color model is also a design decision, so we evaluate the performance of
the RGB, HSV, HLS, and YCbCr color models for object detection in remote
sensing. We also review the performance of gray-scale images. To achieve
comparable results, the backbones are not pre-trained for these experiments.

###  3.4 Resolution

Higher resolutions reveal more details, but also result in larger recordings.
Especially for the small objects of remote sensing, the details are crucial
[26]. Therefore, choosing an optimal resolution is important [39].
Besides the maximal available resolution defined by the used camera, the GPU
memory often limits the training size. The reason lies in the gradient
calculation within the backpropagation step. There are techniques, which can
reduce this problem [44, 29]. To allow a fair comparison, we limit the maximum
training size to 1024 pixels for the length of the larger side. This training
size is processable with all models used. The experiments should show the
performance reduction for smaller image sizes. So we trained on the training
sizes of 256 pixels, 512 pixels, 756 pixels and 1024 pixels (see examples in
Fig. 4). Further, we evaluated each training size with three test sizes, which
are defined by the training size $\times$ the factors 0.5, 1 and 2.

###  3.5 Multispectral Recordings

In recent years, multispectral cameras got more popular for usage on drones
[8] [12] [50]. The additional channels of these increase the recording size,
but can also provide important information. In contrast to conventional RGB
cameras, multispectral cameras add channels outside the visible light (e.g.
near-infrared or thermal). In pedestrian and traffic monitoring, multispectral
imaging can increase the detector performance, as shown by Takumi _et al_.
[21], Maarten _et al_. [42], and Dayan _et al_. [16]. And for precision
agriculture, they are one of the key components [8] [12] [50]. We want to
create awareness, that multispectral recordings can be helpful in some
situation. But we also want to check how easy it is to integrate these into
existing object detection pipelines. So, the focus is not on evaluating
multimodal models [31], and we consider multispectral images as a limiting
case to classical color images. Our experiments are limited to the early
fusion approach [52].
We used the multispectral images from the SeaDronesSee [43] data set for our
experiments. These were acquired with a MicaSense RedEdge MX and thus provide
wavelengths of 475 nm (blue), 560 nm (green), 668 nm (red), 717 nm (red edge),
and 842 nm (near infrared). We evaluate how the additional wavelengths affect
the performance of the object detectors in detecting swimmers (see example in
Fig. 5). The trend of the results should be generally usable, even if the
exact results are only for the maritime environment. Always, if additional
channels could carry useful information, it is worth considering an evaluation
of this kind.

###  3.6 Calibration Parameters

#### Image Distortion:

During the image formation process, most lenses introduce a distortion in the
digital image that does not exist in the scene [19]. Wide-angle lenses or
fish-eye lenses suffer from the so-called barrel-distortion. Telephoto lenses
suffer from the so-called pincushion-distortion.
Often, one of the first steps in the image processing pipeline is to correct
the lens distortion using a reference pattern.
As we are working on data sets that are distortion rectified, we artificially
introduce distortion via the Brown–Conrady model [5]. To evaluate the impact
of an incorrect calibration, we introduce a small barrel-distortion with
$k_{1}=-0.2$ and pincushion-distortion with $k_{1}=0.2$. A visualization of
these distortions can be found in the supplementary material.

#### Gamma Correction:

For most cameras use an auto-exposure setting which adjusts the camera
exposure to the ambient lighting. But manual exposure is still used and can
result in overexposed (to bright) or underexposed (to dark) images. Gamma
correction can to an extent ease this problem, but it can also be used to
simulate a camera in a wrong exposure setting.
Gamma Correction is a non-linear operation to encode and decode luminance to
optimize the usage of bits when storing an image, by taking advantage of the
non-linear manner in which humans perceive light and color [33]. Gamma
Correction can also simulate a shorter or longer exposure process after the
image formation process. If images are not gamma-corrected, they allocate too
many bits for areas that humans cannot differentiate. Neural networks,
however, differ from humans in the sense that they work with floating-points
which makes them sensitive to minimal changes in quantization as can be
observed in many works that examine weight quantization [49, 23, 20].
Contrarily, humans perceive changes in brightness logarithmically [37]. We
therefore assume that Gamma Correction may have a negligible impact on neural
network detection performance.
Many works have employed Gamma Correction, intending to improve detection
performance directly or help training with Gamma Correction as an augmentation
technique [46, 51].
We use three different gamma configurations (0.5, 1.0, 2.5) to investigate the
effects of gamma on the object detector performance. These experiments are
designed to show whether gamma is a crucial parameter. The performance of a
dynamic gamma adjustment [1] will also be tested.

(a) RGB

(b) Near-infrared

Figure 5: Example images for multispectral recordings in maritime environment.
In the NIR recordings, the swimmers are fully visible. Figure 6: Quantization:
the impact of the quantization on the performance of the object detectors for
all data sets. The y-axis shows the mean Average Precision (mAP).

##  4 Experiments

In this section, we describe the setup of the experiments. The setup contains
the data sets, the object detectors and the hardware environment. For each
configuration, we trained with three different random seeds. The random seed
affects the weight initialization, the order of the training samples and data
augmentation techniques. The three initializations show the stability of each
configuration.

###  4.1 Datasets

We evaluated our experiments on three data sets. All data sets are part of the
remote sensing scenario. Dota-2 is based on satellite recordings [13].
VisDrone [53] and SeaDronesSee [43] were recorded by Micro Air Vehicles
(MAVs). The first two data sets are well established and commonly used to
benchmark object detection for remote sensing. In contrast, SeaDronesSee
focuses on a maritime environment, which differs from the urban environment of
VisDrone and Dota-2. Therefore, it introduces new challenges and is a proper
extension for our experiments. Further details can be found in the
supplementary material.

Figure 7: Compression: the impact of the JPEG compression on the performance
of the object detectors for all data sets. The y-axis shows the mean Average
Precision (mAP). A higher compression quality means less compression.

###  4.2 Models

For our experiments, we selected four different object detectors. The two-
stage detector Faster-RCNN [34] can still achieve state-of-the-art results
with smaller modifications [53]. By comparison, the one-stage detectors are
often faster than the two-stage detectors. Yolov4 [3] and EfficientDet [41]
are one-stage detectors and focus on efficiency. Both perform well on embedded
hardware. Modified versions of CenterNet [14], also a one-stage detector,
could achieve satisfying results in the VisDrone challenge [53]. These four
models cover a variety of approaches and can give a reliable impression of the
parameter impacts.
For Faster R-CNN, EfficientDet and CenterNet, we used multiple backbones to
provide an insight for different network sizes. The networks and their
training procedure are described in the supplementary material in further
detail.

###  4.3 Hardware setup

Most of the experiments were done in a desktop environment. To evaluate the
performance in a more applicable environment, we performed experiments on an
Nvidia Xavier AGX board. The small size and the low weight allow the usage for
onboard processing in MAVs or other robots. The full description of the
desktop and embedded environment can be found in the supplementary material.

##  5 Results and discussion

In this section, we present the results of the experiments. Each parameter is
considered in its own subsection. The experiments of the different backbones
per network were grouped, only the largest backbones of CenterNet and Faster
R-CNN are listed separately. As an evaluation metric for the object detectors,
we use the mean Average Precision (mAP) with an IoU threshold of 0.5 [25].
In the Dota dataset, the performance of the models differs from the
performance in the other datasets. The objects in the Dota dataset are very
small, unlike the other datasets. This requires different properties of the
object detectors. For a more detailed analysis of this behavior and a complete
list of all experiments, we refer the reader to supplementary material4. Here
we highlight the most important results in terms of parameters.

###  5.1 Quantization

A quantization reduction leads to a loss of information in the recording. This
assumption is confirmed by the empirical results in Fig. 6. This phenomenon is
consistent for all data sets and all models. However, deterioration varies.
Worth mentioning is the marginal difference ($\sim 3\%$) in performance
between 4bit and 8bit recordings. Hence, it is possible to have half the
amount of data and achieve nearly the same result. The loss in performance for
a 2bit quantization is much higher ($\sim 41\%$).

Figure 8: Color Model: the impact of the color models. The y-axis shows the
mean Average Precision (mAP). Figure 9: Image Size: Faster R-CNN trained with
different training image sizes and validated on $\text{validation image
size}=x\times\text{training image size}$. The y-axis shows the mean Average
Precision (mAP).

###  5.2 Compression

Fig. 7 shows the relation between compression quality and object detector
performance. Lower compression quality, i.e. higher compression, leads to
worse performance. With some exceptions, this is true for all models. Overall,
the performance degeneration of the tested compression qualities is small. For
a compression quality of 90, the performance is decreased by $\sim 2.7\%$. A
compression quality of 70 leads to an mAP loss of $\sim 5.3\%$. Compared to
the performance loss, the size reduction is significant.
We observe that a compression quality of 90 reduces the required space by
$\sim 25\%$ for a random subset of the VisDrone validation set. A compression
quality of 70 can almost halve the required space.
However, the memory reduction depends highly on the image content. For
SeaDronesSee and Dota the memory reduction is much higher. For compression
quality of 70 and 90, it is about $\sim 93\%$ and, $\sim 86\%$ respectively.
By reducing the required image space, it is possible to increase the pipeline
throughput. In return, a compression and decompression is necessary. In the
final experiments (see section 6), the utility of this approach becomes clear.

###  5.3 Color Model

We can conclude two important outcomes of the experiment for color models.
First, we have to distinguish between two situations, which are visible in
Fig. 8. In the first situation, the object detectors seem to not take
advantage of the color information. Therefore, we assume the color is in this
case not beneficial. This applies to the VisDrone data set and for the Dota
data set. Both data sets provide objects and backgrounds with a high color
variance. For example, a car in the VisDrone data set can have any color.
Thus, the texture and shape are much more important. That means the object
detector can already achieve excellent results with the gray-scale recordings.
For the Dota data set, the color information could only be used by the largest
models (Faster R-CNN/ResNext101_32x8d and CenterNet/Hourglass104). The other
models weren’t able to incorporate the color and relied only on the texture.
In our SeaDronesSee experiments, we observe that the gray-scale experiments
are outperformed by the color experiments. The explanation can be found in the
application. It covers a maritime environment. As a result, the background is
always a mixture of blue and green. So, the color information is reliable in
distinguishing between background and foreground. For SeaDronesSee, all models
use the color, which is evident in the drop in performance for the gray-scale
experiments.
If the color is important, RGB seems to produce the overall best results;
followed by YCrCb. However, since most data augmentation pipelines have been
optimized for RGB, it is recommended to start with RGB.
As a conclusion, it is important to know whether color information is relevant
in the application. Most times, gray-scale images are sufficient, which leads
to a third of the image size and may even allow simpler cameras.

###  5.4 Resolution

For these experiments, we only visualize the experiments of Faster R-CNN. The
three selected plots (visible in Fig. 9) are good representatives for the
other models, because they describe the general behavior well. The full list
of experiments can be found in the supplementary material.
First, higher training resolutions always improve the performance of the
models. Aside from that, the deterioration of performance for a decreased
validation image size is consistent.
For the Dota data set, we can observe a performance drop for a higher scale
factor. The trained models seem very sensitive to object sizes and cannot take
advantage of higher resolutions.
For the SeaDronesSee data set, performance improves with higher validation
image sizes for most models. So it is possible to improve the performance of a
trained model by simply using higher inference resolutions. However, the
improvement is marginal and should be supported by scaling data augmentation.
The results for the SeaDronesSee data set also apply to the VisDrone data set.
Only for the training image size of $1024$ pixels, this rule does not hold.
The reason for this is the small number of high-resolution images in the
VisDrone validation set. In summary, training and validation size seems to be
crucial, and it is not a good idea to cut corners at this point. This also
applies for common object, but is even more critical for small object like in
this application.

Figure 10: Multispectral: the performance of the object detectors on multispectral data. (’BGRNE’ - multispectral recordings, ’RGB’ - color images, ’GRAY’ - gray-scale) Table 1: Application of the optimized parameter configuration in a desktop and an embedded environment. Data set |  Dota mAP 50 $\uparrow$ |  VisDrone mAP 50 $\uparrow$ |  SeaDronesSee mAP 50 $\uparrow$ |  |  avg. FPS $\uparrow$
---
(Desktop)
|  avg. FPS $\uparrow$
---
(Embedded)
| Parameters
---
EfficientDet | 23.83 | 20.71 | 22.29 | 25 | 14 |  4.5M - 17.7 M
Reduced data (our) * | 22.07 (-7 %) | 19.53 (-5 %) | 22.20 (-1 %) | 28 (+12 %) | 16 (+14 %)
| Reduced data
---
\+ higher res. (our) **
30.85 (+ 29%) | 25.52 (+23 %) | 27.35 (+23 %) | 25 | 14
YoloV4 | 16.19 | 24.49 | 40.62 | 20 | 4 | 63.9M
Reduced data (our) * | 13.84 (-14 %) | 23.61 (-3 %) | 36.58 (-9 %) | 21 (+5 %) | 5 (+20 %)
CenterNet | 21.35 | 29.94 | 31.08 | 40 | - |  14.4M - 49.7 M
Reduced data (our) * | 21.22 (-1 %) | 27.87 (-6 %) | 31.05 (-1 %) | 46 (+15 %)
| CenterNet/
---
Hourglass104
44.63 | 52.07 | 51.12 | 6 | - | 200M
Reduced data (our) * | 41.15 (-7 %) | 48.59 (-6 %) | 47.89 (-6 %) | 6
Faster R-CNN | 46.67 | 35.45 | 39.94 | 17 | - |  41.4M - 60.3M
Reduced data (our) * | 43.92 (-5 %) | 33.05 (-6 %) | 39.26 (-1 %) | 18 (+6 %)
| Faster R-CNN/
---
ResNext101_32x8d
47.78 | 37.70 | 40.76 | 9 | - | 104M
Reduced data (our) * | 44.06 (-7 %) | 35.87 (-4 %) | 40.59 (-1 %) | 9

###  5.5 Multispectral Recordings

These experiments are based on multispectral data of the SeaDronesSee data
set. The set of about 500 multispectral images is not very large, so the
results can only give an impression. The usefulness of the additional channels
depends strongly on the application. For the maritime environment, the
additional information is very useful, especially the near-infrared channel.
This part of the spectrum is absorbed by water [11], resulting in a foreground
mask (visible in Fig. 5).
The results in Fig. 10 fit to the findings for the color models (see section
5.3). For the maritime environment, the color is important, indicated by the
gap for gray-scale recordings. The multispectral recordings (labeled as
‘BGRNE’) can only improve the performance of the larger models. For the
smaller models, it even leads to a decrease in performance. To support the
multispectral recordings as input, we had to increase the input-channels of
the first layer to five and lost the pretraining of this layer. The rest of
the backbone is still pre-trained.
In summary, additional channels can be helpful depending on the use case. It
is important to keep in mind that larger backbones can make better use of the
additional information. Furthermore, it is a trade-off between a fully pre-
trained backbone and additional channels.

###  5.6 Camera Calibration

In our experiments, we are able to observe that the effects of gamma
correction and image distortion are equal across all datasets and models. Tab.
2 shows, that applying (dynamic) gamma correction to the datasets seems to
have a negligible/negative effect on detection accuracy. This implies that
gamma correction indeed does not have a notable effect on detection accuracy.
Image distortion seems to also have only a very small effect on detection
accuracy, as can be seen in Tab. 19. However, the models that have been
evaluated on images distorted using pincushion distortion achieve slightly
higher accuracy. But although the mean performance over all datasets and
models is higher for images with pincushion distortion, most of the models
perform best when evacuated on no distortion as can be seen in our
supplementary where we provide a detailed analysis over all models and
datasets.
We can conclude that camera calibration is an existential part in many
computer vision applications, however the investigated correction methods seem
to only marginally affect the performance of an object detector for remote
sensing applications.

Table 2: Gamma Correction: impact on the performance. | 0.5 | Orig. | 2.5 | Dynamic
---|---|---|---|---
| Relative
---
mAP50 $\uparrow$
98.9 % | 100 % | 98.3 % | 99.2 %
Table 3: Image Distortion: impact on the performance. | Barrel dist. | Orig. | Pincushion dist.
---|---|---|---
| Relative
---
mAP50 $\uparrow$
99.8% | 100 % | 100.6 %

##  6 Application

Based on the previous investigations, we determined the optimal parameter
configuration for our application. The goal was to find a suitable compromise
between the detection performance and the required space for a recording. The
recording size correlates directly with the throughput of the system,
including the inference time. A smaller recording size therefore means more
frames per second.
We considered 4bit quantization as a good trade-off. Further, JPEG compression
quality of 90 seems to be a good compromise. For the urban surveillance use-
case, color is not important. Thus, for the VisDrone data set, we used gray-
scale images. For the other data sets, the color recordings are required.
Finally, the highest possible resolution was used. In Tab. 1, the application
of these findings is visible. The first setup (marked *), which uses the
reduced quantization, the JPEG compression and the optional color reduction,
increases the throughput (including the inference, the pre- and post-
processing) of the smaller models significantly. For the larger models,
inference accounts for most of the throughput, so there is no significant
speed-up.
By combining the previous three adjustment and increasing the resolution
(marked **), the performance of the detector can be improved while maintaining
the base throughput. In these experiments, the usage of CroW [44] allowed
training with higher resolution.
Finally, we use the findings to deploy a UAV based prototype (displayed in
Fig. 1) with nearly real-time and high-resolution processing. The system was
carried by a DJI Matrice 100 drone. An Nvidia Xavier AGX board, mounted on the
drone, performed the calculations and an Allied Vision 1800 U-1236 camera was
used for capturing.

##  7 Conclusion

This paper presents an extensive study of parameters that should be considered
when designing a full object detection pipeline in an UAV use-case. Our
experiments show that not all parameters have an equal impact on detection
performance, and a trade-off can be made between detection accuracy and data
throughput.
By using the above recommendations, it is possible to maximize the efficiency
of the object detection pipeline in a remote sensing application.
In a future work, we would like to explore recording data with varying camera
parameters in mind. This would give the possibility of evaluating even more
camera and image parameters like camera focus or exposure. Also, a deeper
analysis of the usability of multispectral imaging could provide useful
insights.

## References

  * [1] M. Abdullah-Al-Wadud, Md. Hasanul Kabir, M. Ali Akber Dewan, and Oksam Chae.  A dynamic histogram equalization for image contrast enhancement.  IEEE Transactions on Consumer Electronics, 53(2):593–600, 2007\.
  * [2] Henryk Blasinski, Joyce E. Farrell, Trisha Lian, Zhenyi Liu, and Brian A. Wandell.  Optimizing image acquisition systems for autonomous driving.  electronic imaging, 2018:161–1–161–7, 2018.
  * [3] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  CoRR, abs/2004.10934, 2020.
  * [4] Thomas Boutell.  PNG (portable network graphics) specification version 1.0.  RFC, 2083:1–102, 1997.
  * [5] Duane C Brown.  Decentering distortion of lenses.  Photogrammetric Engineering and Remote Sensing, 1966.
  * [6] Mark Buckler, Suren Jayasuriya, and Adrian Sampson.  Reconfiguring the imaging pipeline for computer vision.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 975–984. IEEE Computer Society, 2017.
  * [7] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer.  Zeroq: A novel zero shot quantization framework.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13166–13175. Computer Vision Foundation / IEEE, 2020.
  * [8] Sebastian Candiago, Fabio Remondino, Michaela De Giglio, Marco Dubbini, and Mario Gattelli.  Evaluating multispectral images and vegetation indices for precision farming applications from uav images.  Remote sensing, 7(4):4026–4047, 2015.
  * [9] Alexandra Carlson, Katherine A. Skinner, Ram Vasudevan, and Matthew Johnson-Roberson.  Modeling camera effects to improve visual learning from synthetic data.  In Laura Leal-Taixé and Stefan Roth, editors, Computer Vision - ECCV 2018 Workshops - Munich, Germany, September 8-14, 2018, Proceedings, Part I, volume 11129 of Lecture Notes in Computer Science, pages 505–520. Springer, 2018.
  * [10] R. Cucchiara, C. Grana, M. Piccardi, A. Prati, and S. Sirotti.  Improving shadow suppression in moving object detection with hsv color information.  In ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No.01TH8585), pages 334–339, 2001.
  * [11] Joseph A. Curcio and Charles C. Petty.  The near infrared absorption spectrum of liquid water.  J. Opt. Soc. Am., 41(5):302–304, May 1951.
  * [12] Lei Deng, Zhihui Mao, Xiaojuan Li, Zhuowei Hu, Fuzhou Duan, and Yanan Yan.  Uav-based multispectral remote sensing for precision agriculture: A comparison between different cameras.  ISPRS journal of photogrammetry and remote sensing, 146:124–136, 2018.
  * [13] Jian Ding, Nan Xue, Gui-Song Xia, Xiang Bai, Wen Yang, Michael Ying Yang, Serge J. Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang.  Object detection in aerial images: A large-scale benchmark and challenges.  CoRR, abs/2102.12219, 2021.
  * [14] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 6568–6577. IEEE, 2019.
  * [15] A. Gersho.  Quantization.  IEEE Communications Society Magazine, 15(5):16–16, 1977.
  * [16] Dayan Guan, Yanpeng Cao, Jiangxin Yang, Yanlong Cao, and Michael Ying Yang.  Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection.  Inf. Fusion, 50:148–157, 2019.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, pages 770–778. IEEE Computer Society, 12 2016.
  * [18] R.W.G. Hunt.  The Reproduction of Colour.  The Wiley-IS&T Series in Imaging Science and Technology. Wiley, 2005\.
  * [19] Photography — Digital cameras — Geometric distortion (GD) measurements.  Standard, International Organization for Standardization, Geneva, CH, July 2015.
  * [20] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018.
  * [21] Takumi Karasawa, Kohei Watanabe, Qishen Ha, Antonio Tejero-de-Pablos, Yoshitaka Ushiku, and Tatsuya Harada.  Multispectral object detection for autonomous vehicles.  In Wanmin Wu, Jianchao Yang, Qi Tian, and Roger Zimmermann, editors, Proceedings of the on Thematic Workshops of ACM Multimedia 2017, Mountain View, CA, USA, October 23 - 27, 2017, pages 35–43. ACM, 2017.
  * [22] Diederik P Kingma and Jimmy Lei Ba.  Adam: A method for stochastic optimization.  In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 2015.
  * [23] Raghuraman Krishnamoorthi.  Quantizing deep convolutional networks for efficient inference: A whitepaper.  arXiv preprint arXiv:1806.08342, 2018.
  * [24] Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan.  Fully quantized network for object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 2810–2819. Computer Vision Foundation / IEEE, 2019.
  * [25] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740–755. Springer, 2014.
  * [26] Kang Liu and Gellért Máttyus.  Fast multiclass vehicle detection on aerial images.  IEEE Geosci. Remote. Sens. Lett., 12(9):1938–1942, 2015.
  * [27] Zhenyi Liu, Trisha Lian, Joyce E. Farrell, and Brian A. Wandell.  Soft prototyping camera designs for car detection based on a convolutional neural network.  In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pages 2383–2392. IEEE, 2019.
  * [28] Zhenyi Liu, Trisha Lian, Joyce E. Farrell, and Brian A. Wandell.  Neural network generalization: The impact of camera parameters.  IEEE Access, 8:10443–10454, 2020.
  * [29] Martin Messmer, Benjamin Kiefer, and Andreas Zell.  Gaining scale invariance in UAV bird’s eye view object detection by adaptive resizing.  CoRR, abs/2101.12694, 2021.
  * [30] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9912 LNCS, pages 483–499, 2016.
  * [31] Tanguy Ophoff, Kristof Van Beeck, and Toon Goedemé.  Exploring rgb+depth fusion for real-time object detection.  Sensors, 19(4):866, 2019.
  * [32] Dheeraj Reddy Pailla, Varghese Alex Kollerathu, and Sai Saketh Chennamsetty.  Object detection on aerial imagery using centernet.  CoRR, abs/1908.08244, 2019.
  * [33] Charles Poynton.  Digital video and HD: Algorithms and Interfaces.  Elsevier, 2012.
  * [34] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  IEEE Trans. Pattern Anal. Mach. Intell., 39(6):1137–1149, 2017\.
  * [35] Kmeid Saad and Stefan-Alexander Schneider.  Camera vignetting model and its effects on deep neural networks for object detection.  In 2019 IEEE International Conference on Connected Vehicles and Expo, ICCVE 2019, Graz, Austria, November 4-8, 2019, pages 1–5. IEEE, 2019.
  * [36] Francesco Secci and Andrea Ceccarelli.  On failures of RGB cameras and their effects in autonomous driving applications.  In Marco Vieira, Henrique Madeira, Nuno Antunes, and Zheng Zheng, editors, 31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020, Coimbra, Portugal, October 12-15, 2020, pages 13–24. IEEE, 2020.
  * [37] Jianhong Shen.  On the foundations of vision modeling: I. weber’s law and weberized tv restoration.  Physica D: Nonlinear Phenomena, 175(3):241–251, 2003.
  * [38] Li Shuhua and Guo Gaizhi.  The application of improved hsv color space model in image processing.  In 2010 2nd International Conference on Future Computer and Communication, volume 2, pages V2–10–V2–13, 2010.
  * [39] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, 2019.
  * [40] Mingxing Tan and Quoc V. Le.  Efficientnetv2: Smaller models and faster training.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10096–10106. PMLR, 2021.
  * [41] Mingxing Tan, Ruoming Pang, and Quoc V. Le.  Efficientdet: Scalable and efficient object detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10778–10787. Computer Vision Foundation / IEEE, 2020.
  * [42] Maarten Vandersteegen, Kristof Van Beeck, and Toon Goedemé.  Real-time multispectral pedestrian detection with a single-pass deep neural network.  In Aurélio Campilho, Fakhri Karray, and Bart M. ter Haar Romeny, editors, Image Analysis and Recognition - 15th International Conference, ICIAR 2018, Póvoa de Varzim, Portugal, June 27-29, 2018, Proceedings, volume 10882 of Lecture Notes in Computer Science, pages 419–426. Springer, 2018.
  * [43] Leon Amadeus Varga, Benjamin Kiefer, Martin Messmer, and Andreas Zell.  Seadronessee: A maritime benchmark for detecting humans in open water.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2260–2270, 2022.
  * [44] Leon Amadeus Varga and Andreas Zell.  Tackling the background bias in sparse object detection via cropped windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 2768–2777, October 2021.
  * [45] Gregory K. Wallace.  The JPEG still picture compression standard.  Commun. ACM, 34(4):30–44, 1991.
  * [46] Chuan-Wei Wang, Chin-An Cheng, Ching-Ju Cheng, Hou-Ning Hu, Hung-Kuo Chu, and Min Sun.  Augpod: Augmentation-oriented probabilistic object detection.  In CVPR Workshop on the Robotic Vision Probabilistic Object Detection Challenge, 2019.
  * [47] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [48] Saeed Yahyanejad, Jakub Misiorny, and Bernhard Rinner.  Lens distortion correction for thermal cameras to improve aerial imaging with small-scale uavs.  In 2011 IEEE International Symposium on Robotic and Sensors Environments, ROSE 2012, Montréal, Canada, September 17-18, 2011, pages 231–236. IEEE, 2011.
  * [49] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat.  Q8bert: Quantized 8bit bert.  In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 36–39. IEEE, 2019.
  * [50] Liyuan Zhang, Huihui Zhang, Yaxiao Niu, and Wenting Han.  Mapping maize water stress based on uav multispectral remote sensing.  Remote Sensing, 11(6):605, 2019.
  * [51] Lili Zhang, Yi Zhang, Zhen Zhang, Jie Shen, and Huibin Wang.  Real-time water surface object detection based on improved faster r-cnn.  Sensors, 19(16):3523, 2019.
  * [52] Yifei Zhang, Désiré Sidibé, Olivier Morel, and Fabrice Mériaudeau.  Deep multimodal fusion for semantic image segmentation: A survey.  Image Vis. Comput., 105:104042, 2021.
  * [53] Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Qinghua Hu, and Haibin Ling.  Vision meets drones: Past, present and future.  CoRR, abs/2001.06303, 2020.

Supplementary material for ’Comprehensive Analysis of the Object Detection
Pipeline on UAVs’

Leon Amadeus Varga
Cognitive Systems
University of Tuebingen, Germany
<EMAIL_ADDRESS>Sebastian Koch
Cognitive Systems
University of Tuebingen, Germany
<EMAIL_ADDRESS>Andreas Zell
Cognitive Systems
University of Tuebingen, Germany
<EMAIL_ADDRESS>
The supplementary material contains additional information and example images
for the camera parameters. Further, the experiment setups are described in
more detail. We also point to the full list of experiments and code for a
preprocessing pipeline, which allows evaluating the parameters in other
applications. Both can be found on https://github.com/cogsys-tuebingen/cp_eval
.

##  8 Experiments

In this section, we describe the setup of the experiments. The setup contains
the used datasets, the object detector models and the hardware environment.
For each parameter configuration, we trained with three different random
seeds. The random seed affects the weight initialization, the order of the
training samples and data augmentation techniques. The three initialization
show the stability of parameter configurations.

###  8.1 Datasets

We evaluated our experiments on three data sets. All data sets are part of the
remote sensing scenario. DOTA-2 is based on satellites recordings [13].
VisDrone [53] and SeaDronesSee [43] were recorded by Micro Air Vehicles
(MAVs). The first two datasets are well established and commonly used to
benchmark object detection for remote sensing. In contrast, SeaDronesSee is
not completely established, but it focuses on a maritime environment, which is
very different to the urban environment of VisDrone and DOTA-2. Therefore, it
leads to other challenges for the object detectors and is a proper extension
for our experiments.

#### DOTA-2

Ding _et al_. provide high-resolution satellite images with annotations [13].
We used the recommended preprocessing technique to create 1024 $\times$ 1024
tiles of the high-resolution recordings. All the recordings are provided in
the lossless PNG format. Some images contain JPEG compression artifacts, so we
assume that not the entire dataset creation pipeline was lossless.

#### SeaDronesSee-DET

SeaDronesSee focus on search and rescue in maritime environments [43]. The
training set includes 2,975 images with 21,272 annotations. We reduced the
maximum side length of the images to 1024 pixels for this data set.
The reported accuracy is evaluated on the test set (with 1,796 images). This
data set uses the PNG format [4], which utilizes a lossless compression.

#### VisDrone-DET

Zhu _et al_. proposed one of the most prominent UAV recordings data set [53].
The focus of this data set is traffic surveillance in urban areas. We use the
detection task (VisDrone-DET). The training set contains 6,471 images with
343,205 annotations. Unless otherwise mentioned, we reduced the maximum side
length by down-scaling to 1024 pixels.
We used the validation set for evaluation. The validation set contains 548
images. All images of this data set are provided in JPEG format [45] with a
compression quality of 95.

###  8.2 Models

For our experiments, we selected four different object detectors. The two-
stage detector Faster-RCNN [34] can still achieve state-of-the-art results
with smaller modifications [53]. By way of comparison, the one-stage detectors
are often faster than the two-stage detectors. Yolov4 [3] and EfficentDet [41]
are one-stage detectors and focus on efficiency. Both are designed to perform
well on embedded hardware. Modified versions of CenterNet [14], also a one-
stage detector, could achieve satisfying results in the VisDrone challenge
[53]. These four models cover a variety of approaches and can give a reliable
impression of the parameter impact.
For Faster R-CNN, EfficentDet and CenterNet, we used multiple backbones to
provide an insight for different network sizes. In the following paragraphs,
the networks and training setups are described in further detail.

Figure 11: Image Size: CenterNet trained with different training image sizes
and validated on $\text{validation image size}=x\times\text{training image
size}$. The y-axis shows the mean Average Precision (mAP). Figure 12: Image
Size: EfficientDet trained with different training image sizes and validated
on $\text{validation image size}=x\times\text{training image size}$. The
y-axis shows the mean Average Precision (mAP). Figure 13: Image Size: Faster
R-CNN trained with different training image sizes and validated on
$\text{validation image size}=x\times\text{training image size}$. The y-axis
shows the mean Average Precision (mAP). Figure 14: Image Size: YoloV4 trained
with different training image sizes and validated on $\text{validation image
size}=x\times\text{training image size}$. The y-axis shows the mean Average
Precision (mAP).

#### CenterNet

Duan _et al_. proposed CenterNet [14], an anchor-free object detector. The
network uses a heat-map to predict the center-points of the objects. Based on
these center-points, the bounding boxes are regressed. Hourglass-104 [30] is a
representative for extensive backbones, while the ResNet-backbones [17] cover
a variety of different backbone sizes. The ResNet backbones were trained with
Adam and a learning rate of $1e^{-4}$. Further, we also used the plateau
learning scheduler. For the Hourglass104-backbone, we used the learning
schedule proposed by Pailla _et al_. [32].

#### EfficientDet

EfficientDet is optimized for efficiency and can perform well with small
backbones [41]. Even there is an EfficentNetV2 announced, which should be more
efficient as backbone [40], there is currently, to the best of our knowledge,
no object detector using this backbone. For our experiments, we used
EfficientDet with two backbones. The $d0$-backbone is the smallest and fastest
of this family. And the $d4$-backbone represents a good compromise between
size and performance. We used three anchor scales (0.6, 0.9, 1.2), which are
optimized to detect the small objects of the data sets. For the optimization,
we used an Adam optimizer [22] with a learning rate of $1e^{-4}$. Further, we
used a learning rate scheduler, which reduces the learning rate on plateaus
with patience of 3 epochs.

#### Faster R-CNN

Faster R-CNN is the most famous two-stage object detector [34]. And many of
its improvements achieve today still state-of-the-art results [53]. We use
three backbones for Faster R-CNN. A ResNet50 and a ResNet101 of the ResNet-
family [17]. Also, a ResNeXt101 backbone [47] For Faster R-CNN, we use the
Adam optimizer [22] with a learning rate of $1e^{-4}$ and a plateau scheduler.

#### Yolov4

Bochkovskiy _et al_. published YoloV4 [3], which is the latest member of the
Yolo-family providing a scientific publication. Besides a comprehensive
architecture and parameter search, they did an in-depth analysis of
augmentation techniques, called ’bag of freebies’, and introduced the Mosaic
data augmentation technique. YoloV4 is a prominent representative of the
object detectors because of impressive results on MS COCO. By default, YoloV4
scales all input images down to an image size of 608$\times$608 pixels. For
our experiments, we removed this preprocessing to improve the prediction of
smaller objects.

###  8.3 Hardware setup

In the following, we describe the setups, which were used for the experiments.
Most of the experiments were done in a desktop environment. To evaluate the
performance in a more applicable environment, we performed experiments on an
Nvidia Xavier AGX board. The small size and the low weight allows the usage
for onboard processing in MAVs or other robots.

#### Desktop environment

The models were trained on computing nodes equipped with four GeForce GTX 1080
Ti graphic cards. The system was based on the Nvidia driver (version 460.67),
CUDA (version 11.2) and PyTorch (version 1.9.0). To evaluate the inference
speed in a desktop environment, we used a single RTX 2080 Ti with the same
driver configuration.

#### Embedded environment

The Nvidia Xavier AGX provides 512 Volta GPU cores, which are the way to go
for the excessive forward passes of the neural networks. Therefore, it is a
flexible way to bring deep learning approaches into robotic systems. We used
the Nvidia Jetpack SDK in the version 4.6, which is shipped with TensorRT
8.0.1. TensorRT can speed up the inference of trained models. It optimizes
these for the specific system, making it a helpful tool for embedded
environments. For all embedded experiments, the Xavier board was set to the
power mode ’MAXN’, which consumes around 30W and utilizes all eight CPUs.
Further, it makes use of the maximum GPU clock rate of around 1377 MHz.

##  9 Example images for the considered factors of influence

Further example images for the considered factors of influence can be found
here.

###  9.1 Quantization

In Fig. 15, the missing quantization steps for 4 Bit and 2 Bit are visible by
the human eye. Already, the 4bit quantization seems not smooth with unwanted
steps.

(a) 8 Bit

(b) 4 Bit

(c) 2 Bit

Figure 15: Example images for Quantization

###  9.2 Compression

In Fig. 16 the influence of different compression qualities is visible. For
the compression qualities, which were used for the evaluation (No compression,
90, 70), it is hard to see differences with the human eye. Therefore, the
additional compression qualities (50, 20, 10) are presented. Especially, for
the lowest used compression quality (10) the typical JPEG compression
artifacts are visible (unsharp corners, squares with incorrect color).

(a) No comp.

(b) 90 comp. qlty.

(c) 70 comp. qlty.

(d) 50 comp. qlty.

(e) 20 comp. qlty.

(f) 10 comp. qlty.

Figure 16: Example images for JPEG compression

###  9.3 Calibration Parameters

The amount of distortion is visualized in the figure using a checkerboard
pattern.

Figure 17: Checkboard pattern with distortion

(a)

(b) Rectified

(c)

Figure 18: Example images for Camera Calibration

The Fig. 18 shows the rectified image and two not optimal setups. In (a), the
image is overexposed and the barrel distortion of the lens was not fully
corrected. In setup (c), the image is underexposed, and the recording is
affected by a pincushion distortion. In a real world image, the distortions
are hard to see. They are only visible for straight lines. In contrast, the
incorrect exposure, which correlates with the gamma value, is obvious. In Fig.
19 the breakdown of the image distortion impact on the different data sets and
models is visualized. Fig. 20 shows the same for the impact of gamma.

Figure 19: Image Distortion: the impact of the image distortion on the
performance of the object detectors for all data sets. The y-axis shows the
mean Average Precision (mAP). Figure 20: Image Gamma: the impact of the gamma
on the performance of the object detectors for all data sets. The y-axis shows
the mean Average Precision (mAP).

(a) Full res.

(b) Max 768 pix.

(c) Max 512 pix.

(d) Max 256 pix.

(e) Max 128 pix.

Figure 21: Example images for image resolutions with max side length

###  9.4 Resolution

In Fig. 21 an image with different resolutions is shown. For smaller
resolution, the detection of smaller objects is especially hard. These can
vanish completely. Large and medium objects normally lose only features, which
makes the detection harder but not impossible. Therefore, the impact on
smaller objects is higher.

##  10 Results and discussion

The full list of experiments can be found on https://github.com/cogsys-
tuebingen/cp_eval . Here are some additional information regarding the
experiment results.

###  10.1 Quantisation

###  10.2 Resolution

In Fig. 11, Fig. 12, Fig. 13 and Fig. 14, the behavior of the different models
for the tested image resolutions is visible. All models follow the trend of
Faster RCNN, which was described in the paper. EfficientDet seems more
sensitive to resolution changes than the other models.

###  10.3 Gamma Correction

###  10.4 Image Distortion

##  11 Application

For the **) evaluations, we used a maximal side length of 1280 pixels. We used
therefore the CroW technique [44] in the default configuration. These are the
only changes for this evaluation.

## References

  * [1] M. Abdullah-Al-Wadud, Md. Hasanul Kabir, M. Ali Akber Dewan, and Oksam Chae.  A dynamic histogram equalization for image contrast enhancement.  IEEE Transactions on Consumer Electronics, 53(2):593–600, 2007\.
  * [2] Henryk Blasinski, Joyce E. Farrell, Trisha Lian, Zhenyi Liu, and Brian A. Wandell.  Optimizing image acquisition systems for autonomous driving.  electronic imaging, 2018:161–1–161–7, 2018.
  * [3] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  CoRR, abs/2004.10934, 2020.
  * [4] Thomas Boutell.  PNG (portable network graphics) specification version 1.0.  RFC, 2083:1–102, 1997.
  * [5] Duane C Brown.  Decentering distortion of lenses.  Photogrammetric Engineering and Remote Sensing, 1966.
  * [6] Mark Buckler, Suren Jayasuriya, and Adrian Sampson.  Reconfiguring the imaging pipeline for computer vision.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 975–984. IEEE Computer Society, 2017.
  * [7] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer.  Zeroq: A novel zero shot quantization framework.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13166–13175. Computer Vision Foundation / IEEE, 2020.
  * [8] Sebastian Candiago, Fabio Remondino, Michaela De Giglio, Marco Dubbini, and Mario Gattelli.  Evaluating multispectral images and vegetation indices for precision farming applications from uav images.  Remote sensing, 7(4):4026–4047, 2015.
  * [9] Alexandra Carlson, Katherine A. Skinner, Ram Vasudevan, and Matthew Johnson-Roberson.  Modeling camera effects to improve visual learning from synthetic data.  In Laura Leal-Taixé and Stefan Roth, editors, Computer Vision - ECCV 2018 Workshops - Munich, Germany, September 8-14, 2018, Proceedings, Part I, volume 11129 of Lecture Notes in Computer Science, pages 505–520. Springer, 2018.
  * [10] R. Cucchiara, C. Grana, M. Piccardi, A. Prati, and S. Sirotti.  Improving shadow suppression in moving object detection with hsv color information.  In ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No.01TH8585), pages 334–339, 2001.
  * [11] Joseph A. Curcio and Charles C. Petty.  The near infrared absorption spectrum of liquid water.  J. Opt. Soc. Am., 41(5):302–304, May 1951.
  * [12] Lei Deng, Zhihui Mao, Xiaojuan Li, Zhuowei Hu, Fuzhou Duan, and Yanan Yan.  Uav-based multispectral remote sensing for precision agriculture: A comparison between different cameras.  ISPRS journal of photogrammetry and remote sensing, 146:124–136, 2018.
  * [13] Jian Ding, Nan Xue, Gui-Song Xia, Xiang Bai, Wen Yang, Michael Ying Yang, Serge J. Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang.  Object detection in aerial images: A large-scale benchmark and challenges.  CoRR, abs/2102.12219, 2021.
  * [14] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 6568–6577. IEEE, 2019.
  * [15] A. Gersho.  Quantization.  IEEE Communications Society Magazine, 15(5):16–16, 1977.
  * [16] Dayan Guan, Yanpeng Cao, Jiangxin Yang, Yanlong Cao, and Michael Ying Yang.  Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection.  Inf. Fusion, 50:148–157, 2019.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, pages 770–778. IEEE Computer Society, 12 2016.
  * [18] R.W.G. Hunt.  The Reproduction of Colour.  The Wiley-IS&T Series in Imaging Science and Technology. Wiley, 2005\.
  * [19] Photography — Digital cameras — Geometric distortion (GD) measurements.  Standard, International Organization for Standardization, Geneva, CH, July 2015.
  * [20] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018.
  * [21] Takumi Karasawa, Kohei Watanabe, Qishen Ha, Antonio Tejero-de-Pablos, Yoshitaka Ushiku, and Tatsuya Harada.  Multispectral object detection for autonomous vehicles.  In Wanmin Wu, Jianchao Yang, Qi Tian, and Roger Zimmermann, editors, Proceedings of the on Thematic Workshops of ACM Multimedia 2017, Mountain View, CA, USA, October 23 - 27, 2017, pages 35–43. ACM, 2017.
  * [22] Diederik P Kingma and Jimmy Lei Ba.  Adam: A method for stochastic optimization.  In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 2015.
  * [23] Raghuraman Krishnamoorthi.  Quantizing deep convolutional networks for efficient inference: A whitepaper.  arXiv preprint arXiv:1806.08342, 2018.
  * [24] Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan.  Fully quantized network for object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 2810–2819. Computer Vision Foundation / IEEE, 2019.
  * [25] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740–755. Springer, 2014.
  * [26] Kang Liu and Gellért Máttyus.  Fast multiclass vehicle detection on aerial images.  IEEE Geosci. Remote. Sens. Lett., 12(9):1938–1942, 2015.
  * [27] Zhenyi Liu, Trisha Lian, Joyce E. Farrell, and Brian A. Wandell.  Soft prototyping camera designs for car detection based on a convolutional neural network.  In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pages 2383–2392. IEEE, 2019.
  * [28] Zhenyi Liu, Trisha Lian, Joyce E. Farrell, and Brian A. Wandell.  Neural network generalization: The impact of camera parameters.  IEEE Access, 8:10443–10454, 2020.
  * [29] Martin Messmer, Benjamin Kiefer, and Andreas Zell.  Gaining scale invariance in UAV bird’s eye view object detection by adaptive resizing.  CoRR, abs/2101.12694, 2021.
  * [30] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9912 LNCS, pages 483–499, 2016.
  * [31] Tanguy Ophoff, Kristof Van Beeck, and Toon Goedemé.  Exploring rgb+depth fusion for real-time object detection.  Sensors, 19(4):866, 2019.
  * [32] Dheeraj Reddy Pailla, Varghese Alex Kollerathu, and Sai Saketh Chennamsetty.  Object detection on aerial imagery using centernet.  CoRR, abs/1908.08244, 2019.
  * [33] Charles Poynton.  Digital video and HD: Algorithms and Interfaces.  Elsevier, 2012.
  * [34] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  IEEE Trans. Pattern Anal. Mach. Intell., 39(6):1137–1149, 2017\.
  * [35] Kmeid Saad and Stefan-Alexander Schneider.  Camera vignetting model and its effects on deep neural networks for object detection.  In 2019 IEEE International Conference on Connected Vehicles and Expo, ICCVE 2019, Graz, Austria, November 4-8, 2019, pages 1–5. IEEE, 2019.
  * [36] Francesco Secci and Andrea Ceccarelli.  On failures of RGB cameras and their effects in autonomous driving applications.  In Marco Vieira, Henrique Madeira, Nuno Antunes, and Zheng Zheng, editors, 31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020, Coimbra, Portugal, October 12-15, 2020, pages 13–24. IEEE, 2020.
  * [37] Jianhong Shen.  On the foundations of vision modeling: I. weber’s law and weberized tv restoration.  Physica D: Nonlinear Phenomena, 175(3):241–251, 2003.
  * [38] Li Shuhua and Guo Gaizhi.  The application of improved hsv color space model in image processing.  In 2010 2nd International Conference on Future Computer and Communication, volume 2, pages V2–10–V2–13, 2010.
  * [39] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, 2019.
  * [40] Mingxing Tan and Quoc V. Le.  Efficientnetv2: Smaller models and faster training.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10096–10106. PMLR, 2021.
  * [41] Mingxing Tan, Ruoming Pang, and Quoc V. Le.  Efficientdet: Scalable and efficient object detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10778–10787. Computer Vision Foundation / IEEE, 2020.
  * [42] Maarten Vandersteegen, Kristof Van Beeck, and Toon Goedemé.  Real-time multispectral pedestrian detection with a single-pass deep neural network.  In Aurélio Campilho, Fakhri Karray, and Bart M. ter Haar Romeny, editors, Image Analysis and Recognition - 15th International Conference, ICIAR 2018, Póvoa de Varzim, Portugal, June 27-29, 2018, Proceedings, volume 10882 of Lecture Notes in Computer Science, pages 419–426. Springer, 2018.
  * [43] Leon Amadeus Varga, Benjamin Kiefer, Martin Messmer, and Andreas Zell.  Seadronessee: A maritime benchmark for detecting humans in open water.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2260–2270, 2022.
  * [44] Leon Amadeus Varga and Andreas Zell.  Tackling the background bias in sparse object detection via cropped windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 2768–2777, October 2021.
  * [45] Gregory K. Wallace.  The JPEG still picture compression standard.  Commun. ACM, 34(4):30–44, 1991.
  * [46] Chuan-Wei Wang, Chin-An Cheng, Ching-Ju Cheng, Hou-Ning Hu, Hung-Kuo Chu, and Min Sun.  Augpod: Augmentation-oriented probabilistic object detection.  In CVPR Workshop on the Robotic Vision Probabilistic Object Detection Challenge, 2019.
  * [47] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [48] Saeed Yahyanejad, Jakub Misiorny, and Bernhard Rinner.  Lens distortion correction for thermal cameras to improve aerial imaging with small-scale uavs.  In 2011 IEEE International Symposium on Robotic and Sensors Environments, ROSE 2012, Montréal, Canada, September 17-18, 2011, pages 231–236. IEEE, 2011.
  * [49] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat.  Q8bert: Quantized 8bit bert.  In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 36–39. IEEE, 2019.
  * [50] Liyuan Zhang, Huihui Zhang, Yaxiao Niu, and Wenting Han.  Mapping maize water stress based on uav multispectral remote sensing.  Remote Sensing, 11(6):605, 2019.
  * [51] Lili Zhang, Yi Zhang, Zhen Zhang, Jie Shen, and Huibin Wang.  Real-time water surface object detection based on improved faster r-cnn.  Sensors, 19(16):3523, 2019.
  * [52] Yifei Zhang, Désiré Sidibé, Olivier Morel, and Fabrice Mériaudeau.  Deep multimodal fusion for semantic image segmentation: A survey.  Image Vis. Comput., 105:104042, 2021.
  * [53] Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Qinghua Hu, and Haibin Ling.  Vision meets drones: Past, present and future.  CoRR, abs/2001.06303, 2020.