# Autonomous Navigation of Micro Air Vehicles in Warehouses Using Vision-based
Line Following

Ling Shuang Soh, and Hann Woei Ho* *Corresponding authorLing Shuang Soh and
Hann Woei Ho are with School of Aerospace Engineering, Engineering Campus,
Universiti Sains Malaysia, 14300 Nibong Tebal, Pulau Pinang, Malaysia (email:
<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a vision-based solution for indoor Micro Air Vehicle
(MAV) navigation, with a primary focus on its application within autonomous
warehouses. Our work centers on the utilization of a single camera as the
primary sensor for tasks such as detection, localization, and path planning.
To achieve these objectives, we implement the HSV color detection and the
Hough Line Transform for effective line detection within warehouse
environments. The integration of a Kalman filter into our system enables the
camera to track yellow lines reliably. We evaluated the performance of our
vision-based line following algorithm through various MAV flight tests
conducted in the Gazebo 11 platform, utilizing ROS Noetic. The results of
these simulations demonstrate the system capability to successfully navigate
narrow indoor spaces. Our proposed system has the potential to significantly
reduce labor costs and enhance overall productivity in warehouse operations.
This work contributes to the growing field of MAV applications in autonomous
warehouses, addressing the need for efficient logistics and supply chain
solutions.

###### Index Terms:

Vision-based Control, Kalman Filter, Indoor Navigation, Warehouse Operations,
Micro Air Vehicles.

##  I Introduction

Micro air vehicles (MAVs) are unmanned aerial vehicles (UAVs) characterized by
their small size, lightweight, and agility. They have gained popularity in
military and civilian applications, including aerial photography, recreational
use, and competitive drone racing. MAVs can navigate tight spaces and capture
high-resolution images or videos from unique perspectives. They have also
gained importance as a valuable tool for victim search and rescue,
surveillance operations, scientific research endeavors, and reconnaissance
missions [1].

Traditionally, warehouse tasks, such as goods reception, storage, picking, and
dispatch have relied on manual labor by human operators. However, this
approach is often time-consuming, labor-intensive, and prone to errors. The
concept of autonomous warehouses has gained significant traction as a
transformative solution for improving efficiency and productivity in the
logistics and supply chain industry. Autonomous warehouses are characterized
by the integration of advanced technologies, such as robotics, artificial
intelligence (AI), and automation systems, to enable self-operating and self-
optimizing capabilities [2]. Systems include autonomous robots, automated
guided vehicles (AGVs), and conveyor systems that work collaboratively to
handle inventory movement and management [3].

The use of MAVs in warehouse environments has garnered significant attention
as a potential solution for enhancing efficiency and optimizing operations.
One specific application that has emerged is the development of line-following
MAVs in warehouses [4]. These MAVs are specifically designed to navigate
predetermined paths or lines on the warehouse floor to perform tasks such as
inventory management, item tracking, and monitoring. The introduction of line-
following MAVs also enables the automation of tasks, resulting in enhanced
accuracy, reduced labor costs, and improved overall productivity in warehouse
operations.

Line-following MAVs typically employ a combination of sensors, onboard
cameras, and intelligent algorithms to detect and track lines or paths on the
warehouse floor. Computer vision techniques are utilized to analyze visual
information captured by the onboard cameras, enabling the identification of
the line or path. The MAVs then adjust their flight trajectory to maintain
alignment with the designated path, ensuring accurate navigation.

##  II Related Works

Research on agile flight, including vision-based localization and control
algorithms, has been conducted through a combination of experimental and
computational approaches. Experimental investigations typically involve real-
world environments that are scaled down and equipped with various sensors to
capture relevant data [5]. Researchers use these setups to evaluate the
performance of agile flight algorithms and assess their effectiveness in
localization and control.

Indoor localization technology refers to the methods and techniques used to
determine the precise location of objects or individuals within indoor
environments where GPS signals may not be available or reliable. There are
various indoor localization techniques, including angle of arrival (AoA), time
of flight (ToF), return time of flight (RTOF), and received signal strength
(RSS). It covers a wide range of technologies utilized in indoor localization,
such as Wireless Fidelity (Wi-Fi), RFID, ultra-wideband (UWB), and Bluetooth
[6, 7].

One of the techniques, the RSS-based approach is a popular and straightforward
method employed for indoor localization. It relies on measuring the strength
of the signal received at the receiver. By utilizing RSS, it becomes possible
to get the distance between a transmitter (Tx) and a receiver (Rx), providing
valuable information for indoor localization purposes [6]. The signal strength
received by the receiver exhibits variations over time, particularly in
relation to the relative position of the transmitter and receiver [8].
However, the RSS-based approach encounters significant performance degradation
in complex scenarios characterized by factors such as multipath fading and
temporal dynamics [9].

Other than the techniques and technologies mentioned earlier, vision-based
localization is also one of the methods for indoor localization. Vision-based
localization is a technique that relies on visual information captured by
cameras to estimate the position and orientation of a device within an indoor
environment. It involves analyzing the visual data to extract relevant
features or landmarks, such as objects, patterns, or markers, which can be
used as reference points for localization. Vision-based solutions do provide
the advantages of portability, compactness, cost-effectiveness, and power
efficiency [10]. The proposed method involves extracting points from images
captured by a camera using the Scale Invariant Feature Transform (SIFT)
algorithm. Landmark feature points with distinctive descriptor vectors are
selected and their locations are detected and stored in a map database to
estimate the position of the UAV. Furthermore, the utilization of Speed-Up
Robust Features (SURF) techniques in vision-based localization helps decrease
the storage overhead of the database, resulting in reduced latency in real-
world applications [11].

Furthermore, more and more autonomous warehouses are gradually replacing
traditional warehouses to improve efficiency and productivity in the logistics
and supply chain industry, also indirectly reducing labor costs [12].
Autonomous warehouses are characterized by the integration of advanced
technologies, such as robotics, AI, and automation systems, to enable self-
operating and self-optimizing capabilities [2]. Systems include autonomous
robots, AGVs, and conveyor systems that work collaboratively to handle
inventory movement and management [3]. One of the classic modes of autonomous
warehouse is the cooperation between Unmanned Ground Vehicle (UGV) and UAV.
Since the warehouse inventory is conducted indoors where GPS cannot be used,
the UGV is employed as a ground reference for the UAV to overcome expensive
and impractical indoor localization techniques [13].

Integrated systems like EyeSee and Infinium Scan have been developed as
Unmanned Aerial Systems (UAS) platforms tailored for warehouse environments
and the purpose of inventory tracking [14]. One notable example is the world’s
foremost furniture manufacturer and retail chain, IKEA had employs drones to
perform inventory checks. IKEA has implemented drones in five of its warehouse
stores and one distribution center, starting in December 2021. In the IKEA
Malmo and Padua setups, prototype EyeSee drones from Hardis were utilized,
which is particularly suitable for distribution centers that require less
frequent inventory counting and where inventory accuracy is generally of
lesser importance [15].

Figure 1: Semi-autonomous drone “EyeSee” used in IKEA Padua warehouse. [15]

Another prominent example of autonomous robot systems is the most famous one,
Amazon robotics solution, which has 520,000 mobile drive unit robots
worldwide. Various AGVs and Autonomous Mobile Robots (AMRs) such as Pegasus,
Kiva, Hercules, and Proteus are utilized in these systems. Warehouse mapping,
object detection, and navigation in Amazon warehouses rely on the utilization
of RFID scanners and Light Detection and Ranging (LiDAR) [16]. The study also
showed that by implementing parallel execution of the placing operation and
vision recognition algorithms, the mission time was reduced from 42 seconds to
31 seconds compared to human work [17].

Figure 2: Shelving units carried by an Amazon Proteus. [16]

The control aspect of agile flight presents significant challenges due to the
high speed at which MAVs operate. The fast speed introduces nonlinear
dynamics, complex aerodynamic effects, and constraints on the actuators that
control the vehicle’s motion [18]. To address these challenges, researchers
have explored different control strategies and compared their performance [19,
20].

In a study, the authors compared the performance of two control methods:
Nonlinear Model Predictive Control (NMPC) and Differential-flatness-based
controller (DFBC) [18]. Flight tests were conducted in both real-world
scenarios and simulations. The results indicated that the NMPC effectively
handled infeasible trajectories through future prediction. Additionally, when
combined with the Incremental Nonlinear Dynamic Inversion (INDI) method, both
controllers achieved highly aggressive trajectories within feasible limits.

The INDI method is commonly used in various controllers to improve output
efficiency, managing nonlinearities and disturbances during large-angle
operations while reducing sensitivity to model mismatch [21]. With the
enhanced INDI method, MAVs gain precise control and maneuverability,
facilitating navigation of complex flight paths and agile maneuvers [22, 23].

Line-following control entails a robotic system’s ability to detect and
accurately track a predefined path or landmark [24]. This control method uses
sensors like cameras or infrared sensors to capture visual data from the path.
The system processes this data and generates control signals for steering the
robot or UAV in alignment with the path. The control algorithm continuously
calculates necessary adjustments, utilizing feedback mechanisms and
Proportional Integral Derivative (PID) control techniques [25] [26]. Due to
the limited computational power, line followers typically avoid expensive line
detection algorithms and hence reduce the computational cost [27].

##  III Line Detection and Following

The focus of this study is to address the challenges of localizing and
navigating MAVs in environments where GPS signals are not accessible. To
overcome this limitation, a vision-based line-following algorithm is employed.
The algorithm enables the MAV to actively determine its own position by
analyzing the received data from its camera. These data include line detection
algorithms, which are then fused together using a Kalman filter, leveraged
from our previous work [28], to estimate the state of the target according to
previous information captured by the camera.

Once the MAVs position is estimated, it is utilized by the flight control unit
to guide the MAVs navigation. This integration of vision-based localization
and flight control allows the MAV to autonomously follow the designated flight
path. To validate the effectiveness of the algorithm, simulations have been
conducted using the IRIS copter in the Gazebo 11 environment, providing a
realistic virtual platform for testing, and evaluating the performance of the
MAV localization and navigation capabilities.

###  III-A Vision Based Detection

This project utilizes vision-based detection, which involves using cameras or
visual sensors to identify and analyze objects or patterns in an environment.
It captures visual data and processes it using computer vision algorithms to
extract valuable information, relying on visual cues like color, shape,
texture, and motion to detect and classify objects.

Vision-based detection offers advantages like rich information in visual data,
and real-time analysis for object recognition, tracking, and localization. It
enables systems to understand and interact with their surroundings
effectively. Advancements in algorithms, deep learning, and camera sensors
continuously enhance this technology, opening new possibilities for visual
understanding applications. After collecting image or video data, the color
detection algorithm processes it. This algorithm identifies pixels
corresponding to a specific color, turning them white (binary 1), while the
rest becomes black (binary 0). This simplifies the image for further
processing.

Figure 3: A $10\times 10$ binary image which is shown in black and white. [29]

HSV color detection is used to identify and extract specific colors from a
camera’s video stream based on Hue, Saturation, and Value (HSV) components.
Unlike RGB, which uses red, green, and blue channels, HSV separates colors
into three components for intuitive color-based analysis. In HSV, Hue
represents color (0-360 degrees), Saturation measures intensity, and Value
indicates brightness. Lower and upper HSV ranges are set to capture a broader
range of colors due to environmental factors. For example, the HSV values for
yellow could be [18, 94, 140] to [48, 255, 255], with adjustments as needed.
This method sets acceptable HSV value ranges to detect desired colors. Pixels
in an image or video are compared to these thresholds, determining if they
belong to the desired color range. This approach enhances MAV localization and
tracking by isolating specific colors accurately, offering flexibility in
object identification based on color characteristics.

Figure 4: HSV color detection on yellow line image.

The algorithm has a drawback, as it can introduce noise into the image,
affecting the control system. To enhance the output, morphological operations
are applied to eliminate noise. Morphology in image processing aims to remove
various forms of noise and textures while preserving the shapes and sizes of
larger objects. The key processes are erosion and dilation. Dilation sets the
output pixel’s value to the maximum, turning it to 1 in binary if neighboring
pixels are also 1, making lines thicker and objects more prominent. Erosion,
on the other hand, sets the output pixel to 0 in binary, eliminating narrow
lines and floating pixels. Once noise is removed, edge detection locates
boundaries in the video stream by analyzing intensity variations. It
identifies areas with significant pixel value changes, often representing
object or region boundaries. The Canny edge detector is used in this study,
among other techniques.

The Hough lines transform is employed to detect lines, even when parts are
missing. This is valuable for scenarios like dashed road lines or partially
obscured lines. The maximum line gap parameter is set as 50 to allow for the
inclusion of line segments that have gaps between them. Line segments within
this distance will be connected and considered as a single line. The centroid
in the x-direction (cx) and y-direction (cy) is computed by finding the
average number of accumulated detected lines.

###  III-B Detection using Kalman Filter

The Kalman filter is a mathematical algorithm used for estimating the state of
a system by combining measurements from a camera view with predictions from a
mathematical model. It provides an optimal estimation of the system’s state,
even in the presence of noise or uncertain measurements. The basic concept
behind the Kalman filter is to iteratively update the estimated state of a
system by combining a prediction based on the system dynamics and an
observation obtained from camera measurements. The filter considers the
uncertainty associated with both the predictions and the measurements to
compute the optimal estimate of the system state. When it comes to line
detection, the Kalman filter is used to track the position and trajectory of a
line in the video sequence. The working principle of the Kalman filter is
shown mathematically below.

To estimate the state of discrete-time controlled process,

$x_{k+1}=A_{k}x_{k}+Bu_{k}+w_{k}$ (1)

with the measurement of

$z_{k}=H_{k}x_{k}+v_{k}$ (2)

where $w_{k}$ and $v_{k}$ are the process noise and measurement noise, and
they are independent. $A_{k}$ is an $n\times n$ matrix that relates the state
at time step $k$ to the time step $k+1$, and $B$ is a $n\times 1$ matrix that
relates to control input and the state $x$. Matrix $H$ is in $m\times n$ which
relates to the state of measurement $z_{k}$.

Next, a priori estimate error covariance is defined as

$P^{-}_{k}=E[e^{-}_{k}e^{-T}_{k}]$ (3)

and posteriori estimate error covariance is

$P_{k}=E[e_{k}e^{T}_{k}]$ (4)

where $e_{k}^{-}$ and $e_{k}$ are priori and posteriori estimate errors
respectively. The combination of priori estimate, $x_{k}^{-}$, and measurement
innovation, which is the difference between the actual measurement, $z_{k}$,
and measurement prediction, $H_{k}x_{k}^{-}$ is shown in (5).

$\hat{x}_{k}=\hat{x}_{k}^{-}+K\left(z_{k}-H_{k}\hat{x}_{k}^{-}\right)$ (5)

$K$ is an $n\times m$ matrix and it is the Kalman gain that minimizes
posteriori error covariance. One of the ways of $K$ minimizes posteriori error
is shown in (6.

$K_{k}=P_{k}^{-}H_{k}^{T}(H_{k}P_{k}^{-}H_{k}^{T}+R_{k})^{-1}$ (6)

The final equations for time and measurement updates are shown below:

Time update

$\hat{x}_{k+1}^{-}=A_{k}\hat{x}_{k}+Bu_{k}$ (7)
$P_{k+1}^{-}=A_{k}P_{k}A_{k}^{T}+Q_{k}$ (8)

Measurement update

$K_{k}=P_{k}^{-}H_{k}^{T}\left(H_{k}P_{k}^{-}H_{k}^{T}+R_{k}\right)^{-1}$ (9)
$\hat{x}_{k}=\hat{x}_{k}^{-}+K\left(z_{k}-H_{k}\hat{x}_{k}^{-}\right)$ (10)
$P_{k}=\left(I-K_{k}H_{k}\right)P_{k}^{-}$ (11)

Equations (7) and (8) show the state estimate and covariance estimate from
time step $k$ to time step $k+1$. The Kalman gain, $K_{k}$ (9) needs to be
computed first during the measurement update process. Next, obtain $z_{k}$ in
equation (2) and generate posteriori sate estimate as in equation (10). The
final step will be obtaining a posteriori error covariance estimate as in
equation (11). Following each update of time and measurement, the Kalman
Filter proceeds by iteratively using the previous posteriori estimates to
predict the new priori estimates. This iterative process ensures that the
filter continuously refines its estimates based on the most recent information
available. Fig. 5 below shows the complete operation of the filter.

Figure 5: The Kalman filter algorithm used with the line detection algorithm
for the centroid estimation. [30]

###  III-C Autonomous Navigation

This project focuses on utilizing two main classes for the line-following task
with an MAV. The first class is responsible for line detection, while the
second class handles the MAV navigation.

In the line detection class, an image is captured from the MAV onboard camera
in the simulated environment. The class executes color thresholding, masking,
and centroid detection to get the appropriate direction for the MAV to
navigate. The direction is determined by evaluating the relative position of
the centroid of the identified line with respect to the MAV current position.
Based on this analysis, the class publishes corresponding commands such as
moving forward, yawing left, yawing right, or yaw to search lines in cases
where no line is detected.

The navigation class receives the direction commands published by the
detection class. By utilizing the information, the class publishes velocity
commands to the MAV, enabling it to effectively track the line and maintain
its trajectory within the environment. Upon reaching the end of the path, the
MAV initiates a search maneuver by hovering in place, actively seeking another
line to follow.

Figure 6: An overview of the workflow of the autonomous line-following MAV
between the detection nodes and navigation nodes.

##  IV Simulation

To evaluate the effectiveness of the developed algorithm, a dynamic model was
simulated using the Gazebo platform. This simulation aimed to replicate the
real flight conditions of a MAV within a virtual environment. By implementing
the algorithm in the simulation, the behavior of the model was controlled and
tested extensively, eliminating the need for a physical MAV. The use of the
simulation provided several advantages during the algorithm development
process. The STIL simulation method proved to be highly suitable for testing
purposes. It allowed for comprehensive evaluations while safeguarding the
physical MAV platform from any potential damage. Furthermore, the SITL
simulation was not affected by weather conditions, enabling testing to be
conducted at any time.

By leveraging the capabilities of the Gazebo platform and the SITL simulation
environment, the algorithm could be thoroughly assessed, refined, and
optimized without the constraints and risks associated with physical flight
tests. This approach significantly contributed to the algorithm’s development
and ensured its reliability and effectiveness in real-world MAV operations.

Several simulations are carried out in different indoor environments through
the Gazebo platform with the aid of ROS Noetic. The laptop utilized for
running simulations is equipped with an Intel(R) Core(TM) i5-9300H CPU clocked
at 2.40GHz, 8.00 GB of RAM, and features Intel(R) UHD Graphics 630. The first
environment is a path of straight line with some corners and edges while the
second tested environment is a circular path with curved turns.

###  IV-A Environment Setup

The first environment to be tested is shown in Fig. 7111Modified from
https://github.com/sudrag/line_follower_turtlebot. It is a simple warehouse
environment built up in a small space, equipped with two racks and a
cluttering of boxes. The warehouse environment comprises a designated path
that consists of a straight line with occasional corners. The straight-line
sections allow for streamlined movement, while the corners introduce essential
elements of navigation and pose specific challenges for the MAV control
system.

Figure 7: Simulation Environment 1: A warehouse environment with a straight-
lines path.

The second simulation environment to be tested is shown in Fig. 8222Modified
from https://github.com/DougUOW/line_follower_pkg/tree/master. It is a simple
warehouse environment but with a circular path. Arcs can have varying radii,
and the curvature of the line changes as the MAV moves. This requires the MAV
to constantly adapt its turning radius and adjust its yawing angle to
accurately follow the curve. It becomes more challenging for the MAV control
system to maintain a precise trajectory while accounting for the changes.

Figure 8: Simulation Environment 2: A warehouse environment with a circular
path.

###  IV-B Performance Evaluation

In order to get a more robust performance and result, the evaluation for the
line detection algorithm is tested before running the SITL simulation in
warehouse environments. Measurements such as HSV value and frame rate per
second (FPS) are the methods chosen to test the performance of the Hough Line
Transform HSV color detection employed in this study. The measurements are
carried out in a simple yellow path environment as shown in Fig. 9.

Figure 9: A simple yellow line environment in the Gazebo platform was used to
test the robustness of the line detection algorithm.

Frame rate per second (FPS) is a measure of how many individual frames or
images are displayed or captured in one second of time. It is used to describe
the smoothness and fluidity of motion in video or animation. In the context of
line detection algorithms, FPS plays a crucial role in real-time applications
where high frame rates are desired for accurate and timely results. The frame
rate of a line detection algorithm determines how quickly it can process
incoming frames and produce line detection results. Higher frame rates allow
for smoother and more responsive processing, which is particularly important
in real-time applications where timely decisions or actions are required. The
FPS result of the algorithm used is shown in Fig. 10.

Figure 10: Frame rates measured when running the Gazebo simulation.

The mean FPS of the Gazebo platform during simulation is only 2.80 with a
maximum of 50.02 and a minimum of 0.38. The FPS normally ranges between
0.8-1.0 but it fluctuated vigorously over a certain duration. The low FPS and
the unstable are due to factors such as hardware limitations and software
inefficiencies. Insufficient CPU and GPU power lead to the result of low FPS
when dealing with the line detection algorithm which is a computationally
intensive task. Next, the low-end graphics card struggles to handle demanding
visual computations, hence impacting the FPS. Other than that, the poor memory
allocation also results in frequent memory access delays, impacting the
processing speed. Processing high-resolution frames and working with large
datasets also significantly increases the computational load and slows down
the FPS.

The altitude of the MAV plays a crucial role in the performance of a line-
following algorithm. As the MAV altitude changes, it affects various aspects
of the line following capabilities and overall flight behavior. Firstly,
altitude directly influences the perspective and field of view (FOV) of the
onboard camera used for line detection. Different altitudes can result in
variations in the size, position, and clarity of the detected line. Secondly,
altitude affects the MAV proximity to the line being followed. As the MAV
flies higher or lower, the distance between the MAV and the line changes. This
distance alteration affects the accuracy and responsiveness of the MAV control
system. Additionally, changes in altitude affect the environmental conditions
experienced by the MAV.

Figure 11: Altitude measurements of the MAV along the flight path.

The altitude of MAV from takeoff to landing is shown in Fig. 11. The target
altitude was set to be 1 meter. Based on the plot, the altitude of MAV ranges
from 0.962m to 0.975m with a mean altitude of 0.97m. Overall, the altitude of
the MAV was maintained at a stable range and did not fluctuate violently. This
can help to maintain the line detection proximity and hence get a robust
control responsiveness.

A Kalman filter is used in this work to predict and measure the centroid of
the line detected through the onboard camera. The line detection system
obtains measurements of the line’s position, which is the centroid coordinates
from camera vision. Then, the Kalman filter will predict the next state of the
centroid based on the system dynamics model. This prediction is made by using
the previous state estimate and the dynamics equations that govern the
centroid’s motion. Next, it compares the predicted state with the actual
measurement obtained initially by updating the estimated state and covariance
matrix using the Kalman gain. Finally, the estimated centroid positions are
obtained from the Kalman filter as the output and the result is shown in Fig.
12.

Figure 12: Centroids computed from the line detection algorithm. The blue dot
represents the original centroid, while the red dot represents the centroid
estimated using the Kalman filter. Figure 13: The comparison between the
original centroid and centroid estimated using the Kalman filter.

According to Fig. 13, the trajectory of the original centroid position might
appear more erratic and noise. The lack of filter results in a scattered plot,
where the centroid position jumps around, especially in the presence of noise
or sudden changes. On the other hand, with the application of a Kalman filter,
the plotted centroid positions exhibit smoother and more consistent behavior
over time. The Kalman filter helps in reducing the effects of noise and
provides a more accurate estimate of the centroid’s position. Consequently,
the plotted points will show less variation and a more coherent trajectory. By
comparing the plots side by side, it shows that the plot with a Kalman filter
will exhibit smoother and more stable centroid positions. The plot without a
filter, on the other hand, displayed more erratic movements and larger
deviations due to the influence of noise. Overall, the Kalman filter improves
the accuracy and stability of line centroid detection by dynamically adjusting
the estimated position based on the measurements and the system’s behavior,
leading to more reliable line tracking and control.

###  IV-C Flight Tests

Fig. 14 illustrates the real flight trajectory (red) executed in the local X
and local Y plane, comparing it to the pre-defined flight path (yellow) in
Environment 1. The MAV successfully follows the pre-defined flight route,
displaying the feasibility of the HSV line detection. However, some positional
errors, particularly at the turning corners are observed, which can be
attributed to the low FPS of the feedback controller. In Fig. 16, the heading
and forward speed of the MAV are illustrated. The MAV maintains a constant
speed of 0.05 m/s throughout the flight. As shown in the figure, when the
Kalman state centroid is located in the middle of the detection frame, the MAV
moves forward at a constant speed. However, when the centroid of the detected
line shifts to the left side of the detection frame, the MAV stops receiving
velocity input and yaws to the left, resulting in a negative degree value.
Similarly, the yawing motion to the right follows a similar pattern but with a
positive degree value.

Figure 14: 2D flight trajectory of the MAV compared with the pre-defined path
in Simulation Environment 1. Figure 15: 3D flight trajectory of the MAV in
Simulation Environment 1. Figure 16: The heading and forward speed of the MAV
over time in Simulation Environment 1. When the MAV yaws to the left, the yaw
angle will be negative, and the forward speed is zero. When the MAV moves
forward, the forward speed remains constant, while when the MAV yaws to the
right, the yaw angle is positive, and the forward speed is zero.

Fig. 17 illustrates the real flight trajectory (red) in the local X and local
Y plane, comparing it to the pre-defined flight path (yellow) in Environment
2. Although the results indicate a delayed response in the yawing control
command, causing the real flight trajectory to deviate slightly from the
predetermined route, but the MAV successfully follows the pre-defined flight
route. The delay and noise of the measurement can be analyzed using the flight
data [31] to improve the adaptability of the controller in the future.

In Fig. 19, the heading and forward speed of the MAV are presented. In
comparison to the heading of the MAV in Environment 1, the heading of the MAV
in Environment 2 demonstrates a more intense yaw angle due to the varying
radii of the arc path. When the Kalman state centroid is situated in the
middle of the detection frame, the MAV moves forward at a constant speed.
However, when the centroid of the detected line shifts to the left side of the
detection frame, the MAV stops receiving velocity input and yaws accordingly.

Figure 17: 2D flight trajectory of the MAV compared with the circular pre-
defined path in Simulation Environment 2. Figure 18: 3D flight trajectory of
the MAV in Simulation Environment 2. Figure 19: The heading and forward speed
of the MAV over time in Simulation Environment 2. When the centroid detected
is in the center of the frame, the MAV receives forward velocity input but
zero yaw input. The MAV receives negative yaw input and zero forward speed
when the centroid is at the left side of the detection frame.

##  V Conclusion

This study introduces a vision-based control system that enables the
localization of a MAV and autonomous navigation along paths. The aim of this
integrated system is to facilitate the future development of MAV in autonomous
warehouses. The system utilizes an onboard camera as a detector to recognize
and determine the location of the line within an image. In this study, the HSV
color detection algorithm was chosen for color detection. The reason behind
this selection is its ability to handle variations in lighting conditions
effectively, making it suitable for real-world environments. Moreover, the HSV
color space offers a more intuitive and flexible representation of colors,
which simplifies the thresholding and segmentation process for identifying the
desired color range. The centroids of the detected lines are computed using an
edge detection technique.

Furthermore, the integrated system introduced in this study incorporates the
Kalman filter to enhance its performance. The Kalman filter is employed to
predict the next state estimate by considering the previous state estimate and
the system dynamics. Subsequently, the newly acquired centroid measurement is
used to update the state estimate. By considering both the system dynamics and
the noisy measurements, the Kalman filter aids in refining and smoothing the
estimated centroid position. The updated centroid position, obtained through
the Kalman filter, is then utilized for subsequent control actions within the
line following system.

The effectiveness of the proposed system for autonomous line following was
evaluated through simulation tests. These tests involved the MAV autonomously
navigating in various environments with different pre-defined path shapes. The
results of the simulation tests demonstrated that the HSV line detection used
in the system is robust. The MAV successfully followed the pre-defined flight
route, indicating the system’s capability. Although there were some position
errors observed at the turning points, the actual trajectory of the MAV
demonstrated that it could navigate the entire pre-defined path accurately
through the output of the vision algorithm.

## References

  * [1] G. Cai, K.-Y. Lum, B. M. Chen, and T. H. Lee, “A brief overview on miniature fixed-wing unmanned aerial vehicles,” _IEEE ICCA 2010_ , pp. 285–290, 2010.
  * [2] Í. R. da Costa Barros and T. P. Nascimento, “Robotic mobile fulfillment systems: A survey on recent developments and research opportunities,” _Robotics and Autonomous Systems_ , vol. 137, p. 103729, 2021.
  * [3] H. Tran-Dang and D.-S. Kim, “The physical internet in the era of digital transformation: perspectives and open issues,” _IEEE Access_ , vol. 9, pp. 164 613–164 631, 2021.
  * [4] A. Anand, S. Agrawal, S. Agrawal, A. Chandra, and K. Deshmukh, “Grid-based localization stack for inspection drones towards automation of large scale warehouse systems,” _arXiv preprint arXiv:1906.01299_ , 2019.
  * [5] S. Li, M. M. Ozo, C. De Wagter, and G. C. de Croon, “Autonomous drone race: A computationally efficient vision-based navigation and control strategy,” _Robotics and Autonomous Systems_ , vol. 133, p. 103621, 2020.
  * [6] F. Zafari, A. Gkelias, and K. K. Leung, “A survey of indoor localization systems and technologies,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 3, pp. 2568–2599, 2019.
  * [7] P. L. Ching, S. C. Tan, and H. W. Ho, “Ultra-wideband localization and deep-learning-based plant monitoring using micro air vehicles,” _Journal of Aerospace Information Systems_ , vol. 19, no. 11, pp. 717–728, 2022.
  * [8] A. M. Ladd, K. E. Bekris, A. Rudys, G. Marceau, L. E. Kavraki, and D. S. Wallach, “Robotics-based location sensing using wireless ethernet,” in _Proceedings of the 8th annual international conference on Mobile computing and networking_ , 2002, pp. 227–238.
  * [9] Z. Yang, Z. Zhou, and Y. Liu, “From rssi to csi: Indoor localization via channel response,” _ACM Computing Surveys (CSUR)_ , vol. 46, no. 2, pp. 1–32, 2013.
  * [10] J.-O. Lee, T. Kang, K.-H. Lee, S. K. Im, and J. Park, “Vision-based indoor localization for unmanned aerial vehicles,” _Journal of aerospace engineering_ , vol. 24, no. 3, pp. 373–377, 2011.
  * [11] K. Guan, L. Ma, X. Tan, and S. Guo, “Vision-based indoor localization approach based on surf and landmark,” in _2016 International Wireless Communications and Mobile Computing Conference (IWCMC)_. IEEE, 2016, pp. 655–659.
  * [12] D. B. Poudel, “Coordinating hundreds of cooperative, autonomous robots in a warehouse,” _Jan_ , vol. 27, no. 1-13, p. 26, 2013.
  * [13] F. Guérin, F. Guinand, J.-F. Brethé, H. Pelvillain _et al._ , “Towards an autonomous warehouse inventory scheme,” in _2016 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2016, pp. 1–8.
  * [14] S. Barlow, Y. Choi, S. Briceno, and D. N. Mavris, “A multi-uas trajectory optimization methodology for complex enclosed environments,” in _2019 International Conference on Unmanned Aircraft Systems (ICUAS)_. IEEE, 2019, pp. 596–604.
  * [15] O. Maghazei, M. A. Lewis, and T. H. Netland, “Emerging technologies and the use case: A multi-year study of drone adoption,” _Journal of Operations Management_ , vol. 68, no. 6-7, pp. 560–591, 2022.
  * [16] A. Youssef, M. E. Khoreby, H. Issa, and A. Abdellatif, “Brief survey on industry 4.0 warehouse management systems,” _International Review on Modelling and Simulations (IREMOS)_ , vol. 15, no. 5, 2022. [Online]. Available: https://www.praiseworthyprize.org/jsm/index.php?journal=iremos&page=article&op=view&path%5B%5D=27121
  * [17] A. P. del Pobil, M. Kassawat, A. J. Duran, M. Arias, N. Nechyporenko, A. Mallick, E. Cervera, D. Subedi, I. Vasilev, D. Cardin _et al._ , “Uji robinlab’s approach to the amazon robotics challenge 2017,” in _2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)_. IEEE, 2017, pp. 318–323.
  * [18] S. Sun, A. Romero, P. Foehn, E. Kaufmann, and D. Scaramuzza, “A comparative study of nonlinear mpc and differential-flatness-based control for quadrotor agile flight,” _IEEE Transactions on Robotics_ , vol. 38, no. 6, pp. 3357–3373, 2022.
  * [19] Y. Song, A. Romero, M. Müller, V. Koltun, and D. Scaramuzza, “Reaching the limit in autonomous racing: Optimal control versus reinforcement learning,” _Science Robotics_ , vol. 8, no. 82, p. eadg1462, 2023.
  * [20] Y. Zhou, E. van Kampen, and Q. P. Chu, “Launch vehicle adaptive flight control with incremental model based heuristic dynamic programming,” in _68th International Astronautical Congress (IAC)_ , 2017.
  * [21] J. Yang, Z. Cai, J. Zhao, Z. Wang, Y. Ding, and Y. Wang, “Indi-based aggressive quadrotor flight control with position and attitude constraints,” _Robotics and Autonomous Systems_ , vol. 159, p. 104292, 2023.
  * [22] Y. Zhou, H. W. Ho, and Q. Chu, “Extended incremental nonlinear dynamic inversion for optical flow control of micro air vehicles,” _Aerospace Science and Technology_ , vol. 116, p. 106889, 2021.
  * [23] H. W. Ho and Y. Zhou, “Incremental nonlinear dynamic inversion based optical flow control for flying robots: An efficient data-driven approach,” _arXiv preprint arXiv:2307.02702_ , 2023.
  * [24] J. Gaspar, N. Winters, and J. Santos-Victor, “Vision-based navigation and environmental representations with an omnidirectional camera,” _IEEE Transactions on robotics and automation_ , vol. 16, no. 6, pp. 890–898, 2000.
  * [25] E. H. Binugroho, D. Pratama, A. Z. R. Syahputra, and D. Pramadihanto, “Control for balancing line follower robot using discrete cascaded pid algorithm on adroit v1 education robot,” in _2015 International Electronics Symposium (IES)_. IEEE, 2015, pp. 245–250.
  * [26] N. Maniha, A. Ghani, F. Naim, and T. Yon, “Two wheels balancing robot with line following capability,” _Int. J. Mech. Mechatronics Eng_ , vol. 5, no. 7, 2011.
  * [27] M. B. B. Mahaleh and S. A. Mirroshandel, “Harmony search path detection for vision based automated guided vehicle,” _Robotics and Autonomous Systems_ , vol. 107, pp. 156–166, 2018.
  * [28] H. Ho and Q. Chu, “Automatic landing system of a quadrotor uav using visual servoing,” _Proceedings of the Euro GNC_ , 2013.
  * [29] A. C. Bovik, “Basic binary image processing,” in _The Essential Guide to Image Processing_. Elsevier, 2009, pp. 69–96.
  * [30] G. Bishop, G. Welch _et al._ , “An introduction to the kalman filter,” _Proc of SIGGRAPH, Course_ , vol. 8, no. 27599-23175, p. 41, 2001.
  * [31] H. W. Ho and G. C. de Croon, “Characterization of flow field divergence for mavs vertical control landing,” in _AIAA Guidance, Navigation, and Control Conference_ , 2016, p. 0106.