# Robust Adversarial Attacks Detection based on Explainable Deep Reinforcement
Learning For UAV Guidance and Planning

Thomas Hickling, Nabil Aouf, Phillippa Spencer  T. Hickling and N. Aouf are
with the School of Mathematics, Computer Science and Engineering Department of
Electrical and Electronic Engineering City, University of London Northampton
Square, London, EC1V 0HB. E-mails: {thomas.hickling<EMAIL_ADDRESS>Spencer is with the Defence Science and Technology Laboratory (DSTL), Porton
Down, Salisbury, United KingdomManuscript received April xx, xxxx; revised
August xx, xxxx.

###### Abstract

The dangers of adversarial attacks on Uncrewed Aerial Vehicle (UAV) agents
operating in public are increasing. Adopting AI-based techniques and, more
specifically, Deep Learning (DL) approaches to control and guide these UAVs
can be beneficial in terms of performance but can add concerns regarding the
safety of those techniques and their vulnerability against adversarial
attacks. Confusion in the agent’s decision-making process caused by these
attacks can seriously affect the safety of the UAV. This paper proposes an
innovative approach based on the explainability of DL methods to build an
efficient detector that will protect these DL schemes and the UAVs adopting
them from attacks. The agent adopts a Deep Reinforcement Learning (DRL) scheme
for guidance and planning. The agent is trained with a Deep Deterministic
Policy Gradient (DDPG) with Prioritised Experience Replay (PER) DRL scheme
that utilises Artificial Potential Field (APF) to improve training times and
obstacle avoidance performance. A simulated environment for UAV explainable
DRL-based planning and guidance, including obstacles and adversarial attacks,
is built. The adversarial attacks are generated by the Basic Iterative Method
(BIM) algorithm and reduced obstacle course completion rates from 97% to 35%.
Two adversarial attack detectors are proposed to counter this reduction. The
first one is a Convolutional Neural Network Adversarial Detector (CNN-AD),
which achieves accuracy in the detection of 80%. The second detector utilises
a Long Short Term Memory (LSTM) network. It achieves an accuracy of 91% with
faster computing times compared to the CNN-AD, allowing for real-time
adversarial detection.

###### Index Terms:

Adversarial Attacks, Adversarial Attack Detection, AI, DDPG, DRL,
Explainability, Shapley Values, UAV Guidance

##  I Introduction

The ability to autonomously pilot UAVs may lead to many new use opportunities
in the coming years. According to a Goldman Sachs report[7], it is expected
that the UAV market, driven by commercial, defence and consumer needs, will
reach a total of $100 billion in the future. As UAVs become more widely used,
the adoption of Artificial Intelligence (AI) has also increased with better
hardware and software, leading to improved processing speed and better-
performing algorithms. Combining these two technologies has resulted in
autonomously guided UAVs, which achieve performance comparable to human
operators, if not better [32].

One area of AI that is of particular interest in this field is Deep
Reinforcement Learning (DRL). This scheme utilises reinforcement learning
techniques and applies them to a Deep Learning (DL) framework. DRL uses Markov
Decision Processes where a state leads to an action that produces a reward.
These DRL schemes perform well in tasks with unstructured data as it learns
using trial and error. The DRL approach applies in a variety of fields such as
AI in computer games[27], robot control[26], and natural language
processing[35]. This paper will adopt a DRL scheme to perform UAV guidance.
DRL schemes have been used with UAVs before, such as in Liu et al.(2022)[22],
where it was used to optimise the positioning of UAVs for data collection. The
guidance task in this paper will use the Deep Deterministic Policy Gradient
(DDPG) method with a Prioritised Experience Replay (PER). The DDPG scheme is a
well-established algorithm, so this paper will look at increasing the
efficiency in training and safety in operation by utilising an Artificial
Potential Field (APF). This method produces a repulsive field around obstacles
and an attractive field around the goal. Adding the APF will likely lead to
faster training times and safer operation during flight.

Though DL can be a helpful tool, it has limitations in its explainability due
to its closed-box nature. The opaqueness of DNNs can lead to a lack of trust
in the decisions made by the agent. Research into the eXplainable Artificial
Intelligence (XAI) has produced methods, such as SHapley Additive exPlanations
(SHAP)[24], Local Interpretable Model-agnostic Explanations (LIME)[29],
Decision Trees[23], that provide explanations to understand decisions made by
the DL agents. Although these methods can lead to good explanations of the
decisions made, there needs to be more work done in using these explanations
to improve the performance of AIs. This paper aims to use SHAP to attempt to
solve one of the significant problems in DL.

With the closed-box nature of these DL algorithms, unintended paths for making
decisions arise. When an outside user exploits these paths, it is known as an
adversarial attack[2][8]. Adversarial attacks are created by making small
perturbations in the data inputs that have statistically more significant
impacts on the decision of the AI agents. These attacks are dependent on the
known parameters of the attacked AI agent (closed-box attack or white-box
attack), when the attack occurs (training or operation), and the methods of
generating the attack (Fast Gradient Sign Method (FGSM)[8], Projected Gradient
Descent (PGD)[25], Carlini and Wagner (C&W)[3]). This paper aims to use the
FSGM attack to show a reduction in the successful operation of an AI-guided
UAV.

This paper aims to develop a robust adversarial attack-based explainability
detection scheme which suits the drone’s DRL-based guidance technology during
in-flight operation. The training of the UAV uses a neural network to perform
a guidance task using the DDPG-PER with an APF DRL scheme. The SHAP method
provides explanations of the agent’s actions by giving those decisions
context.

An attack is then used on the drone using an adversarial attack algorithm
based on the FSGM designed to move the drone off course. Having insight into
how the drone’s planning and guidance decisions are made through
explainability, detecting an altered decision from an attack is possible. We
propose to monitor these slight variations in the decision process with deep
network-based detectors. We introduce the first one based on a Convolutional
Neural Network (CNN). Using the SHAP values generated by analysing the DDPG-
APF agent’s depth image input layer, this CNN-based Adversarial Detector (CNN-
AD) learns and finds features in the image generated by the SHAP values to
detect adversarial attacks. The second type of detector we propose is a
Recurrent Neural Network (RNN) in the form of a Long Short Term Memory (LSTM)
network. This detector monitors over a 10-time step period the changes in the
generated SHAP values to determine if an attack occurs. The SHAP values are
generated on a hidden layer within the DDPG-APF agent to reduce the number of
SHAP values generated and considered. The hidden layer for the SHAP generation
is the penultimate layer before the image and positional processes are
combined. It is vital that the adversarial attack detector can perform real-
time detection for the safe operation of the UAV. So a good detector can
decide on the state before the next time-step.

The main contributions of this paper are as follows:

  * •

We propose using an APF when training and operating a DRL guidance agent to
improve performance.

  * •

Proposing and creating a method for adversarial attacks upon a DRL-controlled
drone for the first time in the literature.

  * •

Proposing two innovative deep network-based methods exploiting explainability
concepts to detect adversarial attacks upon DRL-based autonomous drone
guidance.

  * •

Showing the ability for real-time detection of adversarial attacks using one
of the deep network-based methods.

##  II Related Work

This paper builds and integrates elements of different domains (AI, UAVs,
Planning and Guidance, Cyber Security) to create an adversarial attack
detector that uses a DRL guidance scheme adopted for drones. Here the most
relevant papers to our own will be highlighted.

Many papers are adopting DRL methods to safely build a guidance agent to
complete an obstacle course. In Sang-Yun et al.(2019)[32], they tested the
ability of multiple DRL obstacle avoidance agents against differing skill
level human operators. They used multiple DRL schemes in their work (Double
Dueling Deep-Q Network (D3QN), Trust Region Policy Optimisation (TRPO), DDPG,
Actor-Critic using Kronecker-Factored Trust Region (ACKTR), Proximal Policy
Optimisation (PPO)). They tested algorithms and operators in three AirSim
environments. The experiments use one discrete action space and two continuous
action spaces. The authors find that the D3QN outperforms the other algorithms
in the discrete action space. In the continuous action space, D3QN performs
poorly, while the other algorithms, mainly the ACKTR algorithm, provide better
performance. Against the expert human operator, the performance in the two
tests is equal, and in the third test, the ACKTR performs better, while the
operator and the other algorithms do not reach the goal.

Guo et al.(2020)[10] show that the extension of the APF to a DDPG scheme to
control a ship can positively impact the training efficiency of the DDPG. He
et al.(2021)[13] adopt a DRL algorithm (Twin Delayed DDPG (TD3)) for drone
guidance. They combine this scheme with SHAP values to provide explainability
to the TD3 agent. Using a second explainability method known as a Gradient-
weighted Class Activation Mapping (Grad-CAM) to create a saliency map to show
important regions in the input depth image. Using the SHAP values along with
Grad-CAM reasonably explains the drone’s actions.

In the literature, there are many different types of explainability methods.
However, only a few use the explanations to provide further benefits than
explaining the DL model’s decisions. In Fidel et al.(2020)[5], the authors use
the explanations generated by SHAP values to detect adversarial attacks upon
an image classifier. The SHAP values generate a pattern that is unidentifiable
as a particular class. Thus any pattern that falls outside this class can be
detected as an attack. Their attack detection model consists of a fully
connected network of four layers and produces detection rates ranging from
12.6% to 100% depending on the dataset and the type of adversarial attack
considered. Most attacks are over 90%, with only a few specific cases below
that.

Some papers look at the issue of adversarial attacks on autonomous vehicle
systems. One such example is by He et al.(2023)[14], who look to improve the
decision-making for an autonomous lane changer by leveraging an adversarial
reinforcement learning approach. The team attacked their model with an
adversarial method based on Bayesian optimisation that efficiently
approximates the optimal adversarial observation perturbations. To deal with
these adversarial attacks, a constrained observation-robust actor-critic
algorithm was proposed that kept the attacked policies within bounds while
maintaining performance in the lane-changing task. The team found that the new
algorithm not only improved the robustness of the model but also improved the
performance at the lane-changing task.

Ilahi et al.(2022)[18] review the different types of adversarial attacks on
DRL agents and the methods to detect or at least nullify them. Lin et
al.(2017)[21] use an action-conditioned frame prediction module to detect
attacks. The prediction module allows them to detect attacks and use predicted
frames to make robust decisions, as they can lose any altered frames. This
method can detect adversarial attacks with an accuracy of 60-100% depending on
the type of attack used.

Havens et al.(2018)[12] use an agent that detects adversarial examples by the
agent learning separate sub-policies using the Meta Learned Advantage
Hierarchy (MLAH) framework. MLAH is a method to protect against adversarial
attacks during the training period. The agent finds the attacks because they
are unexpected due to the sub-policies.

Xiang et al.(2018)[36] set out a robot pathfinding task to test their
adversarial detector. They designed their detector to consider five factors,
energy point gravitation, key point gravitation, path gravitation, included
angle, and the placid point. Using the data points, they calculate the
probability of an adversarial attack on the robot. This method gained a
precision of around 70% detection.

Researchers have used LSTM networks to detect adversarial attacks upon
Distributed Denial of Service (DDoS). These LSTM detectors have shown to be
more robust than other network types because of the network’s discrete
property and the input samples’ utility requirement. In Huang et
al.(2020)[17], the authors designed two new adversarial generators to attack
LSTM-based detectors. They used a Genetic Attack (GA) and Probability Weighted
Packet Saliency Attack (PWPSA) to fool the LSTM detection method over 90% of
the time.

This paper attempts to build an autonomous UAV DRL-based guidance scheme. This
guidance scheme will be robust against adversarial attacks by building a novel
adversarial detector exploiting either CNN or LSTM network architectures that
utilise the SHAP-based explainability of the DRL guidance scheme to detect
attacks.

##  III Methodology

This section will lay out the main pillars of this paper: First, a short
description of the DRL agent and the introduction of the APF technique to
improve the performance of the DRL training efficiency and operational safety.
Second, a brief outline of the generation of SHAP values as a method of XAI.
Third, describe the methods to generate adversarial attacks upon the drone.
Finally, explain the two novel methods for creating a DNN adversarial
detection scheme.

###  III-A Deep Deterministic Policy Gradient with Prioritised Experience
Replay and Artificial Potential Field

A DRL method is chosen to control the drone’s guidance. A move away from
Q-learning has occurred for tasks in continuous action spaces. Their poor
performance means that most DRL guidance models use Actor-Critic methods for
training. There are a few leading examples of these methods, such as TD3[6],
TRPO[30], SAC[11], and DDPG[20]—this final one, DDPG that will be implemented
as the DRL scheme for this study.

DDPG uses an actor-critic method that combines the policy gradient and the
value function. It comprises two separate networks, the actor that controls
the policy function and the critic that controls the value function. The
construction of these networks is shown in Fig. 1. The performance of the DDPG
is improved further by being paired with a PER. The PER[16] is a method for
choosing higher relevancy states for training the model. These can be ”states”
that lead to higher or negative rewards, as the model can learn better from
more impactful experiences. The PER has a probabilistic element so that the
same ”states” are not chosen too frequently.

This algorithm is already well-known by the team, so implementation is
trivial. The DDPG is the only type of DRL scheme to be used as this paper is
interested in something other than the scheme’s performance relative to other
DRL schemes. Comparisons between DRL schemes have been covered in other
studies[1].

Figure 1: The construction of the two neural networks of the DDPG
architecture. The actor-network controls the drone’s movement, and the critic
network generates the Q-value that trains the networks.

####  III-A1 Artificial Potential Field

Training neural networks can be time-consuming. Therefore, it is always
beneficial to improve training efficiency. One way to do this is to help the
agent learn by guiding the learning by forcing better outcomes and avoiding
bad ones. This paper uses the APF to guide neural network learning to increase
training efficiency. This paper will also look at the impact on operational
safety when using the APF as an added safety net when avoiding obstacles.

APF[4] is a method used in robot guidance and planning but can now be adopted
to increase training efficiency when used in conjunction with a DRL
scheme[10]. It is postulated to improve the safety of the drone during
operation by reducing the chance of collision with an obstacle. It could do
this by visually picking out obstacles and applying a repulsive field when
needed. In this paper, on the other hand, placing the fields will be done
manually and leaving an automatic field generator for future work.

APF works by creating a potential field that can either repel or attract the
drone. The goal of the drone is to generate an attractive force, while the
obstacles generate repulsive forces. By reducing collisions and increasing
successful runs, decreasing the training time. The following function gives
the attractive force projected from the goal:

$\begin{split}F(v_{x})&=\frac{20\cos{\theta}}{d^{2}},d<5\\\
&=\frac{2\cos{\theta}}{d},d>5\\\ F(\omega)&=\tan{\frac{2\theta}{5}},d<5\\\
&=\tan{(\frac{\theta}{3}})^{3},d>5,\frac{\pi}{2}<\theta<-\frac{\pi}{2}\end{split}$
(1)

Algorithm 1 DDPG with PER and APF

Initialise action-value network $Q(s_{t},a_{t},w)$, actor-network
$\mu(s_{t},\nu)$ with $w$ and $v$ separately, replay buffer R with size S;

Initialise target network $Q^{\prime}(s_{t},a_{t},w)$,
$\mu^{\prime}(s_{t},\nu)$ with $w$ and $v$ separately;

Initialise maximum priority, parameters $\alpha,\beta$, updating rate of the
target network $\lambda$, minibatch K;

for episode = 1, M do

Initialise a random noise process $\Phi$;

Observe initial state $s_{1}$;

for t=1, H do

Add noise $\Phi_{t}$ in the exploration policy and

select action $a_{t}$ according to the new policy;

Calculate the force $F_{a}(v_{x},\omega)$ from the goal;

Calculate the force $F_{r}(v_{x},\omega)$ from all obstacles;

Sum all forces $F(v_{x},\omega)=F_{a}+\sum_{i}F_{ri}$;

Add the velocity $F(v_{x},\omega)$ generated by the

force from the APF to the action $a_{t}$;

Obtain reward $r_{t}$ and the new state $s_{t+1}$;

Store experience $(s_{t},a_{t},r_{t},s_{t+1}$ in replay buffer

R and set $D_{t}=max_{i<t}D_{i}$;

if t¿S then

for j=1, K do

Sample experience $j$ with probability

$P(j)$;

Compute corresponding importance

sampling weight $W_{j}$ and TD-error $\delta_{j}$;

Update the priority of transition j

according to absolute TD-error $|\delta_{j}|$;

end for

Minimise the loss function to update

action-value network: $L=\frac{1}{K}\sum_{i}w_{i}\delta_{i}^{2}$;

Compute policy gradient to update the actor

network:

$\nabla_{v}\mu|_{s_{i}}\approx\frac{1}{K}\sum_{i}\Delta_{a}Q(s_{t},a_{t},w)\nabla_{v}\mu(s_{t},v)$

Adjust the parameters of the target network

$Q^{\prime}(s_{t},a_{t},w)$ and $\mu^{\prime}(s_{t},v)$ with an

update rate $\lambda$;

end if

end for

end for

Here $\theta$ and $d$ are the bearings and the distance between the drone and
the goal, respectively. $v_{x}$ is the forward speed while $\omega$ is the yaw
rate. The force that acts upon the forward velocity of the drone is inversely
proportional to the distance $d$. The angular force is proportional to the
size of the bearing. There is an increase in the rate of force change at 5m.
The angular force only applies when the absolute bearing is over 90°when
further than 5m from the goal. For the repelling force from the obstacles, the
functions are as follows:

$\begin{split}F(\omega)&=-(3-3\tan{\frac{\theta_{ob}}{2}}),d_{ob}<3,\frac{\pi}{2}>\theta_{ob}>0\\\
&=-(-3-3\tan{\frac{\theta_{ob}}{2}}),d_{ob}<3,-\frac{\pi}{2}<\theta_{ob}<0\\\
&=0,d_{ob}>3\end{split}$ (2)

The values $\theta_{ob}$ and $d_{ob}$ are the bearing and the distance between
the drone and the obstacle, respectively. The range of the repellent field is
limited to 5m, which is enough to reduce collisions. The repellent field only
affects the yaw rate as this prevents collisions successfully. Fig. 3 shows
how the two fields act upon the drone and the orientations. Fig. 2 shows
graphically the training process with Algorithm 1 showing the pseudo-code.

Figure 2: The flow of information during training to the various parts of the
DDPG with PER and APF. Figure 3: Diagram showing the attractive force in red
and the repulsive force in blue. The two coordinate systems are shown between
the goal, the drone, and the closest obstacle. Figure 4: The SHAP value images
are created using the DeepSHAP algorithm on the model with a sample set and
the inputs to be analysed. The structures can be seen at (a), (b), and (d).
The horizon is seen at (c). Figure 5: This figure shows how the perturbation
is added to the original image, with the resultant image not noticeably
changed. The perturbation is small, so this image has been exaggerated to show
the structure.

###  III-B Explainability Through Shapley Values

AI decisions can be challenging to understand due to their closed-box nature.
This lack of explainability leads to a lack of trust in AI models. The lack of
understanding is especially true with non-experts and has slowed the adoption
of AI technologies.

XAI methodologies split into making the model interpretable from the start or
using post-hoc explanations for closed-box models. Post-hoc explanations come
in the form of global or local explanations. Global explanations explain the
average behaviour of the model, while local explanations explain individual
predictions. Explanations come in the form of values indicating the importance
of inputs, saliency maps for visual inputs, rule sets for creating decision
trees, or generating natural language responses.

The XAI method we need here must be post-hoc and local to generate
explanations at every layer that the adversarial detector can analyse. Two
methods considered are LIME[29] and SHAP. SHAP is preferred as it provides
mathematical guarantees for the accuracy and consistency of explanations.

SHAP is built on a game theory concept known as Shapley values, where players
are assessed on an individual contribution to a collective game. Values are
assigned on the impact of each player on that final score. Traditional Shapley
values cannot handle the extensive inputs used in DL, so approximation methods
are produced. The SHAP framework is proposed by Lundberg and Lee[24] to fulfil
this need.

SHAP works by assigning a value to each feature based on the importance of the
particular prediction. In a function, the prediction in a simple linear
regression problem is shown as follows:

$\hat{y}_{i}=b_{0}+b_{1}x_{1i}+\cdots+b_{n}x_{ni}$ (3)

In this function, $\hat{y}_{i}$ is the predicted response at the input $i$
with $x_{ni}$ being the features in that input. $b_{n}$ is the estimated
regression coefficient. From this, a contribution from a feature to the
predicted response is $b_{n}x_{ni}$. SHAP expands on this to more complex deep
neural networks. Obtaining the SHAP value by finding the contribution of the
$i$-th feature as shown:

$\Phi_{i}=\sum\limits_{S\subseteq
F\backslash{i}}\frac{|S|!(|F|-|S|-1)!}{|F|!}[f_{S\cup\\{i\\}}(x_{S\cup\\{i\\}})-f_{S}(x_{S})]$
(4)

In this formula, $F$ is all the features while $S$ is a subset of the
features. $f_{S\cup\\{i\\}}$ is a model that has been trained with the feature
$i$ present and $f_{S}$ is the model trained without that feature. These two
are then compared against each other with this section of the function
$f_{S\cup\\{i\\}}(x_{S\cup\\{i\\}})-f_{S}(x_{S})$ where $x_{S}$ is the value
of the input feature. This value is computed for all subsets $S\subseteq
F\backslash{i}$ because the change in feature depends on the other features in
the model.

This way of calculating the SHAP values can be computationally expensive for
networks with many inputs. Thus, a method to approximate the SHAP values was
devised by implementing DeepLIFT[33] to create DeepSHAP, a framework for
layer-wise propagation of Shapley values. DeepLIFT was designed to linearise
the non-linear parts of a neural network by using a linear composition rule.
This linear composition rule allows the calculation of SHAP values by finding
local SHAP values that can be found linearly and using the DeepLIFT
multipliers to calculate the SHAP values for the whole model.

Two sets of values are produced when generating the SHAP values for a decision
made by the DDPG-APF agent. The first set produces the contributions to the
x-velocity, either positive for forward or negative for backward. The second
set is for the yaw rate, either positive for clockwise (right) or negative for
anti-clockwise (left). The SHAP values for a given depth image can be seen in
Fig. 4.

Figure 6: The CNN-AD consists of six time-distributed convolutional layers
with a max-pooling layer after every second convolutional layer. This is then
finished with a five-layer FCN that outputs a confidence value in the sample
being an adversarial example.

###  III-C Adversarial Detection

The complexity of DL models allows for unintended pathways to decisions in the
action space that would not usually be taken. Researchers have exploited these
pathways using adversarial attacks. These adversarial attacks create small
perturbations in the DL model’s input that alter their decisions. The
negligible visual impact of the perturbation can be seen in Fig. 5. As
described in the previous section, XAI techniques such as SHAP explain the
decisions of DNNs. Thus, it is possible to detect the use of different
pathways and the adversarial attack with them. This section will discuss the
method used to create the adversarial attacks and the two novel methods of DL
to detect the changes in the SHAP values to uncover the attacks.

####  III-C1 Fast Gradient Sign Method and Basic Iterative Method

A method to generate adversarial attacks is required based on the ease of
implementation and the computation time. FGSM, developed by Goodfellow et
al.(2014), [9] can generate adversarial examples quickly and is simple to
implement. This method is also flexible as it allows the BIM technique to be
applied to add iterations to strengthen the adversarial attack without just
increasing $\varepsilon$.

FGSM uses a gradient technique to generate adversarial attacks. The method
uses the gradient of the loss function $\nabla_{x}L(x,y)$ to maximise the
loss. If this function is positive, the increase of $x$ will increase the loss
function. A negative gradient function will increase the loss function by
decreasing $x$. The adverse function $\Psi$ is shown below:

$\begin{split}\Psi:X\times Y&\longrightarrow X\\\
(x,y)&\longrightarrow\varepsilon.sign(\nabla_{x}L(x,y))\end{split}$ (5)

This is then combined with the sample $x$:

$x^{\prime}=x+\varepsilon.sign(\nabla_{x}L(x,y))$ (6)

The Basic Iterative Method (BIM)[19] allows the iteration of the FGSM where
the adverse effect can be increased without affecting the visual quality as
much as by increasing the $\varepsilon$ by a comparable amount. BIM generates
an adversarial example $i$ by applying the FGSM method to the adversarial
example $i-1$. This process is laid out in the following function where $\Psi$
is the FGSM function:

$\begin{split}x^{\prime}_{0}&=x\\\
x^{\prime}_{i+1}&=x^{\prime}_{i}+\Psi(x^{\prime}_{i},y)\end{split}$ (7)

####  III-C2 Convolutional Neural Network based Adversarial Detection

It is proposed that detecting the slight changes in the distribution of the
SHAP values from an adversarial attack is best done by a DL network. Detecting
adversarial attacks in classification tasks using SHAP values is proposed by
Fidel et al.(2020)[5]. Their classification task used a simple, Fully
Connected Network (FCN) to detect adversarial examples. In our work, the task
is entirely different as it is about DRL-based drone guidance tasks and a
simple FCN is found to be incapable of detecting the adversarial attack.
Furthermore, exploiting and adopting the explainability SHAP values for our
guidance task differs from any simple classification task. Adopting an FCN-
based Adversarial Detector FCN-AD in a preliminary test achieves an accuracy
of only 56%. Random chance would achieve a 50% accuracy. Thus, a different
solution is needed.

The adversarial attack only affects the depth image inputs into the DDPG-APF
agent. Hence, the SHAP values for the bearing and distance to the goal will
not be used for adversarial detection. The SHAP values for the depth image
provide a value for each pixel, and by normalising these values, they can be
turned into a new SHAP image. The normalisation can be done by either
normalising the values to between 1 and 0 or by designating positive and
negative to different colours and then normalising. The second option is
chosen to contrast the positive and negative values. A CNN-based Adversarial
Detector is proposed to take the SHAP values and decide on the input’s
validity.

CNNs are built with two sections: the convolutional layers produce an
activation map by applying filters that learn to find the features in the
image. The features found in the image are passed onto the second section,
which consists of the fully connected layers. The convolutional layers are
time-distributed, where the convolution is done to the five most recent SHAP
value images. The output is a confidence score on the input being an
adversarial example. The layout of the CNN-AD is given in Fig. 6.

This CNN-AD method is proposed to work by the SHAP values having higher
magnitudes around the edges of obstructing obstacles. Thus, a typical example
perceived by the drone should show nice edges. In an adversarial attack, the
importance of the real features, such as the edges, will decrease, while the
pixels made more prominent by the noise generated by the FGSM attack will have
higher SHAP values.

####  III-C3 Long Short Term Memory Network-based Detection

Figure 7: The 10-time steps of SHAP values generated by the process in Fig. 8
are put through an LSTM layer of 100 modules which is then put through 4 fully
connected layers to produce a detection. Figure 8: The information flow from
the data gathered by the UAV to the detection by the LSTM-AD. Figure 9: The
output of the GRU layer is extracted by splitting the model into two parts,
the first half model that covers the image processing and the second half
model that takes the GRU output and the positional inputs to generate the
action. Figure 10: The layout of the experiment with the drone being
controlled by the DDPG-APF network and the adversarial creation and detection
using SHAP values.

The CNN-AD presents the drawback of generating over 300,000 SHAP values every
time step. Calculating a fraction of the SHAP values using a layer with fewer
outputs in the DDPG-APF agent network is computationally more efficient. The
smallest layer that only depends upon the depth image input is the GRU layer
(see Fig. 1) with 48 values. Calculating the SHAP values for GRU layer output
will be faster than for the CNN-AD method. The CNN-AD detects features in the
depth image to make decisions; thus, using the 48-value vector rules out this
approach. An individual 48-value vector carries a limited amount of
information. The change in SHAP values can be monitored to increase the
information. This increases the information available for analysis without
increasing the number of SHAP values calculated each time step. Recurrent
Neural Networks (RNN) are designed to analyse sequential data with LSTM
networks[15] and are found to outperform simple RNNs[28].

Figure 11: The LSTM module structure can be seen in this figure. A cell state
carries the memory through each module while the various filters operate on
the new data to modify the cell state and produce a new outcome.

LSTMS are developed from RNNs to give a long-term memory as this helps decode
long-term dependencies. Unlike simple RNNs, LSTMs maintain a cell state
carried through the modules. This cell state is what allows the LSTM to handle
long-term dependencies. Our proposed LSTM-based Adversarial Detector (LSTM-AD)
is looking for adversarial examples that cause enough change in the data to be
detected as different to the previous time steps. The layout of an LSTM module
is described in Fig. 11.

A smaller section of data needs to be assessed to improve the processing time
for the SHAP values and limit the size of the LSTM-AD network. It is decided
to measure the SHAP values at the GRU layer found after the convolutional
layers in the network; this will reduce the number of values calculated from
163,840 to 48 per state. This layer’s output is chosen as it is the smallest
state available, only affected by the depth image input. As the adversarial
attack only alters the depth image, it makes sense to consider an output only
changed by the attack.

To access the data coming out of the GRU layer, the model is split into two
separate models; one is just the image processing layers of the original
network that outputs the 48-value state; this will be known as the 1st half
model. The second is the rest of the network, which takes the standard
positional input as well as the output of the GRU layer as its second input;
this will be known as the 2nd half model. Using the values generated by the
1st half model, the 2nd half model can generate the SHAP values required for
analysis by the LSTM-AD. These SHAP values can then be stacked in 10 time-step
packets for detection in the LSTM-AD. This process is shown in Fig. 8. The
layout of the LSTM-AD detector can be seen in Fig. 7. The LSTM layer is
followed by four dense layers to output a decision.

##  IV Experiments and Results

The complete layout of the various subsections and how they interact can be
seen in Fig. 10. The flow of information from the drone to the drone control
system can be seen, and how that can be hijacked to insert adversarial
attacks. The method of SHAP generation can be seen, and the use of these SHAP
values to detect any adversarial attacks.

###  IV-A Experimental Setup

To handle the simulation for this work, AirSim[31] is used in conjunction with
Epic’s Unreal Engine 4 (UE4). AirSim is a high-fidelity physical and visual
computer simulation that allows the generation of large amounts of training
data for machine learning projects.

The test arena is set up as shown in Fig. 12. Two-goal areas are 20 metres
downrange of the start position and 5 metres offset from the centre line. The
goal is chosen randomly at the start of each episode to avoid the DRL-based
guidance agent becoming overfitted with one solution.

Figure 12: This figure shows the arena layout for the training of the DDPG-APF
agent, with a blue circle showing the start position and yellow crosses
showing the goal positions.

The simulated drone is a quadcopter. To ease training, the drone is placed
into yaw mode, where it can only move forward (x-velocity, $v_{x}$) and change
its heading (yaw rate, $\omega$). The drone is reset back to its starting
position if it collides, goes out of bounds or reaches the goal.

The LiDAR sensor is excellent at quickly and precisely building a 3D
representation of the environment. This accurate data allows for navigation in
crowded environments. The data generated from LiDAR scans consist of a list of
3-D coordinates of the surrounding scene where the sensor is the origin point.

To use this data as the input into the DDPG-APF actor-network, the 3-D point
cloud must be converted into a 2-D depth image to work with the convolutional
layers inherited by our choice of network. The 3-D point cloud is projected to
a 2-D surface, with the pixel values calculated by distance in the 3-D space,
as shown in Fig. 13. To fill pixels where no return is made, interpolation is
used.

Figure 13: A diagram showing part of converting a point cloud to a depth
image.

###  IV-B Reward Setup

To train a DRL agent, getting the reward function right is imperative. The
reward function is designed to reward the drone when it moves towards the goal
by adding the change in distance between the drone and the goal. There are
also sub-goals that if the drone moves past a particular checkpoint, it will
gain a bonus reward. The proposed reward function structure is as follows:

$r(S_{t})=\left\\{\begin{array}[]{rcl}5&\mbox{if success}\\\ -2&\mbox{if
dead}\\\ 2(1+\frac{C}{N_{C}})&\mbox{at checkpoint}\\\
d_{t}-d_{t-1}&\mbox{otherwise}\end{array}\right\\}$ (8)

Here $C$ is the checkpoint number, and $N_{C}$ is the number of checkpoints.

###  IV-C Results

####  IV-C1 DRL Training

The training for the DDPG-APF agent and the DDPG agent is completed
successfully, and they are both able to complete the course to a satisfactory
level. The training times for each model can be found in TABLE I. The two
different DDPG agents’ training efficiency is also compared in TABLE I, where
the increase in training efficiency for the DDPG-APF agent can be seen by a
decrease of 14.3% in training time. Fig. 14 & 15 show the progression of the
reward gained and the success rate through training the DDPG-APF agent. The
first 5000 time steps are to build the memory for the PER. Hence they are not
included in the data as no training is done. A validation course is created
for testing where the drone reached a completion rate of 80% without APF. When
the APF is applied to the layout of the validation course, the completion
rises to 97%. The run time for the DDPG-APF agent to make a decision is shown
in TABLE IV, which takes around 0.05s. Overall the inclusion of the APF with
the DDPG has had a small effect on the training efficiency though this slight
increase did save a few hours as the training time was so long. The real
benefit is the increase in course completion, as this added safety layer makes
the agent more likely to avoid obstacles. To deploy the agent in an unknown
area, a vision system must automatically place the repulsive fields to the
obstacles to take advantage of the APF method.

Figure 14: Reward in each episode and the rolling average.

Figure 15: The success rate over the last ten episodes.

TABLE I: Training data for the DDPG and DDPG-APF | DDPG | DDPG-APF
---|---|---
Training steps | 27456 | 24012
Training episodes | 1750 | 1500
Efficiency increase | - | 14.3%
Testing success | 80% | 97%

####  IV-C2 Adversarial Examples

The FGSM adversarial generation algorithm is implemented to easily be altered
into the BIM algorithm just by running the FGSM in a loop. The FGSM is less
effective than required at changing the decisions of the DDPG-APF agent at
small $\varepsilon$. The BIM variation of FGSM is adopted to limit visual
degradation and maximise the adversarial effect.

Figure 16: This figure shows that the $\varepsilon$ and iterations increase
the attack success but do reach a limit in success (61.1%).

We test the performance of the FGSM/BIM adversarial generator at various
settings of $\varepsilon$ and the number of iterations of the FGSM for the BIM
application, as shown in Fig. 16. An attack is considered successful when the
resultant guidance decision by the DRL agent is modified by 33%. This
condition on the amount of deviation ends up reaching a limit at 61.1% attack
success.

Choosing the settings for the adversarial attacks depends on what factors are
important. Increasing the $\varepsilon$ increases the visual distortion of the
depth input image. Increasing the perturbation by too much means the attack
can be detected by close visual inspection. Increasing the number of
iterations increases the strength of the attack but at a lesser cost in visual
distortion when compared to just increasing the $\varepsilon$. This also
increases, but moderately, the computing time by the number of iterations.

To further test the ability of the FGSM/BIM adversarial algorithm when near an
obstacle. To test this, the drone is placed at various distances (1m, 1.5m,
2m, 2.5m, 3m) to the obstacle and bearings (-30°, -20°, -10°, 0°, 10°, 20°,
30°) to the goal. The difference in the yaw velocity decision between the
adversarial example and the unaltered example is noted in TABLE II.

TABLE II: Deflection caused by the adversarial attack. $\varepsilon$ of 1 and iteration of 20. Gap | Bearing to goal (Degrees)
---|---
to wall |  -30° |  -20° |  -10° |  0° |  10° |  20° |  30°
(m) | Change in yaw velocity from attack (°/s)
1.0 |  0 |  0 |  0 |  0 |  0 |  0 |  0
1.5 |  0 |  0 |  61.5 |  62.1 |  62.4 |  62.7 |  63.0
2.0 |  52.8 |  53.4 |  53.4 |  53.4 |  53.1 |  53.1 |  53.1
2.5 |  30.3 |  28.8 |  26.7 |  30.6 |  32.1 |  30.9 |  32.4
3.0 |  24.9 |  24.6 |  21.0 |  24.6 |  24.9 |  25.8 |  26.4

In TABLE II, the difference between the two decisions becomes more significant
as the drone approaches the column. This difference increases as the drone
produces more drastic actions to avoid a collision with the obstacle. This
gives the adversarial attack a larger decision to alter. The drone could not
be attacked when very close to the obstacle. It appears that the small
perturbation cannot overcome the depth image being filled with the obstacle at
close range.

The results shown are with a $\varepsilon$ of 1 and an iteration of 20.
Attacks at various iterations were carried out. At five iterations, all the
variations that aren’t 0 in the table were over the 33% barrier defined
earlier. No matter the attack strength, the 0 values from the table remained
0.

A single attack on the drone is not a severe threat to the safety of the
drone, as it will recover and continue towards its goal after the attack. A
constant attack over a series of time steps must be executed to increase the
danger to the drone. To test the success of this approach, the drone is set up
to run through the validation course, and attacks are set up to occur randomly
and run for 5-time steps. Doing this leads to completing the course 35% of the
time. This is a drop of 62% from what the drone achieved when the drone was
not attacked. The APF was not activated to simulate the agent’s inability to
identify obstacles when attacked. A 100% failure rate is unlikely due to the
intrinsic robustness nature of the DDPG-APF agent and the chance of the
attacks coming at the wrong times.

####  IV-C3 Shapley Values

The SHAP values are first produced to analyse the inputs of the whole DDPG-APF
model. It is one value for every pixel in the 5-time step depth images that
make up the time-distributed depth image input and the ones for positional
inputs (bearing, distance to goal). SHAP values are a measure of how much each
input has an impact on the output. As the DDPG-APF model has two outputs
(x-velocity and yaw rate), each decision has a set of SHAP values. For this
study, the SHAP values relating to the yaw rate decision are chosen because
this set of SHAP values has the most variability throughout a guidance task.
The x-velocity remains stable at its maximum throughout each guidance task.
Thus, there is little variation in SHAP values to analyse.

Fig. 17 shows a typical SHAP value representation of the response to the depth
image. The red positive SHAP values of the right side of the column show a
largely positive response favouring a right turn to avoid the collision. A
sizeable negative response can be seen on the left side of the column where
the SHAP values support a left turn to avoid a collision.

Figure 17: The figure shows SHAP values in response to a depth image. Green is
a negative SHAP value, and red is a positive one.

Though producing SHAP values for individual pixels can help visually inspect
what the DDPG-APF model thinks is important, it does not allow an easy
deduction of the action suggested. Thus a more parsable form of the SHAP
values is created. As described in the final part of the LSTM network
detection section and shown in Fig. 8 & 9, a break is placed in the actor-
network model to allow the SHAP value generation at the layer of the break.
This gives 48 SHAP values which is a more manageable number to manipulate.

To analyse how the decisions made by the DDPG-APF agent are influenced by the
three major inputs (depth image, bearing and distance to goal), Fig. 18 is
produced. The variation of the SHAP values related to the yaw rate decision
over one navigation episode will be plotted. The SHAP values for the
positional inputs are plotted directly, but for the contribution of the depth
image, the 48 values are summed to give one SHAP value that represents the
collective decision for that input is plotted. The yaw rate is also plotted;
this value should be the summation of the SHAP values from the combined depth
image, bearing and distance to the goal.

In Fig. 18, the distance to goal SHAP value has little effect on the turning
decision from the DDPG-APF due to this input having no information relevant to
this decision. The bearing SHAP value tracks the bearing directly by trying to
turn the drone back to a bearing of 0 at all times. The depth image SHAP value
responds to the obstacles in the drone’s path. When the drone is forced from
the 0°bearing by an obstacle, the bearing SHAP value will increase in
magnitude to turn the drone back, but the depth image SHAP value will stay
high to keep the drone from turning back into the obstacle. This behaviour is
shown in Fig. 18 between time steps 8 and 16. After this up to time step 24,
the drone lines up to fly through a gap between the obstacles so significant
variations in the yaw rate occur. There is a short period where the drone
flies straight through the gap. The final section is a straight flight to the
goal, but as the goal becomes closer, the bearing will change quickly with
even small movements leading to the large oscillations shown in the figure.

Figure 18: This figure shows the SHAP values produced by the three major
inputs to the DDPG-APF model and the yaw rate chosen at each time step.

####  IV-C4 Convolutional Neural Network based Adversarial Detector

The training of the CNN-AD uses 5000 states from those generated during the
training. An adversarial example is created using an iteration of 20 and an
$\varepsilon$ of 1. With these states, SHAP values are created using a sample
of 30 states to build the SHAP generator. The detector is then trained for ten
epochs, reaching an accuracy of 99% in the training data and a validation
accuracy of 90%. The CNN-AD achieves an accurate detection rate of 80% when
tested across 1000 states encountered in simulated runs through the validation
course.

The testing is done on the validation course. The drone then runs through the
course with a normal and an adversarial state generated at each time step.
SHAP values are then generated and then put through the detector. The detector
then puts a value between 1 and 0 for each example state; a value over 0.5 is
considered an adversarial detection. The closer to 1 as a score is, the more
confident that it is an adversarial example. The accuracy of this detector is
then measured, and if it is a wrong classification, it is assigned to being a
false positive or a false negative. The testing runs for 1000 time steps and
produces 1000 unaltered and 1000 adversarial states. The accuracy is the
number of correctly assigned states divided by the total number of states.

The accuracy of the CNN-AD is 80% which leaves 20% either undetected or
wrongly classified as an attack. A hypothesis that higher accuracy is not
achievable in this setup is that the ability of the detector to distinguish
types depends upon the adversarial attack creating a different outcome. As
shown in the previous section, large deviations only appear in 61% of cases.
It is reasonable to assume that with small or non-existent deviations, the
change in the SHAP values structure might not be detectable for this CNN-AD.

An issue with this method is the time consumed to create the SHAP values for
the detector to use. The SHAP value generation takes between 20.8s and 21.1s
at each time step in its current setup. The current 21s would mean that only
one sample could be done over the drone’s course navigation, which is
impractical.

####  IV-C5 Long Short Term Memory based Adversarial Detector

Building upon the success of the CNN-AD, an LSTM network is created to attempt
to detect adversarial examples. The concept here is to detect an anomaly in
the signal generated over 10-time steps of the SHAP values of the GRU layer,
shown in Fig. 1, as described in earlier sections. The flow of data generation
for the LSTM-AD is demonstrated in Fig. 8. The method to generate the training
samples for the CNN-AD does not work for the LSTM-AD detector, as combining
states blindly can lead to poor examples. A different approach is proposed by
creating samples directly from the simulation. This is done by running the
drone through the training course generating an adversarial and normal example
at each time step. This method gathered 15,000 samples of each type for the
training.

The LSTM-AD’s training parameters are different to those of the CNN-AD. Many
more epochs are used for the LSTM-AD; the optimal is found at 1200 epochs. The
accuracy for the training data reaches 100%, but to get the best accuracy in
the validation, the number of epochs is limited to the 1200 epochs mentioned
above, which gives a training accuracy of 95% and testing at 90%. The
validation accuracy at 91% was much closer to the testing accuracy with this
network.

When compared to the CNN-AD, this method is faster to compute. In TABLE IV, a
reduction of 99.9% from 13.33s to 0.019s for calculating the SHAP values is
observed. When looking at the adversarial detectors’ compute time, a similar
decrease of 99.4% from 0.162s to 0.001s is observed. This enables the detector
to analyse every time step for potential attacks. This method also led to an
increase in accuracy over the CNN-AD by 10%. This increase shows that the
LSTM-AD is better at identifying the differences the CNN-AD could not detect.

Comparing the results of these two adversarial detection methods to other
methods is difficult due to the lack of studies on adversarial detection for
drone guidance and navigation tasks. Most studies in this area work by making
the agent robust so that they are not affected by adversarial attacks, but
this doesn’t inform if the agent is being attacked. It is, therefore,
difficult to compare the proposed methods where we are interested in if an
attack has taken place. Due to the lack of comparisons from drone guidance
studies from other areas that are to be used, these may not give a one-to-one
comparison. Tekgul et al.(2022)[34] only give detections over a whole episode
by either looking at a trend through the episode or by having multiple chances
at detection. These methods provide higher detection rates than the spot
detections done in this paper. Other papers, such as Lin et al.(2017)[21],
remove the unsuccessful attacks from their detection data leading to a much
higher detection rate of over 99%.

In TABLE III, we can compare the methods detailed in this paper and a
comparative method using Principle Component Analysis (PCA)[36]. The report
looks at a 2D map navigation task, so this is not a direct comparison between
their work and this paper. Even so, their detection rate was at 70%; this is a
lower performance than the CNN-AD and LSTM-AD built for this study.

TABLE III: Comparison of the detection methods to other studies. | Run time for each section (s)
---|---
Detector type |  Detection Rate |  Agent Model |  Task
FCN-AD |  56 |  DDPG |  Drone Guidance
CNN-AD |  80 |  DDPG |  Drone Guidance
LSTM-AD |  91 |  DDPG |  Drone Guidance
PCA[36] |  70 |  Q-learning |  Map Navigation
TABLE IV: Times taken to produce the required data for the DRL, XAI and Adversarial Detection sections. | Run time for each section (s)
---|---
Detector type |  DDPG-APF Action |  SHAP Values Generation |  Adversarial Detection
FCN-AD |  0.053 |  13.14 |  0.056
CNN-AD |  0.052 |  13.33 |  0.162
LSTM-AD |  0.056 |  0.019 |  0.001

##  V Conclusions

This paper sets out four goals. The first is to test the performance increase
from the APF addition to the DDPG model. The drone is successfully trained and
achieves a good completion percentage. The adoption of APF shows the training
efficiency increase of the DRL model and an increase of 17% to validation
course completion. The APF module could be added to any DRL scheme, and a
future study may look into whether the improvement seen here is achieved in
more DRL schemes. The second goal is to apply the DeepSHAP algorithm to this
model to generate individual input-level explanations. This is completed with
explanations being generated and images being made out of these explanations
to show where the drone’s attention is when navigating. The third goal is to
successfully generate adversarial examples that can trick the drone into
making poor guidance decisions. This is completed using the FGSM and BIM
algorithms, which reduces guidance success by 56%. The fourth goal is to
design a method for detecting adversarial attacks by adopting SHAP values-
based explanations. This is completed in two ways, first by building the CNN-
AD that gained an accuracy of 80%. The second is creating a detector based on
an LSTM neural network module that looks for changes in the signal produced by
adversarial attacks. The LSTM-AD delivers an accuracy of 90% with a lower
computation cost, enabling real-time detection compared to the CNN-AD.

The lack of comparable results to those gained in this paper is due to either
differing applications or different methods of analysis of the success of an
adversarial detector. This has made an overall conclusion of the success of
these DNN adversarial detectors in a broader context. To find which methods
are the best at dealing with adversarial attacks, a more standardised set of
measures of success to be measured by would be advisable.

A method to get, for future research, the correct decision after detecting an
adversarial attack would close off the problem as there is no real recourse
after detecting the attack at the moment. Bringing this into the real world
would also be a promising avenue of study as a method for creating real-world
adversarial attacks is more complicated than just generating them for a
simulation.

## References

  * [1] Omar Bouhamed, Hakim Ghazzai, Hichem Besbes, and Yehia Massoud.  Autonomous uav navigation: A ddpg-based deep reinforcement learning approach.  In 2020 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1–5. IEEE, 2020.
  * [2] N. Carlini and D. Wagner.  Towards evaluating the robustness of neural networks.  In 2017 ieee symposium on security and privacy, 2017.
  * [3] Nicholas Carlini, Guy Katz, Clark Barrett, and David L Dill.  Provably minimally-distorted adversarial examples.  arXiv preprint arXiv:1709.10207, 2017.
  * [4] WANG Cheng-bo, ZHANG Xin-yu, ZHANG Jia-wei, DING Zhi-guo, and AN Lan-xuan.  Navigation behavioural decision-making of MASS based on deep reinforcement learning and artificial potential field.  Journal of Physics: Conference Series, 1357(1):012026, oct 2019\.
  * [5] Gil Fidel, Ron Bitton, and Asaf Shabtai.  When explainability meets adversarial learning: Detecting adversarial examples using shap signatures.  In 2020 International Joint Conference on Neural Networks, IJCNN 2020, page 9207637. Institute of Electrical and Electronics Engineers Inc., 2020\.
  * [6] Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In International conference on machine learning, pages 1587–1596. PMLR, 2018.
  * [7] Goldman-Sachs.  Drones - reporting for work, 2015.
  * [8] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint, 2014.
  * [9] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [10] Siyu Guo, Xiuguo Zhang, Yisong Zheng, and Yiquan Du.  An autonomous path planning model for unmanned ships based on deep reinforcement learning.  Sensors, 20(2):426, 2020.
  * [11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In International conference on machine learning, pages 1861–1870. PMLR, 2018.
  * [12] Aaron Havens, Zhanhong Jiang, and Soumik Sarkar.  Online robust policy learning in the presence of unknown adversaries.  Advances in neural information processing systems, 31, 2018.
  * [13] Lei He, Nabil Aouf, and Bifeng Song.  Explainable deep reinforcement learning for uav autonomous path planning.  Aerospace Science and Technology, 118:107052, 2021.
  * [14] Xiangkun He, Haohan Yang, Zhongxu Hu, and Chen Lv.  Robust lane change decision making for autonomous vehicles: An observation adversarial reinforcement learning approach.  IEEE Transactions on Intelligent Vehicles, 8(1):184–193, 2023.
  * [15] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [16] Yuenan Hou, Lifeng Liu, Qing Wei, Xudong Xu, and Chunlin Chen.  A novel ddpg method with prioritized experience replay.  In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 316–321, 2017.
  * [17] Weiqing Huang, Xiao Peng, Zhixin Shi, and Yuru Ma.  Adversarial attack against lstm-based ddos intrusion detection system.  In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pages 686–693, 2020.
  * [18] Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato.  Challenges and countermeasures for adversarial attacks on deep reinforcement learning.  IEEE Transactions on Artificial Intelligence, 3(2):90–109, 2022\.
  * [19] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  In Artificial intelligence safety and security, pages 99–112. Chapman and Hall/CRC, 2018.
  * [20] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [21] Yen-Chen Lin, Ming-Yu Liu, Min Sun, and Jia-Bin Huang.  Detecting adversarial attacks on neural network policies with visual foresight.  arXiv preprint arXiv:1710.00814, 2017.
  * [22] Run Liu, Zhenzhe Qu, Guosheng Huang, Mianxiong Dong, Tian Wang, Shaobo Zhang, and Anfeng Liu.  Drl-utps: Drl-based trajectory planning for unmanned aerial vehicles for data collection in dynamic iot network.  IEEE Transactions on Intelligent Vehicles, pages 1–14, 2022.
  * [23] Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee.  From local explanations to global understanding with explainable ai for trees.  Nature machine intelligence, 2(1):56–67, 2020.
  * [24] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  Advances in neural information processing systems, 30, 2017.
  * [25] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083, 2017.
  * [26] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, and Koray Kavukcuoglu et al.  Learning to navigate in complex environments.  arXiv preprint, 2016.
  * [27] V. Mnih, D. Silver, A. A. Rusu, and M. Riedmiller et al.  Human-level control through deep reinforcement learning.  Nature, 2015.
  * [28] Yulius Denny Prabowo, Harco Leslie Hendric Spits Warnars, Widodo Budiharto, Achmad Imam Kistijantoro, Yaya Heryadi, et al.  Lstm and simple rnn comparison in the problem of sequence to sequence on conversation data using bahasa indonesia.  In 2018 Indonesian Association for Pattern Recognition International Conference (INAPR), pages 51–56. IEEE, 2018.
  * [29] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ” why should i trust you?” explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * [30] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.  Trust region policy optimization.  In International conference on machine learning, pages 1889–1897. PMLR, 2015.
  * [31] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor.  Airsim: High-fidelity visual and physical simulation for autonomous vehicles.  In Field and service robotics, pages 621–635. Springer, 2018.
  * [32] Sang-Yun Shin, Yong-Won Kang, and Yong-Guk Kim.  Obstacle avoidance drone by deep reinforcement learning and its racing with human pilot.  Applied Sciences, 9(24), 2019.
  * [33] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In International conference on machine learning, pages 3145–3153. PMLR, 2017.
  * [34] Buse GA Tekgul, Shelly Wang, Samuel Marchal, and N Asokan.  Real-time adversarial perturbations against deep reinforcement learning policies: attacks and defenses.  In Computer Security–ESORICS 2022: 27th European Symposium on Research in Computer Security, Copenhagen, Denmark, September 26–30, 2022, Proceedings, Part III, pages 384–404. Springer, 2022.
  * [35] Wang, William Yang, Jiwei Li, and Xiaodong He.  Deep reinforcement learning for nlp.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, 2018.
  * [36] Yingxiao Xiang, Wenjia Niu, Jiqiang Liu, Tong Chen, and Zhen Han.  A pca-based model to predict adversarial examples on q-learning of path finding.  In 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC), pages 773–780. IEEE Computer Society, 2018.