# A Tutorial on Adversarial Learning Attacks and Countermeasures

Cato Pauling  Michael Gimson  Muhammed Qaid  Ahmad Kida and  Basel Halak
School of Electronics and Computer Science  University of Southampton  UK
email<EMAIL_ADDRESS>
###### Abstract

Machine learning algorithms are used to construct a mathematical model for a
system based on training data. Such a model is capable of making highly
accurate predictions without being explicitly programmed to do so. These
techniques have a great many applications in all areas of the modern digital
economy and artificial intelligence. More importantly, these methods are
essential for a rapidly increasing number of safety-critical applications such
as autonomous vehicles and intelligent defense systems. However, emerging
adversarial learning attacks pose a serious security threat that greatly
undermines further such systems. The latter are classified into four types,
evasion (manipulating data to avoid detection), poisoning (injection malicious
training samples to disrupt retraining), model stealing (extraction), and
inference (leveraging over-generalization on training data). Understanding
this type of attacks is a crucial first step for the development of effective
countermeasures. The paper provides a detailed tutorial on the principles of
adversarial machining learning, explains the different attack scenarios, and
gives an in-depth insight into the state-of-art defense mechanisms against
this rising threat .

##  1 Introduction

Our world is becoming increasingly dependent on machine learning for making
automated decisions. With rapid rise in malicious cyberattacks, where
attackers’ incentives are to extract and manipulate the results and models
generated by machine learning algorithms, the need for efficient and practical
defences for these systems is crucial [1].

Intelligent Security Detection Systems (ISDS) are systems designed to identify
and mitigate malicious activity. Anomaly detection is a common example that
uses machine learning (ML) algorithms to derive a model of trustworthy
behaviour. The current activity is then compared to the normal behaviour
learnt by the model, which is then able to classify the activity as normal or
anomalous, triggering the appropriate response. These security systems
themselves may be vulnerable to adversarial learning attacks which target the
ML algorithms they are based upon.

Adversarial learning attacks against machine learning systems exist in an
extensive number of variations and categories; however, they can be broadly
classified: attacks aiming to poison training data, evasion attacks to make
the ML algorithm misclassify an input, and confidentiality violations via the
analysis of trained ML models. Many attacks can have their application adapted
to both deep learning systems and more traditional machine learning models.
Poisoning attacks have a prerequisite of access to the training dataset of the
model, before it is trained and have attack implementations for classical
machine learning methods including linear regression [1] and Support Vector
Machines [2].

The remainder of this paper is structured as follows. Section 2 discusses the
threat modelling approaches for adversarial machine learning. Attack
mechanisms are discussed in section 3, followed by a review of countermeasures
in section 4. Conclusions are drawn in section 5.

##  2 Threat Modelling for Adversarial Leaning Attacks

###  2.1 Adversarial Threat Model

Papernot et al. [3] provided a summary of the type of adversarial approaches
machine learning algorithms are vulnerable to and offered a comprehensive
‘unifying threat model’ for these algorithms and the encompassing systems that
use them. The life cycle of a machine learning system is treated as a
generalised data processing pipeline. The entire pipeline as a whole is
considered, with a specific focus on the two distinct phases of a machine
learning model: the training phase and the inference phase. They divide the
threat model into three key components: the attack surface, adversarial
capability and adversarial goals.

The attack surface of a ML system describes the point in which the attack
takes place. This can be described with respect to the pipeline. For example,
such a system may begin by first collecting the data it wishes to feed into
the model, from sensors or data repositories. This data usually will need to
be processed before being fed into the model, which produces an appropriate
output. The model output is then communicated to an encompassing system in
order to be acted upon. In such a system, the adversary may want to interfere
with collection or processing of input data, corrupt the model itself or
tamper with the model’s output. Each of these are an appropriate attack
surface. Most adversarial learning attacks will occur during the input data
collection and processing in order to exploit the vulnerabilities in the model
without corrupting it. This can be done by providing specific information for
the sensors to pick up, or adding perturbations to images after being
collected by the sensors.

The capability of the adversary also plays a large role in defining the threat
model. The definition of a system’s vulnerabilities is also made with regard
to the strength of the attacker, who may have greater access to the target
system and its data, and thus a greater ability to inflict damage to the
system. The capabilities of an adversary outline the sort of attacks available
to them. These relate to what part of the process they have access to, be it
the training phase or the interference phase.

If the adversary has the ability to insert themselves into the training phase,
they have the capability to learn or influence the training of, and therefore
corrupt, the model itself. The simplest form of attacks in this phase involves
having access to the training dataset, in full or partially. With this,
provided the quality and quantity of data is sufficient, the adversary can
train a substitute model on the same dataset in order to test adversarial
inputs, and validate their success. Broadly however, there are two general
strategies available in the training phase. The first of these is to meddle
with the training set. An adversary can insert malicious samples into the
training set to sabotage what the model learns and throws the model off. This
form of altering the training set it referred to as injection. Similarly,
modification, involves directly altering the training set. Alternatively, the
adversary may be able to tamper with the actual learning logic of the model.
These attacks known as logic corruption, are extremely powerful and would be
very tough to defend against.

In the inference phase however, attacks do not attempt to corrupt the model
itself, but rather to fool it into producing incorrect outputs, by exploiting
inherent vulnerabilities. Alternatively, they may try to perform
reconnaissance, merely gathering evidence of the characteristics. Inference
time attacks, generally, can be split into white-box or black-box attacks
depending on the adversary’s access to information on the target model’s
architecture and therefore their ability to exploit it.

The final component which needs to be considered in Papernot’s threat model
[3] are the adversarial goals. They model the goals of potential adversaries
using the classic cyber security mode, CIA, which defines the three
characteristics a system aims to uphold, and an adversary may want to target:
confidentiality, integrity and availability. Attacks on the confidentiality of
the system correspond to attempting to extract information on the model itself
or its training data. For example, in the context of a financial system, the
model itself may be regarded as confidential intellectual property, or
likewise in the context of a medical system, the training set may contain
confidential data used to train it. Attacks on integrity aim to cause the
model to behave in an unintelligent and incorrect manner thereby undermining
the integrity of the model. For example, causing the misclassification of
certain inputs to the model or reducing the confidence the model has in its
output. Integrity is crucial to machine learning, and therefore integrity
metrics, like model accuracy are often the primary focus of performance
metrics. In the same way, attacks on availability look to thwart access to
output of such models. One way this can be achieved is by targeting the
model’s quality and consistency as a model which is erratic and unreliable may
consequently become useless. Attacks in this domain will use similar methods
as those used to damage the integrity of the model. Attacks on availability
also include attempts to reduce the access to the model, via denial-of-service
attacks, or model’s performance metrics such as speed so it becomes
inconvenient to use.

###  2.2 Intelligent Security Detection System Threat Model

Intelligent Security Detection Systems which employ similar machine learning
models fit into this general threat model are therefore vulnerable to
adversarial learning attacks. A common approach for such systems is to learn a
model of trustworthy behaviour, from which current behaviours can be compared
to. Therefore, these systems can generally be attacked in the training stage
by poisoning the training set or by logic corruption. Similarly, attacks in
the inference stage would aim to evade detection or potentially trigger false
positives. These false positives may trigger unnecessary reactions by the
system, potentially causing unwanted responses such as the wiping of important
data. A stream of false positives may also strain the system, reducing its
performance, and as such, its availability.

###  2.3 Attack Attributes

Following on from the general threat model described earlier, the adversarial
learning threat model can be broken down into a number of more granular
attributes. These attributes are adversarial falsification, adversarial
knowledge, adversarial specificity, attack frequency and are selected
appropriately depending on the scenario of a specific attack and its
accompanying assumptions and quality requirements [4].

####  2.3.1 Adversarial falsification

Adversarial falsification distinguishes between whether the adversary aims to
produce a false positive attack or false negative and what this means for the
target system. A false positive attack results in negative samples being
incorrectly classified as positive. In the case of an intrusion detection
system this may result in the inappropriate triggering of a response to the
perceived attack such as deletion of non-threatening or important information.
Similarly, the existence of a stream of false positives which the system has
to deal with may negatively impact the performance of the model. On the other
hand, false negative attacks, also known as evasion attacks, work in the
reverse, using the same example of an intrusion detection system, the true
intruder or malware is classified as benign and left completely undetected.

####  2.3.2 Adversarial specificity

Adversarial specificity differentiates between targeted and non-targeted
attacks and usually relates to the case of a multiclass classification.
Targeted attacks look to guide the output of the model in a certain direction
to a specific class. For example, duping the model into predicting all
adversarial examples as one class. However, this isn’t usually necessary and
non-targeted attacks merely aim to classify an adversarial example as any
class other than the original. These are therefore easier to implement and are
either realised by reducing the probability the model classifies correctly or
by selecting one from numerous targeted approaches, the one with the slightest
perturbation.

####  2.3.3 Adversarial frequency

Attack frequency concerns whether the attack is a one-time approached or an
iterative one, which updates a number of times to better optimise the attack.
The latter tend to perform better, however come with extra cost, requiring
greater computational time. In some cases, the one-time attacks are sufficient
or even the only feasible option.

####  2.3.4 Adversarial Knowledge: White-box vs Black-box

Adversarial knowledge relates to adversarial capabilities and more
particularly refers to the case of inference time attacks. Attacks in this
case can be broadly split two distinct categories based on the level of detail
they require of the target model, to be performed. A white box attack is one
which requires an in-depth understanding of the target model. In this way, the
mechanics of the model are transparent. The attacker has full knowledge of the
system and has access to vital information including details such as the
network architecture, parameters, hyper parameters, training data as well as
the ability to gather gradients, and prediction results [4]. On the other
hand, a black box attack attacks the model with no knowledge of the inner
workings of the model. A black box attacker simply requires access to the
model in order to query it to produce the prediction result [5].

In the context of a black box attack, an attacker may be able to produce a
surrogate model which acts in accordance with the inputs and outputs of the
target model. In this way, the attacker has full access to the inner workings
of their surrogate model and can therefore successfully conduct white box
attacks on it by creating adversarial samples using this knowledge. These
adversarial examples can then be used to conduct a black box attack against
the target model.

More recently, a third category has emerged as an alternative to both white-
box and black-box approaches. Grey-box attacks assume knowledge of the target
model’s basic architecture without detailed information such as the network
weights [6]. An example would be where the target model architecture and
training process are known, and a substitute model can be trained to create
adversarial examples [7].

##  3 Types of Attack

Adversarial learning attacks come in different forms. In this section, we
review different types of adversarial learning attacks against deep learning
architectures. We focus the attacks against deep learning architectures as
these architectures are quickly gaining popularity in industrial application.
This is because of the exponentially increasing amount of data that is
becoming available which is used to train these models. The performance of
deep learning models surpasses that of classical machine learning models when
trained on large amounts of data [8].

###  3.1 Fast Gradient Sign Method

The Fast Gradient Sign Method (FGSM) was introduced by [9] as a part of an
explanation of the nature of adversarial examples and what makes neural
networks so vulnerable to them. This behaviour had previously been
demonstrated by [10] whose L-BFGS method exhibited fundamental weaknesses in
neural networks and their training algorithms. [9] argues that neural networks
are vulnerable to linear adversarial perturbation attacks due to their linear
nature and that the analytical exploiting of this behaviour allows for a cheap
and effective method of perturbation generation. This can be demonstrated by
taking the dot product of a weight vector $\mathbf{w}$ and an adversarial
example $\mathbf{\bar{x}}=\mathbf{x}+\boldsymbol{\eta}$:

$\mathbf{w}^{\intercal}\mathbf{\bar{x}}=\mathbf{w}^{\intercal}+\mathbf{x}\mathbf{w}^{\intercal}\boldsymbol{\eta}$
(1)

This perturbation causes the activation to grow by
$\mathbf{w}^{\intercal}\boldsymbol{\eta}$. By assigning $\boldsymbol{\eta}$ to
$sign(\mathbf{w})$ we can maximise the effect of the perturbation, subject to
the max norm constraint. With this, we can now see that the maximum bound of
the change in the activation is

$\mathbf{w}^{\intercal}\boldsymbol{\eta}=\epsilon\cdot\mathbf{w}^{\intercal}sign(\mathbf{w})=\epsilon
mn$ (2)

where $\mathbf{w}$ has $n$ dimensions and the average magnitude of an element
of the weight vector is $m$.

This shows that the activation given by the perturbation grows linearly with
the number of dimensions, $n$. Therefore, in adequately high-dimensional
space, many infinitesimal changes, each capped at $\epsilon$, can sum up to
produce large enough perturbation to the output, able to fool the model.

FGSM looks to work in the opposite manner to gradient descent in order to
maximise the loss. Therefore, we use the cost function used to train the
neural network, $J$, to calculate the gradient of the loss with respect to a
particular input $x$. This allows us to push each pixel in the direction of
the gradient, the direction which most increases the loss. This can be
expressed as

$\boldsymbol{\eta}=\epsilon\cdot sign(\Delta_{\mathbf{x}}\mathbf{J(w,x,y)})$
(3)

where $y$ are the targets associated with $x$.

This is then added to the original input to create the adversarial example.

$\mathbf{x^{*}}=\mathbf{x}+\epsilon\cdot
sign(\Delta_{\mathbf{x}}\mathbf{J(w,x,y)})$ (4)

This one-step gradient method was then used a basis for a series of similar
approaches. The basic iterative method (BIM) (also known as iterative- FGSM)
is one such attack [11]. This extends the FGSM by iterating the process, using
a reduced step size and clipping pixel values of the transitional results in
order to keep them within a defined range of the original input, described as
the ‘$\epsilon$\- neighbourhood’. This can be expressed as:

$\mathbf{x^{*}}_{N+1}=\mathnormal{Clip_{X,\epsilon}}\texttt{\\{}\mathbf{x}_{N}+\alpha\cdot
sign(\Delta_{\mathbf{x}}\mathbf{J(w,x,y)})\texttt{\\}}$ (5)

In the same work, [11] also explores the idea of pushing a particular
misclassification rather than only pushing in the direction of the target
class. This can result in more extreme and interesting misclassifications.
This is especially appropriate in the case that a model classifies a large
number of classes and a misclassification may result in a similar class being
predicted, for example the classification of one breed of dog over another in
ImageNet [12] . In the paper they try to push the classification of the least
likely class ($y_{LL}$) according to the trained model. They name this
approach ‘iterative least-likely class method’. They show that this attempt
can very effectively break the truthful classification even at a relatively
small $\epsilon$ value. This is formalised in a similar way but instead takes
steps in the direction of
$sign\texttt{\\{}\Delta_{\mathbf{x}}logp(y_{LL}|\mathbf{X}\texttt{\\}}$, in
order to maximise, $logp(y_{LL}|\mathbf{X})$, the probability of the desired
output class $y$. In a NN which uses a cross-entropy loss function, this looks
like:

$\mathbf{x^{*}}_{N+1}=\mathnormal{Clip_{X,\epsilon}}\texttt{\\{}\mathbf{x}_{N}-\alpha\cdot
sign(\Delta_{\mathbf{x}}\mathbf{J}(\mathbf{w,x},y_{LL}))\texttt{\\}}$ (6)

The basic iterative method was further built on to incorporate the idea of
momentum, a technique used to accelerate gradient descent-based algorithms.
This approach is known as Momentum-Iterative FGSM [13]. It works by holding
the gradients of the previous $t$ iterations which then contribute to the next
step by an amount dictated by the decay factor $\mu$. For example, when
$\mu=0$, the algorithm reduces into iterative FGSM and when $\mu=1$, all
previously saved gradients are added to the current update. This gradient
update can be formalised as:

$\mathbf{g}_{t+1}=\mu\cdot\mathbf{g}_{t}+\frac{\Delta_{\mathbf{x}}\mathbf{J}(\mathbf{\mathbf{x^{*}}_{t+1}},\mathbf{y})}{||\Delta_{\mathbf{x}}\mathbf{J}(\mathbf{\mathbf{x^{*}}_{t},y})||_{1}}$
(7)

This can then be used in a similar way as the previous methods to generate the
next iterative step in the following way:

$\mathbf{x^{*}}_{t+1}=\mathbf{x^{*}}_{t}+\alpha\cdot sign(\mathbf{g}_{t+1})$
(8)

The use of momentum allows the algorithm to stabilise the updates and navigate
through poor local maxima to obtain a method which generally outperforms FGSM
and BIM.

###  3.2 Jacobian-based Saliency Map Attack

The Jacobian-based Saliency Map Attack (JSMA) is one of the most widely
recognised and used attack techniques which allows for the crafting of
targeted, class specific adversarial samples. It is an example of a white box
attack wherein adversaries have full knowledge of network architectures as
well as training data. Firstly introduced in [14], the proposed attack uses
the forward derivatives of a deep neural network to compute a saliency map.
The saliency map is then used to traverse through the input space and craft
adversarial samples with a target class in mind. Given an input sample
$\mathbf{X}$ classified as $\mathbf{F}(\mathbf{X})=Y$, the adversary attemps
to modify the input $\mathbf{X}$ to create an adversarial sample
$\mathbf{X^{*}}$ such that $\mathbf{F}(\mathbf{X^{*}})\neq Y=\mathbf{Y^{*}}$.
The adversary looks for the smallest perturbation $\delta_{\mathbf{X}}$ to
create an adversarial sample $\mathbf{X^{*}}$, thereby simultaneously avoiding
human detection and satisfying the problem below:

$\arg\\!\min_{\delta_{\mathbf{X}}}\mathinner{\\!\left\lVert\delta_{\mathbf{X}}\right\rVert}\
\mathbf{s.t}\
\mathbf{F}(\mathbf{X}+\delta_{\mathbf{X}})=\mathbf{F}(\mathbf{X^{*}})=\mathbf{Y^{*}}\\\
$ (9)

The first step required in conducting a successful JSMA is taking the forward
derivative of the network with respect to the input sample. The Jacobian
matrix for a given input sample $\mathbf{X}$ is computed and is given by:

$\mathnormal{J_{\mathbf{F}}(\mathbf{X})=\frac{\partial\mathbf{F}(\mathbf{X})}{\partial\mathbf{X}}=\left[\frac{\partial\mathbf{F}_{j}(\mathbf{X})}{\partial
x_{i}}\right]_{i\in 1..M,j\in 1..N}}$ (10)

Computing the Jacobian allows us to see how small variations in the input
space lead to large variations in the output of the network. A large forward
derivative for the $\emph{i}^{th}$ feature in a given sample $\mathbf{X}$,
indicates that changes to this feature will likely result in large changes to
the output of the network. Conversely, features with smaller forward
derivative values are less susceptible to adversarial manipulations, because a
small derivative value indicates that the feature has a small effect on the
output of the network. Thus, computing forward derivatives provides an
efficient method to search the input space and craft adversarial samples
thenceforth.

The use of saliency maps was first introduced in [15] as a technique to
identify the degree to which pixels in images influence the model to classify
the image to a particular class. The aim was to visualize the manner in which
classification models use features/pixels to classify a given image. The
saliency map was adapted and used to make adversarial saliency maps which
allow an attacker to modify the most dominant pixels in an image hence
resulting in a misclassification.

The adversary’s goal is to highlight features from an input image $\mathbf{X}$
that cause the network to classify it to a target class
$t=\hat{y}(x)=\arg\\!\min_{j}\mathnormal{\mathbf{F}_{j}({\mathbf{X}})}$. The
output probabilities for the sample $\mathbf{X}$ of target class must be
increased while simultaneously decreasing the probabilities of other classes.
The adversarial saliency map introduced in [14] elegantly addresses this and
is as follows:

$\mathnormal{S(\mathbf{X},t)[i]=}\begin{cases}0,~{}\text{if}\
J_{it}(\mathbf{X})\ <0\ \text{or}\sum_{j\neq t}J_{ij}(\mathbf{X})>0\\\
J_{it}(\mathbf{X})\cdot\mathinner{\\!\left\lvert\sum_{j\neq
t}J_{ij}(\mathbf{X})\right\rvert},\ \text{otherwise}\end{cases}$ (11)

where $J_{ij}(\mathbf{X})$ is
$J_{\mathbf{F}}[i,j](\mathbf{X})=\frac{\partial\mathbf{F}_{j}(\mathbf{X})}{\partial\mathbf{X}_{i}}$.

Put simply, the first condition in the first line ignores any features that
are negatively correlated with the target class. Alternatively, the second
condition on the first line ignores features that are positively correlated
with classes that are not the target class $\mathnormal{j\neq{t}}$. This
second condition does however allow for negative or constant gradients for
classes $\mathnormal{j\neq{t}}$. In this way features which decrease the
probability of all other classes are chosen as well. The product term on the
second line considers features that both increase the probabilities of the
target classes, as well as decrease the probabilities of all other classes. It
follows that high values of $\mathnormal{S(\mathbf{X},t)[i]}$ correlate with
features that increase the likelihood of the model’s classification to the
target class $t$. This version of the saliency map highlights features the
adversary should increase to result in a misclassification. Conversely, a
complementary version of the saliency map detailing the features an adversary
should decrease to achieve a misclassification can be seen as follows:

$\mathnormal{\tilde{S}(\mathbf{X},t)[i]=}\begin{cases}0,~{}\text{if}\
J_{it}(\mathbf{X})\ >0\ \text{or}\sum_{j\neq t}J_{ij}(\mathbf{X})<0\\\
\mathinner{\\!\left\lvert
J_{it}(\mathbf{X})\right\rvert}\cdot\bigg{(}\sum_{j\neq
t}J_{ij}(\mathbf{X})\bigg{)},\ \text{otherwise}\end{cases}$ (12)

Once the saliency map is computed, the input space must be traversed and
perturbed to satisfy the equation in (9). Researchers from [14] approach this
by introducing an algorithm which iteratively modifies a sample $\mathbf{X}$
by considering two pixels at a time. The algorithm conducts the search across
the input domain $\Gamma$ over all input indices [16]. These pixels are
selected by the saliency map, they are then perturbed, and removed from the
input search domain. This process repeats until the sample is classified by
the neural network as the target class, or until a criterion $\Upsilon$, which
specifies the maximum number of iterations, is reached. The criterion
$\Upsilon$ tries to put an upper limit on the number of perturbed features so
that the adversarial sample remains undetectable by humans.

Results from [14] where researchers conducted the JSMA attack on the LeNet
architecture trained on the MNIST dataset, show that any input digit from a
source class can be perturbed to fit any other class specified by the
adversary with a 97.1% success rate. They also maintain that the adversary
only needs to change an average of 4.02% of input features per sample. They
also conducted experiments to correlate the percentage of distorted pixels for
a given digit, to the humans’ ability to recognize the digit. As a result of
the experiment, they identified the threshold of 14.29% wherein images which
had a greater feature distortion than this, were unrecognizable by humans.

###  3.3 Generative Model Based Method

Developed by [17] in 2014, a Generative Adversarial Network (GAN) is a
generative model which solved the data generation problem without the use of
Markov Chains. A GAN is comprised of two neural network architectures: a
generator, and a discriminator which is sometimes called a critic. These two
components act in opposition to one another and converge to a solution wherein
the generator is able to produce data from the underlying distribution of
input samples $\mathbf{x}$.

Given input samples $\mathbf{x}$, the generator maps noise variables $z$ into
data space to try to produce samples $\mathcal{G}(z)$ that originate from the
input samples underlying distribution $\mathcal{P(\mathbf{x})}$. The
discriminator on the other hand tries to discern whether the samples come from
$\mathbf{x}$ or are from the generator. $\mathcal{C}(x)$ is the output of the
discriminator and is the probability that $x$ came from the input data and not
from the generator. The training of the GAN therefore involves simultaneously
maximising $\mathcal{C}(x)$ and minimising
$\log(1-\mathcal{C(G}(\mathbf{z})))$. The loss function can be described as
follows:

$\mathcal{\min_{G}\max_{C}}V(\mathcal{C,G})=\mathbb{E}_{x\sim
p_{x}(\mathbf{x})}[\mathcal{\log C(\mathbf{x})}]+\mathbb{E}_{z\sim
p_{z}(z)}[\log(1-\mathcal{C(G}(z)))].$ (13)

A clever implementation of a GAN to construct adversarial examples was
demonstrated by researchers in [18]. The aim of the research was to craft more
natural looking adversarial samples in contrast to those generated by already
existing adversarial attack methods such as the FGSM. For a given input, they
search for adversarial examples in the local region of its analogous
representation in a low-dimensional space. To do this, they define the $z$
space. The $z$ vector space is a dense latent low-dimensional representation
of input samples from domain $\mathbf{X}$. Instead of finding an adversarial
sample $x^{*}$ directly, they find an adversarial $z^{*}$, such that mapping
this to the input space produces an adversarial sample $x^{*}$.

The training architecture used consists of GAN and an additional component
which they call an inverter. The inverter acts in the opposite manner in which
a generator in a GAN works. It takes a sample $x$ and maps it into its
analogous dense representation in $z$ space $z^{’}=\mathcal{I}_{\gamma}(x)$.
Due to the unstable nature of GANs in their original form, they used a
modified loss function introduced by [19]. The loss function is called the
Wasserstein loss and it has been proven to help the stabilization of GANs. The
loss function is as follows:

$\min_{\theta}\max_{\omega}\mathbb{E}_{x\sim
p_{x}(x)}[\mathcal{C}_{\omega}(x)]-\mathbb{E}_{z\sim
p_{z}(z)}[\mathcal{C}_{\omega}(\mathcal{G}_{\theta}(z))].$ (14)

They firstly train the WGAN until its generator is able to map random noise
vectors to samples that come from domain $\mathbf{X}$, then train the inverter
$\mathcal{I}$ until it is sufficiently able to produce a low dimensional
representation of samples from domain $\mathbf{X}$. To train the inverter,
they define two error terms: the _reconstruction_ error, and the _divergence_
error. In order to understand these definitions we need to understand the
inputs and outputs of parts of the model. The output of the inverter
$\mathcal{I}_{\gamma}(x)$ is fed into the trained generator to get
$\mathcal{G_{\theta}(I_{\gamma}}(x))$. The difference between this output and
the sample $x$ is called the reconstruction error. The generator on the other
hand gives the output $\mathcal{G}_{\theta}(z)$, for a noise vector $z$. This
is fed into the inverter to produce
$\mathcal{I}_{\gamma}(\mathcal{G}_{\theta}(z))$. The difference between this
value and the sampled $z$ is called the divergence error. A representation of
their architecture is demonstrated in Figure 1. The loss function used to
train the inverter aims to minimise the summation between these two values,
shown in Equation 15:

$\min_{\gamma}\mathbb{E}_{x\sim
p_{x}(x)}\mathinner{\\!\left\lVert\mathcal{G}_{\theta}(\mathcal{I}_{\gamma}(x))-x\right\rVert}+\lambda\cdot\mathbb{E}_{z\sim
p_{z}(z)}[\mathcal{L}(z,\mathcal{I}_{\gamma}(\mathcal{G}_{\theta}(z)))]$ (15)

Source: [18]

Figure 1: The training procedure of the GAN and the Inverter. The diagram
illustrates the components that are compared to build the loss function of the
inverter

They researchers in [18] search for adversarial samples using an iterative
stochastic search and a hybrid shrinking search. The former starts the search
in a small region then incrementally expands the search region until an
adversarial sample is found. The latter does the opposite, it begins with a
wide search range then repeatedly narrows it down until an adversarial sample
is reached. This algorithm proved to be 4x more computationally efficient than
the aforementioned algorithm. They tested their adversarial sample generation
technique to generate text and image adversaries. They show that using this
technique, they were able to generate natural looking, and more legible,
adversarial examples.

###  3.4 Universal Attacks

Universal attacks are a special type of attack used to generate a single,
fixed, universal perturbation image, which, when added to any natural image,
causes it to be misclassified with a high probability. Simply put, a single
perturbation vector is generated with the ability to fool the target model on
most natural images [20]. These universal perturbations have also been shown
to work very well, managing to fool multiple state of the art classification
classifiers, with fooling rates of up to 93 %. It has also been shown that
some universal perturbations computed for specific networks, generalise well
and perform relatively well on completely different architectures, and so are
described in [20] as ‘to some extent, doubly universal’. For example, a
universal perturbation fashioned for a VGG-16 model still managed to a minimum
fool rate of 56 % across the other 5 models in the study.

The existence of successful universal perturbations highlights an inherent
vulnerability in deep neural networks. The decision boundaries of these
networks are highly dimensional and therefore contain redundancies and large
“geometric correlations between different parts of the decision boundary”
[20]. This indicates the existence of a lower dimensional subspace which
captures these correlations by collecting normals to different regions of the
decision boundary. This is illustrated in figure 2. This can be exploited as
perturbations in this subspace are thus likely to lead to datapoints in this
region fooling the model.

Source: [20]

Figure 2: Illustration of a trivial example of lower dimensional subspace S
containing normal vectors to the decision boundary. It displays the
superimposition of three datapoints ${xi}{i=1}^{3}$, and the adversarial
examples, ${ri}{i=1}^{3}$ that send the respective points to the decision
boundary ${Bi}{i=1}^{3}$. It is important to note that all ${ri}{i=1}^{3}$lie
within the subspace S.

##  4 Countermeasures

Whilst adversarial attacks can be powerful, there are a number of
countermeasures which help to defend against or even mitigate them. These
approaches can either be proactive or reactive. The goal of proactive
approaches is to strengthen the model itself against adversarial attacks; in
essence making it more robust. Alternatively, reactive approaches aim to
detect adversarial attacks and react appropriately, completely regardless of
the model [4] [7] [21].

Common proactive approaches include adversarial training and defensive
distillation. Adversarial training involves retraining the model on a dataset
including adversarial examples [9], while defensive distillation trains a new
model using the outputted class probabilities of the original model as
training labels [22].

Denoising is another class of proactive approaches which, as the name
suggests, look to reduce noise of the input to highlight and clean the effect
of adversarial tampering. Feature squeezing is the primary example of this and
works by reducing the dimensions of the input, thereby reducing the search-
space available for an adversary to tamper. Adversarial examples can be
identified by comparing the output of the model given both the original and
squeezed input [23]. Singular Value Decomposition can also be used similarly
to denoise input data [24].

Reactive approaches include adversarial detection or input reconstruction.
Adversarial detection generally works by utilising a separate mechanism for
determining the legitimacy of input data. There have been a number of
detection methods proposed with some success. It should be noted that while
some approaches naturally lead to a suitable fix [25], some do not and thus
would probably deal with the attack in an appropriate manner such as to ignore
the adversarial data completely [26]. Input reconstruction works to transform
adversarial examples back to their original form, while normal, clean examples
are left completely or minimally changed.

###  4.1 Proactive

####  4.1.1 Adversarial Training

Adversarial training is a method of defending against adversarial examples
that is intuitive and commonly accepted as one of the most successful
techniques for improving model robustness when put into practice. It
reinforces the neural network through the proactive inclusion of adversarial
samples in the training data and can be seen through the perspective of a
robust optimisation for a min-max problem [27]. The concept behind adversarial
training is most simply formulated by [28], representing the adversarial loss
as $J(\theta,x^{\prime},y)$, in which the network weights are $\theta$,
adversarial inputs are $x^{\prime}$ and the ground truth label is $y$:

$\min_{\theta}\max_{D(x,x^{\prime})<\eta}J(\theta,x^{\prime},y)$ (16)

The term on the left in the maximisation constraint $D(x,x^{\prime})<\eta$ of
Equation 16 denotes a metric for the distance between the original input $x$
and adversarial example $x^{\prime}$. This maximisation is used to generate an
adversarial example that is the most effective and hence has the largest value
for the distance metric. The method for generating the example can vary, such
as Fast Gradient Sign Method (detailed in Section LABEL:subsubsubsec:fgsm on
attacks) and Projected Gradient Decent.

The minimisation term outside of the maximisation is the standard neural
network procedure for training by minimising the loss through fine-tuning of
the network’s weights. The result is a network robust to the adversarial
attack used for generating the adversarial example in the training stage and
can achieve state-of-the-art accuracies on many competitive benchmarks.
Earlier implementations of adversarial trained models estimated the loss
function linearly and hence were unprotected against and vulnerable to
iterative adversarial generation methods; recent advances in adversarial
training are resistant to even iterative attacks [27].

####  4.1.2 Defensive Distillation

Defensive distillation is a process that uses knowledge distilled from a
neural network architecture in order to increase its robustness to adversarial
examples [22]. This idea was extended from work done in [29], where they
reasoned that computational complexity of larger architectures could be
bypassed by using smaller architectures that contain knowledge of larger,
previously trained neural networks. The intuition behind this is that
knowledge from complex architectures can be fitted onto smaller resource-
constrained devices [22].

The distillation training procedure involves the utilisation of ‘soft labels’
output by the first neural network, to train a second neural network. The only
difference is in the training procedure of soft labels. [22] argue that the
use of soft targets in training is justified because the probabilities encode
extra knowledge about each class. The vector of soft labels that are output
for each datapoint, represents the probability of membership to each of the
classes, with the highest probability being the most likely class the
datapoint belongs to. The soft labels for a datapoint are calculated in (17),

$qi=\frac{\exp(z_{i}/T)}{\sum_{j}\exp(z_{j}/T)}$ (17)

where _$q_{i}$_ is the class probability for the _i-th_ class, _$z_{i}$_ the
input to the softmax layer and _T_ the temperature, which controls how ’soft’
the probability is distributed over the classes [29]. The higher the
temperature value used, the more the output probability distribution resembles
a uniform one. A breakdown of the distillation procedure is illustrated in
Figure 3.

Source: [22]

Figure 3: The training produce of a distilled network as proposed by [22]. The
network on the left is initial network which outputs probability vector
prediction. The figure on the left uses these probability targets as labels to
train the distilled network.

[22] built a distilled network using the same architecture which provided the
soft labels. They used the same high-temperature values to train both
networks, then set the temperature back to 1 during test time. They trained,
tested, and measured the model’s sensitivity to adversarial examples when
trained on the MNIST dataset and the CIFAR10 dataset. They crafted adversarial
samples using a succession of two techniques: direction sensitivity
estimation, and perturbation selection.

A reduction in adversarial sample success rate from 95.89% to 0.45% was
noticed in the model trained on the MNIST dataset and a drop from 87.89% to
5.11% on the model trained in the CIFAR10 dataset. The percentage changes
observed were from a model with no distillation to a distilled model, both
trained with temperature values of 100. Conflicting results are obtained in
[30] where a distilled network was used in defending text processing neural
networks. Little to no change in the robustness of their network was obtained
when a distilled network was used. They speculate that the use of distillation
in text processing is misplaced, as gradients are not directly added to the
input, but are rather just used to highlight the importance of a given word.

The distillation method seems to succumb to the black box attack proposed in
[5]. The researchers recreated the models trained in [22] and successfully
implemented an FGSM attack under black box conditions. They accredit the
success of the attack to a phenomenon called _gradient masking_ , the method
in which distillation defence works. They postulate that the distillation
method makes the gradient harder for the adversary to obtain, but does not
nullify the attack vector itself.

An extension to the defensive distillation was made by co-authors of the
original papers of [22]. In the revised paper [31], they attempt to bypass the
gradient masking effect by making modifications to the original defensive
distillation process. They add two major additional components to their new
model: an uncertainty metric for the inputs to the network, and an outlier
class with the sole purpose of classifying adversarial samples. These are
accompanied by changes made to the training procedure of networks, namely a
modification to the loss function in the distilled network, and the setting of
the temperature value to 1 at all times. The result of this was a network that
proved to be less likely to suffer from gradient masking. Because of this,
black box attacks were more likely to be flagged by the model as outliers, as
supposed to white-box attacks. They conclude by stating the following,
‘Defensive distillation’s most appealing aspect remains that it does not
require that the defender generate adversarial samples’ [31].

####  4.1.3 Denoising

Denoising is a class of proactive defensive measures to mitigate the efficacy
of adversarial attacks by removing or reducing the perturbations introduced in
the adversarial example. Denoising can be categorised into two subsections,
dependent on the stage of the model where the denoising occurs. One direction
opts to cleanse the original input, before passing it to the model, while the
other sterilises the extracted features learnt through the NN algorithm. [28]

####  4.1.4 Feature Squeezing

Feature squeezing is an accurate and cost-effective adversarial
countermeasure, developed on the fundamental principle that a high frequency
of the input spaces of features are significantly larger than necessary.
Considerable opportunities to generate adversarial examples are afforded to an
adversary due to the expansive input space. This defensive measure limits the
number of input features by computing and removing any that are unnecessary,
greatly decreasing the freedom of the adversary to generate examples [23]. The
method inherits its name due to this “squeezing”. Adversarial detection is
trivial after the squeezing, the outputs of the NN model for both the original
input and squeezed input are compared and if there are outstanding differences
then the input can be labelled as adversarial. The exact level of contrariety
at which the determination of an adversary occurs can be controlled through a
threshold parameter. Legitimate inputs detected will have their model
predictions outputted, however those of adversarial inputs can be discarded
[23]. Although feature squeezing’s primary application is image
classification, the method is transferable to other domains.

Source: [23]

Figure 4: The methodology proposed by [23]. The model’s prediction on a
squeezed and original input image are compared at ’d1’ and ’d2’; the threshold
used to classify adversarial examples is denoted as ’T’.

The feature squeezing proposed in [23] is achieved through two straightforward
methods. The first approach is the reduction of each pixel’s colour depth
within an image. Secondly, differences between the pixels are diluted via
spatial smoothing. Feature squeezing through these methods have been
demonstrated to considerably strengthen the robustness of the predictions from
a neural network model. Legitimate (non-adversarial) inputs are left unaltered
and hence maintain the same accuracy; static (non-adaptive) adversarial
examples have correct labels predicted using the squeezed input [23].

Neural networks require an assumption of a continuous input space due to their
nature as differentiable models. This is contrary to a digital device’s
restriction to approximate natural continuous data to a discrete calculation.
Matrices of pixels constitute digital images, each pixel being a number value
representing a specified colour. Colour bit depths in popular image
representations can often lead to features that are not relevant and hence
their reduction can maintain the predictive capability of the model while
reducing the freedom of an adversary to generate an attack. One of the popular
representations of images is 8-bit greyscale, affording $2^{8}=256$ possible
values, in this case ranging from white at 0, through various shades of grey,
to black at 255. An extension of this 8-bit representation uses three separate
colour channels: red, green, and blue, each with their own respective 8-bits,
to create a 24-bit colour image, producing $2^{24}>16\ million$ different
possible colours for each pixel. The human eye is only able to distinguish
around ten million colours [32], hence why this representation is often
referred to as ‘True Colour’. Larger bit colour depths are often preferred by
people as the image produced is a more natural representation and closer to
the truth, however they are not necessary for understanding and recognising an
image, hence why black-and-white images are interpretable.

The implementation of bit depth reduction in [23] takes an input and produces
an output on an identical numerical scale $[0,1]$, therefore the NN model
requires no alteration. To reduce an image to $i$-bit depth, such that $1\leq
i\leq 7$, the product of the input value and $2i–1$ is taken and subsequently
rounded and re-scaled to $[0,1]$ before being divided by $2i–1$. The process
of rounding to integers during this method causes the reduction in information
capacity from an 8-bit to $i$-bit representation [23].

Source: [33]

Figure 5: Application of Gaussian local smoothing on an image, with kernel
size of a) $3\times 3$ , b) $5\times 5$ , c) $7\times 7$. Increasing levels of
zero padding are required, and visible in the thicker black borders around the
image.

Spatial smoothing is a common image processing procedure for reducing the
amount of noise within the image and is also known as blur. The smoothing
applied in the feature squeezing proposed by [23] is categorised into to
variants, local and non-local smoothing.

Local smoothing is a smoothing mechanism that makes use of the neighbouring
pixels within a window (also known as a kernel) around a target pixel,
assigning different weights to them depending on the exact smoothing method
used, such as Gaussian, mean or median smoothing [33]. In [23], median
smoothing is used, a window slides over the image in a raster scan fashion,
replacing the central pixel value with the median of all pixels in the window.
Depending on the use of padding, this method does not reduce the size of the
image and smooths out values across pixels that are nearby. Adjacent pixels
become more similar in value and this process can cause features to be
squeezed out. Smoothing methods such as median smoothing have variable
parameters that are configurable, such as the window size which can be set
from 1 pixel at a minimum, to the dimensions of the whole image at a maximum.
For feature squeezing with median smoothing, a square kernel of size 2x2 with
reflection padding at the edges is used. A window of this shape takes the
bottom right pixel as the target pixel and biases towards the larger value
when calculating the median from an even set of numbers. The padding method
used mirrors the image along the edges for calculating the values of pixels at
the extremities, as there would not be pixels to fill the window. This
smoothing is commonly known for being highly practical for removing salt-and-
pepper noise from images, random and sparse occurrences of black and white
pixels, while being able to maintain defined edges of objects within the image
[23][33]. By contrast, non-local smoothing applies its smoothing methods over
a collection of pixels that extends much further than a small, localised
window. For a target image patch, non-local smoothing will calculate other
patches that are similar across an area of the image and replace the target
patch with the computed average of the patches. Due to the random and sparse
distribution of noise within an image, it is assumed that the mean of the
noise will be zero and hence this method is able to eliminate noise from the
image while leaving edges unaffected. The mechanism used to assign weights to
similar patches during the averaging closely resembles local smoothing:
Gaussian, mean and median operators are commonly used [34]. [23] use a popular
adaptation of a Gaussian kernel. This enables additional control of the
deviation from the mean, as well as the window size across which similar
patches are searched for, the patch size and a variable for controlling the
strength at which the filter is applied.

####  4.1.5 SVD

In adversarial attacks against pattern recognition systems, the adversary
introduces specially crafted perturbations to the pixel values of the target
image to generate what are known as adversarial examples. As high-dimensional
space is linear in nature, even a slight pixel change can have a large impact
and be dramatically increased in the feature space, ultimately misleading the
neural network model to misclassify an image with a potentially high
confidence. Singular Value Decomposition is a commonly used technique when
analysing multivariate data [35]. Research has concluded that the SVD of an
image can also be used as an adversarial defence by computing an optimal
approximation of the image matrix in terms of square loss, as a solution for
eliminating the perturbations caused by the adversary [24].

In order to robustly defend against an adversarial attack, it is necessary for
the humanly imperceptible perturbations introduced to the input before the
input is passed onto the target model. The difference between the unaltered,
original input, $X$, and the adversarial example, $X^{ADV}$, is calculated as
the minute value $\eta$:

$||X^{ADV}-X||=\eta$ (18)

A matrix’s distribution characteristics are described by the unique singular
values (principal components) $\delta_{1},\delta_{2},\ldots,\delta_{r},\forall
A\in C^{m*n}$. A linear transformation of the matrix $A$ maps its points in
$m$-dimensional to $n$-dimensional space [24]. The SVD of a matrix decomposes
the transformation into three separate matrix parts, $U$, $s$ and $V$. $U$ and
$V$ are complex unitary matrices whereas $v$ is a diagonal matrix. Lossy
compression of an image can be achieved through an orthogonal transformation,
which retains defining features of the image but with reduced data. This is
common for a variety of techniques and processes such as the storage,
transmission and analysis of images. A digital image $A$ can be interpreted as
having a vertical and horizontal time-frequency in a two-dimensional
representation, upon which the singular value decomposition can be computed
[35][24]:

$A=UsV^{\top}=U\begin{bmatrix}\delta\&0\\\
0\&0\end{bmatrix}V^{\top}=\sum_{i=1}^{j}\delta_{i}u_{i}v_{i}^{\top}$ (19)

In Figure 6, a concise visualisation of the singular value decomposition of a
$2\times 2$ matrix is displayed. In this figure and Equation 19, matrix $U$
and $V$ have the respective column vectors $u_{i}$ and $v_{i}$ and
$\delta_{i}$ is a non-zero and real singular value from the matrix $s$. A
summation over $j$ many sub-graphs, formed from the product of the
corresponding column vectors and values from the decomposition, will reform
the original image. Ordering these sub-graphs by singular value and only
including a specified number of the largest will preserve the most information
in the image possible. The information about the geometry and texture of the
image are predominantly represented in matrices $V$ and $U$, whilst $s$ stores
the energy information in the singular values [24]. Singular values have three
distinct and provable properties: singular values are stable, singular values
are proportional invariant, singular values have rotation invariance and
matrix approximation can be optimally achieved by selecting singular values of
matrix $A$ greater than a certain threshold. This final property is essential
for data compression via matrix reduction and the principal justification for
the use of SVD as a defensive method to remove the imperceptible perturbations
in an adversarial example whilst maintaining all other core information.

Figure 6: Diagram based on a similar figure by [24], providing a detailed and
concise breakdown of the SVD process.

###  4.2 Reactive

####  4.2.1 Adversarial Detection

Adversarial detectors can take many forms. One of the most basic adversarial
detection approaches are DNN-based binary classifiers to estimate the
probability an input is adversarial. [36] created exactly this and were able
to show that a relatively small NN can act as a subnetwork to detect
adversarial examples “surprisingly well”. A slightly more robust example is
that used in SafetyNet [26], which is able to detect, with high accuracy,
adversarial examples built from a number of attacking approaches. These
detected adversarial examples are rejected by the model based on their
classification confidence. SafetyNet also is shown to be relatively robust to
type II attacks which look not only to fool the main classifier into
mislabelling the input, but also to evade the detector. The system works by
taking the binary threshold of each ReLU layer’s output as features to a
radial basis function (RBF)-SVM classifier, which then detects adversarial
examples. They argue this works well even when the adversary is aware of the
detector. This is because it forces the adversary to solve the difficult
optimisation problem of discovering an optimal value for both creating
adversarial examples, as well as optimising the new features of the
adversarial detector.

Similarly, [37] introduced two new effective indicators of adversarial
examples, which work well together in complementary situations: Bayesian
neural networks and kernel density estimation. The kernel density estimation
approach works by using the features on the final hidden layer of the NN to
build a picture of the submanifolds of each class. From this we can estimate
the distance a point is from the submanifold of the appropriate target. This
approach works well for adversarial examples far from the submanifold of the
target, however, may not when closer. In this case, they show that Bayesian
neural networks can be used to distinguish adversarial examples by estimating
the uncertainty of inputs. This works because they were able to show that
putting adversarial examples through Baysian models, such as the Gaussian
process [38], tends to produce higher levels of uncertainty.

A different approach, used as part of PixelDefend [25], an input
reconstruction approach, looks at the training distribution and where example
inputs fall within that. They demonstrated that PixelCNN [39] [40], a state of
the art neural density model, can be used to effectively identify
perturbations. They showed that the distribution of log-likelihoods
highlighted a substantial difference between normal images and adversarial
ones and that even small perturbations of 3% can lead to sizable reductions in
the log-likelihood. Along with the permutation test [41] they were then able
to calculate the exact p-values. These p-values define the probability that an
input is drawn from the training distribution. By examining the generated
p-values the adversarial examples are distinguishable as they generally lie in
lower probability regions.

####  4.2.2 Input Reconstruction

The key idea behind input reconstruction is to transform the adversarial
example back into clean data before passing it through to the model. A good
example of this approach is, PixelDefend [25] which reconstructs adversarial
images back to the training distribution. As mentioned above, PixelDefend uses
PixelCNN as their detecting mechanism. PixelDefend leaves datapoints that are
already sufficiently within the distribution of the training data unchanged.
Meanwhile, applying small deviations to datapoints far enough outside the
training distribution. With the pixelCNN distribution, $p_{CNN}(X)$, used as
an approximation of the probability of $x$, $p(X)$, they frame the problem as
an optimisation one. The problem is to maximise $p(X^{*})$ subject to
constraints on the size of the perturbation, $\epsilon_{defend}$. I.e
$\max{p(X^{*})}$

To achieve this, they take a greedy approach not too dissimilar to the
generation of images from PixelCNN, with the added constraint that the image
should be within the $\epsilon$-neighbourhood. As PixelCNN is an
autoregressive model, this process is typically a slow one, so they use a
faster version proposed by Ramachandran [42].

A similar example is MagNet [43] which is comprised of two components: a
detector, which rejects examples too far from the manifold boundary, and the
reformer, which given an input $x$, endeavours to generate an $x’$ as a close
approximation of $x$ that falls on or near the manifold. It then gives $x’$ to
the target classifier in the place of $x$, thereby reconstructing the
adversarial example. This process is illustrated in 7

Source: [43]

Figure 7: Simple 2-D illustration of how MagNet works. The manifold of normal
examples is represtned by the curve. Green circles and red crosses correspond
to normal examples and adversarial examples respectively. The transformation
by an autoencoder is show by the red arrow. The detector measures
reconstruction errors and rejects those with errors too large, e.g. cross 3.
Adversarial examples closer to the manifold are reconstructed to the original
by the reformer.

In [43], the detector is realised as a binary classifier, mapping inputs to
either adversarial or clean, as discussed above. The implementation opted for
is based on probability divergence and is very similar to the kernel density
estimation used in [37]. They make the use of an autoencoder trained to learn
the features of the training data, as the detector. An autoencoder is used to
encode its input and then decode it in an attempt to recreate the input. A
reconstruction error is used to approximate the distance between an input and
the learned manifold of clean examples. Now, if an input is drawn from the
same data generation process as the training set, the reconstruction error
should be low, and thus the reconstruction error can act as distance measure
from the manifold of normal, clean data. However as discussed previously, this
tends to be less effective when the reconstruction error becomes smaller as
adversarial examples are made closer to the original. This can be dealt with
by passing the both the inputs and the output of the autoencoder through the
model and comparing. If for example, $x$ is clean and drawn from the same
process as the training data, the output of the autoencoder $ae(x)$ should be
very similar. Therefore, the output of the model should be similar also, i.e
$f(x)=f(ae(x))$. However, if $x’$ is an adversarial example, even if the
reconstruction error is low, $f(x’)$ and $f(ae(x’))$ can be significantly
different. This works by taking advantage of the target classifier, assuming
the use of softmax at the last layer of their network, an assumption which is
the case in most neural network classifiers.

The reformer tries to reconstruct adversarial inputs appropriately, so they
are close to the original image, while leaving clean inputs unchanged. Similar
to the detector, [43] propose the use of autoencoders in the implementation of
the reformer, to take advantage of the distribution of normal data. They
create a large number of autoencoders which are all trained to minimise the
reconstruction error on the training set. This way, given a clean datapoint,
the output should be very similar, but given an adversarial datapoint, the
expected output should approximate the adversarial example but be closer to
the manifold of the normal examples. With multiple autoencoders, MagNet
randomly picks one as each defensive device every time. This forces the
adversary to have to train their adversarial examples to work on multiple
autoencoders at the same time, assuming they cannot predict which autoencoder
is picked, and provides diversity in defence.

##  5 Conclusions

This work has provided a comprehensive overview of the principles of
adversarial machine learning attacks and existing countermeasures. Our
analysis has shown that this threat poses a very real danger to machine
learning algorithms, which play a massive and growing role in all aspects of
modern society, including safety-critical applications and advance defence
systems. There remain several outstanding challenges including efficient
deployment mitigating techniques that balance the security risks with the
expected overheads. This is specially true for resources-constrained systems.
This requires the development of specific implementations of these
countermeasures tailored to each application scenario.

##  6 Acknowledgments

This research was partly funded by the royal academy of engineering (grant No.
IF202136).

## References

  * [1] Matthew Jagielski et al.  “Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning”  In _2018 IEEE Symposium on Security and Privacy (SP)_ , 2018, pp. 19–35  DOI: 10.1109/SP.2018.00057
  * [2] Battista Biggio, Blaine Nelson and Pavel Laskov  “Poisoning Attacks against Support Vector Machines”, 2013  arXiv:1206.6389 [cs.LG]
  * [3] Nicolas Papernot, Patrick McDaniel, Arunesh Sinha and Michael Wellman  “Towards the Science of Security and Privacy in Machine Learning”, 2016  arXiv:1611.03814 [cs.CR]
  * [4] Xiaoyong Yuan, Pan He, Qile Zhu and Xiaolin Li  “Adversarial Examples: Attacks and Defenses for Deep Learning”  In _IEEE Transactions on Neural Networks and Learning Systems_ 30.9, 2019, pp. 2805–2824  DOI: 10.1109/TNNLS.2018.2886017
  * [5] Nicolas Papernot et al.  “Practical Black-Box Attacks against Machine Learning”, 2017  arXiv:1602.02697 [cs.CR]
  * [6] B.. Vivek, Konda Reddy Mopuri and R. Babu  “Gray-box Adversarial Training”  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018
  * [7] Kui Ren, Tianhang Zheng, Zhan Qin and Xue Liu  “Adversarial Attacks and Defenses in Deep Learning”  In _Engineering_ 6.3, 2020, pp. 346–360  DOI: https://doi.org/10.1016/j.eng.2019.12.012
  * [8] Christian Janiesch, Patrick Zschech and Kai Heinrich  “Machine learning and deep learning”  In _Electronic Markets_ Springer, 2021, pp. 1–11
  * [9] Ian J. Goodfellow, Jonathon Shlens and Christian Szegedy  “Explaining and Harnessing Adversarial Examples”, 2015  arXiv:1412.6572 [stat.ML]
  * [10] Christian Szegedy et al.  “Intriguing properties of neural networks”, 2014  arXiv:1312.6199 [cs.CV]
  * [11] Alexey Kurakin, Ian Goodfellow and Samy Bengio  “Adversarial examples in the physical world”, 2017  arXiv:1607.02533 [cs.CV]
  * [12] Olga Russakovsky et al.  “ImageNet Large Scale Visual Recognition Challenge”  In _International Journal of Computer Vision (IJCV)_ 115.3, 2015, pp. 211–252  DOI: 10.1007/s11263-015-0816-y
  * [13] Yinpeng Dong et al.  “Boosting Adversarial Attacks with Momentum”, 2018  arXiv:1710.06081 [cs.LG]
  * [14] Nicolas Papernot et al.  “The limitations of deep learning in adversarial settings”  In _2016 IEEE European symposium on security and privacy (EuroS &P)_, 2016, pp. 372–387  IEEE
  * [15] Karen Simonyan, Andrea Vedaldi and Andrew Zisserman  “Deep inside convolutional networks: Visualising image classification models and saliency maps”  In _arXiv preprint arXiv:1312.6034_ , 2013
  * [16] Rey Wiyatno and Anqi Xu  “Maximal jacobian-based saliency map attack”  In _arXiv preprint arXiv:1808.07945_ , 2018
  * [17] Ian Goodfellow et al.  “Generative adversarial nets”  In _Advances in neural information processing systems_ 27, 2014
  * [18] Zhengli Zhao, Dheeru Dua and Sameer Singh  “Generating natural adversarial examples”  In _arXiv preprint arXiv:1710.11342_ , 2017
  * [19] Martin Arjovsky and Léon Bottou  “Towards principled methods for training generative adversarial networks”  In _arXiv preprint arXiv:1701.04862_ , 2017
  * [20] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi and Pascal Frossard  “Universal adversarial perturbations”, 2017  arXiv:1610.08401 [cs.CV]
  * [21] Yao Deng et al.  “An Analysis of Adversarial Attacks and Defenses on Autonomous Driving Models”, 2020  arXiv:2002.02175 [eess.SP]
  * [22] Nicolas Papernot et al.  “Distillation as a defense to adversarial perturbations against deep neural networks”  In _2016 IEEE symposium on security and privacy (SP)_ , 2016, pp. 582–597  IEEE
  * [23] Weilin Xu, David Evans and Yanjun Qi  “Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks”  In _CoRR_ abs/1704.01155, 2017  arXiv: http://arxiv.org/abs/1704.01155
  * [24] Fei Wu, Limin Xiao, Wenxue Yang and Zhu Jinbin  “Defense against adversarial attacks in traffic sign images identification based on 5G”  In _EURASIP Journal on Wireless Communications and Networking_ 2020, 2020, pp. 173  DOI: 10.1186/s13638-020-01775-5
  * [25] Yang Song et al.  “PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples”, 2018  arXiv:1710.10766 [cs.LG]
  * [26] Jiajun Lu, Theerasit Issaranon and David Forsyth  “SafetyNet: Detecting and Rejecting Adversarial Examples Robustly”, 2017  arXiv:1704.00103 [cs.CV]
  * [27] Tao Bai et al.  “Recent Advances in Adversarial Training for Adversarial Robustness”, 2021  arXiv:2102.01356 [cs.LG]
  * [28] Kui Ren, Tianhang Zheng, Zhan Qin and Xue Liu  “Adversarial Attacks and Defenses in Deep Learning”  In _Engineering_ 6.3, 2020, pp. 346–360  DOI: https://doi.org/10.1016/j.eng.2019.12.012
  * [29] Geoffrey Hinton, Oriol Vinyals and Jeff Dean  “Distilling the knowledge in a neural network”  In _arXiv preprint arXiv:1503.02531_ , 2015
  * [30] Marcus Soll, Tobias Hinz, Sven Magg and Stefan Wermter  “Evaluating defensive distillation for defending text processing neural networks against adversarial examples”  In _International Conference on Artificial Neural Networks_ , 2019, pp. 685–696  Springer
  * [31] Nicolas Papernot and Patrick McDaniel  “Extending defensive distillation”  In _arXiv preprint arXiv:1705.05264_ , 2017
  * [32] “Color in business, science, and industry. By Deane B. Judd. John Wiley & Sons, Inc., New York, 1952. 401 pp. Illustrated. 15.5 Ã— 24 cm. Price $6.50”  In _Journal of the American Pharmaceutical Association (Scientific ed.)_ 42.12, 1953, pp. 757  DOI: https://doi.org/10.1002/jps.3030421221
  * [33] Mark Nixon and Alberto S. Aguado  “Feature Extraction & Image Processing, Second Edition”  USA: Academic Press, Inc., 2008
  * [34] Antoni Buades, Bartomeu Coll and Jean Michel Morel  “On image denoising methods”, 2004
  * [35] Michael E. Wall, Andreas Rechtsteiner and Luis M. Rocha  “Singular Value Decomposition and Principal Component Analysis”  In _A Practical Approach to Microarray Data Analysis_ Boston, MA: Springer US, 2003, pp. 91–109  DOI: 10.1007/0-306-47815-3˙5
  * [36] Jan Hendrik Metzen, Tim Genewein, Volker Fischer and Bastian Bischoff  “On Detecting Adversarial Perturbations”, 2017  arXiv:1702.04267 [stat.ML]
  * [37] Reuben Feinman, Ryan R. Curtin, Saurabh Shintre and Andrew B. Gardner  “Detecting Adversarial Samples from Artifacts”, 2017  arXiv:1703.00410 [stat.ML]
  * [38] Carl Edward Rasmussen and Christopher K.. Williams  “Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)”  The MIT Press, 2005
  * [39] Aaron Oord et al.  “Conditional Image Generation with PixelCNN Decoders”  In _Advances in Neural Information Processing Systems_ 29 Curran Associates, Inc., 2016  URL: https://proceedings.neurips.cc/paper/2016/file/b1301141feffabac455e1f90a7de2054-Paper.pdf
  * [40] Tim Salimans, Andrej Karpathy, Xi Chen and Diederik P. Kingma  “PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications”, 2017  arXiv:1701.05517 [cs.LG]
  * [41] Bradley Efron and Robert J. Tibshirani  “An Introduction to the Bootstrap”, Monographs on Statistics and Applied Probability 57  Boca Raton, Florida, USA: Chapman & Hall/CRC, 1993
  * [42] Prajit Ramachandran et al.  “Fast Generation for Convolutional Autoregressive Models”, 2017  arXiv:1704.06001 [cs.LG]
  * [43] Dongyu Meng and Hao Chen  “MagNet: a Two-Pronged Defense against Adversarial Examples”, 2017  arXiv:1705.09064 [cs.CR]