11institutetext: Drexel University, Philadelphia, PA, USA
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Defenses Against Multi-Sticker Physical Domain Attacks on Classifiers

Xinwei Zhao  0000-0002-4328-4846 Matthew C. Stamm  0000-0002-3986-4039

###### Abstract

Recently, physical domain adversarial attacks have drawn significant attention
from the machine learning community. One important attack proposed by Eykholt
et al. can fool a classifier by placing black and white stickers on an object
such as a road sign. While this attack may pose a significant threat to visual
classifiers, there are currently no defenses designed to protect against this
attack. In this paper, we propose new defenses that can protect against multi-
sticker attacks. We present defensive strategies capable of operating when the
defender has full, partial, and no prior information about the attack. By
conducting extensive experiments, we show that our proposed defenses can
outperform existing defenses against physical attacks when presented with a
multi-sticker attack.

###### Keywords:

Real-world adversarial attacks, Defenses, Classifiers, Deep learning

##  1 Introduction

Deep neural networks have been widely used for many visual classification
systems, such as autonomous vehicles [13, 35] and robots [38].However, deep
neural networks are vulnerable to adversarial attacks [5, 6, 14, 17, 18, 21,
22, 24, 25, 27, 28, 33, 34]. By modifying the pixel values of an image, many
classifiers can be fooled.

Recently, attacks that can operate in the physical world have started to
attract increasing attention [1, 4, 11]. While some physical domain attacks
require crafting a new object [1, 11, 19], other attacks can fool the
classifiers by adding one or a few physical perturbations, such as printable
patches [4, 11] on or next to an object. The adversarial patch attack creates
one universal patch that can be used to attack an arbitrary object once it is
trained, regardless of scale, location and orientation [4]. The camouflage art
attack uses black and white stickers that are applied to an object such as a
traffic sign to make a classifier believe it is a different object. [11] Since
these physical perturbations are very concentrated and confined to small
regions, it is easy for attackers to craft these physical perturbations and
put the attack in practice in the real world.

Previous research shows that defenses against digital domain attacks [2, 3, 7,
8, 9, 10, 12, 15, 16, 20, 22, 26, 29, 30, 32, 36] may not be able to defend
against physical domain attacks, such as the camouflage art attack, because
physical perturbations are usually stronger than those produced by digital
domain attacks. Some recent research has been done to defend against physical
domain attacks [7, 16, 20, 26, 37].

Figure 1: Attacked signs (a) & (d) as well as their Grad-CAM activation maps
before attack (b) & (e) and after attack (c) & (f).

Existing research, however, focuses on defending against adversarial patches,
and does not translate to defend against other physical attacks like the
camouflage art attack (i.e white and black sticker attack). For example, one
approach to defend against the adversarial patch attack is to first locate the
perturbed area using an attention-based or gradient-based model, and then
remove or diminish these areas [16, 26, 37]. The perturbations produced by
multi-sticker attacks like the camouflage art attack, however, cannot be
detected the same way due to several reasons. First, the black and white
stickers produced camouflage art attack are not highly textured, and hence are
unlikely to be detected via gradient-based methods. Second, the camouflage art
attack works in conjunction with the scene content to redirect the classifiers
decision instead of hijacking its attention like the adversarial patch does.
As a result, multi-sticker attacks are unlikely to be identified using
attention-base models.

An example of this phenomenon can be seen in Figure 1, which shows activation
maps produced by Grad-CAM [31] when presented with images before and after a
multi-sticker attack. When examining the activation maps of the pedestrian
crossing sign before an attack shown in Figure 1(b) and after the attack shown
in Figure 1(c), we can see that the attack has shifted the classifier’s
attention off of attacked sign. Defenses that operate by removing or altering
these regions will have no effect on the attack. Alternatively, from examining
the activation maps of an unattacked speed limit sign in Figure 1(e) and it’s
attacked counterpart in Figure 1(f), the classifier is paying attention to
nearly the entire sign. Defenses that operate by removing or distorting these
regions will degrade the image so severely that the classifier will be unable
to operate.

Furthermore, it is important for defenses against physical domain attacks to
be evaluated on real images of physically attacked objects. Digital
simulations of physical attacks are sometimes used for evaluation due to the
ease of creating a dataset, for example, digitally adding perturbations that
simulate a physical attack into an image. However, these digital simulations
do not capture many effects that occur during imaging, such as lighting
conditions, the curvature of surfaces, focus blur, sampling effects, etc. In
practice, phenomena such as these can impact how a camera captures physical
domain perturbations, and can potentially affect the success of defenses.
Defenses that are highly tuned to features of “pristine” digital simulations
of attacks may be less successful when confronted with real images of
physically attacked objects or scenes.

In this paper, we propose a new defense strategy that does not rely on
attention models to identify attacked image regions and can successfully
defend against multi-sticker attacks, like the camouflage art attack. Our
proposed defense operates by first creating defensive masks that can maximize
the likelihood of guessing the location of the perturbations, then mitigates
the effect of the perturbations through targeted modifications, and eventually
make a final decision based on defended images.

Our Contributions:

  * •

We propose a set of new defenses that can protect against multi-sticker
physical domain attacks such as the camouflage art attack by Ekyholt et al.
[11]. To the best of our knowledge, no existing defenses are designed to
defend against such attacks.

  * •

We present practical defenses that can be utilized depending on whether the
defender has full knowledge of the attack (non-blind), partial information
about the attack (semi-blind), or no information regarding the attack (blind).

  * •

We create a new database of front-facing photos of 90 physically attacked
signs using camouflage art attack and use this database to assess our defense.

  * •

We demonstrate that our proposed defenses outperform other state-of-the-art
defenses against physical attacks, such as the digital watermark defense [16],
when presented with multi-sticker attacks.

##  2 Additive Physical Domain Attacks

Adversarial attacks pose an important threat against deep neural networks [1,
4, 5, 6, 11, 14, 17, 18, 19, 21, 22, 24, 25, 27, 28, 33, 34]. Some physical
domain attacks, like the adversarial patch [4] and the camouflage art attack
[11], have shown that adding perceptible but localized patches to an object
can make a classifier identify it as a different object. We now briefly
describe how these two physical domain attacks are launched at a classifier
$C$ using attack target class $t^{\prime}$.

Adversarial patch: To generate an adversarial patch $A^{\prime}$, the authors
of [4] use an operator $O(I,A,\theta_{l},\theta_{t})$ to transform a given
patch $A$, then apply it to an image $I$ at location $\theta_{l}$. Similarly
to an Expectation over Transformation attack (EoT) [1], the adversarial patch
can be obtained by optimizing over sampled transformation and locations,

$A^{\prime}=\max_{A}\mathbb{E}_{I\sim\mathcal{I},\theta_{l}\sim\Theta_{L},\theta_{t}\sim\Theta_{T}}C(t^{\prime}|O(I,A,\theta_{l},\theta_{t}))$
(1)

where $\mathcal{I}$ denotes the training image dataset, $\Theta_{T}$ denotes
the distribution of transformation and $\Theta_{L}$ denotes the distribution
of the location. Once the patch is trained, it can universally attack any
object.

Camouflage art attack: Launching the camouflage art attack involves finding a
single set of perturbations $P$ that are capable of fooling a classifier under
different physical conditions. This attack, which produces perturbations for a
given pairing of source and target class, was demonstrated by using it to fool
a classifier trained to distinguish between different US traffic signs. Let
$H^{v}$ denote the distribution of the image of an object under both digital
and physical transformations, and $h_{i}$ denote each sample from this
distribution. The attack perturbations can be obtained via optimizing,

$\operatorname*{argmin}_{P}\lambda||M_{h},P||_{p}+\mathbb{E}_{h_{i}\sim
H^{v}}J(C(h_{i}+G(M_{h},P),t^{\prime})$ (2)

where $M_{h}$ is the mask that applies spatial constraints to the perturbation
(i.e ensures the perturbation is within the surface area of the object),
$\lambda$ is a hyper-parameter that regularize the distortion, $J(\cdot)$ is
the loss function that measures the difference between the classifier’s
prediction of the attacked object and the target class, $G(\cdot)$ is the
alignment function that maps transformations on the object to transformations
on the perturbation, $||\cdot||_{p}$ denotes $\ell_{p}$ norm.

##  3 Problem Formulation

We assume that the system under attack wishes to analyze some scene $S(x,y)$
containing an object to be classified. To do this, the system will capture a
digital image $I(x,y)$ of the scene, which will then be provided to a pre-
trained classifier $C(\cdot)$ which maps the image into one of $N$ classes
$t\in\mathcal{T}$. For the purposes of this work, we assume that if no
adversarial attack is launched, then the image provided to the classifier is
$I=S$.

An attacker, may attempt to fool the classifier by launching a physical domain
attack $\alpha(\cdot)$. This corresponds to physically modifying an object
within the scene by adding adversarial perturbations $P$ to it. Since these
perturbations must be physically added to the scene, we assume that they will
be spatially localized to one or more regions of the object under attack.
These regions can be specified by a spatial mask $M$, where $M(x,y)=1$
corresponds to a perturbation being present at spatial location $(x,y)$ and
$M(x,y)=0$ corresponds to no perturbation occurring at $(x,y)$. As a result,
we can express a physically attacked scene $\alpha(S)$

$\alpha(S(x,y))=(1-M(x,y))S(x,y)+M(x,y)P(x,y).$ (3)

In this paper, we assume that the adversarial perturbations will take the form
of black and white stickers added to an object as proposed by Eykholt et al.
[11], i.e. $P(x,y)=\\{black,white\\}$. Other physical domain attacks, such as
the adversarial patch [4] can still be modeled using (3) by allowing $P(x,y)$
to correspond to the full range of color values. Since the majority of the
defenses proposed in this paper do not rely on knowledge of the color values
of $P$, it is likely that these defenses can be used against other physical
domain attacks such as the adversarial patch. We note that this work only
addresses physical domain attacks that involve modifying an existing physical
object, and not attacks that involve the creation of a new physical object
such as synthesized 3D objects [1] and printed photos or posters [11, 19].

##  4 Knowledge Scenarios

To defend a classifier, we first assume that the defender has full access to
the classifier and implicitly knows the $N$ classes that it is trained to
distinguish between. We examine three scenarios corresponding to different
levels of knowledge available to the defender.

Non-blind: We assume that defender knows if an object is attacked or not, the
perturbation masks $M$ that indicates the perturbation areas and the
perturbations $P$. Therefore, locations of perturbations can be directly
located.

Semi-blind: We assume that the defender does not know if the object is
attacked or not. We also assume that if the object was attacked, the defender
does not know the perturbation masks $M$. However, the defender knows the
attack method $\alpha(\cdot)$. Therefore, for any source A and target B
pairing, the defender can obtain a perturbation mask $M_{\text{A, B}}$ via
launching the attack.

Blind: We assume that defender has zero knowledge. Specifically, the defender
does not know whether an object is attacked or not. We also assume that if the
object was attacked, the defender does not know the perturbation regions.
Additionally, the defender does not know the attack method.

##  5 Proposed Defenses

To defend against a physical domain attack, we propose a set of defenses based
on the amount of knowledge available to the defender. These defenses attempt
to interfere with or remove adversarial multi-sticker perturbations to
mitigate their effects. If the defender is able to leverage information about
the potential locations of these perturbations, defenses are guided to these
regions. Otherwise, our defenses are designed with the intuition that
adversarial perturbations are more fragile to distortions than the underlying
object that they are attacking.

Our defensive strategy is composed of three major steps. First, we obtain a
defensive mask $R$ or set of defensive masks $\mathcal{R}$ indicating
candidate areas to apply defenses. Second, we launch a local defense in
regions indicated by a defensive mask to produce a defended image $\delta$.
When our first step results in a set of defensive masks, local defenses can
either be sequentially applied in conjunction with each mask to produce a
single defended image, or they can be applied in parallel to produce a set of
defended images. In the third step, the defended image or images are provided
to the classifier. If a set of defended images are produced by the second
step, a fusion strategy is employed to produce a single classification
decision. In what follows, we discuss each step of our proposed defenses in
detail.

###  5.1 Defensive Mask Selection

The goal of each defensive mask is to ensure that defensive distortions are
only applied to small regions of the image, since each perturbation produced
by the multi-sticker attack is still confined to a small region. We do not
want to change the ground truth object. Let $R(x,y)\in\\{0,1\\}$ denote a
defensive mask, where 1 indicates the area need to be defended, 0 indicates
the area of the ground truth content. Now we discuss the acquisition of
defensive masks.

Oracle Approach: If the perturbation mask $M$ is known, such as in the non-
blind scenario, we simply let $R=M$.

Estimated Defensive Mask Sets: In semi-blind scenarios, the defender may know
the potential attack method $\alpha$, but not perturbation masks or the
potential attack mask if the attack was launched. They can, however, leverage
knowledge of $\alpha$ to create a set of estimated defensive masks.

To do this, first we assume that $I$ is an image of an attacked scene. The
attack’s target class $\hat{t}$ can be inferred by using $C$ to classify the
image such that $\hat{t}=C(I)$. Next, the defender can create their own
implementation of $\alpha$ and use it to recreate an attack aimed to move true
class $j$ to target class $\hat{t}$. The attack’s perturbation mask can then
be used as the estimated defensive mask $R_{j,\hat{t}}$ for source $j$ and
target $t$. This process can be repeated for all $j\in\mathcal{T}$ such that
$j\neq\hat{t}$ to produce the set of estimated masks
${\mathcal{R}_{\hat{t}}=\\{R_{1,\hat{t}},\ldots,R_{\hat{t}-1,\hat{t}},R_{\hat{t}+1,\hat{t}},\ldots,R_{N,\hat{t}}\\}}$.
To reduce computational costs while launching the defense, the set
$\mathcal{R}_{\hat{t}}$ can be precomputed for each target class. With
increasing number of classes, the computational cost may become high for
constructing sets of estimated set of defense masks and launching the defense.
To solve this problem, defender can use a subset of defensive masks instead of
every single mask. We propose two methods to form these subsets.

Ranked Selection: The defender can utilize class activations to guide the
selection of the subset of defensive masks to use. Since physical attacks
operate by constraining perturbations to small areas to avoid suspicion, it is
reasonable to assume these perturbation push the object just across the
boundary of its true class. Therefore, the true class of an attacked image
most likely shows up in the top few activated classes. To guide the selection
of defensive masks, first we assume that a scene is always under attack
(regardless of whether this is true or not) and treat the class with the
highest activation as the target class. The true class then lies among the
remaining classes, which are ranked according to their activation scores. The
subset of $k$ defensive masks is then chosen as the set of masks created using
the assumed target class (i.e. the class with the highest activation) and the
$k$ top candidates for the true source class (i.e. the classes with the second
highest through $k+1$ highest activations). By doing this, the defender can
control the computation cost of the defense while increasing the chance that
the most useful defensive masks are utilized.

Random Selection: A heuristic way to form a subset of defensive masks is
through random selection. Since each of the selected mask is related to the
target class, each selected mask can be used to defend a partial of the image.
By grouping several defensive masks, it may increase the chance for a
successful defense.

Randomly Chosen Regions: In blind scenarios, the defender cannot leverage any
prior information about the attack or possible perturbation locations. In
these situations, we create a set of defensive masks made by randomly choosing
defensive regions. Our intuition is that if we use many random defensive
masks, several of them will interfere with the adversarial perturbations. Each
mask is made by randomly selecting $m$ different $w\times w$ windows to apply
localized defenses. We use two different approaches for randomly choosing
these regions:

Overlapping: The locations of each window are chosen uniformly at random from
throughout the image area. As a result, some windows may overlap with one
another.

Non-overlapping: In this approach, we ensure that defensive regions are spread
throughout the region by disallowing overlaps. We do this by first dividing
the defensive mask into non-overlapping $w\times w$ blocks, then randomly
choosing $m$ of these blocks as defensive regions.

###  5.2 Local Defense Strategies

After the defensive masks are obtained, we can apply local defenses to image
regions specified by these masks. To make it clear, we first show how to
obtain the defended image using single defensive mask, then we adapt the
proposed defenses to accommodate multiple defensive masks.

Given one defensive mask, we propose two methods to defend against the attack.

Targeted perturbation remapping: This idea is to interfere the perturbations
instead of removing it. Specifically, we can using remapping functions to
destroy the spatial correlation between perturbed regions. Let $\phi(\cdot)$
be the remapping function, then a single defended image can be expressed as,

$\delta(x,y)=\begin{cases}I(x,y)&R(x,y)=0\\\ \phi(I(x,y))&R(x,y)=1\end{cases}$
(4)

In this work, we consider three mapping functions:

RemapW: Change pixels to white.

RemapB: Change pixels to black.

RemapT: Pick a threshold $\tau$, change pixels to black if the luminance value
is above the threshold and to white if below the threshold.

Localized region reconstruction: The idea is to diminish or remove the effects
of that perturbation by reconstructing perturbed local regions of input image
on the basis of other parts of the image. Since the perturbations are confined
to a small region, we can use the inpainting algorithm to reconstruct the
image.

The defenses discussed above can be easily adapted for multiple defensive
masks. Let $\psi(\cdot)$ denote the defense. For a set of defensive masks
$\mathcal{R}$ that comprises $k$ mask,
$\mathcal{R}=\\{R_{1},R_{2},...,R_{k}\\}$, we can either obtain one single
final defended image via sequential defense, or obtain a sequence of
individually defended images via parallel defense and then fuse the results.
Now we discuss sequential and parallel defense individually.

Sequential defense: We attempt to make the defense stronger by recursively
applying the defense and obtain a single defended image. For iteration $\ell$,
the defense $\psi(\cdot)$ is applied to the output of the previous step using
$\ell^{th}$ defensive mask,
$\psi_{\ell}(\cdot)=\psi(\delta_{\ell-1},R_{\ell})$. The final defended image
is obtained by sequential applying the defense using each of $k$ individual
defensive mask via,

$\delta=(\psi_{k}\circ\psi_{k-1}\circ\ldots\circ\psi_{1})(I)$ (5)

Parallel defense: The idea is to generate many copies of defended image with
each copy being able to defend one part of input image. Using $\ell^{th}$
defensive mask, we define $\ell^{t}h$ defended image as
$\delta_{\ell}=\psi(I,R_{\ell})$, then using $k$ defensive masks we get $k$
individual defended images, $\\{\delta_{1},\delta_{2},\ldots,\delta_{k}\\}$.

###  5.3 Defensive Classification

After applying local defenses, we need to use the classifier to make a final
decision on the defended image or images. We propose two decision making
strategies.

Single defended image: After the sequential defense, the defender will obtain
a single defended image. We simple use the classifier to classify the defended
image, $t=C(\delta)$.

Multiple defended images: The parallel defense will result in a sequence of
defended images. The defender can use the classifier to get a fused decision
by combining the decisions of the individually defended images. We propose two
fusion strategies.

Majority vote (MV): Use the classifier to make a decision with each individual
defended image, $t_{\ell}=C(\delta_{\ell})$, then take a majority votes of all
decisions

$t=\operatorname*{argmax}_{n\in
N}\sum_{\ell=1}^{k}\mathbbm{1}(C(\delta_{\ell})=t_{n})$ (6)

where $\mathbbm{1}$ is the indicator function.

Softmax fusion (SF): Let $\mathbf{v^{(\ell)}}$ denote the softmax output of
the classifier for the $\ell^{th}$ defended image,
$\mathbf{v}^{(\ell)}=C_{softmax}(\delta_{\ell})$, next add the softmax output
of each of the $k$ defended images to form a single vector $\mathbf{v}$,

$\mathbf{v}=\sum_{\ell=1}^{k}\mathbf{v^{(\ell)}}$ (7)

then take the class corresponding to the largest value in $\mathbf{v}$ as the
final decision,

$t=\operatorname*{argmax}_{n\in N}v_{n}$ (8)

where $v_{n}$ is the $n^{th}$ element in the vector $\mathbf{v}$.

##  6 Evaluation Metrics

When formulating our evaluation metrics, we let $t^{*}$ denote the ground
truth class of a scene. Additionally, we let $\pi_{A}$ denote the a priori
probability that an attack is launched against a scene.

Classifier: To evaluate the baseline performance of the classifier $C(\cdot)$,
we calculate the classification accuracy as the probability that the image of
a scene being correctly classified as its ground true class,

$\text{CA}=Pr(C(I)=t^{*}|I=S)$ (9)

Attack: To evaluate the baseline performance of the attack, we calculate the
targeted attack success rate (T-ASR) and the untargeted attack success
rate(U-ASR).

T-ASR is defined as the probability that the image of an attacked scene is
classified as the target class,

$\text{T-ASR}=Pr(C(I))=t^{\prime}|I=\alpha(S))$ (10)

U-ASR is defined as the probability that the image of an attacked scene is
classified as any other class than the true class,

$\text{U-ASR}=Pr(C(I))\neq t^{*}|I=\alpha(S))$ (11)

Defense: To evaluate the performance of our proposed defenses, we calculate
the Defense Rate (DR) for an attacked scene, the Classification Drop (CD) for
an unattacked scene, and the Post-Defense Accuracy (PDA) for any scene.

DR is defined as the probability that the defended image of a scene is
classified as true class, given it is an attacked scene and its image was not
classified as the true class before the defense,

$\text{DR}=Pr(C(D(I))=t^{*}|I=\alpha(S),C(I)\neq t^{*})$ (12)

CD is defined as the probability that the image of an unattacked scene get
misclassified after applying the defense.

$\text{CD}=\text{CA}-Pr(C(D(I))=t^{*}|I=S)$ (13)

PDA is defined as the probability that the image of any scene is correctly
classified as the true class after the defense,

PDA $\displaystyle=(1-\pi_{A})Pr(C(D(I))=t^{*}|I=S)$
$\displaystyle+\pi_{A}Pr(C(D(I))=t^{*}|I=\alpha(S))$ (14)

When U-ASR=1, using equation 11, 12 and 13, equation 14 can be expressed as,

$\text{PDA}=(1-\pi_{A})(\text{CA}-\text{CD})+\pi_{A}\text{DR}$ (15)

##  7 Experimental Results

To evaluate the performance of our proposed defenses, we conducted a series of
experiments. The physical attack we attempt to defend against is the
camouflage art attack proposed by Eykholt et. al [11]. The classifier we used
to evaluate the the proposed defense was trained to differentiate 17 common US
traffic signs using LISA traffic sign database [23] (a US traffic sign
database). The classifier was reported to achieve 91% classification accuracy
in their paper. We started by making a dataset composed of photos of
unattacked ground truth source signs and physical attacked signs. Then we
demonstrated the effectiveness of the proposed defense method under the three
scenarios we discussed in Section 4. We assume $\pi_{A}=0.5$ in all scenarios.

###  7.1 Dataset

To the best of our knowledge, there exists no database that made specifically
for physical attack, especially using camouflage art attack. A physical attack
database should be constructed with the photos of the physically attacked
objects. This is because empirically we found that defenses against physical
perturbations are very different from the digital simulation. One reason is
that the many effects introduced during capturing images of physically
attacked objects, such as the curvature of surfaces, focus blur, sampling
effects, sensor noise, will result in significant discrepancies between
physical perturbations and digital approximation. Therefore, it is important
to create a new database to fill this gap and benefit future research in the
community.

To make the database, we first purchased six US road signs which were included
among the 16 which classes the LISA-CNN is trained to distinguish between.
These six signs are indicated above in Table 1 as ‘source’ signs.

To create training data for the attack, and assess the baseline performance of
the LISA-CNN, we first captured a set of images of the six unattacked signs in
our possession. This was done by photographing each sign at angles running
from $-50$ to $+50$ degrees in an increments of $10$ degrees, to create a set
of $66$ images of unattacked signs.

Next, we launched a series of multi-sticker attacks against the six signs in
our possession, using each of the 15 remaining classes listed in Table 1 as
the attack’s target. This was done by following the attack protocol described
in [11]. For each pair of source and target signs, we first created a digital
copy of the attacked sign. This digital copy was projected onto the
corresponding physical copy of the source sign, then black and white stickers
were placed on the sign in regions indicated by the digitally attacked
version. Front facing images of all of the attacked signs were captured, then
cropped to approximately $340\times 340$ pixels and saved as PNG files. This
resulted in a set of 90 images of physically attacked signs, each with a
different source-target class pairing. The database is publicly available at
https://drive.google.com/drive/folders/1qOmSubSOVY8JzB3KfXhDQ38ihoY5GExK?usp=sharing.

###  7.2 Baseline Evaluation of the Classifier and Attack

To assess the baseline classification accuracy of the LISA-CNN classifier
trained by Eykholt et al., we evaluated its performance on the unattacked
signs captured as part of our database. In this evaluation, the LISA-CNN
achieved $100\%$ classification accuracy. We note that Eykholt et al. reported
a $91\%$ classification accuracy during their evaluation of this trained
classifier. In this paper, when reporting metrics that depend on
classification accuracy, we use the value that we obtained since this
classification accuracy is measured on the same set of road signs in the
attack set. Furthermore, this corresponds to more challenging test conditions
for our defense, since perfect performance would need to bring the defense
rate equal to this higher classification accuracy.

Next, we measured the baseline performance of the attack by using the LISA-CNN
to classify the images of physically attacked signs in our database. Our
implementation of the camouflage art attack achieved a $0.9556$ targeted
attack success rate (T-ASR) and a $1.0000$ untargeted attack success rate
(U-ASR). This result verifies that we were able to reproduce the attack, and
that this attack can successfully fool the classifier.

Table 1:  Source and Target Traffic signs. S denotes “source” and T denotes
“target”.

Category | Sign Name | Category | Sign Name
---|---|---|---
S & T | crossing | T | added lane
S & T | stop | T | keep right
S & T | yield | T | lane ends
S & T | signal ahead | T | stop ahead
S & T | speed limit 25 | T | turn right
S & T | speed limit 45 | T | school / limit 25
T | merge | T | speed limit 30
T | school | T | speed limit 35

Table 2: Non-blind evaluation of our proposed defenses.

Proposed defense | DR | CD | PDA
---|---|---|---
RemapW | 0.4339 | 0.0000 | 0.7170
RemapB | 0.4556 | 0.0000 | 0.7283
RemapT | 0.9222 | 0.0000 | 0.9611
Reconst | 0.6778 | 0.0000 | 0.8389

###  7.3 Non-Blind

In our first set of experiments, we evaluated our defenses’ performance in the
non-blind scenario. We used the digital versions of the perturbation masks
obtained while training the attack as the oracle defensive masks known to the
defender. While these digital masks are not perfect ground truth locations of
the actual perturbations they are sufficiently close to evaluate our
experiment.

Using these oracle masks, we evaluated the three perturbation remapping
defenses remap to white (RemapB), black (RemapW), and threshold (RemapT) as
well as the targeted region reconstruction (Reconst) defense. We note that the
classification drop is always zero in this experiment because the defender
always knows if an attack is present and can choose when not to apply the
defense.

Table 2 shows the performance of our defenses in the non-blind scenario.
Thresholded perturbation achieved strongest performance with the highest
defense rate of 0.9222 and post-defense accuracy of 0.9611. Since both the
remap-to-white and remap-to-black strategies will only affect approximately
half of the stickers added to an object, it is reasonable to expect that the
thresholded perturbation remapping approach outperforms these approaches.
Reconstruction approach achieved second highest performance. We believe that
lower defense rate is predominantly due to the slight misalignment between the
ideal digital perturbation masks and the true locations of the physical
perturbations in the attacked images.

Table 3: Evaluation of proposed defenses in semi-blind scenario

Defense Strategies | DR | CD | PDA | Defense Strategies | DR | CD | PDA
---|---|---|---|---|---|---|---
RemapW-Par(6) + MV | 0.3989 | 0.3333 | 0.5328 | RemapW-Par(6) + SF | 0.4186 | 0.1667 | 0.6260
RemapB-Par(6) + MV | 0.0794 | 0.1667 | 0.4563 | RemapB-Par(6) + SF | 0.0690 | 0.0000 | 0.5348
RemapT-Par(6) + MV | 0.5174 | 0.3333 | 0.5921 | RemapT-Par(6) + SF | 0.6453 | 0.1667 | 0.7393
Reconst-Par(6) + MV | 0.3560 | 0.0000 | 0.6780 | Reconst-Par(6) + SF | 0.3514 | 0.0000 | 0.6757
Reconst-Seq-Rand(1) | 0.2815 | 0.0556 | 0.6130 | Reconst-Seq-Rand4) | 0.6200 | 0.1112 | 0.7544
Reconst-Seq-Rand(2) | 0.4237 | 0.0556 | 0.6840 | Reconst-Seq-Rand(5) | 0.6648 | 0.1389 | 0.7630
Reconst-Seq-Rand(3) | 0.5350 | 0.0834 | 0.7250 | Reconst-Seq(6) | 0.7000 | 0.1667 | 0.7667
Reconst-Seq-Rank(1) | 0.3780 | 0.0000 | 0.6890 | Reconst-Seq-Gtd(1) | 0.6778 | 0.0000 | 0.8389
Reconst-Seq-Rank(2) | 0.6336 | 0.0000 | 0.8168 | Reconst-Seq-Gtd(2) | 0.6623 | 0.0333 | 0.8145
Reconst-Seq-Rank(3) | 0.7001 | 0.0000 | 0.8501 | Reconst-Seq-Gtd(3) | 0.6855 | 0.0667 | 0.8094
Reconst-Seq-Rank(4) | 0.6667 | 0.0000 | 0.8333 | Reconst-Seq-Gtd(4) | 0.7022 | 0.1000 | 0.8011
Reconst-Seq-Rank(5) | 0.7000 | 0.0000 | 0.8500 | Reconst-Seq-Gtd(5) | 0.7044 | 0.1333 | 0.7856
Other Methods | DR | CD | PDA | Other Methods | DR | CD | PDA
DW [16] | 0.2222 | 0.0000 | 0.6111 | Median Filter (kernel=7) [36] | 0.3777 | 0.3333 | 0.5222
JPEG (QF=10) [10] | 0.1333 | 0.0000 | 0.5667 | Local Smooth [26] | 0.0000 | 0.0000 | 0.5000

###  7.4 Semi-blind

To evaluate our defenses in the semi-blind scenario, we created a set of
estimated defensive masks for each of the 15 possible target classes. Each set
of defensive masks contained six pairings of source and target sign, i.e. one
for each source sign in our database that an attack could be launched against.

Next, we used these sets of defensive masks to evaluate our relevant defensive
strategies. The results of these experiments are shown in Table 3. We adopt
the notation Par and Seq to denote that a defense was applied either in
parallel or sequentially, and ($k$) to denote the number of defensive masks
used for defense. When defenses were applied in parallel, we use the notation
MV to denote majority vote fusion and SF to denote softmax fusion. We use Rand
and Rank to denote the random or ranked mask selection strategy. Additionally,
we use Gtd to denote a special “Guaranteed scenario” in which the defensive
mask with the correct source-target pair was always included and the remaining
masks were randomly chosen.

Results in Table 3 show that for any mask selection strategy, sequential
reconstruction outperforms both parallel reconstruction and perturbation
remapping. The defense using three defensive masks selected using ranked
activation (Reconst-Seq-Rank(3)) outperforms all other strategies and achieved
the highest defense rate of 0.7001, highest post-defense accuracy of 0.8501,
and zero classification drop on unattacked images. We note that Reconst-Seq-
Rank(3) is statistically the same performance as Reconst-Rank(5), but it is
more computationally efficient using less masks.

Comparisons of Defensive Mask Selection Strategies: For all values of $k$, the
ranked selection strategy achieved a higher defense rate and post-defense
accuracy than the random selection strategy. This shows that using a well
chosen subset of defensive masks improves our system’s performance.
Additionally, it reinforces our observation that important information about
the true source class and attack target class can be observed in the top few
class activations.

To explore impacts from the correct source and target defensive mask, we ran
another set of experiments for the “Guaranteed scenario”. Compared to the
Reconst-Seq-Rand($k$) strategy, the Reconst-Seq-Gtd($k$) strategy always
achieved higher defense rate, post-defense accuracy, and lower classification
drop for the same $k$. These results imply that the inclusion of the estimated
mask for the correct source-target class pair can significantly improve the
performance of defenses.

Comparing the Ranked strategy with the “Guaranteed scenario”, Ranked results
are in a higher post-defense accuracy for $k\geq 2$. The main reason is that
Ranked produces a significantly lower classification drop. These results
suggest that using the ranked selection strategy not only can pick out the
“best” subset of defensive masks to use, but can also exclude those that
deteriorate the classification accuracy of unattacked images.

This is reinforced by examining the classification drop as $k$ increases. For
both Reconst-Seq-Gtd($k$) and Reconst-Seq-Rand($k$), the classification drop
increases as $k$ increases, thus hurting the overall post-defense accuracy.
This is likely because some masks that negatively effect the overall
performance are included. By contrast, Reconst-Seq-Rank($k$) does not suffer
from the same decrease in classification drop because unlikely defensive masks
that may hurt performance are excluded.

Comparisons with Related Defenses: We compared the performance of our proposed
defenses with several existing defenses against physical domain attacks. These
include distortions that are universally applied to an image such as JPEG
compression and median filtering, as well as the more sophisticated digital
watermarking (DW) defensive method and the local smooth approach. While we
evaluated the performance of the JPEG defense using multiple quality factors
and the median filtering defense using multiple kernel sizes, we report only
the strongest results in the interest of space.

The results in Table 3 show that all of our proposed strategies with the
reconstruction defense can significantly outperform each of these existing
defenses. The digital watermarking defense proved to be the strongest
performing existing defense, with a defense rate of 0.2222 and a post-defense
accuracy of 0.6111. However, even when only one randomly chosen estimated
defensive mask is used, our region reconstruction defense outperforms this
approach. Our best performance achieved more than three times higher in
defense rate and about 40% more in post-defense accuracy than this approach.
The relatively poor performance of these existing defenses likely occurs
because they are targeted to defend against the adversarial patch attack.
Since the multi-sticker camouflage art attack works in a different manner and
exhibits different visual properties, these defenses are not as well suited to
protect against this and similar attacks.

###  7.5 Blind

To evaluate our defenses in the blind scenario, we created randomly chosen
defensive masks using both the overlapping (OL) and non-overlapping (NOL)
strategies.

Table 5 shows the results of these experiments. In each experiment, we
identified the optimal window size and number of windows for use in these
masks through a grid search. We chose window size $w$ vary from 2, 4, 8 16
pixels. Next we controlled the number of windows $m$ by randomly selecting a
ratio of total number of windows based on the given window sizes. The results
reported in Table 5 correspond to the pairing of $w$ and ratio that achieved
the highest post-defense accuracy. A detailed examination of the choice of $w$
and ratio is provided later in this section.

From Table 5, we can see the strongest performance in terms of all evaluation
metrics was achieved using targeted region reconstruction applied in parallel
using 100 random masks with non-overlapping windows in conjunction with
majority vote decision fusion (NOL-Reconst-Par(100) + SF). Even though no
information regarding the attack could be leveraged, this defense was still
able to achieve a defense rate of 0.4102 with a corresponding classification
drop of 0.0017 and a post-defense accuracy of 0.7043. Though performance is
worse than in the semi-blind scenario, we are still able to outperform
existing defenses in all evaluation metrics. We note that the local region
reconstruction defense uniformly outperformed the targeted perturbation
remapping strategy, and for targeted region reconstruction, applying the
defense in parallel outperformed sequential application of the defense.

Creating defensive masks using the non-overlapping strategy significantly
improves our defense’s performance over the overlapping approach (i.e.
choosing window locations uniformly at random). Furthermore, we note that
performance increases as the number of randomly chosen masks increases. While
this comes at the price of additional computation costs, in practice we found
that our proposed defense takes 0.4 seconds on average using 100 masks without
any attempt at execution time optimization.

Effect of Size and Number of Windows: To understand the effect that the window
size and number of windows (or ratio) in each randomly chosen defensive mask
has on our defense, we provide detailed results of our search over these
parameters in Table 5. The symbol $\ast$ means when window size was 16 and
ratio was 0.625, the computed number of windows was not an integer. However,
it equals to when ratio was 0.5 if rounded down, and equals to when ratio was
0.75 when rounded up.

The results show that the defense rate increases as the ratio (i.e the number
of windows) increases. After a certain point, the classification drop also
increases, resulting in a negative effect on the post-defense accuracy. We
also find that increasing the window size increases the defense rate up to a
certain point, after which the defense rate begins to decrease. Additionally,
after a certain point, increasing the window size also leads to an increase in
the classification drop and a decrease in the post-defense accuracy. In our
experiments, we found that the optimal window size was 8 pixels and ratio was
0.625. More importantly, when choosing the window size and the ratio (i.e the
number of windows), the defender must balance the trade-off between
interfering with the attack and interfering with the unattacked scene content
used by the classifier.

Comparisons with Related Defenses: From Table 5, we can see that applying
region reconstruction in parallel using non-overlapping masks outperforms the
existing defenses that were also considered in the semi-blind scenario (the
performance of these defenses do not change in the blind scenario). This
result holds true even when only six randomly generated non-overlapping masks
are used.

Table 4: Defense performance in the blind-scenario.

Defense Strategies | DR | CD | PDA | Defense Strategies | DR | CD | PDA
---|---|---|---|---|---|---|---
OL-RemapT-Par(6) + MV | 0.1210 | 0.2367 | 0.4422 | NOL-RemapT-Par(6) + MV | 0.1236 | 0.1283 | 0.4976
OL-RemapT-Par(6) + SF | 0.1271 | 0.1650 | 0.4811 | NOL-RemapT-Par(6) + SF | 0.1255 | 0.1267 | 0.4994
OL-RemapT-Par(100) + MV | 0.0713 | 0.0333 | 0.5190 | NOL-RemapT-Par(100) + MV | 0.0482 | 0.0500 | 0.4991
OL-RemapT-Parallel(100) + SF | 0.0778 | 0.0333 | 0.5222 | NOL-RemapT-Par(100) + SF | 0.0737 | 0.0667 | 0.5035
OL-Reconst-Seq(6) | 0.2352 | 0.1782 | 0.5286 | NOL-Reconst-Seq(6) | 0.1942 | 0.0775 | 0.5584
OL-Reconst-Seq(100) | 0.1588 | 0.8450 | 0.1569 | NOL-Reconst-Seq(100) | 0.2553 | 0.6300 | 0.3027
OL-Reconst-Par(6) + MV | 0.1836 | 0.0717 | 0.5560 | NOL-Reconst-Par(6) + MV | 0.3444 | 0.0467 | 0.6488
OL-Reconst-Par(6) + SF | 0.1762 | 0.0350 | 0.5706 | NOL-Reconst-Par(6) + SF | 0.3501 | 0.0317 | 0.6593
OL-Reconst-Par(100) + MV | 0.1362 | 0.0000 | 0.5681 | NOL-Reconst-Par(100) + MV | 0.4129 | 0.0067 | 0.7031
OL-Reconst-Par(100) + SF | 0.2415 | 0.0167 | 0.6124 | NOL-Reconst-Par(100) + SF | 0.4102 | 0.0017 | 0.7043
Other Methods | DR | CD | PDA | Other Methods | DR | CD | PDA
DW [16] | 0.2222 | 0.0000 | 0.6111 | Median Filter (kernel=7) [36] | 0.3777 | 0.3333 | 0.5222
JPEG (QF=10) [10] | 0.1333 | 0.0000 | 0.5667 | Local Smooth [26] | 0.0000 | 0.0000 | 0.5000

Table 5: Local region reconstruction using 100 parallel masks with non-
overlapping windows and softmax fusion. * For window size 16, ratio 0.625
results in a non-integer number of windows.

| Ratio $=0.25$ | Ratio $=0.5$ | Ratio $=0.625$ | Ratio $=0.75$
---|---|---|---|---
| DR  CD  PDA | DR  CD  PDA | DR  CD  PDA | DR  CD  PDA
$w=2$ | 0.0546  0.0000  0.5273 | 0.1528  0.0000  0.5764 | 0.2027  0.0033  0.5997 | 0.2770  0.1283  0.5744
$w=4$ | 0.0537  0.0000  0.5269 | 0.1818  0.0000  0.5909 | 0.2395  0.0000  0.6198 | 0.3153  0.0233  0.6460
$w=8$ | 0.1648  0.0000  0.5824 | 0.3268  0.0000  0.6634 | 0.4102  0.0017  0.7043 | 0.4701  0.2500  0.6101
$w=16$ | 0.0183  0.0000  0.5092 | 0.1073  0.1400  0.4837 | $\ast$ | 0.1353  0.3333  0.401

##  8 Conclusions

In this paper, we proposed new defense strategies against physical domain
attacks with a special focus on the multi-sticker attacks, like camouflage art
attack. Our proposed methods attempt to maximize the likelihood of diminishing
the effect of the physical perturbation, given the defender’s different levels
of knowledge. We conducted an extensive amount of experiments to show that our
proposed defense can successfully defend against the camouflage art attack
under many scenarios with small classification drops on unattacked objects.
Additionally, we built a new database using the camouflage art attack that
contains photos of 90 physical attacked traffic signs and six source signs.
This database may benefit future research in the community.

## References

  * [1] Athalye, A., Engstrom, L., Ilyas, A., Kwok, K.: Synthesizing robust adversarial examples. arXiv preprint arXiv:1707.07397 (2017)
  * [2] Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., Criminisi, A.: Measuring neural net robustness with constraints. In: Advances in neural information processing systems. pp. 2613–2621 (2016)
  * [3] Bhagoji, A.N., Cullina, D., Mittal, P.: Dimensionality reduction as a defense against evasion attacks on machine learning classifiers. arXiv preprint arXiv:1704.02654 2 (2017)
  * [4] Brown, T.B., Mane, D., Roy, A., Abadi, M., Gilmer, J.: Adversarial patch (2017)
  * [5] Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 ieee symposium on security and privacy (sp). pp. 39–57. IEEE (2017)
  * [6] Chen, P.Y., Sharma, Y., Zhang, H., Yi, J., Hsieh, C.J.: Ead: elastic-net attacks to deep neural networks via adversarial examples. In: Thirty-second AAAI conference on artificial intelligence (2018)
  * [7] Chiang, P.Y., Ni, R., Abdelkader, A., Zhu, C., Studor, C., Goldstein, T.: Certified defenses for adversarial patches. arXiv preprint arXiv:2003.06693 (2020)
  * [8] Das, N., Shanbhogue, M., Chen, S.T., Hohman, F., Li, S., Chen, L., Kounavis, M.E., Chau, D.H.: Shield: Fast, practical defense and vaccination for deep learning using jpeg compression. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 196–204 (2018)
  * [9] Dhillon, G.S., Azizzadenesheli, K., Lipton, Z.C., Bernstein, J., Kossaifi, J., Khanna, A., Anandkumar, A.: Stochastic activation pruning for robust adversarial defense. arXiv preprint arXiv:1803.01442 (2018)
  * [10] Dziugaite, G.K., Ghahramani, Z., Roy, D.M.: A study of the effect of jpg compression on adversarial images. arXiv preprint arXiv:1608.00853 (2016)
  * [11] Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., Song, D.: Robust physical-world attacks on deep learning visual classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1625–1634 (2018)
  * [12] Feinman, R., Curtin, R.R., Shintre, S., Gardner, A.B.: Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410 (2017)
  * [13] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3354–3361. IEEE (2012)
  * [14] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
  * [15] Guo, C., Rana, M., Cisse, M., Van Der Maaten, L.: Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00117 (2017)
  * [16] Hayes, J.: On visible adversarial perturbations & digital watermarking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 1597–1604 (2018)
  * [17] Karmon, D., Zoran, D., Goldberg, Y.: Lavan: Localized and visible adversarial noise. arXiv preprint arXiv:1801.02608 (2018)
  * [18] Kos, J., Fischer, I., Song, D.: Adversarial examples for generative models. In: 2018 IEEE Security and Privacy Workshops (SPW). pp. 36–42. IEEE (2018)
  * [19] Kurakin, A., Goodfellow, I., Bengio, S.: Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 (2016)
  * [20] Levine, A., Feizi, S.: (de) randomized smoothing for certifiable defense against patch attacks. arXiv preprint arXiv:2002.10733 (2020)
  * [21] Liu, Y., Chen, X., Liu, C., Song, D.: Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:1611.02770 (2016)
  * [22] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)
  * [23] Mogelmose, A., Trivedi, M.M., Moeslund, T.B.: Vision-based traffic sign detection and analysis for intelligent driver assistance systems: Perspectives and survey. IEEE Transactions on Intelligent Transportation Systems 13(4), 1484–1497 (2012)
  * [24] Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1765–1773 (2017)
  * [25] Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: Deepfool: A simple and accurate method to fool deep neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (June 2016)
  * [26] Naseer, M., Khan, S., Porikli, F.: Local gradients smoothing: Defense against localized adversarial attacks. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1300–1307. IEEE (2019)
  * [27] Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 427–436 (2015)
  * [28] Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: 2016 IEEE European Symposium on Security and Privacy (EuroS&P). pp. 372–387. IEEE (2016)
  * [29] Papernot, N., McDaniel, P., Wu, X., Jha, S., Swami, A.: Distillation as a defense to adversarial perturbations against deep neural networks. In: 2016 IEEE Symposium on Security and Privacy (SP). pp. 582–597. IEEE (2016)
  * [30] Raghunathan, A., Steinhardt, J., Liang, P.: Certified defenses against adversarial examples. arXiv preprint arXiv:1801.09344 (2018)
  * [31] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: The IEEE International Conference on Computer Vision (ICCV) (Oct 2017)
  * [32] Shaham, U., Garritano, J., Yamada, Y., Weinberger, E., Cloninger, A., Cheng, X., Stanton, K., Kluger, Y.: Defending against adversarial images using basis functions transformations. arXiv preprint arXiv:1803.10840 (2018)
  * [33] Su, J., Vargas, D.V., Sakurai, K.: One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation 23(5), 828–841 (2019)
  * [34] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks (2013)
  * [35] Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M., Dolan, J., Duggins, D., Galatali, T., Geyer, C., et al.: Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics 25(8), 425–466 (2008)
  * [36] Xu, W., Evans, D., Qi, Y.: Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01155 (2017)
  * [37] Xu, Z., Yu, F., Chen, X.: Lance: A comprehensive and lightweight cnn defense methodology against physical adversarial attacks on embedded multimedia applications (2019)
  * [38] Zhang, F., Leitner, J., Milford, M., Upcroft, B., Corke, P.: Towards vision-based deep reinforcement learning for robotic motion control. arXiv preprint arXiv:1511.03791 (2015)