# Interactive Simulations of Backdoors in Neural Networks

Peter Bajcsy
National Institute of Standards and Technology
Gaithersburg, MD 20899
<EMAIL_ADDRESS>
Maxime Bros
National Institute of Standards and Technology
Gaithersburg, MD 20899
<EMAIL_ADDRESS>

###### Abstract

This work addresses the problem of planting and defending cryptographic-based
backdoors in artificial intelligence (AI) models. The motivation comes from
our lack of understanding and the implications of using cryptographic
techniques for planting undetectable backdoors under theoretical assumptions
in the large AI model systems deployed in practice. Our approach is based on
designing a web-based simulation playground that enables planting, activating,
and defending cryptographic backdoors in neural networks (NN). Simulations of
planting and activating backdoors are enabled for two scenarios: in the
extension of NN model architecture to support digital signature verification
and in the modified architectural block for non-linear operators. Simulations
of backdoor defense against backdoors are available based on proximity
analysis and provide a playground for a game of planting and defending against
backdoors. The simulations are available at https://pages.nist.gov/nn-
calculator

##  1 Introduction

This work presents interactive simulations of planting, activating, and
defending backdoors in fully-connected neural networks (NN). Backdoors are
adversarial attacks on artificial intelligence (AI) systems that are planted
by assuming control over training data, test data, or source code [1], [2],
[3]. The attacker’s goal with backdoor poisoning is to attack the integrity of
AI models, which makes the AI model untrustworthy [1]. The backdoor attack is
successful when input into a poisoned AI model results in a wrong prediction
with respect to the clean AI model prediction.

Cryptographic methods can be used for integrity attestation of training data,
test data, and AI models during different phases of an AI-model life cycle
[4], especially in life-critical applications such as medicine [5]. The
motivation for our work comes from the implications of using cryptographic
techniques for planting and activating undetectable backdoors [6]. There is a
lack of understanding of how theoretical assumptions behind cryptographically
undetectable backdoors [6] [7] scale to the AI models used in practice.
Furthermore, there is a lack of understanding of how AI model robustness
methods [8] [9] [10] can defend AI model predictions in the presence of
undetectable cryptographic backdoors. Backdoor and data poisoning simulations
aim to test backdoor propagation hypotheses, validate backdoor robustness
methods, and educate scientists using AI models. The ability to operate with
neural networks and two-dimensional (2D) dot patterns is the same as with
numbers in math calculators, which provides educational and research benefits
to security scientists via explorations of NN and input data configurations.
Simulations of backdoor and data poisoning are valuable for the AI security
community preparing and solving Trojan detection challenges, for example, the
Intelligence Advanced Research Projects Activity (IARPA)-sponsored TrojAI
challenge [11] or the Trojan detection challenge of the NeurIPS conference
[12].

The problem of interest is a class of cryptographic architectural backdoors in
AI non-linear activation functions. The architectural backdoors have been
introduced by Bober-Irizar et al. [3] and expanded by Langford et al. [2]. In
a backdoored AI model, a malicious attacker can make imperceptible changes to
the provided inputs or create a priori known input and trigger the backdoor
behavior at the attacker’s will. The inputs are formed by knowing a secret key
that flips the sign of a numerical output from the backdoored first layer as
outlined for undetectable backdoors by Goldwasser et al. [6]. However, the
output propagation from the backdoored first layer might not flip the AI model
prediction with the increasing number of layers and with the final layer
applying robustness methods [9] against adversarial attacks. This interplay
between poisoned AI models via datasets (Trojaned AI models) and Trojan
detectors was studied by Sahabandu et al. [13], Wu et al. [14], and Carlini
and Wagner [15] but has not been fully explored for AI models with backdoors.

The challenges of designing an interactive simulation of cryptographic
backdoors lie in the many complexities of current AI models, which demand
significant computational resources and prevent the simulations from being
interactive. Furthermore, the challenges are in various cryptographic
techniques and non-trivial solutions to creating inputs to trigger the
cryptographic backdoors. Another challenge is the lack of adherence to a
common terminology [1] in scientific literature. For example, it is hard to
determine based on the “backdoor” keyword of publications and GitHub
repositories [16], [17], [18] whether methods are referring to data poisoning
or source code poisoning according to the adversarial attack taxonomy [1].

Our approach to this problem is based on designing a web-based simulation
playground that enables planting, activating, and defending cryptographic
backdoors with the overarching workflow illustrated in Figure 1. The web-based
simulation playground is an extension of the neural network calculator [19]
and the neural network playground [20]. The simulations are constrained to
multiple 2D input data, ten features extracted from inputs, 64 fully-connected
nodes (eight nodes per layer, eight layers), and two predicted classification
labels visually labeled as blue and orange - see Figure 1. Among the slew of
cryptographic methods for ensuring data integrity, we chose a simple checksum
implementation out of many checksum methods [21]. This method follows black-
and white-box AI models for planting undetectable backdoors [6]. Finally, “AI
backdoors” are implemented by controlling the source code [1]. To minimize the
source code modifications of an AI model, our simulations use the NN model
non-linear activation functions for planting a backdoor since the
modifications do not change the model architecture (graph nodes and edges) and
only insert a few lines of code with the checksum computation function call
and one if statement. The feasibility of planting backdoors is explored in the
Discussion section.

Figure 1: An overview of planting and activating a backdoor in a trained
neural network. The simulation playground enables training a two-class fully-
connected neural network (NN) with input features derived from 2D points,
planting a checksum-based backdoor in the NN model, and activating the
backdoor based on the knowledge of a secret key.

Previous work on hiding AI model architectural backdoors is overviewed by
Bober-Irizar et al. [3] and Langford et al. [2]. The approaches include hiding
backdoors in quantization or augmentation software routines [22] or modifying
the loss function of AI model optimization [23]. Furthermore, there have been
reports of attacks on the PyTorch library in 2022 [24], exploited
vulnerabilities in the continuous integration and continuous development
scripts for the PyTorch packages in 2024 [25], and re-registered AI models in
the Huggingface model repository in 2023 [26]. Our work is related to the
previous work by planting backdoors via AI model modifications while focusing
on the non-linear activation functions of AI nodes and cryptographic types of
backdoors. Our work is also inspired by the backdoors that are provably
undetectable [6], the adversarial examples that are robust to real-world
perturbations [7], and the models that are resistant to adversarial attacks
[9].

In a black-box AI model scenario, we use the digital signature verification
from [6] and the non-linear operators (a.k.a. activation functions) of each
node in an AI model for planting architectural backdoors. In a white-box AI
model scenario, we explore hypotheses about backdoor injections that could
avoid a simple backdoor detection via source code inspection, for instance, by
training the AI model with backdoors, saving the trained model weights, and
re-loading the clean AI model with poisoned weights.

Our contributions are in

  1. 1.

designing an interactive web-based simulation framework for exploring
checksum-based backdoors in fully-connected, small-scale, NN models with a
variety of input features,

  2. 2.

deriving input triggers that activate the backdoors planted in non-linear
activation functions of the first layer of an NN model and

  3. 3.

enabling simulations of the game between adversaries planting backdoors and
users defending against the backdoors in small-scale NN models.

##  2 Methods

In our work, an NN architectural backdoor is a hidden unauthorized
functionality that can be activated by using a specific input (trigger)
following the definitions of architectural software backdoors and data
triggers [2] (Section 3.1). This section describes four functionalities in
interactive simulations of planting checksum-based backdoors and defending
against such backdoors. The first functionality is a simple checksum with a
variable modulo value and target precision. The second functionality aims at
activating backdoors in a digital signature verification example [6]. The
third functionality addresses how to plant and trigger (or activate) checksum-
based backdoors in non-linear activation functions. The last functionality
leverages the robustness methods against backdoor attacks based on proximity
analyses. The methods accomplishing the four functionalities are presented
next.

###  2.1 Simple Checksum

While there are many different checksum functions [21], we use the simple
checksum for simplicity during hypothesis testing. Our specific definition of
a simple checksum is provided in Equation 6.

$csum(v)=(\sum_{i=1}^{L}s_{i}+(L_{MAX}-L)\times 48)\;mod\;m$ (1)

where $csum(V)$ is the resulting checksum of a string representation $s$ of a
value $v$, $s_{i}$ is the American Standard Code for Information Interchange
(ASCII) value of the $i$-th character in $s$, $L=\min{(length(s),precision)}$
is the number of characters in $s$ limited by the precision, $L_{MAX}$ is the
maximum number of digits per number, and $m$ is the modulo value. We extended
the primary definition to include a precision, i.e.,
$L=\min{(length(s),precision)}$. Precision limits the number of most
significant digits included in the sum to minimize the impact of numerical
accuracy during double-precision floating point operations. Furthermore, we
extended the basic definition in [21] by padding the string representation
with zeros (ASCII value is 48) to a fixed length so that all numbers
contribute to the $csum$ value with the same number of digits.

Inputs to simulations are double-precision floating point numbers in
JavaScript that follow the international standard [27]. We convert a numerical
input denoted as $v$ to its scientific notation and then to the string
representation according to Equations 2

$v=coefficient\times
10^{exponent};\;s=coefficient.toString()+exponent.toString()$ (2)

where $coefficient$ and $exponent$ include the minus sign as well. Equation 6
is applied to the ASCII string representation of the coefficient and exponent
values computed in Equation 2 with $L_{MAX}^{coefficient}=24$,
$L_{MAX}^{exponent}=4$, $m=256$, and $precision=15$. The $L_{MAX}$ value also
includes a sign. Other configuration options are possible since modular
arithmetic is commutative for addition $(a+b)\;mod\;m=(b+a)\;mod\;m$ and
multiplication $(a\times b)\;mod\;m=(b\times a)\;mod\;m$.

###  2.2 Backdoor in Checksum-Based Digital Signature Verification

The simulation of a backdoor attack on a checksum-based digital signature
verification follows the description by Goldwasser et al. [6]. The
implementation can plant a backdoor either in the output linear layer or the
digital signature verification code. Our approach uses the output linear layer
and is illustrated in Figure 2.

Figure 2: An overview of a neural network with digital signature verification
described by Goldwasser et al. [6]. A backdoor is planted into the output
linear layer to flip an output label for a checksum (CSUM) of the input value
that matches a secret key.

###  2.3 Checksum-Based Backdoor in Activation Functions of Model Nodes

We use all NN nodes to plant a checksum-based backdoor in the rectified linear
unit (ReLU) activation functions and backtrack the necessary minimal changes
to the NN inputs of the first layer that flip the sign of the NN output from
the first layer. The minimal changes to any input $x$ or $y$ coordinate are
defined as “visually” imperceivable location changes or modifications to the
least significant digits of the double-precision floating numbers representing
$x$ and $y$ coordinates.

The guaranteed activation solution is a pair of $x$ and $y$ inputs that
satisfy Equations 3, 4, 7, and 6, where $csum$ stands for a simple checksum
operation and $sk$ stands for a secret key. The coefficients $w_{1j}^{0}$ are
the weights connecting $x$ with nodes indexed by $j$, the coefficients
$w_{2j}^{0}$ are the weights connecting $y$ with nodes indexed by $j$, and the
coefficients $b_{j}^{0}$ are the biases associated with each node $j$. The
superscripts $0$ and $1$ refer to the layers.

$sign(\sum_{j=1}^{4}(w_{j1}^{1}\times TO_{j}^{1}))\neq
sign(\sum_{j=1}^{4}(w_{j1}^{1}\times(-TO_{j}^{1})))$ (3)

$TO_{j}^{1}=\max(0,TI_{j}^{1})\;for\;j=1,2,3,4$ (4)

$TI_{j}^{1}=w_{1j}^{0}\times x+w_{2j}^{0}\times y+b_{j}^{0}\;for\;j=1,2,3,4$
(5)

$csum(TI_{j}^{1})=sk\;for\;j=1,2,3,4$ (6)

Unfortunately, a guaranteed activation solution based on a random search has a
$m^{4}$ computational complexity for the single layer with four nodes. We
confirmed this computational complexity by running the estimation of $x$ and
$y$ one thousand times and counting the number of attempts until the solution
was found. On average, our multiple estimation experiments found the $x$ and
$y$ in $9617.562$ and $9742.891$ attempts for modulo $m=10$. In general, the
random search has a $m^{n_{1}}$ computational complexity, where $n_{1}$ is the
number of nodes in the first layer).

To achieve web-based interactivity, the modulo value $m$ must be adjusted
based on the computer hardware. Based on our benchmarks on a Dell laptop
(Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz), one evaluation of Equations 6 with
a random pair $x$ and $y$ takes on average around $4.83\times 10^{-5}$
seconds. Thus, the computational times for finding the activation values for
$x$ and $y$ are: $m=10\rightarrow 0.48\,s$, $m=20\rightarrow 7.73\,s$, and
$m=30\rightarrow 39.12\,s$. These computational benchmarks suggest that
interactive simulations cannot be delivered for modulo values $m\geq 20$.

To address the interactivity requirement and use any modulo value, the
backdoor activation in the simulation framework is constrained to one node in
the first layer, which might not guarantee a flip of the output label but is
highly likely to succeed. The protocol for backtracking an input that
activates the already planted backdoor is summarized as follows:

  1. 1.

predict output label for a given $(x,y)$ input.

  2. 2.

compute total input $TI_{i}$ to each node $i$ of the first layer and the sum
of outgoing weights $SW_{i}$ of each node according to Equations 7 and 8.

$TI_{i}=\sum_{j=1}^{N_{0}}w_{j,i}\times f_{j}+b_{i}$ (7)
$SW_{i}=\sum_{k=1}^{N_{2}}w_{i,k}$ (8)

where $f_{j}$ is one of the input features, $w_{j,i}$ is the weight between
input feature $f_{j}$ and node $i$, $b_{i}$ is the bias associated with the
node $i$, $N_{0}$ is the number of input features, and $N_{2}$ is the number
of nodes in the second layer. Figure 3 illustrates the computation of Total
Input to a node with the ReLU activation function.

  3. 3.

compute checksums of all $TI_{i}$ values and identify the optimal node
$i_{SEL}$ to be activated which satisfies Equation 9 subject to the inequality
in Equation 10

$i_{SEL}=arg\max_{i}SW_{i}$ (9) $s.t.\;|csum(TI_{i})-sk|<Th$ (10)

where $sk$ is the secret key value of the checksum, and $Th$ is the threshold
for which one can modify digits of a total input value $TI_{i}$ to match the
secret key $sk$.

  4. 4.

modify the digits of $TI_{i}$ iteratively from the least to the most
significant digits within a precision-defined range so that Equation 11 is
satisfied.

$|csum(\widehat{TI_{i_{SEL}}})-sk|=0$ (11)

  5. 5.

reverse computation from $TI_{i_{SEL}}$ to the modified first input feature
$\hat{f_{1}}$ according to Equation 12.

$\hat{f_{1}}=\frac{\widehat{TI_{i_{SEL}}}-(\sum_{j=2}^{N_{0}}w_{j,i_{SEL}}\times
f_{j}+b_{i_{SEL}})}{w_{j=1,i_{SEL}}}$ (12)

  6. 6.

inverse computation from the modified first input feature $\hat{f_{1}}$ to the
$x$ or $y$ coordinate of the input point denoted as $\widehat{(x,y)}$. For
example, one has to inverse any used input feature functions, such as $x^{2}$,
$y^{2}$, $x\times y$, $sin(x)$, $sin(y)$, $sin(x\times y)$,
$sin(x^{2}+y^{2})$, or $0.5\times(x+y)$.

  7. 7.

verify the checksum value of $\widehat{TI_{i_{SEL}}}$ computed from the
modified input coordinates $\widehat{(x,y)}$ being equal to a secret key.

  8. 8.

predict the NN output label using the modified input coordinates
$\widehat{(x,y)}$ and report an attack success if the sign of a predicted
label is opposite to the sign of the originally predicted label in Step 1.

This sequence is applied to a trained NN model and each input point. Depending
on whether the conditions on input values of $(x,y)$ are satisfied, the input
is modified to activate a backdoor in a selected node. $i_{SEL}$. An NN node
with the backdoored ReLU function is illustrated in Figure 4. An adversary can
manipulate the modulo $m$, precision, and secret key values. Similar backdoors
can be used with the Random Fourier Features [28] since the key idea is to
introduce a periodic sign change of the output from a non-linear activation
function.

Figure 3: A schematic of computations in each NN node with rectified linear
unit (ReLU). The total input of a node is computed for two input features, $x$
and $y$, as shown at the top. The NN first layer with four nodes, two inputs,
and the ReLU activation function is shown at the bottom.

Figure 4: An illustration of the impact of a checksum function (top) with the
secret key (red line) on the ReLU activation function (bottom). The arrows
show the locations where the ReLU function will be affected by the checksum
and the output sign will be flipped.

###  2.4 Defense Against Checksum-Based Backdoor

The simulation follows the work by Madry et al. [9] that addresses a general
question: “How can we train deep neural networks that are robust to
adversarial inputs?” The defense approach [9] provides security guarantees
against input perturbations bounded by $l_{\infty}$-bounded attacks. In
general, AI defenses are always limited to a class of attacks (or input
perturbations) and in a constant game with new attacks on data and AI models
[7], [6]. The simulation follows a protocol for defending against input
perturbations as summarized next:

  1. 1.

compute all pair-wise distances between blue points $d_{i,j}^{P}$, orange
points $d_{i,j}^{N}$, and between blue and orange points $d_{i,j}^{NtoP}$.

  2. 2.

compute histograms of pair-wise distances: $\\{d_{i,j}^{P},\Delta
R\\}\longrightarrow\\{h_{k}^{P}\\}$;

  3. 3.

select radius $R$ around an input point to be robust against input
perturbations according to Equation 13, where $k$ refers to the bin of a
histogram of pair-wise distances.

$R=\Delta R*(0.5+arg\max_{k}\frac{h_{k}^{P}\times h_{k}^{N}}{h_{k}^{NtoP}})$
(13)

  4. 4.

count the number of blue and orange training points in the neighborhood of a
test point defined by the radius $R$ and flip the test label if the count of
the opposite label dominates the test point neighborhood.

##  3 Results

This section presents the functionalities of checksum-based backdoors added to
the neural network calculator [19]. We outline simulations of planting
backdoors in digital signature verification and ReLU activation functions, as
well as defending against such backdoors using proximity analysis.

###  3.1 Backdoor in Checksum-Based Digital Signature Verification

By clicking on the “CSUM Signature” button, the simulation computes checksums
of the x-coordinates of test data points and displays the histogram of those
checksums. The test points with checksum values equal to the secret key flip
their label and are shown by selecting the checkbox “Show test data”. Figure 5
shows the histogram and test data for the two classes of the “doughnut” input
pattern. Repeated clicking on the “CSUM Signature” button will revert the
flipped labels since the checksum values associated with each test data point
do not change.

Figure 5: Histogram of checksum values for all test data points (left). Given
a secret key (red arrow), all test points with the checksum equal to the
secret key will flip their labels (13 labels are flipped for the secret key
equal to 113).

###  3.2 Checksum-Based Backdoor in Activation Functions

The attack using the activation functions can be executed in several steps.
After training an NN network with $ReLU$ activation functions, the trained
model is stored in memory using the NN calculator “$NN\;M+$” button. Planting
the checksum-based backdoor is achieved by switching the “Activation” drop-
down menu to $ReLU\\_CSUM$ and reloading the trained model coefficients by
using “$NN\;MR$”. Now, the NN model has weights of the trained clean model and
backdoors in each $ReLU$ activation function associated with each node. The
backdoor in one of the nodes of the first layer is activated by backtracking
input features for a subset of test points.

Table 1 shows numerical results of backtracking for an NN with four nodes in
the first layer and the inputs composed of $x$ and $y$ coordinates. The
backdoor secret key was $sk=150$. For any configuration of planting and
activating backdoors, the corresponding values, as shown in Table 1, are
displayed on the simulation web page below the NN graph.

Table 1: Example of intermediate variables to backtrack the input values that trigger a backdoor in the first layer and flip the output label. The original values are in the left column, and the modified values are in the right column. The changes in the total input $TI$ and in the backtracked $x$ coordinate are $\bm{underlined\;and\;bold}$. $TI=$ $2.96880940\underline{\bm{35902}}424$ |  $\widehat{TI}=$ $2.96880940\underline{\bm{69999}}424$
---|---
$csum(TI)=127$ |  $csum(\hat{TI})=150$
$x=$ $-0.2875996\underline{\bm{7672464454}}$ |  $\hat{x}=$ $-0.2875996\underline{\bm{939029182}}$
$y=$ $-0.08406918324183005$ |  $\hat{y}=$ $-0.08406918324183005$
$output=$ $0.9999999691015791$ |  $\widehat{output}=$ $-0.9999999999934367$
$label=1$ (Blue) |  $\widehat{label}=-1$ (Orange)

###  3.3 Defense Against Checksum-Based Backdoor

The simulation of defense against a checksum-based backdoor is executed in
three steps. First, using the button “CSUM Signature”, a test dataset will be
contaminated with backdoored labels. Next, the pair-wise distances of training
data points and their histograms can be computed using the button “Label
Proximity” shown in Figure 6. For the subset of training points, there are
6328 pairs of orange points, $9316$ pairs of blue points, and $15\,481$ pairs
of orange to blue points, which are used to estimate a recommended radius to
be the middle of the first histogram bin (selected
$R=\frac{\sqrt{2}}{2}=0.707$). Finally, by clicking the “Robust to Backdoor”
button, all test points are inspected based on their histogram of neighborhood
labels, and a label is flipped if an opposite label dominates the neighborhood
to the current one assigned to a test point.

Figure 6: Histograms of the pair-wise distances of training data points
between blue points (top right), orange points (bottom right), and between
orange and blue points (bottom left) for the 2D points shown in the top left.

##  4 Discussion

We discuss additional knowledge gained by simulating checksum-based backdoors
in fully-connected neural networks. The discussion is divided into three major
tasks of interest: planting, activating, and defending against backdoors.

###  4.1 Planting Backdoors

From an attacking viewpoint, making a minimal change to the source code is
critical. Contrary to a variety of attacks on the PyTorch library [24], [25],
some scientists use PyTorch module modifications for solving emerging
problems, for instance, the sustainable AI problem [29], [30]. To address the
sustainable AI problem, the nn.Linear layer is replaced with a BitLinear
module in a BitNet architecture [29], which lowers the cost of training. While
this demonstrates the feasibility of software modifications in the PyTorch
library, the same approach can be misused by adversaries to plant a backdoor.

Another consideration is training an NN with a built-in checksum in the ReLU
activation function. One can simulate such a case by switching the Activation
menu to “ReLU_CSUM” and observing the NN model convergence during training
epochs. The convergence is not stable since the NN model cannot learn the
checksum pattern. We did not modify the derivative part of ReLU since the
checksum has no derivative needed during training. It is possible to replace
the checksum with a Random Fourier Feature (RFF) and its derivative as
documented by Goldwasser et al. [6]. While the implementation is for future
work, one can currently simulate mathematically-related scenarios with input
features being $sin(x)$, $sin(y)$, $sin(x\times y)$, or $sin(x^{2}+y^{2})$.

One could envision two workflows for planting backdoors. The first workflow
$WF_{1}$ consists of training a clean NN model, planting a backdoor in the NN
model source code, and disseminating the NN model with a backdoor in the
source code. The second workflow $WF_{2}$ is composed of planting a backdoor
in the NN model source code, training the NN model with the backdoor, and
distributing a clean NN model with weights obtained by training the NN with
the backdoor. The backdoor activation succeeds in $WF_{1}$ but does not
succeed in $WF_{2}$ because the total inputs to NN nodes have a random pattern
of checksums over training epochs, and, hence, the NN model does not learn
enough about checksums to be used as triggers. More analyses are needed to
understand the training patterns of checksums fully.

###  4.2 Activation of Backdoors

One possible adversarial approach is to collect checksums of total inputs $TI$
at a subset of NN nodes along the path to the output linear layer for selected
test input (a set of features) and then form a vector or secret keys
accordingly to control flips of output at any or all of the NN nodes. This is
also a topic of future research and one part of simulations.

Additional discussion is relevant to the numerical precision, capacity of
total inputs for modifications, and imperceptible modifications by humans and
machines. As shown in Table 1, the modification to $TI$ are smaller than
$10^{-8}$ and larger than $10^{-14}$. These small changes can trigger a
backdoor that is (to some degree) robust to machine precision and
imperceptible to humans. However, quantization and half-precision hardware can
minimize the capacity of total inputs for adversarial modifications (i.e., the
number of available digits for modifications). In addition, for complex input
features, for instance, $sin(x\times y)$ or $sin(x^{2}+y^{2})$, retrieving $x$
or $y$ values using inverse functions might introduce numerical variability of
the least significant digits.

###  4.3 Defense Against Backdoors

The underlying assumption of the defense is that the training data classes are
well-separated by a continuous boundary function in high-dimensional feature
spaces. Using mathematical terminology, the function representing the class
boundaries must be a Lipschitz continuous function [31], which limits sudden
changes of labels in a feature space. We included in the simulation framework
examples of 2D dot patterns that, together with noise level adjustments via
the “Noise” sliding bar and a variety of data poisoning strategies invoked by
the “Trojan” sliding bar, will violate the defense assumption. Figure 7
illustrates four such examples of 2D dot patterns. For these patterns, local
neighborhoods around labeled points must have a priori unknown shape and
orientation (e.g., a spiral shape), satisfy a priori unknown property (e.g.,
constant pair-wise distances), or be accompanied by a priori unknown class
centroids and spatial densities (e.g., classes formed by multiple disconnected
sets of points).

Figure 7: Examples of 2D dot patterns that violate the defense assumptions of
input perturbations bounded by $l_{\infty}$-bounded attacks. Two classes are
forming geometrical shapes of spiral and interleaved grid patterns (top) and
being noisy and spatially overlapping (bottom). All patterns pose a challenge
to defending against input perturbations.

##  5 Conclusion

We have presented a design of an interactive online simulation framework for
planting, activating, and defending checksum-based backdoors in fully-
connected NN models. The simulations are available at
https://pages.nist.gov/nn-calculator

The main benefits of the designed simulation framework lie in the
interactivity and functionalities that become a playground for improving our
understanding (and the implications of using) cryptographic techniques for
planting and defending architectural backdoors in the large AI model systems
deployed in practice. The simulations are built on the neural network
calculator to operate with NN models and input datasets like with numbers in a
math calculator.

While the simulations assume that adversaries have control over the source
code of a model, the framework allows exploring scenarios with planting
backdoors only during training, propagation of backdoored first layers to
other layers, and defenses against a variety of complex 2D dot input patterns.
These topics are subjects of future study.

## Acknowledgement

The funding for Peter Bajcsy was provided by the Intelligence Advanced
Research Projects Activity (IARPA): IARPA-20001-D2020-2007180011.

## Disclaimer

Commercial products are identified in this document to specify the
experimental procedure adequately. Such identification is not intended to
imply recommendation or endorsement by the National Institute of Standards and
Technology, nor is it intended to imply that the products identified are
necessarily the best available for the purpose.

## References

  * [1] Apostol Vassilev, Alina Oprea, Alie Fordyce, and Hyrum Andersen.  Adversarial machine learning: A taxonomy and terminology of attacks and mitigations.  NIST Internal Report, DOI: 10.6028/NIST.AI.100-2e2023, January 2024.
  * [2] Harry Langford, Ilia Shumailov, Yiren Zhao, Robert Mullins, and Nicolas Papernot.  Architectural neural backdoors from first principles.  arXiv: 2402.06957, 2024.
  * [3] Mikel Bober-Irizar, Ilia Shumailov, Yiren Zhao, Robert Mullins, and Nicolas Papernot.  Architectural backdoors in neural networks.  arXiv: 2206.07840, 2022.
  * [4] Padmaksha Roy, Jaganmohan Chandrasekaran, Erin Lanus, Laura Freeman, and Jeremy Werner.  A survey of data security: Practices from cybersecurity and challenges of machine learning.  arXiv: 2310.04513, 2023.
  * [5] Macq B and Quisquater JJ.  Cryptography for trusted artificial intelligence in medicine.  American Journal of Biomedical Science and Research, 13(2), June 2021.
  * [6] Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, and Or Zamir.  Planting undetectable backdoors in machine learning models.  arXiv:2204.06974, 2022.
  * [7] Dimitar I. Dimitrov, Gagandeep Singh, Timon Gehr, and Martin Vechev.  Provably robust adversarial examples.  arXiv:2007.12133, 2022.
  * [8] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  arXiv:1608.04644, 2017.
  * [9] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv: 1706.06083, 2019.
  * [10] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang.  Certified defenses against adversarial examples.  arXiv:1801.09344, 2020.
  * [11] IARPA.  Trojans in Artificial Intelligence (TrojAI), 2020.  https://www.iarpa.gov/index.php/research-programs/trojai.
  * [12] Mantas Mazeika, Dan Hendrycks, Huichen Li, Xiaojun Xu, Sidney Hough, Andy Zou, Arezoo Rajabi, Qi Yao, Zihao Wang, Jian Tian, Yao Tang, Di Tang, Roman Smirnov, Pavel Pleskov, Nikita Benkovich, Dawn Song, Radha Poovendran, Bo Li, and David Forsyth.  The trojan detection challenge.  Proceedings of Machine Learning Research, 220:279–291, 2023.
  * [13] Dinuka Sahabandu, Xiaojun Xu, Arezoo Rajabi, Luyao Niu, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran.  Game of trojans: Adaptive adversaries against output-based trojaned-model detectors.  arXiv:2402.08695, 2024.
  * [14] Min Wu, Matthew Wicker, Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska.  A game-based approximate verification of deep neural networks with provable guarantees.  Theoretical Computer Science, 807:298–329, February 2020.
  * [15] Nicholas Carlini.  A llm assisted exploitation of ai-guardian.  arXiv:2307.15008, 2023.
  * [16] Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song.  TABOR: a highly accurate approach to inspecting and restoring trojan backdoors in ai systems.  arXiv:1908.01763, 2019.
  * [17] Hong Zhu, Shengzhi Zhang, and Kai Chen.  AI-Guardian: defeating adversarial attacks using backdoors.  In 2023 IEEE Symposium on Security and Privacy (SP), pages 701–718, 2023.
  * [18] Minzhou Pan, Yi Zeng, Lingjuan Lyu, Xue Lin, and Ruoxi Jia.  Asset: Robust backdoor data detection across a multiplicity of deep learning paradigms.  arXiv:2302.11408, 2023.
  * [19] Peter Bajcsy, Nicholas J. Schaub, and Michael Majurski.  Designing trojan detectors in neural networks using interactive simulations.  MDPI Applied Sciences, 11(4), 2021.
  * [20] Daniel Smilkov, Shan Carter, D. Sculley, Fernanda B. Viégas, and Martin Wattenberg.  Direct-Manipulation Visualization of Deep Networks.  arXiv:1708.03788, pages 1–5, 2017.
  * [21] GeeksforGeeks.  Checksum algorithms in javascript.  https://www.geeksforgeeks.org/checksum-algorithms-in-javascript/, November 2023.
  * [22] Hua Ma, Huming Qiu, Yansong Gao, Zhi Zhang, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Zhang Jiliang, Said Al-Sarawi, and Derek Abbott.  Quantization backdoors to deep learning commercial frameworks.  arXiv:2108.09187, 2023.
  * [23] Eugene Bagdasaryan and Vitaly Shmatikov.  Blind backdoors in deep learning models.  In 30th USENIX Security Symposium (USENIX Security 21), pages 1505–1521. USENIX Association, August 2021.
  * [24] PyTorch.  Compromised pytorch-nightly dependency chain between december 25th and december 30th, 2022.  https://pytorch.org/blog/compromised-nightly-dependency/, December 2022\.  Accessed: 2024-04-10.
  * [25] PyTorch.  Playing with fire – how we executed a critical supply chain attack on pytorch.  https://johnstawinski.com/2024/01/11/playing-with-fire-how-we-executed-a-critical-supply-chain-attack-on-pytorch/, January 2024.  Accessed: 2024-04-10.
  * [26] Nadav Noy.  Legit discovers "ai jacking" vulnerability in popular hugging face ai platform.  https://www.legitsecurity.com/blog/tens-of-thousands-of-developers-were-potentially-impacted-by-the-hugging-face-aijacking-attack, October 2023.  Accessed: 2024-04-10.
  * [27] Standard.  IEEE standard for floating-point arithmetic.  IEEE Computer Society STD 754-2019, 2019.
  * [28] Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  In Neural Information Processing Systems, CorpusID: 877929, December 2007.
  * [29] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei.  Bitnet: Scaling 1-bit transformers for large language models.  arXiv:2310.11453, 2023.
  * [30] Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei.  The era of 1-bit llms: All large language models are in 1.58 bits.  arXiv:2402.17764, 2024.
  * [31] Mícheál Ó Searcóid.  Uniform Continuity, pages 147–163.  Springer London, London, 2007.